In the multi-armed bandit problem, the decision-maker must choose each period a single project to work on. From the chosen project she receives an immediate reward that depends on the current state of the project. Next period the chosen project makes a stochastic transition to a new state, but projects that are not chosen remain in the same state. What happens in a two-armed bandit context if projects not chosen do not remain in the same state? We derive two sufficient conditions for the optimal policy to be myopic: either the transition function for chosen projects has in a certain sense uniformly stronger stochastic dominance than the transition function for unchosen projects, or both transition processes are normal martingales, the variance of which is independent of the history of process choices.

We extend large exceedence results for i.i.d. -valued random variables to a class of uniformly recurrent Markov-additive processes and stationary strong-mixing processes. As in the i.i.d. case, the results are proved via large deviations estimates.

Well-known inequalities for the spectral gap of a discrete-time Markov chain, such as Poincaré's and Cheeger's inequalities, do not perform well if the transition graph of the Markov chain is strongly connected. For example in the case of nearest-neighbour random walk on the n-dimensional cube Poincaré's and Cheeger's inequalities are off by a factor n. Using a coupling technique and a contraction principle lower bounds on the spectral gap can be derived. Finally, we show that using the contraction principle yields a sharp estimate for nearest-neighbour random walk on the n-dimensional cube.

The application of the generalised ballot theorem to queueing theory leads to elegant results for the simple M/G/1 queue. It is thought that such results are not possible for more general M/G/1-type queues. We, however, derive a batch ballot theorem which can be applied to derive the first passage distribution matrix, G, for the general M/G/1-type queue.

We consider the problem of stochastic comparison of general GARCH-like processes for different parameters and different distributions of the innovations. We identify several stochastic orders that are propagated from the innovations to the GARCH process itself, and we discuss their interpretations. We focus on the convex order and show that in the case of symmetric innovations it is also propagated to the cumulated sums of the GARCH process. More generally, we discuss multivariate comparison results related to the multivariate convex and supermodular orders. Finally, we discuss ordering with respect to the parameters in the GARCH(1, 1) case.

An optimal coupling is a bivariate distribution with specified marginals achieving maximal correlation. We show that optimal couplings are totally positive and, in fact, satisfy a strictly stronger condition we call the nonintersection property. For discrete distributions we illustrate the equivalence between optimal coupling and a certain transportation problem. Specifically, the optimal solutions of greedily-solvable transportation problems are totally positive, and even nonintersecting, through a rearrangement of matrix entries that results in a Monge sequence. In coupling continuous random variables or random vectors, we exploit a characterization of optimal couplings in terms of subgradients of a closed convex function to establish a generalization of the nonintersection property. We argue that nonintersection is not only stronger than total positivity, it is the more natural concept for the singular distributions that arise in coupling continuous random variables.

A basic process of simple queueing theory is S(t), an integer valued stochastic process which represents the “number of clients present, including the one in service” at epoch t. The queueing context physically limits S(t) to non-negative values, but if this impenetrable barrier is removed thereby permitting S(t) to assume negative values as well, we have a “randomized random walk” in which S(t) represents the position at time t of a mythical particle which moves on the x-axis according to the rules of the walk. The nomenclature “randomized random walk” is due to Feller (1966a). S(t) changes by unit positive or negative amounts, and our basic assumption is that the time intervals τ/σ separating successive positive/negative steps are i.i.d. with continuous d.f.'s. A(t)/B(t) (A(0)/B(0) = 0) possessing p.d.f.'s. a(t)/b(t): τ and σ are further assumed to be statistically independent. When A(t) = 1 – e–λt and B(t) = 1 – e–μt, we have a generalization of the M/M/1 queueing process which we shall denote by M/M. The walk generated by A(t) = 1 – e–λt and B(t) arbitrary may similarly be denoted by M/G. For G/M we clearly mean A(t) arbitrary and B(t) = 1 – e–μt. M/M has received extensive treatment elsewhere (cf. Feller (1966a), (1966b); Takács (1967); Gibson (1968); Conolly (1971)), and will not be considered here. Our interest centers on certain aspects of M/G and G/M, but since each is the dual of the other (loosely, G/M is M/G “upside down”) it is sufficient to restrict attention to one of them, as convenient; pointing out, where necessary, the interpretation in terms of the other.

An expression for the first-passage density of Brownian motion to a curved boundary due to Daniels and Lerche is shown to give the same result as a different form due to the author. The equivalence is extended to continuous Gaussian Markov processes.

Some consequences of a modified repair system for Phillips' (1981a, b) model for a two-component system are discussed. In the original model, both components are repaired whenever a revealed fault occurs; in the modified model only faulty components are repaired. Specifically (i) the distribution of time from the initial state up to discovery of an unrevealed fault, (ii) the expected proportion of time during which there exists an unrepaired fault, and (iii) the distribution of number of revealed faults up to and including the one which leads to a discovery of an unrevealed fault, are obtained. The theory is illustrated by examples, based on specific distributions for the times between repairs and occurrences of the two types of faults. A characterization of the exponential distribution is indicated.

In this paper we consider maximum likelihood estimation of the rate constant for stochastic rth-order reactions based on observation of the level of the system at time t > 0. Conditions are found for which the waiting time until the nth. reaction event is asymptotically normal, as the initial number of molecules and the number of reaction events become large. This distributional result is used to derive an approximate estimator which is shown for the second-order case to be close to the exact maximum likelihood estimate over a wide range of percentage completion.

Present developments in computer performance evaluation require detailed analysis of N-dimensional random walks on the set of lattice points in the first 2N-ant of Recent research has shown that for the two-dimensional case the inherent mathematical problem can often be formulated as a boundary value problem of the Riemann–Hilbert type. The paper is concerned with a derivation and analysis of an identity for the first entrance times distributions into the boundary of such random walks. The identity formulates a relation between these distributions and the zero-tuples of the kernel of the random walk; the kernel contains all the information concerning the structure of the random walk in the interior of its stage space. For the two-dimensional case the identity is resolved and explicit expressions for the entrance times distributions are obtained.

A summation formula is given for computing the spectrum of intervals of a geometric branching Poisson process from the cumulant generating function of the associated counting process.

This paper presents a degree-two probability lower bound for the union of an arbitrary set of events in an arbitrary probability space. The bound is designed in terms of the first-degree Bonferroni summation and pairwise joint probabilities of events, which are represented as weights of edges in a Hamilton-type circuit in a connected graph. The proposed lower bound strengthens the Dawson–Sankoff lower bound in the same way that Hunter and Worsley's degree-two upper bound improves the degree-two Bonferroni-type optimal upper bound. It can be applied to statistical inference in time series and outlier diagnoses as well as the study of dose response curves.

In this paper we present a method for obtaining explicit results for some two-sided boundary problems involving sums of independent random variables with alternating distributions. We apply the method to finding the first passage time to either one of two finite barriers, and to some situations arising in queueing and dam theory. The results can be expressed in terms of a finite sum of simple repeated integrals (or sums) of known functions (cf. formulae (3.6)– (3.11)).

Let (Xt) and (Yt) be continuous-time Markov chains with countable state spaces E and F and let K be an arbitrary subset of E x F. We give necessary and sufficient conditions on the transition rates of (Xt) and (Yt) for the existence of a coupling which stays in K. We also show that when such a coupling exists, it can be chosen to be Markovian and give a way to construct it. In the case E=F and K ⊆ E x E, we see how the problem of construction of the coupling can be simplified. We give some examples of use and application of our results, including a new concept of lumpability in Markov chains.

In this paper we analyze players' long-run behavior in evolutionary coordination games with imperfect monitoring in a large population. Players can observe signals corresponding to other players' unseen actions and use the proposed simple or maximum likelihood estimation algorithm to extract information from the signals. In the simple learning process we find conditions for the risk-dominant and the non-risk-dominant equilibria to emerge alone in the long run. Furthermore, we find that the two equilibria can coexist in the long run. In contrast, the coexistence of the two equilibria is the only limit distribution under the maximum likelihood estimation learning algorithm. We also analyze the long-run equilibria of other 2x2 symmetric games under imperfect monitoring.

It is shown how to estimate centroids and other measures of location of an unknown embedded subset of a bounded specimen by random sectioning.

Consider a number of events in a probability space. Let X be a random variable that is the number of events that occur. Given some of the moments of the distribution of X, it is possible to obtain bounds on the probability that at least one event occurs. The best possible bounds are given here. If there are many equiprobable events that are d- wise independent, and d is even, then the probability that at least one event happens is at least 1 — O(µ–d/2), where μ = E(X).

The asymptotic behaviour of many locally branching epidemic models can, at least to first order, be deduced from the limit theory of two branching processes. The first is Whittle's (1955) branching approximation to the early stages of the epidemic, the phase in which approximately exponential growth takes place. The second is the susceptibility approximation; the backward branching process that approximates the history of the contacts that would lead to an individual becoming infected. The simplest coupling arguments for demonstrating the closeness of these branching process approximations do not keep the processes identical for quite long enough. Thus, arguments showing that the differences are unimportant are also needed. In this paper we show that, for some models, couplings can be constructed that are sufficiently accurate for this extra step to be dispensed with.

Birth–death processes form a natural class where ideas and results on large deviations can be tested. We derive a large-deviation principle under an assumption that the rate of jump down (death) grows asymptotically linearly with the population size, while the rate of jump up (birth) grows sublinearly. We establish a large-deviation principle under various forms of scaling of the underlying process and the corresponding normalization of the logarithm of the large-deviation probabilities. The results show interesting features of dependence of the rate functional upon the parameters of the process and the forms of scaling and normalization.