Here we introduce two simple models: simple cubic random packing and random packing by Hamming distance. Consider the packing density γ d of dimension d by cubic random packing. From computer simulations up to dimension 11, γ d+1/γ d seems to approach 1. Also, we give simulation results for random packing by Hamming distance and discuss the behavior of packing density when dimensionality is increased. For the case of Hamming distances of 2 or 3, d–α fits the simulation results of packing density where α is an empirical constant. The variance of packing density is larger when k is even and smaller when k is odd, where k represents Hamming distance.

In this paper, we study the new better than used in expectation (NBUE) and new worse than used in expectation (NWUE) properties of Markov renewal processes. We show that a Markov renewal process belongs to a more general class of stochastic processes encountered in reliability or maintenance applications. We present sufficient conditions such that the first-passage times of these processes are new better than used in expectation. The results are applied to the study of shock and repair models, random repair time processes, inventory, and queueing models.

This paper considers the best-choice problem with a random number of objects having a known distribution. The optimality equation of the problem reduces to an integral equation by a scaling limit. The equation is explicitly solved under conditions on the distribution, which relate to the condition for an OLA policy to be optimal in Markov decision processes. This technique is then applied to three different versions of the problem and an exact value for the asymptotic optimal strategy is found.

We examine the existence of limiting behavior, or stability, for storage models with shot noise input and general release rules. The shot noise feature of the input process allows the individual inputs to gradually enter the store.

We first show that a store under the unit release rule is stable if and only if the traffic intensity is less than one; this extends the classic result of Prabhu (1980) to the case of shot noise input. The stability of the unit release rule store and various stochastic orderings are then used to derive a sufficient condition for a store with a general release rule to be stable. Finally, we show that when restricted to a compact state space, our storage model is always stable.

An important component of the paper is the methodology employed: coupling and stochastic monotonicity play a key role in analyzing the non-Markov processes encountered.

This study is concerned with finite Markov decision processes whose dynamics and reward structure are unknown but the state is observable exactly.

We establish a learning algorithm which yields an optimal policy and construct an adaptive policy which is optimal under the average expected reward criterion.

This paper is concerned with a discrete-time population model in which a new individual entering the population at time t can produce offspring for the first time at time t + 2 and then subsequently at times t + 3, t + 4, ···. The numbers of offspring produced on each occasion are independent random variables each with the distribution of Z for which EZ = m <∞, and individuals have independent lines of descent. This model is contrasted with the corresponding Bienaymé-Galton-Watson one. If Xn denotes the number of individuals in the population at time n, it is shown that z–nXn almost surely converges to a random variable W, as n→∞, where Various properties of W are obtained, in particular W > 0 a.s. if and only if EZ | log Z | < ∞ Results are also given on the rate of convergence of to z when Var Z < ∞ and these display a surprising dependence on the size of z.

For a sequence ξ1, ξ2, · ·· of i.i.d. random variables let X0 = 0 and Xk = max(Xk–1 + ξ k, 0) for k = 1, 2, ···. Let . These stopping times are used in Page's (1954) one-sided cusum procedures and are also important in queueing theory. Various asymptotic properties of Nx are derived.

Correction terms for the diffusion approximation to the maximum and ruin probabilities for a random walk with small negative drift, obtained by Siegmund (1979) in the exponential family case, are extended by different methods to some non-exponential family cases.

Consider a sequence of Bernoulli trials between players A and B in which player A wins each trial with probability p∈ [0, 1]. For positive integers n and k with k ≦ n, an (n, k) contest is one in which the first player to win at least n trials and to be ahead of his opponent by at least k trials wins the contest. The (n, 1) contest is the Banach match problem and the (n, n) contest is the gambler's ruin problem. Many real contests (such as the World Series in baseball and the tennis game) have an (n, 1) or an (n, 2) format. The (n, k) contest is formulated in terms of the first-exit time of the graph of a random walk from a certain region of the state-time space. Explicit results are obtained for the probability that player A wins an (n, k) contest and the expected number of trials in an (n, k) contest. Comparisons of (n, k) contests are made in terms of the probability that the stronger player wins and the expected number of trials.

As an extension of the discrete-time case, this note investigates the variance of the total cumulative reward for continuous-time Markov reward chains with finite state spaces. The results correspond to discrete-time results. In particular, the variance growth rate is shown to be asymptotically linear in time. Expressions are provided to compute this growth rate.

In this paper, we discuss the problem of the pricing of American-style options in the exponential Lévy security market model. This model is typically incomplete, and we derive the explicit bounds of the interval of no arbitrage prices and the related optimal stopping moments for American put options and American call options in both finite and infinite horizon time. We consider a large class of Lévy processes.

An urn contains black and red balls. Let Z n be the proportion of black balls at time n and 0≤L<U≤1 random barriers. At each time n, a ball b n is drawn. If b n is black and Z n-1<U, then b n is replaced together with a random number B n of black balls. If b n is red and Z n-1>L, then b n is replaced together with a random number R n of red balls. Otherwise, no additional balls are added, and b n alone is replaced. In this paper we assume that R n =B n . Then, under mild conditions, it is shown that Z n →a.s. Z for some random variable Z, and D n ≔√n(Z n -Z)→𝒩(0,σ2) conditionally almost surely (a.s.), where σ2 is a certain random variance. Almost sure conditional convergence means that ℙ(D n ∈⋅|𝒢n )→w 𝒩(0,σ2) a.s., where ℙ(D n ∈⋅|𝒢n ) is a regular version of the conditional distribution of D n given the past 𝒢n . Thus, in particular, one obtains D n →𝒩(0,σ2) stably. It is also shown that L<Z<U a.s. and Z has nonatomic distribution.

Existing results on the asymptotic behaviour of solutions to two dynamic programming equations involving non-negative matrices are reviewed and strengthened in certain directions. The results are then applied to strategies for harvesting of a small population so as to optimise its survival potential in a limited environment.

An adaptive-type exponential smoothing, motivated by an insurance tariff problem, is treated. We consider the process Zn = ß(Zn –1)Xn +(1 – ß (Zn–1))Zn–1, where Xn are i.i.d. taking values in the interval [0, M], M ≦ ∞ and ß is a monotonically increasing function [0, M] → [c, d], 0 < c < d < 1.

Together with (Zn), we consider the ordinary exponential smoothing Yn = αXn + (1 – α)Yn –1 where α is a constant, 0 < α < 1. We show that (Yn) and (Zn) are geometrically ergodic Markov chains (in the case of finite interval we even have uniform ergodicity) and that EYn, EZn converge to limits EY, EZ, respectively, with a geometric convergence rate. Moreover, we show that Ez is strictly less than EY = EXn.

This paper considers a semi-infinite storage model, of the type studied by Senturia and Puri [13] and Balagopal [2], defined on a Markov renewal process, {(Xn, Tn), n = 0, 1, ·· ·}, with 0 = T0 < T1 < · ··, almost surely, where Xn takes values in the set {1, 2, ·· ·}. If at Tn, Xn = j, then there is a random ‘input' Vn (j) (a negative input implying a demand) of ‘type' j, having distribution function Fj(·). We assume that {Vn (j)} is an i.i.d. sequence of random variables, taken to be independent of {(Xn, Tn)} and of {Vn (k)}, for k ≠ j, and that Vn (j) has first and second moments. Here the random variables Vn (j) represent instantaneous ‘inputs' (a negative value implying a demand) of type j for our storage model. Under these assumptions, we establish certain limit distributions for the joint process (Z(t), L(t)), where Z(t) (defined in (2)) is the level of storage at time t and L(t) (defined in (3)) is the demand lost due to shortage of supply during [0, t]. Different limit distributions are obtained for the cases when the ‘average stationary input' ρ, as defined in (5), is positive, zero or negative.

The problem of determining the asymptotic form of the stock vector n(t) in a continuous time Markovian manpower model is solved for asymptotically exponential recruitment functions {R(t)}. A new approach to the limiting behaviour of some manpower systems with given total sizes {N(t)} is then given by means of time-inhomogeneous Markov processes.

Inspired by a PDE–ODE system of aggregation developed in the biomathematical literature, we investigate an interacting particle system representing aggregation at the level of individuals. We prove that the empirical density of the individual converges to the solution of the PDE–ODE system.

Motivated by a problem in neural encoding, we introduce an adaptive (or real-time) parameter estimation algorithm driven by a counting process. Despite the long history of adaptive algorithms, this kind of algorithm is relatively new. We develop a finite-time averaging analysis which is nonstandard partly because of the point process setting and partly because we have sought to avoid requiring mixing conditions. This is significant since mixing conditions often place restrictive history-dependent requirements on algorithm convergence.

The study of simple solar energy storage models leads to the question of analyzing the equilibrium distribution of Markov chains (Harris chains), for which the state at epoch (n + 1) (i.e. the temperature of the storage tank) depends on the state at epoch n and on a controlled input, acceptance of which entails a further decrease of the temperature level. Here we study the model where the input is exponentially distributed. For all values of the parameters involved an explicit expression for the equilibrium distribution of the Markov chain is derived, and from this we calculate, as one of the possible applications, the exact values of the mean of this equilibrium distribution.

Let {W(t), 0 ≦ t < ∞} be the standard Wiener process. The probabilities of the type P[sup0≦t ≦ TW(t) − f(t) ≧ 0] have been extensively studied when f(t) is a deterministic function. This paper discusses the probabilities of the type P{sup0≦t ≦ TW(t) − [f(t) + X(t)] ≧ 0} when X(t) is a stochastic process. By taking compound Poisson processes as X(t), the paper gives procedures for finding such probabilities.