IN this chapter we consider several fundamental optimization problems involving graphs: all-pairs shortest paths, minimum cuts, and minimum spanning trees. In each case, deterministic polynomial time algorithms are known, but the use of randomization allows us to obtain significantly faster solutions for these problems. We show that the problem of computing all-pairs shortest paths is reducible, via a randomized reduction, to the problem of multiplying two integer matrices. We present a fast randomized algorithm for the min-cut problem in undirected graphs, thereby providing evidence that this problem may be easier than the max-flow problem. Finally, we present a linear-time randomized algorithm for the problem of finding minimum spanning trees.

Unless stated otherwise, all the graphs we consider are assumed to be undirected and without multiple edges or self-loops. For shortest paths and min-cuts we restrict our attention to unweighted graphs, although in some cases the results generalize to weighted graphs; we give references in the discussion at the end of the chapter.

All-pairs Shortest Paths

Let G(V,E) be an undirected, connected graph with V = ﹛l,…,n﹜ and \E\ = m. The adjacency matrix A is an n x n 0-1 matrix with Aij = Aji = 1 if and only if the edge (i, j) is present in E. Given A, we define the distance matrix D as an n x n matrix with non-negative integer entries such that Dij equals the length of a shortest path from vertex i to vertex j.

THE last decade has witnessed a tremendous growth in the area of randomized algorithms. During this period, randomized algorithms went from being a tool in computational number theory to finding widespread application in many types of algorithms. Two benefits of randomization have spearheaded this growth: simplicity and speed. For many applications, a randomized algorithm is the simplest algorithm available, or the fastest, or both.

This book presents the basic concepts in the design and analysis of randomized algorithms at a level accessible to advanced undergraduates and to graduate students. We expect it will also prove to be a reference to professionals wishing to implement such algorithms and to researchers seeking to establish new results in the area.

Organization and Course Information

We assume that the reader has had undergraduate courses in Algorithms and Complexity, and in Probability Theory. The book is organized into two parts. The first part, consisting of seven chapters, presents basic tools from probability theory and probabilistic analysis that are recurrent in algorithmic applications. Applications are given along with each tool to illustrate the tool in concrete settings. The second part of the book also contains seven chapters, each focusing on one area of application of randomized algorithms. The seven areas of application we have selected are: data structures, graph algorithms, geometric algorithms, number theoretic algorithms, counting algorithms, parallel and distributed algorithms, and online algorithms. Naturally, some of the algorithms used for illustration in Part I do fall into one of these seven categories.

SOME of the most notable results in theoretical computer science, particularly in complexity theory, have involved a non-trivial use of algebraic techniques combined with randomization. In this chapter we describe some basic randomization techniques with an underlying algebraic flavor. We begin by describing Freivalds’ technique for the verification of identities involving matrices, polynomials, and integers. We describe how this generalizes to the Schwartz-Zippel technique for identities involving multivariate polynomials, and we illustrate this technique by applying it to the problem of detecting the existence of perfect matchings in graphs. Then we present a related technique that leads to an efficient randomized algorithm for pattern matching in strings. We conclude with some complexity-theoretic applications of the techniques introduced here. In particular, we define interactive proof systems and demonstrate such systems for the graph non-isomorphism problem and the problem of counting the number of satisfying truth assignments for a Boolean formula. We then refine this concept into that of an efficiently verifiable proof and demonstrate such proofs for the satisfiability problem. We indicate how these concepts have led to a completely different view of classical complexity classes, as well as the new results obtained via the resulting insight into the structure of these classes.

Most of these techniques and their applications involve (sometimes indirectly) a fingerprinting mechanism, which can be described as follows. Consider the problem of deciding the equality of two elements x and y drawn from a large universe U.

IN this chapter we apply randomization to hard counting problems. After defining the class #P, we present several #P-complete problems. We present a (randomized) polynomial time approximation scheme for the problem of counting the number of satisfying truth assignments for a DNF formula. The problem of approximate counting of perfect matchings in a bipartite graph is shown to be reducible to that of the uniform generation of perfect matchings. We describe a solution to the latter problem using the rapid mixing property of a suitably defined random walk, provided the input graph is sufficiently dense. We conclude with an overview of the estimation of the volume of a convex body.

We say that a decision problem Π is in NP if for any YES-instance I of Π, there exists a proof that I is a YES-instance that can be verified in polynomial time. Equivalently, we can cast the decision problem as a language recognition problem, where the language consists of suitable encodings of all YES-instances of Π. A proof now certifies the membership in the language of an encoded instance of the problem. Usually the proof of membership corresponds to a "solution” to the search version of the decision problem II: for instance, if II were the problem of deciding whether a given graph is Hamiltonian, a possible proof of this for a Hamiltonian graph (YES-instance) would be a Hamiltonian cycle in the graph. In the counting version of this problem, we wish to compute the number of proofs that an instance / is a YES-instance.

The theory of numbers plays a central role in several areas of great importance to computer science, such as cryptography, pseudo-random number generation, complexity theory, algebraic problems, coding theory, and combinatorics, to name just a few. We have already seen that relatively simple properties of prime numbers allow us to devise k-wise independent variables (Chapter 3), and number-theoretic ideas are at the heart of the algebraic techniques in randomization discussed in Chapter 7.

In this chapter, we focus on solving number-theoretic problems using randomized techniques. Since the structure of finite fields depends on the properties of prime numbers, algebraic problems involving polynomials over such fields are also treated in this chapter. We start with a review of some basic concepts in number theory and algebra. Then we develop a variety of randomized algorithms, most notably for the problems of computing square roots, solving polynomial equations, and testing primality. Connections with other areas, such as cryptography and complexity theory, are also pointed out along the way.

There are several unique features in the use of randomization in number theory. As will soon become clear, the use of randomization is fairly simple in that most of the algorithms will start by picking a random number from some domain and then work deterministically from there on. We will claim that with high probability the chosen random number has some desirable property. The hard part usually will be establishing this claim, which will require us to use non-trivial ideas from number theory and algebra.

IN this chapter we study several ideas that are basic to the design and analysis of randomized algorithms. All the topics in this chapter share a game-theoretic viewpoint, which enables us to think of a randomized algorithm as a probability distribution on deterministic algorithms. This leads to the Yao ‘s Minimax Principle, which can be used to establish a lower bound on the performance of a randomized algorithm.

Game Tree Evaluation

We begin with another simple illustration of linearity of expectation, in the setting of game tree evaluation. This example will demonstrate a randomized algorithm whose expected running time is smaller than that of any deterministic algorithm. It will also serve as a vehicle for demonstrating a standard technique for deriving a lower bound on the running time of any randomized algorithm for a problem.

A game tree is a rooted tree in which internal nodes at even distance from the root are labeled MIN and internal nodes at odd distance are labeled MAX. Associated with each leaf is a real number, which we call its value. The evaluation of the game tree is the following process. Each leaf returns the value associated with it. Each MAX node returns the largest value returned by its children, and each MIN node returns the smallest value returned by its children. Given a tree with values at the leaves, the evaluation problem is to determine the value returned by the root.

IN Chapters 1 and 2, we bounded the expected running times of several randomized algorithms. While the expectation of a random variable (such as a running time) may be small, it may frequently assume values that are far higher. In analyzing the performance of a randomized algorithm, we often like to show that the behavior of the algorithm is good almost all the time. For example, it is more desirable to show that the running time is small with high probability, not just that it has a small expectation. In this chapter we will begin the study of general methods for proving statements of this type. We will begin by examining a family of stochastic processes that is fundamental to the analysis of many randomized algorithms: these are called occupancy problems. This motivates the study (in this chapter and the next) of general bounds on the probability that a random variable deviates far from its expectation, enabling us to avoid such custom-made analyses. The probability that a random variable deviates by a given amount from its expectation is referred to as a tail probability for that deviation. Readers wishing to review basic material on probability and distributions may consult Appendix C.

Occupancy Problems

We begin with an example of an occupancy problem. In such problems we envision each of m indistinguishable objects ("balls") being randomly assigned to one of n distinct classes ("bins"). In other words, each ball is placed in a bin chosen independently and uniformly at random.

IN this chapter we will study some basic principles of the probabilistic method, a combinatorial tool with many applications in computer science. This method is a powerful tool for demonstrating the existence of combinatorial objects. We introduce the basic idea through several examples drawn from earlier chapters, and follow that by a detailed study of the maximum satisfiability (MAX-SAT) problem. We then introduce the notion of expanding graphs and apply the probabilistic method to demonstrate their existence. These graphs have powerful properties that prove useful in later chapters, and we illustrate these properties via an application to probability amplification.

In certain cases, the probabilistic method can actually be used to demonstrate the existence of algorithms, rather than merely combinatorial objects. We illustrate this by showing the existence of efficient non-uniform algorithms for the problem of oblivious routing. We then present a particular result, the Lovász Local Lemma, which underlies the successful application of the probabilistic method in a number of settings. We apply this lemma to the problem of finding a satisfying truth assignment in an instance of the SAT problem where each variable occurs in a bounded number of clauses. While the probabilistic method usually yields only randomized or non-uniform deterministic algorithms, there are cases where a technique called the method of conditional probabilities can be used to devise a uniform, deterministic algorithm; we conclude the chapter with an exposition of this method for derandomization.

IN this chapter we present some general bounds on the tail of the distribution of the sum of independent random variables, with some extensions to the case of dependent or correlated random variables. These bounds are derived via the use of moment generating functions and result in “Chernoff-type” or “exponential" tail bounds. These Chernoff bounds are applied to the analysis of algorithms for global wiring in chips and routing in parallel communications networks. For applications in which the random variables of interest cannot be modeled as sums of independent random variables, martingales are a powerful probabilistic tool for bounding the divergence of a random variable from its expected value. We introduce the concept of conditional expectation as a random variable, and use this to develop a simplified definition of martingales. Using measure theoretic ideas, we provide a more general description of martingales. Finally, we present an exponential tail bound for martingales and apply it to the analysis of an occupancy problem.

The Chernoff Bound

In Chapter 3 we initiated the study of techniques for bounding the probability that a random variable deviates far from its expectation. In this chapter we focus on techniques for obtaining considerably sharper bounds on such tail probabilities.

The random variables we will be most concerned with are sums of independent Bernoulli trials; for example, the outcomes of tosses of a coin. In designing and analyzing randomized algorithms in various settings, it is extremely useful to have an understanding of the behavior of this sum.

Summary

The probabilities of 2 mutually exclusive alternatives are combined by addition. Equivalent probabilities can be combined by averaging. Probabilities derived from 2 independent estimates of the same distribution of scores can be combined by Stouffer's Z method or by Fisher's chi squared method. Independent probabilities of the same event should be combined using the Bayes product of odds. The conjoint probability of 2 independent events with unknown outcomes is calculated by multiplication. There are 3 general approaches for dealing with problems with multiple uncertainties. Decision makers can ask an expert professional to give a judgment. They can follow conventional practices, or they can make a formal analysis like a decision analysis or cost-benefit analysis.

Combining probabilities

Table 2.1 lists 6 different methods of combining probabilities. Table 2.2 gives the numerical values when each method is used to combine a probability of .1 with probabilities ranging from .05 through .5. In the column headings of Table 2.2 the methods are numbered 1 through 6, from the method giving the highest combined probabilities on the left to the method giving the lowest combined probabilities on the right.

Summary

Lack of confidence can produce apparent overconfidence. This can happen when people estimate the probability of correctness of their answers to 2-choice general knowledge questions, using a one-sided scale of probability extending only from .5 through 1.0. It can also happen in setting uncertainty bounds on unknown quantities using the fractile method. People show apparent overconfidence when rating the probability of correctness of their answers in other conditions: when performing an impossible task that is not judged to be impossible; when judging the probability of correctness of their answers from their familiarity with the area of knowledge instead of from their detailed knowledge of the questions asked; and when rating the probability of correctness of their answers using a logarithmic scale with very long odds.

Apparent overconfidence can be reduced by training with feedback, by employing trained experts who are warned against overconfidence, perhaps by reversing stimuli and responses in setting uncertainty bounds, and by making the response contraction bias oppose overconfidence. Overconfidence in predicting people's future progress encourages expert interviewers to rely on their clinical judgment and neglect the objective measures that they are given.

Lack of confidence can produce apparent overconfidence

Well-calibrated people have a pretty good idea of what they do and do not know, and of what they can and cannot do well. Calibration can be assessed by comparing people's average success at a task with their average estimates of success (Adams, 1957).

Summary

The same pair of outcomes can be presented in different frames or contexts. Changing the frame can be made to reverse the preferences for the outcomes. The heuristic bias could be said to be using the frame in choosing between the outcomes. Anyone who is able to avoid being biased by the frame could be said to follow the normative rule.

Changing the reference level frame can be made to change a sure gain into a sure loss, and so to reverse the preferences. The probabilities that distinguish 2 options can be isolated from the probabilities that both options share. Changing the frame by isolating the distinguishing probabilities can be made to change a probable gain into a sure gain, and so to reverse the preferences for the 2 options. The choice between a small sure loss and a larger less probable loss can be framed as a preference or as an insurance premium. The small sure loss is more often chosen when framed as insurance.

In theory, bias by frames could be avoided, provided the cover story and options together present all the facts. But ordinary respondents would probably have to be warned against the bias and be taught how to deal with it. Changing the frame from the probability of survival to the probability of death may reduce the apparent difference between surgical and medical treatments, and so the preference for surgical treatment.

Summary

Of the investigations described in this book, 4 show that trained experts perform better in their area of expertise than do ordinary respondents, who are usually untrained students. Another 4 investigations show that statistically trained students perform better on tests of probability than do statistically naive students. In addition, 6 investigations show that a relatively brief training in the laboratory can improve untrained people's dealing with probabilities.

Training improves dealing with probabilities

The results reported in this book demonstrate that many untrained people fail to deal with probabilities correctly, and that they can be trained to perform better. The picture is fairly typical of the literature on dealing with probabilities (Edwards, 1983, p. 508; Phillips, 1983 p. 526). However, the results do not abolish the need to investigate the effects of training. In everyday life novel tasks often have to be performed without special training. It is important to find out what heuristics untrained people use, how quickly they learn the normative rules, and how successfully they then apply them.

Experts

Entry 1.1 A in Table 15.1 (Lichtenstein and Fischhoff, 1980a, Table 1; see Chapter 3, Employ trained experts) describes the performance of 8 experts in subjective probability. The experts do not show apparent overconfidence in their answers to 2-choice general knowledge questions, when using a one-sided rating scale with probabilities ranging only from .5 through 1.0.