As already presented, the Internet routing system is partitioned into tens of thousands of independently administered Autonomous Systems (ASs). The Border Gateway Protocol (BGP) is the de facto inter-domain routing protocol that maintains and exchanges routing information between ASs. However, the BGP was designed based on the implicit trust between all participants and does not employ any measure to authenticate the routes injected into or propagated through the system. Therefore, virtually any AS can announce any route into the routing system, and sometimes the bogus routes can trigger large-scale anomalies in the Internet. A canonical example occurred on April 25, 1997, when a misconfigured router maintained by a small service provider (AS7007) in Virginia, USA, injected incorrect routing information into the global Internet and claimed to have optimal connectivity to all Internet destinations. As a result, most Internet traffic was routed to this small ISP. The traffic overwhelmed the misconfigured and intermediate routers, and effectively crippled the Internet for almost two hours. Since then, many such events have been reported, some of them due to human mistakes, others due to malicious activities that exploited vulnerabilities in the BGP in order to cause large-scale damage. For example, it is common for spammers to announce an arbitrary prefix and then use that prefix to send spam from the hijacked address space, making the trace back and the spammer identity discovery much more difficult.

Since the 1950s, voice and video services such as telephony and television have established themselves as an integral part of everyone's life. Traditionally, voice and video service providers built their own networks to deliver these services to customers. However, tremendous technical advancements since the 1990s have revolutionized the mode of delivery of these services. Today, these services are delivered to the users over the Internet, and we believe that there are two main reasons for this: (i) delivering services over the Internet in IP packets is much more economical for voice and video service providers and (ii) the massive penetration of broadband (i.e. higher bandwidth) Internet service has ensured that the quality of voice and video services over the Internet is good enough for everyday use. The feasibility of a more economical alternative for voice and video services attracted many ISPs including Comcast, AT&T and Verizon, among several others, to offer these services to end users at a lower cost. However, non-ISPs, such as Skype, Google, Microsoft, etc. have also started offering these services to customers at extremely competitive prices (and, on many occasions, for free).

From an ISP's perspective, traffic classification has always been a critical activity for several important management tasks, such as traffic engineering, network planning and provisioning, security, billing and Quality of Service (QoS). Given the popularity of voice and video services over the Internet, it has now become all the more important for ISPs to identify voice and video traffic from other service providers for three reasons.

Since the late 1990s there has been significant interest and attention from the research community devoted to understanding the key drivers of how ISP networks are designed, built and operated. While recent work by empiricists and theoreticians has emphasized certain statistical and mathematical properties of network structures and their behaviors, this part of the book presents in great detail an optimization-based perspective that focuses on the objectives, constraints and other drivers of engineering design that will help the community gain a better insight into this fascinating world and enable the design of more “realistic” models.

In this chapter we introduce the area of IP network design and the factors commonly used to drive such a process. Our discussion revolves around IP-over-WDM networks, and we define the network design problem as the end-to-end process aimed at identifying the “right” IP topology, the associated routing strategy and its mapping over the physical infrastructure in order to guarantee the efficient utilization of network resources, a high degree of resilience to failures and the satisfaction of SLAs.

We start by providing a high-level overview of the IP-over-WDM technology. We highlight the properties of the physical and IP layers (the IP layer is also known as the logical layer), we discuss their relationship, and introduce the terminology that will be extensively used in the following chapters. Then, we introduce the processes encountered in IP network design and their driving factors.

Due to the challenges of obtaining an AS topology annotated with AS relationships, it is infeasible to use the valley-free rule to identify redistribution path spoofing in the work. Alternatively, we apply the direction-conforming rule to the AS topology annotated with directed AS-links to carry out the detection. The following theorems show that the direction-conforming rule actually shows roughly equivalent efficiency.

Theorem D.1

For an observer AS, a valley-free path in the AS topology annotated with AS relationships must be “direction-conforming” in the corresponding AS topology annotated with inferred directed AS-links.

Theorem D.2

1. (1) For a Tier-1 AS, the direction-conforming paths in the AS topology annotated with inferred directed AS-links must be valley-free in the real AS topology annotated with AS relationships.

2. (2) For a non-Tier-1 AS, except the redistribution path-spoofing paths launched by the provider ASs, the direction-conforming paths must be valley-free.

In order to prove these theorems, we first investigate the mapping between the real AS topology annotated with AS relationships and the inferred AS topology annotated with directed AS-links.

Note that, similar to the analysis in the text, we assume that the inferred topology is “ideally” complete, namely it contains all legitimate directed AS-links that the observer AS should see. In order to infer a complete AS topology comprising of directed AS-links based on the route announcements from the observer AS, we assume an ideal inference scenario, in which the AS connections and relationships do not change over the inference period and every AS tries all possible valid routes.

Change detection is a fundamental problem arising in many fields of engineering, in finance, in the natural and social sciences, and even in the humanities. This book is concerned with the problem of change detection within a specific context. In particular, the framework considered here is one in which changes are manifested in the statistical behavior of quantitative observations, so that the problem treated is that of statistical change detection. Moreover, we are interested in the on–line problem of quickest detection, in which the objective is to detect changes in real time as quickly as possible after they occur. And, finally, our focus is on formulating such problems in such a way that optimal procedures can be sought and found using the tools of stochastic analysis.

Thus, the purpose of this book is to provide an exposition of the extant theory underlying the problem of quickest detection, with an emphasis on providing the reader with the background necessary to begin new research in the field. It is intended both for those familiar with basic statistical procedures for change detection who are interested in understanding these methods from a fundamental viewpoint (and possibly extending them to new applications), and for those who are interested in theoretical questions of change detection themselves.

The approach taken in this book is to cast the problem of quickest detection in the framework of optimal stopping theory.

Introduction

In Chapter 5, we considered the quickest detection problem within the framework proposed by Kolmogorov and Shiryaev, in which the unknown change point is assumed to be a random variable with a given, geometric, prior distribution. This formulation led to a very natural detection procedure; namely, announce a change at the first upcrossing of a suitable threshold by the posterior probability of a change. Although the assumption of a prior on the change point is rather natural in applications such as condition monitoring, there are other applications in which this assumption is unrealistic. For example, in surveillance or inspection systems, there is often no pre–existing statistical model for the occurence of intruders or flaws.

In such situations, an alternative to the formulations of Chapter 5 must be found, since the absence of a prior precludes the specification of expected delays and similar quantities that involve averaging over the change–point distribution. There are several very useful such formulations, and these will be discussed in this chapter.

We will primarily consider a notable formulation due to Lorden, in which the average delay is replaced with a worst–case value of delay. However, other formulations will be considered as well.

As in the Bayesian formulation of this problem, optimal stopping theory plays a major role in specifying the optimal procedure, although (as we shall see) more work is required here to place the problems of interest within the standard optimal stopping formulation of Chapter 3.

Introduction

In Chapter 4, we considered the problem of optimally deciding, with a cost on sampling, between two statistical models for a set of sequentially observed data. Within each of these two models the data are homogeneous; that is, the data obey only one of the two alternative statistical models during the entire period of observation. In most of the remainder of this book, we turn to a generalization of this problem in which it is possible for the statistical behavior of observed data to change from one model to another at some unknown time during the period of observation. The objective of the observer is to detect such a change, if one occurs, as quickly as possible. This objective must be balanced with a desire to minimize false alarms. Such problems are known as quickest detection problems. In this and subsequent chapters, we analyze several useful formulations of this type of problem. Again, our focus is on the development of optimal procedures, although the issue of performance analysis will also be considered to a degree.

A useful framework for quickest detection problems is to consider a sequence Z1, Z2,… of random observations, and to suppose that there is a change point t ≥ 1 (possibly t = ∞) such that, given t, Z1, Z2, …, Zt−1 are drawn from one distribution and Zt, Zt+1, …, are drawn from another distribution.

The problem of detecting abrupt changes in the statistical behavior of an observed signal or time series is a classical one, whose provenance dates at least to work in the 1930s on the problem of monitoring the quality of manufacturing processes. In more recent years, this problem has attracted attention in a wide variety of fields, including climate modeling, econometrics, environment and public health, finance, image analysis, medical diagnosis, navigation, network security, neuroscience, other security applications such as fraud detection and counter–terrorism, remote sensing (seismic, sonar, radar, biomedical), video editing, and even the analysis of historical texts. This list, although long, is hardly exhaustive, and other applications can be found, for example, in. These cited references only touch the surface of a very diverse and vibrant field, in which this general problem is known variously as statistical change detection, change–point detection, or disorder detection.

Many of these applications, such as those in image analysis, econometrics, or the analysis of historical texts, involve primarily off–line analyses to detect a change in statistical behavior during a pre–specified frame of time or space. In such problems, it is of interest to estimate the occurrence time of a change, and to identify appropriate statistical models before and after the change. However, it is not usually an objective of these applications to perform these functions in real time.

Introduction

This chapter will formulate and solve the classical sequential detection problem as an optimal stopping problem. This problem deals with the optimization of decision rules for deciding between two possible statistical models for an infinite, homogeneous sequence of random observations. The optimization is carried out by penalizing, in various ways, the probabilities of error and the average amount of time required to reach a decision. By optimizing separately over the error probabilities with the decision time fixed, this problem becomes an optimal stopping problem that can be treated using the methods of the preceding chapter. As this problem is treated in many sources, the primary motivation for including it here is that it serves as a prototype for developing the tools needed in the related problem of quickest detection.

With this in mind, both Bayesian and non–Bayesian, as well as discrete– and continuous–time formulations of this problem will be treated. In the course of this treatment, a set of analytical techniques will be developed that will be useful in the solution and performance analysis of problems of quickest detection to be treated in subsequent chapters. Specific topics to be included are Bayesian optimization, the Wald—Wolfowitz theorem, the fundamental identity of sequential analysis, Wald's approximations, diffusion approximations, and Poisson approximations.

Sequential testing displays certain advantages over fixed sample testing in that it helps the user reach a decision between two hypotheses after a minimal average number of experiments.

Introduction

In Chapter 4, we considered the problem of optimally deciding between two hypotheses on the state of the environment given a constant cost of sampling for each additional observation. Within each of these two models the data are homogeneous; that is, the data obey only one of the two alternative statistical models during the entire period of observation. In Chapters 5 and 6 on the other hand, we considered the problem of optimally detecting an abrupt change in the mechanism generating the data from one regime to another. In this chapter, we examine several generalizations and modifications of these basic sequential decision–making problems, in which various of the assumptions are relaxed so as to provide more practical solutions.

We will first address the problem of decentralized sequential detection. In this setting information becomes available sequentially at distinct sensors, rather than at a common location, as in the models considered in the preceding chapters. In general, these sensors communicate a summary message to a central fusion center (which may or may not also be receiving information on its own), which must ultimately decide about the state of the environment. Various sensor configurations are possible for decentralized detection. See for example. One of the main advantages of the decentralized setting over its centralized counterpart is the reduced communication requirements of such a configuration.