To save content items to your account,
please confirm that you agree to abide by our usage policies.
If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account.
Find out more about saving content to .
To save content items to your Kindle, first ensure no-reply@cambridge.org
is added to your Approved Personal Document E-mail List under your Personal Document Settings
on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part
of your Kindle email address below.
Find out more about saving to your Kindle.
Note you can select to save to either the @free.kindle.com or @kindle.com variations.
‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi.
‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.
Our last chapter is devoted to entropy. With this excuse we first present Shannon’s information theory, including the derivation of his entropy, and the enunciations and proofs of the source coding theorem and of the noisy-channel coding theorem. Then, we consider dynamical systems and the production of entropy in chaotic systems, termed Kolmogorov–Sinai entropy. For non-experts or readers who require a memory jog, we make a short recap of statistical mechanics. That is just enough to tie up some knots left untied in Chapter 4, when we developed large deviations theory for independent variables. Here we generalize to correlated variables and make one application to statistical mechanics. In particular, we find out that entropy is a large deviations function, apart from constants. We end with a lightning fast introduction to configurational entropy in disordered complex systems. Just to give a tiny glimpse of … what we do for a living!
Chapter 4 takes up the question of book size, including format (folio, quarto, etc.) as well as the adjectives applied to books (big, large, little, etc.). The rhetoric of book size gave people a way to talk about information.
Based on the long-running Probability Theory course at the Sapienza University of Rome, this book offers a fresh and in-depth approach to probability and statistics, while remaining intuitive and accessible in style. The fundamentals of probability theory are elegantly presented, supported by numerous examples and illustrations, and modern applications are later introduced giving readers an appreciation of current research topics. The text covers distribution functions, statistical inference and data analysis, and more advanced methods including Markov chains and Poisson processes, widely used in dynamical systems and data science research. The concluding section, 'Entropy, Probability and Statistical Mechanics' unites key concepts from the text with the authors' impressive research experience, to provide a clear illustration of these powerful statistical tools in action. Ideal for students and researchers in the quantitative sciences this book provides an authoritative account of probability theory, written by leading researchers in the field.
Language models can produce fluent, grammatical text. Nonetheless, some maintain that language models don’t really learn language and also that, even if they did, that would not be informative for the study of human learning and processing. On the other side, there have been claims that the success of LMs obviates the need for studying linguistic theory and structure. We argue that both extremes are wrong. LMs can contribute to fundamental questions about linguistic structure, language processing, and learning. They force us to rethink arguments and ways of thinking that have been foundational in linguistics. While they do not replace linguistic structure and theory, they serve as model systems and working proofs of concept for gradient, usage-based approaches to language. We offer an optimistic take on the relationship between language models and linguistics.
Fish swimming together in schools interact via multiple sensory pathways, including vision, acoustics and hydrodynamics, to coordinate their movements. Disentangling the specific role of each sensory pathway is an open and important question. Here, we propose an information-theoretic approach to dissect interactions between swimming fish based on their movement and the flow velocity at selected measurement points in the environment. We test the approach in a controlled mechanical system constituted by an actively pitching airfoil and a compliant flag that simulates the behaviour of two fish swimming in line. The system consists of two distinct types of interactions – hydrodynamic and electromechanical. By using transfer entropy of the measured time series, we unveil a strong causal influence of the airfoil pitching on the flag undulation with an accurate estimate of the time delay between the two. By conditioning the computation on the flow-speed information, recorded by laser Doppler velocimetry, we discover a significant reduction in transfer entropy, correctly implying the presence of a hydrodynamic pathway of interaction. Similarly, the electromechanical pathway of interaction is identified accurately when present. The study supports the potential use of information-theoretic methods to decipher the existence of different pathways of interaction between schooling fish.
Chapter 8 explores the asymptotic regime of quantum information processing, beginning with quantum typicality, which illustrates the convergence of quantum states toward a typical form with increasing copies. This leads to the asymptotic equipartition property (AEP), indicating that with a high number of copies, probability vectors become uniformly distributed. The method of types is introduced next, a tool from classical information theory that classifies sequences based on their statistical properties. This is crucial for understanding the behavior of large quantum systems and has implications for quantum data compression. Advancing to quantum hypothesis testing, the chapter outlines efficient strategies for distinguishing between two quantum states through repeated measurements. Central to this is the Quantum Stein’s lemma, which asserts the exponential decline in the error probability of hypothesis testing as the sample size of quantum systems increases. The chapter highlights the deep interplay between typicality, statistical methods, and hypothesis testing, laying the groundwork for asymptotic interconversion of quantum resources.
Information processing is a process of uncertainty resolution. Information-theoretic constructs such as surprisal and entropy reflect the fine-grained probabilistic knowledge which people have accumulated over time. The information-theoretic constructs explain the extent of processing difficulty that people encounter, for example when comprehending language. Processing difficulty and cognitive effort in turn are a direct reflection of predictability.
This enthusiastic introduction to the fundamentals of information theory builds from classical Shannon theory through to modern applications in statistical learning, equipping students with a uniquely well-rounded and rigorous foundation for further study. Introduces core topics such as data compression, channel coding, and rate-distortion theory using a unique finite block-length approach. With over 210 end-of-part exercises and numerous examples, students are introduced to contemporary applications in statistics, machine learning and modern communication theory. This textbook presents information-theoretic methods with applications in statistical learning and computer science, such as f-divergences, PAC Bayes and variational principle, Kolmogorov's metric entropy, strong data processing inequalities, and entropic upper bounds for statistical estimation. Accompanied by a solutions manual for instructors, and additional standalone chapters on more specialized topics in information theory, this is the ideal introductory textbook for senior undergraduate and graduate students in electrical engineering, statistics, and computer science.
Convergence of the expectation-maximization (EM) algorithm to a global optimum of the marginal log likelihood function for unconstrained latent variable models with categorical indicators is presented. The sufficient conditions under which global convergence of the EM algorithm is attainable are provided in an information-theoretic context by interpreting the EM algorithm as alternating minimization of the Kullback–Leibler divergence between two convex sets. It is shown that these conditions are satisfied by an unconstrained latent class model, yielding an optimal bound against which more highly constrained models may be compared.
This chapter introduces communication and information theoretical aspects of molecular communication, relating molecular communication to existing techniques and results in communication systems. Communication models are discussed, as well as detection and estimation problems. The information theory of molecular communication is introduced, and calculation of the Shannon capacity is discussed.
We develop and demonstrate a computationally cheap framework to identify optimal experiments for Bayesian inference of physics-based models. We develop the metrics (i) to identify optimal experiments to infer the unknown parameters of a physics-based model, (ii) to identify optimal sensor placements for parameter inference, and (iii) to identify optimal experiments to perform Bayesian model selection. We demonstrate the framework on thermoacoustic instability, which is an industrially relevant problem in aerospace propulsion, where experiments can be prohibitively expensive. By using an existing densely sampled dataset, we identify the most informative experiments and use them to train the physics-based model. The remaining data are used for validation. We show that, although approximate, the proposed framework can significantly reduce the number of experiments required to perform the three inference tasks we have studied. For example, we show that for task (i), we can achieve an acceptable model fit using just 2.5% of the data that were originally collected.
This Element tries to discern the known unknowns in the field of pragmatics, the 'Dark Matter' of the title. We can identify a key bottleneck in human communication, the sheer limitation on the speed of speech encoding: pragmatics occupies the niche nestled between slow speech encoding and fast comprehension. Pragmatic strategies are tricks for evading this tight encoding bottleneck by meaning more than you say. Five such tricks are reviewed, which are all domains where we have made considerable progress. We can then ask for each of these areas, where have we neglected to push the frontier forward? These are the known unknowns of pragmatics, key areas, and topics for future research. The Element thus offers a brief review of some central areas of pragmatics, and a survey of targets for future research. This title is also available as Open Access on Cambridge Core.
This chapter describes new music in Paris in the late 1960s, the period when the young spectral composers were students at the Paris Conservatoire. It opens with an account of Messiaen’s composition class and how elements such as neumes and Messiaen’s analyses of Debussy and Ravel informed Grisey’s, Murail’s, and Levinas’s emerging musical sensibilities. After giving a brief biographical account of those latter composers and Roger Tessier, the chapter touches on serialism’s changing status at a time when it had begun to be taught at the Paris Conservatoire; the effect of May ’68 on the Conservatoire’s pedagogy and on musical mores more generally among young composers; Fifth Republic France’s increased funding for new music festivals in regional cities such as Royan; Boulez and Xenakis’s profiles as the two most influential composers in France; and collectives, aleatoricism, and music theatre in post-1968 composition. The chapter closes with an account of Grisey’s early student works, in particular their creative adaptation of Messiaen’s personnages sonores concept towards the construction of audibly distinct musical figures, which would become a key element in Grisey’s musical style.
The composer-performer collective l’Itinéraire, founded in 1973 by Murail and Tessier, was the de facto platform for spectral music in France. This chapter shows how l’Itinéraire formed with the aim of establishing an organ for the music of the youngest generation of French composers, and how, with the demise of Boulez’s Domaine Musical ahead of the opening of IRCAM, l’Itinéraire fortuitously found itself positioned as the successor to that Parisian new music series, endorsed by Messiaen and recipient of a state subsidy. The chapter details how Grisey composed Périodes, a l’Itinéraire commission and the first work composed from his Les Espaces acoustiques cycle, and how Grisey attended the acoustics laboratory at the Université Paris VI Jussieu, where, alongside lessons in musical acoustics, he absorbed the work of Abraham Moles on the application of information theory to art. The chapter also shows how Murail began to incorporate the models of spectral harmonicity and periodicity into his music from Tigres de verre onwards, and it explores the relationship of these instrumental compositional techniques to the computer sound synthesis work of Risset and Chowning.
The book’s conclusion situates spectral music as a modernist musical movement. It shows how spectral music reprises many of serialism’s concerns, albeit on a more psychoacoustically accurate level. It relates the debates between Levinas and the other spectral composers to an older debate about formalism in art between Flaubert and Sand. Finally, the book concludes by situating Grisey as the founder of spectral music.
Spectral music as a distinct movement began in 1976, when, within a few days of each other, Murail’s Mémoire-érosion and Grisey’s Partiels were both premiered by Ensemble l’Itinéraire. This chapter explores how, driven by the theorist Dufourt, the young composers associated with l’Itinéraire developed a theoretical identity in contradistinction to Boulez and IRCAM. As well as detailing the salient qualities of Grisey and Murail’s music in this period, the chapter explores the diverse spectral music of Dufourt, Levinas, and Tessier. Dufourt’s works Erewhon, La tempesta d’après Giorgione, and Saturne engage with insights regarding sound related to his encounters with Risset and Chowning. Levinas’s works like Appels foregrounded sonic parasitism and a dramatic spectacle far removed from the more reserved forms of Murail, of which the chapter shows Levinas to have been at times a public critic. Tessier’s music in this period was expressionistic and explored electroacoustic resources. As well as detailing these various spectral sub-currents, the chapter explores the role of l’Itinéraire’s performers in helping to develop performing techniques adequate to the spectral writing.
The introductory chapter to Gérard Grisey and Spectral Music: Composition in the Information Age situates the book’s historical narrative by focusing on correspondence between Grisey and Dufourt in 1980 discussing what name they should give their common musical movement:’ spectral music’ or ‘liminal music’. This matter of naming indicates the compositional values the composers prioritised: movement over stasis, thresholds over states, psychoacoustic phenomena over traditional notes and pitches. The chapter then gives an overview of the book’s argument that spectral music developed from serialism through embracing information theory and developments in psychoacoustics and computer sound synthesis. Inasmuch as it arose in France but depended on developments that occurred at Bell Telephone Laboratories in the USA, spectral music was transatlantic in origin and signified a paradigm shift in musical composition.
The first in-depth historical overview of spectral music, which is widely regarded, alongside minimalism, as one of the two most influential compositional movements of the last fifty years. Charting spectral music's development in France from 1972 to 1982, this ground-breaking study establishes how spectral music's innovations combined existing techniques from post-war music with the use of information technology. The first section focuses on Gérard Grisey, showing how he creatively developed techniques from Messiaen, Xenakis, Ligeti, Stockhausen and Boulez towards a distinctive style of music based on groups of sounds mutating in time. The second section shows how a wider generation of young composers centred on the Parisian collective L'Itinéraire developed a common vision of music embracing seismic developments in in psychoacoustics and computer sound synthesis. Framed against institutional and political developments in France, spectral music is shown as at once an inventive artistic response to the information age and a continuation of the French colouristic tradition.
This paper proposes a linear quadratic approximation approach to dynamic nonlinear rationally inattentive control problems with multiple states and multiple controls. An efficient toolbox to implement this approach is provided. Applying this toolbox to five economic examples demonstrates that rational inattention can help explain the comovement puzzle in the macroeconomics literature.
Since the 1960s Mastermind has been studied for the combinatorial and information-theoretical interest the game has to offer. Many results have been discovered starting with Erdős and Rényi determining the optimal number of queries needed for two colours. For $k$ colours and $n$ positions, Chvátal found asymptotically optimal bounds when $k \le n^{1-\varepsilon }$. Following a sequence of gradual improvements for $k\geq n$ colours, the central open question is to resolve the gap between $\Omega (n)$ and $\mathcal{O}(n\log \log n)$ for $k=n$. In this paper, we resolve this gap by presenting the first algorithm for solving $k=n$ Mastermind with a linear number of queries. As a consequence, we are able to determine the query complexity of Mastermind for any parameters $k$ and $n$.