My aims in preparing this second edition have been to simplify and clarify the discussion, wherever this could be done without diluting the content, and to update the text in the light of developments during the last 17 years. The discussion of non-locality and particularly the Bell inequalities in Chapter 3 is an example of both of these. The proof of Bell's theorem has been considerably simplified, without, I believe, damaging its validity, and reference is made to a number of important experiments performed during the last decade of the twentieth century. I am grateful to Lev Vaidman for drawing my attention to the unfairness of some of my criticisms of the ‘many worlds’ interpretation, and to him and Simon Saunders for their attempts to lead me to an understanding of how the problem of probabilities is addressed in this context. Chapter 6 has been largely rewritten in the light of these, but I am sure that neither of the above will wholeheartedly agree with my conclusions.

Chapter 7 has been revised to include an account of the influential spontaneous-collapse model developed by G. C. Ghiradi, A. Rimini and T. Weber. Significant recent experimental work in this area is also reviewed. There has been considerable progress on the understanding of irreversibility, which is discussed in Chapters 8, 9 and 10. Chapter 9, which emphasised ideas current in the 1980s, has been left largely alone, but the new Chapter 10 deals with developments since then.

The last two chapters have described two extreme views of the quantum measurement problem. On the one hand it was suggested that the laws of quantum physics are valid for all physical systems, but break down in the assumed non-physical conscious mind. On the other hand the many-worlds approach assumes that the laws of physics apply universally and that a branching of the universe occurs at every measurement or measurement-like event. However, although in one sense these represent opposite extremes, what both approaches have in common is a desire to preserve quantum theory as the one fundamental universal theory of the physical universe, able to explain equally well the properties of atoms and subatomic particles on the one hand and detectors, counters and cats on the other. In this chapter we explore an alternative possibility, that quantum theory may have to be modified before it can explain the behaviour of large-scale macroscopic objects as well as microscopic systems. We will require that any such modification preserves the principle of weak reductionism discussed towards the end of Chapter 5.

The first point to be made is that the problems we have been discussing seem to make very little difference in practice. As we emphasised in Chapter 1, quantum physics has been probably the most successful theory of modern science. Wherever it can be tested, be it in the exotic behaviour of fundamental particles or the operation of the silicon chip, quantum predictions have always been in complete agreement with experimental results.

Albert Einstein's comment that ‘God does not play dice’ sums up the way many people react when they first encounter the ideas discussed in the previous chapters. How can it be that some future events are not completely determined by the way things are at present? How can a cause have two or more possible effects? If the choice of future events is not determined by natural laws does it mean that some supernatural force (God?) is involved wherever a quantum event occurs? Questions of this kind trouble many students of physics, though most get used to the conceptual problems, say ‘Nature is just like that’ and apply the ideas of quantum physics to their study or research without worrying about their fundamental truth or falsity. Some, however, never get used to the, at least apparent, contradictions. Others believe that the fundamental processes underlying the basic physics of the universe must be describable in deterministic, or at least realistic, terms and are therefore attracted by hidden-variable theories. Einstein was one of those. He stood out obstinately against the growing consensus of opinion in the 1920s and 30s that was prepared to accept indeterminism and the lack of objective realism as a price to be paid for a theory that was proving so successful in a wide variety of practical situations. Ironically, however, Einstein's greatest contribution to the field was not some subtle explanation of the underlying structure of quantum physics but the exposure of yet another surprising consequence of quantum theory.

A completely different interpretation of the measurement problem, one which many professional scientists have found attractive if only because of its mathematical elegance, was first suggested by Hugh Everett III in 1957 and is known variously as the ‘relative state’, ‘many-worlds’ or ‘branching-universe’ interpretation. This viewpoint gives no special role to the conscious mind and to this extent the theory is completely objective, but we shall see that many of its other consequences are in their own way just as revolutionary as those discussed in the previous chapter.

The essence of the many-worlds interpretation can be illustrated by again considering the example of the 45° polarised photon approaching the H/V detector. Remember what we demonstrated in Chapters 2 and 4: from the wave point of view a 45° polarised light wave is equivalent to a superposition of a horizontally polarised wave and a vertically polarised wave. If we were able to think purely in terms of waves, the effect of the H/V polariser on the 45° polarised wave would be simply to split the wave into these two components. These would then travel through the H and V channels respectively, half the original intensity being detected in each. Photons, however, cannot be split but can be considered to be in a superposition state until a measurement ‘collapses’ the system into one or other of its possible outcomes. Up to now, we have argued that collapse must inevitably happen somewhere in the measurement chain – even if only when the information reaches a conscious mind.

In this chapter we ask if there is some aspect of the nature of the thermodynamic changes that occur in measuring processes that could clearly distinguish them from the kind of process for which reversibility is a possibility and to which pure quantum theory can be applied. This idea has been suggested on a number of occasions, and was developed in the 1980s by Ilya Prigogine, who won a Nobel prize in 1977 for his theoretical work in the field of irreversible chemical thermodynamics. The starting point of the approach by Prigogine and his co-workers is a re-examination of the validity of the ergodic principle, which leads to the idea of the Poincaré recurrence. We made the point in the last chapter that, in the simple case of a single particle confined to a rectangle, unless the starting angle has a special value the particle trajectory will fill the whole rectangle and the particle will sooner or later revisit a state that is arbitrarily close to its initial state. By this we meant that, although the initial and final states cannot in general be precisely identical, we can make the difference between the initial and final positions and velocities as small as we like by waiting long enough. The implicit assumption is that the future behaviour of the system will not be significantly affected by this arbitrarily small change in state; its behaviour after the recurrence will then be practically the same as its behaviour was in the first place.

The previous chapter surveyed part of the rich variety of physical phenomena that can be understood using the ideas of quantum physics. Now that we are beginning the task of looking more deeply into the subject we shall find it useful to concentrate on examples that are comparatively simple to understand but which still illustrate the fundamental principles and highlight the basic conceptual problems. Some years ago most writers discussing such topics would probably have turned to the example of a ‘particle’ passing through a two-slit apparatus (as in Figure 1.2), whose wave properties are revealed in the interference pattern. Much of the discussion would have been in terms of wave–particle duality and the problems involved in position and momentum measurements, as in the discussion of the uncertainty principle in the last chapter. However, there are essentially an infinite number of places where the particle can be and an infinite number of possible values of its momentum, and this complicates the discussion considerably. We can illustrate all the points of principle we want to discuss by considering situations where a measurement has only a small number of possible outcomes. One such quantity relating to the physics of light beams and photons is known as polarisation. In the next section we discuss it in the context of the classical wave theory of light, and the rest of the chapter extends the concept to situations where the photon nature of light is important.

Now that we have completed our survey of the conceptual problems of quantum physics, what are we to make of it all? One thing that should be clear is that there is considerable scope for us all to have opinions and there is a disappointing lack of practicable experimental tests to confirm or falsify our ideas. Because of this, I intend to drop the use of the conventional scientific ‘we’ in this chapter and to use the first person singular pronoun wherever I am stating an opinion rather than describing an objective fact or a widely accepted scientific idea. This is not to say that everything in the earlier chapters has been free of personal bias, but I have tried to maintain a greater basis of objectivity there than will be appropriate from now on.

I first want to refer briefly to a way of thinking about philosophical problems known as ‘positivism’. Encapsulated in Wittgenstein's phrase ‘of what we cannot speak thereof should we be silent’, positivism asserts that questions that are incapable of verification are ‘non-questions’, which it is meaningless to try to answer. Thus the famous, if apocryphal, debate between mediaeval scholars about how many angels can dance on the point of a pin has no content because angels can never be observed or measured so no direct test of any conclusion we might reach about them is ever possible. Opinions about such unobservable phenomena are therefore a matter of choice rather than logical necessity.

Quantum mechanics is our most successful physical theory. It underlies our very detailed understanding of atomic physics, chemistry, and nuclear physics, and the many technologies to which physical systems in these regimes give rise. Additionally, relativistic quantum mechanics is the basis for the standard model of elementary particles, which very successfully gives a partial unification of the forces operating at the atomic, nuclear, and subnuclear levels.

However, from its inception the probabilistic nature of quantum mechanics, and the fact that “quantum measurements” in the orthodox formulation appear to require the intervention of non-quantum mechanical “classical systems,” have led to speculations by many physicists, mathematicians, and philosophers of science that quantum mechanics may be incomplete. Among the Founding Fathers of quantum theory, Einstein and Schrödinger were both of the opinion that quantum mechanics is in some way unsatisfactory, and this view has been amplified in more recent profound work of John Bell, among others. In an opposing camp, many others in the physics, mathematics, and philosophy communities have attempted to provide an interpretational foundation in which quantum mechanics remains a complete and self-contained system. Among the Founding Fathers, Bohr, Born, and Heisenberg maintained that quantum mechanics is a complete system, and a number of recent proposals have been made to improve upon or to provide alternatives to their “Copenhagen Interpretation.” The debate continues, and has spawned an enormous literature.

In this chapter we set up a classical Lagrangian and Hamiltonian dynamics for matrix models. The fundamental idea is to set up an analog of classical dynamics in which the phase space variables are non-commutative, and the basic tool that allows one to accomplish this is cyclic invariance under a trace. Since no assumptions about commutativity of the phase space variables (such as canonical commutators/anticommutators) are made at this stage, the dynamics that we set up is not the same as standard quantum mechanics. Quantum mechanical behavior will be seen to emerge only when, in Chapters 4 and 5, we study the statistical mechanics of the classical matrix dynamics formulated here.

In Section 1.1, we introduce our basic notation for bosonic and fermionic matrices, and give the cyclic identities that will be used repeatedly throughout the book. In Section 1.2, we define the derivative of a trace quantity with respect to an operator, and give the basic properties of this definition. In Section 1.3, we use the operator derivative to formulate a Lagrangian and Hamiltonian dynamics for matrix models. In Section 1.4, we introduce a generalized Poisson bracket appropriate to trace dynamics, constructed from the operator derivative defined in Section 1.2, and give its properties and some applications. Finally, in Section 1.5 we discuss the relation between the trace dynamics time evolution equations, and the usual unitary Heisenberg picture equations of motion obtained when one assumes standard canonical commutators/anticommutators.

Bosonic and fermionic matrices and the cyclic trace identities

We shall assume finite-dimensional matrices, although ultimately an extension to the infinite-dimensional case may be needed. The matrix elements of these matrices will be constructed from ordinary complex numbers, and from complex anticommuting Grassmann numbers.

In Section 2.4, we illustrated the trace dynamics formalism by constructing the trace dynamics analog of a simple field theory model, in which a Dirac fermion interacts with a scalar Klein–Gordon field. Much of the recent literature in quantum field theory has concerned itself with supersymmetric theories, in which invariance under the Poincaré group has been extended to invariance under the graded Poincaré group, and theories of this type are considered likely to play a central role in the ultimate unification of the forces. Our aim in this chapter (which can be omitted on a first reading) is to show that the trace dynamics formalism naturally extends to globally supersymmetric theories. Specifically, we shall see that, when there is a global supersymmetry, there is a conserved trace supersymmetry current with a time-independent trace supercharge Qα, that together with the trace four momentum obeys the Poincaré supersymmetry algebra under the generalized Poisson bracket of Eq. (1.11a). We shall illustrate this statement with three concrete examples, the trace dynamics versions (Adler 1997a,b) of the Wess–Zumino model (Section 3.1), the supersymmetric Yang–Mills model (Section 3.2), and the so-called “matrix model for M theory” (Section 3.3). These three examples are worked out using component field methods; we close in Section 3.4 with a short discussion of a superspace approach, and of the obstruction that prevents the construction of a trace dynamics theory with local supersymmetry.

The Wess–Zumino model

We begin with the trace dynamics transcription of the Wess–Zumino model.

To keep the discussion of this book self-contained, a number of topics that are briefly mentioned in the text are treated in more detail in the appendices that follow.

The notation of the appendices follows that generally used in the text. Throughout this book, we indicate sums explicitly, except that the usual Einstein summation convention is used for sums over Greek letter four-vector and tensor indices, and a summation convention is used for the indices i, j in Section 3.3. Our Minkowski metric convention is ημν = diag(1, 1, 1, –1), and we have taken the velocity of light to be unity, so that c does not appear in the equations. However, Planck's constant is generally retained (except in the formulas of Sections 1.5 and 6.3 through 6.5 where we set = 1), because our approach implies that it has a dynamical origin.

Fermionic quantities are represented throughout by Grassmann numbers. Real Grassmann numbers are constructed as products of a basis Xr of real Grassmann elements, that obey the anticommutative algebra {Xr, Xs} = 0 for all r, s, which implies that the square of any Grassmann element vanishes. Complex Grassmann numbers have real and imaginary parts that are real Grassmann numbers. A product of an even number of Grassmann basis elements, multiplied by a complex number coefficient, is called even grade or bosonic, and a product of an odd number of Grassmann basis elements, with a complex number coefficient, is called odd grade or fermionic. Grassmann integration is defined by and so is effectively the same as differention from the left.