An exciting development of the 21st century is that the 20th-century vision of mechanized program verification is finally becoming practical, thanks to 30 years of advances in logic, programming-language theory, proof-assistant software, decision procedures for theorem proving, and even Moore's law which gives us everyday computers powerful enough to run all this software.

We can write functional programs in ML-like languages and prove them correct in expressive higher-order logics; and we can write imperative programs in C-like languages and prove them correct in appropriately chosen program logics. We can even prove the correctness of the verification toolchain itself: the compiler, the program logic, automatic static analyzers, concurrency primitives (and their interaction with the compiler). There will be few places for bugs (or security vulnerabilities) to hide.

This book explains how to construct powerful and expressive program logics based on separation logic and Indirection Theory. It is accompanied by an open-source machine-checked formal model and soundness proof, the Verified Software Toolchain (VST), formalized in the Coq proof assistant. The VST components include the theory of separation logic for reasoning about pointer-manipulating programs; indirection theory for reasoning with “step-indexing” about first-class function pointers, recursive types, recursive functions, dynamic mutual-exclusion locks, and other higher-order programming; a Hoare logic (separation logic) with full reasoning about control-flow and data-flow of the C programming language; theories of concurrency for reasoning about programming models such as Pthreads; theories of compiler correctness for connecting to the CompCert verified C compiler; theories of symbolic execution for implementing foundationally verified static analyses.

Synopsis: Specification of the interface between CompCert and its clients such as the VST separation logic for C light, or clients such as proved-sound static analyses and abstract interpretations. This specification takes the form of an operational semantics with a nontrivial memory model. The need to preserve the compiler's freedom to optimize the placement of data (in memory, out of memory) requires the ability to rename addresses and adjust block sizes. Thus the specification of shared-memory interaction between subprograms (separately compiled functions, or concurrent threads) requires particular care, to keep these renamings consistent.

A static analysis is an algorithm that checks (or calculates) invariants of a program based on its syntactic (static) structure, in contrast to a dynamic analysis which observes properties of actual program executions. Static analysis can tell us properties of all possible executions, while dynamic analysis can only observe executions on particular inputs.

A sound static analysis is one with a proof that any invariants checked by the analysis will actually hold on all executions. A foundationally sound analysis is one where the soundness proof is (ideally) machine-checked, (ideally) with respect to the machine-language instruction-set architecture specification—not the source language—and (ideally) with no axioms other than the foundations of logic and the ISA specification.

Some of the first foundationally sound static analyses were proof-carrying code systems of the early 21st century [5, 45, 35, 3]. It was considered impractical (at that time) to prove the correctness of compilers, so these proof-carrying systems transformed source-language typechecking (or Hoare logic [14]) phase by phase through the compilation, into an assembly-language Hoare logic.

With the existence of foundationally correct compilers such as CompCert, instead of proof-carrying code we can prove the soundness of a static analysis from the source-language semantics, and compose that proof with the compiler-correctness proof. See for example the value analysis using abstract interpretation by Blazy et al. [22]

Some kinds of static analysis may be easier to prove sound with respect to a program logic than directly from the operational semantics.

The imperative programming paradigm views programs as sequences of commands that update a memory state. A memory model specifies memory states and operations such as reads and writes. Such a memory model is a prerequisite to giving formal semantics to imperative programming languages, verifying properties of programs, and proving the correctness of program transformations.

For high-level, type-safe languages such as ML or the sequential fragment of Java, the memory model is simple and amounts to a finite map from abstract memory locations to the values they contain. At the other end of the complexity spectrum, we find memory models for shared-memory concurrent programs with data races and relaxed (non sequentially consistent) memory, where much effort is needed to capture the relaxations (e.g. reorderings of reads and writes) that are allowed and those that are guaranteed never to happen [1].

For CompCert we focus on memory models for the C language and for compiler intermediate languages, in the sequential case and with extensions to data race-free concurrency. C and our intermediate languages feature both low-level aspects such as pointers, pointer arithmetic, and nested objects, and high-level aspects such as separation and freshness guarantees. For instance, pointer arithmetic can result in aliasing or partial overlap between the memory areas referenced by two pointers; yet, it is guaranteed that the memory areas corresponding to two distinct variables or two successive calls to malloc are disjoint.

Dijkstra presented semaphore-based mutual exclusion as an extension to a sequential language [37]. Posix threads present Dijkstra-Hoare concurrency as an extension to a sequential language [55]. O'Hearn presented concurrent separation logic (CSL) as an extension to separation logic, in which all the rules of sequential separation logic still hold [71].

Can we really model concurrency as an extension to sequentiality? Boehm explains why it is very tricky to explain shared-memory concurrency as an extension to a sequential language [24]. But we have taken great care to specify our language's external-interaction model (Chapter 33), in order to do this soundly.

Therefore we do something ambitious: we present the semantic model of CSL, for the C language, in the presence of an optimizing compiler and weak cache coherency, as a modular extension to our semantic model for sequential separation logic. This chapter is based on Aquinas Hobor's PhD thesis [49, 51] and on current work by Gordon Stewart.

Concurrent separation logic with first-class locks. O'Hearn's presentation of CSL had several limitations, most importantly a lack of first-class locks (locks that can be created/destroyed dynamically, and in particular can be used to control access to other locks). Hobor et al. [51] and Gotsman et al. [44] independently extended CSL to handle first-class locks as well as a number of other features.

Chapter 30 explains our CSL with first-class locks.

In building a Hoare logic over a language specified by an operational semantics, we can view the assertions of the Hoare logic as predicates on the states of the operational semantics. But we do this in two layers: first, relate the operational states into semantic worlds; then, apply the Hoare assertions to the worlds. We do this for several reasons: The operational states may contain information that we want to hide from the Hoare assertions, particularly control information (program counter, control stack) that is not supposed to be visible to a Hoare-logic assertion. The semantic worlds may contain information that we want to hide from the operational semantics, such as step-indexes, ghost variables, predicates in the heap (for modeling first-class functions or mutex locks), and permission-shares. For convenience in reasoning, we require Liebniz equality on worlds—equivalent worlds should be equal—but operational states do not generally satisfy Liebniz equality. In our two different toy examples (Chapter 9 and Chapter 39) we have separated operational states from semantic worlds, and we do so for the VST Separation Logic for CompCert C light.

Some components of the world are subject to separating conjunction (*), and others are not. Generally, to “heap” or “memory” or any other addressable component of the world, we apply separation in order to reason about antialiasing. Nonaddressable local variables (and the addresses of addressable local variables) do not require separation, because there is no aliasing to worry about.

Synopsis: Separation logic is a formal system for static reasoning about pointer-manipulating programs. Like Hoare logic, it uses assertions that serve as preconditions and postconditions of commands and functions. Unlike Hoare logic, its assertions model anti-aliasing via the disjointness of memory heaplets. Separation algebras serve as models of separation logic. We can define a calculus of different kinds of separation algebras, and operators on separation algebras. Permission shares allow reasoning about shared ownership of memory and other resources. In a first-order separation logic we can have predicates to describe the contents of memory, anti-aliasing of pointers, and simple (covariant) forms of recursive predicates. A simple case study of straight-line programs serves to illustrate the application of separation logic.

In constructing program logics for particular applications, for particular languages, 20th-century scientists had particular difficulty with pointers—aliasing and antialiasing, updates to state, commuting of operators; and self-reference—recursive functions, recursive types, functions as first-class values, types and predicates as parameters to functions. The combination of these two was particularly difficult: pointers to mutable records that can contain pointers to function-values that operate on mutable records.

Two strains of late-20th-century logic shed light on these matters.

• Girard's 1987 linear logic [43] and other substructural logics treat logical assertions as resources that can be created, consumed and (when necessary) duplicated. This kind of thinking led to the Ishtiaq/O'Hearn/Reynolds invention in 2001 of separation logic [56, 79], which is a substructural Hoare logic for reasoning about sharing and antialiasing in pointer-manipulating programs.

• Scott's 1976 domain theory [84] treats data types as approximations, thereby avoiding paradoxes about recursive types and other recursive definitions/specifications: a type can be defined in terms of more approximate versions of itself, which eventually bottoms out in a type that contains no information at all. These ideas led to the Appel/McAllester/Ahmed invention in 2001 of step-indexing [10, 2] which is a kind of practical domain theory for computations on von Neumann machines.

Many 21st-century scientists have generalized these methods for application to many different program logics (and other applications); and have formalized these methods within mechanical proof assistants.