Knowing the perceptions that an animal controls and the brain-based rules that regulate which actions are performed may not be sufficient to explain the overt behaviour performed. The reason for this is that the animal’s particular body anatomy and the structure of the environment within which the actions are performed may constrain them to a particular form. To be certain that it is perceptions or intrinsic motor organisation that is responsible for the behaviour being expressed, the role of body anatomy and environmental context needs to be assessed. These are the third and fourth principles of behavioural organisation. Without taking these principles into account, the behavioural markers selected for measurement may be misleadingly attributed to neural causes and evolutionary processes.

As an empirical science, the study of animal behaviour involves measurement. When an animal engages in a series of actions, such as exploring or catching prey, the problem becomes that of identifying suitable components from that stream of action to use as markers suitable to score. The markers selected reflect the observer’s hypothesis of the organisation of the behaviour. Unfortunately, in most cases, researchers only provide heuristic descriptions of what they measure. To make the study of animal behaviour more scientific, the hypotheses underlying the decision of what to measure should be made explicit so as to allow them to be tested. Using hypothesis testing as a guiding framework, several principles that have been shown to be useful in identifying behavioural organisation are presented, providing a starting point in deciding what markers to select for measurement.

Some actions are intrinsic motor units that can be concatenated as needed to solve a problem. In contrast, many actions are correlated with one another when engaging in particular types of behaviour, such as predation, mating or aggression. Moreover, how those actions are associated depend on intrinsic rules of organisation. Consequently, selecting markers to measure as if they are independent may be misleading. For example, scoring success in grasping a piece of food may fail to reveal that different combinations of limb movements may be capable of comparable rates of success. The lack of independence among actions and the potentially misleading conclusions that can be drawn from end-point measures point us to the second principle – understanding the intrinsic organisation of behaviour. Knowing something about the intrinsic organisation of a behavioural sequence can be critical in identifying markers that reflect that organisation.

Knowing the perceptions that an animal controls and the brain-based rules that regulate which actions are performed may not be sufficient to explain the overt behaviour performed. The reason for this is that the animal’s particular body anatomy and the structure of the environment within which the actions are performed may constrain them to a particular form. To be certain that it is perceptions or intrinsic motor organisation that is responsible for the behaviour being expressed, the role of body anatomy and environmental context needs to be assessed. These are the third and fourth principles of behavioural organisation. Without taking these principles into account, the behavioural markers selected for measurement may be misleadingly attributed to neural causes and evolutionary processes.

As an empirical science, the study of animal behaviour involves measurement. When an animal engages in a series of actions, such as exploring or catching prey, the problem becomes that of identifying suitable components from that stream of action to use as markers suitable to score. The markers selected reflect the observer’s hypothesis of the organisation of the behaviour. Unfortunately, in most cases, researchers only provide heuristic descriptions of what they measure. To make the study of animal behaviour more scientific, the hypotheses underlying the decision of what to measure should be made explicit so as to allow them to be tested. Using hypothesis testing as a guiding framework, several principles that have been shown to be useful in identifying behavioural organisation are presented, providing a starting point in deciding what markers to select for measurement.

When engaging in some behaviour, some actions by an animal are more likely to resonate with us as observers than others and those impressions often form the basis for the behavioural markers that we choose for measurement. However, as much as possible, we should view the context from the animal’s perspective as it is what is important to them that guides their behaviour. Of what the animal may be able to sense, some sensations rank as perceptions that are relevant to the animal in that context. Moreover, in dynamic situations, it is often those perceptions that the animal seeks to stabilise. This means the behaviour controls those perceptions, so many actions can be explained as being compensatory. Without knowing what an action is compensating for, actions may be mistakenly abstracted as markers to measure. So, the first principle is to identify the perceptions that are relevant to the animal.

The inferior temporal cortex (ITC) is the highest echelon within the visual stream concerned with processing visual shape information. The Felleman and Van Essen diagram (Chapter 1, Figure 1.5) places the hippocampus at the top. While visual responses can be elicited in the hippocampus, people with bilateral lesions to the hippocampus can still see very well. A famous example is a patient known as H. M., who had no known visual deficit but gave rise to the whole field of memory studies based on his inability to form new memories. The hippocampus is not a visual area and instead receives inputs from all sensory modalities (Chapter 4).

We have been traveling through the wonderful territory of the visual cortex, examining the properties of different brain areas and neural circuits, learning about how animals and their neurons respond to visual stimuli and what happens when different parts of the visual cortex are lesioned or artificially stimulated. It is now time to put all this biological knowledge into a theory of visual recognition and to instantiate this theory through a computational model that can see and interpret the world. En route toward this goal, here we start by discussing how scientists describe neural circuits using computational models and define the basic properties of neural networks.

Around the 1950s, a wealth of behavioral experiments had characterized many phenomenological aspects of visual perception that begged for a mechanistic explanation (Chapter 3). Lesion studies had provided a compelling case that damage to circumscribed brain regions led to specific visual processing deficits (Chapter 4). These lesion studies pointed to specific brain areas to investigate visual processing, especially the primary visual cortex in the back of the brain. In addition, the successful use of microelectrode electrical recordings had led to direct insights about the function of neurons within the retinal circuitry (Chapter 2). The time was ripe to open the black box of the brain and begin to think about how vision emerges from the spiking activity of neurons in the cortex.

We want to understand the neural mechanisms responsible for visual cognition, and we want to instantiate these mechanisms into computational algorithms that resemble and perhaps even surpass human performance. In order to build such biologically inspired visually intelligent machines, we first need to define visual cognition capabilities at the behavioral level. What types of shapes can be recognized, and when and how? Under what conditions do people make mistakes during visual processing? How much experience and what type of experience with the world is required to learn to see? To answer these questions, we need to quantify human performance under well-controlled visual tasks. A discipline with the picturesque and attractive name of psychophysics aims to rigorously characterize, quantify, and understand behavior during cognitive tasks.