If we are given an optical imaging system, one thing we would like to know is how much light will be available to our sensors. Because all sensors have limited operating ranges, controlling the amount of light irradiating the sensors is very important to obtain the best usage of the sensors and for the best quality of our images. In order to study the energy flow of light through the various stages of image formation, we have to carefully define the concepts and terms that we are going to use. The study and measurement of optical energy flow are the subject of radiometry.

Over the years several nomenclature systems have been proposed for light measurement and although there is still some debate on the subject, the units and terms proposed by the CIE have gained general acceptance. These units and terms are described in detail in the CIE publication International Lighting Vocabulary [187]. They have also been adopted by the American National Standards Institute (ANSI Z7.1–1967) and recommended by the publications of the (US) National Bureau of Standards [700, p. 8]. We will describe the radiometric concepts using CIE units and terms.

Concepts and definitions

The concepts and measurements of optical energy flow in radiometry are traditionally defined using geometrical optics. For example, optical rays are used to define the cone associated with a light beam and the path by which a ray is transmitted from one medium to another is determined by Snell's law. As a consequence of this idealization, many concepts lose their meanings when the spatial dimension is reduced to an infinitely small distance.

Introduction

The reproduction of colors in many imaging applications requires “faithful reproduction” of the color appearances in the original scenes or objects. Since faithfulness is in the eyes of the beholder, the objective of color reproduction needs careful examination.

For some applications, the objective is easy to define in exact, physical terms, but turns out to be very difficult to achieve: for example, a mail-order catalogue has to reproduce the image of merchandise just as it will look when the customer actually holds it in his hand. The only true solution is to reproduce the spectral reflectance of the object, including the fluorescent property of the original material if any. The Lippman method and the microdispersion method [429, pp. 5–8], which come close to reproducing the spectra, are simply too expensive and inconvenient to use.

For some other applications, the objective is very difficult to define in exact, physical terms, and there is no known method of achieving the objective systematically. For example, there is no convenient method of reproducing the wide dynamic luminance range of an outdoor scene on a bright sunny day on a reflection print, which has only a very limited dynamic range. In this case, it is difficult to specify the physical criteria for a good or faithful color reproduction. Furthermore, it is likely that there will not be a reproduction that most people agree to be the best possible reproduction. The subject may well fall into the domain of artistic impression, which is still far beyond our limited understanding.

The beauty of the golden sky at sunset, the splendor of peacock feathers, and the glorious spectacle of fireworks are displays of changing color. Our visual sense is greatly enriched by our perception and appreciation of colors. Although our color perception seems to be direct and effortless, it is a very interesting subject of immense complexity, as are other aspects of our visual perception. In the last 70 years, we have made a lot of progress in understanding the physics, chemistry, optics, physiology, psychophysics, anatomy, neural science, and molecular biology of human color vision, but we are still very far from being able to describe exactly how it works. Therefore, practical use of color requires certain empirical rules. These rules, which are by no means perfect, are based on many years of experimentation and observation, and they form the empirical foundation of colorimetry, the science of measuring color.

The basic measurement of a color stimulus is its spectral power distribution as a function of wavelength (or frequency). The spectral power distribution of a reflecting surface is the product of the spectral power distribution of the illumination and the spectral reflectance factor of the surface. Although the same spectral power distribution may produce different color sensations, depending on its surroundings, background, illumination, and viewing geometry, all physical specifications of color stimuli start from their spectral power distributions. The link between the objective physics and the subjective perception is provided by photometry and colorimetry. These two fields of scientific study attempt to quantify the capacity of light stimuli to produce color sensation.

In our discussion of radiometry, light flux is measured in terms of power or energy. However, even under identical viewing conditions, equal power of light of different wavelengths does not produce equal brightness sensation in our visual perception. (In an extreme case, even a kilowatt infrared source will not help us to see.) Therefore, radiometric quantities are not always meaningful in our visualworld, especially in the field of illumination engineering. For example, in order to illuminate a library reading room, we need to know “how much” visible light our chosen light sources will provide for reading. For these types of applications, we need to measure light flux in quantities that are representative of its visual impact, such as brightness. Photometry deals with measurements of visible light in terms of its effectiveness to produce the “brightness” sensation in the human visual system. Given two stimuli of different spectral compositions, the basic goal of photometry is to set up a quantitative procedure for determining which stimulus will appear “brighter” or more luminous to an average observer.

Measuring visual quantities of light is complicated because light stimuli of different spectral compositions produce complex perceptions of light, such as bright red or dark green. It is not easy (if not impossible) to order these different color sensations along a single, intensive scale. In fact, years of research have not produced a completely satisfactory solution. However, the applications are so important that an agreed-upon, incomplete solution is better than no solution at all.

Introduction

Images are often considered as records of the physical scenes that we have seen. Therefore, we wish to have images that reproduce the visual impressions of the original scenes as we remember them. Among the various attributes that contribute to the total visual impression, tone and color are two of the most important factors. Tone reproduction is the process of reproducing the visual brightness/lightness impression of the original scene in an image. Similarly, color reproduction refers to the process of reproducing the visual color impression. Although color perception involves brightness/lightness perception, the two topics will be discussed separately, with the implied, narrower definitions that tone reproduction deals with luminance perception and color reproduction chrominance perception. However, it should be understood that there are interactions and trade-offs between the two processes. The criteria and goals of tone reproduction vary from application to application, and we will mainly be interested in consumer imaging applications.

Since the success of a tone reproduction is finally judged by human observers, there are at least two separate systems involved in a tone reproduction task, i.e., the imaging system and the human visual system. Therefore, it is convenient to divide any tone reproduction into three processes: (1) the subjective process that specifies what a desired reproduction should be in terms of visual impression, (2) the psychophysical (translation) process that converts the perceptual criteria as specified in the subjective process into physically quantifiable criteria, and (3) the objective process that deals with calibrating and controlling image devices to achieve the desired reproduction in terms of physical quantities.

In conventional photography, from capturing an image on a color negative to getting a color print back from a photofinisher, most consumers do not have to bother with measuring light and color because the entire image chain (from film to reflection print) has been designed to reproduce good tone and color, and the uncontrolled variables left are taken care of by the automatic focus and exposure algorithms in the camera, and automatic color–density balance algorithms in the photofinishing printer. The conventional photographic system is a closed system and no user intervention is required. In electronic imaging, the situation is quite different. Digital cameras, film scanners, paper scanners, color monitors, and color printers are not manufactured to the same system specifications. Therefore, these imaging devices require careful calibration to ensure that they work together to reproduce good color images. Figure 16.1 shows a general block diagram for a color imaging application. The image data from an input device have to be calibrated so that they can be manipulated and processed according to the algorithm specifications. Similarly, the image processing algorithm outputs a digital image in a chosen color metric and that image has to be converted through an output calibration so that the desired color image can be reproduced by the output device.

Let us say, we have a nice color picture (reflection print) of our relatives and friends, and wewould like to produce more copies of the picture to send to each of them. We use a scanner to digitize the picture into a digital file which is then sent to a color printer for printing.

Color images of scenes and objects can be captured on photographic film by conventional cameras, on video tape by video cameras, and on magnetic disk or solid-state memory card by digital cameras. Digital color images can be digitized from film or paper by scanners. In this chapter, we will cover these major color image acquisition devices. Photographic film has the longest history and still offers a convenient, low-cost, high-quality means for capturing color images. For this reason, it is very useful to understand the photographic processes, photographic film and photographic paper, because they are often the sources of many color images that we will encounter. They have some unique properties that influence how film-originated digital color images should be processed by computers. The next in importance is the solid state sensors, of which charge-coupled devices (CCDs) are the most widely used so far, with others (such as complementary metal-oxide–semiconductor (CMOS) sensors) gaining in popularity. Scanners are devices that are used to digitize images from film and paper. They are the main devices for generating high-quality digital color images. Most scanners use CCD sensors, except some high-end graphic arts scanners that use photomultiplier tubes. Digital cameras are becoming more and more competitive with photographic films in terms of image quality and convenience. Most digital cameras today also use CCD sensors. Each of these devices has different characteristics and unique image processing problems. They are discussed separately.

General considerations for system design and evaluation

Color image acquisition systems are designed under a lot of practical constraints. Many system components are designed and manufactured separately.

Introduction

In this chapter, we will study lens aberrations and their effects on light distributed on the image plane. We would like to calculate the image irradiance for a given optical imaging system, especially when there is defocus because this is the most frequent problem in consumer images. First, we derive the relation between the scene radiance and the image irradiance for an ideal optical imaging systemwhich has no lens aberrations and is in perfect focus. Next, we study how the distribution of light on the image plane is affected by some defects in the optical imaging process. The theory of wavefront aberrations is formulated and it is used to calculate the point spread function (PSF) and the OTF in the presence of focus error. Results from geometrical optics and physical optics are compared.

Some terms are used very often in the discussion of image light distribution. Sometimes, however, they are defined differently by different authors. We will define some of these terms here based on the international standard as specified in ISO 9334. The image of an ideal point object is a two-dimensional function, f(x, y), on the image plane, on which the coordinates (x, y) are defined. If we normalize this function so that it integrates to 1, the normalized f(x, y) is the PSF of the imaging system. The Fourier transform of the PSF is the OTF, F(νx, νy), where νx and νy are the horizontal and vertical spatial frequencies in the image plane. By the definition of the PSF, the OTF is equal to 1 at zero frequency, i.e., F(0, 0) = 1. An OTF can be a complex function.

Introduction

Color images are formed by optical imaging systems from physical scenes that are composed of three-dimensional matter interacting with light. Light radiated from light sources is reflected, refracted, scattered, or diffracted by matter. As a result of all these light–matter interactions, light is redistributed spatially and temporally to create the physical scenes that we see and take pictures of. The study of color imaging science, thus, should begin with the light-field formation process of the physical scenes. This is what we mean by scene physics. The necessity for studying such a subject arises not simply to generate realistic color images by computer graphics. It is also driven by our need to understand and model the scene physics to develop computational algorithms that can adjust the color balance and tone scale of color images automatically so that optimal reproduction and display can be achieved.

General description of light reflection

Our discussion of reflection (Fresnel equations) in Section 5.4.1 assumes that the object surface is perfectly smooth, flat, and isotropic. However, the surfaces of real objects are almost never like that. In order to characterize light reflection and scattering from surfaces, we need a more general way to describe the optical property of surface reflection.

Although surface reflection is a well-studied subject, the terms used in the literature have not yet been standardized. Difficulties arise not only with the definitions, but also with the underlying concept of measurement and the models of the assumed physical processes. Let us start by treating light as rays (geometric optics) and see what can happen as light interacts with a rough surface.

The performance of a color imaging system is often evaluated by the image quality it can deliver to the user. Image quality can be evaluated physically (objective image quality) or psychophysically (subjective or perceptual image quality) or both. In this chapter, we will discuss some of the metrics and procedures that are used in image quality measurements. Objective image quality measures, such as resolving power, noise power spectrum, detective quantum efficiency (DQE), and system MTF, are well defined and can often be measured consistently [64]. However, they may not be directly correlated with the perceived image quality. Therefore psychophysical procedures are used to construct metrics that relate to the subjective image quality. Given our inadequate understanding of image perception, one may even argue that the definitive quality evaluation can only be done by human observers looking at images and making judgments. Therefore, the subjective quality rating is the only reliable metric for image quality. Although this statement is true, it does not help us much in developing better imaging systems because human judgment is too time-consuming, costly, and not always consistent. Objective image quality metrics are needed for many product optimizations and simulations.

In the past (before 1970), image quality was often measured on a system level. With the advance and availability of digital imaging devices, quality metrics for individual digital images have also been developed. These image-dependent image quality measures are becoming more and more important because they can be used to detect and correct problems before images are displayed or printed. An automatic correction algorithm for individual images requires a reliable image quality metric that can be computed from a digital image [489, 672].

Having studied radiometry, colorimetry, and the psychophysics of our visual perception, we now have the appropriate background to study the subject of color order systems. This is a subject that is often discussed on an intuitive level, but the concepts and the logic of color order systems can be much better appreciated if we have a proper knowledge of the physics of color and the psychophysics of human color perception. Therefore, we have delayed discussion of this subject until now. Color order systems are important in applications because they provide some practical solutions for many color problems in our daily life, such as how to specify the paint color we want and how to coordinate the colors of furniture. Color order systems are also quite important for the explicit expression of our theoretical thinking and understanding of how we perceive colors, such as the opponent-color processes.

Introduction

How many colors can we distinguish? The number is estimated to be more than one million [713]. How do we accurately communicate with each other about a particular color without actually showing a real sample? Obviously our vocabulary of color names is too limited for this purpose. A system is needed to order all possible colors according to certain chosen attributes in a well-defined manner so that any color can be specified by its attributes in the system. In principle, a color order system can be designed purely on a conceptual level [519]. However, for the convenience of practical use, most color systems are implemented as collections of physical color samples. This makes them easy to understand and easy to use, and means it is easy to make approximate interpolation between colors.

The optics of the eye imposes the upper bound on the image details that can be seen by the visual system. It is important to understand and be able to model this limit of image quality under various viewing conditions, so that the performance by imaging systems can be properly optimized. However, it should be pointed out that the optical characteristics of the human eye are constantly changing throughout life, and there are also very significant variations among individuals. In this chapter, we will first describe the important features of the anatomy of the eye. Since the anatomy shows a structure too complicated to model in detail, we will then describe two simplified optical models of the eye: the reduced eye and the schematic eye. These models are very useful because they allow us to make good estimates of geometrical metrics for our retinal images. We will discuss some optical properties of the ocular media and the eye as awhole. We will also touch on the mechanism of accommodation and pupil control. Finally, we will describe how to put together a computational model of the eye optics for calculating the optical quality of the retinal image. Such a model will allow us to performmore detailed analyses under various viewing conditions and for different stimuli.

Before our discussion on visual optics, we need to define “visual angle” as a measure of image size and retinal distance. Since the image size of an object on the retina depends on its distance from the eye, it is often more convenient to use visual angle to specify object size or retinal distance.

To understand the capturing, the processing, and the display of color images requires knowledge of many disciplines, such as image formation, radiometry, colorimetry, psychophysics, and color reproduction, that are not parts of the traditional training for engineers. Yet, with the advance of sensor, computing, and display technologies, engineers today often have to deal with aspects of color imaging, some more frequently than others. This book is intended as an introduction to color imaging science for engineers and scientists. It will be useful for those who are preparing to work or are already working in the field of color imaging or other fields that would benefit from the understanding of the fundamental processes of color imaging.

The sound training of imaging scientists and engineers requires more than teaching practical knowledge of color signal conversion, such as YIQ to RGB. It also has to impart good understanding of the physical, mathematical, and psychophysical principles underlying the practice. Good understanding ensures correct usage of formulas and enables one to come up with creative solutions to new problems. The major emphasis of this book, therefore, is to elucidate the basic principles and processes of color imaging, rather than to compile knowledge of all known systems and algorithms. Many applications are described, but they serve mainly as examples of how the basic principles can be used in practice and where compromises are made.

Color imaging science covers so many fields of research that it takes much more than one book to discuss its various aspects in reasonable detail.

Imaging is a mapping from some properties of the physical world (object space) into another representation of those properties (image space). The mapping can be carried out by changing the propagation of various types of physical signals. For example, medical ultrasound imaging is the mapping of the acoustic properties of the body tissue into their representation in the transmitted or reflected intensity of the acoustic field. The mapping is carried out by the absorption, scattering, and transmission of the acoustic energy. Optical imaging, the formation of an optical representation separate from the original objects, is a mapping carried out mostly by changing the directions of the electromagneticwaves coming from the objects. Insofar as light can be treated as rays, the spatial mapping from a point in the object space to a point in the image space can be studied geometrically. This field is called the geometrical theory of optical imaging. Situations arise when the wave nature of the light has to be dealt with explicitly. This field is called the physical (or wave) theory of optical imaging. Of course, there are other cases where the quantum nature of the light is the dominant characteristics to be considered.

In this and the next chapter we will study only the basic concepts and processes of optical imaging. The three main subjects to be studied are geometric optics, physical optics, and the radiometry of imaging.

Digital image processing is a field that has diverse applications, such as remote sensing, computer vision, medical imaging, computer graphics, graphic arts, pattern recognition, and industrial inspection. There have been many books that cover the general topics of digital image processing in varying depths and applications (e.g., [86, 165, 262, 351, 363, 456, 457, 594, 752, 776, 807, 841]). Readers are encouraged to consult these books for various operations and algorithms for digital image processing. Most of the books deal with monochromatic images. When dealing with color images, there are several concepts that are inherently quite different. For example, if we treat the RGB signals at a pixel as a three-dimensional vector, a color image becomes a vector field, while a monochromatic image is a scalar field. Typical operations, such as the gradient of an image, have to be thought over again because simply repeating the same scalar operation three times is often not the best thing to do. Another important reason for much of the required rethinking is that our visual perception of a color image is usually described in terms of luminance–chrominance color attributes, not RGB color channels. A color image simply provides much more information than a monochromatic image about the scene, its material properties and its illumination. We have to think and rethink about how to extract the additional information more effectively for the applications we have in mind. In this chapter, we will study some basic issues and explore some new concepts for formulating old problems which we might have encountered when working on monochromatic image processing.