A Unified Color and Contrast Age-Dependent Visual Content Adaptation

Abstract

We present a unified color and contrast content adaptation (UCA) method designed to modify images while preserving observers’ preferences for white color and detail level. Our method is based on newly released models of the color matching functions (CMF) and contrast sensitivity function (CSF) incorporating age-dependent components. Using CIE-2006 physiological model in combination with an extension of Barten’s CSF model, our technique adjusts the display white point and local contrast to correspond to the viewer’s age-related color and contrast response preference. The results of a subjective evaluation show the effectiveness of the method on a large range of image types and confirmed that combining both color and contrast is the preferred approach for content adaptation to viewers’ characteristics.

1 Introduction

Human color and contrast perception differ from person to person, not only in terms of vision deficiency, but also among color-normal observers. However, this variability is not taken into account in current display technologies, and it is assumed that a single average standard observer can represent the entire population.

Historically, the objective of colorimetry was to integrate properties of the human color vision into the measurement of visible light to define visual equivalents of colored stimuli. More recently, its objective has been to provide procedures that enable quantifying color matches and differences [1]. In that sense colorimetry is based on the assumption that everyone’s color response can be quantified with the CIE standard observer functions, which predict the average viewer’s response to the spectral content of light. However, individual observers may have slightly different response functions, which may cause disagreement about which colors match and which do not. For colors with smoothly varying (broad) spectra, the disagreement is generally small, but for colors mixed using a few narrow-band spectral peaks, differences can be as large as 10 CIELAB units [2].

The human visual system (HVS) works over a remarkably wide range of illumination thanks to two classes of photoreceptors, rods and cones [3]. In daylight, vision relies mainly on three types of cones photoreceptors, and is referred as photopic vision. In low light conditions, only rods are active, and this is referred as scotopic vision. In between photopic and scotopic conditions, both cones and rods operate simultaneously, and this is referred as mesopic vision. Contrast sensitivity is a very important measure, especially under dim illumination, the presence of fog or glare, or when the contrast between objects and their background is reduced. For instance, an activity that requires a good contrast sensitivity for safety is driving at night. Numerous studies showed that older people have a significantly lower contrast sensitivity than young people [4, 5]. This suggests that activities that rely on visual performance could be impacted by this decline in sensitivity. One of the many challenges is the reduced visual performance in executing everyday tasks for older people compared to younger ones. The effect of age on human vision has been extensively studied in order to identify and understand the mechanisms of this degradation [6, 7]. The contrast sensitivity function (CSF), in particular, has become very popular within the vision research community, and thanks to recent advances in image processing, it becomes possible to model in real time its settings according to the effective age of the observer. In terms of displays, it allows to increase the image contrast following the preference of the observer.

Although the large variability of color and contrast perception among individual observers is a well-established fact, age is not the only factor that contributes to a change in the sensitivity of the HVS. In the same age group, the inter-observer variability of the sensitivity is high [8]. For instance, the response of the optical components of the HVS may be altered dramatically by surgical treatments. Furthermore, CIE-2006 age parameters do not match the real observer properties and cannot thoroughly describe age-induced effects of visual perception [9]. There is not enough correlation between the real age and the age-dependent parameters incorporated in CIE-2006 model [9].

In this study, we aim at develo** a unified visual method to correct the effect of age on color perception and contrast sensitivity, with a strong emphasis on the personalization of the viewing experience. Given the large differences between different viewers, we will introduce a method to determine the “effective age” of a specific observer. The rest of the paper is organized as follows. In Sect. 2, we will mention some of the previous studies in contrast and color enhancement, with more emphasis on those that target specific observers. Section 3 will provide a detailed overview of our method. In Sect. 4, we will describe the subjective user study we conducted, and discuss the results. We will finish in Sect. 5 by drawing some conclusion and paths for future studies.

2 Background

Color and contrast enhancement are among the most studied fields in image processing. Their aim is to improve the image visual quality based on a specific metric, a particular application, or targeted to a specific observer’s preference. Many algorithms have been proposed by the research community. One of the simplest techniques is histogram equalization, which does not provide good results in preserving the local details and natural look and color of the image [10]. To overcome these limitations, researchers developed other histogram algorithms such as local and adaptive techniques [11]. Another more sophisticated method is the Retinex theory and its variations [12, 13], whose objective is to imitate the perception of the human visual system by performing a comparison between the target pixel and the referenced white point.

Some of the techniques that attempt to improve the image content to a particular set of viewers have also been developed. Peli and Peli [14] were among the first to investigate the use of data from spatial frequency content for contrast enhancements targeting visually impaired people. Lawton [15] used the CSF of the target patient to adapt the content by enhancing the contrast to the most important frequencies of that individual. More recently, Choudhury and Medioni [16] presented an approach which aims at improving the visual quality of images for both people with normal vision and patients with low-vision. This method separates the image into illumination and reflectance components and then corrects only the illumination component while trying to achieve color constancy. In [17], an adaptive color and contrast enhancement method for digital images is proposed. The intensity, contrast and color are modified and applied to the three channels in RGB space, based on features of HVS. In [18], researchers proposed a data-dependent gamma correction for applications to elderly vision. This approach converts RGB components by setting the gamma according to each pixel value of the hue, saturation and lightness contrast.

All those methods - either addressed only one aspect of image enhancement - or did not incorporate a personalization aspect to include the observer preference. Our method, on the other hand, is designed to enhance an image color and contrast based on user preference. Moreover, it incorporates a luminance retargeting component that adapts to the ambient light and the target display peak luminance. Finally, it is designed to be fully compatible with the next generation of wide color gamut displays.

3 Age-Based Content Adaptation Method

Two parts of our age-based visual content adaptation method consist of a color modification based on the observer white preference and a contrast enhancement based on the observer’s preferred level of detail. The first stage of our method is a white balancing technique that allows for observers color variation, combined with a gamut expansion algorithm that maps the content from the sRGB color space to the target display wide gamut. The second stage consists of a retargeting technique that modifies the local contrast based on the observer preferred level of detail. The Fig. 1 presents the detailed components of our algorithm, which will be explained in the next section.

Fig. 1.

The diagram of the proposed age-dependent content adaptation algorithm

3.1 White Balance

We make use of the CIE-2006 model of age-based observer color-matching functions (CMF), which establishes a method for computing LMS cone-responses to spectral stimuli [19]. We use this model to discover the range of expected variation rather than predict responses from age alone.

The most important step in this first part is to adjust the display white point to correspond to the viewer’s age-related color response and preference. Our two inputs are CIE-2006 observer age and black body temperature. From these parameters and detailed measurements of the OLED RGB spectra and default white balance, we compute the white balance multipliers (r, g, b) using the procedure presented in Fig. 2.

Fig. 2.

The diagram of the adaptive white balance method

Display Calibration. The first step of the white balance method is to normalize the OLED primary spectra $r(\lambda )$, $g(\lambda )$ and $b(\lambda )$ so they sum to the current display white point $x_{w} = 0.3127$, $y_{w} = 0.3290$, $z_{w} = 1- x_{w} - y_{w}$. This is achieved by multiplying by the CIE 1931 standard observer curves $\bar{x}(\lambda )$, $\bar{y}(\lambda )$ and $\bar{z}(\lambda )$, and solving for the RGB scaling $\varvec{J}=\left( J_{r},J_{g},J_{b}\right) ^{T}$ that produces the measured xy-chromaticity.

$$\begin{aligned} M_{\bar{x}\bar{y}\bar{z} \rightarrow SPD_{r,g,b}}\varvec{J} = \left( \begin{array}{ccc} \bar{x}(\lambda ) r(\lambda ) &{} \bar{x}(\lambda ) g(\lambda ) &{} \bar{x}(\lambda ) b(\lambda ) \\ \bar{y}(\lambda ) r(\lambda ) &{} \bar{y}(\lambda ) g(\lambda ) &{} \bar{y}(\lambda ) b(\lambda ) \\ \bar{z}(\lambda ) r(\lambda ) &{} \bar{z}(\lambda ) g(\lambda ) &{} \bar{z}(\lambda ) b(\lambda ) \end{array} \right) \,\,\varvec{J} = y_{w}^{-1}\left( x_{w},y_{w},z_{w} \right) ^{T} \end{aligned}$$

(1)

where $M_{\bar{x}\bar{y}\bar{z} \rightarrow SPD_{r,g,b}}$ is the $3\times 3$ matrix from emission to response spectra. The normalized spectra are then given by: $(r,g,b)_{n}(\lambda ) = (r,g,b)(\lambda )J_{r,g,b}$.

Age-Based LMS. From the LMS cone responses $\overline{LMS}_{age}(\lambda ) = (\bar{l}_{age}(\lambda ), \bar{m}_{age}(\lambda ) , \bar{s}_{age}(\lambda ))$ for the given age, based on the CIE-2006 physiological model, and the black body spectrum for the specified target color temperature $M_{e}(\lambda ,T)$ ([20], p. 83), we compute the age-based LMS cone responses

$$\begin{aligned} \mathbf {w}_{\overline{LMS}} = \frac{\sum _{\lambda }\left[ \overline{LMS}_{age}(\lambda )M_{e}(\lambda , T)\right] }{\sum _{\lambda }\overline{LMS}_{age}(\lambda )} \end{aligned}$$

(2)

Matching Primary Settings. Next, we compute the $3 \times 3$ matrix corresponding to the LMS cone responses to the OLED RGB primary spectra, and solve the linear system to determine the RGB factors $\mathbf {x}= \left( x_{r}, x_{g}, x_{b} \right)$ that achieve the desired black body color match.

$$\begin{aligned} M_{LMS \rightarrow SPD_{n}(\lambda )_{r,g,b}} \mathbf {x}= \left( \begin{array}{ccc} \bar{l}(\lambda )r_{n}(\lambda ) &{} \bar{l}(\lambda )g_{n}(\lambda ) &{} \bar{l}(\lambda )b_{n}(\lambda ) \\ \bar{m}(\lambda )r_{n}(\lambda ) &{} \bar{m}(\lambda )g_{n}(\lambda ) &{} \bar{m}(\lambda )b_{n}(\lambda ) \\ \bar{s}(\lambda )r_{n}(\lambda ) &{} \bar{s}(\lambda )g_{n}(\lambda ) &{} \bar{s}(\lambda )b_{n}(\lambda ) \end{array} \right) \mathbf {x}= \mathbf {w}_{\overline{LMS}} \end{aligned}$$

(3)

Finally, divide these white balance factors by the maximum of the three, such that the maximum factor is 1. With $m = max(x_{r}, x_{g}, x_{b})$, we get the linear factors: $r = \frac{x_{r}}{m}$, $g = \frac{x_{g}}{m}$ and $b = \frac{x_{b}}{m}$.

These are the linear factors we will apply to each RGB pixel to map an image to the desired white point. Note that there are two degrees of freedom in the input, age and color temperature, and two degrees of freedom in the output, since one of the RGB factors is always 1.0.

Fig. 3.

Left: LMS cone responses using CIE-2006 age model. Right: Difference in D65 white appearance relative to a 25 year-old reference subject on a Samsung AMOLED display (Galaxy Tab S 10.5) for 2$^{\circ }$ and 10$^{\circ }$ patches.

The left graph in Fig. 3 shows the color matching functions (CMF) for some ages using the CIE-2006 age model. The right graph displays the difference in D65 white appearance relative to a 25 years-old reference subject on a Samsung AMOLED display (Galaxy Tab S 10.5) for 2$^{\circ }$ and 10$^{\circ }$ patches. These results illustrate the large variability among individual observers.

The result of this stage is an image compensated for the viewer white balance preference.

The next main goal of this first part is to map the image to the target display wide gamut. We use a hybrid color map** (HCM) [21] that is designed to preserve a selected region in chromaticity space while exploiting the larger gamut of the intended target display. This method preserves the earth and flesh tones while expanding the most saturated colors to the destination gamut.

3.2 Contrast Sensitivity Function

Human visual system ability to perceive and identify objects in the environment varies as a function of the object size, distance, contrast and orientation [7]. The contrast sensitivity function (CSF) extends and enriches the limited information given by the measures of acuity by evaluating the visual efficiency of an individual for the perception and identification of objects over a wide range of sizes, distances and orientations [7]. The CSF is usually derived by measuring the minimum contrast needed to detect sinusoidal grating patterns with different spatial frequencies expressed in cpd (cycles/degree) [7]. Since low-contrast thresholds are associated with high levels of visual sensitivity, the reciprocal of the threshold is calculated and plotted as a function of the spatial frequency.

Fig. 4.

Left: Illustration of the contrast sensitivity function and the threshold contrast. Right: sensitivity loss for an observer compared to our reference observer of 25 years. Spatial frequency is 10 cpd and the adaptation luminance at 100 $\text {cd}/\text {m}^2$

Numerous studies report a consistent pattern of change related to age in the CSF gathered under well-lit conditions (photopic vision) [4, 6]. For targets of spatial frequency between 4 and 18 cpd, the decline of the contrast sensitivity is about 0.3 log units on average across the second half of adult life [4, 5, 22]. The left graph in the Fig. 4 shows an example of the CSF as a function of spatial frequency (cpd), while the right graph shows the sensitivity loss between an observer of a given age a, and our reference observer of 25 years. The two graphs are computed with the help of Eqs. 7 and 8 (see Sect. 3.)

Local Contrast Retargeting. In this section, we will define the most important parts related to the local contrast retargeting. For a more complete treatment of the topic, please refer to the excellent studies in [23, 24].

Measures of contrast usually used by the vision community is the Michelson Contrast M, and the logarithmic contrast G when calculating image contrast in a multi-scale representation [24].

$$\begin{aligned} M = \frac{L_{max} - L_{min}}{L_{max} + L_{min}},\quad G = \frac{1}{2}log_{10}\left( \frac{L_{max}}{L_{min}}\right) \end{aligned}$$

(4)

where $L_{max}$ and $L_{min}$ are the maximum and the minimum luminance values of a sine wave.

The local contrast modification will follow the methodology developed by Wanat and Mantiuk [24], where the authors proposed a luminance retargeting technique that modifies the perceived colors and contrast of an image to match the appearance under different luminance levels. The authors formulated the condition for matching contrast as the matching of their supra-thresholds contrast, i.e. that the difference between the physical contrast M and the detection threshold $M_t$ must be constant across different luminance levels:

$$\begin{aligned} M - M_t = \tilde{M} - \tilde{M}_t \end{aligned}$$

(5)

where M and $\tilde{M}$ are the Michelson Contrast observed at different luminances.

The detection threshold $M_t$ is estimated by the CSF function

$$\begin{aligned} M_t = \frac{\varDelta L}{L} =\frac{1}{CSF_{hdrvdp}(\rho , L_a)} = \frac{1}{S\cdot CSF(\rho , L_a)} \end{aligned}$$

(6)

where $\rho$ is the spatial frequency in cycles per degrees, $L_a$ is the adaptation luminance in $\text {cd}/\text {m}^2$, and S is the absolute sensitivity of the HVS, necessary to adjust the CSF for a particular experiment. To compute the CSF from Eq. 6, we make use of the formula by Mantiuk et al. [23], and we assume it is equal to the CSF of our reference age of 25 years.

To integrate the age factor in Eq. 6, we employ a new model of the contrast sensitivity function [25], which is an extension of the one proposed by Barten [26] with age dependencies a.

$$\begin{aligned} CSF_B(\rho , L_a, a) = \frac{1}{m_{t}(\rho , a)} = \frac{M_{opt}(\rho , a)}{2k(a)}\sqrt{\frac{(XYT)(\rho )}{\varPhi _{ph}(a) + \varPhi _{0}(a)/M_{lat}^{2}(\rho , a)}} \end{aligned}$$

(7)

where $m_t$ is the modulation threshold, $M_{opt}(\rho )$ the optical MTF (modulation transfer function) that describes the behavior of the input signal passing through the optical elements of the eye, k the signal to noise ratio, X, Y, and T are the spatial and temporal integration area of the eye, $\varPhi _{ph}$ is the photon noise that describes the statistical fluctuations in the number of incident photons absorbed by the photoreceptors. $\varPhi _{0}$ is the neural noise, and $M_{lat}(\rho )$ is the lateral inhibition term [26]. The block diagram of the used model is shown in Fig. 5, where the components that are age dependent are depicted in green, and $m_n$ denotes the average modulation of the internal noise. We use the model from Eq. 7 to predict the sensitivity loss between our reference observer (25 years), and an observer at a given age a.

$$\begin{aligned} CSF_{loss}(\rho , L_a, a) = CSF_B(\rho , L_a, 25) - CSF_B(\rho , L_a, a) \end{aligned}$$

(8)

Fig. 5.

Block diagram of the processing of information and noise according to Barten’s CSF model [26] (Color figure online)

The CSF loss function will depend on both the spatial frequency f and the adaptation luminance $L_{a}$. Thus, the CSF from the Eq. 6 will be modified depending on the observer’s level of detail preference which corresponds to the observer’s effective age.

$$\begin{aligned} M_t =\frac{1}{CSF_{hdrvdp}(\rho , L_a, a)} = \frac{1}{CSF_{hdrvdp}(\rho , L_a) - CSF_{loss}(\rho , L_a, a) } \end{aligned}$$

(9)

Figure 6 shows CSF data measurements for two age groups (24y, 73y), and for 3 luminance levels (0.107, 3.38, 107  $\text {cd}/\text {m}^2$) as a function of the spatial frequency (in cpd) [5]. We notice the sensitivity loss for the elderly group compared to the young. The loss occurs at all luminance levels, and is more pronounced for higher spatial frequencies.

Fig. 6.

Left: Data measurements from the paper [5] for two age groups, 24y (continuous line) and 73y (dotted line), and 3 luminance levels. Right: Sensitivity loss (young vs old).

To perform the local contrast retargeting, the image is decomposed into low-pass and high-pass bands using the Laplacian pyramid, and computed for the log luminance values. The localized broadband contrast is defined by [24]:

$$\begin{aligned} c(x,y) = \sqrt{(g_\sigma *[l(x,y)-(g_\sigma * l)(x,y)]^2)(x,y)} \end{aligned}$$

(10)

where $*$ denotes the convolution and $g_\sigma$ is the Gaussian kernel and $\sigma$ is the standard deviation of $g_\sigma$.

The spatial frequency $\rho$ can be calculated from $\sigma$ (given in pixels) and the angular display resolution of the display $R_{ppd}$ (in pixels per visual degree) as:

$$\begin{aligned} \rho = 2^{-(k+1)}R_{ppd} \end{aligned}$$

(11)

where $k=1,...,N$ is the level of the pyramid, and $k=1$ represents the finest level.

Contrast retargeting is performed as a local enhancement of the Laplacian pyramid [24]:

$$\begin{aligned} \tilde{P}_k(x,y) = P_k(x,y).m_k(x,y) \end{aligned}$$

(12)

$P_k$ refers to the source image pyramid level, and m is the contrast modification defined by:

$$\begin{aligned} m_k(x,y) = \frac{c_k(x,y)-G(M_t)+G(\tilde{M_t})}{c_k(x,y)} \end{aligned}$$

(13)

The reconstruction of the enhanced image in terms of the contrast is done by summing all processed levels of the pyramids $\tilde{P}_k(x,y)$ with the addition of the base band (which was not modified by the local contrast step).

Figure 7 shows the result of our algorithm when applied to a test image. We easily notice the skin color and white are more natural in the processed image, as well as the details which are more pronounced. The compensated image looks natural in both color and contrast, and the content is personalized to the specific observer’s preferences.

Fig. 7.

Proposed unified content adaptation - Left: unprocessed, Right: compensated for both color and contrast. (Effective age for color: 57, contrast: 54). (Color figure online)

4 Experimental Validation

We subjectively evaluated the performance of our unified age-dependent content adaptation model using the pairwise comparison approach introduced in [27]. The experiment was run in the same room, which has a relatively constant illuminance of 50 Lux. Based on internal measurements, this corresponds to a target luminance of 32  $\text {cd}/\text {m}^2$ for the visualization tablet. This value was kept constant during the experiment. The visualization tablet is the Samsung AMOLED Galaxy Tab S 10.5, with a resolution of $2560 \times 1600$, and was at 50 cm distance from the observer, about 90$^{\circ }$ relative to the observer’s eyes.

To determine the observer’s effective age, the observer was presented with two videos that show the changing of an image’s color and details separately. In the first video, they were offered a 1-D axis control to find their preferred white point settings, which corresponds to the age-related color dimension. In the second video, their preferred level of detail setting, which corresponds to the age-related contrast dimension.

We used 10 images processed by 3 different color models, our unified color and contrast content adaptation -UCA, only color adaptation -CCA, and original image - SDS (same drive signal). For comparative studies, previous research [28] indicated that a size between 8 and 25 subjects is sufficient to provide statistically significant results. We asked 30 naive observers to compare the presented result. Our observers were asked to pick their preferred image of the pair. For each observer, 30 total pairs of images were displayed using the Samsung AMOLED Galaxy Tab S, 10 pairs for CCA:UCA, 10 pairs for CCA:SDS, and 10 pairs for SDS:UCA. Displaying the image pairs was randomized, and a black image is shown for 2 s between each consecutive image pairs. The observers were instructed to select one of the displayed images as their preferred image based on the overall feeling of color and details level. Our observers consist of 7 females and 23 males from the age of 23 to 60. On average, the whole experiment took about 10 min to complete.

Figure 8 shows processing results for color and contrast (UCA) with original images (SDS) and only color content adaptation (CCA). A few of the images include both indoor and outdoor scenes with a lot of details, and portraits where people’s faces are close to the camera. Our method keeps the white color, skin and earth tones natural while enhancing the local contrast for better visualization.

We used the pairwise comparison method with just noticeable difference (JND) evaluation in our experiment. This method has been recently used for subjective evaluation in the literature [24, 27, 29]. We used the Bayesian method of Silverstein and Farrel [30], which maximizes the probability that the pairwise comparison results accounts for the experiment under the assumptions of equal variances and uncorrelated distributions. During an optimization procedure, a quality value for each image is calculated to maximize the probability, modeled by a binomial distribution. Since we have 3 conditions for comparison (UCA, CCA, SDS), this Bayesian approach is suitable. It is more common when we compare a large number of conditions and it is known for being robust to unanimous answers.

Fig. 8.

Content adaptation examples with original images. UCA - our unified content adaptation, SDS - original image, and CCA - only color content adaptation. (Effective age for color: 38, contrast: 40) (Color figure online)

Fig. 9.

Subjective evaluation results of pairwise comparison representing as JND values for each 10 images including error bars which denote 95% confidence intervals calculated by bootstrap**. UCA: our proposed unified age-based color and contrast visual content adaptation, SDS: original image, and CCA: only color content adaptation. (Color figure online)

Figure 9 shows the results of the subjective evaluation calculating the JND value using the definition in [27]. For discriminating between choices, only relative differences can be used. Thus, the absolute JND values are not very relevant taken alone. A method that has a high JND value is desirable over methods with smaller JND values, where 1 JND corresponds to 75% discrimination threshold. The Fig. 9 represents the JND value for each scene, rather than the average value, because JND is a relative value that can also be meaningful when compared with others. In the Fig. 9, we also represent the confidence intervals with 95% probability for each JND. To calculate the confidence intervals, we used a numerical method, known as bootstrap** [31]. We generated 500 random sampling, then computed 2.5$^{th}$ and 97.5$^{th}$ percentiles for each JND point. Because both JND and confidence intervals for JND are relative values, the JND values for the mode we choose as a reference, SDS, are equal. For 9 of the images, our proposed unified age content adaptation is the most preferred method with JND differences of 0.31−2.2 between it and the second most preferred method. We also notice when only color is compensated, observers preferred the unprocessed in 6 of the images. The reason of this result in our opinion, is that people are more familiar in seeing unprocessed images in a variety of displays. For instance, in the images where there is water (Trevi), and sky (Colosseum), the colors in the processed images are very different from the SDS mode, which makes it difficult for people to decide without having a reference.

5 Conclusion

We presented a method that adapts the color and the contrast to the observer’s effective age, and showed that this method was preferred over adapting the color only (CCA) and same drive signal (SDS) methods on average in images containing natural white and high detail levels. The key to our method is to merge a white balance age-based adjustment procedure with a local contrast enhancement based on a CSF model that incorporates the observer’s effective age. In the future, we hope to refine our procedure to determine the observer’s effective age and extend our approach for all ranges of luminance (particularly mesopic and scotopic vision).