Non-Photorealistic Rendering for High Dynamic Range Images

1


To mimic visual perception of human painters, a
non-photorealistic rendering (NPR) method based on
strokes is proposed for rendering high dynamic range
images (HDR). It enables smooth guidance of the
strokes generated in the rendering process on
account of the improvement in computing the
gradient values especially in the regions that have
particularly high or low luminance values.
Experimental results show that angles of gradient
values obtained from HDR images have a reduction
in the averaged error of up to 57.5% in comparison to
that of conventional digital images. A partial
experiment on incorporating HDR images into other
NPR styles such as dithering shows the wide
compatibility of HDR images in providing the source
information for a NPR process.

Keywords: Image Processing, High Dynamic Range,
Rendering, Non-Photorealistic


1. INTRODUCTION
In contrast to the conventional quest for realism in the field
of rendering, a new quest for creating imageries which are
more artistic and easy for communication is growing [1].
Rendering methods developed for enabling a variety of
expressive styles are referred to as NPR [2]. Typical NPR is
usually inspired by artistic works e.g. the painting [3],
pen-and-ink illustration [4], and so on. Digital images of
real scenes are usually utilized as the reference for
generating a NPR result [1]. Limited dynamic range of
digital images, however, imposes restriction on the
performance of NPR methods. With the development of
image processing technologies, HDR images have attracted
attentions from researchers [5][6][7][8][9]. The limitation
on performance of NPR methods resulting from low
dynamic range of digital images can be removed when HDR
images are incorporated into the rendering process.
The proposal mainly focuses on blending HDR images into
conventional stroke-based NPR methods such as painterly
rendering proposed by Hertzmann [3]. The Achromatic
brightness channel of a HDR image provides the gradient
value for guiding the path of strokes. The modification in
the stroke-based rendering process [3] is supposed to obtain
longer strokes, which is rewarding in such a rendering style.
A tone mapped HDR image or a well-exposed digital image
of the scene is chosen to provide color information for the
rendering process depending on the scene. These changes in
source image of the rendering process enable a preservation
of details in resulting images.
In order to evaluate the proposal, an image dataset
composed of three scenes are built. Firstly, errors of angles
occurred in gradient computation for both a HDR image and
generated digital images are figured out by simulating a
photo capturing process in which the radiance map used for
this experiment is a superposition of three Gaussian
functions. Then, improvement in length of strokes after
blending HDR images into a typical stroke-based rendering
method [3] is also tested on the built image dataset.
Relevant concepts of HDR and NPR, the evaluation
criterions and the proposed rendering method are covered in
2. In 3, details for evaluation through experiments and
intuitive rendering results of the image dataset are
demonstrated.
2. EVALUATION OF HDR GRADIENT AND
FLOW OF NPR FOR HDR IMAGES
2.1. High Dynamic Range Imaging
In real world, light strength in different regions of the same
scene may vary acutely; therefore, the whole dynamic range
cannot be effectively covered by normal roll films or CCD
sensors. Solutions ranging from hardware [10] to image
processing techniques [6] have been proposed to capture the
dynamic range of scene radiance in its entirety, among
which the method proposed by Debevec et al [6] for
constructing HDR images is widely accepted due to its
easiness to implement and its compatibility for different
image acquisition hardware. There are mainly two steps for
the method proposed by Debevec et al [6]. Firstly, the
response curve of the imaging sensor is obtained through a
recovery process by using a set of photographs with
different shutter time. Recovering the response curve of an
image sensor is realized by minimizing the objective
function
( ) ( ) | | { }

= =
=
N
i
P
j
j i ij ij
t E Z g Z w
1 1
2
ln ln
( ) ( ) | |

+ =
' ' +
1
1
2
max
min
Z
Z z
z g z w (1)
where
i
E
and
i
t A
denote the radiance of the pixel at
location i and the shutter time of image j, ( ) z w is a
Jiajun LU
*1
, Fangyan DONG
*1
, and Kaoru HIROTA
*1


*1
Dept of Computational Intelligence & Systems Science, Tokyo Institute of Technology,
G3-49, 4259 Nagatsuta, Midori-ku, Yokohama 226-8502, Japan
E-mail: {lu, tou, hirota}@ hrt.dis.titech.ac.jp

Non-Photorealistic Rendering for
High Dynamic Range Images
Non-Photorealistic Rendering for High Dynamic Range Images


2


weighting function for different intensity values, and g(z) is
the log inverse function of the sensor response function. The
second step is the fusion process to generate the radiance
map by combining the set of differently exposed
photographs using the recovered response curve.
To display HDR images in normal displays other than those
designed for professionals, contrast ratio of HDR images
has to be compressed. The process of compressing the HDR
images is usually referred to as tone mapping, and methods
such as bilateral filtering [7], gradient domain compression
[9] have been put forward to accomplish this task. After
tone mapping, a compressed version of a HDR image is
obtained. For well-designed tone mapping algorithms, both
the color information and the details of the original scene
can be effectively preserved.
2.2. Stroke Guidance by Gradient of HDR Images
Non-photorealistic rendering methods usually simulate the
painting process of human painters for generating a
rendering result [1]. Painterly rendering [3] is one typical
NPR method, of which a rendering result is shown in Fig. 1.


Fig. 1 Rendering result of lena in a painterly style.

The rendering process for painterly style imitates the
placement of strokes onto the canvas as skilled human
painters. A path for guiding each stroke consists of a set of
control points, and relative position for sequential control
points is determined by the gradient values of the realistic
image [3]. In this paper, this methodology for computing the
guiding path of long strokes is inherited, but images used
for computing gradient are replaced by the radiance map of
HDR images.
An intuitive explanation for the improvement in precision
for gradient computation of HDR images can be that the gap
between discrete gray levels in conventional digital images
are smoothen if a HDR image is used for gradient
computation. The quantitative evaluation for the accuracy
of gradient computation is given in 3.
2.3. Evaluation for Gradient Computation of HDR
Images
Its usually hard to quantitatively evaluate the performance
of NPR algorithms [11]. As is mentioned in 2.2, gradient
values play a crucial role in guiding the strokes in the
painterly rendering process [3]. So, instead of directly
evaluating rendered images, errors occurred in gradient
computation of HDR images are estimated.
For any real scene, its difficult to obtain its radiance map.
Thus, the gradient value of the radiance map is also not
easily obtainable. Instead of using digital images shot in a
real scene, a set of virtual images are generated by
simulating the imaging process of a typical digital camera.
The light intensity (radiance map) of the closed form of the
virtual scene ( ) y x I
F
, is given. The only tunable parameter
in the simulation process is the shutter time. Using a set of
predefined shutter time, a virtual image set
( ) ( ) { } n m I n m I I
k
, ,..., ,
1
=
is generated. The HDR image
( ) n m I
HDR
, is constructed from this generated image set I
using the reconstruction method proposed by Debevec et al
[6].
The angle of the exact gradient value in a fixed location
( ) n m, of the virtual scene can be computed using
( )
( ) ( )
n y m x
F F
F
x
y x I
y
y x I
n m
= =

|
|
.
|

\
|
c
c
c
c
=
,
1
, ,
tan , u
. (2)
In order to compute the angle of the gradient value in
location ( ) n m, of a HDR image, Sobel operator is utilized
in accord with gradient computation method used in [3].
The derivative value of ( ) n m I
HDR
, along horizontal
direction is computed as
( ) ( ) n m I n m Gx
HDR HDR
,
1 0 1
2 0 2
1 0 1
, -
(
(
(

=
. (3)
Similarly, the derivative value along vertical direction is
calculated using
( ) ( ) n m I n m Gy
HDR HDR
,
1 2 1
0 0 0
1 2 1
, -
(
(
(

=
. (4)
Then, the formula used for computing the angle of the
gradient value of ( ) n m I
HDR
, is given
( ) ( ) ( ) ( ) n m Gx n m Gy n m
HDR HDR HDR
, , tan ,
1
= u . (5)
The gradient value of any generated image ( ) n m I
NOR
, in
image set I can be computed as
( ) ( ) ( ) ( ) n m Gx n m Gy n m
NOR NOR NOR
, , tan ,
1
= u . (6)
Then, error occurred in computing the gradient value of
( ) n m I
HDR
, is measured by figuring out the averaged
absolute difference between ( ) n m
HDR
, u and ( ) n m
F
, u as
( ) ( )

= =
= A
M
m
N
n
HDR F HDR
n m n m
MN
1 1
, ,
1
u u u , (7)
where M and N denote number of pixels in columns and
rows for the HDR image.
For evaluation of the error occurred in gradient
computation of ( ) n m I
NOR
, , the averaged absolute
difference
NOR
u A is calculated in the same manner as
HDR
u A . More details of this evaluation process are given in
3.
2.4. Workflow for NPR of HDR Images
A workflow for the proposed rendering process of the HDR
images is given in Fig. 2. The basic structure of this method
is inherited from the painterly rendering method proposed
by Hertzmann [3].
Non-Photorealistic Rendering for High Dynamic Range Images


3




Fig. 2 Rendering process for HDR images.
The first step is to construct a HDR image through an image
fusion process using the method proposed by Debevec et al.
[6]. Then, gradient values and a tone-mapped image of the
constructed HDR image are separately computed. Gradient
values of the HDR image are calculated to guide the strokes
in the rendering process. The tone-mapped image is used to
provide the color information for rendering. Well
established tone mapping algorithms [7][9] can be utilized
for compressing the HDR image. Note that a well-exposed
image of the scene can also be used to provide the color
information instead of a tone-mapped image depending on
the illumination condition of the scene. Combining the
gradient values and the color information from the
tone-mapped image, a rendered image is achieved by
placing all the strokes.
3. EXPERIMENTS ON GRADIENT OF HDR
IMAGES AND RESULTS OF NPR
3.1. Error Estimation in Gradient Computation of
HDR Images

Gradient value of HDR images plays an important role in
guiding strokes in proposed NPR method. In the first
experiment, errors occurred in gradient computation of
HDR images are estimated.
The images used for constructing the HDR image are
generated by simulating the imaging process of a digital
camera as mentioned in 2.3. The scene radiance function
( ) y x I
F
, used for generating virtual images is the
superposition of three two-dimensional Gaussian functions
( ) ( ) ( ) ( ) y x g y x g y x g y x I
F
, , , ,
3 3 2 2 1 1
o o o + + = , (8)
where
1
,
2
and
3
are scaling factors for each of the three
Gaussian functions, of which the peak and standard
deviation are listed in Table 1. A plot of the scene radiance
function ( ) y x I
F
, is depicted in Fig. 3.

Table 1: Parameters for scene radiance function ( ) y x I
F
,
.

Gaussian
Function
Gaussian
Peak
Standard
Deviation
g
1
(160,50) 60.0
g
2
(80,190) 60.0
g
3
(240,190) 60.0


Fig. 3 A 3-D plot for the scene radiance function ( ) y x I
F
, of
which each value is given by (8).

In this experiment, a set of virtual images is shot by
simulating the process of taking digital images. The shutter
time is tuned for generating each individual photograph.
Shutter time used in this experiment is listed in Table 2.

Table 2: Shutter time used in generating virtual images.

Image No. 1 2 3 4 5 6 7
Shutter time (s) 0.6 0.8 1.0 1.2 1.5 2.0 2.5

Two samples from the generated image set I are given in
Fig. 4 along with their shutter time. Then, a HDR image is
constructed from this set of differently exposed images
using the response curve obtained by minimizing the
objective function (1) [6]. Response function of the imaging
sensor for this simulation is given as
( )
( ) ( )
( ) ( )

<
< s
< s + +
>
=
0 3 0
0 0 3 10 1 ln 13 37 5 127
0 3 0 5 127 10 1 ln 13 37
0 3 255
. if ex ,
ex . if - , ex * . . round
. ex , if . ex * . round
. if ex ,
ex f
, (9)
where ex is the log exposure of the radiance value.
Non-Photorealistic Rendering for High Dynamic Range Images


4


After a HDR image is constructed from the generated image
set I, ( ) n m
F
, u , ( ) n m
NOR
, u and ( ) n m
HDR
, u are figured
out using (2), (5) and (6).


(a) 0.6s (b) 1.5s
Fig. 4 Two generated images with shutter time.

For each pixel, ( ) ( ) n m n m
HDR F HDR
, , u u u = A and
( ) ( ) n m n m
NOR F NOR
, , u u u = A are computed to measure
errors which occur in gradient computation. If the absolute
angle difference
HDR
u A
or
NOR
u A
is larger than 90, then its
value is replaced with
HDR
u A 180 or
NOR
u A 180 . A
plot for both
HDR
u A
and
NOR
u A
is demonstrated in Fig. 5.
There exist a few locations with higher values than the rest
vast majority. This is due to the fact that the derivative along
horizontal direction is approaching zero on these locations,
which results in rapid change in gradient values.


(a)

(b)
Fig. 5 A plot for
HDR
u A
and
NOR
u A
. (a) Errors in computing the
gradient value of the constructed HDR image. (b) Errors in
computing the gradient values of a generated image (shutter time =
0.6) from the image set I.

HDR
u A
and
NOR
u A
are calculated using (7) to give a
quantitative evaluation for the performance in terms of the
averaged error in gradient estimation of HDR images, of
which the results are shown in Fig. 6. In comparison to the
lowest value of
NOR
u A
(shutter time = 0.6) among the
generated image set I,
HDR
u A achieves a reduction of error
by nearly 57.5%.


Fig. 6 Averaged error in gradient computation of the HDR image
(
HDR
u A ) and the generated images (
NOR
u A
). For all the
generated images in image set I, their
NOR
u A
values are
computed.

3.2. Evaluation of Stroke Length
In the second experiment, length of the strokes in rendering
process is chosen as the criterion for evaluation.
As briefly mentioned in 2.4, the proposed rendering method
is formed by blending the gradient estimation mechanism of
HDR images into painterly rendering method [3]. The
length of strokes formed in the proposed rendering process
is compared with the original method [3] to show the
change in stroke length.
An image dataset of three different scenes is built as
depicted in Fig. 7. Note that only a portion of the images for
each scene is demonstrated.


(a) (b) (c)

(d) (e) (f)

(g) (h) (i)
Fig. 7 Image sets used for constructing HDR images.

Non-Photorealistic Rendering for High Dynamic Range Images


5


In the painterly rendering method [3], a smoothly filtered
version of the original image is used for guiding the strokes.
To give scope to the gradient values of HDR images, the
original HDR image are directly utilized without filtering
for computing the gradient value in the proposed method. In
this experiment, the smoothing operation in the painterly
rendering [3] is also cancelled for a more equitable
comparison.
The tone mapping process of HDR images are done using
the bilateral filtering method proposed by Durand and
Dorsey [7].
Histograms of the stroke length information for rendering a
bowl (scene of the second row in Fig. 7) using the original
method and the proposed method are listed in Fig. 8. In the
rendering process for this HDR image, a well-exposed
digital image of the scene is used for providing the color
information instead of the tone mapped HDR image.


Fig. 8 Histogram of length of strokes formed in the adjusted
oil-painting like rendering process.

As is shown in Fig. 8, the length of strokes has a tendency of
becoming longer if a HDR image is used in providing the
gradient information for guiding the strokes. The minimum
length for a stroke in the rendering process is set to 7, but
some strokes with lengths shorter than 7 exist. This is
mainly due to the difficulty in computing the gradient value
in some uniform regions inside an image and also due to
short strokes originated from the locations close to the
border of the image.
If smoothing filtering is added before gradient computation
both in the proposed rendering method and in the painterly
rendering method [3], then the stroke length formed in both
methods are almost the same. The limited improvement in
gradient values of HDR images can be easily influenced by
the smoothing operation before computing the gradient
value for guiding the strokes. And the smoothing operation
can provide a better guidance for the strokes in some
uniform regions of an image by propagating the intensity
value of each pixel.

3.3. Rendering Results for HDR Images
The rendered results for three scenes in Fig. 7 are depicted
in Fig. 9. For all the three scenes shown in the left column,
color information in the rendering process of the HDR
image is provided by tone mapped HDR images using a
bilateral filtering method [7]. For the rendering results
shown in the right column, normal 8bit color digital images
are the provider of the color information.
Comparing the results shown in Fig. 9(a) and Fig. 9(b), the
holder of the cup rendered from HDR image (left side)
demonstrates an improvement in structure and color. For the
scene shown in Fig. 9(e) and Fig. 9(f), the rendering result
for HDR image of the scene (left side) preserves more
details especially in the window region, which is closer to
humans perception of vision. Also the color of Fig. 9(e) is
more vivid than that of Fig. 9(f). The rendering results of
HDR images seem to be darker than those of the
conventional 8bit color digital images, which is mainly due
to the tone mapping methods applied in this experiment.


(a) (b)

(c) (d)

(e) (f)
Fig. 9 Rendering results of HDR images and normal images for
three scenes. Rendered results in the left column are based on
HDR images, and the results in the right column are obtained from
normal digital images.

For other stroke based rendering methods like the
decorative mosaic method proposed by Hausner [13], the
proposal can also be applied to preserve more details of the
scene. The result for rendering HDR images from the third
scene shown in Fig.7 using the decorative mosaic method is
shown in Fig. 10 in comparison with that of a normal digital
image. The orientations of the mosaics in Fig. 10(a) are
determined using the gradient values of the HDR image.
And the colors of the mosaics are given by the tone mapped
HDR image.


(a)
Non-Photorealistic Rendering for High Dynamic Range Images


6



(b)
Fig. 10 Rendering results of a HDR image and a normal image
using the decorative mosaic style. (a) Rendering results of a HDR
image. (b) Rendering results of a normal image.

The mosaics formed in Fig. 10(a) convey more color and
structural information of the scene than those in Fig. 10(b).
For other NPR styles like dithering [1], tone mapped HDR
image can also be used as a good information source for the
rendering process. In Fig. 11, the rendering result in
dithering style for both a HDR image and a conventional
digital image is shown. The scene used in Fig. 11 is
displayed in the third row of Fig. 7.


(a) (b)
Fig. 11 Rendering results of HDR image and normal image in
dithering style. (a) Rendering results for a HDR image of the third
scene in Fig. 7. (b) Rendering results for a conventional image of
the third scene in Fig. 7.

Improvement in stroke placement and preservation of
details are the two main benefits when applying the
proposed rendering method for HDR images. Apart from
the rendering styles mentioned in this paper, this idea can be
further extended to varying NPR methods, which may
provide a new choice for those who want to achieve an
impressive demonstration of a certain scene.
4. CONCLUSION
A stroke-based NPR method for HDR images is provided in
this paper. Experimental results show that a HDR image can
achieve a reduction of 57.5% in terms of angle error of
gradient values compared with the gradient values obtained
from conventional digital images. Change in length of
strokes is observed when the proposed framework is
compared with the painterly rendering style proposed by
Hertzmann [3], using gradient value computed from HDR
images in its stroke guidance step. Preservation of details in
over-exposed regions of the scene can be easily observed in
the rendered images. This detail preservation property of
HDR images in NPR process is crucial in reflecting human
perception in visual information. For designers and artists
who seek approaches to more natural illustrations of certain
scenes in non-photorealistic styles, potential rendering tools
can be extended from the proposed framework. Moreover,
how other aspects of HDR images like color and texture
may influence NPR process are to be further studied.

Acknowledgements
This work has been supported by Japan Society for the
Promotion of Science (JSPS) under grant KAKENHI
B21300080.

REFERENCES:
[1] T. Strothotte and S. Schlechtweg, Non-Photorealistic Computer
Graphics: Modeling, Rendering and Animation, San Francisco:
Morgan Kaufmann, 2002.
[2] http://en.wikipedia.org/wiki/Non-photorealistic_rendering
[3] A. Hertzmann, Painterly Rendering with Curved Brush Strokes of
Multiple Sizes, In Proc. of SIGGRAPH98, pp. 453-460, 1998.
[4] G. Winkenbach, and D. H. Salesin, "Computer-Generated
Pen-and-Ink Illustration", In Proc. of SIGGRAPH 94, pp. 91-100,
1994.
[5] W. Zhang, X. An, and S. Pan, An Improved Recursive Retinex for
Rendering High Dynamic Range Photographs, In Proc. of
ICWAPR11, pp. 121-125, 2011.
[6] P. E. Debevec and J. Malik, Recovering High Dynamic Range
Radiance Maps from Photographs, In Proc. of SIGGRAPH 97, pp.
369-378, 1997.
[7] F. Durand and J. Dorsey, Fast Bilateral Filtering for the Display of
High-Dynamic-Range Images, ACM Transactions on Graphics, vol.
21, no. 3, pp. 257-266, 2002.
[8] C. Schlick, Quantization Techniques for Visualization of High
Dynamic Range Pictures, In Proc. of the 5th Eurographics
Rendering Workshop, pp. 7-20, 1994.
[9] R. Fattal, D. Lischinski, and M. Werman, Gradient Domain High
Dynamic Range Compression,ACM Transactions on Graphics, vol.
21, no. 3, pp. 249-256, 2002.
[10] H. Graf, C. Harendt, T. Engelhardt, C. Scherjon, K. Warkentin, H.
Richter, and J. N. Burghartz, High Dynamic Range CMOS Imager
Technologies for Biomedical Applications, IEEE Journal of
Solid-State Circuits, vol. 1, no. 1, pp. 281-289, 2009.
[11] A. Santella and D. DeCarlo, Visual Interest and NPR: an Evaluation
and Manifesto, In Proc. of NPAR04, pp. 71-78, 2004.
[12] P. M. Hall, J. P. Collomosse, Y. Song, P. Shen, and C. Li, RTcams:
A New Perspective on Non-Photorealistic Rendering from
Photographs, IEEE Transactions on Visualization and Computer
Graphics, vol. 13, no. 5, pp. 966-979, 2007.
[13] A. Hausner, Simulating Decorative Mosaic, In Proc. of
SIGGRAPH 01, pp. 573-578, 2001.