AI and Cultural Heritage

Semantic
Classification
of Byzantine Icons
Paraskevi Tzouveli, Nikos Simou, Giorgios Stamou, and Stefanos Kollias,
National Technical University of Athens

R
ecent advances in Web technologies that ensure quick, effective infor-

mation distribution have created the ideal terrain for spreading vari-

ous cultures through cultural-content presentation. Nowadays, we are wit-

This system uses nessing the publication of a huge amount of cultural information on the Web.

fuzzy description In the past, people gathered cultural in-
formation from physical objects (such as
logics and patterns books, sculptures, statues, and paintings).
Now, digital collections have become com-
to automatically mon practice for most cultural-content
providers.
determine the sacred However, as the amount of the Web’s cul-
tural content grows, search and retrieval
figure depicted procedures for that content become increas-
ingly difficult. Moreover, Web users need
in an icon. more efficient ways to access huge amounts
of content. So, researchers have proposed so-
phisticated browsing and viewing technolo-
gies, raising the need for detailed metadata
that effectively describes the cultural con-
tent. Several annotation standards have been
developed and implemented, and Semantic
Web technologies provide a solution for the
semantic description of collections on the
Web.1 Unfortunately, the semantic annota-
tion of cultural content is time consuming
and expensive, making it one of the main
difficulties of cultural-content publication.
Therefore, the need for automatic or semi­
automatic analysis and classification of cul-
tural assets has emerged.
To meet this need, researchers have pro-
posed using image analysis methods (for
some examples, see the sidebar on the next
page). However, these methods use domain
knowledge for only low-level analysis. Also,
two main difficulties have arisen in using
image analysis to automate annotation and
classification of cultural digital assets. The
first is the failure of semantic image seg-
mentation and image analysis algorithms in
some real-world conditions. This is due to
the high variability of content and environ-
mental parameters (luminance and so on),
which makes the problem complex. The sec-
ond difficulty is the extensive and vague na-
ture of domain knowledge (at least for some
cultural content), which complicates formal
knowledge representation and reasoning.
However, some cultural domains are ap-
propriate for automatic analysis and classi-
fication methods. Byzantine icon art is one
of them. The predefined image content and
the low variability of the image characteris-
tics support the successful application of im-
age analysis methods. Consequently, we’ve
developed a system that exploits traditional

March/April 2009 1541-1672/09/$25.00 © 2009 IEEE 35

Published by the IEEE Computer Society
AI and Cultural Heritage
Related Work on Image Analysis for Cultural Heritage
T
he July 2008 IEEE Signal Processing Magazine pre-
sented some particularly interesting applications of
image analysis to the cultural-heritage domain. Ale-
jandro Ribes and his colleagues described problems con-
cerning the design of multispectral cameras as well as the
analysis of the Mona Lisa in the context of the digitization
of paintings.1 Anna Pelagotti and her colleagues proposed
a novel multispectral digital-imaging technique for clus-
tering image regions in a set of images containing similar
characteristics when exposed to electromagnetic radia-
tion. 2 Their technique provides material localization and
identification over a painting’s entire surface. C. Richard
Johnson and his colleagues described how three research
groups developed systems that identify artists on the basis
iconographic guidelines to automatically analyze icons.
Our system detects a set of concepts and properties de-
picted in the icon, according to which it recognizes a sa-
cred figure’s face, thus providing semantic classification.
Byzantine Iconography
Byzantine art refers to the art of Eastern Orthodox states
that were concurrent with the Byzantine Empire. Certain
artistic traditions born in the Byzantine Empire, particu-
larly regarding icon painting and church architecture, have
continued in Greece, Bulgaria, Russia, and other Eastern
Orthodox countries up to the present. The icons usually
depict sacred figures from the Christian faith, such as Je-
sus, the Virgin Mary, the apostles, and the saints.
At the beginning of the 17th century, iconography man-
uals represented the most important source of inspiration
for icon painters. The 16th-century monk Dionysios from
Fourna is credited as the author of the prototypical man-
ual, Interpretation of the Byzantine Art. 2 In it, he defines
the rules and iconographic patterns that painters still use
to create icons.
Byzantine iconography follows a unique convention of
painting. The artistic language of Byzantine painters is
characterized by apparent simplicity, overemphasized flat-
ness, unreal and symbolic colors, lack of perspective, and
strange proportions. The sacred figures are set beyond real
time and real space through the use of gold backgrounds.
The most important figure in an icon is depicted frontally,
focusing on the eyes, facial expression, and hands. Clarity
is the rule, not only in describing shapes but also in arrang-
ing figures in balanced compositions so that the narrative’s
actions are clear. The figures have large eyes, enlarged ears,
long, thin noses, and small mouths, each painted following
specific rules.
When depicting various figures, the artist is obliged to
observe the manual’s rules regarding posture, hair and
beard style, apparel, and other attributes. Specific sacred
36 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS
of brushwork characteristics that art historians use for au-
thentication. 3 Finally, Howard Leung and his colleagues re-
ported on the preservation of ancient Chinese calligraphy. 4
References
1. A. Ribes et al., “Studying That Smile,” IEEE Signal Processing
Magazine, vol. 25, no. 4, 2008, pp. 14–26.
2. A. Pelagotti et al., “Multispectral Imaging of Paintings,” IEEE
Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 27–37.
3. C.R. Johnson et al., “Image Processing for Artist Identifica-
tion,” IEEE Signal Processing Magazine, vol. 25, no. 4, 2008,
pp. 37–48.
4. H. Leung, S.T.S. Wong, and H.H.S. Ip, “In the Name of Art,” IEEE
Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 49–54.
figures of the Christian faith are defined by specific facial
characteristics. For example, according to Dionysios, Jesus
is illustrated as a young figure with long, straight, thick,
black hair and a short, straight, thick, black beard.
According to these rules, the area in which the sacred
figure will be painted is separated into equal segments. The
first segment contains the head, focusing on the eyes and
facial expression. This segment contains the sacred fig-
ure’s most distinguishable features—for example, hair and
beard style. The remaining segments contain the sacred fig-
ure’s body, focusing on the hands and arms. The body can
be depicted in different poses—for example, standing, sit-
ting, from the head to the waist, and from the head to the
thorax.
The head should be elliptical. The major axis is along
the vertical part of the head and is equal to four times the
height of the nose (H). (According to Dionysios, the nose
serves as the metric for producing harmonious facial and
body proportions.) The minor axis is along the horizontal
and is equal to 3H. The head segment can be separated into
four equal vertical sections of height H. The hair is in the
first part, the forehead is in the second, the nose is in the
third, and the area between the mustache and the chin is in
the fourth.
A Byzantine painter applies four base colors in layers on
the face segment. The darking is a dark color for the base
of the face segment and the darkest shadows. The proplas-
mos is a less-dark color for lighter shadows. The sarkoma
is for flesh, and the lightening is for highlights, containing
a good amount of white and being light enough to show off
against the base color. The absolute value of these colors
differs from icon to icon.
From Art Manual to Semantic Representation
Although the knowledge in Dionysios’s manual concerns
vague concepts such as “long hair,” “young face,” and so
on, it’s quite strict and formally described. Consequently,
we can create an ontological represen- Sacred-figure
Feature recognition
tation of this knowledge using OWL. extraction
In this way, the ontology’s axiomatic Semantic
skeleton will provide the terminology segmentation
and restrictions for Byzantine icons. Semantic
The only problem is that even a interpretation
well-tuned image analysis algorithm Byzantine icon
can’t produce perfect results for a analysis
wide set of data with different char- Byzantine
icon
acteristics. Moreover, many Byzan- database
tine icons might be misinterpreted
owing to their age, further affecting Assertional
knowledge
the results. Finally, the vague nature
Reasoning
of terms such as distance, color, and engine
length introduces fuzziness in the cat- Terminological
egorization of objects. For example, knowledge
sometimes a sacred figure represents Knowledge representation
a young face, but only to a certain and reasoning
degree. In such cases, standard de-
scription logics (DLs) are insufficient
because they can’t deal with vague,
uncertain information. To handle Figure 1. The Byzantine icon classification system’s architecture. The analysis
such problems, researchers have pro- module extracts information about the icon, and the knowledge representation and
posed fuzzy DLs, based on fuzzy sets reasoning module uses this information to infer implicit knowledge that categorizes
and fuzzy logic, which are suitable for the icon.
handling imprecise information. 3
To classify the important facial features of Byzantine a large set of heuristics that facilitated the design of the
icons and detect the sacred figures having specific charac- subsystem modules and helped us select the most ap-
teristics, we combine image analysis algorithms with the propriate semantic segmentation and feature extraction
expressive power of fuzzy DLs. Using the rules and pat- methods.
terns described in Dionysios’s manual, we constructed a
fuzzy knowledge base that’s populated by the results of im- Semantic segmentation. This module divides the icon into
age analysis of the digitized paintings. Using fuzzy reason- regions visualizing the icon’s physical objects and their
ing and on the basis of the defined terminology, our system parts.
can classify the figures in Byzantine icons. The first step of semantic segmentation is face detection.
For this, we adopted a fast method based on support vec-
The Icon Classification System tor decision functions.4 This method first uses integral im-
Figure 1 shows the general architecture of our Byzantine aging for very fast feature evaluation based on Haar basis
icon classification system. It consists of the Byzantine icon functions. Then, it uses a simple, efficient classifier based
analysis subsystem and the knowledge representation and on the AdaBoost learning algorithm to select a small num-
reasoning subsystem. ber of critical visual features from a very large set of po-
tential features. Finally, this method combines classifiers
Byzantine Icon Analysis in a “cascade,” quickly discarding the image’s background
This subsystem performs the image processing and analy- regions while concentrating computation on promising
sis tasks necessary to detect a set of primitive concepts and face-like regions. An example of this method’s results is
properties (such as face and long hair) and form assertional the green rectangle in the first image of the face detection
knowledge that semantically describes the icon’s content. It box in Figure 2 (see next page).
consists of modules for semantic segmentation, feature ex- The next step is localization of the eyes. Be-
traction, and semantic interpretation. cause the nose’s height should be equal to the dis-
The first step of system development was the interpre- tance of one retina from the other, that distance is a
tation, with the aid of Byzantine iconography experts, critical parameter for segmentation. The semantic-
of Dionysios’s manual. This procedure provided us with segmentation module performs eye localization with the

March/april 2009 www.computer.org/intelligent 37

AI and Cultural Heritage

Face detection
extracts the face
and its basic weights and model eyes to deter-
components mine an eye’s presence and position
(nose and eyes). (indicated by the red and green dots
in the second image in the face de-
tection box of Figure 2).
Face component Base color After eye localization, the mod-
detection analysis extracts
detects the most the base color
ule detects the nose. Let H denote the
important components layers (darking, distance between the eyes and con-
of the face (beard, proplasmos. sequently the nose’s length. Accord-
moustache, cheek, sarkoma, and ing to Dionysios, the nose starts on a
forehead, and hair, lightening).
respectively). small horizontal line whose center is
a distance of H/5 below the midpoint
between the eyes. We use a Hough
s1 Image description transform to find this line. Therefore,
Feature extraction s: Face we determine the nose’s ending line
automatically extracts s2 s1: Hair
features such as
by applying a Hough transform at a
s2: Forehead
distance, color, s3 s3: Cheek distance H down vertically from the
and length for s4: Mustache starting line.
each face s5: Beard The computation of H leads to the
component and s4 (s, s1): hasComponent
for some ...
detection of the face’s other main com-
segments s5 Length(s1) = 2.8*H ponents—the hair, forehead, cheek,
of the face ... mustache, and beard—because they are
components. darking(s1) = 88% well defined in Dionysios’s manual.
...
The module detects the four base
colors by applying an Otsu segmen-
Semantic Formal assertions tation algorithm 6 to a restricted area
interpretation s: Face of the face segment (see the base
gives meaning to BelowEarsHair AboveEarsHair s1: Hair color analysis box in Figure 2), de-
Membership

specific features, 1 s2: Forehead

termining a range of intensity values
properties, and 0.7 s3: Cheek
relationships of s4: Mustache for the pixels.
0.3
face components s5: Beard To segment each part of the face, we
with the aid (s, s1): hasComponent use H together with Dionysios’s pro-
of fuzzy 0 2.8 5 ...
partitions. Hairlength s1: AboveEarsHair = 0.7 portion guidelines to define the seeds
Fuzzy partitioning s1: BelowEarsHair = 0.3 for graph-cut segmentation.7 The cost
... functions used as soft constraints in-
corporate boundary and part infor-
mation for each of the five segmented
Figure 2. Image analysis. First, the algorithm detects the sacred figure’s face region, parts. To achieve segmentation of
eyes, and nose. Then, it extracts the hair, forehead, cheek, mustache, and beard each part, we compute the minimum
parts together with the face’s base color layers. Further analysis of the extracted cut that’s the global optimum among
parts provides information about characteristic features. Finally, the algorithm
all the segmentations satisfying the
produces a semantic interpretation for each of these features, together with formal
assertions.
constraints (see the face-component-
detection box in Figure 2).

aid of geometric information from the eyes and the sur-
rounding area. 5 We use a training set of sacred-figure
eyes to create vector maps, and we use principal com-
ponents analysis to derive eigenvectors for these maps.
For a candidate face segment, the algorithm 5 projects
the length and its angle maps on the spaces spanned
by the eigenvectors that were evaluated during train-
ing. In that way, we use the similarity of the projection
Feature extraction. This module automatically ex-
tracts features from the segmented parts, 8 providing
additional information regarding each part’s length,
color, shape, and texture. This information constitutes
the characteristic values of the features, providing an
image description set (see the image-description sec-
tion of the feature extraction box in Figure 2).
Dionysios assigns four features to hair:

38 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

• color (dark, gray, or white),
• density (thin or thick),
• length (above the ears or below the ears, and whether the
figure has a braid on the left, right, or both shoulders), and
• form (straight, wild, or curly).
The feature extraction module categorizes hair color (see
Figure 2, feature extraction, segment s1) using the intervals
of Otsu segmentation. It estimates the density and form by
combining a Canny edge detector and a morphological fi l-
ter and by evaluating the mean value of the curvatures of
the detected edges. It fi nds the hair’s length using H to-
gether with Dionysios’s proportion guidelines.
Next, the module determines information about the sa-
cred figure’s age by computing the
number of wrinkles on the forehead
(s2). Using the Hough transform, the the module determines
module defi nes the number of verti-
cal and horizontal lines in the fore- information about
head, which represent the wrinkles.
A young sacred figure has zero or few
wrinkles, whereas an older sacred fig-
the sacred figure’s age
ure has many.
The cheek (s3) provides additional
by computing the
information about age. According to
Dionysios, a small number of wrin-
number of wrinkles
kles on the larger (left or right) part
of the cheek indicates a young sacred on the forehead.
figure. Conversely, an older sacred fig-
ure’s cheek will have a greater number
of wrinkles (and therefore a greater proportion of the colors
used for shadows).
Finally, the module defi nes the feature sets for the mus-
tache (s4) and beard (s5). Using techniques similar to those
for hair analysis, it analyzes the mustache’s color and shape.
It also extracts information on the beard’s length (short, not
very long, long, down to the waist, or down to the knees),
color (fair or dark), form (straight, wild, or curly) and shape
(sharp, circle-like, circle, wide, or fluffy).
Semantic interpretation. This module represents the ex-
tracted information in terms of the sacred-figure ontology.
Representing the face and its main components is
straightforward: we simply write assertions such as s1:
Hair (see the semantic-interpretation box in Figure 2).
However, each component’s properties are vague. For ex-
ample, you can’t always directly determine on the basis of
length whether the sacred figure has long hair. This dif-
ficulty is due to the vagueness of the defi nition of length
and the imprecision introduced by feature extraction (see,
for example, the result of the segmentation of the hair in
the feature extraction box in Figure 2). Nevertheless, we
March/april 2009 www.computer.org/intelligent
can express an assertion in a fuzzy manner, with the aid of
fuzzy set theory. This procedure is critical for the knowl-
edge representation system.
Suppose that a specific feature’s values lie in the inter-
val X. A fuzzy set A of X associates each value x of X
to a real number in the interval [0, 1], with the value of
fA(X) at x representing the grade of membership of x in
A. So, we defi ne an appropriate fuzzy partition (a set of
fuzzy sets) of X and connect each fuzzy set of the parti-
tion with a subconcept of the terminology, defi ned by that
fuzzy set’s property. For example, having extracted the
value 2.8H for the length of hair (s1), we determine that
s1 has a membership value of 0.7 for AboveEarsHair and
0.3 for BelowEarsHair (see the graph in the semantic-
interpretation box in Figure 2). The
defi nition of the fuzzy partitions is
based on the heuristics extracted from
Dionysios’s manual.
Knowledge representation
and reasoning
This subsystem consists of terminolog-
ical and assertional knowledge and a
reasoning engine.
These types of knowledge are
the basic components of a knowl-
edge-based system based on DLs, 9 a
structured knowledge-representation
formalism with decidable-reasoning
algorithms. DLs have become popular,
especially because of their use in the Semantic Web (as in
OWL DL, for example). DLs represent a domain’s impor-
tant notions as concept and role descriptions. To do this,
DLs use a set of concept and role constructors on the ba-
sic elements of a domain-specific alphabet. This alphabet
consists of a set of individuals (objects) constituting the do-
main, a set of atomic concepts describing the individuals,
and a set of atomic roles that relate the individuals. The
concept and role constructors that are used indicate the ex-
pressive power and the name of the specific DL. Here, we
use SHIN, an expressive subset of OWL DL that employs
concept negation, intersection, and union; existential and
universal quantifiers; transitive and inverse roles; role hier-
archy; and number restrictions.
Terminological and assertional knowledge. The termino-
logical knowledge is an ontological representation of the
knowledge in Dionysios’s manual. The assertional knowl-
edge is a formal set of assertions describing a specific icon
in terms of the terminological knowledge. For example, the
terminology contains axiomatic knowledge such as “a Jesus
Face is depicted as a young face with long, straight, thick,
39
AI and Cultural Heritage
black hair and a short, straight, thick, black beard,” whereas
“the face depicted in the icon is a face with long hair in some
degree” is a relative assertion.
The knowledge base has two main components. The
terminological component (TBox) describes the relevant
notions of the application domain by stating properties of
concepts and roles and their interrelations. TBox is actu-
ally a set of concept inclusion axioms of the form C v D,
where D is a superconcept of C, and concept equivalence
axioms of the form C ≡ D, where C is equivalent to D.
The assertional component (ABox) describes a concrete
world by stating properties of individuals and their spe-
cific interrelations. To deal with the vagueness introduced
in our case, we use f-SHIN, a fuzzy extension of SHIN. 3 In
f-SHIN, ABox is a finite set of fuzzy assertions of the form
〈a : C⋈n〉, 〈(a, b) : R⋈n〉, where ⋈ stands for ≥, >, ≤, or
<, for a, b ∈ I. Intuitively, a fuzzy assertion of the form 〈a
: C ≥ n〉 means that the membership degree of a to concept
C is at least equal to n. (A formal definition of the syntax
and semantics of fuzzy DLs appears elsewhere. 3,10)
In the Byzantine-icon-analysis domain, the individuals
are the segments that the semantic-segmentation module
extracted. The DL concepts classifying the individuals are
determined by the properties measured as features during
feature extraction (for example, BlackHair and AboveE-
arsHair). The DL roles mainly describe partonomic
(mereological) hierarchies and spatial relations (for ex-
ample, the axiom isLeftOf– = isRightOf indicates that
the role isLeftOf is the inverse of isRightOf, and the
axioms Trans(isLeftOf) and Trans(isAboveOf) indicate
that isLeftOf and isRightOF are transitive).
The terminology’s main concepts are Figure, Face, and
FaceComponent. They’re connected with the partonomic
role hasPart and its subroles hasSegment and hasCom-
ponent, to indicate whether a segment is a subsegment of
another segment and so on. Using these roles, we describe
the partonomic hierarchy as follows:
Face ≡ ∀hasComponent.FaceComponent
Figure ≡ ∃hasSegment.Face
FaceComponent v Beard t Cheek t Forehead t Hair
t Mustache
We describe relevant assertions as follows:
〈s:Face ≥ 1.0〉
〈s1:Hair ≥ 1.0〉
〈(s, s1):hasSegment ≥ 1.0〉
40 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS
…
The taxonomy is constructed using the properties mea-
sured as features during feature extraction. Here are some
examples of terminological axioms defining the concept
hierarchy:
AboveEarsHair v Hair
OldCheek v Cheek
OldForehead v Forehead
HairyBeard v Beard
Restrictions concerning the taxonomy mainly describe dis-
jointness and exhaustiveness, forming axioms such as
AboveEarsHair v ¬BelowEarsHair
Hair v AboveEarsHair t BelowEarsHair
Using these concepts, we define more expressive descrip-
tions that specify some specific face characteristics on the
basis of Dionysios’s manual. For example, we define a
young man’s face as
YoungMansFace ≡ ∃hasComponent.BlackHair u
∃hasComponent.(NotVeryLongBeard u BlackBeard)
u ∃hasComponent.YoungForehead u ∃hasPart.
YoungCheek u ∃hasPart.BlackMustache
We similarly define concepts specifying the characteris-
tics of some popular sacred figures. For example, we define
JesusFace as
JesusFace ≡ YoungMansFace u ∃hasComponent.
(BraidOnTheLeftShoulder u HairyHair
u StraightHair) u ∃hasComponent.
(CircleLikeBeard u StraightBeard)
The reasoning engine. This module uses the terminologi-
cal knowledge and the assertions to recognize the sacred
figure. Our system uses FiRE, the Fuzzy Reasoning Engine
(www.image.ece.ntua.gr/~nsimou/FiRE), which fully sup-
ports f-SHIN.
Figure 3 illustrates how the engine performs reasoning
and classification for recognizing Jesus as the sacred figure.
We use FiRE’s greatest lower bound (GLB) reasoning ser-
vice, which evaluates an individual’s greatest possible degree
of participation in a concept. The concepts of interest in this
case, YoungMansFace and JesusFace, only include conjunc-
 YoungMansFace � �hasComponent.
BlackHair � hasComponent.
�.(NotVeryLongBeard � BlackBeard)
� �hasComponent.YoungForehead
� �hasPart.YoungCheek � �hasPart.
BlackMoustache

JesusFace � YoungMansFace � JesusFace

s1 �hasComponent.(BraidOnTheLeftShoulder
s2 � HairyHair � StraightHair)
� �hasComponent.(CircleLikeBeard
s3 � StraightBeard)
TBox
Reasoning
s4 engine
s: Face
s5 s1: Hair …
(s, s1): hasComponent …

s1: BraidOnTheLeftShoulder = 0.78

s1: HairyHair = 0.96
s1: StraightHair = 0.86
s1: BlackHair = 0.88
s2: YoungForehead = 0.82
s3: YoungCheek = 0.87
s4: StraightMoustache = 1.0
s4: BlackMoustache = 0.83
s5: CircleLikeBeard = 0.79
s5: StraightBeard = 0.92
s5: NotVeryLongBeard = 1.0 Knowledge
s5: BlackBeard = 0.84 ABox representation
and reasoning

Figure 3. A reasoning example. The extracted information from image analysis constitutes the assertional component (ABox) of
the knowledge base; the terminological component (TBox) is defined on the basis of Fourna’s specification. These components
form the input to the fuzzy reasoning engine, which infers information about the icon.

tions, so we use the fuzzy intersection operator for the mini-
mum. So, the GLB of segment s (the face) in YoungMansFace
is 0.82, whereas the GLB of segment s in JesusFace is 0.78.
In other words, the extracted features indicate that the face
depicted is YoungMansFace to a degree of 0.82 and is Je-
susFace to a degree of 0.78.
Results
We evaluated our system on a database, provided by the
Mount Sinai Foundation in Greece, containing 2,000 digi-
tized Byzantine icons dating back to the 13th century. The
icons depict 50 different characters; according to Diony-
sios, each character has specific facial features that makes
him or her distinguishable.
Evaluation of the Byzantine-icon-analysis subsystem pro-
duced promising results. The subsystem’s mean response
time was approximately 15 seconds on a typical PC.
In the semantic-segmentation module, the face detection
submodule reached 80 percent accuracy. In most cases, the
failure occurred in icons with a destroyed face area. If the
submodule detected the face, it almost always detected the
eyes and nose. The base-color-analysis submodule defined
the color model used by the icon’s artist, which the fea-
ture extraction module later used. Using the nose’s height,
which can be easily determined, and taking into account
the manual’s rules, the face-component-detection submod-
ule estimated the positions of the background and fore-
ground pixels for every part of the face. These positions
constituted the input to the graph-cut algorithm. This sub-
module achieved 96 percent accuracy.
The feature extraction module extracted features for
every icon segment. Then, using the fuzzy partitions, the
semantic-interpretation module interpreted each segment’s
specific features and properties and the relationship of face
parts. Thus, that module determined each feature’s degree
of membership in a specific class, thereby creating an image
description.
To evaluate overall system performance, we used
precision and recall. Table 1 (on the next page) pres-
ents results for 20 of the sacred-figure classes. The
more distinctive facial features the sacred figure in a
class contained, the better our method performed (for

March/april 2009 www.computer.org/intelligent 41

AI and Cultural Heritage

Table 1. Evaluation of the icon classification system.

No. of images
Class No. of icons in class No. of images classified correctly classified Precision Recall

Jesus 70 67 58 0.87 0.83

Virgin Mary 60 54 43 0.80 0.72
Peter 50 44 38 0.86 0.76
Paul 50 44 39 0.89 0.78
Katerina 40 35 30 0.86 0.75
Ioannis 40 37 29 0.78 0.73
Luke 40 33 28 0.85 0.70
Andrew 40 33 29 0.88 0.73
Stefanos 30 27 24 0.89 0.80
Konstantinos 40 36 30 0.83 0.75
Dimitrios 40 36 31 0.86 0.78
Georgios 40 37 32 0.86 0.80
Eleni 40 37 31 0.84 0.78
Pelegia 20 17 15 0.88 0.75
Nikolaos 40 38 33 0.87 0.83
Basileios 40 37 31 0.84 0.78
Antonios 30 27 20 0.74 0.67
Eythimios 25 23 18 0.78 0.72
Thomas 35 32 26 0.81 0.74
Minas 20 17 14 0.82 0.70

example, for the Jesus class). Conversely, our method
performed worse on figures with similar facial features,
such as women.
sify paintings on the basis of unmixed colors and dense
textures.
Acknowledgments
Our research has been partly supported by the MORFES (extraction
and modeling of figure in Byzantine paintings of Mount Sinai monu-

W e’re developing a version of our system that takes

into account information about clothes associated
with specific types of figures (such as apostles, kings, an-
gels, and monks) and clothing accessories (kerchiefs and
so on) and nonclothing accessories (books, crosses, and so
on) associated with specific sacred figures. For example,
the system could incorporate the knowledge that icons of
the Virgin Mary depict her wearing a kerchief with three
stars. Such information might help in classifying figures
with similar facial features.
Furthermore, an integrated version of our system could
eventually date icons and attribute them to a particular
artist or artistic school, on the basis of features specific
to each artistic period. Such information can help classify
an icon even if the specific sacred figure is difficult to rec-
ognize. We’re also investigating applying our method to
other art movements such as impressionism, trying to clas-
ments using measures) project, funded by the EU and the Greek Sec-
retariat for Research and Development. We thank the Mount Sinai
Foundation for providing access to the rich data set and especially
archeologist Dimitris Kalomirakis for his participation in knowl-
edge extraction and his continuing support during this research.
Finally, we thank the anonymous reviewers for their constructive
comments.
References
1. T. Berners-Lee, J. Hendler, and O. Lassila, “The Semantic
Web,” Scientific American, May 2001, pp. 34–43.
2. Dionysios tou ek Phourna, Hermeneia tis zografikis technis
(interpretation of the Byzantine art), B. Kirschbaum, 1909.
3. G. Stoilos et al., “Reasoning with Very Expressive Fuzzy De­­
scription Logics,” J. Artificial Intelligence Research, vol. 30,
no. 5, 2007, pp. 273–320.
4. P. Viola and M.J. Jones, “Robust Real-Time Face Detection,”
Int’l J. Computer Vision, vol. 57, no. 2, 2004, pp. 137–152.
5. S. Asteriadis et al., “An Eye Detection Algorithm Using Pixel
to Edge Information,” Proc. 2nd IEEE-EURASIP Int’l Symp.

42 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS


Control, Communications, and Sig­
nal Processing (ISCCSP 06), 2006;
www.eurasip.org/Proceedings/Ext/
ISCCSP2006/defevent/papers/cr1124.
pdf.
6. N.A. Otsu, “A Threshold Selection
Method from Gray-Level Histograms,”
IEEE Trans. Systems, Man, and Cyber-
netics, vol. 1, no. 9, 1979, pp. 62–66.
7. Y. Boykov and M.-P. Jolly, “Interactive
Graph Cuts for Optimal Boundary and
Region Segmentation of Objects in N-D
Images,” Proc. Int’l Conf. Computer
Vision (ICCV 01), vol. 1, IEEE Press,
2001, pp. 105–112.
8. R.C. Gonzalez and R.E. Woods, Digi-
tal Image Processing, 3rd ed., Prentice
Hall, 2008.
9. F. Baader et al., The Description Logic
Handbook: Theory, Implementation
and Applications, Cambridge Univ.
Press, 2002.
10. G. Stoilos et al., “Uncertainty and the
Semantic Web,” IEEE Intelligent Sys-
tems, vol. 21, no. 5, 2006, pp. 84–87.
For more information on this or any other
computing topic, please visit our Digital Li-
brary at www.computer.org/csdl.
Th e A u t h o r s
Paraskevi Tzouveli is pursuing her PhD in image and video analysis at the National Tech-
nical University of Athens. Her research interests include image and video analysis, in-
formation retrieval, knowledge manipulation, e-learning, and digital cultural heritage
systems. Tzouveli received her Diploma in electrical and computer engineering from the
National Technical University of Athens. Contact her at tpar@image.ntua.gr.
Nikos Simou is a PhD student in the National Technical University of Athens Depart-
ment of Electrical and Computer Engineering and a member of the Image, Video, and
Multimedia Systems Laboratory’s Knowledge Technologies Group. His research interests
include knowledge representation under certain and uncertain or imprecise knowledge
based on crisp and fuzzy description logics, ontologies, and the implementation and op-
timization of reasoning systems and applications for the Semantic Web. Contact him at
nsimou@image.ntua.gr.
Giorgios Stamou is a lecturer in the National Technical University of Athens School of
Electrical and Computer Engineering and a member of the university’s Image, Video, and
Multimedia Systems Laboratory, heading the lab’s Knowledge Technologies Group. His
main research interests include knowledge representation and reasoning, theory of uncer-
tainty, machine learning, semantic interoperability, and digital archives. Contact him at
gstam@cs.ntua.gr.
Stefanos Kollias is a professor in the National Technical University of Athens (NTUA)
School of Electrical and Computer Engineering and the director of the Image, Video, and
Multimedia Systems Laboratory. His research interests include image and video process-
ing, analysis, retrieval, multimedia systems, computer graphics and virtual reality, artificial
intelligence, neural networks, human-computer interaction, and digital libraries. Kollias
received his PhD in signal processing from NTUA. He’s a member of the European Com-
mission expert group in digital libraries and of the EC committee on semantic interoper­
ability, leading NTUA participation in DL projects, EDLnet, VideoActive, Europeana
Connect, Europeana Network 1.0, and EU Screen. Contact him at stefanos@cs.ntua.gr.

March/April 2009 • IEEE Intelligent Systems
Advertising Personnel
Marion Delaney
IEEE Media,
Advertising Dir.
Phone: +1 415 863 4717
Email: md.ieeemedia@
ieee.org
Marian Anderson
Sr. Advertising
Coordinator
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: manderson@
computer.org
Sandy Brown
Sr. Business
Development Mgr.
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: sb.ieeemedia@
ieee.org
AdvertiSER Information
Advertising Sales Midwest/Southwest Product:
Representatives Darcy Giovingo
Phone: +1 847 498 4520 US East
Fax: +1 847 498 5911 Joseph M. Donnelly
Email: dg.ieeemedia@ieee.org Phone: +1 732 526 7119
Recruitment:
Email: jmd.ieeemedia@ieee.org
Mid Atlantic Northwest/Southern CA
Lisa Rinaldo Tim Matteson US Central
Phone: +1 732 772 0160 Phone: +1 310 836 4064 Darcy Giovingo
Fax: +1 732 772 0164 Fax: +1 310 836 4067 Phone: +1 847 498 4520
Email: lr.ieeemedia@ Email: tm.ieeemedia@ Fax: +1 847 498 5911
ieee.org ieee.org Email: dg.ieeemedia@ieee.org
New England Japan US West
John Restchack Tim Matteson Lynne Stickrod
Phone: +1 212 419 7578 Phone: +1 310 836 4064 Phone: +1 415 931 9782
Fax: +1 212 419 7589 Fax: +1 310 836 4067 Fax: +1 415 931 9782
Email: j.restchack@ieee.org Email: tm.ieeemedia@ieee.org Email: ls.ieeemedia@ieee.org
Southeast Europe Europe
Thomas M. Flynn Hilary Turnbull Sven Anacker
Phone: +1 770 645 2944 Phone: +44 1875 825700 Phone: +49 202 27169 11
Fax: +1 770 993 4423 Fax: +44 1875 825701 Fax: +49 202 27169 20
Email: flynntom@ Email: impress@ Email: sanacker@
mindspring.com impressmedia.com intermediapartners.de

March/april 2009 www.computer.org/intelligent 43