Applied Intelligence 12, 251–268 (2000)

°
c 2000 Kluwer Academic Publishers. Manufactured in The Netherlands.

Incremental Iterative Retrieval and Browsing for Efficient Conversational

CBR Systems

IGOR JURISICA
University of Toronto, Faculty of Information Studies, 140 St. George St., Toronto, ON M5S 3G6, Canada
jurisica@fis.utoronto.ca

JANICE GLASGOW
Queen’s University, Department of Computing and Information Science, Kingston, ON K7L 3N6, Canada
janice@qucis.queensu.ca

JOHN MYLOPOULOS
University of Toronto, Department of Computer Science, 6 King’s College Rd., Toronto, ON M5S 3H5, Canada
jm@ai.utoronto.ca

Abstract. A case base is a repository of past experiences that can be used for problem solving. Given a new
problem, expressed in the form of a query, the case base is browsed in search of “similar” or “relevant” cases.
Conversational case-based reasoning (CBR) systems generally support user interaction during case retrieval and
adaptation. Here we focus on case retrieval where users initiate problem solving by entering a partial problem
description. During an interactive CBR session, a user may submit additional queries to provide a “focus of
attention”. These queries may be obtained by relaxing or restricting the constraints specified for a prior query. Thus,
case retrieval involves the iterative evaluation of a series of queries against the case base, where each query in the
series is obtained by restricting or relaxing the preceding query.
This paper considers alternative approaches for implementing iterative browsing in conversational CBR systems.
First, we discuss a naive algorithm, which evaluates each query independent of earlier evaluations. Second, we
introduce an incremental algorithm, which reuses the results of past query evaluations to minimize the computation
required for subsequent queries. In particular, the paper proposes an efficient algorithm for case base browsing and
retrieval using database techniques for incremental view maintenance. In addition, the paper evaluates scalability
of the proposed algorithm using its performance model. The model is created using algorithmic complexity and
experimental evaluation of the system performance.

Keywords: knowledge base technology, case-based reasoning, performance evaluation, context-based iterative
browsing and retrieval

1. Introduction analytical processing, web databases [4], conversa-

There are many applications where one could effec-
tively obtain information and support collaboration us-
ing iterative browsing [1–3]. It is particularly useful
for exploratory search in complex domains and for
problem-solving tasks that involve considering alter-
native situations (e.g., examining “what-if” scenarios).
Such applications include knowledge mining, on-line
tional case-based reasoning [5], exploratory retrieval
and browsing [6]. Although the proposed approach to
incremental retrieval has wider applicability, here we
focus on its application to achieve efficient retrieval for
case-based reasoning (CBR) systems.
The research presented in this paper has three pri-
mary objectives: (i) to define an iterative browsing
system for conversational case retrieval; (ii) to adapt
252 Jurisica, Glasgow and Mylopoulos
an incremental view maintenance algorithm used in
database systems for use in CBR for efficient iterative
retrieval and browsing; and (iii) to provide a theoretical
performance model for the proposed incremental case
retrieval system. Results of performance evaluation
suggest that a conversational CBR system with an in-
cremental retrieval method would scale up for large and
complex case bases. However, in some well-defined
and relatively-small domains, traditional CBR algo-
rithms currently provide an adequate solution for case
retrieval.
1.1. Case-Based Reasoning
Case-based reasoning (CBR) is a process of remember-
ing and reusing experience in the form of cases stored
in a case base.1 It relies on the concept that similar
problems have similar solutions. Informally, a case is
a problem-solution-feedback triplet. Given a partially
developed case—a new problem expressed in the form
of a query, the case base is browsed in search of “sim-
ilar” or “relevant” cases. These cases are then adapted
(either by an expert or by the system) to produce a so-
lution for the input problem. The final step of CBR
involves evaluating the solution and producing feed-
back information.
The decision-making process finds similarities be-
tween a problem at hand and cases in the case base,
and reuses stored solutions to solve new problems. The
more relevant a case is to a problem, the less adaptation
is needed and the more precise the solution. Thus, the
quality of reasoning depends on the quality and quan-
tity of cases in a case base. With an increased number of
unique cases, the problem-solving capabilities of CBR
systems improve while their efficiency may decrease.
1.2. Conversational CBR
Conversational CBR systems generally support user
interaction during case retrieval and adaptation. In this
paper we propose a retrieval algorithm for efficient and
effective iterative browsing for CBR systems. During
an interactive CBR session, a user may submit several
queries to provide a “focus of attention” (note that we
assume that the queries are related to a given problem-
solving session). These queries may be obtained by
modifying a partial description of a case, e.g., relaxing
or restricting the constraints specified for a prior query.2
Thus, case retrieval involves the iterative evaluation of
a series of queries against the case base, where each
query in the series is obtained by restricting or relaxing
the preceding query.
In order to support iterative browsing, a system must
be able to respond to a series of queries, where the sys-
tem or the user may alter a current query and re-submit
it for further evaluation. This process may be repeated
until the desired quality and quantity of cases is ob-
tained. The iterative process is aimed at maintaining
high recall while improving precision of the case re-
trieval. It is essential that the process is efficient, so
that the overall system is scalable.
To be effective, a conversational CBR system should
limit the number of questions asked and control their
relevancy [7]. Questions are selected based on the num-
ber and quality of retrieved cases. To increase relevancy
of cases, the user may need to specify criteria for simi-
larity assessment. We have previously demonstrated
that variable-context similarity-based retrieval can im-
prove competence of a CBR system [8, 9]. Since the
criteria are likely to change during the retrieval, the
process of altering it is iterative. Here we hypothesize
that incremental retrieval can improve the efficiency of
CBR systems.
1.3. Incremental Iterative Browsing
To illustrate our approach to conversational CBR,
imagine a partial representation of a medical case base
for an in vitro fertilization (IVF) domain [9]. Indivi-
dual patients are identified by age, IVF cycle, pro-
tocol used, and diagnosis of infertility. During case-
based reasoning, it may be desired to modify the
relevance of individual attributes during similarity as-
sessment. In addition, constraints on attribute values
may also be modified. Since these changes affect what
cases are retrieved from a case base, we call this con-
text modification.3 Context can be either relaxed or
restricted. Intuitively, relaxation results in retrieving
more cases and restriction results in retrieving fewer
cases. Since relaxation and restriction can be inter-
twined, context modification is a process analogous
to iterative browsing. As an example, consider the re-
laxation of a context specified in Fig. 1.4 Individual
attributes are grouped into three categories of the same
priority. During retrieval, cardinality criteria are used
to change a requirement that all attributes in a given
category must match to a relaxed requirement that only

Figure 1. Context relaxation by reducing the number of attributes required to match and by enlarging the set of allowable values for an attribute.
2 attributes must match. We call this relaxation reduc-
tion, since we reduce number of attributes in a category
required to match. We can also relax requirements on
attribute values.
This paper considers alternative approaches for im-
plementing iterative browsing in conversational CBR
systems. First, we discuss a naive algorithm, which
evaluates each query independent of earlier evalua-
tions. Thus, after the query is modified, the system
searches the case base again. Second, we introduce
an incremental algorithm, which reuses the results of
past query evaluations to minimize the computation re-
quired for subsequent queries. Incremental algorithms
have previously been successfully applied to database
systems for view maintenance [10–13]. In particular,
the paper proposes an efficient algorithm for case base
browsing and retrieval using database techniques for
incremental view maintenance. Although this may re-
quire additional storage, overall performance is usu-
ally improved. Retrieval is based on a nearest-neighbor
matching algorithm [14], which was modified as fol-
lows: (i) attributes are grouped into categories of dif-
ferent importance to help control the matching process
and diminish the negative effect of irrelevant attributes
Efficient Conversational CBR Systems 253
on performance; (ii) an explicit context is used during
similarity assessment to ensure that only relevant cases
are retrieved; and (iii) incremental context transforma-
tions are applied during query relaxation to speed-up
query-processing.
1.4. Outline
Next we describe TA3, a case-based decision-support
system with incremental retrieval.5 Section 3 presents
theory and implementation of context modification. An
incremental approach to context relaxation and restric-
tion is described in Section 4. Section 5 introduces a
performance model and presents a performance evalu-
ation of TA3. Section 6 discusses related research, our
experience, and future directions.
2. System Description
TA3 comprises all modules of a CBR system: case
representation, case retrieval, evaluation, adaptation
and presentation. In addition, it supports case base or-
ganization and knowledge mining. TA3 uses flexible
case representation, effective and efficient incremental
254 Jurisica, Glasgow and Mylopoulos
Figure 2. TA3 architecture.
retrieval algorithm. Figure 2 depicts the overall archi-
tecture, data and control flow.
A case, C, is a representation of a real-world experi-
ence in a particular representation scheme. A case will
be represented as a finite set of attribute-value pairs
(descriptors):6
C = {ha0 · V0 i, ha1 · V1 i, . . . , han · Vn i}.
Individual attributes and values are grouped into one
or more categories, which introduces additional struc-
ture to a case representation. Category membership is
defined either using a knowledge-discovery algorithm
[9, 15] or using domain knowledge (if available). Dif-
ferent constraints may be ascribed to different cate-
gories, i.e., individual groups of attributes. This reduces
the impact of irrelevant attributes on system compe-
tence by selectively using individual categories during
matching [16, 17].
Context plays an important role during case retrieval.
Our primary goal is to retrieve only useful cases, i.e.,
cases highly relevant to a problem represented as a par-
tially developed target case (a query). Since the use-
fulness of individual attributes during matching varies,
we propose to incorporate the notion of a context to
define what attributes are relevant in a given situation.
2.1. A Context
A context explicitly defines the attributes to be used dur-
ing similarity assessment and any constraints that may
be applicable to the attribute values: Context = {ha0 ·
C V0 i, . . . , hak ·C Vk i}, where ai is an attribute name and
constraint C Vi specifies the set of “allowable” values
for attribute ai . A context specifies relevant attributes
and how “close” an attribute value must be in order to
satisfy the context.

A context is formed using the most relevant attributes
for a given target case; similarity is determined as a
closeness of values for the prescribed attributes, taking
attribute value distribution and the task being solved
into account. The attributes and constraints for a con-
text can be specified by a user or derived automatically.
In general, the context can be specified using the fol-
lowing scenarios: (i) Task-based retrieval: The user
specifies the task to be solved and the system selects an
appropriate context. (ii) Query-by-example: The user
selects a source case from a case base to be used as a
problem example. Case attributes and values are then
used as attributes and constraints in a context. (iii) The
user has enough domain knowledge7 to specify the con-
text directly. (iv) Retrieval-by-reformulation: The user
submits an initial unrefined query, reviews the resulting
solution, and then iteratively modifies and resubmits
the query. This approach is best suited for repository
browsing.
If the interpretation of the case in a case base (a
source case) satisfies the given constraints for the spec-
ified attributes, then it is considered relevant for a
given context. A case C satisfies a context Context,
denoted sat(C, Context), if and only if for all pairs
hai .C Vi i ∈ Context, there exists a pair hai · Vi i ∈ C
such that Vi is in C Vi :
sat(C, Context) iff ∀ai hai · C Vi i ∈ Context
→ ∃Vi hai · Vi i ∈ C ∧ Vi ∈ C Vi .
All cases relevant to a given context are considered
similar for the context.8 The process of retrieving rel-
evant cases can then be described as a constraint satis-
faction process [18].
2.2. Cardinality and Value Constraints
Two types of constraints can be imposed on attributes
and their values. Cardinality constraints specify the
number of attributes required to match for a particular
category. This is an extension of an x-of-n matching
algorithm [17], since different categories may have dif-
ferent matching requirements, i.e., different x. Individ-
ual categories are ordered according to the importance
of their attributes. If no priority is assigned to a cat-
egory, they are accessed sequentially. If the case has
only one category, the matching is equivalent to x-of-
n matching. However, if more categories are defined,
then important attributes may require n-of-n matching,
Efficient Conversational CBR Systems 255
less important attributes x-of-n matching (for x < n)
and irrelevant attributes may be eliminated from con-
sideration altogether.
Value constraints specify constraints on attribute val-
ues. They include: (i) instance matching—the attribute
value in the source case must match the attribute value
in the target case; (ii) set matching—the attribute value
in the source case must be included in the set of al-
lowed values specified in the context; (iii) interval
matching—the attribute value in the source case must
be within an interval specified by the context; and (iv)
domain matching—the attribute value in the source
case must be an instance of the constraint in the context.
Variable-context similarity-based retrieval is mono-
tonic [8]. For the purpose of case retrieval, the sim-
ilarity relation maps a Context and a CaseBase onto
the set of cases Answer in the CaseBase that sat-
isfy the Context. Given a CaseBase and a Context,
the retrieval function returns a non-empty set of rel-
evant cases Answer ⊆ CaseBase such that all cases in
Answer satisfy the given Context. The retrieval func-
tion is complete in the sense that it returns all relevant
cases and only relevant cases.
3. Context Relaxation and Restriction
An explicitly defined context controls the closeness of
retrieved cases. If too many or too few relevant cases
are retrieved using the initial context, then the system
automatically transforms the context or the user man-
ually modifies it. The context transformation process
controls the quality and quantity of retrieved cases and
thus when transforming the context, the system may re-
turn an approximate answer quickly or may spend more
resources to calculate a more accurate answer [19]. An
approximate answer can be iteratively improved, so
that the change between an approximate and an accu-
rate answer is continuous. This is an important feature
for bounded-resource computation [20, 21].
We propose two context transformations as a foun-
dation for supporting iterative retrieval and browsing:
relaxation—to retrieve more cases, and restriction—to
retrieve fewer cases.
3.1. Context Relaxation
A context Context1 is a relaxation of a context Context2 ,
denoted Context1 Â Context2 , if and only if the set of
256 Jurisica, Glasgow and Mylopoulos

attributes for Context1 is a subset of the set of attributes Thus, the attribute value could alternatively be relaxed
for Context2 and for all attributes in Context1 , the set from 28 to for example, 26–30. The constraint of at-
of constraints in Context2 is a subset of the constraints tribute 1 in category 3 is also relaxed from constraint
in Context1 . As well, contexts Context1 and Context2 tubal to constraint set {tubal, endo}.
are not equal.
We implement context relaxation as reduction
3.2. Context Restriction
and generalization.9 Reduction removes an attribute-
value pair from a context, either permanently or
A context can be similarly iteratively restricted by mak-
dynamically—given x-of-n matching, the number of
ing it progressively more specific, i.e., allowing fewer
attributes required to match is reduced from x to y,
cases to satisfy it. As in the relaxation algorithm, ex-
where 0 < y < x ≤ n. Generalization relaxes the
pansion and specialization algorithms produce a par-
context by enlarging the set of allowable values for an
tial order of contexts. In general, only one category
attribute.
is restricted at a time, using categories with the lower
The relaxation technique can advantageously be
priority first.
used for returning answers to a specific query as well as
A Context1 is a restriction of a Context2 , denoted
returning related answers [22]. Without an automatic
Context1 ≺ Context2 , if and only if Context2 is a relax-
query relaxation, users would need to submit alterna-
ation of Context1 : Context1 ≺ Context2 iffContext2 Â
tive queries. The restriction technique works analo-
Context1 . We implement context restriction as expan-
gously, but is used mainly for controlling the amount of
sion and specialization. Expansion, an inverse opera-
returned information, preventing information overload.
tion to reduction, strengthens constraints by enlarging
Since the search for relaxed or restricted contexts could
the number of attributes that are required to match. Spe-
be infinite, there must be a mechanism for controlling
cialization strengthens constraints by removing values
it, either by user intervention (via user preferences) or
from a constraint set for an attribute. This may lead to
by other means (e.g., limiting number of consecutive
a decreased number of cases that satisfy the resulting
context modifications).
context.
As an example, consider the relaxation of a context
Contexts can also be iteratively restricted to re-
specified in Fig. 1. This context was created from a
trieve successively fewer cases. There are two possible
partially developed problem case. In order to increase
implementations of context restriction: expansion—
versatility, we grouped attributes into three categories.
strengthening constraints by enlarging the number
Individual attributes identify age, IVF cycle, protocol
of attributes required to match and specialization—
used, and diagnosis of infertility. Assume that all cat-
strengthening constraints by removing values from a
egories have the same priority and value constraints
constraint set for an attribute (see Fig. 3).
but they require a different number of attributes to
match, i.e., different cardinality criteria are imposed. It
is not possible to apply reduction on category 1, since 3.3. Naive Iterative Retrieval Algorithm
there is only one attribute. Category 2 and 3 can be
relaxed using reduction. Some(2), in the cardinality Conversational case-based reasoning and case base
criteria for category 3, implies that at least two of the browsing require that the query (and the context) be
attribute-value pairs need to match for the whole cate- iteratively changed, by restricting or relaxing it. The
gory to match. This constraint could be further relaxed process of restricting and relaxing contexts can be re-
to Some(1). Additionally, category 2 could be relaxed peated and interwoven until the agent is satisfied with
from Some(2) to Some(1), as illustrated in Fig. 1. the quantity and the relevance of the retrieved cases. It
In the second scenario, value constraints are relaxed. is apparent that after the context is modified by relax-
Since all categories required instance matches, context ation/restriction, the system must re-evaluate the query,
values could be updated by using domain knowledge. i.e., to search the case base for similar cases. A naive
Thus, the constraint for the AGE attribute could be approach takes the new query and submits it to the sys-
changed to 24–29 (as shown in Fig. 1), as an imme- tem (see Fig. 4). A more sophisticated approach could
diate generalization of 28. In addition, a user-guided take advantage of an already processed query by incre-
relaxation may be a better option, since the user might mentally modifying its result [23]. Thus, the system
have domain knowledge not represented in the system. will not search the whole case base again, but will only
 Efficient Conversational CBR Systems 257

Figure 3. Context restriction by enlarging the number of attributes required to match and by reducing the set of allowable values for an attribute.

Figure 4. Naive iterative case retrieval algorithm. Context is initialized with the attributes and constraints from the input case. Special counters
are used to prevent repeating context restrictions and relaxations forever. Context transformations modify attributes of the least important
category first. Only one category is transformed at a time.
258 Jurisica, Glasgow and Mylopoulos
modify set of retrieved relevant cases accordingly. In
the next section, we introduce such an algorithm.
4. Incremental Context Modifications
The basic idea of incremental query processing is to
store query results and reuse them to compute related
queries [10]. Incremental algorithms have previously
been successfully applied to database systems for view
maintenance [10–13]. Incremental view maintenance
algorithms avoid a complete query re-computation by
changing only relevant parts of the answer or view.
Usually, an incremental approach is substantially more
efficient than a naive one. Efficiency improvement is
higher when several consecutive changes to the query
are required (i.e., during iterative browsing), or when a
large information base is used. Small updates to the
query generally produce only small changes to the
query result. Then an incremental approach performs
only local changes to the query [13].
In general, an incremental view maintenance al-
gorithm handles deletions, negations, updates, aggre-
gation and recursion. Various approaches have been
proposed to tackle these problems. One well known
incremental view maintenance algorithm is a count-
ing algorithm, which supports delete and re-derive op-
erations [24]. It assumes universal materialization of
predicates and stores counts of the number of deriva-
tions to be associated with each tuple. Another al-
gorithm derives production rules to maintain selected
SQL views, namely views without duplicates, aggre-
gation and negation [11].
Here we propose an incremental retrieval algorithm
for CBR systems based on a nearest-neighbor matching
algorithm [14] with the following modifications:
1. grouping attributes into categories of different im-
portance to gain a fine-grain control over the match-
ing process, and diminish the negative effect of
irrelevant attributes on competence;
2. using an explicit context during similarity assess-
ment to recall only relevant cases; and
3. accelerating query processing via incremental con-
text transformation during query relaxation.
After the context is modified by relaxation/restri-
ction the amount of necessary modification to a query
in an incremental approach can be, to some degree,
controlled by collecting and using extra information
produced during query evaluation. Although this may
require additional storage, the overall efficiency is
usually improved. Thus, during an incremental com-
putation the result of a query is reused to evaluate a
subsequent query more efficiently.
If the number of attributes per case is significantly
smaller than the total number of cases in the case base
then incremental context modification outperforms
computation of the answer from scratch. Our system
supports incremental context modification when partial
results are kept at the attribute or category level. The
first approach requires extra storage space, but is more
versatile and is thus suitable when context is changed
frequently and when cases have fewer attributes. The
second approach is a compromise between efficiency
gain from an incremental approach and modest stor-
age requirement. It is useful for less frequent context
changes and for case bases containing cases with a large
number of attributes. Next, we explain the rationale be-
hind the first approach. (The second approach works
analogously.)
Consider context relaxation with the naive and incre-
mental retrieval algorithms (see Figs. 4 and 5). During
k iterative context modifications the naive case retrieval
algorithm requires k iterations, while the incremental
algorithm handles it with only a single iteration. Al-
though the initial evaluation is the same for both algo-
rithms, the incremental approach stores partial match-
ing results on the attribute level in the set of cases
Answeri . If a case is a member of Answeri it means
that it satisfies the constraint on attributei .
During an iterative retrieval of cases, the system
modifies constraints on attributes. A naive retrieval al-
gorithm (see Fig. 4) produces Answer0 by determining
which cases in a case base satisfy constraints on all at-
tributes defined in the context, i.e., for all Answer.ai .
The incremental retrieval algorithm reuses Answer.ai
to produce the set of relevant cases Answer0 (see Fig. 5).
Relaxing attribute ai changes a set of matching cases
included in Answer.ai . However, all remaining partial
answers remain unchanged. Thus, Answer0 can be con-
structed by adding cases that satisfy both the initial
context and the relaxed constraint on attribute ai to
the Answer. Restricting attribute ai results in creat-
ing Answer0 by removing cases that do not satisfy
additional constraints from Answer. Determining satis-
fiability only requires testing if the case in the set of
retrieved cases must be removed either because it needs
an excluded value to match or because it cannot match
an added attribute.

Figure 5. Incremental case retrieval algorithm. Initial Answer is set to ∅. A user specifies LowerLimit and UpperLimit to set a desired amount
of cases to be retrieved.
The idea of incremental context relaxation and
restriction has evolved from the notion of diffe-
rential queries [25]. First, parts of the context af-
fected by the transformation are determined. Second,
only affected parts of context are recomputed. We
express this process using context addition and
difference.10 Thus, the incremental context transfor-
mation algorithm can be formalized as follows:
Context0 = Context + Context+ − Context− , where Co-
ntext + and Context− denote all attributes and their
associated constraints that need to be added or
removed.
Without loss of generality, it is assumed that
Context+ and Context− are contexts with a single
attribute-constraint pair. Because the context transfor-
mation process is iterative, more complex constraints
can be created through multiple iterations. Next we
formalize context transformations for our incremental
retrieval algorithm (see Fig. 5) and show how the final
set of retrieved cases (Answer0 ) can be constructed us-
ing partial results.
Efficient Conversational CBR Systems 259
Reduction involves removing an attribute-value pair
from a context. This can be done either permanently—
an attribute ak is removed from the context: Context0 =
Context − ak , or dynamically—an m-of-n matching is
used. Thus, for the reduced context, the resulting set of
cases is generated as a union of the partial results of the
set of cases that satisfy individual attribute constraints
of the original context (Answer.ai ), without consider-
ing the constraints on the removed attribute:
Answer0 = ∩Answer.ai /Answer.ak .
Expansion involves adding an attribute-value pair
to a context: Context0 = Context + Context+ , where
Context+ = {hai , {Vi }i}. Thus, removing cases that
do not satisfy the constrained context from the set of
cases generates the set of retrieved cases:
Answer0 = Answer ∩ Answer.ai0 ,
where Ci ∈ Answer.ai0 iff sat(Ci , Context+ ).
260 Jurisica, Glasgow and Mylopoulos
Generalization involves enlarging the set of
allowable values for a given context: Context0 =
Context + Context+ , where Context+ = {hai , {Vi }i}.
Thus, the set of retrieved cases is generated as an in-
tersection of the set of cases that satisfy the original
context and the set of cases that satisfy the context
change Context+ :
Answer0 = Answer ∩ Answer.ai0 ,
where Ci ∈ Answer.ai0 iff sat(Ci , Context+ ).
Specialization involves removing values from a
constraint set for an attribute defined in a context:
Context0 = Context − Context− , where Context− =
{hai , {Vi }i}. Thus, the set of retrieved cases (Answer0 )
is generated by removing cases that do not satisfy the
restricted context from Answer:
Answer0 = Answer ∩ Answer.ai0 ,
where Ci ∈ Answer.ai0 iff sat(Ci , Context− ).
5. Performance Evaluation
Various approaches can be used to evaluate system
performance. Available methods evaluate either com-
petence of the system or its scalability. The former
measures the capabilities of the system and can be
assessed by precision/recall or accuracy/coverage mea-
sures. Scalability assesses system dependence on im-
portant factors, such as case base size, case and con-
text complexity, and number of context transformation
applied during iterative case retrieval.
We have tested both the competence and scalability
of the proposed system [26]: (i) learning control—
solving the inverse kinematic task for a three-link
spherical, angular robot [27], (ii) classification into a
continuous class—predicting the rise time of a ser-
vomechanism in terms of two continuous gain set-
tings and two discrete choices of mechanical linkages
[27], (iii) prediction in medicine—suggesting a cost-
effective treatment for in-vitro fertilization patients
without compromising the probability for successful
pregnancy [9], (iv) letter recognition [8] and (v) soft-
ware reuse and program understanding—similarity-
based retrieval of software artifacts [28, 29]. In [30] we
show how a generic CBR system prototype TA3 can
be custom-tailored to satisfy specific requirements in
individual domains. Errico and Jurisica [31] show how
case-based reasoning can be applied to user modeling
to support adaptive agent-based systems for electronic
commerce. Here we report on efficiency improvement
achieved by using an incremental query modification.
Precision and recall are used to evaluate information
retrieval systems, while accuracy and coverage evalu-
ate machine-learning algorithms. The former bench-
marks are used to assess how good the retrieval is,
i.e., how much relevant information is retrieved and
how much irrelevant information is included. The lat-
ter benchmarks are used to measure accuracy of the
learned rule and the portion of an information base it
covers. Given the nature of information retrieval sys-
tems, when recall rises, generally precision decreases
and vice versa. In contrast, variable-context similar-
ity assessment helps to achieve both high precision
and high recall. Two iterative retrieval approaches yield
high precision and recall simultaneously:
1. Conservative approach: The objective is to prevent
retrieving irrelevant items, even if some relevant
items are missed. After the initial set of relevant
items is available, we could iteratively consider
neighbors and include only relevant ones (so that
precision is not decreased). This approach is ap-
propriate if the user is knowledgeable in the prob-
lem domain, since the retrieval process must be
constrained—to eliminate irrelevant matches, and
similarity-based—to retrieve relevant neighbors of
initial matches. The conservative approach can be
explained as precision-oriented retrieval with itera-
tive query tuning to increase recall, while preserving
high precision.
2. Novice approach: The emphasis is on retrieving
all relevant items, even at the price of including
some irrelevant ones. After the initial pool of items
is retrieved, irrelevant items are iteratively removed.
This approach is suitable for novices, i.e., users who
do not have initial knowledge about the structure of
the repository. Thus, they may not specify a query
that ensures high precision, but using the result and
feedback information they may iteratively prune the
set of retrieved items. The novice approach can be
explained as a recall-oriented retrieval with itera-
tive query tuning to increase precision, while pre-
serving perfect recall.
From the above description it follows that flexible
criteria are required to guide and control retrieval. Usu-
ally, information retrieval systems are tuned for spe-
cific applications; thus, supporting either precision-
or recall-oriented retrieval. Next, we describe how
variable-context similarity assessment can be used to

support flexible retrieval while ensuring high precision
and high recall.
Variable-context similarity assessment assures that
all cases that satisfy the current context are similar
with respect to the context. Thus, precision and recall is
100% with respect to the given context. However, we
have to take into account that the original query may
be relaxed or restricted. Thus, we make a distinction
between precision according to the query (precisionq )
and precision according to the user (precisionu ). Sim-
ilarly, we distinguish recall with respect to a query
(recallq ) from recall according to a user’s original re-
quest (recallu ).
After making this terminological distinction, we re-
turn to the high precision and recall claim. We can
prove that precisionq and recallq is 100%. However,
precisionu and recallu may or may not be equal to
100%, depending on the context.
For scalability evaluation we define a performance
model of TA3. The performance model simulates
TA3’s scalability as a function of case base size,
case representation complexity, query complexity, and
context-modification strategy used. As a validation of
this model, we compare experimental efficiency eval-
uation and scalability evaluation of TA3 on respective
case base sizes, case representation complexity, and
context complexity.
5.1. The Performance Model of TA3
From the algorithms presented in Figs. 4 and 5 it fol-
lows that k consecutive context modifications require
k applications of the naive algorithm. Thus, a naive
approach requires k × |CaseBase| × |Context| evalua-
tions. In contrast to this, an incremental algorithm is
evaluated only once to produce Answerpart and subse-
quent context modifications are handled by incremental
changes to the result.
Assuming that the initial retrieval result |Answerpart |
is substantially smaller than the case base size
|CaseBase|, producing Answer0 incrementally is sig-
nificantly more efficient. For all categories that are not
affected by a context modification, no recomputation
is necessary. For categories affected by a change, lo-
cal changes handle the recomputation as presented in
earlier. It should be noted that both algorithms could
be improved by indexing, which avoids accessing all
cases in a case base.
Iterative retrieval is based on the algorithm defined
in Fig. 4. The complexity of iterative retrieval depends
not only on the number of cases in a case base but
also on the user criteria, the number of retrieved cases,
Efficient Conversational CBR Systems 261
and the number of consecutive iterations. A perfor-
mance model of individual context transformations is
presented in below.
Iterative retrieval with standard reduction depends
on the case base size, context, and number of iterative
context reductions:
à !
Xk
|CaseBase| |Context| + k|Context| − i ,
1
where |Context| is the context complexity, |CaseBase|
is the case base size, and k is the number of iterations.
Iterative retrieval with incremental reduction de-
pends on the case base size, context, set of returned
cases, and number of iterative context reductions:
à !
Xk
|Context| |CaseBase| + k|Answer| − |Answer| i ,
1
where |Context| is the context complexity, |CaseBase|
is the case base size, |Answer| is the number of returned
cases, and k is the number of iterations.
Iterative retrieval with standard expansion depends
on the case base size, context, and number of iterative
context expansions:
à !
Xk
|CaseBase| k|Context| + i ,
1
where |Context| is the context complexity, |CaseBase|
is the case base size, and k is the number of iterations.
Iterative retrieval with incremental expansion de-
pends on the case base size, context, set of returned
cases, and number of iterative context expansions:
à à !!
X k
|Context| |CaseBase| + |Answer| k + i ,
1
where |Context| is the context complexity, |CaseBase|
is the case base size, |Answer| is the number of returned
cases, and k is number of iterations.
Iterative retrieval with standard generalization de-
pends on the case base size, context, and number of
iterative context generalizations:
|CaseBase||Context|(k + 1),
where |Context| is the context complexity, |CaseBase|
is the case base size, and k is number of iterations.
Iterative retrieval using incremental generalization
depends on the case base size, context, and number of
iterative context generalizations:
|CaseBase|(k+ |Context|),
262 Jurisica, Glasgow and Mylopoulos
where |Context| is the context complexity, |CaseBase|
is the case base size, and k is number of iterations.
Iterative retrieval with standard specialization de-
pends on the case base size, context, and number of
iterative context specializations:
|CaseBase||Context|(k + 1),
where |Context| is the context complexity, |CaseBase|
is the case base size, and k is number of iterations.
Iterative retrieval with incremental specialization
depends on the case base size, context, set of returned
Figure 6. Cost of retrieval for standard and incremental reduction as a function of case base size (|CB|) and size of the context (|Context|).
Ten consecutive relaxations are considered.
cases, and number of iterative context specializations:
|Context|(|CaseBase| +k |Answer|),
where |Context| is the context complexity, |CaseBase|
is the case base size, |Answer| is the number of returned
cases, and k is number of iterations.
Next we show efficiency and scalability evaluation
using the performance model of TA3. On the basis of
experiments from real-world domains, we assume that
ten cases are returned in average, and that ten consec-
utive context transformations are performed. It should

Figure 7. Cost of retrieval for standard and incremental expansion as a function of case base size (|CB|) and size of the context (|Context|).
Ten consecutive restrictions are considered.
be noted that if fewer than ten cases are returned, or
more than ten consecutive context transformations are
performed, then the incremental approach is even more
efficient than a naive algorithm.
5.2. Efficiency and Scalability Evaluation of TA3
We have used the complexity of incremental implemen-
tation of context transformations to create a model that
fits TA3’s performance on real-world domains [8, 9].
We parameterize this model to determine the scalability
of TA3 with respect to (1) case base size, (2) query
Efficient Conversational CBR Systems 263
complexity (context), and (3) number of consecutive
iterations for individual context transformations. In
the model we assumed that partial evaluations are kept
on attribute level. Thus, more storage space is required,
but the system is more efficient when frequent changes
to the context are made.
Presented results support the claim that the incre-
mental context manipulation is more efficient than
the naive approach. Performance improvement is in-
creased when the case base size is substantial, cases
have many attributes and several subsequent context
modifications are required. The results presented in
264 Jurisica, Glasgow and Mylopoulos
Figure 8. Cost of retrieval for standard and incremental generalization as a function of case base size (|CB|) and size of the context (|Context|).
Ten consecutive relaxations are considered.
Figs. 6–9 show that the incremental approach improves
performance by 70% on average. However, for sim-
ple retrievals—small case base and/or no consecu-
tive context modifications—the naive approach is more
efficient.
6. Conclusions
Research in database management systems has long
been producing scalable incremental algorithms for
view maintenance. Gupta et al. [13] present an incre-
mental algorithm for recomputing the view in response
to changes in the view definition. The authors consider
an SQL Select-From-Where-GroupBy, Union and Ex-
cept views, and present local adaptation strategies using
the old view materialization. Their methods for adapt-
ing the Where part of an SQL view are similar to our
context modifications. However, they do not support
cardinality relaxation and restriction. Blakeley et al.
[25] introduced an incremental view maintenance algo-
rithm that supports only base relations updates. In con-
trast, TA3 also supports updates to context (i.e., views).

Figure 9. Cost of retrieval for standard and incremental specialization as a function of case base size (|CB|) and size of the context (|Context|).
Ten consecutive restrictions are considered.
FRANK is a case retrieval system applied in a medi-
cal domain for back-injury diagnosis [32]. The user
provides a description of a patient’s symptoms and
selects from a hierarchy of report types. The user’s
top-level considerations are filtered through the sys-
tem’s processing using a flexible control mechanism.
The plan is then selected based on the report type. The
task mechanism controls queries to the case base—if a
query is not successful, then the system resubmits the
query with altered initial values. Thus, it is similar to
our notion of iterative query modification.
Efficient Conversational CBR Systems 265
CBR has previously been successfully applied in var-
ious domains [33]. However, in many of these studies,
only small case bases were considered, often because
the implemented system could not efficiently support
retrieval for larger case bases. Various approaches, such
as indexing [34–36] and selective forgetting [37, 38],
have been proposed to improve the performance of
CBR systems without decreasing competence. Parallel
architectures have also been suggested as a way of in-
creasing efficiency [39]. Research on conversational
CBR systems has focused on reducing the number
266 Jurisica, Glasgow and Mylopoulos
of questions posed and increasing their relevancy
[7], supporting dialogue inferencing [40], case base
authoring [41], and maintaining case bases [42, 43].
Respective approaches have been proposed that in-
tegrate the CBR paradigm with model-based reason-
ing [44], machine learning [45], and high-performance
query tools [46, 47]. Our approach to performance
improvement is based on incremental context modifi-
cation, an approach which has previously proven suc-
cessful in database applications.
Results presented in this paper support the claim that
incremental context manipulation is generally more ef-
ficient then a naive approach. An incremental approach
improves retrieval efficiency without reducing system
competence. Performance improvement is increased
when a large case base is used, when cases have many
attributes or when several subsequent context modifica-
tions are required. In other words, this approach is most
suitable for iterative browsing in complex domains and
thus conversational case-based reasoning systems may
benefit from it. For simple retrievals involving a small
case base and/or no consecutive context modifications,
the naive approach may be preferable since time per-
formance is not an issue and no extra storage space is
required.
The performance model simulates TA3’s scalability
as a function of case base size, case representation com-
plexity, query complexity, and context-modification
strategy used. As a validation of this model, we com-
pare experimental efficiency evaluation and scalability
evaluation of TA3 on respective case base sizes, case
representation complexity, and context complexity.
The proposed incremental algorithm is general in the
sense that it is applicable for retrieval in any automated
reasoning or decision-support system that represents
knowledge as attribute-value pairs and problems are
solved by iteratively accessing and using previously
derived information.
In addition to evaluations done earlier, we are cur-
rently applying the proposed system to two biomedical
applications: reproductive medicine and crystallogra-
phy experiment design [48–50]. In both application
areas the case base is large and complex, and the rea-
soning is done iteratively. Preliminary results show
that an incremental conversational CBR works well for
symbolic data. However, in both domains we need to
extend the representation of domain knowledge to in-
clude image representations. At the present time, we
use image feature extraction algorithms to transform
images into symbolic information, so that the proposed
incremental context manipulation works [51]. We are,
however, considering modifications of the incremental
methods for image processing.
Acknowledgments
The research described in this paper was supported by
Natural Sciences and Engineering Research Council,
Communications and Information Technology Ontario
and IBM Canada.
Notes
1. A reader may find detailed descriptions of CBR process and
systems in [52]. More recent research directions are presented
in [53] and and practically-oriented descriptions of CBR can be
found in [54, 55].
2. This modification can be done by the user or the system.
3. Context modifications are similar to knowledge manipulation
transmutations [56].
4. More details are provided in Section 3.1.
5. Additional details on the theory, performance evaluation and
complexity analysis of TA3 are provided in [26].
6. Note that this is a low-level representation and even complex
case representations can be created using this schema.
7. Relevant attributes can be located using a knowledge-discovery
algorithm [9].
8. This is similar to the notion of “similarity in the context of a
given set of descriptors” [56, 57].
9. Reduction and generalization implement cardinality and value
constraints respectively.
10. We define context addition and difference in terms of standard
set-theoretic operations. Individual attributes could be added,
removed or constraints on these attributes could be expanded or
restricted.
References
1. P. Constantopoulos and E. Pataki, “A browser for software
reuse,” in Proc. of the CAiSE’92, edited by P. Loucopoulos,
Berlin: Springer, 1992, pp. 304–326.
2. T.P. Martin, H.-K. Hung, and C. Walmsley, “Supporting brows-
ing of large knowledge bases,” Technical report, Department
of Computing and Information Science, Queen’s University,
Kingston, ONT, 1992.
3. M.B. Twidale, D.M. Nichols, and C.D. Paice, “Browsing is
collaborative process,” Information Processing & Management,
vol. 33, no. 6, pp. 761–783, 1997.
4. R.J. Miller, O.G. Tsatalos, and J.H. Williams, “DataWeb: Cus-
tomizable database publishing for the web,” IEEE MultiMedia,
pp. 14–21, 1997.
5. D.W. Aha and L.A. Breslow, “Refining conversational case rea-
soning,” in Proc. of the 2nd International Conference on Case-
Based Reasoning, Providence, RI, 1997, pp. 267–278.
6. K. Hammond, R. Burke, and K. Schmitt, “A case-based approach
to knowledge navigation,” in Leake, 1996, pp. 125–136, 1996.

7. D.W. Aha, L.A. Breslow, and T. Maney, “Supporting conver-
sational case-based reasoning in an integrated reasoning frame-
work,” Technical Report AIC-98-006, Naval Research Labora-
tory, Navy Center for Applied Research in Artificial Intelligence,
Washington, DC, 1998.
8. I. Jurisica and J. Glasgow, “Case-based classification using
similarity-based retrieval,” International Journal of Artificial In-
telligence Tools. Special Issue of IEEE ICTAI-96 Best Papers,
vol. 6, no. 4, pp. 511–536, 1997.
9. I. Jurisica, J. Mylopoulos, J. Glasgow, H. Shapiro, and R. Casper,
“Case-based reasoning in IVF: Prediction and knowledge min-
ing,” AI in Medicine, vol. 12, no. 1, pp. 1–24, 1998.
10. L. Bækgaard and L. Mark, “Incremental computation of time-
varying query expressions,” IEEE Trans. on Knowledge and
Data Engineering, vol. 7, no. 4, pp. 583–589, 1995.
11. S. Ceri and J. Widom, “Deriving production rules for incremental
view maintenance,” in VLDB-91, Barcelona, Spain, 1991, pp.
577–589.
12. T. Griffin and L. Libkin, “Incremental maintenance of views
with duplicates,” in ACM SIGMOD, San Jose, CA, 1995, pp.
328–339.
13. A. Gupta, I. Mumick, and K. Ross, “Adapting materialized views
after redefinitions,” in ACM SIGMOD, San Jose, CA, 1995, pp.
211–222.
14. D. Wettschereck and T. Dietterich, “An experimental compar-
ison of the nearest neighbor and nearest hyperrectangle algo-
rithms,” Machine Learning, vol. 19, no. 1, pp. 5–27, 1995.
15. J. Frawley and G. Piatetsky-Shapiro, Knowledge Discovery in
Databases, AAAI Press, 1991.
16. D. Aha, “Tolerating noisy, irrelevant and novel attributes in
instance-based learning algorithms,” International Journal of
Man—Machine Studies, vol. 36, no. 2, pp. 267–287, 1992.
17. J. Ortega, “On the informativeness of the DNA promoter
sequences domain theory,” Journal of Artificial Intelligence
Research, vol. 2, pp. 361–367, Research Note, 1995.
18. P.R. Thagard, K.J. Holyoak, G. Nelson, and D. Gotchfeld, “Ana-
log retrieval by constraint satisfaction,” Artificial Intelligence,
vol. 46, pp. 259–310, 1990.
19. I. Jurisica, “Supporting flexibility. A case-based reasoning
approach,” in The AAAI Fall Symposium. Flexible Computa-
tion in Intelligent Systems: Results, Issues, and Opportunities,
Cambridge, MA, 1996.
20. B. D’Ambrosio, “Process, structure, and modularity in reason-
ing with uncertainty,” in Uncertainty in Artificial Intelligence
edited by R. Shachter, T. Levitt, L. Kanal, and J. Lemmer, vol.
4, pp. 15–25, 1990.
21. E. Horvitz, “Reasoning under varying and uncertain resource
constraints,” in Proc. of AAAI-88, 1988, pp. 111–116.
22. T. Gaasterland, “Restricting query relaxation through user con-
straints,” in Proc. Int. Conf. on Intelligent and Coop. Inf. Systems,
Rotterdam, 1993, pp. 359–366.
23. F. Bancilhon, “Naive evaluation of recursively defined relations,”
Knowledge Base Management Systems, edited by M. Brodie and
J. Mylopoulos, pp. 165–178, 1986.
24. A. Gupta, I. Mumick, and V. Subrahmanian, “Maintaining views
incrementally,” in Proc. of the 12th ACM SIGACT-SIGMOD-
SIGART Symposium on Principles of Database Systems, 1993,
pp. 157–166.
25. J.A. Blakeley, P.-A. Larson, and F.W. Tompa, “Efficiently up-
dating materialized views,” in ACM-SIGMOD, 1986, pp. 61–71.
Efficient Conversational CBR Systems 267
26. I. Jurisica, “TA3: Theory, implementation, and applications
of similarity-based retrieval for case-based reasoning,” Ph.D.
dissertation, University of Toronto, Department of Computer
Science, Toronto, Ontario, 1998.
27. I. Jurisica and J. Glasgow, “A case-based reasoning approach to
learning control,” in 5th Int. Conf. on Data and Knowledge Sys-
tems for Manufacturing and Engineering, DKSME-96, Phoenix,
AZ, 1996.
28. I. Jurisica, “Similarity-based retrieval for diverse Bookshelf soft-
ware repository users,” in IBM CASCON Conference, Toronto,
Canada, 1997, pp. 224–235.
29. H. Dayani-Fard and I. Jurisica, “Reverse engineering by mining
dynamic repositories,” in 5th Working Conference on Reverse
Engineering (WCRE’98), Honolulu, Hawaii, 1998, pp. 174–
182.
30. I. Jurisica and B. Nixon, “Building quality into case-based rea-
soning systems,” in CAiSE*98, 1998, Lecture Notes in Computer
Science.
31. B. Errico and I. Jurisica, “Adaptive agent-based systems for the
Web: An application to the NECTAR project,” in AAAI Spring
Symposium on Intelligent Agents in Cyberspace, Stanford, CA:
AAAI Press, 1999.
32. E.L. Rissland, J.J. Daniels, Z.B. Rubinstein, and D.B. Skalak,
“Case-based diagnostic analysis in a blackboard architecture,”
in Proc. of AAAI-93, 1993.
33. I.D. Watson, Applying Case-Based Reasoning: Techniques for
Enterprise Systems, San Francisco, CA: Morgan Kaufmann Pub-
lishers, 1997.
34. R. Barletta and W. Mark, “Explanation-based indexing of cases,”
in Proc. of AAAI-88, 1988, pp. 541–546.
35. A. Ram, “Indexing, elaboration and refinement: Incremental
learning of explanatory cases,” Machine Learning, vol. 10, no. 3,
pp. 201–248, 1993.
36. C.M. Seifert, “Case-based learning—predictive features in in-
dexing,” Machine Learning, vol. 16, nos. 1, 2, pp. 37–56,
1994.
37. B. Smyth and M.T. Keane, “Remembering to forget: A
competence-preserving case deletion policy for case-based rea-
soning systems,” in Proc. of the 14th IJCAI. Montreal, Quebec,
1995, pp. 377–382.
38. H. Watanabe, K. Okuda, and S. Fujiwara, “A strategy for forget-
ting cases by restricting memory,” IEICE Trans. on Information
and Systems, vol. E78D, no. 10, pp. 1324–1326, 1995.
39. E. Sumita, N. Nisiyama, and H. Iida, “The relationship between
architectures and example-retrieval times,” in Proc. of AAAI,
Seattle, 1994, pp. 478–483.
40. H. Shimazu, A. Shibata, and K. Nihei, “Case-based retrieval
interface adapted to customer-initiated dialogues in help desk
operations,” in Proc. of the 12th National Conference on Artifi-
cial Intelligence. Seattle, WA, 1994, pp. 553–564.
41. K.M. Gupta, “Case base engineering for large scale industrial
applications,” in AAAI Spring Symposium Series on Knowledge
Management, Stanford, CA, 1997.
42. K. Racine and Q. Yang, “Maintaining unstructured case bases,”
in Proc. of the 2nd International Conference on Case-Based
Reasoning, Providence, RI, pp. 553–564, 1997.
43. B. Smyth, “Case base maintenance,” in 11th International
Conference on Industrial Engineering Applications of Artifi-
cial Intelligence and Expert Systems (IEA/AIE’98). Benicassim,
Castellon, Spain, 1998, pp. 507–516.
268 Jurisica, Glasgow and Mylopoulos
44. M.P. Feret and J.I. Glasgow, “Combining case-based and model-
based reasoning for the diagnosis of complex devices,” Applied
Intelligence, vol. 7, no. 1, pp. 57–78, 1997.
45. L.A. Breslow and D.W. Aha, “NaCoDAE: Navy conversational
decision aids environment,” Technical Report AIC-97-018,
Naval Research Laboratory, Navy Center for Applied Research
in Artificial Intelligence, Washington, DC, 1997.
46. H. Munoz-Avila, J.A. Hendler, and D.W. Aha, “Conversational
case-based planning,” Review of Applied Expert Systems, vol. 5,
pp. 163–174, 1999.
47. I. Jurisica and J. Glasgow, “An efficient approach to itera-
tive browsing and retrieval for case-based reasoning,” in 11th
International Conference on Industrial Engineering Applica-
tions of Artificial Intelligence and Expert Systems (IEA/AIE’98),
Benicassim, Castellon, Spain, 1998, Vol. 2 of Lecture Notes in
Computer Science, LNAI 1416, pp. 535–546.
48. J. Luft, M. Bianca, I. Jurisica, P. Rogers, J. Glasgow, S. Fortier,
and G. DeTitta, “An opening strategy for macromolecular crys-
tallization: Case-based reasoning and the exploitation of a
precipitation reaction outcome database. in Conference of the
American Crystallography Association (ACA99), Buffalo, NY.
An abstract for an oral presentation, 1999.
49. J. Luft, M. Bianca, L.M. Owczarczak, D.R. Weeks, I. Jurisica,
P. Rogers, J. Glasgow, S. Fortier, and G. DeTitta, “The devel-
opment of high throughput methods for macromolecular micro-
batch crystallization,” in Recent Advances in Macromolecular
Crystallization, San Diego, CA, An abstract for an oral presen-
tation, 1999.
50. I. Jurisica, G. DeTitta, J. Luft, J. Glasgow, and S. Fortier,
“Knowledge management in scientific domains,” in AAAI-99
Workshop on Exploring; Synergies of Knowledge Manage-
ment and Case-Based Reasoning, Orlando, FL, pp. 30–34,
1999.
51. J. Glasgow and I. Jurisica, “Integration of case-based and image-
based reasoning,” in AAAI’98 Workshop on Case-Based Reason-
ing, edited by D.W. Aha, Madison, WI, pp. 67–74, 1998.
52. J.L. Kolodner, Case-Based Reasoning, San Mateo, CA: Morgan
Kaufmann, 1993.
53. D. Leake (ed.), Case-Based Reasoning: Experiences, lessons
and future directions, AAAI Press, 1996.
54. I.D. Watson, Applying Case-Based Reasoning: Techniques for
Enterprise Systems, San Francisco, CA: Morgan Kaufmann
Publishers, 1997.
55. R. Bergmann, S. Breen, M. Goker, M. Manago, and S. Wess, De-
veloping Industrial Case-Based Reasoning Applications: The
INRECA Methodology, Berlin: Springer, 1999.
56. R.S. Michalski, “Inferential theory of learning: Developing
foundations for multistrategy learning,” in Machine Learning:
A Multistrategy Approach, vol. IV, 1994.
57. A.M. Collins and R.S. Michalski, “The logic of plausible-
reasoning: A core theory,” Cognitive Science, vol. 13, pp. 1–49,
1989.