Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Artificial Intelligence In Medicine

journal homepage: www.elsevier.com/locate/artmed

Towards automatic encoding of medical procedures using convolutional

neural networks and autoencoders
Yihan Denga, , André Sanderb, Lukas Faulstichb, Kerstin Deneckea
⁎

a
Bern University of Applied Sciences, Medical Informatics, Biel, Switzerland
b
ID Information und Dokumentation im Gesundheitswesen GmbH & Co. KGaA, Berlin, Germany

ARTICLE INFO ABSTRACT

Keywords: Classification systems such as ICD-10 for diagnoses or the Swiss Operation Classification System (CHOP) for
Automatic encoding procedure classification in the clinical treatment are essential for clinical management and information ex-
Deep learning change. Traditionally, classification codes are assigned manually or by systems that rely upon concept-based or
Classification system rule-based classification methods. Such methods can reach their limit easily due to the restricted coverage of
Autoencoder
handcrafted rules and of the vocabulary in underlying terminological systems. Conventional machine learning
Convolutional neural networks
approaches normally depend on selected features within a human annotated training set. However, it is quite
laborious to obtain a well labeled data set and its generation can easily be influenced by accumulative errors
caused by human factors. To overcome this, we will present our processing pipeline for query matching realized
through neural networks within the task of medical procedure classification. The pipeline is built upon con-
volutional neural networks (CNN) and autoencoder with logistic regression. On the task of relevance determi-
nation between query and category text, the autoencoder based method has achieved a micro F1 score of
70.29%, while the convolutional based method has reached a micro F1 score of 60.86% with high efficiency.
These two algorithms are compared in experiments with different configurations and baselines (SVM, logistic
regression) with respect to their suitability for the task of automatic encoding. Advantages and limitations are
discussed.

1. Introduction
In order to claim costs to the health insurance as well as for clinical
documentation purposes, hospitals and physicians are legally bound to
encode diagnoses and procedures with classification codes from re-
levant classification systems. In Switzerland, these are ICD-10-GM for
diagnoses and the Swiss Operation Classification System (CHOP)1 for
clinical and surgical treatments.
1.1. Challenges in automatic encoding
In this study, we are focusing on the task of automatic encoding
based on CHOP. In total, the CHOP system consists of 18 different ca-
tegories and over 14,000 different classification codes. In order to
realize automatic encoding or search for relevant classification codes,
respectively, rule-based classification systems have been developed to
achieve an automatic encoding [1]. As input, physicians and coding
assistants use free-text queries within their search whereas the output is
a set of possible classification codes. Consider the example in Fig. 1: A
medical documentation assistant wants to encode a procedure reflected
by the keywords Chirurgie CT und MR (surgery, computed tomography
and magnetic resonance in English). For this query, the assistant needs
to conduct a top-down dictionary look-up within the CHOP classifica-
tion system, which goes from the top level chapter C0 (measurement
and intervention) down to subcategory 00.3 (computer assisted sur-
gery). Subcategory 00.3 is further divided in several subcategories: the
category computer assisted surgery with “CT and MR” represented by
“Computergesteuerte Chirurgie mit mehreren Datenquellen” is one of
the six subcategories under “computer assisted surgery”. All sub-
categories of 00.3 achieve at least a partial match with the query
whereas the subcategories differ only slightly. The best results for
partial matches are achieved for the codes 00.31 and 00.32 that contain
at least one of the imaging procedures requested. Based on their
background knowledge, the encoding assistants are able to decide the

Corresponding author.
⁎

E-mail address: yihan.deng@bfh.ch (Y. Deng).

1
https://www.bfs.admin.ch/bfs/de/home/statistiken/gesundheit/nomenklaturen/medkk/instrumente-medizinische-kodierung.assetdetail.1940914.html, ac-
cessed on September 9, 2017.

https://doi.org/10.1016/j.artmed.2018.10.001
Received 20 October 2017; Received in revised form 7 September 2018; Accepted 3 October 2018
0933-3657/ © 2018 Elsevier B.V. All rights reserved.

Please cite this article as: Deng, Y., Artificial Intelligence In Medicine, https://doi.org/10.1016/j.artmed.2018.10.001
Y. Deng et al.
Fig. 1. Query scenario and snippet from CHOP on category computer assisted surgery.
correct code is 00.35 (computer assisted surgery with multiple data
sources). However, for an automatic encoding this would require se-
mantic knowledge that CT and MR are imaging data sources.
In encoding systems, automatic extraction approaches and rule-
based encoding approaches are employed to handle the user query and
determine the matching between query and category text. Such
methods extract specific information from query and category text
automatically and support in classifying them according to standard
medical terminologies [2]. Hence, comprehensive rules are specified
beforehand considering all the possible query situations, which is often
highly time-consuming. It is also difficult to achieve completeness and
correctness in the rule set. Conventional machine learning approaches
aim at learning correlation between query and CHOP encoding auto-
matically and deriving latent models from large data sets. Although
manually chosen features have proven to be effective for some specific
classification tasks, the biases caused by feature selection at each pro-
cessing step is unavoidable. For this kind of encoding with multiple
subcategories, it is particularly difficult to obtain sufficient training
examples for each sub-category to make the classifier differentiable for
instance in the corresponding classes. Without these examples, a
trained classifier would have a reduced ability of discrimination for
each subcategory and would produce a large number of false positive
codes.
With the resurgence of neural networks and deep learning techni-
ques, the end-to-end methods for data representation learning and ob-
ject/pattern recognition in both, text and image, have advanced sub-
stantially. For a classification task based on natural language text,
words and relations between words can be learned through semantic
preserving feature vectors [3]. Besides, the convolutional neural net-
works (CNN) [4,5] enable a minimal manual pre-processing on the
input data and learn the feature maps through applying a sliding
window on the original input according to a pre-defined filtering size
and stride. The features corresponding to different filters can be se-
lected by the convolution and pooling process (max pooling or average
pooling). In contrast to supervised deep neural networks, the un-
supervised pre-training approaches like restricted Boltzmann machine
(RBM) [6], deep belief nets (DBN) [7] and autoencoder [8] provide the
possibility of stepwise adaptation. The latent representation of high
dimensional input data can be layer-wise reduced and fine-tuned at
2
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
each layer for specific tasks.
In this paper, we introduce and evaluate a pipeline based on neural
networks to support the automatic CHOP encoding. More specifically,
the goal is to determine the best model of neural networks for the CHOP
encoding and in this way, assigning a correct CHOP code to a query
text. We are assessing different models based on neural networks with
respect to their performance (relevance determination in precision,
recall and F1). Moreover, we assess the impact of a semantic enrich-
ment of the query using a semantic knowledge on the classification
performance.
Based on aforementioned challenges and objectives, we will address
the following research questions:
• Which type of neural networks is better suited for the considered
matching task in terms of efficiency: CNN or autoencoder?
• Given the positive influence of semantic enrichment for the sake of
balancing between query and category text, which method can be
applied to deal with the sparseness of the data set caused by a large
vocabulary space of enrichment?
• Is autoencoder a suitable method for the knowledge integration?
Does semantic enrichment of the feature set by concepts of a se-
mantic network impact the classification accuracy?
• Is layer-wise pre-training of autoencoder suited for representation
for the task of semantic matching?
1.2. Paper organization
In Section 2, we summarize the related work. Moreover, we also
introduce the characteristics of our task in contrast to the related work.
In Section 3, we present our data material that is used for training and
evaluation. Section 4 includes the formal definition of the CHOP query
matching problem in Section 4.1 and the methods we proposed for the
CHOP query matching problem in Section 4.3. Besides, a working ex-
ample for input normalization and enrichment is described in Section
4.2. The CNN based method and autoencoder based methods are de-
scribed in Sections 4.3.1 and 4.3.2, respectively. Section 4.4 and 4.5
describe the implementation and experimental settings of our methods.
The results and efficiency for relevance determination are presented in
Section 5. After summarizing our principal findings and pointing to
Y. Deng et al.
limitations in Section 6, respectively, we refer to directions of future
work in Section 6.3.
2. Related work
In this section, we will firstly introduce a fraction of relevant
methods for medical document tagging and classification based on
neural networks. As next, the specific neural networks for representa-
tion learning and knowledge integration will be presented in detail.
Finally, the insights provided by the previous work and the difference of
our approaches in contrast to the existing methods will be highlighted.
2.1. Neural networks, medical document classification and tagging
Neural networks have been employed for a variety of tasks in the
medical domain. Miotto et al. [9] provided a comprehensive summary
for applying deep learning technologies in the medical domain.
Through a review of 32 recent research papers regarding four domains
of clinical applications (clinical imaging, electronic health records,
genomics and mobile health), the related deep models such as CNN,
recurrent neural networks (RNNs), restricted Boltzmann machine
(RBM) and autoencoder (AE) have been discussed. The potentials and
opportunities of deep models relating to our encoding task are the
feature enrichment, the incorporation of expert knowledge and the
interpretability of the model. The consideration of these aspects could
potentially improve the model performance and also increase the ac-
ceptance of deep learning methods in medical use cases.
Similar as the CHOP classification in the sense of medical procedure
indexing, the medical subject headings thesaurus (MeSH) is a hier-
archical medical thesaurus for indexing biomedical literature. It con-
tains 16 top categories and 27,455 main headings.2 The assignment of
MeSH tags to free text based on neutral networks depends on a suitable
feature representation and a algorithm for the relevance determination.
Peng et al. [10] proposed DeepMeSH, an approach for MeSH tag-
ging based on unsupervised deep representation learning. The docu-
ment to vector (d2v) [11] and tf-idf feature embeddings combined with
MeSHLabeler [12] achieved the best score on the task of large scale
semantic indexing in the 2017 BioASQ3 challenge [13] (task 5a). Fur-
ther, with an F1-score of 0.6323, the DeepMeSH method yielded an
improvement of 12% and 2% respectively, in comparison to the two
baseline index algorithms: Medical text indexer (MTI) with an F1-
Measure of 0.5637 and MeSHLabeler with an F1-Measure of 0.6218.
Du et al. [14] provided another possibility of feature representation
for MeSH indexing beside d2v. They employed the bidirectional re-
current neural network (BRNN) and an auxiliary regression mechanism
to conduct the primary multi-label classification. The composed serial
structure between terms can therefore be extracted. The algorithm has
outperformed the state-of-art baseline Medical Text Indexers (MTI) in
F1-score with 0.6220 and reached a higher precision (0.77) than
DeepMeSH (0.70). One earlier attempt conducted by Rios and Kavuluru
[15] used CNNs to assign MeSH terms to biomedical articles. Paper
abstract of publications listed in Pubmed have been processed by
multiple-layer CNNs proposed by Kim [16]. The simple CNN structure
with single convolution and pooling layer achieved an absolute im-
provement of over 3% in macro F1-score on selected subsets of MeSH
terms compared to the baseline MTI method.
The aforementioned examples of MeSH tagging demonstrate the
strong potentials of the application of neural networks for medical
document classification (tagging) and show the importance of re-
presentation learning. One lesson learned from the method design is
that a comprehensive representation combining multiple levels of
salient features can largely enhance the performance of the follow-up
classification. In the follow section, we will take a further look at the
2 4
https://www.nlm.nih.gov/mesh, accessed on August 23, 2018.
3
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
deep representation learning based on unsupervised neural networks.
2.2. Unsupervised representation learning using neutral networks
In contrast to the training with annotated data by supervised
learning, the unsupervised deep learning method trains the model with
the input data itself (reconstruction of input). Based on the MNIST digit
recognition data set [17], Larochelle et al. [18] have evaluated the
greedy layer-wise pre-training of a deep network. The results confirmed
that the layer-wise pre-training can largely increase the effectiveness of
the network initialization and enable a rational starting point for the
classification. It has also proved that the training can achieve a rela-
tively large benefit by the layer-wise pre-training, in particular, when
the amount of training data is small. In the clinical domain, Miotto et al.
[19] have used a three-layer stacked denoising autoencoder (SDA) to
learn a deep patient representation based on electronic health records.
The representation was then employed to predict disease risk using
random forest as classifiers. The evaluation was performed on 76,214
patients comprising 78 diseases from diverse clinical domains within a
time window up to one year. On the task of disease prediction, the SDA
based methods have significantly outperformed other dimension re-
duction methods such as principal component analysis (PCA), k-mean
on the task of disease prediction.
In our task, a suitable representation learning method is essential for
balancing the difference between query and CHOP category text.
Beyond that, the representation learning should also be able to facilitate
the knowledge integration from external medical knowledge base.
2.3. Word embedding and autoencoders for knowledge integration
Another important trend of representation learning is the integra-
tion of external knowledge through unsupervised learning. In the gen-
eral domain, word embedding with semantic enrichment has been
evaluated by Yu and Dredze [20] and Celikyilmaz et al. [21]. The
knowledge from terminologies like Wordnet3 or PPDB4 is exploited in
these approaches. The type of relations and weights of the relations
between concepts have been combined with a word embedding model
(word2vec) linearly. The linearly combined models were trained on
different data sources: the word embedding was firstly trained on the
input corpus based on continuous bag of words (CBOW). Then, the
normalized weights from related words found in the knowledge re-
source were used to update the weight parameter of CBOW and opti-
mized according to the same lost function. Yu and Dredze's method
achieved a 19% improvement in mean reciprocal rank (MRR) on the
task of finding semantically related words using the embeddings, while
Celikyilmaz et al. have achieved around 2% of improvement in F1
measure on the task of semantic tagging within one movie dataset [22].
Faruqui et al. [23] have further improved the knowledge integration
through a post-processing step by conducting belief propagation on a
graph obtained from lexicon-derived relational information. These
methods outperformed the prior approaches developed by Yu and
Dredze [20] with 5% (sentiment analysis [24]) to 20% (Synonym Se-
lection (TOEFL) [25]) improvement in accuracy.
Yu et al. [26] used semantic hashing based on autoencoder to
support short text understanding and retrieval. More specifically, the
short text representations were enriched with concepts and their co-
occurring concepts from Probase [27]. After the enrichment, the
stacked autoencoder was applied to reduce the dimension of the re-
presentation. The obtained hashing representation of short text has
yielded to an improvement of 20% to 30% on the task of news retrieval.
On the task of classifying Wikipedia sentences, the proposed method
has obtained an improvement of 10% (in comparison with other types
3
https://wordnet.princeton.edu, accessed on September 9, 2017.
http://www.cis.upenn.edu/∼ccb/ppdb/, accessed on September 9, 2017.
Y. Deng et al.
of enrichment) to 18% in accuracy (in contrast to non-enrichment).
In general, the deep neural networks have approximated a latent
representation with a suitable dimension gradually through non-linear
transformation. The obtained deep representation was proved to be
useful on aforementioned classification tasks.
In the biomedical domain, a large number of ontologies and ter-
minologies (e.g. SNOMED CT,5 UMLS6 Radlex,7 OntoMed8) provide
valuable knowledge on clinical concepts and their relations. Based on
the findings from the general domain as presented before, we hy-
pothesize that integrating such knowledge bases into the representation
of texts will introduce in-depth rich semantics to the representation and
will lead to a quality improvement for the classification tasks. However,
to our best knowledge, the knowledge enrichment for the purpose of
medical encoding with neural networks has not been well studied. In
particular, the enrichment of German medical text for the task medical
encoding is still an open question. Hence, the potentials of word em-
bedding and autoencoder for the knowledge integration will be in-
vestigated within our experiments.
2.4. Our task in contrast to related work
Compared with the document tagging with MeSH headings, a spe-
cific challenge in our work is that the structure of the query and the
CHOP category description differ substantially from texts and queries
considered in related work. One encoding query or one CHOP category
description normally contains only one or a few biomedical concepts.
Complex sentence structures can neither be found in the query nor in
the category text. Instead, a nominal style is used. However, we have to
consider a high-dimensional vocabulary space. In contrast to public
accessible medical knowledge bases in English and other open knowl-
edge bases in the general domain, German medical knowledge bases are
either unavailable (there is no free accessible translation of SNOMED
CT available in German) or they are managed by industrial companies
and optimized for special commercial purposes. The access and appli-
cation of those knowledge bases may therefore be limited by the in-
terface in terms of concept enrichment, i.e., the search of similar con-
cepts can only be conducted based on provided function for the
similarity comparison. Hence, a suitable method for representation
learning and enrichment should be designed in consideration of avail-
able interfaces provided by selected external knowledge bases.
To deal with these challenges, we will exploit several types of
methods to enable a knowledge-based semantic matching in the CHOP
encoding. In view of the aforementioned state-of-the-art analysis
(neural networks for medical text tagging, unsupervised representation
learning, knowledge integration), we will implement and evaluate a
baseline method using CNN due to its known performance and effi-
ciency on the task of text classification and tagging. Secondly, a suitable
representation will be determined for query and category text in our
specific task. The possibility of knowledge integration in the neural
network will be investigated with a particular focus on the knowledge
enriched matching based on unsupervised representation learning [8],
Besides, the semantic matching with conventional machine learning
methods (SVM, logistic regression) will also be tested as the baseline of
the neural network based methods.
We hypothesize that the application of neural networks is a suitable
technique to complement the conventional method and allows to ef-
fectively use the available features of the data set.
5
https://www.snomed.org/snomed-ct/what-is-snomed-ct, accessed on
September 9, 2017.
6
https://www.nlm.nih.gov/research/umls/, accessed on September 9, 2017.
7 10
https://www.rsna.org/RadLex.aspx, accessed on September 9, 2017.
8
http://www.onto-med.de/ontologies/, accessed on September 9, 2017.
4
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
3. Data set
The queries and associated CHOP codes have been collected from
logs of a bug-tracker of the current rule-based system ID DIACOS®,9
including both, open (unresolved) and resolved search pairs. The search
pairs are grouped into search issues. One search issue stands for the
encoding of a series of similar cases (see example in Fig. 1). Currently,
about 90% of the query and category description pairs can be processed
with the knowledge base ID MACS®.10 About 85% of the matching pairs
are completely solved in terms of accuracy. The relevance of the CHOP
category description and the user query has been annotated manually
after the data gathering. Annotations were conducted by customers and
internal testers and were quality-assured by encoding experts, taking
into account the official coding guidelines for CHOP. In order to eval-
uate the annotation quality, a random sampling within a subset of the
annotated pairs including 15% all available unique codes has been
conducted. When a query was labeled as relevant with the correct ca-
tegory description, the data pair was considered as correctly annotated
pair. Based on this criterion, the annotated data set has reached a match
rate over 98.6%.
In summary, the data set underlying this work comprises 24,092
query-category training pairs, consisting of relevant (7,522) and irre-
levant (16,570) pairs. There are 856 unique queries and 14,898 unique
categories in the training pairs. The query and CHOP category have a
many-to-many relationship. This many-to-many relationship and the
similarity between category text under one category issue may lead to
difficulties in the model training. However, we will implement incre-
mental matching rules: The classifiers will be indexed according to the
CHOP structure, updated and redeployed in case of new log data or
category changes.
4. Methodology
In this section, the definition of data representation on the basis of
data pre-processing will be introduced in Section 4.1. For illustration
purposes, a working example of all representation types is presented in
Section 4.2. In Section 4.3, details are provided on our methods based
on CNN and autoencoder for the task of relevance determination. Fi-
nally, the implementation and settings of the experiments are specified.
4.1. Data representation
In order to process pairs of query and category text with neural
network based methods, a suitable pre-processing approach should be
employed to transform the raw data pairs into the vector representa-
tions. In the following two subsections, we will firstly introduce the pre-
processing method (normalization and enrichment) for query and ca-
tegory pairs. Subsequently, the vector representation will be proposed
considering the output of pre-processing and the requirements of
follow-up networks.
4.1.1. Data pre-processing
In this work, we consider the problem of assigning one or more
CHOP codes to a free-textual query. Within the CHOP classification
system, to each CHOP code a category description, i.e. a short free-
textual category description is given. For normalization and re-
presentation purposes, these category descriptions and queries have
been mapped to concepts of a medical terminology through the ter-
minology server ID MACS®, a software provided by the German com-
pany ID Information und Dokumentation im Gesundheitswesen. ID
9
https://www.id-berlin.de/en/products/1-coding/1-id-diacos/, accessed on
September 9, 2017.
https://www.id-berlin.de/en/products/8-terminology/15-id-macs/, ac-
cessed on September 9, 2017.
Y. Deng et al.
MACS® contains a manually curated medical ontology, which is a
German derivative of an early version of SNOMED called Wingert
Nomenclature [28]. ID MACS®11 has been in use in more than 1,000
hospitals in Germany, Switzerland and other countries in the past two
decades. Since then the company has developed and extended the ID
MACS® continuously. Its quality is assured by various manual and au-
tomated processes. It has been found useful in applications such as
semantic search and drug safety.
One challenge of our data set is that queries differ from CHOP ca-
tegory descriptions in several aspects, which makes it difficult to match
queries with descriptions: queries are short (often single word), am-
biguous (containing abbreviations or typos), use broader or narrower
terminology than the category description texts, may contain irrelevant
information. For addressing this issue, we generate a normalized and
enriched feature vector by including related concepts of a term from the
ID MACS® semantic network. The enrichment aims at bridging the gap
between queries and categories for the matching task. For instance, in
the case of a query containing broader concepts than the category de-
scription, the enriched query vector would contain also the narrower
concepts occurring in the category description. Our intuition to perform
this pre-processing approach is confirmed by Koopman et al. [29]. They
concluded from a user study with pre-defined queries and topics that
the human process of novel keyword choosing can largely facilitate the
retrieval of target biomedical documents. Our approach based on se-
mantic query enrichment and relation prediction through a deep neural
network may emulate this human keyword selection process. Through
our proposed method of query expansion, the discrepancy between
short and ad hoc query and well-formulated category text will be re-
duced. A more distinguishable representation can be generated for re-
levance prediction. The concept vector for query and category text has
therefore two variants: the first concept space only contains concepts
recognized in a query or category text, which sum up to 5,919 concepts.
The second dictionary contains the enriched vocabulary with 20,193
concepts, which includes the parent nodes and ancestor nodes of each
concept found in a text as retrieved from the medical semantic network
ID MACS®. The concept vector can be looked up in the case specific
vector dictionary through query id and category id.
4.1.2. Representation definition
After the data pre-processing, the CHOP classification problem can
be considered as a matching task between the concept vectors of a
query and a category text description. In our task, two types of vector
representations are defined for query and category text: (1) an em-
bedding representation learned from concept sequences of query cate-
gory pairs exploited by the CNN based approaches ((A) in Fig. 2), and
(2) a frequency-based vector representation for autoencoder-based
methods ((B) and (C) in Fig. 2), while the frequency based vector can be
derived from either non-enriched concept sequences or enriched con-
cept sets.
For representation of query and category pairs based on concept
embedding, a task-specific concept embedding is trained based on a
normalized concept sequence from query and category pairs. For this
purpose, the word embedding framework word2vec [3] is applied. One
query and category pair contains a sequence of normalized concepts,
while the word2vec algorithm generates the embedding representation
referring to the co-occurrence of the concepts within the scope of one
query and category pair. Since no enrichment has been applied to the
concepts, the original sequence of these concepts is kept. Based on this
setting of embedding, let ∥ be the concatenation operation and x i k
be the k dimensional concept vector referring to the ith concept in one
query with nq concepts. Similarly, let x j k be the k dimensional
concept vector referring to the jth concept in one category text with nc
11
https://www.id-berlin.de/static/files/products/id_logik.pdf, accessed on
August 19, 2018.
5
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
concepts.
k
Q1: nq = x1 ||x2 ||…||x nq , x i (1)
k
C1:nc = x1 ||x 2 ||…||x nc , x j (2)
The query and category text are padded to n with zero if necessary,
where the n corresponds to the size of the non-enriched vocabulary
space for both, query and category text. In this way, we get the con-
catenated query and category pair:
P1:2 * n = Qn ||Cn (3)
This processing step is required since a padded input with same length
facilitates the batching and the convolution and pooling process in
CNN.
For the representation of the frequency-based input for autoencoder
(B) and (C), the normalized query and category text share the same
vocabulary space. Hence, the vector of query and category description
is represented with the same dimension. Let x be the frequency of one
concept l appearing in the query of category text, while l is the index of
this concept in the concept dictionary. The dimension of one vector
represents the full size of the vocabulary space of one corresponding
concept dictionary. The value of the frequency of the corresponding
index l is obtained by computing the frequency divided by the count of
concepts in the vector of a query or category description. The concepts,
which have not appeared in the vector, are set to zero. A query with m
dimensions is therefore denoted as
Q1: m = (x1, x2, …, xl, …, xm ) (4)
where xm indicates the frequency of the term at the corresponding index
m. Similarly, the value of x′ in C represents the frequency of a concept x′
in one category text description. A category textual description is
therefore denoted as
C1:m = (x1 , x 2 , …, xl , …, x m) (5)
The enriched vector representation contains more concepts and also
higher dimensions than the non-enriched vector. The non-enriched
vector contains 1 or 0 to indicate the occurrence of one concept, while
the enriched vector contains additionally the normalized percentage
value received from a similarity function to show the frequency of
concept appearance. The frequency-based input vector has eliminated
the original sequence of terms. The sequence of the dimension corre-
sponds to the concept sequence in the vocabulary dictionary. Based on
the representation, the objective of our model is to learn the relevance
(relevant or irrelevant) of C for category encoding Q.
4.2. Example of vector representation, normalization and enrichment
To illustrate the details of the data representation, normalization
and enrichment we consider a concrete example of the single term
query “liquorfistel”. One CHOP category text for the corresponding
code 03.59.12 is the category description “Liquorfistelverschluss am
Rückenmark” (“closure of a spinal liquor fistula”) (see Fig. 2). The
query is mapped by ID MACS® to two concepts associated with the
terms “Fistulisation” and “Liquor” (internal concept ids are 1226 and
3389) while the CHOP category text is represented by the concepts
“closure of a fistula” (id: 1493) and “Liquor spinals” (id: 3410). The
vector representation of this text is a sparse vector with two concept
entries corresponding to the dimensions of the related concept dic-
tionary space from ID MACS®.
As it has already been explained in Section 4.1.2, the query vector is
represented by (1226, 3389). Meanwhile, the vector with frequencies is
represented by a vector of the form (0 … 0, 1, 0 … 0, 1, 0 … 0) having
two non-zero entries at the positions of the two concepts to which the
query has been mapped to. All texts of the data set with positive and
negative (query/category) pairs are represented in this way. In the
enriched vector representation, additional concept entries are added to
Y. Deng et al.
Fig. 2. Example for input normalization and enrichment. (A) shows one example of the non-enriched concept vector and the corresponding embedding of query and
category text. (B) indicates the non-enriched concept vector with frequency. (C) represents the enriched concept vector with frequency.
each vector, namely those concepts that have been retrieved as ances-
tors of the originally mapped concepts. This is realized by a similarity
function. The similarity function employs the class hierarchy and re-
lations of the ID MACS® semantic network to determine the relatedness
of two concepts. The value generated by the similarity function re-
presents the frequency of one concept appearing in the representation
of one query or category text. For instance, an entry for spinal column
with value 0.86 is added to the sparse vector representation of the
target text, since spinal column is related to spinal liquor in our se-
mantic network. Similarly, a concept entry for “fistula extirpation” with
value 0.89 is added because this concept describes another fistula-re-
lated operation which is connected in our semantic network to the
concept “closure of a fistula”. Hence, the enriched frequency vector has
a form of (0 … 0, 1, 0.86, 0 … 0, 0.89, 1, 0 … 0, 0 … 0). The non-en-
riched concept vector is the input for the embedding layer of CNN
(labeled with (A)). The two frequency-based vector representations
((B), (C)) are the input for autoencoder, SVM or logistic regression.
4.3. Matching methods with CNN and autoencoder
In our pipeline and experiments, we exploit mainly two types of
models based on neural networks (see Fig. 3).
The first approach is based on CNN whereas the second type of
network is a classification model connected to a representation layer
trained with autoencoders. The CNN are implemented as baseline to
compare with the autoencoder based method. Since the original se-
quence of text has been eliminated after the semantic enrichment and
the sequence between concepts is a crucial prerequisite for the em-
bedding and convolution in CNN, we will only apply the holistic
knowledge integration for the autoencoder method. Additionally, the
linear-kernel SVM and logistic regression with L2 regularization are
applied on both, enriched and non-enriched input, as generic baselines
to compare with the neural network-based approaches.
6
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
4.3.1. Matching with CNN
A CNN is a type of feed-forward artificial neural networks, which
has been successfully applied to both, image processing [30] and nat-
ural language processing [31]. Our architecture exploits a CNN with
one layer of convolution and one layer of max pooling. In the con-
volutional layer, different filter sizes can be defined to cover the po-
tential semantic scope in the query and category description. The ar-
chitecture is therefore similar to the CNN employed by Kim and Shen
[16,32]. As input for the training, the vectors of the query and category
text from the training set are concatenated as one vector. The output of
the network is the relevance of a category text with respect to a query.
The features are obtained from multiple filters (3, 4, 5); the convolution
is selected through max-pooling and fully connected into one feature
vector. The filter size 3, 4, 5 have been determined based on the length
statistics of our training corpus as most of the concept vectors for query
and category description have been mapped to 3–5 concepts.
Based on the formulas (1)–(3) in Section 4.1.2, a pair of padded
query and CHOP category description is modeled as
P = Qn ||Cn = {x1, x2 , …, x i , …, x n , x1 , …, x j , …, xn} (6)
Since query and category text share the same vocabulary after the
normalization and also the same index in the concept dictionary, the
query and category pair can be transformed into: P = {x1, …, x2n}. The
convolution process is conducted through filters with predefined size,
which slides through the concatenated concept vector P. As it has al-
ready been mentioned in Section 4.1.2 shortly after formula (3), the
dimension n represents the padded length of a query or a category
vector. In this network, a filter of size h generates a windows of xi:i+h.
The corresponding features are obtained through an activation function
f with a bias term b ∈ in form of
Convolutional_features = f (w × x i : i + h 1 + b) (7)
For each dimension of concept embedding (size k in presentation
Y. Deng et al. Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx

Fig. 3. Data flow for the training: the bug-tracker logs (query-category text pairs) are processed with the concept dictionary mapper. A non-enriched concept vector
dictionary is generated for concept embedding, while enriched vectors are prepared for autoencoder based representation learning. The representation is trained and
cross validated for relevance determination. As is illustrated in Fig. 2, (A) represents the non-enriched concept vector for concept embedding. (B) indicates the non-
enriched concept vector with frequency. (C) represents the enriched concept vector with frequency.

definition: 128) we have assigned one filter to slide through all the
possible windows in the concept vector. As it can be seen in Fig. 4, a
feature vector with length of n − h + 1 is generated at each dimension
of the embedding. Next, the maximum value in the feature vector is
pooled out through the max pooling process. Next, the maximum value
in the feature vector is pooled out through the max pooling process so
that a fully connected layer can be created. At last, the softmax layer is
attached to conduct the classification.
In order to achieve an optimized network configuration for the CNN
experiment, we have configured and evaluated the outcome of the
embedding based on annotated relevance metrics [33]. The rectified
linear unit (RELU) has been applied as activation function; filter win-
dows of 3, 4, 5 with 128 dimensional feature maps are applied. We have
chosen the size of 128 to ensure a fast calculation of similarity metrics.
The dropout rate has been defined as 0.5 whereas an L2 constraint (s) of
0.01 has been chosen to avoid over-fitting. The batch size – chosen

Fig. 4. CNN for relevance determination of CHOP query and category text matching, filter sizes with 3, 4, 5 are selected.

7
Y. Deng et al.
Fig. 5. The basic autoencoder with three layers. The network tries to reproduce
its input at the output layer.
through incremental experiments using the evaluation set – is 64. These
settings were selected considering the performance and computing ef-
ficiency.
4.3.2. Holistic matching based on an enriched representation
An autoencoder is an unsupervised learning algorithm that auto-
matically learns features from unlabeled data [8]. It is actually a three-
layer neural network and the learning process consists of two main
stages, namely the encoder and the decoder. It can be seen as a feed-
forward neural network trained to reproduce its input at the output
layer. The encoder takes a vector x d
as input and encodes it to
hidden feature h d
within hidden layer through a deterministic
mapping, that is h = f(Wx + b), where f is the activation function, W is
a d × d′ dimensional weight matrix. The decoder needs to reconstruct
the input: the code h is further decoded to y, which is seen as a pre-
diction of x that is y = g(W′h + b′), where g is a similar activation
function, W′ is the transposed tied weight with W, b′ is a d dimensional
bias vector illustrated in Fig. 5.
The semantic matching based on a single layer convolution and
pooling network has only represented one trend of textual semantic
matching [5,34,26]. However, for applications in the medical domain,
knowledge bases and existing terminologies are a valuable resource to
enable the clinical interoperability and logical reasoning. These re-
sources are expected to balance the input and to facilitate relevance
determination by bypassing the unnecessary connection in the value
propagation chain. However, the knowledge is not always easy to be
integrated. For the mechanism based on neural networks, especially,
after the sparse coding and semantic enrichment, we normally obtain
vectors indicating higher-level features and additional knowledge to-
gether with extended dimensions [35]. With the aim of the knowledge
utilization based on the local proximity and probability from the raw
input data, a suitable mechanism to integrate the knowledge in the
representation layer is required. In this work, we propose a mechanism
based on autoencoder to enable biomedical knowledge injection at the
phase of representation learning. In comparison with the stacked au-
toencoder applied on handwritten digit data set [36,37], our method
has been adapted to process short text. Instead of learning the re-
presentation from training corpus only, we have employed a German
biomedical terminology ID MACS® as prior knowledge to train stacked
autoencoder and to evaluate the influence of external knowledge on
representation learning. Derived from the same aforementioned re-
presentation formulas (4) and (5) in Section 4.1.2, the holistic matching
process can be described with the following formulas:
Mvector = Qm ||Cm
where Q and C represent the query and category text whereas Mvector
stands for the concatenated vector, m is the shared dimension of vo-
cabulary in both query and category text,
Vrich = fMacs (Mvector )
where Vrich stands for enriched vector. The function fMacs represents the
enrichment approach provided by ID MACS® semantic network and
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
concept mapper, more specifically, it is based on the aforementioned
similarity function s(c, c′) (see Section 4.2), which assigns a pair of
medical concepts a similarity value in the range of [0, 1], in particular, s
(c, c′) = 1,
h i = f (x i ) (10)
is the encoder function and
ri = g (hi ) (11)
is the decoder function. Let Vrich = g(f(Vrich)) be the optimization
target. We will use the same concept mapping process and similarity
function to generate the enriched input vector. The autoencoder will
learn to reduce dimensions of the enriched vector through the encoder
and decoder process and to minimize the loss function L:
L (Vrich, g (f (Vrich ))) (12)
where the hi vector is the latent representation used in the relevance
classification. As a consecutive layer after the dimensional reduction
layer, another layer based on denoising autoencoder is added to retrain
the output from the previous layer. The difference to the previous layer
is that the noise derived from noise distribution D, such as Gaussian,
Masking noise, salt-and-pepper, etc., [38] is used to corrupt the original
input and the autoencoder can then try to differentiate the target pairs
within the noise environment, so that a more generalized representa-
tion can be learned. We define the data corruption function as qc. In
contrast to the original data Vrich, the data corruption process
q ( Vrich|Vrich ) has been pushed away from the original data distribution.
So the loss function can be defined as:
L (Vrich, g (f ( Vrich ))) (13)
where Vrich represents the enriched vector mixed with predefined
noise.
In the perspective of manifold learning, the learning process by the
denoising autoencoder is aimed at projecting the corrupted examples
back to the low dimensional manifold through loss function mini-
mization, so that the obtained representation can yield a clearly higher
generalization level than autoencoder based on a normal data set [37].
This assertion matches the attributes of our training data perfectly due
to the high similarity between the training set within one CHOP cate-
gory (search issue). The performance of the denoising layer in our case
will be evaluated. As far as the vector size at each layer is concerned, we
employed the empirical definition for the vector size: each layer re-
duces the vector size approximately to half of the vector size of its
previous layer. For the concept vector without enrichment, a layer di-
mension with vector sizes: (11838)-5919-2969-1479-500 is applied by
the four autoencoders’ layers (see Fig. 7). For the concept vector with
enrichment, a layer size configuration with vector sizes: (40278)-
20000-8000-4000-1500-500 has been employed. The current definition
enables an efficient experiment with a comparable outcome. A fine
tuning of the vector size and decreasing interval is out of the scope of
this paper.
The deep autoencoder extends the single two layer structure into
multiple layers of symmetrical neural nets to increase the approxima-
tion capacity. As a comparable benchmark to the stacked denosing
autoencoder, we will also evaluate the non-stop deep autoencoder. The
training of representation will be fully supervised and the layer attri-
butes will not be initialized in a layer-wise manner. The sigmoid
function will be applied at each layer as activation function and the
(8)
cross entropy loss is computed as objective of minimization. The con-
crete steps of proposed approaches will be introduced in the following
Section 4.3.2.1.
4.3.2.1. Pre-training of frequency based vectors. The pre-training is an
(9)
important technique to let each layer of a neural network be initialized
correctly. At this phase, we employed the unsupervised method based
on autoencoder to obtain a compact high level representation of the
8
Y. Deng et al.
Fig. 6. Relevance classification based on stacked autoencoder. The details of the pre-training module are illustrated in Fig. 7.
Fig. 7. Pre-training architecture implementing 4 stacked denoising autoencoders.
input data, meanwhile, the input information is attained through these
pre-training steps as much as possible. Based on the pre-training
network illustrated in Fig. 6, the vector enriched by ID MACS® is
concatenated into one vector in form of Query ∥ Category. The vectors
are multiple-hot coded with concepts. The networks are initialized layer
by layer with denoising mechanism:
(1) Stepwise initialization: In Fig. 6, the main architecture of our
mapping process is presented. With the enriched query and category
text concatenation, the four nested autoencoders are initialized. We
followed the principle of a greedy layer-wise initialization [38]. The
greedy algorithm optimizes each piece of the solution independently,
one layer at a time, rather than jointly optimizing all layers. Specifi-
cally, greedy layer-wise pre-training proceeds one layer at a time,
training the kth layer while keeping the previous layers. The lower
layers (which are trained first) are not adapted after the upper layers
have been introduced. In our pre-training model illustrated in Fig. 7, we
9
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
also follow this two phase protocol. The pre-training can then be un-
derstood as the first part of an entire two phase protocol that combines
the pre-training phase and a supervised learning phase. The supervised
learning phase involves training a classifier on top of the features
learned in the pre-training phase. At the same time, the supervised
classification can fine-tune the entire network learned in the layer-wise
pre-training phase.
As a comparable experiment, we will also evaluate the non-stop
deep autoencoder, which conducts the latent learning without inter-
ruption. The F1 measure of the classification and the efficiency of the
training will be compared.
(2) Enabled with denoising mechanism: A denoising autoencoder
minimizes
L (x , g (f (x ))) (14)
Y. Deng et al.
Fig. 8. Using autoencoder to learn latent representation directly on an enriched single query and category concept dictionary.
where x̃ is a copy of x that has been corrupted by some form of noise.
The denoising autoencoders must therefore undo this corruption rather
than simply copying their input. The adding of noise has firstly pushed
the original data distribution away from the target low dimension
manifold, while the learning process has to project the noised input
data back onto the “manifold” [36]. With the input from the last hidden
layer hl:
qD = g (hl + N )
where N is the Gaussian noise and g is the activation function, the de-
noising process is expected to achieve a higher generalization than
ordinary autoencoders.
4.3.2.2. Alternative representation learning: applying autoencoder only on
query and category dictionary. The aforementioned approaches start all
with a concatenated vector of query and category text. There is another
way of training which can largely reduce the training time by
representation learning only on the CHOP concept dictionary
(representation learning before query/category concatenation) (see
Fig. 3). The concept dictionary contains the single vector
representation for query and category. The corresponding vector
representation can be obtained through query/category id as
dictionary key. The query and category text can then be assigned
with a short representation separately before the concatenation of the
query and the category text see Fig. 8. This alternative approach learns
the representation locally based on the vocabulary vector
representation without supervised tuning. In comparison with the
autoencoder with fully connected query and category text vector,
illustrated in Fig. 6, the alternative method has concentrated on the
learning of intrinsic relations within a single query or a single category
text. It is expected that the relatively low dimension of the
representation leads to a short training time and a fast converging.
For the experiment with autoencoder with supervised classification,
the configuration is realized as follows: we employed four autoencoders
consecutively to reduce the dimension directly after the enriched input.
The noise is added to the encoding step of each of the four layers, while
a denoising mechanism has been employed at each denoising layer. The
sigmoid activation function and cross entropy loss are calculated at
each layer. The stochastic gradient decent (SGD) is used as optimization
method. As is already mentioned in Section 4.3.2, through each
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
autoencoder, the vector size will be reduced to the half of the previous
layer. In the last step, the supervised classifier has received a 500 di-
mensional vector to conduct the relevance classification. We have de-
fined a batch size of 128 to do the training. This configuration is de-
signed with the consideration of both performance expectation and
training cost. Since we conduct experiments with limited hardware
resources (single GPU with 12G memory), an acceptable training time
and cost is essential for a comparison based on different approaches.
(15)
4.4. Implementation
We implemented the proposed architecture with the Tensorflow
framework.12 Tensorflow was chosen, since it provides comprehensive
toolkits for the construction of embedding and neural networks [39].
The computational network works as a road map of the data processing
workflow, whereas the real data is not loaded into the computing
graph. The input is only defined with sizes and attributes within pla-
ceholders. The real value input will be triggered only after the session
has been initialized through queue feeding or dictionary feeding, while
queue is defined for asynchronous tensor input and dictionary is used to
input the data with a small amount statically. This deferred form of
graph computing can also optimize the resource utilization and scal-
ability during the processing. For the creation of a CNN, Tensorflow has
provided interfaces like tf.nn.embedding_lookup, tf.nn.conv2d
and tf.nn.max_pool. The convolution and pooling process can be
created and configured intuitively to constitute our proposed CNN ar-
chitecture. In this implementation, we have also used the Python APIs
from the package scikit-learn13 and numpy14 to implement the bench-
mark.
4.5. Experiments
Three groups of methods are tested: the conventional machine
learning methods (SVM, logistic regression), the convolutional pooling
based method and the autoencoder based relevance determination. The
12
https://www.tensorflow.org, accessed on September 12, 2017.
13
http://scikit-learn.org/stable/, accessed on September 9, 2017.
14
http://www.numpy.org, accessed on September 9, 2017.
10
Y. Deng et al. Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx

Table 1 Table 2
Baseline classification results according to precision, recall, F1, specificity (true Pre-trained autoencoders with classifier (logistic regression). If not mentioned,
negative rate = true negatives/(true negatives + false positives)) (all in micro the experiments are enabled with denoising layer. In the last experiment, the
average) for relevance determination based on different configurations. 3, 4, 5 denoising layer is disabled for comparison. Specificity refers to the true nega-
represent that three filters of size 3, 4, 5 have been applied, respectively. The tive rate, true negative rate = true negatives/(true negatives + false positives).
embedding was trained on cross-validation fold (4/5 of training data). SVM and
Configuration (all enriched) Precision Recall F1 Specificity
logistic regression are evaluated on both enriched and non-enriched input.
Configuration Precision Recall F1 Specificity Deep autoencoder (non-stop training 79.32% 54.76% 64.14% 93.6%
on dictionary)
CNN with single convolution and 78.32% 49.76% 60.86% 93.86% Layer-wise autoencoder on 85.21% 59.56% 70.29% 95.3%
pooling 3, 4, 5 Ada L2 0.01 concatenated input
SVM classifier linear kernel with non- 30.04% 25.52% 27.60% 72.9% Layer-wise alternative autoencoder: 84% 55.9% 68% 95.17%
enriched input representation learned on
Logistic regression L2 Liblinear with 23.24% 18.66% 20.70% 71.7% dictionary
non-enriched input Layer-wise alternative autoencoder 80.32% 52.46% 63.47% 94.14%
SVM classifier linear kernel with 28.34% 31.89% 30.01% 63.2% dictionary denoising disabled
enriched input
Logistic regression L2 Liblinear with 21.01% 19.31% 19.65% 66%
enriched input are better recognized than the relevant pairs in general. The SVM and
Deep autoencoder (non-stop training 73.12% 45.26% 55.26% 93.78%
logistic regression with enriched input have reached a higher recall
on dictionary) with non-enriched
input
(increasing with 1–6%) but a lower specificity percentage (decreasing
Layer-wise autoencoder on 74.41% 55.76% 63.75% 94% with 5–9%) than non-enriched input. However, even with the enriched
concatenated input with non- input, the outcomes of the SVM and logistic regression are still out-
enriched input performed by the CNN with concept embedding without depending on
external knowledge.
The enriched versions of logistic regression has achieved a higher
performance of the latter will be compared to the first two methods that
precision than its own recall whereas the SVM with enriched input
are considered as baseline. We chose different inputs and configurations
reaches a higher recall (31.89%) than precision. Generally, the en-
of these methods with the aim of optimizing the performance. As far as
richment setting (SVM, logistic regression) showed an improvement on
the input is concerned, the CNN method with single convolutional and
F1 measures ranging from 1% to 4% due to an increase in recall. In
pooling layer has been fed with concept embedding learned from the
order to give a reference for the autoencoder with enriched frequency
non-enriched concept vector. The SVM, logistic regression and auto-
vector, we have extended our baseline group with two autoencoder
encoder based methods have been tested on both, the non-enriched
configurations with non-enriched frequency vector. The deep auto-
concept vectors and the enriched frequency vectors. We tested the au-
encoder based on dictionary reached a micro F1 score of 55.26%, which
toencoder with denoising layer. Three different types of autoencoders
shows 5% of decrease in F1 score in contrast to CNN, while the layer-
(non-stop deep autoencoder, layer-wise autoencoder with merged
wise autoencoder based on concatenated input has shown a slightly
input, alternative autoencoder on dictionary) with logistic regression
better F1 measure (3%) than the CNN based on the non-enriched fre-
were tested in parallel. Precision, recall, F1 measure, specificity (all in
quency vector. The layer-wise autoencoder based on concatenated
Micro) have been compared. In the evaluation, all methods have gone
input has outperformed all the other baseline methods since it has
through a 5-fold validation. For the concept embedding, the embedding
achieved the highest score for recall (55.76%). Through the layer-wise
is conducted based on the cross-validation fold (4/5 of the entire data
pre-training with fine-tuning, this configuration have gained a 10%
corpus). The representations learned by autoencoder are derived from
improvement in recall in contrast to the non-stop deep autoencoder
enriched or non-enriched frequency dictionary, while the layer-wise
based on dictionary with the same non-enriched input.
autoencoder with concatenated input has also been trained on cross-
validation fold (second configuration (line) in Table 2).
5.2. Performance of enriched autoencoder-based method
5. Results
The performances of the enriched autoencoder-based method using
In this section, we will first describe the results of the proposed pre-training have been presented in Table 2. Different types of auto-
baseline methods. Second, the performance of the autoencoder-based encoders resulting from varying pre-processing steps are evaluated on
approaches will be presented. Finally, the efficiency aspects of training the enriched frequency vectors illustrated in (C) in Fig. 2. All models of
and deployment of the proposed methods are discussed. autoencoders were evaluated based on the same enriched input.
As illustrated in Table 2, the layer-wise autoencoder based on
5.1. Performance of baseline classification concatenated input has achieved the best F1 measure. The deep auto-
encoder trained on the enriched frequency dictionary has also reached
Table 1 shows the precision, recall and F1 measure of our baseline high precision (84%) and moderate recall of 55%. The enrichment led
model (all in micro average) on non-enriched and enriched input. The to an improvement of the F1 measure between 4% to 10% in compar-
CNN with single convolution and pooling layer has been trained on ison with the CNN method. The effectiveness of denoising layers has
embeddings obtained from the concept vector (A) demonstrated in been confirmed: the same configuration without denoising mechanism
Fig. 2. We can see that the CNN with the embedding significantly
outperforms the other two conventional machine learning methods Table 3
with an F1 measure of 60.86%. Based on a frequency-based concept Learning efficiency of the proposed methods based on neural networks.
vector with enrichment, the SVM classifier with linear kernel reached Configuration Efficiency of training (iterations)
an F1 measure of 30.01%, while the logistic regression with L2 reg-
ularization has yielded an F1 measure of 19.65%. With the same con- Deep autoencoder (non-stop training) 14,000
Layer-wise trained autoencoder Stopped by each training layer
figuration, SVM and logistic regression with the non-enriched concept
Autoencoder trained on dictionary 8,900
vector have achieved F1 measures of 27.6% and 20.7%, respectively. CNN 3, 4, 5 Ada L2 0.01 6,000
The specificity (true negative rate) indicates that the irrelevant pairs

11
Y. Deng et al.
showed a decrease in precision (4% less) and recall (3% less) in contrast
to the same configuration with denoising. In all configurations, the ir-
relevant pairs were more accurately predicted than the relevant pairs
according to the score of specificity and the classifier tends to over-
classify (overkill) the relevant as irrelevant.
5.3. Efficiency of proposed methods
Given the same input, the count of iterations on different model
configurations before the model convergence (changes _ o f Accuracy@
dev < 5%) can be used to indicate the speed of model converging. The
count can therefore be seen as an indicator of efficiency by training. As
it can be seen in Table 3, the CNN-based method with single convolu-
tion and pooling layer achieves the best efficiency due to its efficient
embedding mechanism and the pooling with short sliding windows (3,
4, 5). Since most of the time costs in autoencoders are consumed by the
representation learning phase, we only compare the iterations and stops
in the representation phase. Regarding the efficiency of deployment,
Tensorflow provides a comprehensive library for product ready model
serving. With the required resource stored as docker container, the
chosen model (ckpt file) is transformed into a protocol buffer (pb) file.
The frozen model can be deployed as classifier for relevance prediction.
The inference and reconfiguration with the model is highly efficient.
For the query normalization and enrichment, the ID MACS® provides
response time at millisecond level, while the representation generation
through frozen model using pb file can also expect a low response time.
6. Discussion and conclusion
The current results have shown a clear potential of different models
based on neural networks in clinical procedure classification. In con-
trast to previous research, we have conducted a model design and
evaluation in a relatively narrow application domain (clinical proce-
dure encoding). We achieved better performance using autoencoder-
based representation learning compared to the baseline methods im-
plemented with CNN and conventional machine learning methods. The
methods are evaluated with a subset of CHOP categories and 24,092
corresponding query-category pairs. With the insight yielded through
the experiment, more work will be done to exploit the full CHOP ca-
tegory with more training data.
6.1. Discussion of results
Within current settings of autoencoder, the non-stop pre-training is
better suited for processing a large balanced data set, while layer-wise
pre-training should be applied when the data set is relatively small and
unequally sampled. In our experiment, the layer-wise pre-training
clearly yielded improvement compared to the application of auto-
encoders without layer-wise initialization. We believe that the layer-
wise pre-training has brought the model into the rational subspace
which leads to better optimization effectiveness and prevents the model
from getting skewed by the imbalanced input. The enrichment has
shown a positive influence on the matching performance. In our ex-
periment, the enriched input processed by the layer-wise pre-trained
autoencoder and supervised classifier achieved the best outcomes on
the task relevance determination.
In comparison with the performance of baseline methods (SVM and
logistic regression) with enriched input presented in Table 1, the non-
linear transformation conducted by a layer-wise autoencoder has shown
a clearly better result. We believe that this performance augmentation
originates from the in-depth latent representation learned through
multi-layer autoencoder. In addition, the separate layers have also been
fine-tuned globally with the supervised classification and get adapted to
the task on the fly. These flexibilities made the autoencoder-based re-
presentation learning clearly more attractive than the static linear in-
tegration of a knowledge weight based on word embedding, since the
12
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
latter method can only be trained once based on a training set.
According to the results presented in Table 2, the local feature
within one vector of query or one category text can already yield a high
F1 measure while the inter-query-category text features have appar-
ently higher training costs but a slightly better performance on the test
set. Hence, the feature selection should be determined by balancing
training costs and performance requirements. As a potential extension,
we can make use of the query and category specific features, such as the
exclusive concept in category, as negation rules to complement the
current trained classifier. In the current training set, we have only
eliminated the exclusive terms. In the next step, we can transform the
vector pairs with exclusive terms into negative training examples.
6.2. Summary of principal findings
Through experiments, we compared different configurations of two
models based on neural networks (CNN and autoencoder-based classi-
fication) to solve the matching issues. The CNN-based methods, SVM
and logistic regression are used as baseline to compare with the auto-
encoder based methods. The principal findings are:
1. A suitable vector representation for a high dimensional sparse data
set has been identified for the clinical encoding task: concept em-
bedding is well suited for the input with original sequence while a
frequency vector is a good representation for semantic enriched
input.
2. The suitability of stepwise pre-training and non-stop representation
learning based on autoencoder have been assessed: for text re-
presentation learning on a small data amount, stepwise pre-training
can be necessary and useful for a better generalization.
3. The usefulness of the knowledge enrichment has been confirmed. It
has been shown that the enrichment process can facilitate the se-
mantic matching and increase the model generalization in the test
set. More specifically, the autoencoder with pre-training can facil-
itate the knowledge integration based on the enriched frequency
vector. The denoising layer can also make moderate contributions to
the performance in terms of model generalization.
4. CNN has been proven to be a suitable method for concept matching
in the biomedical context. However, the embedding process and the
sliding windows at convolutional layer rely on the co-occurrence
information within the input sequence. For those tasks with se-
quence-less input, CNNs should be avoided or applied with adap-
tation. In this work, for a non-enriched input with original sequence,
CNN can achieve a moderate prediction performance with a high
training efficiency.
5. The unsupervised pre-training based on autoencoder may slightly
decrease the converging speed but makes the model more generic. It
achieves a better result in the cross-validation, while the knowledge
can be integrated in the pre-training more smoothly.
6.3. Limitations and future work
One limitation is that the quality of layer-wise pre-training can only
be judged by the result of matching prediction. The status concerning
the weight and biases as well as the status of the model at each layer is
still unclear. An investigation method should be developed to present
the changes of models, so that the correlation between the hyper-
parameter and model status can be determined and the performance
can be fine-tuned.
Secondly, the external knowledge base has been used in a relatively
simple way, the enrichment was conducted based on hypernyms and
hyponyms in the hierarchical terminology. The graphical features and
the conditional probabilities between the concepts have not been
exploited. As a next step, the additional knowledge reasoning can be
conducted by Bayes’ rules to guide the matching process.
Last but not least, the current data set (24,092 pairs) is relatively
Y. Deng et al.
small for deep fine-tuning. A more comprehensive evaluation based on
more log data from the production server (over 1 million pairs) has
been planned.
During the training, we have not considered the exclusive concept
separately (certain concepts that must not appear in the category text
has been provided in each CHOP category) due to the capacity of the
representation vector. The possible improvements are either the de-
velopment of a separate classifier to generate an exclusive decision or
transforming vector pairs with negative concept into negative training
examples.
As a next step, we would like to employ a larger data set to perform
a more comprehensive layer-wise pre-training. The status of the model
at each layer will be investigated to make the training more inter-
pretable. Beyond, additional features from category texts such as ex-
clusive concept and the category hierarchy will be used to complement
the current classifiers, so that the training in deep networks can be
accelerated and the performance of the encoding can be improved. At
last, we will analyze the hardware requirement of the proposed neural
network based method. A hardware for deployment in the real pro-
duction will be selected.
Acknowledgement
This work is partially supported by ID Information und
Dokumentation im Gesundheitswesen GmbH & Co. KGaA, Berlin,
Germany. Our gratitude also goes to Marie-Anne Pinheiro for her va-
luable comments. We thank the anonymous reviewers for their careful
reading of our manuscript and their many insightful comments and
suggestions.
References
[1] Li S-T, Chen C-C, Huang F. Conceptual-driven classification for coding advise in
health insurance reimbursement. Artif Intell Med 2011;51(1):27–41. https://doi.
org/10.1016/j.artmed.2010.10.003.
[2] Faulstich L, Mueller F, Sander A. Automated clinical documentation from electronic
health records. Medizinische Informatik, Biometrie und Epidemiologie, GMDS
2010, 55. Jahrestagung der GMDS, Mannheim, September 2010 2010:547–50.
[3] Mikolov T, Sutskever I, Chen K, Corrado G, Dean J. Distributed representations of
words and phrases and their compositionality, CoRR abs/1310.4546. URL http://
arxiv.org/abs/1310.4546.
[4] Hu B, Lu Z, Li H, Chen Q. Convolutional neural network architectures for matching
natural language sentences. Advances in neural information processing systems 27:
Annual conference on neural information processing systems 2014, December 8–13,
2014, Montreal, Quebec, Canada 2014:2042–50. URL http://papers.nips.cc/paper/
5550-convolutional-neural-network-architectures-for-matching-natural-language-
sentences.pdf.
[5] Shen Y, He X, Gao J, Deng L, Mesnil G. A latent semantic model with convolutional-
pooling structure for information retrieval. Proceedings of the 23rd ACM interna-
tional conference on conference on information and knowledge management,
CIKM’14. New York, NY, USA: ACM; 2014. p. 101–10. https://doi.org/10.1145/
2661829.2661935.
[6] Salakhutdinov R, Mnih A, Hinton G. Restricted Boltzmann machines for colla-
borative filtering. Proceedings of the 24th international conference on machine
learning, ICML’07. New York, NY, USA: ACM; 2007. p. 791–8. https://doi.org/10.
1145/1273496.1273596.
[7] Hinton GE, Osindero S, Teh Y-W. A fast learning algorithm for deep belief nets.
Neural Comput 2006;18(7):1527–54. https://doi.org/10.1162/neco.2006.18.7.
1527.
[8] Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural
networks. Science 2006;313(5786):504–7. https://doi.org/10.1126/science.
1127647.
[9] Miotto R, Wang F, Wang S, Jiang X, Dudley JT. Deep learning for healthcare: re-
view, opportunities and challenges. Brief Bioinf 2017. https://doi.org/10.1093/
bib/bbx044.
[10] Peng S, You R, Wang H, Zhai C, Mamitsuka H, Zhu S. Deepmesh: deep semantic
representation for improving large-scale mesh indexing. Bioinformatics
2016;32(12):70–9. https://doi.org/10.1093/bioinformatics/btw294.
[11] Le Q, Mikolov T. Distributed representations of sentences and documents. In: Xing
EP, Jebara T, editors. Proceedings of the 31st international conference on machine
learning, Vol. 32 of proceedings of machine learning research. Beijing, China:
PMLR; 2014. p. 1188–96http://proceedings.mlr.press/v32/le14.html.
[12] Liu K, Peng S, Wu J, Zhai C, Mamitsuka H, Zhu S. Meshlabeler: improving the
accuracy of large-scale mesh indexing by integrating diverse evidence.
Bioinformatics 2015;31(12):i339–47. https://doi.org/10.1093/bioinformatics/
btv237.
13
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
[13] Nentidis A, Bougiatiotis K, Krithara A, Paliouras G, Kakadiaris IA. Results of the
fifth edition of the BioASQ challenge. BioNLP, 2017, Vancouver, Canada, August 4,
2017 2017:48–57. https://doi.org/10.18653/v1/W17-2306.
[14] Du Y, Pan Y, Ji J. A novel serial deep multi-task learning model for large scale
biomedical semantic indexing. 2017 IEEE international conference on bioinfor-
matics and biomedicine (BIBM) 2017:533–7. https://doi.org/10.1109/BIBM.2017.
8217704.
[15] Rios A, Kavuluru R. Convolutional neural networks for biomedical text classifica-
tion: application in indexing biomedical articles. Proceedings of the 6th ACM
conference on bioinformatics, computational biology and health informatics,
BCB’15. New York, NY, USA: ACM; 2015. p. 258–67. https://doi.org/10.1145/
2808719:2808746.
[16] Kim Y. Convolutional neural networks for sentence classification. Proceedings of
the 2014 conference on empirical methods in natural language processing, EMNLP
2014, October 25–29, 2014, Doha, Qatar. A meeting of SIGDAT, a special interest
group of the ACL 2014:1746–51. URL http://aclweb.org/anthology/D/D14/D14-
1181.pdf.
[17] Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to docu-
ment recognition. Proc IEEE 1998;86(11):2278–324. https://doi.org/10.1109/
5:726791.
[18] Larochelle H, Bengio Y, Louradour J, Lamblin P. Exploring strategies for training
deep neural networks. J Mach Learn Res 2009;10:1–40. URL http://dl.acm.org/
citation.cfm?id=1577069.1577070.
[19] Miotto R, Li L, Kidd BA, Dudley JT. Deep patient: an unsupervised representation to
predict the future of patients from the electronic health records. Scientific reports
2016.
[20] Yu M, Dredze M. Improving lexical embeddings with semantic knowledge.
Proceedings of the 52nd annual meeting of the association for computational lin-
guistics, ACL 2014, June 22–27, 2014, Baltimore, MD, USA, Volume 2: Short papers
2014:545–50. URL http://aclweb.org/anthology/P/P14/P14-2089.pdf.
[21] Celikyilmaz A, Hakkani-Tür D, Pasupat P, Sarikaya R. Enriching word embeddings
using knowledge graph for semantic tagging in conversational dialog systems. AAAI
– Association for the Advancement of Artificial Intelligence; 2015. URL https://
www.microsoft.com/en-us/research/publication/enriching-word-embeddings-
using-knowledge-graph-for-semantic-tagging-in-conversational-dialog-systems/.
[22] Hakkani-Tür D, Celikyilmaz A, Heck L, Tur G, Zweig G. Probabilistic enrichment of
knowledge graph entities for relation detection in conversational understanding.
Proceedings of Interspeech, ISCA – International Speech Communication
Association 2014. URL https://www.microsoft.com/en-us/research/publication/
probabilistic-enrichment-of-knowledge-graph-entities-for-relation-detection-in-
conversational-understanding/.
[23] Faruqui M, Dodge J, Jauhar SK, Dyer C, Hovy E, Smith NA. Retrofitting word
vectors to semantic lexicons. Proceedings of the 2015 conference of the North
American Chapter of the Association for computational linguistics: human language
technologies. Association for Computational Linguistics; 2015. p. 1606–15. URL
http://www.aclweb.org/anthology/N15-1184.
[24] Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng A, et al. Recursive deep
models for semantic compositionality over a sentiment treebank. Proceedings of the
2013 conference on empirical methods in natural language processing. Association
for Computational Linguistics; 2013. p. 1631–42. URL http://www.aclweb.org/
anthology/D13-1170.
[25] Landauer TK, Dumais ST. A solution to Plato's problem: The latent semantic analysis
theory of acquisition, induction, and representation of knowledge. Psychol Rev
1997;104:211–40.
[26] Yu Z, Wang H, Lin X, Wang M. Understanding short texts through semantic en-
richment and hashing. IEEE Trans Knowl Data Eng 2016;28(2):566–79. https://doi.
org/10.1109/TKDE.2015.2485224.
[27] Wu W, Li H, Wang H, Zhu KQ. Probase: a probabilistic taxonomy for text under-
standing. Proceedings of the 2012 ACM SIGMOD international conference on
management of data, SIGMOD’12. New York, NY, USA: ACM; 2012. p. 481–92.
https://doi.org/10.1145/2213836.2213891.
[28] Wingert F. Automated indexing based on snomed. Methods Inf Med 1984;24:27–34.
https://doi.org/10.1055/s-0038-1635350.
[29] Koopman B, Zuccon G, Bruza P. What makes an effective clinical query and querier?
J Assoc Inf Sci Technol 2017;68(11):2557–71. https://doi.org/10.1002/asi.23959.
[30] Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolu-
tional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ, editors.
Advances in neural information processing systems 25. Curran Associates, Inc.;
2012. p. 1097–105. URL http://papers.nips.cc/paper/4824-imagenet-classification-
with-deep-convolutional-neural-networks.pdf.
[31] Kalchbrenner N, Grefenstette E, Blunsom P. A convolutional neural network for
modelling sentences. CoRR abs/1404.2188. 2014. URL http://arxiv.org/abs/1404.
[32] Shen Y, He X, Gao J, Deng L, Mesnil G. Learning semantic representations using
convolutional neural networks for web search. Proceedings of the 23rd interna-
tional conference on world wide web, WWW’14 companion. New York, NY, USA:
ACM; 2014. p. 373–4. https://doi.org/10.1145/2567948.2577348.
[33] Deng Y, Faulstich L, Denecke K. Concept embedding for relevance detection of
search queries regarding CHOP. MEDINFO 2017: precision healthcare through in-
formatics – Proceedings of the 16th world congress on medical and health infor-
matics, Hangzhou, China, 21–25 August, 2017 2017:2333. https://doi.org/10.
3233/978-1-61499-830-3-1260.
[34] Huang P-S, He X, Gao J, Deng L, Acero A, Heck L. Learning deep structured se-
mantic models for web search using clickthrough data. Proceedings of the 22nd
ACM international conference on information & knowledge management, CIKM’13.
New York, NY, USA: ACM; 2013. p. 2333–8. https://doi.org/10.1145/2505515.
2505665.
Y. Deng et al.
[35] Lee H, Battle A, Raina R, Ng AY. Efficient sparse coding algorithms. In: Schölkopf B,
Platt JC, Hoffman T, editors. Advances in neural information processing systems 19.
MIT Press; 2007. p. 801–8. URL http://papers.nips.cc/paper/2979-efficient-sparse-
coding-algorithms.pdf.
[36] Vincent P, Larochelle H, Bengio Y, Manzagol P-A. Extracting and composing robust
features with denoising autoencoders. Proceedings of the 25th international con-
ference on machine learning, ICML’08. New York, NY, USA: ACM; 2008. p.
1096–103. https://doi.org/10.1145/1390156.1390294.
[37] Goodfellow I, Bengio Y, Courville A. Deep learning. MIT Press; 2016. URL http://
14
Artificial Intelligence In Medicine xxx (xxxx) xxx–xxx
www.deeplearningbook.org.
[38] Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol P. Stacked denoising auto-
encoders: learning useful representations in a deep network with a local denoising
criterion. J Mach Learn Res 2010;11:3371–408. URL http://portal.acm.org/
citation.cfm?id=1953039.
[39] Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, et al. TensorFlow: large-
scale machine learning on heterogeneous systems. 2015 Software available from:
URL https://www.tensorflow.org/.