WWW.C-CHEM.
ORG SOFTWARE NEWS AND UPDATES
QSARINS: A New Software for the Development, Analysis,
and Validation of QSAR MLR Models
Paola Gramatica,* Nicola Chirico, Ester Papa, Stefano Cassani, and Simona Kovarich
QSARINS (QSAR-INSUBRIA) is a new software for the develop-
ment and validation of multiple linear regression Quantitative
Structure-Activity Relationship (QSAR) models by Ordinary
Least Squares method and Genetic Algorithm for variable
selection. This program is mainly focused on the external vali-
dation of QSAR models. Various tools for explorative analysis
of the datasets by Principal Component Analysis, prereduction
of input molecular descriptors, splitting of datasets in training
and prediction sets, detection of outliers and interpolated or
extrapolated predictions, internal and external validation by
different parameters, consensus modeling and various plots
Introduction
Quantitative Structure-Activity Relationship (QSAR) models,
when correctly developed and rigorously validated, are highly
useful for screening and prioritization of chemicals without
experimental data or even before their synthesis in the safe
chemical design approach. Their use in regulation was also
suggested in the European legislation of chemicals REACH,[1]
in particular to reduce experimental costs and tests on ani-
mals. However, the users of QSAR models should be aware
that QSAR modeling is not a trivial approach and that it is not
sufficient to push a button and find a correlation.[24] In
recent years, particular attention has been devoted by the
QSAR community to the validation of QSAR models, with par-
ticular emphasis on applicability domain (AD) and predictiv-
ity,[513] and the famous OECD principles for the validation of
QSARs models for their application in regulation[14] have been
established to increase the confidence on the reliability of
data predicted by QSAR models. Various parameters for check-
ing the model predictivity, and specifically, for external valida-
tion on chemicals not used in model development, have been
suggested.[5,6,1521] In a recent comparison,[20,21] several prob-
lems for some of these parameters have been highlighted and
the use of more than one validation criterion and of graphical
plots has been proposed. However, it is not common to have
different statistical parameters for validation, nor a pool of use-
ful graphical visualizations contemporaneously calculated in a
single software. We here propose a new software, QSARINS
(QSAR-INSUBRIA), based on the experience of the QSAR
Research Unit of Insubria that allows to develop multiple linear
regression (MLR) QSAR models, by ordinary least squares
(OLS), carefully verified and thoroughly validated according to
the chemometric approach. In addition, QSARINS implements
various tools for multivariate analysis, applicable for the explor-
ative study of data at different steps of the modeling process
and for decision making in the selection of the best models.
for visualizations are implemented. QSARINS is a user-friendly
platform for QSAR modeling in agreement with the OECD Prin-
ciples and for the analysis of the reliability of the obtained
predicted data. The Insubria Persistent Bioaccumulative and
Toxic (PBT) Index model for the prediction of the cumulative
behavior of new chemicals as PBTs is implemented. Addition-
ally, QSARINS allows the user to validate single models, prede-
veloped using also different software. V
C 2013 Wiley Periodicals,
Inc.
DOI: 10.1002/jcc.23361
Workflow of the QSAR Approach in QSARINS
The main steps involved in the development and analysis of a
QSAR model can be summarized as: (1) Preparation, analysis,
and setup of the input dataset, (2) Model calculation based on
selected descriptors, (3) Model exploration, validation, and
selection.
The first step in QSAR modeling is the preparation of a suit-
able dataset, which is then used as the input for the QSAR
model generation by means of statistical analysis. This dataset
consists of the experimental responses to be modeled and the
corresponding set of molecular structure descriptors for each
chemical, which are calculated using appropriate software. The
preparation of a good input dataset is the first crucial point in
any QSAR approach; the relevance of data curation has been
the subject of relevant papers.[24] Usually, the number of cal-
culated descriptors is high (hundreds, sometimes even thou-
sands),[22,23] to have the possibility to represent different
features of the chemical structure in different ways. Thus, a lot
of them could be intercorrelated and redundant giving very
similar structural information. QSARINS eases the objective pre-
reduction of the molecular descriptors by means of a set of
options (listed in the following section Importing and reduc-
ing input dataset).
The next important step is the dataset analysis, i.e., the
check of the data distribution in the chemical and
P. Gramatica, N. Chirico, E. Papa, S. Cassani, S. Kovarich
Department of Theoretical and Applied Sciences, QSAR Research Unit in Envi-
ronmental Chemistry and Ecotoxicology, University of Insubria, Via Dunant 3,
21100, Varese, Italy. E-mail: paola.gramatica@uninsubria.it
Contract grant sponsor: European Union through the project CADASTER
FP7-ENV-2007-1-212668; Contract grant sponsor: Regione Lombardia and
University of Insubria for a post-doc fellowship (to N.C.)
C 2013 Wiley Periodicals, Inc.
V
Journal of Computational Chemistry 2013, 34, 21212132 2121
 SOFTWARE NEWS AND UPDATES WWW.C-CHEM.ORG
Figure 1. QSARINS workflow.
experimental space, highlighting possible outliers and/or par-
ticular clusters (see sections Dataset analysis and Data
setup).
In the QSAR practice, a school of thought and the one fol-
lowed here recommends to develop and propose models
always verifying their predictivity on new chemicals,[5,6,1013]
using a dataset, that is, training set, for establishing the quan-
titative structure-activity relationship and then validating the
models performances by means of one or more external data-
sets, i.e. prediction set(s) (which has not been used in the
models calculations). The training and the prediction sets are
extracted from the available experimental dataset, and QSAR-
INS provides a set of tools for the data splitting (see section
Data setup) procedure.
Once the data selection has been performed, the next step
is the development of the QSAR model. Models are calculated
in QSARINS by means of the MLR, using the OLS method (see
section Model calculation). The model descriptors can be a
priori selected by the modeler or by the software. It would be
ideally possible to use all the combinations of the available
descriptors for models calculation; however, the number of
combinations is usually so big, that it is impossible to calcu-
late the models for them all. Thus, to reduce the time for
model calculation, it is here proposed to use preliminarily
all combinations (all subset models) only for a small number
of descriptors per model to explore all the low dimension
combinations, and then to apply a genetic algorithm (GA)[24]
procedure for the development of models based on a big-
ger number of descriptors (see section Descriptors
selection).
During the GA selection, a long list of calculated models is
obtained: the next step is to select the best one. This analy-
sis is complex, because model performances vary depending
on the different criteria which are used to evaluate them.
First, a model must have at least a high ability to reproduce
the data used to calculate it, that is, a good fitting. Then, a
model must be verified to have a high capacity to predict por-
tions of the training dataset, using techniques collectively
known as cross-validation (CV) or internal validation. If the
model passes this verification, the model can be defined as
robust and stable. To guarantee its ability in providing reliable
2122 Journal of Computational Chemistry 2013, 34, 21212132
predictions on new chemicals, it must have good performan-
ces in predicting the external dataset(s): this procedure is
called external validation. All calculated models can be
checked, selected, filtered (see section View and select mod-
els), and validated using different techniques within QSARINS
(see section Model validation).
After performing all these steps in QSARINS, a QSAR model
can be selected as the best and various graphs for verifying
the predicted data distribution, the errors, the AD, etc. can be
plotted. The selection of the best model can be done consid-
ering various validation parameters altogether by the Multicri-
teria Decision Making (MCDM).[25] Since the GA-MLR approach
allows to obtain several similarly good models, but based on
different descriptors, these models can be cumulatively
exploited to obtain better predictions by consensus.[26] QSAR-
INS has two different tools, based on Principal Component
Analysis (PCA), for the selection of the most diverse models to
be averaged in a consensus modeling. The schematic workflow
of the steps followed by QSARINS and the main available
methods for each one are reported in Figure 1.
The aim of a model is also to be applicable for making predic-
tions in the future when data for new chemicals become avail-
able: in QSARINS it is also possible to store any single model,
even developed by other software, to validate it by various
parameters, and to apply the model later to new chemicals.
Preparation, Analysis, and Setup of the Input
Dataset
Importing and reducing input dataset
A typical input dataset for QSAR analysis is, in its basic form, a
matrix containing the studied chemicals in the rows and the
corresponding descriptors values (independent variables X, cal-
culated by means of suitable software e.g., DRAGON,[22]
PaDEL-Descriptor,[23] or others) and the experimental responses
(dependent variable Y: the end-points to be modeled) in the
columns.
To mitigate the redundancy of intercorrelated descriptors, a
prereduction of descriptors is performed based on an objec-
tive selection that uses only the independent variables X. The
WWW.CHEMISTRYVIEWS.COM
 WWW.C-CHEM.ORG
Figure 2. Import data dialog. More details in Supporting Information.
descriptors to discard are identified by: (1) tests of identical val-
ues (constant variables); (2) pair-wise correlations (according to
a user defined cutoff value; suggestion: 95%) calculating the cor-
relation among all couples of descriptors and, if a couple is
found to be highly correlated, the descriptor with the higher
correlation with the other descriptors is deleted. Descriptors can
also be deleted if the percentage of the compounds sharing the
same value is too high (suggestion: 80%).
This last option is useful for counter descriptors (e.g., num-
ber of carbon atoms, number of a functional group, etc.), for
instance in cases where only few chemicals have a certain
functional group that could be selected just for fitting the
data, but would not be useful for future predictions.
Sometimes the input dataset contains special columns rela-
tive to the dataset splitting predefined by the modeler, report-
ing which compounds will be used for model calculations (1:
training set), which will be used for its validation (2: prediction
set(s)), and which could be excluded from computations (0:
excluded). In these cases, it is possible to verify the correlation
of the descriptors and the consequent selection only using the
training set chemicals.
QSARINS provides all the aforementioned procedures, as
reported in the Import data dialog of Figure 2.
Dataset analysis
Once data are imported, they can be inspected by various
graphs (see examples in Fig. 3). In fact, before the data setup
and modeling, it is useful and suggested to display the profile
and distribution of the compounds in the dataset, in terms of
structural and experimental domains. The distribution of the
compounds for each descriptor and for each response is plot-
ted/visualized in View variables/objects profiles. The correla-
tion among data, and the scatter plots of each couple of data,
can be also visualized.
SOFTWARE NEWS AND UPDATES
These kinds of data inspections can be performed at any step
in the QSAR model development, to continuously check the
quality of the data selected by the subsequent procedures.
To further explore the distribution of the studied com-
pounds in the chemical structural space, a multivariable
explorative analysis can be performed by PCA of molecular
descriptors in the data setup section.
Data setup
In this section, the QSAR developer can select the compounds
and the descriptors for the development of the model. The
Data setup dialog allows for the selection of: (1) the molecular
descriptors to include in the subsequent variable selection pro-
cedure, (2) the response (endpoint) to be modeled and (3) the
status of the molecules under study (training, prediction,
excluded) (see Fig. 4).
Every molecular descriptor can be normalized over the
range of values of all the corresponding chemical compounds,
by pressing the Normalize Var button.
At this step, it could be useful to explore the chemical space of
the studied dataset by the PCA of the chosen descriptors (see Fig.
5). By means of a score plot (upper graph in Fig. 5), it is possible
to check the existence of molecules clustered by similar struc-
tures and/or to detect strong structural outliers, while the loading
plot (lower graph in Fig. 5) allows to identify those descriptors
that highly influence the occurrence of these clusters/outliers.
In addition, the QSAR developer (or the user) can apply in
QSARINS three different kinds of splitting of the compounds in
the dataset: (1) completely random splitting, setting a pre-
ferred random percentage of the compounds to be put aside
in the prediction set; (2) by ordered response; (3) by structure,
ordering the PC1 score of the molecular descriptors. In options
2 and 3, the first and the last chemicals in the training set are
by default always included.
Journal of Computational Chemistry 2013, 34, 21212132 2123
 SOFTWARE NEWS AND UPDATES WWW.C-CHEM.ORG

Figure 3. View Data dialog. The main dialog on the upper position reports tabulated data and allows graphical display options. The dialog on the middle
is an example of data scatter plot in the space of two arbitrary molecular descriptors. The dialog on the lower position plots the distribution of the experi-
mental data.

At this step, the modeling descriptors could be selected is the random error (called also model residual). The intercept
either a priori by the QSAR developer, or automatically by the (b0 ) and the coefficients (bj ) are thus to be estimated.
procedure explained in the Descriptor selection section. The eq. (1) can be rewritten in a more compact form using
the matrix notation:

Model Calculation Based on Selected y5Xb1e (2)

Descriptors
where y is the responses vector, b the vector of the coeffi-
Modeling method in QSARINS cients, and e the vector of the errors. X is the matrix of the
model, where the columns are the descriptors.
The datasets used in QSAR analysis are, as already mentioned,
In this software, to estimate the vector of the coefficients,
composed of descriptors that should be correlated with the
the OLS technique is used:
corresponding experimental responses. At this step it is neces-
sary to apply a quantitative method able to find the existing _  21
relationship between a limited number of structural descrip- b 5 XT X XT y (3)
tors and the modeled response. _

In QSARINS, the used method is the MLR approach that can where b is the vector that estimates the b vector of the coef-
be exemplified by the following formula: ficients, XT the transposed X matrix and 21 is the inverse
matrix operation.
Xn
The OLS minimizes the sum of squares of the difference
yi 5b0 1 bj xij 1ei (1)
j51
between the experimental responses and the ones calculated
by the model. To work properly, the OLS assumes that: (1) a
where a linear relationship is computed between the studied linear relationship exists between the descriptors and the
responses (yi ) and the selected values of the descriptors (xij ); ei response, (2) the response errors are independent and similarly

2124 Journal of Computational Chemistry 2013, 34, 21212132 WWW.CHEMISTRYVIEWS.COM

 WWW.C-CHEM.ORG SOFTWARE NEWS AND UPDATES

Figure 4. Data setup dialog. More details in Supporting Information.

Figure 5. Examples of PCA of data. The upper dialog is a score plot of the molecular compounds while the lower dialog is the corresponding loading plot
of the selected descriptors. Both axes report the chosen PCA components.

Journal of Computational Chemistry 2013, 34, 21212132 2125

 SOFTWARE NEWS AND UPDATES WWW.C-CHEM.ORG
distributed, (3) the descriptors are not too correlated among
them (for this reason QSARINS control this point both in the
preselection step and by the QUIK rule,[27] see the section
Filtering during calculation), (4) there are more compounds
than modeling descriptors (a ratio that should be always
higher than 5:1).
Once the coefficients of the model are calculated, it is possi-
ble to obtain the vector of the y^, as in the following formula:
_  21
y^ 5Xb 5X XT X XT y5Hy (4)
where H is the leverage (or hat) matrix that relates the calcu-
lated and the experimental responses. The diagonal elements
of the hat matrix hii are useful to determine the distance of
the i object from the centre of the chemical space of the
model,[68,11,28] thus, for checking the structural AD of the
model (see in the Single model details section).
Descriptor selection
After setting and exploring the data, a pool of molecules with
the corresponding descriptors and the studied end-point
becomes available for model calculation. QSAR models are
characterized by their dimension, that is, the number of mod-
eling descriptors, and a limited number of modeling descrip-
tors, related to the studied response, must be selected from
the available pool. Calculating all models corresponding to all
descriptors combinations is feasible only for small dimensions
(often 23): this technique is called all subset and is suggested
in the QSAR practice, having verified in our experience the util-
ity of exploring all the low dimension models before applying
any selection tools.
If satisfying models are not obtained by the all subset pro-
cedure and higher dimension models are needed, a different
method, able to find the best combinations without exploring
them all, should be applied: in QSARINS this is done using a
GA procedure.[24] This technique is able to explore a wide
range of solutions, searching for the best ones, by maximizing
(or minimizing) a selected fitness function. This is done mim-
icking the natural selection, where the best solutions replace
the less performing. In biological terms, one would say that
the best genes in the population replace the less fitting. In
our case, every descriptor represents a gene, and a set of
descriptors represents a chromosome. The fitness of a chromo-
some is related to the corresponding model performances.
Starting with a pool of chromosomes, small subsets of chro-
mosomes are picked at random, and the best become parents
(this technique, used in QSARINS, is called tournament selec-
tion[24]). Couples of parent chromosomes are then crossed at a
random position (crossing-over), thus obtaining the offspring,
whose chromosomes are a mixing of the parent ones. If
among the new chromosomes one or more of them outper-
form the less fitting in the parent population, these chromo-
somes will substitute the less performing. Repeating the
aforementioned procedure many times, and introducing also
random mutations (descriptor substitution) in the chromo-
somes, the result at the end of the process is a population of
2126 Journal of Computational Chemistry 2013, 34, 21212132
models with better performances than the models introduced
at the beginning.
In order to avoid a completely random beginning of the GA,
in QSARINS, the best set of descriptors extracted from the all
subset procedure is used as the core of the chromosomes of
the initial population.
In QSARINS, the tuning of the GA can be done varying the
population size, the mutation rate, and the number of
generations.
A fundamental option is the selection of the fitness function
to be used by GA. This function is used to calculate the per-
formance of each model: in QSARINS, R2adj , LOF,[29] Root Mean
Square Error in cross validation (RMSECV ), and Q2LOO are pro-
vided (all formulas are reported in details in Supporting Infor-
mation). R2adj and LOF detect the possible overfitting of a
model so, used as fitness functions, are useful to select models
with high fitting with the minimum number of descriptors.
However, it is important to note that they are fitting criteria,
so they provide no information on the predictive ability of the
models. For this reason, it is here suggested to use Q2LOO as fit-
ness function for the selection of predictive models. The graph
that plots Q2LOO vs LOF can help in the selection of the models
with the best compromise between high internal predictivity
and small dimension.
Alternatively to the all subset and GA techniques, QSARINS
also provides the possibility to add iteratively, one by one, all
the descriptors to a known model (whose descriptors are kept
fixed in the process). This allows checking whether there is an
improvement in the model performances due to such
additions.
Filtering during calculation
During a typical QSAR session, thousands of alternative models
are calculated following the descriptors selection. In order to
avoid useless cluttering of the final output model list, it is pos-
sible to exclude them beforehand if they do not pass a mini-
mum quality test. If requested by the QSAR developer, the
QUIK rule[27] can be applied. This method tests whether the
total correlation among the block of descriptors (KXX ) is higher
than the correlation among them and the responses (KXY ), that
is, a model is excluded if KXY 2KXX < dK , where dK is a user
defined threshold value. In fact, a model makes sense if the
correlation among the descriptors is smaller than the correla-
tion among them and the response, that is, the model has low
multicollinearity and good correlation with the modeled
response. Thus, dK must be positive and should be as high as
possible.
Model Exploration, Validation, and Selection
View and select models
After calculation and storage of the models, it is essential to
display them in a form that makes their sorting and further
selection (see Fig. 6) user friendly. In QSARINS, it is possible to
sort the calculated models, visualizing them all or only a
selected portion of the best models, according to various
WWW.CHEMISTRYVIEWS.COM
 WWW.C-CHEM.ORG
Figure 6. View and select models dialog. More details in Supporting Information.
criteria: fitting (highest R2 ), robustness (highest Q2LOO ), stability
(lowest R2 -Q2LOO ), the lowest descriptor correlation and the
highest correlation with the response (low KXX and high dK ),
and the root mean squared errors on (a) training calculation
(RMSETR), (b) training prediction by leave one out (LOO)
(RMSECV ), and (c) external prediction set (RMSEEXT ). It is
obvious that the most stable, predictive, and more generaliz-
able model must have the lowest difference between the fit-
ting, the CV and the external parameters and, as a
consequence, have RMSE values as similar as possible.
An additional and interesting criterion for an overall evalua-
tion of the model population is to assess the relative impor-
tance of the modeling descriptors, evaluated by counting their
occurrences among all listed models. The more often a
descriptor is represented among models, (even if in combina-
tion with other alternative descriptors), the more it is likely to
be a good modeling descriptor. In addition, the more repre-
sented descriptors could also be used for a subsequent proce-
dure, that is, an all subset calculation, to further explore other
combinations.
It is here also proposed to prefer models with the lowest
number of bad predictions (Y-outliers) and chemicals far from
the training structural domain (X-outliers, out of the structural
AD, also for unknown chemicals without experimental data).
QSARINS allows for sorting the models according to these
counts. The Y-outliers are those with standardized residuals
higher than a user definable threshold (suggested 2.53r),
while X-outliers are those with a leverage value (taken from the
SOFTWARE NEWS AND UPDATES
hat matrix diagonal) higher than the cutoff value h* 53p0 /n,
where p0 is the number of model variables plus one, and n is
the number of the objects used to calculate the model.[68,11,28]
Model validation
Model calculation is the basic step in QSAR analysis, but is not
sufficient to guarantee the model validity: it is essential to
check various model performances, that is, fitting, stability by
CV, and capability to predict new chemicals (external valida-
tion).[5] In addition, it is important to verify that the model is
not obtained by chance. These performances are measured by
appropriate formulas/methods of various validation
criteria.[20,21]
The model fitting is evaluated by the coefficient of determi-
nation R2 , and a modified form R2adj which also assesses the
convenience of adding a new descriptor to a model. Both
these criteria evaluate how well the model reproduces the
data used to create it and this is an essential step in model
evaluation. However, these criteria say nothing about the abil-
ity of the model to predict new data or whether the model is
just the result of chance correlation. If a model is unstable,
small changes in the data lead to big changes in the predic-
tions, and the checking of model robustness is called internal
validation.
A well known criterion for internal validation, widely used in
the QSAR practice, is Q2LOO applying the CV technique, that is,
by iteratively excluding one compound from the dataset
Journal of Computational Chemistry 2013, 34, 21212132 2127
 SOFTWARE NEWS AND UPDATES
Figure 7. Example of plot of LMO validations and Y-scrambled models compared with the original model.
(LOO), and then computing a model with the remaining com-
pounds, making a prediction for the excluded one. If internal
predictions are good, that is, Q2LOO has a high value compara-
ble to R2 , the model is considered to be internally stable or
robust. However, a perturbation of only one compound at a
time is too weak to demonstrate real model robustness. In
QSARINS, the stronger Leave-More (or Many)-Out (LMO) tech-
nique is also included: this technique studies the behavior of
the model when a larger number of compounds is excluded.
Random samples of a defined percentage of compounds are
excluded iteratively, the model is calculated using the remain-
ing data, and then its performances are calculated predicting
the excluded compounds. The model under analysis can be
considered stable if the R2 and Q2 values calculated in every
LMO iteration, and their averages (R2LMO and Q2LMO ), are close
to the R2 and Q2LOO values of the model (see upper graph in
Fig. 7).
To demonstrate that the model is not the result of chance
correlation, the Y-scrambling procedure can be applied. In this
procedure, the responses are shuffled at random, so no corre-
lation between them and the descriptors should exist. As a
consequence, the performances of the corresponding
scrambled models should decrease drastically. In this case, if
2128 Journal of Computational Chemistry 2013, 34, 21212132
WWW.C-CHEM.ORG
the original model under validation is good, the values of R2
and Q2 of every iteration, and their averages (R2YS and Q2YS ),
must be far and much smaller from the values of the original
model (see lower graph in Fig. 7).
To compare the different R2 and Q2 values in the x axis of
Figure 7, the correlation between the model descriptors and
the response (KXY ) is reported. It is intuitive that the compared
values must be very similar among them and also with compa-
rable KXY values in the LMO procedure, while in Y-scrambling
the scrambled models must not only have R2 and Q2 values
lower then the original model, but also lower KXY values.
In addition to the aforementioned techniques, two varia-
tions of the classical Y-scrambling are applied in QSARINS: ran-
dom responses and random descriptors.[30] In the random
responses procedure, random values are picked within the
range of the responses and used as end-points for new model
calculations, instead of randomly shuffling the actual responses
as in the classical Y-scrambling. The same logic is used in the
random descriptors procedure, applied to demonstrate that
models based on chance numbers are very different from the
original model: random values are chosen within an arbitrary
range, comparable with true values found for descriptors. As a
consequence, for original successful models not obtained by
WWW.CHEMISTRYVIEWS.COM
 WWW.C-CHEM.ORG
Figure 8. Single model dialog. More details in Supporting Information.
chance, such techniques destroy the structure among data
and, as a consequence, the performances of the scrambled/
random models are very low.
Once a model is internally validated and chance correlation
is excluded, external validation can be performed, that is, the
model is checked for its ability to predict new compounds.
This is done by applying the model equation, obtained on the
training set, to one or more prediction data set(s), that is, the
excluded compounds that have never been used in model cal-
culation, and measuring the performances by means of differ-
ent criteria, such as: RMSEEXT,[9] Q2F1 ,[15] Q2F2 ,[16] Q2F3 ,[17] rm
2 plus
2 [18] [1921]
Drm , CCC, and the Golbraikh and Tropsha method.[5]
Such a set of different criteria, and not just one, are here cal-
culated and compared because we have analyzed them all in
depth and demonstrated that they are not always in agree-
ment in measuring the model predictivity.[20,21] Thus, it would
be best to investigate them all before selecting a model for its
performances in external predictivity.
Since the models can be evaluated by means of different
validation criteria, a method to evaluate model performances
is to count how many validation criteria are fulfilled. This can
be done by first setting a threshold of acceptance for every
validation criterion (default values are set in QSARINS, as sug-
gested in our previous work,[21] in a designated dialog box,
but can be changed by the user).
However, it is important to note that the selection of the
best models depends on the QSAR modeler preference,
SOFTWARE NEWS AND UPDATES
aims, and application. To help the user selection, exploiting
contemporaneously all the available information regarding the
model validation, the sort of the models according to the
MCDM procedure[25] is implemented in this software. This pro-
cedure can be applied for fitting, cross validation, and external
validation criteria separately, or together.
After the above evaluations, a manual selection of the mod-
els can be also performed. The user can decide to choice only
one model and to use it for prediction of new additional
chemicals (in Single model use and implementation section)
or to select a certain number of good models, with similar per-
formances, but based on different descriptors, which could
then be used in a subsequent consensus modeling (see sec-
tion Consensus modeling), that allows to obtain a combined
average model, which usually provides better predictions.
Single model details
The QSAR developer needs to explore the models further in
full details. This can be done by: (1) looking at the model
equation, (2) verifying the relevance of each descriptor by the
standardized coefficients, (3) checking the confidence intervals
for each descriptor, (4) verifying the values of the various crite-
ria used for the validation, (5) looking at the statistics concern-
ing all compounds both in tabulated and in plotted form, as
reported in Figures 8 and 9. At this point all the information
of the studied chemicals (e.g., ID, CAS, experimental values,
Journal of Computational Chemistry 2013, 34, 21212132 2129
 SOFTWARE NEWS AND UPDATES WWW.C-CHEM.ORG

Figure 9. Upper left graph: scatter plot of experimental endpoint data versus predicted by LOO. Upper right graph: plot of experimental data versus resid-
uals (experimentalpredicted data by LOO). Lower left graph: Hat diagonal values versus standardized residuals (Williams plot). Lower right graph: Hat
diagonal values versus predicted data (Insubria graph).

predictions, hat values, residuals, and standardized ones) are
displayed.
One first supporting tool for the assessment of the model
quality is the correlation matrix of the used descriptors: in
addition to the already mentioned KXX value, it is proposed to
verify the degree of correlation among each couple of model-
ing descriptors in each model.
A deeper analysis of a QSAR model can be done by visualiz-
ing a series of informative plots, that are useful in finding
some specific anomalies:[21] (a) the scatter plot of the experi-
mental versus the predicted data, (b) the plot of the residuals
(c) the Williams plot for the study of the model AD of training
and prediction chemicals, highlighting the X- and Y-outliers,
and (d) the Insubria graph for the study of the structural AD
on compounds without experimental data.[4] All the above
mentioned graphs of validation techniques can be produced
by QSARINS, either by applying the model equation or the
LOO technique.
The scatter plot of the experimental versus the predicted
responses (see upper left graph in Fig. 9) easily detects sys-
tematic trends or data clustering and, if any, strong outliers in
the data.

2130 Journal of Computational Chemistry 2013, 34, 21212132 WWW.CHEMISTRYVIEWS.COM

 WWW.C-CHEM.ORG
The plot of the residuals (see upper right graph in Fig. 9)
allows to evaluate the deviations from the ideal matching
between experimental and predicted data and to detect
anomalous trends.
The Williams plot detects the outliers for the response (Y-
outliers) and those for the structure (X-outliers). It consists of
plotting the standardized residuals on the y-axis and the lever-
age values from the hat matrix diagonal on the x-axis (see
lower left graph in Fig. 9). Concerning the residuals, all the
chemicals falling above or below the user defined threshold
are not well predicted and thus considered as outliers. Too
many outliers, particularly those underestimated, are sympto-
matic of a poor model and this is the reason of implementing
the counting of the outliers, as reported in the View and
select models section. Leverage values represent the degree
of influence that the structure of every single chemical has on
the model. A compound with high leverage in a QSAR model
is the driving force for the variable selection if this compound
is in the training set (good leverage). A high leverage com-
pound in the prediction set is detected as far from the chemi-
cal domain of the training compounds, thus it could lead to
unreliable predicted data, being the result of substantial
extrapolation of the model. Therefore, the structural informa-
tion of the chemicals included in the training set could be not
sufficient for a reliable prediction of chemicals lying outside of
the training-AD.
Any QSAR model can also be applied to predict the end-
point for the molecules lacking an experimental response: it is
always possible to evaluate their position in comparison to the
structural AD and compare their predictions with those of the
compounds that have experimental values. To visually explore
them, the Insubria graph, which plots the hat diagonal values
versus the predicted responses, (see lower right graph in Fig.
9) has been recently proposed by our group.[4]
Consensus modeling
A single QSAR model is not likely to include all the aspects of
the molecular structures, since it is built up using a small sub-
set of the available descriptors. An individual QSAR model may
overemphasize some aspects, underestimate others, and
ignore many important features completely. In the QSAR mod-
eling approach based on GA-MLR, as implemented in QSAR-
INS, a long list (a population) of models, with similar good
performances and based on different descriptors, is available.
The descriptors selected in the various models are usually dif-
ferent, thus potentially representing multiple aspects of the
molecular structures. It is thus reasonable to make an average
model or consensus model from the chosen list, in order to
make more accurate predictions. This consensus modeling can
take into account simultaneously several peculiar aspects of
some particular chemical structures. The consensus QSAR
modeling has already demonstrated its utility in many QSAR
studies[26,31,32] providing better predictive ability than the
majority of individual models.
To catch as much variation as possible among the list of
models, the selection of the more diverse models is feasible in
SOFTWARE NEWS AND UPDATES
QSARINS by the PCA of the prediction residuals[26] or of the
different modeling molecular descriptors. The consensus pre-
dicted data are obtained by averaging the predictions of the
selected models (both applying the model equation in fitting
and in LOO cross validation). The supporting statistics such as:
standard deviations, the minimum and maximum values, and
the average of the diagonal hat matrix values (these values
allow to understand if the chemicals are outside the AD of the
consensus model) are also provided. The predictions from
each single model can be also averaged, using different
weights in the Weighted Consensus Modeling.[33] Finally, the
AD of the consensus model can be plotted as Williams plot
and/or Insubria graph.
Single model use and implementation
The aim of the QSAR practice is to calculate a model able to
make good predictions for new chemicals, also for not yet syn-
thesized ones (chemical design). QSARINS eases this task in two
steps. The first is the possibility to store any MLR-OLS model
(also developed by other software) for a later application. The
second is the possibility to obtain predictions for new chemicals
automatically applying the model developed in QSARINS. In
both cases it is possible to verify the model performances by
the various validation criteria and plots available in the software.
The Persistent Bioaccumulative and Toxic (PBT) Index model,
developed within our research group,[34] for the prediction of
the cumulative behavior of new chemicals as PBTs, is provided
by default as a special feature of QSARINS (current version
1.2). The original and published PBT Index model was based
on DRAGON descriptors. The molecular descriptors were
selected by GA and their ability in predicting chemicals, not
participating to model development (training: 54 chemicals)
was preliminarily verified on external chemicals (126 chemi-
cals): Q2F1 5 80.72%; R2EXT 5 89.27%; RMSEEXT 5 0.72. We had
also applied this model to an additional 70 chemicals without
PBT Index values for the prediction of their PBT behavior.
The model now implemented in QSARINS software is the
following, redeveloped using the same descriptors calculated
by the open-source software PaDEL-Descriptor:[23]
PBT Index 5 21.46 (60.10) 1 0.64 (60.03) nX 1 0.22 (60.01)
nBondsM0.39 (60.07) nHBDon_Lipinksi0.06 (60.03) MAXDP2
n5180; R2 588:89%; Q2LOO 588:25%; Q2LMO 588:20%;
R2YS 50:02; RMSETR 50:51
Additional models generated by the Insubria group and the
related QSAR Model Reporting Format[35] will be included in
the future updated versions of QSARINS. Furthermore, a data-
base of 3D chemical structures, and experimental endpoints,
collected and modeled by the Insubria group in the last 15
years, will be also added in QSARINS.
Conclusions
QSARINS is proposed as a new software for the calculation,
analysis, validation, and application of QSAR MLR-OLS models
Journal of Computational Chemistry 2013, 34, 21212132 2131
 SOFTWARE NEWS AND UPDATES WWW.C-CHEM.ORG
according to the OECD principles. It allows to import and
reduce input data by automated procedures and to visualize
the results both in tabulated and graphical form. A flexible
data setup system is then proposed before the calculation of
the models, including an explorative study of the chemicals
space by PCA (based on molecular descriptors) and different
approaches for the dataset splitting in training and prediction
sets. Descriptors selection is then performed by different
methods including the all subset and GA techniques. Once the
models are obtained, they can be listed and filtered using dif-
ferent methods. Models can be evaluated as a comparative list
or singularly for a more detailed analysis. Different validation
techniques can also be applied to evaluate their performances.
Strong emphasis has been devoted to the external validation
by comparing various criteria, by combining them using the
MCDM procedure, and to the visual inspection of the model
data by informative plots.
After model evaluation, a consensus modeling based on the
best of them can be performed in order to obtain usually
more reliable predictions. The best models can be saved for
future application. In QSARINS, there is the possibility to rede-
velop, check, validate, and apply to any new chemicals single
MLR models, previously developed also by other tools.
The PBT Index model is here implemented for the detection
of PBT chemicals from their structural information only. QSAR-
INS, that runs under Microsoft Windows 2000 or more recent
versions, is available free of charge for academia and research
centers only, and can be requested from the authors by
e-mail. Further details are available in the web site: www.qsar.it
and in the help-function in the software.
Acknowledgement
Dr. Leon van der Wal is acknowledged for his critical comments on
the manuscript and English revision.
Keywords: QSAR  modeling software  OLS regression 
validation  plots
How to cite this article: P. Gramatica, N. Chirico, E. Papa, S.
Cassani, S. Kovarich. J. Comput. Chem. 2013, 34, 21212132.
DOI: 10.1002/jcc.23361
Additional Supporting Information may be found in the
online version of this article.
[1] Regulation (EC) No 1907/2006 of the European Parliament and of the
Council (18/12/2006) concerning REACH. Available at: http://ec.euro-
pa.eu/environment/chemicals/reach/reach_intro.htm, accessed on Feb-
ruary 13, 2013.
[2] A. Tropsha, Mol. Inf. 2010, 29, 476.
[3] D. Fourches, E. Muratov, A. Tropsha, J. Chem. Inf. Model. 2010, 50,
1189.
2132 Journal of Computational Chemistry 2013, 34, 21212132
[4] P. Gramatica, S. Cassani, P.P. Roy, S. Kovarich, C.W. Yap, E. Papa, Mol. Inf.
2012, 31, 817.
[5] A. Golbraikh, A. Tropsha, J. Mol. Graph. Model. 2002, 20, 269.
[6] A. Tropsha, P. Gramatica, V.K. Gombar, QSAR Comb. Sci. 2003, 22, 69.
[7] L. Eriksson, J. Jaworska, A. Worth, M. Cronin, R.M. McDowell, P.
Gramatica, Environ. Health Perspect. 2003, 111, 1361.
[8] T.I. Netzeva, A.P. Worth, T. Aldenberg, R. Benigni, M.T. D. Cronin, P.
Gramatica, J.S. Jaworska, S. Kahn, G. Klopman, C.A. Marchant, G. Myatt,
N. Nikolova-Jeliazkova, G.Y. Patlewicz, R. Perkins, D.W. Roberts, T.W.
Schultz, D.T. Stanton, J.J. M. van de Sandt, W. Tong, G. Veith, C. Yang,
ATLA 2005, 33, 155.
[9] O.A. Aptula, N.G. Jeliazkova, T.W. Schultz, M.T. D. Cronin, QSAR Comb.
Sci. 2005, 24, 385.
[10] T. Oberg, Atmos. Environ. 2005, 39, 2189.
[11] P. Gramatica, QSAR Comb. Sci. 2007, 26, 694.
[12] T. Puzyn, A. Gajewicz, A. Rybacka, M. Haranczyk, Struct. Chem. 2011,
22, 873884.
[13] T. Puzyn, A. Mostrag-Szlichtyng, A. Gajewicz, M. Skrzynski, A.P. Worth,
Struct. Chem. 2011, 22, 795.
[14] OECD Principles (2004). Available at: http://www.oecd.org/dataoecd/
33/37/37849783.pdf, accessed on February 13, 2013.
[15] M. Shi, H. Fang, W. Tong, J. Wu, R. Perkins, R.M. Blair, W.S. Branham,
S.L. Dial, C.L. Moland, D.M. Sheehan, J. Chem. Inf. Comput. Sci. 2001,
41, 186.
[16] G. Sch rmann, R. Ebert, J. Chen, B. Wang, R. K
uu uhne, J. Chem. Inf.
Model. 2008, 48, 2140.
[17] V. Consonni, D. Ballabio, R. Todeschini, J. Chemometrics, 2010, 24, 194.
[18] P.K. Ojha, I. Mitra, R.N. Das, K. Roy, Chemom. Intell. Lab. Syst. 2011, 107,
194.
[19] L.I. Lin, Biometrics 1989, 45, 255.
[20] N. Chirico, P. Gramatica, J. Chem. Inf. Model. 2011, 51, 2320.
[21] N. Chirico, P. Gramatica, J. Chem. Inf. Model. 2012, 52, 2044.
[22] DRAGON for Windows (Software for molecular Descriptor Calculation),
Talete srl. Available at: http://www.talete.mi.it, accessed on February
13, 2013.
[23] PaDEL Descriptor. Available: http://padel.nus.edu.sg/software/padelde-
scriptor, accessed on February 13, 2013.
[24] R.L. Haupt, S.E. Haupt, In Practical Genetic Algorithms, 2nd ed.; Wiley,
New Jersey, 2004.
[25] H.R. Keller, D.L. Massart, J.P. Brans. Chemom. Intell. Lab. Syst. 1991, 11,
175.
[26] P. Gramatica, P. Pilutti, E. Papa, J. Chem. Inf. Comput. Sci. 2004, 44,
1794.
[27] R. Todeschini, V. Consonni, A. Maiocchi, Chemom. Intell. Lab. Syst.
1999, 46, 13.
[28] A.C. Atkinson, In Plots, Transformations and Regression; Clarendon
Press, Oxford, 1985.
[29] J.H. Friedman, Ann. Stat. 1991, 19, 1.
[30] C. Rucker, G. Rucker, M. Meringer, J. Chem. Inf. Model. 2007, 47, 2345.
[31] H. Zhu, A. Tropsha, D. Fourches, A. Varnek, E. Papa, P. Gramatica, T.
Oberg, P. Dao, A. Cherkasov, I.V. Tetko, J. Chem. Inf. Model. 2008, 48,
766.
[32] B. Bhhatarai, W. Teetz, T. Liu, T. Oberg, N. Jeliazkova, N. Kochev, O.
Pukalov, I. Tetko, S. Kovarich, E. Papa, P. Gramatica, Mol. Inf. 2011, 30,
189.
[33] J. Li, B. Lei, H. Liu, S. Li, X. Yao, M. Liu, P. Gramatica, J. Comput. Chem.
2008, 29, 2636.
[34] E. Papa, P. Gramatica, Green Chem. 2010, 12, 836.
[35] ECHA report. Available at: http://echa.europa.eu/documents/10162/
13655/pg_report_qsars_en.pdf, accessed April 8, 2013.
Received: 9 April 2013
Revised: 3 June 2013
Accepted: 4 June 2013
Published online on 21 June 2013
WWW.CHEMISTRYVIEWS.COM