Speaker Adaptation with MLLR Transformation

Unsupervised speaker adaptation for Sphinx4

For building an improved acoustic model there are two methods. One of them
needs to collect data from a speaker and train the acoustic model set. Thus
using the speaker's characteristics the recognition will be more accurately.
The disadvantage of this method is that it needs a large amount of data to be
collected to have a sufficient model accuracy.
The other method, when the amount of data available is small from a new
speaker, is to collect them and by using an adaptation technique to adapt the
model set to better fit the speaker's characteristics.
The adaptation technique used is MLLR (maximum likelihood linear regression)
transform that is applied depending on the available data by generating one or
more transformations that reduce the mismatch between
an initial model set and the adaptation data. There is only one transformation
when the amount of available data is too small and is called global adaptation
transform. The global transform is applied to every Gaussian component in the
model set. Otherwise, when the amount of adaptation data is large, the number
of transformations is increasing and each transformation is applied to a
certain cluster of Gaussian components.
To be able to decode with an adapted model there are two important classes that
should be imported:
import edu.cmu.sphinx.decoder.adaptation.Stats;
import edu.cmu.sphinx.decoder.adaptation.Transform;
Stats Class estimates a MLLR transform for each cluster of data and the
transform will be applied to the corresponding cluster. You can choose the
number of clusters by giving the number as argument to
createStats(nrOfClusters) in Stats method. The method will return an object
that contains the loaded acoustic model and the number of clusters. This
important to collect counts from each Result object because based on them we
will perform the estimation of the MLLR transformation.
Before starting collect counts it is important to have all Gaussians clustered.
So, createStats(nrOfClusters) will generate an ClusteredDensityFileData object
to prepare the Gaussians. ClusteredDensityFileData class performs the clustering
using the "k-means" clustering algorithm. The k-means clustering algorithm aims
to partition the Gaussians into k clusters in which each Gaussian belongs
to the cluster with the nearest mean. It is interesting to know that the problem
of clustering is computationally difficult, so the heuristic used is the
Euclidean criterion.
The next step is to collect counts from each Result object and store them
separately for each cluster. Here, the matrices regLs and regRs used in
computing the transformation are filled. Transform class performs the actual
transformation for each cluster. Given the counts previously gathered and the
number of clusters, the class will compute the two matrices A (the
transformation matrix) B (the bias vector) that are tied across the Gaussians
from the corresponding cluster. A Transform object will contain all the
transformations computed for an utterance. To use the adapted acoustic model it
is necessary to update the Sphinx3Loader which is responsible for
loading the files from the model. When updating occurs, the acoustic model is
already loaded, so setTransform(transform) method will replace the old means
with the new ones.

Now, that we have the theoretical part, let s see the practical part. Here is
how you create and use a MLLR transformation:
Stats stats = recognizer.createStats(1);
recognizer.startRecognition(stream);
while ((result = recognizer.getResult()) != null) {
stats.collect(result);
}
recognizer.stopRecognition();
// Transform represents the speech profile
Transform transform = stats.createTransform();
recognizer.setTransform(transform);
After setting the transformation to the StreamSpeechRecognizer object,
the recognizer is ready to decode using the new means. The process
of recognition is the same as you decode with the general acoustic model.
When you create and set a transformation is like you create a
new acoustic model with speaker's characteristics, thus the accuracy
will be better.
For further decodings you can store the transformation of a speaker in a file
by performing store( FilePath , 0) in Transform object.
If you have your own transformation known as mllr_matrix previously generated
with Sphinx4 or with another program, you can load the file by performing
load( FilePath ) in Transform object and then to set it to an Recognizer object.