Relevance and Matrix adaptation in Learning Vector Quantization (GRLVQ, GMLVQ and LiRaM LVQ)

M. Biehl, P. Schneider and K. Bunte,
University of Groningen,
Department of Computing Science, Intelligent Systems

This page contains an overview of Generalized Matrix Learning Vector Quantization (GMLVQ) and an example concerning the corresponding Matlab script. With the Matlab script you can use the GLVQ variants Generalized Relevance LVQ (GRLVQ) and GMLVQ for classification, feature selection and linear dimensionality reduction.


Overview

Generalized Learning Vector Quantization (GLVQ) is a prototype based classification technique. The aim is to adapt a set of labeled prototypes, such that most samples in their receptive fields belong to the same class. The classification takes place by a winner-takes-all scheme: a new data sample is classified to the class given by the nearest prototype.
One extension of this algorithm is called Generalized Relevance LVQ (GRLVQ)[1]. Here a relevance vector is included in the distance measurement to scale the input dimensions according to their importance with respect to the classification task. This not only might improve the classification performance, but may also be used for feature selection.

As many classification algorithms also LVQ crucially depends on the distance measure used. Distance learning is introduced in the Generalized Matrix LVQ (GMLVQ)[2]. Here a distance measure based on the quadratic form d(x,w) = (x-w)tAtA(x-w) is adapted aiming at optimizing the discrimination of the classes. The eigenvalue spectrum of the semi-definite matrix tAtA tend to be dominated by the largest eigenvalue. A Regularization [3] penelizes this effect and leads to more homogeneous eigenvalue profiles. This is particularly useful if a low rank approximation is desired where the number of non vanishing eigenvalues should not fall below a given limit. Note that the regularization might become time consuming, because of the use of the pseudo inverse. The Limited Rank Matrix LVQ (LiRaM LVQ) [4] finds this low rank approximations. This variant optimizes a rectangular matrix A, which can be used for example for discriminative linear dimensionality reduction or to spare computational effort when the number of dimensions is very high.

The Localized GMLVQ (LGMLVQ) works with a more complex model using local matrices either attached to each prototype or in a classwise manner. This lead to non-linear decision boundaries instead of the piecewise linear ones in the global formulation. Note, that the computational effort increases dependent on the number of matrices and the dimension.

Scripts and Parameters

The toolbox supports the GRLVQ, GMLVQ and LGMLVQ algorithms. The file demo.m includes examples for the usage of all these algorithms. The following section gives an overview of the most important features. Use help for detailed information.
demo.m Examples for the usage.
Preprocessing
nFoldCrossValidation.m performs an n-fold cross validation on a labeled or unlabeled data set and returns an index list for the folds as cell array.
zscoreTransformation.m normalize the data to have zero mean and unit variance.
Algorithms
GRLVQ_train.m trains the Generalized Relevance LVQ algorithm and returns a model struct.
GRLVQ_classify.m classifies the given data with the trained model.
GMLVQ_train.m trains the Generalized Matrix LVQ algorithm and returns a model struct.
GMLVQ_classify.m classifies the given data with the trained model.
GMLVQ_project.m linear projection of the given data with the given GMLVQ model and a given target dimension. If the target dimension is smaller than the number of rows of the matrix in the model the projection might be accompanied with a loss of classification accuracy.
LGMLVQ_train.m trains the Localized Generalized Matrix LVQ algorithm and returns a model struct.
LGMLVQ_classify.m classifies the given data with the trained model.

Example for a minimal call using default parameters:
function model = GMLVQ_train(trainSet, trainLab)
Example for optional parameters:
function [model,initialization,trainError,testError,costs] = GMLVQ_train(trainSet,trainLab,...
'PrototypesPerClass',2,'dim',2,'regularization',0)

Example

load data/segment.dat;
data = segment(:,1:end-1);data(:,std(data)==0) = []; % feature 3 is constant -> exclude it
label= segment(:,end);
indices = nFoldCrossValidation(data,'labels',label,'nb_samples',300,'nb_folds',1,...
                                    'splits','random','comparable',1);
trainSet = data(indices{1},:);
trainLab = label(indices{1});

testIdx = 1:length(label);
testIdx(indices{1}) = [];
testSet = data(testIdx,:);
testLab = label(testIdx);

disp('preprocess the data using zscore');
[trainSet, zscore_model] = zscoreTransformation(trainSet);
testSet = zscoreTransformation(testSet, 'parameter', zscore_model);

GMLVQ_model = GMLVQ_train(trainSet, trainLab,'dim',18,'PrototypesPerClass',1);
GMLVQ_model_rank2 = GMLVQ_train(trainSet, trainLab,'dim',2,'PrototypesPerClass',[2,1,2,1,3,2,1]);

estimatedTrainLabels = GMLVQ_classify(trainSet, GMLVQ_model);
trainError = mean( trainLab ~= estimatedTrainLabels );
fprintf('GMLVQ: error on the train set: %f\n',trainError);
estimatedTestLabels = GMLVQ_classify(testSet, GMLVQ_model);
testError = mean( testLab ~= estimatedTestLabels );
fprintf('GMLVQ: error on the test set: %f\n',testError);

dataprojection = GMLVQ_project([trainSet;testSet], GMLVQ_model, 2);
gscatter(dataprojection(:,1),dataprojection(:,2),[trainLab;testLab],'','o',4,'off','dim 1','dim 2');

References

[1] B. Hammer and T. Villmann, "Generalized relevance learning vector quantization", Neural Networks, 15, 1059-1068, 2002. [pdf],[bib]

[2] Petra Schneider, Michael Biehl and Barbara Hammer, "Adaptive Relevance Matrices in Learning Vector Quantization", Neural Computation, vol. 21, nb. 12, pp. 3532-3561, 2009. [pdf],[bib]

[3] P. Schneider, K. Bunte, B. Hammer and M. Biehl, "Regularization in Matrix Relevance Learning", IEEE Transactions on Neural Networks, vol. 21, nb. 5, pp. 831-840, 2010. [pdf],[bib]

[4] K. Bunte, P. Schneider, B. Hammer, F.-M. Schleif, T. Villmann and M. Biehl, "Limited Rank Matrix Learning - Discriminative Dimension Reduction and Visualization", Neural Networks, vol. 26, nb. 4, pp. 159-173, 2012. [pdf],[bib]

The toolbox includes the Fast Limited Memory Optimizer fminlbfgs.m written by Dirk-Jan Kroon.
Last changed: 2012-12-07