Linear Discriminant Analysis (LDA)

Its Linear Discriminant Analysis (LDA) to start off with this new year.  To make the points clear and easy to understand, the definitions used for explaining the concepts is kept short.

To understand the Linear Discriminant Analysis, first we need to understand the term called “Dimensionality Reduction“.

What is Dimensionality Reduction ?

The technique to reduce dimensions by removing the redundant and dependent features by transforming the features from higher dimensional space to lower dimensional space.

There are 2 major types of Dimensionality Reduction :

1. Supervised Dimensionality Reduction Technique
2. Unsupervised Dimensionality Reduction Technique

Supervised Dimensionality Reduction Technique

Supervised technique is one, where the labels are taken into consideration for dimensionality reduction.

Examples – Neural Networks (NN) , Mixture Discriminant Analysis (MDA), Linear Discriminant Analysis (LDA)

Unsupervised Dimensionality Reduction Technique

Unsupervised technique is one, where there is no need for class labels for dimensionality reduction.

Examples – Principal Component Analysis (PCA), Independent Component Analysis (ICA), Non-negative matrix factorisation (NMF).

What is Linear Discriminant Analysis ?

LDA – Linear Discriminant Analysis, a dimensionality reduction technique, which transforms the features into lower dimensional space that maximises the ratio of between-class variance to within-class variance, thereby maximum class separability.

There are 2 types of LDA:

• Class-dependent LDA
• Class-Independent LDA

Class-Dependent LDA

It is one of the technique where, one separate low dimensional space is created for each class to project its data.

Class-Independent LDA

The method where each class is considered as separate class and one low dimensional space is created for all the classes to project it data points.

Steps to calculate LDA :

1. Calculate the separability between the different classes – Between-class variance
2. Calculate the distance between mean & samples of each class – Within-class variance
3. Construct the low dimensional space which maximises the between-class variance and minimises the within-class variance.

By default, the Linear Discriminant Analysis (LDA) uses Class-Independent method for dimensionally reduction the features.

Problems with Class-dependent LDA method

• It requires more CPU time, as separate low dimension class is created for each class.
• It leads to Small Sample Size problem

Problems with LDA

• Linear Discriminant Analysis (LDA) fails to find the low dimensional space, if number of dimensions are higher than the number of samples in the data (Singularity).

Solutions

• Removing the null space within the class matrix
• Using intermediate subspace – Principal Component Analysis (PCA)
• Regularisation

References

https://people.revoledu.com/kardi/tutorial/LDA/Numerical%20Example.html

https://www.researchgate.net/publication/316994943_Linear_discriminant_analysis_A_detailed_tutorial