Model drift is change in pattern of features and labels after the deployment compared with the features and labels that we used to train the model.
Types of Drift
Drift can takes place in either feature or the label, based on that it is classified as
a) Data Drift → When the pattern of a given feature changes after model deployment
b) Concept Drift → When the pattern of the label changes after model deployment
Why Model Drift
The model drift happens due to the nature of change that happens in domain based on seasonality /environment change. One of the typical example is the Face recognition system that is used for attendance, mobiles, laptop e.t.c due to pandemic as everyone is wearing mask, the model will not work thereby the performance degrades drastically. This scenario is called as Model Drift due to data, because the model is trained using the photo without Mask and now inferences using Face Mask.
How to identify Drift
Model drift can be evaluated using below metrics
a) Statistical Measures — Mean, Median , Minimum , Maximum Value
b) Population Stability Index (PSI)
c) Kolmogorov-Smirnov test (KS Test)
d) K-L Divergence
Statistical Measures
We can tabulate statistical measures for instance Mean for all the features in a tabular column for both the data that is used for training as well as the data that we predicted so far after model deployment. Measure the difference between the two quantities , if the variation is high, then it concludes the drift in data.
Population Stability Index (PSI)
The idea here is , take a feature, and covert the data into bins, for e.g. if we have temperature feature, then create a table row for bin values of temperature 0–10 , 11–20,21–30,31–40 e.t.c and for each of these bins find the percentage of records that falls in each of the category for the data that is used for training and also for the data that was predicted after model deployment. After we formulate the table for a feature, PSI is calculated as
PSI=∑((Train%−Prediction%) * ln(Train% / Prediction%))
PSI < 0.1 → Distribution is similar
PSI < 0.2 → Distribution has changed to some extent
PSI >= 0.2 → Distribution changed a lot
From the above metric, we can see the how well the distribution of incoming data for prediction is aligned with the data that is used for training.
Kolmogorov-Smirnov test (KS Test)
This is a hypothesis testing method which is also a metric to find the difference between the two distributions. The null hypothesis in this case is both the distributions are same.
The idea here is to find the Cumulative Distribution of data that is used to train and the data that is incoming post the model deployment and observe the difference.
The test statistic will be the supremum of difference between the test data distribution and the prediction data distribution
If we fail to reject the null hypothesis , then it means that both the distributions are not same and hence the evidence for data drift.
K-L Divergence
It quantifies the difference between two probability distributions.
P(x) can be treated as probability distribution of Train Data of a feature and Q(x) can be probability distribution of the feature data that comes to our model after we deploy. Large KL Divergence value indicates distributions are not similar and there is possibility of data drift.
Here is a code snippet taken from https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
Conclusion
Model Drift can be plugged as a machine learning pipeline which comes after the deployment phase and the results of this can be used to take decision on retraining the model.
