Machine learning models degrade in production as the world changes and the model becomes less fit for the task. However, it is often challenging to catch the model decay on time, especially if the model does not have an immediate feedback loop. In this case, one can resort to monitoring proxy metrics instead of measuring model quality directly.
Evaluating statistical drift in the data inputs and model predictions is one practical approach to this problem. In this talk, I will discuss the methods to test and monitor for data drift, some challenges, and limitations and share practical tips on incorporating drift detection in the model maintenance playbook.