𝗛𝗼𝘄 𝘁𝗼 𝗮𝗱𝗱 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗺𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 𝗮𝗻𝗱 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 to your ML system.
Your model is exposed to performance degradation after it is deployed to production.
That is why you need to monitor it constantly.
The most common way to monitor an ML model is to compute its metrics.
But for that, you need the ground truth.
𝗜𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻, 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗮𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗰𝗮𝗹𝗹𝘆 𝗮𝗰𝗰𝗲𝘀𝘀 𝘁𝗵𝗲 𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵 𝗶𝗻 𝟯 𝗺𝗮𝗶𝗻 𝘀𝗰𝗲𝗻𝗮𝗿𝗶𝗼𝘀:
1. near real-time: you can access it quite quickly
2. delayed: you can access it after a considerable amount of time (e.g., one month)
3. never: you have to label the data manually
.
𝗙𝗼𝗿 𝘂𝘀𝗲 𝗰𝗮𝘀𝗲𝘀 𝟮. 𝗮𝗻𝗱 𝟯. 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗾𝘂𝗶𝗰𝗸𝗹𝘆 𝗰𝗼𝗺𝗽𝘂𝘁𝗲 𝘆𝗼𝘂𝗿 𝗺𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗶𝗻 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝘄𝗮𝘆:
- store the model predictions and GT as soon as they are available (these 2 will be out of sync -> you can't compute the metrics right away)
- build a DAG (e.g., using Airflow) that extracts the predictions & GT computes the metrics in batch mode and loads them into another storage (e.g., GCS)
- use an orchestration tool to run the DAG in the following scenarios:
1. scheduled: if the GT is available in near real-time (e.g., hourly), then it makes sense to run your monitoring pipeline based on the known frequency
2. triggered: if the GT is delayed and you don't know when it may come up, then you can implement a webhook to trigger your monitoring pipeline
- attach a consumer to your storage to use and display the metrics (e.g., trigger alarms and display them in a dashboard)
.
If you want to see how to implement a near real-time monitoring pipeline using Airflow and GCS, check out my article: 𝘌𝘯𝘴𝘶𝘳𝘪𝘯𝘨 𝘛𝘳𝘶𝘴𝘵𝘸𝘰𝘳𝘵𝘩𝘺 𝘔𝘓 𝘚𝘺𝘴𝘵𝘦𝘮𝘴 𝘞𝘪𝘵𝘩 𝘋𝘢𝘵𝘢 𝘝𝘢𝘭𝘪𝘥𝘢𝘵𝘪𝘰𝘯 𝘢𝘯𝘥 𝘙𝘦𝘢𝘭-𝘛𝘪𝘮𝘦 𝘔𝘰𝘯𝘪𝘵𝘰𝘳𝘪𝘯𝘨. Link in the comments ↓
#mlops #machinelearning #data
-----
💡 Follow me for daily lessons about ML engineering and MLOps.
I lead API governance at Bloomberg, also known as the API Evangelist, studying the tech, business & politics of APIs since 2010, and Presidential Innovation Fellow during the Obama administration - opinions are my own.
8moBut I could have done 125!!! ;-)