# Monitoring & drift

> Monitoring and drift is watching quality after launch, and catching it when a model update or a change in the data moves behavior off the launch set.

Watching quality and behavior after launch, when data or models change.

Track: Ship & Maintain
Access: free
Card: https://buildsparkai.com/monitoring-drift
Plain text: https://buildsparkai.com/monitoring-drift.md
Last checked: 2026-10-18

## What it is

NIST's AI Risk Management Framework Playbook, MEASURE 2.4, says to monitor the functionality and behavior of an AI system in production. It describes drift as the system no longer meeting the assumptions and limitations of the original design, because the environment changes over time. It says to compare indicators you see in production with the same indicators from pre-deployment tests, and to alert when new inputs or outputs differ from those tests. LangSmith's evaluation docs describe regression tests as checks that a new version does not degrade cases the current version handles, and they say those tests typically run when the model or the architecture changes. A launch-week pass does not stay true by itself. Re-run the saved eval set when the model name changes and when the documents you retrieve change.

## Why it matters

**Cost.** Re-running the set costs the same kind of tokens as the original eval. Run it when the model, the prompt, or the retrieved documents change. This card does not add a price.

**Quality.** Drift can be a new model snapshot or a corpus that no longer matches the questions. NIST's playbook asks whether the data is still representative of the operational environment. For a retrieval product, that is whether the documents still match what users ask.

**Risk.** An alert that nobody owns is a log line. Name who looks at a failed re-run, and what they do before the next release. Uptime does not tell you the close time is still right.

## Try today

Write the model name and the date next to the latest eval result. The next time the model name or the retrieved documents change, run the same set and compare the failures with that result.

## Common mistake

Watching uptime and spend, and calling that quality monitoring. A fast, cheap reply can still be wrong. NIST's measure is about behavior.

## Self-check

What does NIST's playbook mean by drift on this card?

The system no longer meets the assumptions of the original design as conditions change

## Practice and deep

Coming soon. A launch result with the model name and date, then the same set run again after a model or corpus change.

Coming soon. How NIST MEASURE 2.4 describes production monitoring and drift, and how LangSmith describes regression tests when the model changes.

## Sources

- [NIST AI RMF Playbook, Measure](https://airc.nist.gov/airmf-resources/playbook/measure/)
- [LangSmith, Evaluation types](https://docs.langchain.com/langsmith/evaluation-types)

Opened 2026-10-18. Vendor, framework, or standards docs. This card does not add a rate, a usage count, or a score of its own. Numbers quoted from a source are labelled vendor-reported or as that document's own words.