Ship & Maintain

How a workflow is evaluated, watched, and kept within cost and privacy limits after it ships.

No cards on this track are live yet. Concept cards are free, including the ones that are not written yet. Names without a card are not links. Practice for this track is the Ship & Maintain pack, and that pack is coming soon.

Production evaluation Coming soon Free card
A repeatable eval set you run on changes that will face real users.
Observability Coming soon Free card
Logs and traces of prompts, tool calls, and replies so a failure can be inspected.
Cost forecasting Coming soon Free card
Estimating spend from token volume, rates, and how often a workflow runs.
Latency Coming soon Free card
How long a request takes, including retrieval, tools, and the model.
Fallbacks & rate limits Coming soon Free card
What the product does when a model errors, slows down, or hits a rate limit.
Privacy & data handling Coming soon Free card
What you send, what is stored, and what should never enter a prompt.
Safety Coming soon Free card
Limits on harmful or disallowed use, in the product and in the model.
Monitoring & drift Coming soon Free card
Watching quality and behavior after launch, when data or models change.
Fine-tuning vs prompting vs RAG Coming soon Free card
Choosing among a prompt change, retrieval, and a trained update.
Licensing Coming soon Free card
What the model and the data allow you to ship.

Ship & Maintain pack Pricing