AI for business leaders · Lesson 5 of 5

Running it after launch

AI systems fail quietly, so someone has to be watching.

Marksai · Free course · About 10 minutes

The idea

Ordinary software tends to fail loudly: a page does not load, or an error appears. AI systems fail quietly. The system keeps answering while the answers get worse, because the questions have shifted, the data has changed, or the model underneath was updated. Nothing breaks, so nothing is reported.

This means launch is not the end of the work. Someone has to be watching, and the system has to be built so that it can be watched.

Four habits

  • Instrument. The system records what it was asked, what it answered, and what it cost.
  • Monitor. Quality and cost are measured against a baseline, and a named person is told when they move.
  • Diagnose. When an answer is wrong, the cause is found, not only the symptom.
  • Improve. Fixes ship as routine, and are checked against the baseline.

Cloud teams have run large services this way for years. The same discipline applies to AI.

What this means for your budget

Plan for running the system, not only building it. That means time from a named owner, the tools to see how the system is doing, and an allowance for the cost of use, which grows as adoption grows. A project budget that ends at launch has left out the part that decides whether the investment pays back.

Five questions

  1. If its answers got worse next month, how would you find out?
  2. Can you see what it was asked, what it answered, and what it drew on?
  3. Is there a baseline that tells you whether a change made it better or worse?
  4. Who is named as responsible when something looks wrong?
  5. Do you know what it costs to run, and what drives that cost?

Exercise · about 10 minutes

Answer the five questions for one AI system you run or are planning. For each one you cannot answer, write down who would need to be involved to answer it.

One thing to take to your team

Ask who is watching each AI system you have, by name.

For more depth, read our whitepaper Running AI like a cloud service.

You have finished the course

You now have a sorted list of tasks, a scored first use case, a named stage and exit, the outline of a usage policy, and five answers about running a system. That is a sound basis for a first conversation with whoever will build it. If you would like that conversation to be with us, book 30 minutes.