Managing AI workflows as a production operation

Before working in GTM, I worked in semiconductors, including process optimization at Cree and manufacturing analytics at Cypress. That experience shapes how I think about one person overseeing a business process carried out by agents. An operator on an automated line depends on process engineers who refine recipes and on equipment and facilities teams who keep the line running within specification. Yield engineers investigate defects that may start several steps before anyone detects them. Counting only the operator would leave those people out of the staffing estimate.

A machine can follow its recipe and still produce something unusable. NIST's CHIPS Metrology Program describes how inadequate measurement can reduce yield and quality and increase cost. Running the line faster can produce more defective parts before anyone notices. An AI workflow could have a similar problem. For example, it might prepare a renewal proposal, get approval and update the CRM while using the wrong contract throughout. A count of completed runs would miss the error.

For that workflow, a measure closer to yield would track renewals completed within policy without correction. Looking at the results by contract type and account segment would help the team notice a problem affecting only some customers. A good overall rate could conceal repeated mistakes on an unusual contract. The team would still need to inspect examples to understand what went wrong and where it started.

The manufacturing comparison has limits. A price can be checked against a rate card, but people may disagree about the best way to help a customer, and it could take months to see the result. A model can help review recommendations, but people who understand the business need to decide what a good answer looks like and check whether the model agrees for sensible reasons. They may have to leave some cases unresolved rather than force a pass or fail.

Those tests need to change as the business changes. After a prompt, model or connector update, someone has to run them and investigate any new failures. The people who previously prepared renewal materials could help with this if they have training and time set aside for it. An engineer fixing a connector may still need a sales or finance colleague to explain why an answer is wrong. A staffing plan needs to give those colleagues time to help.

Cost per usable result

Several agents may each read the account history and make repeated attempts before producing a proposal a person can approve. Anthropic reported in June 2025 that multi-agent systems in its data used about 15 times the tokens of chat interactions. That is specific to its systems, but it illustrates why the cost of a chat response is a poor starting point for budgeting a longer workflow. The team needs to measure the full run, including failed attempts and the time people spend correcting the result.

Lower model prices can help without reducing the total bill. Stanford's 2025 AI Index reported a greater than 280-fold decline in inference cost at roughly GPT-3.5-level performance on MMLU between November 2022 and October 2024. A team could still spend more if it processes more cases or makes more model calls per case. Code can handle a rate-card lookup, leaving model calls for work where testing shows they help. The system should also stop after a set number of failed attempts and send the case to someone who can investigate it.

Work in progress during an outage

A provider outage might leave one proposal approved but not sent and another waiting for account data. The team needs a record of where each case stopped and which actions succeeded. Otherwise, restarting a workflow could send the same proposal twice. Switching to another model helps only if that model has been tested for the job and doesn't depend on the same service that failed.

The remaining team may be too small to handle the usual volume manually. It needs to know which customer commitments are urgent and which work can wait. When service returns, the unfinished cases will arrive alongside new work, so recovery could take longer than the outage itself. Before promising a turnaround time, the manager needs to check how many cases the team can handle and how it will catch up on the ones that had to wait.