A multi-agent system where each agent hits a hosted API grows in cost with every user and every step, and nothing in the architecture caps it. Moving inference in-house is the obvious answer, and the obvious answer is where most attempts stall.
Why self-hosting alone fails
The instinct is to swap the API for an open-weights model and stop. But base models know how to talk, not how to use your tools. They generate fluent text and struggle with exactly the things an agent needs.
- Generate fluent text
- Answer questions
- Summarise content
- Knowing when to call a function
- Building correct parameters
- Chaining several calls
- Knowing when the task is done
Build the function-calling dataset
No dataset exists for your specific tools, so you have to create it. Each sample needs the user query, the full schema of available tools, the correct call, and — the part most people skip — the decision that the task is finished. Production traffic is the best source: real queries carry the edge cases you would never invent.
The staged path, and what each stage is worth
Applied together, this approach can take a multi-agent bill from around $18K a month to roughly $2.1K — an 88% reduction — without giving up capability. The order matters: fine-tuning is what makes the other two possible, because routing and caching are only safe once the local model is good enough to trust with the traffic they send it.
