When customer data leaves your infrastructure to reach a hosted API, you are paying for the privilege of a compliance risk. Both halves resolve the same way: bring the model to the data instead of sending the data to the model.
A local model handles more than expected
The assumption that local models cannot match hosted quality is true for general tasks and often false for narrow ones. Test several open models on your actual queries, pick the one with acceptable behaviour in your domain, fine-tune on your own data, and measure on held-out production questions.
Tuned this way, a local model can absorb roughly 80% of traffic, leaving genuine multi-step reasoning to escalate. That step alone can account for about 60% of the total reduction. The tuned model is not a compromise — within months it tends to beat the general hosted model on your specific domain, because it trains on real edge cases from real users.
Semantic caching, done properly
Standard caching compares strings, which fails immediately with natural language because people phrase the same question differently every time.
- —“What is the interest rate?”
- —“How much is the interest right now?”
- —“Can you tell me the rate on savings?” — a different question, and it must miss.
Embedding the query and matching on similarity catches the first two and correctly rejects the third. Where exact-match caching catches a couple of percent of traffic, semantic caching can reach around a third.
Combined, this approach can take a customer-service AI bill from roughly $52K a month to about $4.8K — a 91% reduction — while removing personal data from third-party APIs for the majority of traffic.
