Every few months a new model claims the top of some benchmark, and every few months we get asked whether we have switched Romeo over to it. Usually the honest answer is no, not yet, and the reason has little to do with raw capability.
Latency budgets, cost per action, and — most of all — consistency under our specific prompts and tools matter more day to day than a benchmark score. A model that is 2% smarter but twice as slow, or occasionally ignores a tool schema, is a worse choice for an assistant that needs to feel dependable.
We also plan for the model layer to fail or degrade, the same way we plan for any other dependency — with fallback behavior, not just a hope that the API stays up.
The model is one component in a system. Most of the engineering work is in the rest of the system.