I was skimming through Microsoft’s SUTRADHARA paper, and here are two best practices for agent latency that you might find helpful…
One of the underrated latency improvements during a tool call is to dispatch the tool call before the provider stream ends.
A model can emit a complete tool call while it is still streaming later calls or text, so waiting for the whole message turns two independent clocks into one serial clock.
SUTRADHARA, a paper from Microsoft, does exactly this, dispatching tool calls incrementally during decoding, and it cuts end-to-end latency by 10%.
The second is treating tool schemas and results the way you would treat any oversized dataset: do not load it all up front.
Represent deferred tools with compact searchable metadata and hydrate the real schema only when selected. Cursor reported a 46.9% reduction in total agent tokens after moving to this kind of dynamic context discovery.