We do not think about token budgets until something breaks - the app stops working, a sentence gets cut off midway, etc. Fun fact: token budgets are more of a system design problem :)
Token budgets matter because they are the edge condition for everything an LLM does. A model is capped by its context window. Go beyond it, and you get failures. Stay too far under it, and you are making your system less intelligent.
Token budgets are a system design problem.
What fills a context window is not random, and hence it does not need to be treated as random. It is system prompts, conversation history, retrieved documents, tool call results, and reasoning chains. Every one of these is an architectural decision.
If your RAG pipeline dumps 10 unranked chunks into the context, that is a retrieval design problem. If your conversation history grows unbounded across turns, that is a state management problem. A prompt instruction does not fix either.
System design gives you levers - chunking strategy, context window allocation, history summarization, tiered retrieval, parallelization, etc. These decisions compound, and you need a rock-solid design and data flow to build a robust AI system.
By the way, a well-designed system does more with a 32k context window than a poorly designed one with 200k - and at a fraction of the cost.
Models also behave differently near their context limit. Attention degrades, key details get dropped, and output quality falls non-linearly. You cannot prompt your way around that. You design around it.
The prompt is the last 5% - the system is the other 95%.