End-to-end tests do not work at scale, nor does staging infrastructure. Past a certain number of services, both turn into a liability, and a big one at that. It is nearly impossible to run a “company-in-a-box” environment at scale.
Staging is meant to be a smaller, calmer copy of production. In practice, it becomes its own fragile system. A shared database, shared services, and everything around them all need to be up, on the right version, and in the right state at the same time.
Running tests across services on top of that staging infrastructure is even more fragile. Test for service A now depends on service B being deployed, service C having the right data, and the staging database not being mid-migration because of someone else’s change.
One flaky service and the whole suite goes red, for a reason that has nothing to do with the code being tested.
When that happens, nobody owns the failure, and it somehow becomes the responsibility of a central team (most likely DevOps, SRE, or Platform. Yesss… been in the same situation a decade ago).
What works instead is each service owning its own tests and mocking the responses of the services it depends on. No shared staging database. No waiting on five other teams to deploy before you can run a single test suite.
This gives you a clear separation of concerns. A service is responsible for its own contract and its own correctness, nothing more. Cross-service behavior gets verified through contract tests between two teams, not through a shared environment that everyone touches and no one owns.
It also shifts accountability to the left because there is no one to blame. If a mock does not match reality, that is a contract problem between two teams, caught early by the team that owns the code.
Fewer moving parts, less shared infrastructure to babysit.
Somehow, every company struggles with, there is no end to this problem, and there is probably a billion-dollar business to be built solving it :)