If you want to fundamentally challenge an assumption and innovate, try building things from scratch.
Building something from zero takes away every assumption you would otherwise inherit from an existing solution. You naturally start focusing on, “Does this even need to be true?”
That is the real reason greenfield projects (starting from scratch) often produce radical designs. Inherited code comes with inherited constraints. A blank slate has none, so you have to justify every choice from first principles. That, makes you think.
Kafka is a good example of this. Every message queue before it - JMS, RabbitMQ, you name it - agreed on one thing without really questioning it: once a consumer reads a message, delete it. A queue is a mailbox, not an archive.
When LinkedIn built Kafka from scratch, they were not trying to build a faster queue. They asked a more fundamental question: why delete data the broker already has?
So they went in the opposite direction.
On paper, that is a worse deal for a simple use case. More disk. More moving parts for the client to manage. But it made things possible that traditional queues simply could not do: replaying old messages, running multiple consumer groups over the same stream at their own pace, and treating the log itself as the source of truth.
Apache Spark is another example. Instead of optimizing disk-based processing, it did in-memory computation. Columnar databases made a similar shift by rethinking how data should be grouped and stored.
The point is, you do not stumble into designs like these by patching an existing system. You get there by starting over and refusing to accept what has been handed to you.