LinkedIn leverages Apache Beam to power their Notification Platform.
They built a fully managed stream processing platform on top of Apache Beam to process 4 trillion events daily across their services in real-time, involving 3000 pipelines. Here are 4 interesting use cases they leverage Apache Beam for.
-
Beam powers their anti-abuse platform to process user activity events from Kafka in near-real time, labeling abusive actions and detecting abusive users in seconds.
-
Beam helped them transition from an offline (24 to 48 hours) ML feature generation pipeline to a real-time generation happening in seconds or even milliseconds.
-
They use Beam and Samza together to generate notification content, compute the target audience, and deliver in near real-time.
They shared some really interesting details about their system, architecture, and implementation nuances in their blog; give it a read.