Kafka, Samza and the Unix Philosophy of Distributed Data
Accepted version
Peer-reviewed
Repository URI
Repository DOI
Change log
Authors
Kleppmann, Martin https://orcid.org/0000-0001-7252-6958
Kreps, J
Abstract
Apache Kafka is a scalable message broker, and Apache Samza is a stream processing framework built upon Kafka. They are widely used as infrastructure for implementing personalized online services and real-time predictive analytics. Besides providing high throughput and low latency, Kafka and Samza are designed with operational robustness and long-term maintenance of applications in mind. In this paper we explain the reasoning behind the design of Kafka and Samza, which allow complex applications to be built by composing a small number of simple primitives – replicated logs and stream operators. We draw parallels between the design of Kafka and Samza, batch processing pipelines, database architecture, and the design philosophy of Unix.
Description
Keywords
Journal Title
IEEE Data Engineering Bulletin
Conference Name
Journal ISSN
Volume Title
38
Publisher
IEEE