Tim Berglund

Kafka as a Platform: the Ecosystem from the Ground Up

Tuesday, 3:15 PM EST

Kafka has become a key data infrastructure technology, and we all have at least a vague sense that it is a messaging system, but what else is it? How can an overgrown message bus be getting this much buzz? Well, because Kafka is merely the center of a rich streaming data platform that invites detailed exploration.

In this talk, we’ll look at the entire open-source streaming platform provided by the Apache Kafka and Confluent Open Source projects. Starting with a lonely key-value pair, we’ll build up topics, partitioning, replication, and low-level Producer and Consumer APIs. We’ll group consumers into elastically scalable, fault-tolerant application clusters, then layer on more sophisticated stream processing APIs like Kafka Streams and KSQL. We’ll help teams collaborate around data formats with schema management. We’ll integrate with legacy systems without writing custom code. By the time we’re done, the open-source project we thought was Big Data’s answer to message queues will have become an enterprise-grade streaming platform, all in 90 minutes.

Managing Schemas in Kafka

Tuesday, 5:00 PM EST

On the inside, Kafka is schemaless, but there is nothing schemaless about the worlds we live in. Our languages impose type systems, and the objects in our business domains have fixed sets of properties and semantics that must be obeyed. Pretending that we can operate without competent schema management does us no good at all.

In this talk, we’ll explore our how the different parts of the open-source Kafka ecosystem help us manage schema, from KSQL’s data format opinions to the full power of the Confluent Schema Registry. We will examine the Schema Registry’s operations in some detail, how it handles schema migrations, and look at examples of client code that makes proper use of it. You’ll leave this talk seeing that schema is not just an inconvenience that must be remedied, but a key means of collaboration around an enterprise-wide streaming platform.

Four Distributed Systems Architectural Patterns

Thursday, 10:45 AM EST

Developers and architects are increasingly called upon to solve big problems, and we are able to draw on a world-class set of open source tools with which to solve them. Problems of scale are no longer consigned to the web’s largest companies, but are increasingly a part of ordinary enterprise development. At the risk of only a little hyperbole, we are all distributed systems engineers now.

In this talk, we’ll look at four distributed systems architectural patterns based on real-world systems that you can apply to solve the problems you will face in the next few years. We’ll look at the strengths and weaknesses of each architecture and develop a set of criteria for knowing when to apply each one. You will leave knowing how to work with the leading data storage, messaging, and computation tools of the day to solve the daunting problems of scale in your near future.

Streaming Data with Apache Kafka

Thursday, 1:30 PM EST

The toolset for building scalable data systems is maturing, having adapted well to our decades-old paradigm of update-in-place databases. We ingest events, we store them in high-volume OLTP databases, and we have new OLAP systems to analyze them at scale—even if the size of our operation requires us to grow to dozens or hundreds of servers in the distributed system. But something feels a little dated about the store-and-analyze paradigm, as if we are missing a new architectural insight that might more efficiently distribute the work of storing and computing the events that happen to our software. That new paradigm is stream processing.

In this workshop, we’ll learn the basics of Kafka as a messaging system, learning the core concepts of topic, producer, consumer, and broker. We’ll look at how topics are partitioned among brokers and see the simple Java APIs for getting data in and out. But more than that, we’ll look at how we can extend this scalable messaging system into a streaming data processing system—one that offers significant advantages in scalability and deployment agility, while locating computation in your data pipeline in precisely the places it belongs: in your microservices and applications, and out of costly, high-density systems.

Come to this workshop to learn how to do streaming data computation with Apache Kafka!

Streaming Data with Apache Kafka

Thursday, 3:15 PM EST

The toolset for building scalable data systems is maturing, having adapted well to our decades-old paradigm of update-in-place databases. We ingest events, we store them in high-volume OLTP databases, and we have new OLAP systems to analyze them at scale—even if the size of our operation requires us to grow to dozens or hundreds of servers in the distributed system. But something feels a little dated about the store-and-analyze paradigm, as if we are missing a new architectural insight that might more efficiently distribute the work of storing and computing the events that happen to our software. That new paradigm is stream processing.

In this workshop, we’ll learn the basics of Kafka as a messaging system, learning the core concepts of topic, producer, consumer, and broker. We’ll look at how topics are partitioned among brokers and see the simple Java APIs for getting data in and out. But more than that, we’ll look at how we can extend this scalable messaging system into a streaming data processing system—one that offers significant advantages in scalability and deployment agility, while locating computation in your data pipeline in precisely the places it belongs: in your microservices and applications, and out of costly, high-density systems.

Come to this workshop to learn how to do streaming data computation with Apache Kafka!