Real-time Change Stream Processing with Apache Flink

Length: 60 minutes

Room: Room 3

Abstract

Log-based change data capture (CDC) is a key component of the modern data streaming stack, used for data replication, feeding search indexes, low-latency data warehouse updates, and more. Merely taking data from A to B often isn't enough though; instead, change event streams, as for instance created using Debezium, may need to be filtered or routed based on event contents, multiple streams be joined, continuous queries be updated, etc. Enter Apache Flink: it lets you do stateful stream processing on change event feeds. Join us for this session and learn about * Implementing streaming queries on CDC events with the Flink data stream API and Flink SQL * Aggregating and enriching change data events * Different deployment options: Kafka Connect vs. Flink CDC In a demo we'll put all these open-source components into action, showing how to set up a data streaming pipeline from your operational database to a live dashboard within minutes.

Day & time

Thursday, September 07, 2:20 – 3:20 PM

Intended audience

This talk is geared towards folks (software engineers, architects, tech leads) who want to learn about processing changes in their data in real-time, using the open-source stack of Apache Kafka (for data streaming), Apache Flink (for stream processing), and Debezium (for log-based change data capture). Attendees should have some understanding of the fundamentals of Kafka.

• Gunnar Morling

  Gunnar Morling is a software engineer and open-source enthusiast by heart, currently working at Decodable on stream processing based on Apache Flink. In his prior role as a software engineer at Red Hat, he led the Debezium project, a distributed platform for change data capture. He is a Java Champion and has founded multiple open source projects such as JfrUnit, kcctl, and MapStruct. Gunnar is an avid blogger (morling.dev) and has spoken at a wide range of conferences like QCon, Java One, and Devoxx. He lives in Hamburg, Germany.

  

  @gunnarmorling