Itās just two more weeks until Current ā24! Organized by the fine folks over at Confluent, itās the event of the year in the data streaming and processing space. This year, Current is back to Austin,TX, on September 17-18.
As always, the agenda looks great, with two days fully packed with talks around real-time and event-driven architectures, Apache Kafka and Flink, change data capture, streaming analytics, and much more. Itās going to be really hard to choose which sessions to attend, but below are a few ones which piqued my interest and which Iāll definitely try to attend.
Are you going to be at Current too? If so, letās meet and chat on real-time data streaming, stream processing, and everything in between! If you donāt find me at the Decodable booth, you can catch me at my talk (details below), in Breakout Room 6 where Iām going to be a room host on Wednesday morning, or roaming the conference halls in between.
Atomic Dual-write Recipes with Kafka Two Phase Commit (KIP-939)
Wed Sep 18, 4:00 PM - 4:45 PM
Itās a common requirement for applications to update state within their own (typically relational) database while also sending a message to Kafka, for instance to notify other services about that data change. But without transactions spanning across the database and Kafka, doing so is prone to failure and should be avoided (remember, friends donāt let friends do dual writes). KIP-939 is promising to change this, enabling Kafka to participate in distributed two-phase transactions. This is gonna be a game changer for Kafka, and I canāt wait to learn more about the details in this session by Confluentās Artem Livshits.
Streaming Queries Without Compromise
Wed Sep 18, 4:00 PM - 4:45 PM
One of the most exciting data research papers last year was DBSP: Automatic Incremental View Maintenance for Rich Query Languages, providing a fresh look at the problem of efficient incremental view maintenance (IVM) and winning the best research paper award at the 2023 conference of Very Large Databases (VLDB). This session by Mihai Budiu and Leonid Ryzhyk of Feldera discusses the core ideas behind DBSP and should be an absolute must watch for everyone interested in IVM.
Optimizing Apache Kafka with GraalVM: Faster, Leaner, and Performant
Wed Sep 18, 3:00 PM - 3:45 PM
Who doesnāt love their Kafka brokers to start faster and consume less memory? Thanks to ahead-of-time compilation with GraalVM, this is not a dream any longer, and itās just great to see that the Kafka project has picked up this technology, providing a container image with Kafka compiled into a native binary since version 3.7.0. A while ago, I took this for a spin, and Kafka would start up in 150 ms on my laptop. So great for integration testing, for instance! Join Krishna Agarwal and Vedarth Sharma of Confluent to learn all about this cool feature.
Flinking Enrichment: Shouldn't This Be Easier?
Wed Sep 18, 2:00 PM - 2:45 PMĀ
Enriching events on a Kafka topic with contextual data is one very common use case for stream processing with Apache Kafka. Thereās multiple ways for doing so, including Flinkās DataStream API with AsyncIO, Flink SQL joins, and others. What are the pros and cons of the different approaches, when to use which one, which pitfalls and traps you should avoid? David Anderson of Confluent is going to answer all these questions, and surely more, in this talk which I think will be a great watch for all you Flink practitioners out there.
Sentiment Analysis in Action with Apache Flink: Building Your Real-time Pipeline
Tue Sep 17, 3:00 PM - 3:45 PM
Are we at peak AI yet? I donāt really know, but for sure we see more and more AI-related use cases also around data streaming, be it feeding data from operational data stores to vector databases, building retrieval-augmented generation (RAG) architectures, or real-time sentiment analysis of streaming data, as in this talk by Aivenās Olena Kutsenko. If youāre curious about how to build your own real-time API data pipeline using Apache Kafka and Flink, this is a session you should check out.
Change Data Capture & Kafka: How Slack Transitioned to CDC with Debezium & Kafka Connect
Wed Sep 18, 4:00 PM - 4:45 PM
āThe CDC pipeline slashed cost by millions and slashed latency from 24 hours to less than 10 minutes.ā Do you need any more reasons for adopting Debezium and CDC? Having worked on Debezium for several years, I am always loving to learn about success stories from people adopting change data capture, in particular when itās such large scale users like Slack, who also contributed to Debeziumās connector for Vitess. Really looking forward to this session by Slack engineers Joseph Thaidigsman and Tom Thornton.
Addressing Streaming ETL Pipelines Challenges: Delving into Flink CDC
Wed Sep 18, 5:00 PM - 5:45 PM
Flink CDC has made a mature leap forward with its version 3.0, evolving into a complete end-to-end solution for data pipelines, providing powerful solutions for (re-)snapshotting specific tables, handling schema changes, and more. I havenāt seen an awful lot of information around it, besides the initial announcement and a handful of blog posts, so I am really looking forward to this session by Xu Bangjiang of Alibaba Cloud. I might have some questions about that usage of YAML, though ;)
Enabling Flink's Cloud-Native Future: Introducing Disaggregated State in Flink 2.0
Tue Sep 17, 5:00 PM - 5:45 PM
Relying on node-local state for checkpoints and savepoints isnāt ideal when running Apache Flink in cloud environments, for instance when considering containerized Flink workloads which may be moved around compute nodes. The Flink community is currently in the process of addressing these challenges with FLIP-423, targeting Flink 2.0. Yuan Mei of Alibaba is going to provide an overview of this work in this session, which should be highly relevant to everyone running Flink in the cloud, on Kubernetes, etc.
Community Events
Besides the large number of regular break-out sessions, there will also be a variety of community events. I am looking forward to the Apache FlinkĀ® Ask Me Anything session at the Meetup Hub, with folks from Confluent, LinkedIn, and Apple being ready to answer all your questions on the popular stream processing platform. Thereāll be the announcement of the Data Streaming Awards, an unofficial 5K run/walk un-organized by my fellow Decoder Robin Moffatt, and much more.
As you can see, quite a few cool things are in store for Current ā24. I canāt wait for the community to come together and am looking forward very much to seeing you in Austin!