Iceberg

TLDR Iceberg tracks every Flink checkpoint as an immutable snapshot. That gives you time travel queries on a streaming pipeline for free, with about 30-second granularity. You can add columns to a live Iceberg table while Flink is writing to it. Old rows get NULLs for the new columns, and the pipeline keeps running without rewriting data or taking downtime. DuckDB reads the same Iceberg tables without Trino running. Three different query engines can hit the same data without copying anything. ...

TLDR I built an open-source streaming lakehouse: Kafka ingests events, Flink processes them, Iceberg stores them as tables, Trino queries them, and Superset visualizes them. One make demo command runs it all locally. Even when your data lands in Iceberg automatically (as it does with Confluent Tableflow), you still need a query engine and a visualization layer. This stack builds that full picture with open-source components. MinIO went closed-source, so I switched to SeaweedFS (thanks to Robin Moffatt’s research). And Flink’s dependency management is still a jar-shaped nightmare. ...

GitHub Repository | Duration: 2 hours | Difficulty: Beginner to Advanced Overview This hands-on workshop walks you through building a complete real-time data pipeline using Confluent Cloud. You’ll stream live cryptocurrency price data from the CoinGecko API, process it with Apache Flink SQL, and materialize it as Apache Iceberg tables using Tableflow — all queryable through DuckDB. What You’ll Learn Set up and configure a Kafka cluster on Confluent Cloud Create topics and ingest live data using HTTP Source Connector Materialize Kafka topics as Iceberg tables with Tableflow Write Flink SQL for real-time stream processing Query real-time and historical data with DuckDB ...