Ryan Blue This year at Subsurface, my talk was on my favorite developments in Iceberg in the last year. The one I’m most excited about is the addition of tags and branches to the Iceberg spec. What are tags and branches? They are named references to versions of the data…

by Ryan Blue I’m excited to announce that, as of this morning, we have opened up sign-ups for Tabular. If you can hardly sit still because you’re so eager to sign up, go create a free account! Then come back for a quick tour. It’s amazing to be at this…

Iceberg updates Flink support for inspecting metadata tables (Liwei Li) Flink write and read support for branch and tag (Amogh and Liwei Li) Flink read and write support for Avro GenericRecord (Steven Wu) Implemented branch commits for all operations (Namratha Keshavaprakash & Amogh Jahagirdar) DDL support for tags and branches (Amogh, Liwei…

Iceberg updates Added support in Spark for storage-partitioned joins. Storage partitioned joins are like bucketed joins, but more generic. (Anton Okolnychyi) Added Spark changelog readers, which is another step toward reading Iceberg tables as CDC streams. (Yufei Gu) Fixed an important NaN bug (#6517) (Russel Spitzer) Updated Arrow to automatically set tuning…

This blog will demonstrate how to load data from an Iceberg table into PyArrow or DuckDB using PyIceberg. The code is publicly available on the docker-spark-iceberg Github repository that contains a full docker-compose stack of Apache Spark with Iceberg support, MinIO as a storage backend, and a REST catalog as…

Iceberg Updates Version 1.1 was released, which added API stability guarantees, in addition to these highlights: Puffin Stats Files Branching and Tagging Table Scan Metrics Changelog Scans Spark FunctionCatalog Trino support for the REST catalog, details available here More details can be found here. A spec for “materialized views” in Iceberg was…

Author: Ryan Blue at Tabular The Apache Iceberg community just released a new version, 1.1.0. In this post, we’ll explore some of the recent highlights: Puffin Stats Files Branching and Tagging Table Scan Metrics Changelog Scans Spark FunctionCatalog API stability Iceberg 1.1 comes on the heels of the 1.0 release that added…

Catching up with all the exciting Iceberg community news from October and November 2022. Iceberg Updates Iceberg v1.1.0 released Spark: Integration to read from branches and tags Spark: added initial APIs to use Iceberg metadata for aggregate queries Spark: Support for Spark 3.0 has been dropped Flink: 1.16 support was added, 1.13…

This is a walkthrough on how to use the PyIceberg CLI. For this walkthrough, we’re going to use the docker-spark-iceberg repository that contains a full stack of Spark with Iceberg support, MinIO as a storage backend and a REST catalog as a catalog. This is a companion piece to our…