Ryan Blue I’ve moved around for work quite a bit in my career, mostly when I worked for the government and was lucky enough to have professional movers. By the time I moved last, the process was familiar, with everything packed up on day 1 and hauled off on day…

by Ryan Blue Analytic databases are quietly going through an unprecedented transformation — one that will fundamentally change the industry for the better, by freeing data that’s being held hostage. This change is coming, but when is an open question. Let me explain. Databases are made up of two equally…

Iceberg updates Java: Flink sink adds custom partitioner to better distribute traffic for bucket partitioned tables (Thanks, Sergio!) Java: AWS, GCP, and Azure bundles (Thanks, Bryan!) Java: Azure FileIO (Thanks, Bryan!) Java: Delete file in job planning optimizations (Thanks, Anton!) Java: Fixed branches with empty tables (Thanks, ConeyLiu!) Rust: Merged TableMetadata (including…

by Eduard Tudenhoefner Using Airbyte with Tabular Airbyte is an open-source ELT (extract, load, transform) platform that allows you to sync data from applications, APIs, and databases. These are then immediately loaded into data warehouses, data lakes, and other destinations. …

Iceberg updates Apache Iceberg 1.3.1 was released on July 25, 2023. The 1.3.1 release addresses various issues identified in the 1.3.0 release. Core Table Metadata parser now accepts null for fields: current-snapshot-id, properties, and snapshots (#8064) Hive Fix HiveCatalog deleting metadata on failures in checking lock status (#7931) Spark Fix RewritePositionDeleteFiles failure…

by Fokko Driesprong Happy to announce that PyIceberg 0.4.0 has been released, and is packed with many new features! PyIceberg is a pure Python implementation for reading Iceberg tables into you favorite engine. This new release is a major step in the maturity of PyIceberg. You can now expect a…

by Jason Reid The growing need for a secure data lake We’re experiencing two powerful business trends that seem at odds with each other: more workers are using more data in more ways to deliver value to their employer, while at the same time, regulations and consumer expectations around data privacy, governance, and compliance are sharply increasing. It’s…

by Ryan Blue The MERGE pattern is the first design I discuss in this series that maintains data directly in a mirror table rather than applying the latest set of changes at read time using a view. This is the go-to solution for most people, and it generally works well…

by Ryan Blue The last two weeks have seen several major announcements about Apache Iceberg: Google announced that Apache Iceberg support in BigLake and BigQuery is now generally available (GA), calling Iceberg “the standard for organizations building open-format lakehouses.” Snowflake announced major performance improvements for Iceberg tables, along with future…