Iceberg updates
The community is currently actively working on View support, Multi-Table Transactions, and a new pyiceberg 0.4.0 release. Below is a list of noteworthy items that made it into Iceberg.
Subscribe for a central pulse on all videos Iceberg and Community Syncs.
- OOM fix caused by Avro decoder caching
- Multiple shuffle partitions per file
- View metadata implementation
- View support for InMemoryCatalog
- API for multi-table commits
- Catalog Transaction API is still in draft
- Flink: Split ordering based on Sequence Number
- Adding new Iceberg Events Calendar iso that people can self-sign-up for the Iceberg sync.
PyIceberg updates
PyIceberg 0.4.0 has been released, which brings:
- Support for converting Parquet schemas into Iceberg ones
- Support for reading data using FSSpec.
- Support fetching a limited number of rows to quickly peek into a dataset.
- Reduced the number of calls to the object store with PyArrow>=12.0.0.
- Speed up queries using the Iceberg metrics.
- Ability to do SQL style filters: row_filter=‘passengers >= 3’.|
- SigV4 support for the REST catalog.
- A complete makeover of the docs site.
- Support for positional deletes.
- Ability to set table properties.
- And many bugs have been fixed!
More information can be found on the project site, and the package is available on PyPI.
Iceberg in the industry
- Oracle adds cross cloud support for Iceberg from ADW
- Google BigQuery Iceberg support is now GA
- LakeFS adds support for Iceberg
- Berlin Buzzwords — Fokko Driesprong: Tip of the Iceberg (video)
- Snowflake — Ryan Blue: Getting Started With Apache Iceberg With Project (video)
- Trino — Ryan Blue: CDC patterns in Apache Iceberg (video)
- Informatica — Informatica Announces SuperPipe for Snowflake with Up to 3.5x Faster Data Integration and Replication
Blogs from the community
- Kristof Martens — Enhance Your ETL Ingestion: Unlocking the Power of the Apache Iceberg Table Format
- Jonathan Merlevede — Upserting Data using Spark and Iceberg
- Rui Li — How Bilibili Builds OLAP Data Lakehouse with Apache Iceberg
- James Malone — Apache Iceberg or Snowflake Table format?
- Amazon — Choosing an open table format for your transactional data lake on AWS
- Tabular — Intro to the Iceberg Kafka Connect sink
- Tabular — Hello, World of CDC!
- Tabular -CDC Data Gremlins
Iceberg in the news
- Silicon Angle: Cloudera creates observability tool to help enterprises manage cloud costs
- Silicon Angle: Snowflake Summit will reveal the future of data apps. Here’s our take.
- Datanami: Cloudera Sees Iceberg Everywhere
- Best Stocks: Snowflake Announces Revolutionary Advancements to its Data Cloud Platform at Snowflake Summit 2023
Keep up to date on all things iceberg
Watch for new blog posts added to the Blogs page
See the community Contribute guide to learn how to start contributing to Iceberg
Join the Apache Iceberg workspace on Slack using the invite link
Subscribe to the Apache Iceberg mailing list
