The lakehouse community spent this week deciding how change itself should work. Apache Parquet opened a formal vote to adopt versioned releases for breaking changes, borrowing a governance model Iceberg refined over years. Apache Polaris canceled a vote on its semantic model API to align with Apache Ossie, the freshly incubating semantics project that opened its dev list this week. Across Iceberg, Arrow, and Parquet, contributors debated who owns statistics, which format features deserve to survive, and how far specs should bend to serve AI/ML workloads. The connective tissue is governance: building rules for evolving open formats without breaking millions of tables.
Releases kept pace: Polaris shipped 1.6.0, Arrow Rust shipped 59.1.0, and Iceberg Rust 0.10.0 and Arrow 25.0.0 entered final voting rounds.
Apache Iceberg
The Rust implementation dominated release activity. Iceberg-rust has become the foundation for pyiceberg-core (Python bindings), DataFusion integration, and lightweight services that bypass the JVM. When this library ships a bug, the blast radius crosses ecosystems. Danny Jones and Shawn Chang opened the vote for Iceberg Rust 0.10.0 RC2 on July 1, drawing verification from L. C. Hsieh, Matt Butrovich, Neelesh Salian, Renjie Liu, and Xin Huang. Issues surfaced during verification, and by July 8 Jones returned with a vote on RC3. The willingness to cut a third candidate rather than wave through a flawed second one shows how seriously the Rust community takes its release checklist (ASF license headers, clean pyiceberg-core builds).
The week's most consequential design debate concerned table statistics. Query engines rely on statistics (distinct value counts, data distributions) to choose good execution plans. Iceberg lets engines write statistics files attached to a snapshot. Trouble starts in multi-engine deployments: Spark handles ingestion, Trino/Dremio serves queries, Impala/Flink sits in the mix. Dzeri96 raised concerns about how Iceberg handles existing statistics files; Gábor Kaszab pushed back across nine messages. Core problem: when engine A writes statistics for snapshot X and engine B later computes its own, B can overwrite A's work. One proposal would create a new snapshot whenever statistics get computed. Kaszab argued this breaks the mental model—snapshots today mark data changes, not metadata additions. Another proposal would bind multiple statistics files to one snapshot, keyed by engine. Kaszab questioned how readers match files to engines in practice (does Impala X.Y know it can read stat files written by Spark A.B?). The thread closed without resolution.
Security and access control produced a second substantial discussion. William Hyun proposed extending the REST catalog's delegated access (temporary storage credentials scoped to a table) with file-level access delegation. Today, delegation operates at table scope. If a consumer should only see a subset of partitions, administrators either over-provision access or fragment tables. Hyun's proposal uses pre-signed URLs during scan planning to make partition-scoped sharing practical without restructuring tables. Concretely, a catalog planning a scan for a restricted consumer returns signed links only for files that consumer is entitled to see. Nevin Zheng added a related thread on updating the Read Restrictions proposal to adopt the Expressions Spec and IDReference.
The primary key conversation resurfaced. Iceberg tables have no primary keys—the format grew up serving analytical scans over immutable files. Change data capture pipelines spend enormous effort reconstructing key semantics. Chandra Sekhar K described production experiments with primary-key-oriented semantics for CDC and mutable-data workloads. His team treats the primary key as table metadata and builds behavior on top: key-aware write semantics, storage organization, compaction strategy, changelog generation. He framed this as complementary to the existing constraints proposal (which limits PRIMARY KEY and UNIQUE to informational metadata). The distinction sketches a two-layer future: spec standardizes how keys are declared; engines opt into richer key-aware behavior above that line.
Geospatial work is quietly becoming an active frontier. Sunmin Lee proposed declaring row-level bounding box covering columns in Iceberg metadata, letting readers prune data files using spatial bounds. Seyed Muhammad Mahdi Hoseini followed with a separate proposal for QuadTree-inspired physical spatial partitioning. Together these sketch a spatial stack: partition data by spatial cells on write, prune by bounding boxes on read. Iceberg v3 introduced geometry and geography types; these proposals move from type support to performance engineering.
The variant type effort keeps a steady drumbeat. Variant is the shredded, binary-encoded type for semi-structured data spanning Iceberg and Parquet specs. Neelesh Salian posted notes and a recording from the July 2 Variant Sync, with action items tracked in a shared document. Anurag Mantripragada opened a related design question about column update file representation.
Spec hygiene rounded out the week. Daniel Weeks opened threads on clarifying schema JSON type string serialization and clarifying valid source types for the identity transform. Sung Yun raised a write-path gap for field-id-bound policy during schema evolution and a question about field id handling in the REST spec.
Other items: Szehon Ho called a vote to add specific-name to the UDF spec. Alexander Löser continued collation support discussion. Daniel Weeks flagged an upcoming Iceberg Terraform provider release. Tanmay Rauth proposed a table_properties_log metadata table. Renjie Liu suggested trimming CI runner time by running JDK 21 checks only on main and nightly builds.
Apache Polaris
The Polaris story of the week is a vote that did not happen. The cancellation of the vote on the semantic model API matters because it shows Polaris aligning with Apache Ossie, the freshly incubating semantics project. Polaris also shipped 1.6.0.
What to Do Now
If you use Iceberg in multi-engine deployments, watch the statistics and access delegation threads—they directly affect query performance and security. If you work with CDC pipelines, track the primary key discussion. For spatial workloads, review the bounding box and partitioning proposals. And if you maintain an Iceberg implementation, participate in the spec hygiene threads to prevent future bugs.
