Apache Iceberg Table Format

Overview & Architectural Motivation

Apache Iceberg is an open table format designed for massive scale analytic datasets. Originally developed at Netflix and now a top-level Apache Software Foundation project, Iceberg was built to replace directory-based table conventions (such as Hive) that fail on cloud object storage.

In legacy data lake architectures, tables were defined simply as file paths inside directories. This caused major operational failures:

• Renaming directories in cloud object stores requires O(n) copy operations.
• Concurrent writes lead to corrupted or partially written state.
• Readers frequently see incomplete files during ongoing batch ingest.
• Changing table partitions required rewriting entire datasets.

Apache Iceberg addresses this by defining table state through an explicit metadata tree rather than physical directory paths. Every read and write transaction is executed against an immutable snapshot, bringing full serializable ACID transactions to cloud object storage.

Architecture & System Flow

Internal Mechanics & Specifications

Iceberg achieves robust performance and reliability through a multi-tiered metadata hierarchy:

Snapshot Isolation

Every mutating operation (insert, update, delete, compaction) creates a new snapshot. Readers always inspect an isolated, point-in-time view of the table. Writers commit new snapshots through atomic compare-and-swap operations at the catalog level.

Metadata Architecture

1. Table Metadata JSON: Tracks schema definitions, partition specifications, current snapshot ID, and historical snapshot log.
2. Manifest List: An Avro file containing one entry for every manifest file participating in the snapshot. It tracks partition ranges and summary statistics for each manifest.
3. Manifest Files: Avro files listing individual Parquet data and delete files, complete with lower and upper column bounds, file byte offsets, and row counts.
4. Data Files: The underlying physical Parquet files containing the actual table records.

Hidden Partitioning

In legacy systems, partition keys were physical directory columns (e.g. year=2026/month=03). Queries that failed to supply the exact directory expression forced full table scans. Iceberg introduces hidden partitioning: partition values are derived dynamically from existing fields using transform functions (such as day(event_time) or bucket(100, user_id)). Queries filter directly on event_time without knowing the underlying partition layout, and Iceberg automatically prunes unneeded partitions.

Schema Evolution via Field IDs

Iceberg tracks every column using an immutable, unique integer field ID rather than by column name or ordinal position. Renaming, reordering, adding, or deleting columns is an instantaneous metadata-only change that never breaks historical data reads.

Role in the Open Lakehouse and AI Architecture

Apache Iceberg is the foundational table standard of the Open Lakehouse.

By abstracting physical file storage behind an open metadata specification, Iceberg allows diverse processing engines (Dremio, Spark, Flink, Trino) to safely access the same tables at the same time. One engine can stream real-time events while another runs complex analytical queries, completely eliminating the need for ETL copies between storage layers.

For AI and machine learning architectures, Iceberg provides native table branching, tagging, and time travel. Data scientists can reproduce the exact training data state used to train an AI model months prior simply by passing the historic snapshot ID or timestamp in the SQL query.

Ecosystem Interoperability & Open Interfaces

Iceberg enjoys universal ecosystem support across open architecture tools:

• Plugs natively into Apache Polaris and any Iceberg REST Catalog endpoint.
• Uses Apache Parquet for high-performance physical columnar data files.
• Provides native connectors for Dremio Sonar, Apache Spark, Trino, Presto, and Apache Flink.
• Python ecosystem access via PyIceberg allows Python scripts, DuckDB instances, and ML pipelines to read and write Iceberg tables directly without requiring a JVM.

Production Engineering Considerations

Operating Iceberg tables at petabyte scale requires maintaining metadata health:

Compaction Schedules

Row-level updates and deletes generate delete files (position deletes or equality deletes). Run regular merge-on-read compaction routines to rewrite data files and eliminate delete file lookup overhead.

Snapshot Expiration

Unexpired snapshots hold references to older data files on object storage. Run scheduled snapshot expiration jobs to prune historical commits and reclaim storage space.

Manifest Optimization

Frequent small commits can produce too many manifest files. Use the Iceberg rewriteManifests procedure to consolidate manifests and accelerate query planning.