a file is just bytes
One column came back in 3 milliseconds from one file and 114 milliseconds from the other.
CSV: 44.5 MB
Parquet: 13.7 MB (3.3x smaller)
read only the 'price' column:
CSV 114 ms (had to parse every byte of the file)
Parquet 3 ms (read only that column's bytes)Both files had two million logical records with fields named id, category, price, and qty. That is not quite the same as saying they contained the same information. The CSV carried characters and separators but no authoritative data types. The Parquet file carried a schema, typed values, null information, encodings, compressed pages, statistics, and offsets. A CSV reader had to infer that 53.44 was a floating point number. A Parquet reader did not.
The size and time above are an older recorded run. The generator and its exact environment were not preserved well enough to recreate 13.7 MB. The deterministic probe now in the repository produces a 48.4 MB CSV and a 21.0 MB Parquet file on CPython 3.10.14 with pandas 2.3.3, PyArrow 21.0.0, and NumPy 1.26.4. I will keep the old observation because it is what started the investigation. I will not invent a random seed or a distribution change to explain a file I can no longer regenerate.
The result still poses a useful problem. The logical rows are close enough to compare, yet the physical work is radically different. Somewhere between read_csv and read_table, one reader is asked to touch fewer bytes and perform less conversion. I first blamed the storage device. A warm read made that explanation difficult to sustain.
I had also recorded a convenient exception. In a best of nine sweep, CSV appeared to win at one hundred rows by 27 microseconds:
rows CSV ms Parquet ms winner
100 0.232 0.259 CSV
1,000 0.323 0.235 Parquet
10,000 1.118 0.311 Parquet
100,000 10.392 0.866 Parquet
1,000,000 94.463 6.677 ParquetCalling that a boundary was unjustified. A minimum selects the luckiest observation, and a 27 microsecond difference is smaller than disturbances the probe did not control. I reran the small cases as paired distributions. Each format was warmed first. The order was randomized on every pair. The table reports the median and the tenth and ninetieth percentiles across 41 reads:
rows format median ms p10 ms p90 ms runs
100 CSV 0.223 0.214 0.269 41
100 Parquet 0.177 0.172 0.216 41
1,000 CSV 0.327 0.303 0.379 41
1,000 Parquet 0.179 0.172 0.194 41
10,000 CSV 1.246 1.182 1.359 41
10,000 Parquet 0.261 0.240 0.277 41
100,000 CSV 11.031 10.358 11.758 41
100,000 Parquet 0.965 0.903 1.039 41The alleged boundary disappeared. On this rerun Parquet won even at one hundred rows. That does not prove Parquet always wins on tiny inputs. It proves the first experiment could not locate a stable crossover. Parser implementation, schema inference, file width, compression, process startup, and cache state can all move it. Below one millisecond, reporting one best sample with three decimal places made the conclusion look much firmer than the experiment was.
the bytes the parser could not skip
Open a CSV and the arrangement is obvious: one row per line, fields separated by commas, in row order.
id,category,price,qty
0,electronics,53.44,7
1,books,9.021,3Every field of row 0, then every field of row 1, and so on, row-major. It’s human-readable and it’s the reason reading one column is slow: the price values are scattered through the file, one per row, with everything else in between, so to collect them all you have to read the entire file and parse past the columns you don’t want. There’s no way to get to just the prices without walking the whole thing, because they’re not together.
I wrote out a one column CSV of two million floats, 16 MB, and timed copying the file bytes into memory against parsing them into a numeric column:
narrow CSV (16 MB, one float column):
raw read bytes only : 2.6 ms
read + parse to float: 104.0 msThe probe warmed the file first. The 2.6 milliseconds therefore describe the operating system’s page cache path on this machine, not the storage device. Sixteen megabytes divided by 0.0026 seconds is about 6.2 GB/s, a plausible memory copy rate and no evidence about the SSD. The other hundred milliseconds are spent finding record boundaries, recognizing a sign, accumulating decimal digits, applying the exponent, and producing an IEEE 754 double two million times. That run establishes that parsing dominated this warm case. It does not establish what a cold read from another device would do.
Text is not merely a bulky representation of the same machine value. The bytes 5, 3, ., 4, 4 describe a number according to a grammar. A reader has to decide whether an empty field is null or an empty string, whether 01 is an integer or an identifier, what time zone a timestamp uses, and what to do when row 900,000 contains unknown in a column previously inferred as numeric. A binary typed format moves those choices to the writer. Its reader still validates headers, decodes values, and may decompress pages. It does not repeat CSV’s type inference and decimal conversion for every field.
the same characters became different values
I reduced the schema problem to 36 bytes:
code,reading
00123,NA
00007,3.5There are at least two coherent interpretations. code may be an integer, in which case leading zeroes are formatting and its values are 123 and 7. It may be a string identifier, in which case 00123 and 00007 are the values. NA may be a conventional marker for a missing measurement or the literal two character reading produced by some instrument.
The CSV bytes contain no declaration that selects between those meanings. Pandas 2.3.3 with its defaults chose integers and a nullable looking floating point result:
dtypes={'code': 'int64', 'reading': 'float64'}
values={'code': [123, 7], 'reading': [nan, 3.5]}The leading zeroes vanished. The literal NA became a missing floating point value represented here as nan. Reading the identical bytes with two explicit choices produced another valid result:
pd.read_csv(path, dtype=str, keep_default_na=False)dtypes={'code': 'object', 'reading': 'object'}
values={'code': ['00123', '00007'], 'reading': ['NA', '3.5']}Neither result can be called the unique meaning of the CSV. The caller supplied part of the format in its options. This is why saying the CSV and Parquet file have the “same information content” is unsafe even if every printed field looks alike.
I parsed the CSV once with an explicit Arrow schema containing two string fields, wrote that table to Parquet, and read it back:
schema:
code: string
reading: string
values:
code: ['00123', '00007']
reading: ['NA', '3.5']The Parquet footer carries the physical and logical schema needed to reproduce those two string arrays. A conforming reader does not infer code from its first few values. That changes both correctness and cost. The writer performs the schema decision once. Readers can allocate buffers and select decoders before visiting every value.
A stored schema does not guarantee that the writer chose the right meaning. If the writer labels local timestamps as UTC or converts an identifier to an integer, Parquet preserves the mistake more reliably. Logical types also sit above physical primitives. A decimal may be stored in fixed length bytes with precision and scale metadata. A timestamp may be stored as an integer count with a unit. Compatibility depends on readers understanding those annotations.
Schema therefore moves ambiguity rather than abolishing it. With CSV, interpretation is often repeated at each read unless a separate schema is maintained. With Parquet, interpretation is committed into file metadata and later changes require explicit compatibility rules. That distinction becomes operational when thousands of files form one table. One file saying qty: int32 and a later file saying qty: string cannot be repaired by arranging the bytes more cleverly. The table layer must decide whether to reject, promote, cast, or expose a schema evolution.
Parquet makes the opposite choice, columnar, but my first diagram flattened one important layer. A file does not normally store the entire id column followed by the entire category column. It divides rows into row groups. Inside each row group, it stores one contiguous column chunk for each column. A column chunk contains pages. The Parquet file layout and its conceptual model specify that nesting:
file
row group 0
id column chunk [page][page]...
category column chunk [dictionary page][data page]...
price column chunk [page][page]...
qty column chunk [page][page]...
row group 1
id column chunk [page][page]...
category column chunk [dictionary page][data page]...
price column chunk [page][page]...
qty column chunk [page][page]...
footerThe row group keeps records aligned. Row 12,345 has one value in each column chunk of its group even though those values are physically separated. The column chunk is the unit a reader can select. Pages inside it are the units on which encoding and compression operate. That distinction matters later because statistics can exist at row group or page granularity, and because a dictionary encoder can give up after one data page without changing the rest of the file.
Reading price now means reading the price chunk from every selected row group. A query over four row groups makes at least four relevant ranges, not one whole file column range. It can still skip every id, category, and qty chunk. Passing the needed column names to the file reader is called projection pushdown because the selection reaches the layer that chooses byte ranges. On a wide table, this can prevent most of the file from entering memory at all.
I’d been saying “two of your fifty columns” as a figure of speech, so I built the actual fifty-column table, two million rows, and read two columns out of it and then all fifty, from both formats:
50-col table: CSV 778 MB, Parquet 547 MB
read 2 of 50 cols : Parquet 11 ms CSV 1899 ms
read all 50 cols : Parquet 200 ms CSV 4950 msTwo things in there matter more than the raw times. First, on Parquet, asking for two of fifty columns is eighteen times cheaper than asking for all fifty, 11 ms against 200, because projection pushdown reads two column blocks and never touches the other forty-eight. That ratio is the pushdown, and it grows with the width of the table: at four columns the column-read advantage was 19x, at fifty columns Parquet’s two-column read is 172 times faster than CSV’s. Second, and this is the one that surprised me, on CSV asking for two of fifty columns is only 2.6 times cheaper than reading everything, 1899 ms against 4950. I’d assumed usecols=["c0","c1"] would be fast because I only wanted two columns. It isn’t, because the parser still has to walk every byte of every row to find where columns c0 and c1 end, which means finding every comma in the row, which means parsing past all fifty fields whether you keep them or not. Row-major storage doesn’t let you ask for a column cheaply even when the library pretends to; the layout forbids it before the API ever gets a say.
The word “knows” still needed an execution path. A Parquet file begins and ends with the four bytes PAR1. Immediately before the closing magic are four little endian bytes containing the metadata length. If a file is N bytes long and that integer is M, the serialized metadata begins at N - 8 - M. This is enough information for a reader to find the footer from the tail without scanning from the front.
I wrote a smaller uncompressed file so the ranges would be easy to account for: 40,000 rows, three fixed width columns, four row groups. It was 802,501 bytes. Its final eight bytes said the metadata was 1,653 bytes, so the footer began at byte 800,840. PyArrow’s reported serialized_size agreed with the integer I decoded directly.
The column chunk offsets made the nested layout visible:
row_group=0: id@4+80075, price@80079+80075, quantity@160154+40059
row_group=1: id@200213+80075, price@280288+80075, quantity@360363+40059
row_group=2: id@400422+80075, price@480497+80075, quantity@560572+40059
row_group=3: id@600631+80075, price@680706+80075, quantity@760781+40059The @ number is the starting byte and the number after + is the chunk’s compressed size. Compression was disabled, but the metadata and page headers mean that 10,000 * 8 value bytes occupy 80,075 file bytes for each int64 or double chunk. The first row group holds all three chunks before the second row group starts. The four price chunks are separated by the other columns.
Footer metadata tells a reader what it may request. It does not guarantee that a library performs one exact seek and read. Buffering, read coalescing, remote filesystem adapters, page indexes, and threading can change the syscall pattern. I wrapped the local file object and logged the read calls PyArrow 21.0.0 made with prebuffering and worker threads disabled:
open for ['price']:
1 call, 65536 bytes, range [736965, 802501)
read ['price']:
4 calls, 320300 bytes
[80079,160154) [280288,360363)
[480497,560572) [680706,760781)
read ['id','price','quantity']:
12 calls, 800836 bytesOpening read the final 64 KiB rather than issuing separate eight byte and 1,653 byte requests. The footer fit inside that buffer. Reading only price then requested exactly the four ranges predicted by the metadata, 320,300 bytes in total. Reading all columns requested 800,836 bytes, everything between the opening magic and the footer. The projection saved about sixty percent of the file in this three column fixture. It did not save the footer read, and it did not merge the separated chunks into a fictional single price block.
That trace is one local PyArrow path, not part of the file format contract. A remote reader might coalesce nearby chunks into a larger range because paying for extra bytes can be cheaper than paying for another network request. A memory mapped reader might fault pages as they are accessed rather than calling read for each chunk. The stable fact is that the footer contains enough offset and length information to avoid unrelated chunks. The request strategy belongs to the reader.
The footer comes last because it describes offsets that are only known after column chunks have been written. The official format permits metadata to be written after the data in a single pass. Calling the format write once is still an operational shorthand rather than a prohibition on changing bytes. A program could rewrite the file. It would need to preserve a valid ordering and rebuild offsets, lengths, statistics, and the footer. Appending a few rows in place is not a cheap local edit.
The cost has moved to the writer. A CSV writer can receive one row, format its fields, append a newline, and forget the row. It does not need to know what values arrive tomorrow in order to make today’s bytes meaningful. If the process stops, the file may contain a valid prefix plus an incomplete final record that a recovery tool can inspect.
A Parquet writer cannot make the same streaming promise with constant state. Dictionary encoding needs to remember distinct values and their indexes until it emits a dictionary and data page. Statistics need running minima, maxima, and null counts. Compression needs a page of encoded input. The writer also needs column chunk offsets and lengths for the footer. It can bound memory by closing pages and row groups, but the chosen bounds affect compression, dictionary reuse, metadata count, and later pruning.
This gives row group size three roles at once:
- It bounds how much row data and encoder state the writer may buffer.
- It defines the coarsest unit that footer statistics can reject.
- It resets column chunks, dictionaries, compression streams, and metadata entries.
Those roles pull in different directions. Smaller groups can reduce writer memory and sharpen pruning. Larger groups can amortize headers and preserve more encoding and compression context. A writer library can spill or pipeline buffers, so row group size is not identical to peak process memory, but it remains a major physical boundary.
The footer also changes failure behavior. Data pages may already be present when a writer crashes, yet a normal reader cannot discover a complete valid file without final metadata and closing magic. Salvaging those pages would require reconstructing schema and chunk metadata, not merely trimming one broken line. This article does not run a crash recovery experiment, so that statement stays at the format mechanism.
In place append is awkward for the same reason. Adding a new row group after the closing footer would require writing new data and a new footer that includes old and new groups, then arranging a safe publication point. Object storage usually cannot mutate the tail of an existing object in place anyway. Table systems avoid the problem by writing another immutable Parquet file and committing metadata that includes it. The manifest race near the end is therefore a consequence of the footer layout, not an unrelated distributed systems detour.
I did not benchmark CSV and Parquet write throughput here. Encoding can save enough output bytes to offset its CPU cost, and asynchronous storage can change the result. The prediction is only that the Parquet writer must maintain more structured state before publication. A write benchmark would need memory usage, encoded bytes, CPU time, flush behavior, row group size, codec, and durability settings, not one wall clock number.
I wanted to see the layout instead of believing in it, so I regenerated the table and cracked the footer open with pyarrow.parquet.ParquetFile(path).metadata, walking every row group and summing each column’s total_compressed_size:
CSV: 48.4 MB
Parquet: 21.0 MB
row groups: 2, total rows: 2000000
per-column compressed bytes on disk (Parquet):
id 8.563 MB
category 0.504 MB
price 10.931 MB
qty 1.004 MBThe price column alone accounts for 10.9 MB, more than half the file. It contains rounded samples from uniform(1, 1000), close to two million distinct floating point values. There is little repeated structure for the chosen encoding and compressor to exploit. The four value category column occupies only 0.504 MB.
This breakdown explains the reproducible 21.0 MB file. It does not explain the historical 13.7 MB file. Saying that a different random seed created enough price repetition was an attractive story, but two million samples from the same uniform distribution should have similar aggregate compressibility. The original input is gone, so a changed generator, schema, codec, library version, or measurement mistake remains possible. The honest range is not “2.3x to 3.3x depending on noise.” The honest statement is that the current input gives 2.3x, while an older unrepeatable run recorded 3.3x.
why the same data got three times smaller
The columnar layout also explains the size. When every value in a block is the same type, all the prices are doubles, all the categories are strings, the data compresses far better than a CSV’s mixed rows, because the compressor sees a long run of similar things instead of a repeating jumble of number-comma-string-comma-number. But the bigger win is encoding, and the category column shows it starkly. It has four distinct values, electronics, books, toys, food, repeated across two million rows:
CSV stores 'category' as ~12 MB of repeated strings
Parquet: 4 distinct values -> a dictionary {0:electronics, 1:books, 2:toys, 3:food}
+ 2,000,000 two-bit codes = 0.5 MB + a tiny dictionaryCSV writes the full word “electronics” every single time it appears, twelve megabytes of the same handful of strings over and over. Parquet notices there are only four distinct values, builds a dictionary mapping each to a small integer code, and stores two million two-bit codes plus the tiny dictionary, half a megabyte instead of twelve. This is dictionary encoding, and it’s why low-cardinality columns (a few distinct values, many rows, countries, categories, status flags) nearly vanish in Parquet. Then run-length encoding squashes it further: a column that’s sorted or clumpy has long runs of the same code, and “the value 1, forty thousand times” stores as a count and a value instead of forty thousand copies. The data didn’t lose anything, you can reconstruct every original string, it just stopped storing the redundancy that CSV writes out in full.
The phrase “two million two bit codes” is close to the measured size, but it hides how those codes become bytes. With four dictionary entries, the largest index is 3, whose binary representation needs two bits. The raw lower bound for two million indexes is therefore:
2,000,000 values * 2 bits/value = 4,000,000 bits
4,000,000 bits / 8 bits/byte = 500,000 bytesThe measured column chunk was 0.504 MB. The difference from 500,000 bytes includes the four dictionary strings, page headers, encoded run headers, levels, statistics, and file metadata. The arithmetic does not prove every bit’s location, but it explains why the result lands so close to half a megabyte.
Dictionary width changes in steps. A fifth distinct value does not need 2.32 bits in the data stream. It raises the width from two bits to three because every fixed width index slot must represent values 0 through 4. The raw index budget jumps from 500,000 to 750,000 bytes for two million rows. It stays at three bits through eight dictionary entries, then becomes four bits at nine.
The data page does not write each two bit index as a separately addressable object. RLE_DICTIONARY begins with the bit width and encodes indexes through Parquet’s hybrid run length and bit packed encoding. A repeated sequence can be represented as a run length plus one value. Less repetitive indexes are packed into groups. Four categories chosen randomly do not automatically form long runs, so the half megabyte result mostly follows bit packing. Sorting the same category values would create long equal runs and give run length encoding more to remove.
Dictionary indexes are local to a column chunk. Index 1 in row group 0 need not name the same string as index 1 in row group 1. A reader cannot compare those integers across chunks without consulting each chunk’s dictionary. This locality lets a writer reset state at row group boundaries and adapt to each portion of the data. It also explains part of the cost of very small row groups: dictionaries and their headers are rebuilt repeatedly.
The dictionary page itself stores the distinct byte array values with PLAIN, which for a variable length byte array includes a length followed by that many bytes. The following data page stores dictionary indexes. That division is why the page inspector later sees PLAIN before RLE_DICTIONARY even when every data value uses the dictionary.
That 28x on category (I measured the raw text at 14.0 MB against 0.504 MB in Parquet, close enough to the 12-to-0.5 sketch above) made me want to know where the dictionary stops paying off. Four distinct values is a fairy tale. Real columns have hundreds or millions. So I built a cardinality ladder: the same two million rows, four columns, each column a different number of distinct values, and I wrote it twice, once letting Parquet use dictionaries and once forcing them off with use_dictionary=False, so I could see the counterfactual:
column distinct dict on dict off
cat4 4 0.504 MB 4.254 MB (8.4x)
cat1000 1000 2.512 MB 6.422 MB (2.6x)
near_uniq ~2M 13.502 MB 13.096 MB (0.97x, dict slightly LARGER)
rand_f64 ~2M 16.560 MB 16.002 MB (0.97x)The dictionary’s value generally falls as cardinality rises, but cardinality alone does not locate a universal inversion point. Value length, frequency distribution, page size, compression, and writer policy also matter. With four values this input gets an 8.4x win. With one thousand it gets 2.6x. With nearly unique values, the mapping plus index stream is slightly larger than plain values.
The footer reports a set of encodings used by a column chunk. Seeing both PLAIN and RLE_DICTIONARY in that set still does not prove fallback. A dictionary page stores its dictionary values with PLAIN, while the following data pages can all use dictionary indexes. Metadata merges those two distinct page roles into one set.
the page where the dictionary stopped
The missing evidence was below the metadata API. PyArrow ships the C++ Parquet headers and libraries, so I wrote a page inspector around ParquetFileReader::RowGroup, GetColumnPageReader, and PageReader::NextPage. The relevant loop is small:
auto row_group = reader->RowGroup(row_group_index);
auto pages = row_group->GetColumnPageReader(column_index);
while (auto page = pages->NextPage()) {
if (page->type() == parquet::PageType::DICTIONARY_PAGE) {
auto dictionary =
std::static_pointer_cast<parquet::DictionaryPage>(page);
print(dictionary->num_values(), dictionary->encoding());
} else {
auto data = std::static_pointer_cast<parquet::DataPage>(page);
print(data->num_values(), data->encoding());
}
}This code does not decode values. It asks the library’s page reader to parse headers and reports each page’s role and encoding. I generated two 40,000 row string columns with compression disabled, a 4 KiB dictionary size limit, 16 KiB target data pages, and 256 value write batches. The low cardinality column repeated four strings. The high cardinality column generated a unique 42 byte string for every row.
The low cardinality file had two pages:
page=0 type=DICTIONARY bytes=44 values=4 encoding=PLAIN
page=1 type=DATA_V1 bytes=10089 values=40000 encoding=RLE_DICTIONARYThat is the ambiguity the footer could not resolve. PLAIN belongs to the dictionary page. Every data value is represented by a dictionary index.
The high cardinality file exposed the transition:
page=0 type=DICTIONARY bytes=10752 values=256 encoding=PLAIN
page=1 type=DATA_V1 bytes=265 values=256 encoding=RLE_DICTIONARY
page=2 type=DATA_V1 bytes=21511 values=512 encoding=PLAIN
page=3 type=DATA_V1 bytes=21511 values=512 encoding=PLAIN
...
page=79 type=DATA_V1 bytes=13447 values=320 encoding=PLAINThe writer accumulated the first batch of 256 unique strings, emitted a dictionary containing those strings, encoded the first 256 data values as indexes, and wrote later values plain. The 10,752 byte dictionary page is larger than the configured 4 KiB limit because the writer tests the growing dictionary at batch boundaries. The option is a fallback threshold, not a promise that the emitted page will end at exactly that many bytes.
This is the event the size ladder only suggested. The column chunk legitimately mixes one dictionary encoded data page with later plain data pages. A reader must retain the dictionary while it decodes the first page, then switch decoders when the next page header says PLAIN. The Parquet encoding specification defines the encodings. The observed decision point belongs specifically to the PyArrow 21.0.0 writer and these batch and page settings.
one changed byte still looked like a number
The page inspector trusted the page bytes once their headers parsed. That left a physical question unanswered. If one bit changes in a data page, does the reader notice, or does it return a different value with the correct type?
Parquet page headers can contain a CRC, a checksum calculated from the page data. A writer calculates the checksum before storing the page. A verifying reader calculates it again from the bytes it received. Equal checksums do not prove the page came from a trusted writer, but unequal checksums prove the bytes changed.
The format’s checksumming rule uses standard CRC32 over the serialized page representation and excludes the page header itself. That boundary has two consequences. Verification can identify the ordinal of a damaged page without hashing the entire file, which is useful when a reader requests one column chunk. It does not protect header fields through that page CRC. Corruption in a page header may be caught because lengths or encodings become impossible, but that is structural validation rather than the checksum described here. Footer integrity, encryption metadata, and whole object integrity are separate boundaries again.
In PyArrow 21.0.0, write_table exposes this as write_page_checksum=True. The default is false. read_table exposes page_checksum_verification=True. Its default is also false. Writing checksums and verifying them are separate decisions.
I wrote integers 0 through 19,999 as an uncompressed, nondictionary column with 16 KiB target pages and checksums enabled. The first data page began at byte 4, immediately after the opening magic. I copied the file and toggled the lowest bit of the byte at file offset 104:
with corrupted.open("r+b") as stream:
stream.seek(data_page_offset + 100)
original_byte = stream.read(1)
stream.seek(data_page_offset + 100)
stream.write(bytes([original_byte[0] ^ 0x01]))Reading the corrupted file with verification disabled completed successfully:
flipped file byte at offset=104
unchecked read completed
changed_values=1
first changed value:
row=2 expected=2 observed=1099511627778The observed number differs from 2 by 1,099,511,627,776, which is 2^40. The bit toggle landed in one of row 2’s higher order bytes. The page framing, value count, and integer type all remained plausible, so the decoder had no structural reason to reject it. A valid int64 emerged. It was simply the wrong integer.
The same read with page verification enabled stopped before returning the array:
verified read rejected corrupted page:
could not verify page integrity,
CRC checksum verification failed for page_ordinal 0
verified read accepted the intact fileThis establishes one bit corruption path for an uncompressed data page. Compression can make some corruptions visible because a codec rejects an invalid stream, but that is not an integrity contract for all bit changes. Some altered compressed streams still decode. An uncompressed page has no codec layer to complain. A page checksum gives the reader an explicit comparison independent of the value decoder.
The checksum has limits. CRC is designed to detect accidental corruption, not a malicious modification. Anyone able to change the page can calculate a matching CRC. Cryptographic authenticity requires a keyed message authentication code or a signature, plus a trusted key and metadata binding. A page checksum also does not tell the reader how to recover. It can identify a bad page, after which the system still needs another replica, a source file, or a failed query.
CRC32 stores only 32 result bits, so different page contents can share a checksum. Under an idealized model where an accidental corruption produces an independent uniformly distributed checksum, an undetected mismatch has probability 1 / 2^32, about one in 4.29 billion. Real faults are not required to follow that model. The polynomial is chosen to detect useful classes of common transmission errors, but the number should not be turned into a universal corruption probability. The experiment establishes detection of the injected bit flip, not a measured silent corruption rate.
Checksums at other layers do not make this one redundant in every path. Memory, storage devices, filesystems, transport protocols, and object stores may each validate portions of a transfer. Their coverage begins and ends at different boundaries. A page checksum travels with the format and can be verified after the bytes have crossed those layers and before the values are accepted. Whether its CPU and space cost is worthwhile depends on the threat and failure model.
I did not benchmark checksum overhead or inject corruption into compressed pages, page headers, indexes, or the footer. The measured claim is exact: one changed payload byte passed an unchecked PyArrow read as a wrong but valid integer, and the stored page CRC caused a verified read to reject it.
There’s a second knob under the size number that has nothing to do with encoding, and I’d been treating it as one thing when it’s really two. Encoding transforms values according to their type and structure. Compression applies a general byte codec to page data after encoding. They stack.
The first codec table reported the minimum of seven warm reads. That repeats the same statistical mistake as the original small file sweep. I rewrote it as 21 rounds with the ten (codec, selection) actions shuffled in every round. The files were warmed first. These are page cache reads, and the table reports median, tenth percentile, and ninetieth percentile in milliseconds:
codec size MB selection median p10 p90
none 34.56 price 2.89 2.34 4.68
none 34.56 all 15.47 14.32 18.62
snappy 21.01 price 5.75 5.39 8.16
snappy 21.01 all 15.86 14.05 19.93
gzip 12.70 price 17.93 17.58 18.34
gzip 12.70 all 21.76 19.85 23.35
zstd 16.16 price 9.85 8.84 11.87
zstd 16.16 all 15.88 12.57 17.63
lz4 20.61 price 3.94 3.57 5.41
lz4 20.61 all 15.03 13.53 21.94The uncompressed file is the biggest at 34.56 MB and produced the lowest median single column time, 2.89 ms. Gzip produced the smallest file and the slowest median single column time, 17.93 ms. Gzip’s narrow spread also shows that low variance does not imply low latency. LZ4’s full read has a 15.03 ms median and 21.94 ms ninetieth percentile. Selecting a codec from one minimum would have hidden that tail.
The experiment changed only the codec when writing, and randomized read order to reduce drift confounding. Decompression is a credible cause of the systematic single column differences, but the timer still combines reading, decompression, page decoding, allocation, and scheduling. It does not measure codec CPU time in isolation.
There is also no winning codec independent of data and machine. Gzip made this table smaller than zstd, while zstd read faster than gzip. LZ4’s file was close to Snappy’s and its observed column read was lower. PyArrow 21.0.0 documents snappy as the default for write_table, which explains this probe’s implicit choice. It does not make Snappy the correct answer for every engine or workload. A choice needs at least the column distributions, storage cost, read frequency, available CPU, and a measured latency or throughput target. The five warm minima above are a first comparison, not a codec policy.
The dictionary is a string trick. Numbers get their own encodings, and this is where I found the most embarrassing number in the whole investigation, sitting in my own file the entire time. The id column is 0, 1, 2, ..., 2000000, perfectly sequential, and the footer reported it at 8.5 MB. There’s an encoding, DELTA_BINARY_PACKED, that stores the differences between consecutive values instead of the values, and for a sequential column every difference is 1, which packs down to almost nothing. I wrote the id column with delta encoding and compared it to plain:
id column (0..2M sequential int64):
PLAIN 16.00 MB
DELTA_BINARY_PACKED 0.040 MB (403x)Forty kilobytes. The most predictable column imaginable collapses by four hundred times when stored as deltas. The default call did not select this encoding, but the probe did not establish why, and a Parquet format version does not by itself require a writer to choose delta encoding. The controlled claim is narrower: with dictionary encoding disabled, explicitly setting column_encoding={"id": "DELTA_BINARY_PACKED"} reduced this sequential int64 column from 16.00 MB to 0.040 MB. That saving can be valuable, but it also adds an encoding decision that should be tested against the actual reader fleet.
I then made a comparison that changed two variables without noticing. The original “default” side allowed dictionary encoding, while the BYTE_STREAM_SPLIT side disabled it because PyArrow does not combine those choices for the same column. The result could not isolate byte stream splitting. I reran it with dictionaries disabled on both sides.
BYTE_STREAM_SPLIT regroups the bytes of fixed width values by byte position. For a double, all first bytes are placed together, then all second bytes, until eight streams have been formed. If signs and exponents repeat, a compressor can find structure that was separated by the remaining mantissa bytes. The corrected result for these rounded uniform values was:
price column (2M random float64, dictionary disabled on both):
codec PLAIN MB BYTE_STREAM_SPLIT MB plain/split
none 16.00 16.00 1.00x
snappy 10.40 14.10 0.74x
zstd 11.79 12.26 0.96xWithout compression both layouts contain the same 16 million value bytes plus small framing overhead. With Snappy, byte stream splitting was 3.70 MB larger. With zstd it was 0.47 MB larger. That does not mean the encoding is generally harmful. It means these values and codecs did not reward the regrouping. A claim about sensor readings or a clustered price distribution needs another input experiment. I did not run that experiment here.
There’s a third thing a column stores that I’d left out entirely: absence. Parquet represents whether an optional value exists with definition levels. For a flat optional field, level zero can mean null and level one can mean present. Those levels are encoded with the hybrid run length and bit packed representation rather than being specified as a literal one bit bitmap on disk. Repeated equal levels can cost less than one bit per row after run length encoding. Irregular patterns also require run headers and page framing. The null encoding description is deliberately expressed in levels because nested lists and structs need more states than present or absent.
CSV has no type aware null stream. In a one column CSV, a missing value can be represented by an empty record, which still requires a newline but not a comma. In a wider row it is delimited by separators. Either way, the reader must interpret the field according to a convention. I made one column progressively emptier, from all present to ninety nine percent null, and wrote it both ways:
0% null (2,000,000 values): Parquet 16.56 MB CSV 36.32 MB
50% null ( 999,488 values): Parquet 8.85 MB CSV 21.15 MB
90% null ( 200,264 values): Parquet 2.33 MB CSV 9.04 MB
99% null ( 19,749 values): Parquet 0.27 MB CSV 6.30 MBAt ninety nine percent null, the value stream contains about twenty thousand floats rather than two million. The definition level stream still describes two million logical positions. The result is 0.27 MB. The CSV still has two million records and occupies 6.30 MB. The measurement establishes the aggregate sizes. It does not let me subtract an exact “one bit per null” cost because page headers, encodings, compression, text formatting, and line endings are included in the totals.
Flat optional floats need only two logical states, but definition levels exist for a harder reason. A columnar file still has to reconstruct nested records. Consider three people whose phones field is a list of optional strings:
person A: phones = ["111", "222"]
person B: phones = []
person C: phones = null
person D: phones = [null]Flattening the phone strings into one leaf column loses distinctions. An empty list and a null list contribute no string value. The second phone for A must be attached to A rather than interpreted as person B’s first phone. The null element for D must remain an element inside an existing list.
The conventional three level Parquet list schema has an optional outer list, a repeated list group, and an optional element. Its maximum definition level for the leaf is 3. A level describes how far down that path a logical value exists:
definition 0: the outer phones field is null
definition 1: phones exists, but the list is empty
definition 2: a list element exists, but that element is null
definition 3: the string value existsThe exact maximum depends on the schema’s optional and repeated nodes. The idea is stable: the definition level preserves which enclosing structures exist even when there is no leaf value to store.
A repetition level answers the other question. Level 0 starts a new parent record. Level 1 on "222" says this leaf value continues the repeated list started by "111" for the same person. More deeply nested repeated structures can require more repetition levels.
The three streams now have separate jobs:
values: "111", "222"
definition levels: which lists and elements exist
repetition levels: which values continue the same repeated parentA reader walks levels and consumes a value only when the definition level reaches the leaf’s maximum. It can rebuild null, empty, and repeated structures without storing each original record as one contiguous blob. The cost is that projection of a nested leaf still needs its level streams, and decoding is more than copying fixed width values.
This is why reducing definition levels to “a bitmap” is safe only for the flat optional case as a mental model. Even there, the physical bytes use the hybrid run length and bit packed encoding. In nested data, the levels are small integers carrying structural states. The official Parquet null and nested encoding description gives the schema dependent cases. I did not add a nested size benchmark here. The measured null table is flat, and its results should remain scoped to that shape.
reading only the rows you want, too
Projection skips columns; the matching trick skips rows. Parquet doesn’t store a column as one giant block, it breaks the table into row groups (chunks of rows), and for each column in each row group it records the min and max value. So a query with a filter can look at the metadata and skip entire chunks without reading them:
'data.parquet' has 2 row groups; each stores per-column min/max.
filter category=='books', read 'price': 18 ms (498,977 matching rows)If you filter for price > 1000 and a row group’s recorded max price is 50, the engine knows without reading a single value that nothing in that chunk can match, and skips it. That’s predicate pushdown: pushing the “where” clause down to the storage layer so whole swaths of the file are never touched. Combine the two, read only the columns in your select, only the row groups that could match your where, and a query against a terabyte table can end up reading a few gigabytes, which is the entire reason columnar formats took over analytics. The query does less work because the file was arranged to let it.
a bound could reject but never confirm
Minimum and maximum values are not a miniature copy of the column. If a chunk has min=10 and max=20, a predicate for x = 25 can reject the chunk. A predicate for x = 15 cannot confirm that 15 exists. The chunk might contain only 10 and 20. Bounds authorize skipping when a match is impossible. Otherwise they return “maybe,” just like the Bloom filter later in the investigation.
Correctness depends on using a bound in the safe direction:
predicate x < 5, chunk min=10 -> reject
predicate x > 25, chunk max=20 -> reject
predicate x = 15, range [10,20] -> cannot reject
predicate x = 25, range [10,20] -> rejectNulls and IEEE floating point NaNs make the state richer. A null is absence according to the schema. A NaN is a present floating point value with comparison behavior unlike ordinary numbers. In particular, equality and ordered comparisons with NaN are false under the usual IEEE rules. Treating NaN as an ordinary largest or smallest number would make some pruning rules unsound.
I wrote four one column files with PyArrow 21.0.0 and inspected their row group statistics:
mixed:
values=[1.0, NaN, 3.0, null, -2.0]
has_min_max=true min=-2.0 max=3.0
null_count=1 num_values=4
all_nan:
has_min_max=false min=None max=None
null_count=0 num_values=4
all_null:
has_min_max=false min=None max=None
null_count=4 num_values=0
write_statistics=false:
statistics=NoneThe mixed chunk’s finite numbers determine the bounds. NaN is present, so it contributes to num_values, but it does not extend the finite minimum or maximum. The null contributes to null_count and not to num_values. With all NaNs there is no usable minimum or maximum. With all nulls there are no values to bound. Disabling statistics removes the structure entirely.
A reader evaluating is_nan(x) cannot use the finite [-2, 3] bounds to claim the mixed group has no match. A reader evaluating x > 100 can reject that group under the relevant floating point filter semantics because the finite maximum is 3 and NaN does not satisfy the ordered comparison. A group with no usable bounds must normally remain a candidate unless another statistic answers the predicate.
Writers may omit statistics globally or by column. Readers may decline to trust statistics produced by implementations with known ordering bugs or ambiguous logical types. Binary and string bounds also have ordering rules, and some systems truncate long bounds to cap metadata size. A truncated lower bound must round in a direction that remains no greater than every value, and an upper bound must remain no smaller than every value. Saving a few footer bytes is not allowed to create a false negative.
The timing probe counts groups whose PyArrow statistics could overlap the integer interval. It does not assume every Parquet file has statistics or that every type has useful bounds. Predicate pushdown is a negotiation between the expression, available metadata, type ordering, and reader support. When any of those pieces cannot prove impossibility, correctness requires reading more bytes.
That last sentence is truer than I first understood, because I quietly assumed the min/max pruning just works, and it doesn’t always. Min/max can only skip a row group if that group’s range actually excludes your filter, and whether it does depends on how the rows were ordered when the file was written. If your data is sorted on the filter column, each row group covers a tight, non-overlapping slice of the domain, and a point lookup lands in exactly one of them. If the data is shuffled, every row group’s min is near the global minimum and its max is near the global maximum, so every group’s range spans your filter and the min/max prunes nothing. Same values, same filter, and the pruning is either total or useless depending only on sort order. I built both to watch it: four million rows, an integer key, one hundred thousand rows per row group, forty row groups, written once sorted and once shuffled. Then I asked each file for the tiny slice k between 5,000,000 and 5,000,100, and before running the filter I walked the footer myself and counted how many row groups even could contain a match:
sorted : 40 row groups, 1 can match by min/max, 49 rows returned, 1.18 ms
shuffled : 40 row groups, 40 can match by min/max, 49 rows returned, 11.13 msOne row group could match versus forty, with the same forty nine result rows and a bit under ten times the observed wall time. On the sorted file the footer rules out thirty nine groups. On the shuffled file every group’s range is roughly [0, 10,000,000], so row group min and max cannot reject any of them for this predicate.
That is not the same as saying an unsorted file has no predicate pushdown. The reader still receives the predicate and may apply it during decoding. Other columns may have useful statistics. Bloom filters or page indexes may reject work that row group ranges cannot. The controlled result is that shuffling this particular filter column destroyed this particular min and max pruning opportunity.
I chose forty row groups by hand up there, which raised a question I’d been ignoring: where do row groups come from, and why did the original two-million-row file split into exactly two? The answer is a single default. The pyarrow writer targets about 1,048,576 rows per row group, so two million rows falls into two groups without anyone asking. That number is a policy, not a law, and it’s the pruning granularity, so I swept it on a sorted file and watched the trade:
rows/group groups file MB footer KB groups hit by a point lookup
2,000,000 1 24.09 0.6 1/1 (100.0%)
500,000 4 25.86 1.4 1/4 ( 25.0%)
100,000 20 31.81 5.2 1/20 ( 5.0%)
20,000 100 30.85 24.3 1/100 ( 1.0%)
4,000 500 29.52 117.7 1/500 ( 0.2%)Finer row groups prune harder in this sorted point lookup. At four thousand rows per group, the metadata selects one group out of five hundred. With one group, statistics can select only the entire file. The footer grows from 0.6 KB to 117.7 KB because every group repeats column metadata and statistics.
The file size is not monotonic: 24.09, 25.86, 31.81, 30.85, then 29.52 MB. My earlier explanation assigned all of that growth to lost compression context, which the measurement cannot support. A new row group also starts new column chunks, page headers, dictionaries, statistics, and encoder state. Dictionary choices can change at the boundary. Different value counts alter page packing. Compression context is one mechanism among several, and the nonmonotonic result is a warning against pretending the aggregate file size identifies one of them.
The practical compromise depends on query selectivity and object size as well as metadata overhead. A file scanned in full gains nothing from five hundred row groups. A sorted column serving narrow range filters may gain a great deal. A remote reader may prefer a chunk large enough to amortize each request. The PyArrow default used here is a library policy, not a natural unit.
the index that sits beside the pages
Row group statistics answer a coarse question: can any value in this entire chunk satisfy the predicate? A 100,000 row chunk can still contain many data pages. If only one page overlaps a narrow range, reading and decoding the other pages is avoidable in principle.
The Parquet page index moves page statistics out of individual page headers and places two index structures near the footer. A ColumnIndex stores null counts and min and max values in page order. An OffsetIndex stores the location and compressed size of each page. The first says which pages may match. The second says where their bytes live. They are separate because some readers need row locations without comparing values, and because separating indexes lets a reader fetch only the metadata it needs.
For a sorted column, page ranges can be ordered and nonoverlapping:
page 0: min=0 max=9,999
page 1: min=10,000 max=19,999
page 2: min=20,000 max=29,999
page 3: min=30,000 max=39,999A predicate for value = 23,417 can reject pages 0, 1, and 3 from the ColumnIndex, find page 2’s byte range in the OffsetIndex, and read that range. If the same values are shuffled, each page may have a minimum near zero and a maximum near 39,999. The index remains correct but no longer selective. Sorting did not make the index valid. It made the summaries useful.
Page boundaries also complicate row reconstruction. Suppose the predicate is evaluated on category, but the query returns price. The matching category page identifies a span of row numbers. The reader needs the price pages covering the same rows, even though the two columns may have different compressed page sizes and physical offsets. Offset indexes carry first row positions so those independently paged columns can be aligned.
I did not measure page index pruning in this article. PyArrow 21.0.0 can write a page index with write_page_index=True, but its write_table documentation says the index is not yet used by PyArrow’s reader. Reporting a faster PyArrow filter and attributing it to a page index would therefore be false. The mechanism explains why pages exist below row groups and what a capable reader can do with their summaries. The measured pruning results above stop at row groups.
I’d described projection and predicate pushdown as two tricks and never run one query through both, so I built a ten-column, ten-million-row table, sorted by category, and asked the one question that uses everything: SUM(price) WHERE category == 'books'. Four ways, from the dumbest to the sharpest:
10M rows, 10 cols: CSV 1793 MB, Parquet 846 MB
4746.0 ms CSV: parse all 10 columns, filter, sum
171.6 ms Parquet: read all 10 columns, filter, sum
30.3 ms Parquet: read only category + price, filter, sum
9.0 ms Parquet: projection + predicate pushdownThat’s one query getting five hundred times faster without its meaning changing at all, and the nice thing is that each step is a different saving stacking on the last. Going from CSV to reading the whole Parquet file is 28x, and that’s pure format, no parsing text into floats. Reading only the two columns the query names instead of all ten is another 5.7x, and that’s projection, the eight columns I never mention are never touched. Handing the category == 'books' filter to the reader so it skips the row groups that hold no books is another 3.4x on top, and that’s predicate pushdown, the sorted layout letting whole chunks get ruled out by their min/max before a single price is read. None of these is Parquet being fast; each is Parquet being asked less, and they multiply because they cut along independent axes, columns and rows and encoding. The 4746 and the 9.0 are the same sum over the same books, and the entire gap between them is arrangement.
the buffer was contiguous only after decoding
The file’s column chunks are convenient inputs for columnar computation, but the bytes on disk are not automatically the array a CPU sums. A dictionary encoded string page contains indexes and a dictionary. A delta encoded integer page contains a base and packed differences. A compressed page contains codec output. Definition levels are interleaved with the information needed to reconstruct nulls. The reader parses and decodes those representations into an in memory array.
For a fixed width Arrow array, the in memory representation can have a values buffer containing adjacent machine values and a separate validity bitmap. That layout is suitable for a native bulk loop. Variable width strings use an offsets buffer plus a data buffer instead. The columnar property survives the file transition, but the representation changes.
I read the price column and summed the same two million values three ways:
SUM(price) over 2,000,000 values:
pyarrow.compute : 0.22 ms sum=1001263586.21
numpy native loop : 0.22 ms sum=1001263586.21
python loop : 61.61 ms sum=1001263586.21The timing establishes a roughly 280x gap between the Python loop and these native bulk operations on this run. It does not establish a SIMD width. That would require inspecting generated instructions or collecting appropriate hardware counters. A compiled scalar loop with low interpreter overhead could also beat the Python loop by a large factor. Calling the 0.22 milliseconds “sixteen values per instruction” was more precision than the measurement contained.
I had also described NumPy as viewing the same buffer without checking whether the Parquet read returned one Arrow array. It returned a ChunkedArray, with one chunk per row group in the controlled three group probe. One NumPy array cannot alias three disjoint value buffers as one contiguous range. PyArrow rejects the demand explicitly:
chunked.to_numpy(zero_copy_only=True)
# ValueError: zero_copy_only must be False for pyarrow.ChunkedArray.to_numpyEach individual fixed width, nonnullable chunk can be viewed without a copy:
for chunk in chunked.chunks:
view = chunk.to_numpy(zero_copy_only=True)
assert view.__array_interface__["data"][0] == chunk.buffers()[1].addressThe probe compared the Arrow value buffer address with the NumPy data address for all three chunks and they matched. Calling combine_chunks() produced a new Arrow values buffer at an address different from every original chunk. NumPy then viewed that new consolidated buffer without another copy.
There are therefore three different events that the phrase “zero copy” can hide:
- Parquet pages are read and decoded into Arrow buffers. That generally creates the in memory representation.
- Several Arrow chunks may be consolidated into one new buffer. That copies values.
- NumPy may create an array view over one compatible Arrow buffer. That step can avoid a copy.
The sum probe calls combine_chunks() before timing NumPy, so the consolidation cost is outside the reduction time. The Arrow compute kernel can operate on the chunked input directly. Comparing their reduction times is useful. Claiming that both started from the identical unchanged disk buffer is not.
This connects to the native boundary investigation and the CPU and GPU loop investigation more precisely than the earlier story did. Contiguous typed buffers let a native kernel amortize dispatch and traverse predictable addresses. File layout makes it possible to fetch only the relevant column chunks. Decoding creates buffers appropriate for execution. Those advantages reinforce one another, but they are separate stages with separate possible copies.
the filter that could only say no
Row group ranges are poor summaries for a shuffled high cardinality key because the minimum and maximum can span almost the entire domain. A Bloom filter asks a narrower question: is this exact value definitely absent? It keeps a bit array and maps each inserted key to several bit positions. A lookup maps the candidate to the same positions. If any required bit is zero, the key could not have been inserted. If all are one, the key may be present, or other keys may have set the same bits. The structure has false positives but no false negatives when the hashing and implementation are correct.
The probe here implements a conventional Bloom filter with two 64 bit hashes used to derive several positions. It is not the Bloom filter stored by the Parquet specification. Parquet defines a split block Bloom filter made of 256 bit blocks, with eight 32 bit words per block and one bit selected in each word. The on disk structure lives with a column chunk, not as the one coarse sidecar per file I built. The toy still tests the probability and skip behavior, but its byte layout must not be mistaken for a Parquet implementation.
I split ten million shuffled string keys across fifty Parquet files and built one toy filter per file. Then I asked for an absent key, once by scanning every file and once by checking filters first:
10,000,000 rows across 50 files; total bloom size 11.98 MB (9.6 bits/row)
absent-key lookup, no filter : opened 50/50 files, 2432.7 ms
absent-key lookup, bloom : opened 0/50 files, 0.05 ms
speedup on the 'not here' answer: 51,622xZero files opened. The Bloom filter answered “definitely not in any of these” from twelve megabytes of bit arrays without touching a single Parquet file, in fifty microseconds against two and a half seconds. That’s the whole promise of the thing, the “definitely not” that lets a query skip a file it never opens, and it’s the same instinct as everything else in the post: keep a tiny summary arranged so you don’t have to read the big thing.
The twelve megabytes follows from the model used to size the toy filter. With n inserted keys, m bits, and an optimally chosen number of hash positions, the approximate false positive probability leads to m/n = -ln(p) / ln(2)^2. For p = 0.01, that is 9.59 bits per key. Ten million keys therefore need about 95.9 million bits, or 11.99 MB in decimal units. The allocated 11.98 MB differs through integer rounding.
The units catch an easy error. Bits per key multiplied by keys gives bits, not bytes. Dividing by eight is necessary before comparing with the 11.98 MB allocation. Every factor of ten reduction in the target probability adds about 4.79 bits per key under this idealized model.
Except I got suspicious of the 51,622x, because it’s the kind of number that’s true and misleading at the same time. I’d picked one absent key, and it happened to trip none of the fifty filters. So I probed a hundred thousand different absent keys and counted how many slipped past at least one filter and forced a wasted file-open:
false-positive check: of 100k absent keys, 39,530 slipped past >=1 filter
(39.53% of lookups still had to open a file for nothing)Forty percent. If fifty filter outcomes were independent and each had a one percent false positive probability, the chance that at least one said “maybe” would be 1 - (1 - 0.01)^50, or 39.5 percent. The measured 39.53 percent is close.
Independence is an assumption, not something that formula proves. The filters have distinct inserted keys but share a hash construction and sizing policy. Correlations could move the union result. The agreement says independence is a useful model for this probe, not a law for arbitrary filter sets.
The operational quantity is also not simply “a lookup has a false positive.” It is the number and cost of unnecessary column chunks opened after all available pruning has run. Partition pruning may reduce fifty candidates to two before Bloom filters are consulted. A positive filter on remote storage may cost a request and a page decode. A positive filter on an already cached chunk may be cheap. Choosing a filter size requires the candidate count and miss cost, not only a per filter percentage.
the manifest that two writers wanted
One file has one footer and one set of row groups. A table assembled over time has another question: which files count right now? A directory listing is not a sufficient answer. A writer may have uploaded a data file but crashed before committing it. A compaction may replace ten small files with one large file while old readers still need the previous snapshot. Two writers may both base changes on the same earlier state.
I built a local mechanism small enough to expose those decisions. The data files are immutable Parquet files. A _manifest directory holds immutable numbered JSON files. Each JSON file contains the complete list of data files visible at that version. This is a teaching probe for an immutable log, not an implementation of Iceberg or Delta Lake.
Three sequential appends produced:
commit v1: +mon.parquet rows=2 sum=30
commit v2: +tue.parquet rows=3 sum=60
commit v3: +wed.parquet rows=6 sum=210The bytes in mon.parquet do not change when Tuesday arrives. Version 2’s manifest names Monday and Tuesday. Version 3 names all three days. Reading an older version means reading its manifest and opening only the files it names:
v1: rows=2 days=['mon.parquet', 'mon.parquet']
v2: rows=3 days=['mon.parquet', 'mon.parquet', 'tue.parquet']
v3: rows=6 days=['mon.parquet', 'mon.parquet',
'tue.parquet',
'wed.parquet', 'wed.parquet', 'wed.parquet']The repeated names are values in the day column, one per row. They are not duplicate manifest entries. This toy makes snapshot reads cheap in one dimension because old data files remain immutable and old manifests remain addressable. It makes them expensive in another dimension because every manifest repeats the entire live file list. Production formats introduce more levels so a new snapshot can reuse existing manifest structures.
The first version of the probe appeared to reject a stale writer:
current = max(existing_versions)
if current != expected_version:
raise Conflict
write_json(".4.json.tmp")
os.rename(".4.json.tmp", "4.json")It checked writer A and then writer B sequentially, so writer B always observed A’s commit. That never exercised the race the prose claimed to measure.
Running the checks concurrently reveals a worse bug. Both writers can read version 3 before either publishes. Both prepare a file named 4.json. POSIX rename gives atomic replacement, which means a reader does not see a partially copied directory entry. It does not give compare and set. If 4.json already exists, a later rename can replace it. Writer A can report success, writer B can replace A’s manifest, and A’s data file becomes invisible. Atomic visibility prevented a torn JSON file while doing nothing to prevent a lost update.
The publication primitive has to mean “create version 4 only if version 4 does not exist.” On the local filesystem probe, I used link:
def publish_manifest(log, expected_version, add_file, barrier=None):
if current_version(log) != expected_version:
return False
body = build_manifest(expected_version, add_file)
temporary = unique_temporary_name(expected_version + 1)
final = log / f"{expected_version + 1}.json"
write_json_and_fsync(temporary, body)
if barrier is not None:
barrier.wait()
try:
os.link(temporary, final)
fsync_directory(log)
return True
except FileExistsError:
return False
finally:
temporary.unlink()A hard link creates another name for the already written inode. Creating the final name fails with EEXIST if another writer has created it. It does not replace the winner. This works only within a filesystem that supports the required hard link and atomic directory entry semantics. An exclusive create followed by a carefully designed write protocol is another local option, but readers must never accept a partially written final file.
I put a barrier immediately before os.link. Both threads had already observed version 3. Both had written and synced their temporary manifests. They then attempted to create 4.json concurrently:
concurrent v4: winner=A, loser=B, visible=thu_A.parquet
loser's immutable data file remains orphaned: thu_B.parquet
retry v5: both files visible, rows=8
power-loss durability was not testedThe identity of the winner is nondeterministic. This recorded run happened to choose A. The assertion only requires exactly one successful link, exactly one visible Thursday file in version 4, and the losing data file still present on disk. The loser rereads version 4, adds its file to that file list, and publishes version 5. Version 5 contains both appends.
That orphan is not an accident in the probe. Data must be written before the metadata that makes it visible. If a writer loses the commit race or crashes between those steps, its immutable data may remain unreferenced. A cleanup process cannot delete every unreferenced file immediately because another live writer may have created a file and not committed it yet. It needs an age threshold, writer coordination, or a reachability rule that respects in progress commits. Snapshot expiration and orphan cleanup are part of the table’s correctness boundary, not mere housekeeping.
The fsync calls close a different gap from the no replace link. Atomicity asks what concurrent readers and writers can observe. Durability asks what survives a crash or power loss. The probe writes the data file, syncs it, writes and syncs the temporary manifest, links the final name, then syncs the manifest directory. That ordering intends to avoid a durable manifest referring to a data file whose contents were still only in volatile cache.
I did not pull the power cord, inject filesystem faults, or remount after a crash. Filesystems and storage devices differ in the guarantees behind fsync. The result proves the concurrent publication race in a live process. It does not prove crash durability. Printing that limitation from the probe is more useful than allowing the presence of fsync to impersonate a recovery test.
The real Iceberg specification uses a richer tree. A table metadata file points to snapshots. A snapshot points to a manifest list. That list describes manifest files, and each manifest tracks data files or delete files with partition information, counts, and bounds. Reusing unchanged manifests avoids copying the entire table inventory into every new snapshot. Sequence numbers and status fields distinguish added, existing, and deleted entries.
My JSON file collapses those layers into one list. It demonstrates visibility, conflict, retry, orphan creation, and historical reads. It does not demonstrate Iceberg schema evolution, partition evolution, row level deletes, sequence number rules, or catalog integration. Calling it “the spine of Iceberg” was too broad. It is one small shape that immutable table metadata also uses.
the footer was too far away five thousand times
The manifest has another job even when there is only one writer. Planning a filter requires finding candidate data files. If the only statistics live in each Parquet footer, the planner has to open every file before deciding that most files are irrelevant.
I generated five thousand small local Parquet files, recorded each file’s minimum and maximum, and timed two planning paths. One opened every footer. The other loaded one 414 KB JSON manifest containing the copied bounds:
5000 files, planning a query needs every file's min/max:
open every footer : 200.9 ms
read one manifest : 2.93 msBoth paths were warm local reads in one process. The 69x ratio does not measure an SSD and says nothing directly about object storage latency. It measures library calls, metadata parsing, file object work, and cached byte access on this machine. The manifest path performs one parse over a compact object rather than five thousand Parquet metadata opens.
For remote storage, request count becomes another term, but multiplying five thousand by an assumed network latency would be a poor prediction. Readers issue requests concurrently, coalesce ranges, cache metadata, retry failures, and face service limits. The safe deduction is structural: five thousand independently stored footers expose up to five thousand metadata objects to locate and fetch, while a manifest can place the needed planning summaries in far fewer objects. The time saved must be measured with the actual store, client, concurrency, cache, and table shape.
A manifest is itself a file that can become too large. If every snapshot contains millions of entries in one JSON document, parsing that document becomes the new scan. Hierarchical manifests, partition summaries, manifest pruning, caching, and periodic rewriting move the boundary again. The same rule has repeated from CSV through pages to table metadata: a useful layout makes the next decision without touching bytes irrelevant to that decision.
the object store no longer behaved like the old story
The old text said S3 lacked strong read after write consistency and atomic conditional updates. That history cannot be used as a description of the current service.
Amazon S3 now documents strong read after write consistency for successful writes and deletes, including subsequent GET and LIST operations. It also documents conditional writes: If-None-Match: * can require that a key not already exist, and If-Match can require that the current object’s entity tag match the value the writer observed.
Those operations can implement pieces of the local protocol without pretending object storage has rename. A writer can put immutable data under a unique key. It can attempt to create an immutable version key only if absent. A design with one mutable metadata pointer can update that pointer only if its entity tag still matches the base version. A precondition failure means another writer changed the state and the loser must rebase or abort.
Strong visibility and a conditional write do not make a collection of object updates one transaction. The data object, manifest object, and metadata pointer are separate writes. A crash can occur between them. Readers need a rule that treats only objects reachable from a successfully committed metadata state as visible. Writers need unique names or content addressed names to avoid overwriting one another’s staged files. Cleanup still has to distinguish abandoned objects from live uncommitted work.
S3 is also one object store with specific current semantics. Another store, an S3 compatible implementation, a proxy cache, or a catalog may expose a different set of conditional operations and failure modes. Table formats separate logical snapshot rules from catalog and storage integrations partly because the commit primitive is an environmental dependency.
Iceberg’s specification requires an atomic swap of table metadata as part of commit. A catalog can provide that swap. In deployments where a storage service’s conditional operation satisfies the integration’s requirements, an external database is not automatically mandatory. The claim to test is not “object stores cannot compare and set.” It is “does this deployment provide the exact atomic update, visibility, durability, and recovery behavior the table commit assumes?”
the original column can now be predicted
The 3 millisecond column read at the beginning is still an archived observation, and the 13.7 MB input that produced it is still missing. Nothing added later repairs that provenance.
The mechanism is no longer mysterious, though. A reader starts near the tail, obtains row group and column chunk metadata, and requests the chunks that contain price. In the traced fixture, that meant four ranges totaling 320,300 bytes instead of twelve ranges totaling 800,836 bytes. Each chunk then crosses page headers, optional decompression, level decoding, and value decoding before it becomes one or more Arrow buffers. A native reduction can traverse those typed buffers without two million Python dispatches. If NumPy needs one array from several chunks, consolidation introduces a copy that the word “columnar” does not erase.
The size is predictable only after the writer choices and value distribution are known. Four repeated strings reward dictionary indexes. Forty thousand unique strings force the measured writer to switch from one dictionary encoded data page to plain pages. Sequential integers collapse under delta encoding. These uniform rounded floats did not reward byte stream splitting with Snappy or zstd. Nulls remove absent values from the value stream while definition levels preserve logical positions.
Skipping rows depends on summaries being selective. Sorted row groups let min and max reject thirty nine of forty chunks in the range probe. Shuffled groups did not. A page index can repeat that decision at finer granularity, but PyArrow 21.0.0 did not use it on reads, so no page level speedup appears in the measurements. The toy Bloom filter could reject a particular absent key without opening data, yet fifty one percent filters composed into about a 39.5 percent chance that at least one would say maybe.
At table scale, the byte arrangement becomes an agreement. A manifest says which immutable files count. The first rename probe only looked atomic because it scheduled the writers one after another. The corrected concurrent test forces both writers to race for the same version name, lets one no replace publication win, leaves the losing data file orphaned, and requires a retry. Current S3 conditional writes can supply related primitives, but the recovery rules still belong to the table design.
A file is just bytes. The useful question is which bytes must be read, decoded, copied, compared, or made durable before the next decision can be correct. The CSV required the parser to visit characters for fields the query did not return. The Parquet footer let the reader avoid those fields. The page header told it when the dictionary stopped. The manifest let a reader ignore a data file that existed but never committed. The original time difference is not a property of the filename extension. It is the accumulated consequence of those decisions.