Training data I/O: sharded formats and GIL-free loading¶
Scope: the two decisions that determine whether an input pipeline can keep GPUs fed, taken above the level of tuning knobs. First, the on-disk layout: individual sample files versus sequentially-readable shards. Second, the concurrency model: processes, threads, or a free-threaded interpreter. Knob-level tuning of num_workers, prefetch_factor, and pinned-memory copies is in data-loading pipeline tuning; the storage fabric underneath is in storage and data platform; device-direct paths are in GPUDirect Storage.
Two primary sources. Hira et al., "Scalable and Performant Data Loading" (arXiv 2504.20067, submitted 2025-04-23, last revised 2026-03-10), describing SPDL. Aizman et al., "High Performance I/O For Large Scale Deep Learning" (arXiv 2001.01858, 2020-01-07), describing AIStore and WebDataset. Repository metadata checked 2026-07-30:
facebookresearch/spdl(BSD-2-Clause, Python, 397 stars, last push 2026-07-24),NVIDIA/aistore(MIT, Go, 1,903 stars, last push 2026-07-30),webdataset/webdataset(BSD-3-Clause, Python, 3,150 stars, last push 2026-02-09). No benchmark was reproduced: this host has no GPU and no dataset at scale. The Python block is this page's own model of the two mechanisms, executed and asserted.
flowchart LR
ST["Storage: object store or parallel FS"] --> LAY{"On-disk layout"}
LAY -->|"one file per sample"| SMALL["Latency-bound: one round trip per sample"]
LAY -->|"tar/Parquet shards"| SHARD["Bandwidth-bound: one round trip per shard"]
SHARD --> CONC{"Concurrency model"}
CONC -->|"process pool"| PROC["No GIL, pays IPC + per-worker memory"]
CONC -->|"thread pool"| THR["Cheap, but only scales if work releases the GIL"]
CONC -->|"free-threaded Python"| FT["No GIL serial fraction at all"]
PROC --> GPU["GPU fed"]
THR --> GPU
FT --> GPUWhat it is¶
Two independent bottlenecks starve GPUs, and they need different fixes.
The layout problem. A dataset stored as millions of individual files makes every sample a separate request. The job becomes latency-bound rather than bandwidth-bound, and the metadata traffic can degrade a shared filesystem for every other tenant on it. The fix is to pack samples into large, sequentially-readable shards. WebDataset does this with plain POSIX tar files containing concatenated samples, readable as an ordinary IterableDataset. AIStore is the storage side of the same argument: a scalable, easy-to-deploy system built for the access patterns deep-learning jobs actually have, rather than the patterns general-purpose distributed filesystems were designed for. The original paper compares these against local SSDs, single-node NFS, and HDFS.
The concurrency problem. Once reads are sequential, the bottleneck moves to preprocessing: decode, resize, augment. Python's global interpreter lock makes this hard to parallelize with threads, which is why the standard answer has been a process pool. Processes work but duplicate the entire pipeline per worker and pay serialization plus a copy for every batch crossing the boundary.
SPDL's observation is that when preprocessing functions release the GIL entirely, they can run concurrently in a thread pool, avoiding both costs. The library is framework-agnostic and its design follows directly from that constraint: restrict the thread pool to functions that release the GIL, and run everything else elsewhere. It also supports asynchronous execution for the data-acquisition stage, which is I/O-bound rather than CPU-bound, and avoids an extra copy when handing a buffer to NumPy or PyTorch.
The reported results: iterating ImageNet 74% faster than the PyTorch DataLoader while using 38% less CPU and 50 GB less memory, keeping a ViT-B/16 training run fed, and a further 33% throughput improvement on the free-threaded Python 3.13t build with no code changes.
Why the mechanisms behave the way they do¶
Both fixes have a sharp structure that the executed model below makes concrete.
Thread scaling is Amdahl's law with the GIL-held fraction as the serial part. If a fraction f of per-item work releases the GIL, the ceiling on thread-pool speedup is 1 / (1 - f) regardless of how many threads you add. At f = 0.5 that ceiling is 2x, and going from 4 threads to 32 buys 21% more throughput. This is why "just add threads" has historically failed for Python data loading, and why SPDL's design constraint is not an implementation detail but the whole idea.
Sharding converts a latency problem into a bandwidth problem, and then stops helping. One file per sample spends 97.8% of its time on per-object latency in the modelled setup. At 1,000 samples per shard that falls to 4.2%, and 10,000-sample shards land within 0.44% of the pure bandwidth floor. Past that point, larger shards buy nothing and cost shuffling granularity.
When to use what¶
Shard the data whenever the dataset has more than a few hundred thousand samples, or whenever it lives on a shared parallel filesystem where a metadata storm affects other tenants. There is essentially no counter-argument at scale; the only costs are a preprocessing step and coarser shuffling.
Use a thread pool when your preprocessing genuinely releases the GIL: native decoders, C or C++ extensions, and I/O. This is where SPDL applies.
Stay with a process pool when preprocessing is Python-heavy and holds the GIL. The executed model is blunt about this: under its assumed IPC cost, processes win on raw throughput until the GIL-release fraction exceeds 0.989. The honest framing is that threads win on CPU and resident memory across a much wider range than they win on throughput, and the paper's own reported figures are a throughput gain and a 38% CPU reduction and a 50 GB memory reduction together.
Move to free-threaded Python if your pipeline is GIL-bound and your dependencies support it. The gain is largest exactly where the GIL hurt most: the model shows a pipeline at f = 0.85 gaining 225% at 16 threads, and one at f = 0.50 gaining 750%.
Do not shard if your access pattern is genuinely random single-sample lookup rather than epoch-order streaming, or if you need per-sample deletion for compliance reasons; a shard is the unit of both access and rewriting.
How to size it¶
Run: python3 dataio.py.
import numpy as np
# --- 1. Why a thread pool only works if the work releases the GIL --------
# If a fraction f of per-item work runs with the GIL released, the remaining
# (1 - f) serializes across threads. That is Amdahl's law with serial
# fraction (1 - f), and it is the constraint SPDL's design is built around.
def thread_speedup(f: float, n_threads: int) -> float:
assert 0.0 <= f <= 1.0 and n_threads >= 1
return 1.0 / ((1.0 - f) + f / n_threads)
print("[1] thread-pool speedup by GIL-release fraction f (Amdahl, serial = 1-f)")
print(f"[1] {'f':>6} | " + " ".join(f"{n:>6}t" for n in (2, 4, 8, 16, 32)) + " ceiling")
for f in (0.50, 0.80, 0.95, 0.99, 1.00):
row = " ".join(f"{thread_speedup(f, n):6.2f} " for n in (2, 4, 8, 16, 32))
ceiling = float("inf") if f == 1.0 else 1.0 / (1.0 - f)
print(f"[1] {f:6.2f} | {row} {ceiling:8.1f}x")
assert np.isclose(thread_speedup(0.95, 10_000), 1 / 0.05, rtol=1e-2), "ceiling is 1/(1-f)"
assert thread_speedup(0.5, 32) < 2.0, "half-GIL work cannot exceed 2x no matter the thread count"
# The practical consequence: adding threads to a pipeline that holds the GIL
# is nearly free of benefit. SPDL therefore restricts the thread pool to
# functions that release the GIL and runs the rest elsewhere.
gain = thread_speedup(0.50, 32) / thread_speedup(0.50, 4) - 1.0
print(f"[1] f=0.50: going from 4 to 32 threads (8x the workers) buys "
f"{100 * gain:.0f}% more throughput ({thread_speedup(0.50, 4):.2f}x -> "
f"{thread_speedup(0.50, 32):.2f}x against a {1 / (1 - 0.50):.0f}x ceiling)")
assert gain < 0.25, "8x the threads must buy well under 25% when half the work holds the GIL"
# --- 2. Threads versus processes: the GIL against IPC --------------------
# A process pool sidesteps the GIL entirely but pays serialization and a copy
# for every batch that crosses the process boundary.
print()
T_ITEM = 0.010 # 10 ms of preprocessing per sample
SER_PER_ITEM = 0.0016 # serialize + IPC + deserialize per sample
def thread_throughput(f: float, n: int) -> float:
return thread_speedup(f, n) / T_ITEM
def process_throughput(n: int) -> float:
return n / (T_ITEM + SER_PER_ITEM)
for f in (0.60, 0.90, 0.99):
print(f"[2] f={f:.2f}:")
for n in (4, 16):
th, pr = thread_throughput(f, n), process_throughput(n)
winner = "threads" if th > pr else "processes"
print(f"[2] n={n:2d}: threads {th:8.1f} items/s, processes {pr:8.1f} items/s "
f"-> {winner}")
assert thread_throughput(0.60, 16) < process_throughput(16), "low GIL release favours processes"
assert thread_throughput(0.99, 16) > process_throughput(16), "high GIL release favours threads"
# Processes also duplicate the whole pipeline per worker, which is where the
# reported memory difference comes from; the IPC term above is only the
# per-batch cost, not the resident-set cost.
grid = np.linspace(0.5, 1.0, 5001)
cross = float(grid[np.argmax([thread_throughput(f, 16) > process_throughput(16) for f in grid])])
print(f"[2] crossover at 16 workers: threads beat processes on THROUGHPUT only once "
f"f exceeds {cross:.3f}")
assert cross > 0.95, "on throughput alone, processes win across most of the range"
print("[2] so the case for threads is not raw throughput at moderate f: it is CPU and "
"resident memory, since a process pool duplicates the whole pipeline per worker")
# --- 3. Free-threaded Python removes the serial fraction ----------------
# On a build with the GIL disabled, the (1 - f) term stops being serial.
print()
F_REAL = 0.85 # a realistic mixed pipeline
for n in (8, 16):
gil = thread_speedup(F_REAL, n)
nogil = float(n) # no serial fraction from the GIL
print(f"[3] n={n:2d}: with GIL {gil:5.2f}x, free-threaded {nogil:5.2f}x "
f"({100 * (nogil / gil - 1):5.1f}% more throughput, no code change)")
assert thread_speedup(F_REAL, 16) < 16.0, "the GIL caps threaded scaling below linear"
# SPDL reports a 33% throughput gain on Python 3.13t without code changes.
# That magnitude implies the pipeline's serial fraction was already small.
implied_f = 1.0 / (1.0 + 1.0 / 0.33) # solve nogil/gil = 1.33 in the large-n limit
print(f"[3] a reported 33% gain implies the GIL-held fraction was already small; "
f"a pipeline at f=0.50 would instead gain {100 * (16 / thread_speedup(0.50, 16) - 1):.0f}%")
assert 16 / thread_speedup(0.50, 16) > 8.0, "GIL-bound pipelines have far more to gain"
# --- 4. Small files versus shards: the metadata storm -------------------
# Reading M samples as individual objects pays a per-object latency M times.
# Packing them into shards pays it once per shard.
print()
M, SAMPLE_B = 1_000_000, 110_000
LAT_S, BW_BPS = 0.004, 1.2e9 # per-object latency, per-stream bandwidth
def read_seconds(m: int, per_shard: int) -> float:
assert per_shard >= 1
shards = int(np.ceil(m / per_shard))
return shards * (LAT_S + per_shard * SAMPLE_B / BW_BPS)
for k in (1, 64, 1_000, 10_000):
t = read_seconds(M, k)
label = "one file per sample" if k == 1 else f"{k:,} samples/shard"
print(f"[4] {label:22s} -> {t / 3600:7.3f} h "
f"(latency share {100 * (np.ceil(M / k) * LAT_S) / t:5.1f}%)")
assert read_seconds(M, 1) > 10 * read_seconds(M, 1_000), "sharding removes the latency wall"
# At one file per sample the job is latency-bound; at a large shard size it is
# bandwidth-bound and no further sharding helps.
bw_floor = M * SAMPLE_B / BW_BPS
print(f"[4] pure-bandwidth floor = {bw_floor / 3600:.3f} h; "
f"10,000-sample shards are within {100 * (read_seconds(M, 10_000) / bw_floor - 1):.2f}% of it")
assert read_seconds(M, 10_000) < 1.01 * bw_floor, "large shards converge to the bandwidth floor"
# Adversarial: sharding is not free. A shard is the unit of random access, so
# shuffling granularity drops from M to M/K unless a shuffle buffer is used.
for k in (1, 1_000, 10_000):
print(f"[4] {k:>6,} samples/shard -> {int(np.ceil(M / k)):>9,} independently shufflable units")
assert np.ceil(M / 10_000) < np.ceil(M / 1_000) < M, "bigger shards mean coarser shuffling"
print("\nAll assertions passed.")
Executed output:
[1] thread-pool speedup by GIL-release fraction f (Amdahl, serial = 1-f)
[1] f | 2t 4t 8t 16t 32t ceiling
[1] 0.50 | 1.33 1.60 1.78 1.88 1.94 2.0x
[1] 0.80 | 1.67 2.50 3.33 4.00 4.44 5.0x
[1] 0.95 | 1.90 3.48 5.93 9.14 12.55 20.0x
[1] 0.99 | 1.98 3.88 7.48 13.91 24.43 100.0x
[1] 1.00 | 2.00 4.00 8.00 16.00 32.00 infx
[1] f=0.50: going from 4 to 32 threads (8x the workers) buys 21% more throughput (1.60x -> 1.94x against a 2x ceiling)
[2] f=0.60:
[2] n= 4: threads 181.8 items/s, processes 344.8 items/s -> processes
[2] n=16: threads 228.6 items/s, processes 1379.3 items/s -> processes
[2] f=0.90:
[2] n= 4: threads 307.7 items/s, processes 344.8 items/s -> processes
[2] n=16: threads 640.0 items/s, processes 1379.3 items/s -> processes
[2] f=0.99:
[2] n= 4: threads 388.3 items/s, processes 344.8 items/s -> threads
[2] n=16: threads 1391.3 items/s, processes 1379.3 items/s -> threads
[2] crossover at 16 workers: threads beat processes on THROUGHPUT only once f exceeds 0.989
[2] so the case for threads is not raw throughput at moderate f: it is CPU and resident memory, since a process pool duplicates the whole pipeline per worker
[3] n= 8: with GIL 3.90x, free-threaded 8.00x (105.0% more throughput, no code change)
[3] n=16: with GIL 4.92x, free-threaded 16.00x (225.0% more throughput, no code change)
[3] a reported 33% gain implies the GIL-held fraction was already small; a pipeline at f=0.50 would instead gain 750%
[4] one file per sample -> 1.137 h (latency share 97.8%)
[4] 64 samples/shard -> 0.043 h (latency share 40.5%)
[4] 1,000 samples/shard -> 0.027 h (latency share 4.2%)
[4] 10,000 samples/shard -> 0.026 h (latency share 0.4%)
[4] pure-bandwidth floor = 0.025 h; 10,000-sample shards are within 0.44% of it
[4] 1 samples/shard -> 1,000,000 independently shufflable units
[4] 1,000 samples/shard -> 1,000 independently shufflable units
[4] 10,000 samples/shard -> 100 independently shufflable units
All assertions passed.
Three things to take from that output.
The GIL-release fraction is the parameter that decides the concurrency model. Everything else is secondary. Measure it before choosing: profile how much of your per-sample wall time is spent inside functions that release the GIL. If you cannot answer that question, you cannot predict whether threads will help.
The throughput crossover in section 2 is sensitive to the assumed IPC cost, and this model's parameters are not the paper's. At 1.6 ms of serialization per sample, processes hold the throughput lead until f is very high. SPDL reports measuring a 74% throughput win in practice, which implies its comparison saw a larger per-batch process overhead than modelled here, plus the per-worker pipeline duplication this model does not price at all. Treat section 2 as a structural argument about which forces compete, not as a prediction of your numbers.
Shard size has an optimum and then a penalty. Going from 1,000 to 10,000 samples per shard buys 0.44% of remaining headroom and costs a tenfold reduction in independently shufflable units, from 1,000 down to 100. With 100 units, shard-order shuffling alone gives poor randomness and a shuffle buffer becomes mandatory rather than optional.
How to run it in production¶
- Target roughly 100 MB to 1 GB per shard and verify against the bandwidth floor rather than copying a number. The model shows where the curve flattens for a given sample size and per-object latency; both vary by an order of magnitude across storage backends.
- Always pair large shards with a shuffle buffer. Shard-order shuffling is not sample shuffling. This is the most common correctness-adjacent mistake in a sharded pipeline, and it shows up as a training curve that is subtly worse for no visible reason.
- Split shards across workers and ranks at shard granularity, so reads stay sequential within each shard.
- Measure CPU and resident memory, not only throughput. If a process pool is at throughput parity but using 38% more CPU, that CPU is unavailable to the rest of the node, and on a GPU node the host is often the constrained resource.
- Test the free-threaded build before committing to it. The claim is a gain with no code changes, but dependency support for free-threaded builds is uneven and a single extension that does not support it forces a fallback.
- Keep the acquisition stage asynchronous and separate from the CPU stage. Fetching bytes is I/O-bound and does not belong in the same pool as decode.
How to maintain it¶
- Re-measure the GIL-release fraction after dependency upgrades. A decoder that switches from a native path to a pure-Python fallback silently converts a thread-scaling pipeline into a serial one.
- Re-shard when sample size changes materially. Shard size in bytes is what matters; a change in resolution or sequence length moves the optimum.
- Watch for the metadata storm returning. New preprocessing stages that write per-sample artifacts reintroduce the small-file problem through the back door.
- Track the storage layer's own currency. AIStore is actively developed; WebDataset moves more slowly. Both were pushed within the last six months as of this writing.
Failure modes¶
- Adding threads to a GIL-bound pipeline. Costs nothing, achieves nothing, and looks like a tuning problem rather than a structural one.
- Sharding without a shuffle buffer. Silently degrades training quality; no error, no obvious signal.
- Shards so large that shuffling collapses. The extreme of the previous item, visible in the model as 100 shufflable units for a million samples.
- Process pool memory exhaustion on many-core hosts. Each worker duplicates the pipeline; the resident-set cost scales with worker count, not with data.
- Assuming a reported benchmark transfers. The 74%, 38%, and 50 GB figures are for a specific ImageNet pipeline on the authors' hardware.
- Free-threaded build with an unsupported extension. Falls back silently to worse behaviour or fails at import in a way that is hard to attribute.
- Fixing the loader when storage is the bottleneck. If the job is latency-bound on small files, no amount of loader concurrency helps; fix the layout first.
Open questions and validation¶
- No benchmark here was reproduced. This host has no GPU, and neither the ImageNet nor the AIStore comparisons were run.
- The model's parameters (per-item cost, IPC cost, per-object latency, bandwidth) are illustrative. Substitute measurements from your own stack before using any number.
- The AIStore and WebDataset paper dates from 2020; the storage landscape has moved, and its specific comparisons against HDFS and NFS should be read as historical rather than current.
- Whether SPDL's advantage holds for video pipelines, which the paper notes are especially demanding, was not evaluated here.
- The free-threaded Python result is a single reported figure on one pipeline; the model shows the gain depends entirely on the pipeline's GIL-held fraction, so it is not a transferable constant.
References¶
- Hira et al., "Scalable and Performant Data Loading", arXiv 2504.20067: https://arxiv.org/abs/2504.20067
- SPDL repository: https://github.com/facebookresearch/spdl
- Aizman et al., "High Performance I/O For Large Scale Deep Learning", arXiv 2001.01858: https://arxiv.org/abs/2001.01858
- AIStore repository: https://github.com/NVIDIA/aistore
- WebDataset repository: https://github.com/webdataset/webdataset
Related: Data-loading pipeline tuning · Storage and data platform · GPUDirect Storage · NUMA affinity and CPU pinning · NVIDIA DALI pipeline · DeepSeek 3FS filesystem