Skip to content
Markdown

Training data I/O: sharded formats and GIL-free loading

Scope: the two decisions that determine whether an input pipeline can keep GPUs fed, taken above the level of tuning knobs. First, the on-disk layout: individual sample files versus sequentially-readable shards. Second, the concurrency model: processes, threads, or a free-threaded interpreter. Knob-level tuning of num_workers, prefetch_factor, and pinned-memory copies is in data-loading pipeline tuning; the storage fabric underneath is in storage and data platform; device-direct paths are in GPUDirect Storage.

Two primary sources. Hira et al., "Scalable and Performant Data Loading" (arXiv 2504.20067, submitted 2025-04-23, last revised 2026-03-10), describing SPDL. Aizman et al., "High Performance I/O For Large Scale Deep Learning" (arXiv 2001.01858, 2020-01-07), describing AIStore and WebDataset. Repository metadata checked 2026-07-30: facebookresearch/spdl (BSD-2-Clause, Python, 397 stars, last push 2026-07-24), NVIDIA/aistore (MIT, Go, 1,903 stars, last push 2026-07-30), webdataset/webdataset (BSD-3-Clause, Python, 3,150 stars, last push 2026-02-09). No benchmark was reproduced: this host has no GPU and no dataset at scale. The Python block is this page's own model of the two mechanisms, executed and asserted.

What it is

Two independent bottlenecks starve GPUs, and they need different fixes.

The layout problem. A dataset stored as millions of individual files makes every sample a separate request. The job becomes latency-bound rather than bandwidth-bound, and the metadata traffic can degrade a shared filesystem for every other tenant on it. The usual fix is to pack samples into large, sequentially-readable shards. WebDataset does this with plain POSIX tar files containing concatenated samples, readable as an ordinary IterableDataset. AIStore is the storage side of the same argument: a scalable, easy-to-deploy system built for the access patterns deep-learning jobs actually have, rather than the patterns general-purpose distributed filesystems were designed for. The original paper compares these against local SSDs, single-node NFS, and HDFS.

Sharding is a trade, not a free win, and the costs land in three places. Shuffling granularity drops: sequential shard reads mean you shuffle shards and then shuffle within a bounded buffer, rather than sampling uniformly over the dataset, so the effective randomness depends on shard size and buffer depth. Mutability goes away: correcting or deleting one sample in a packed tar means rewriting a shard, which makes label fixes and per-sample deletion requests expensive. Random access goes away with it, so anything that needs to fetch a specific sample by index (targeted evaluation, debugging a bad example, curriculum orders computed offline) needs a separate index or a separate copy. Pick shard size against those three, not only against read throughput.

The concurrency problem. Once reads are sequential, the bottleneck moves to preprocessing: decode, resize, augment. In CPython this is where the global interpreter lock bites, but the constraint is narrower than "threads do not work". The language reference is specific: "only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation)", the GIL "limits the performance gains of threading when it comes to CPU-bound tasks", and "threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously".1 So a thread pool already scales for the fetch stage, and it scales for any decode that happens inside a C extension that drops the lock. What does not scale is preprocessing that stays in Python bytecode, which is why the standard answer has been a process pool. Processes work but duplicate the entire pipeline per worker and pay serialization plus a copy for every batch crossing the boundary.

SPDL's observation is that when preprocessing functions release the GIL entirely, they can run concurrently in a thread pool, avoiding both costs. The library is framework-agnostic and its design follows directly from that constraint: restrict the thread pool to functions that release the GIL, and run everything else elsewhere. It also supports asynchronous execution for the data-acquisition stage, which is I/O-bound rather than CPU-bound, and avoids an extra copy when handing a buffer to NumPy or PyTorch.

The reported results: iterating ImageNet 74% faster than the PyTorch DataLoader while using 38% less CPU and 50 GB less memory, keeping a ViT-B/16 training run fed, and a further 33% throughput improvement on the free-threaded Python 3.13t build with no code changes. Note what that build is: free-threaded builds are documented as experimental as of 3.13, they disable the GIL to enable "true parallel execution of threads", and they are "not available by default".1 Treat the 33% as the headroom a GIL-free interpreter unlocks for an already thread-shaped pipeline, not as a number you get by switching interpreters on a process-pool loader.

Why use it

Both fixes have a sharp structure that the executed model below makes concrete.

Thread scaling is Amdahl's law with the GIL-held fraction as the serial part. If a fraction f of per-item work releases the GIL, the ceiling on thread-pool speedup is 1 / (1 - f) regardless of how many threads you add. At f = 0.5 that ceiling is 2x, and going from 4 threads to 32 buys 21% more throughput. This is why "just add threads" has historically failed for Python data loading, and why SPDL's design constraint is not an implementation detail but the whole idea.

Sharding converts a latency problem into a bandwidth problem, and then stops helping. One file per sample spends 97.8% of its time on per-object latency in the modelled setup. At 1,000 samples per shard that falls to 4.2%, and 10,000-sample shards land within 0.44% of the pure bandwidth floor. Past that point, larger shards buy nothing and cost shuffling granularity.

When to use it (and when not)

Shard the data whenever the dataset has more than a few hundred thousand samples, or whenever it lives on a shared parallel filesystem where a metadata storm affects other tenants. There is essentially no counter-argument at scale; the only costs are a preprocessing step and coarser shuffling.

Use a thread pool when your preprocessing genuinely releases the GIL: native decoders, C or C++ extensions, and I/O. This is where SPDL applies.

Stay with a process pool when preprocessing is Python-heavy and holds the GIL. The executed model is blunt about this: under its assumed IPC cost, processes win on raw throughput until the GIL-release fraction exceeds 0.989. The honest framing is that threads win on CPU and resident memory across a much wider range than they win on throughput, and the paper's own reported figures are a throughput gain and a 38% CPU reduction and a 50 GB memory reduction together.

Move to free-threaded Python if your pipeline is GIL-bound and your dependencies support it. The gain is largest exactly where the GIL hurt most: the model shows a pipeline at f = 0.85 gaining 225% at 16 threads, and one at f = 0.50 gaining 750%.

Do not shard if your access pattern is genuinely random single-sample lookup rather than epoch-order streaming, or if you need per-sample deletion for compliance reasons; a shard is the unit of both access and rewriting.

Architecture

flowchart LR
  ST["Storage: object store or parallel FS"] --> LAY{"On-disk layout"}
  LAY -->|"one file per sample"| SMALL["Latency-bound: one round trip per sample"]
  LAY -->|"tar/Parquet shards"| SHARD["Bandwidth-bound: one round trip per shard"]
  SHARD --> CONC{"Concurrency model"}
  CONC -->|"process pool"| PROC["No GIL, pays IPC + per-worker memory"]
  CONC -->|"thread pool"| THR["Cheap, but only scales if work releases the GIL"]
  CONC -->|"free-threaded Python"| FT["No GIL serial fraction at all"]
  PROC --> GPU["GPU fed"]
  THR --> GPU
  FT --> GPU

How to use it

Run: python3 dataio.py.

import numpy as np

# --- 1. Why a thread pool only works if the work releases the GIL --------
# If a fraction f of per-item work runs with the GIL released, the remaining
# (1 - f) serializes across threads. That is Amdahl's law with serial
# fraction (1 - f), and it is the constraint SPDL's design is built around.


def thread_speedup(f: float, n_threads: int) -> float:
    assert 0.0 <= f <= 1.0 and n_threads >= 1
    return 1.0 / ((1.0 - f) + f / n_threads)


print("[1] thread-pool speedup by GIL-release fraction f (Amdahl, serial = 1-f)")
print(f"[1] {'f':>6} | " + " ".join(f"{n:>6}t" for n in (2, 4, 8, 16, 32)) + "   ceiling")
for f in (0.50, 0.80, 0.95, 0.99, 1.00):
    row = " ".join(f"{thread_speedup(f, n):6.2f} " for n in (2, 4, 8, 16, 32))
    ceiling = float("inf") if f == 1.0 else 1.0 / (1.0 - f)
    print(f"[1] {f:6.2f} | {row}  {ceiling:8.1f}x")
assert np.isclose(thread_speedup(0.95, 10_000), 1 / 0.05, rtol=1e-2), "ceiling is 1/(1-f)"
assert thread_speedup(0.5, 32) < 2.0, "half-GIL work cannot exceed 2x no matter the thread count"
# The practical consequence: adding threads to a pipeline that holds the GIL
# is nearly free of benefit. SPDL therefore restricts the thread pool to
# functions that release the GIL and runs the rest elsewhere.
gain = thread_speedup(0.50, 32) / thread_speedup(0.50, 4) - 1.0
print(f"[1] f=0.50: going from 4 to 32 threads (8x the workers) buys "
      f"{100 * gain:.0f}% more throughput ({thread_speedup(0.50, 4):.2f}x -> "
      f"{thread_speedup(0.50, 32):.2f}x against a {1 / (1 - 0.50):.0f}x ceiling)")
assert gain < 0.25, "8x the threads must buy well under 25% when half the work holds the GIL"

# --- 2. Threads versus processes: the GIL against IPC --------------------
# A process pool sidesteps the GIL entirely but pays serialization and a copy
# for every batch that crosses the process boundary.
print()
T_ITEM = 0.010          # 10 ms of preprocessing per sample
SER_PER_ITEM = 0.0016   # serialize + IPC + deserialize per sample


def thread_throughput(f: float, n: int) -> float:
    return thread_speedup(f, n) / T_ITEM


def process_throughput(n: int) -> float:
    return n / (T_ITEM + SER_PER_ITEM)


for f in (0.60, 0.90, 0.99):
    print(f"[2] f={f:.2f}:")
    for n in (4, 16):
        th, pr = thread_throughput(f, n), process_throughput(n)
        winner = "threads" if th > pr else "processes"
        print(f"[2]   n={n:2d}: threads {th:8.1f} items/s, processes {pr:8.1f} items/s "
              f"-> {winner}")
assert thread_throughput(0.60, 16) < process_throughput(16), "low GIL release favours processes"
assert thread_throughput(0.99, 16) > process_throughput(16), "high GIL release favours threads"
# Processes also duplicate the whole pipeline per worker, which is where the
# reported memory difference comes from; the IPC term above is only the
# per-batch cost, not the resident-set cost.
grid = np.linspace(0.5, 1.0, 5001)
cross = float(grid[np.argmax([thread_throughput(f, 16) > process_throughput(16) for f in grid])])
print(f"[2] crossover at 16 workers: threads beat processes on THROUGHPUT only once "
      f"f exceeds {cross:.3f}")
assert cross > 0.95, "on throughput alone, processes win across most of the range"
print("[2] so the case for threads is not raw throughput at moderate f: it is CPU and "
      "resident memory, since a process pool duplicates the whole pipeline per worker")

# --- 3. Free-threaded Python removes the serial fraction ----------------
# On a build with the GIL disabled, the (1 - f) term stops being serial.
print()
F_REAL = 0.85           # a realistic mixed pipeline
for n in (8, 16):
    gil = thread_speedup(F_REAL, n)
    nogil = float(n)    # no serial fraction from the GIL
    print(f"[3] n={n:2d}: with GIL {gil:5.2f}x, free-threaded {nogil:5.2f}x "
          f"({100 * (nogil / gil - 1):5.1f}% more throughput, no code change)")
assert thread_speedup(F_REAL, 16) < 16.0, "the GIL caps threaded scaling below linear"
# SPDL reports a 33% throughput gain on Python 3.13t without code changes. In
# this explicit n-worker Amdahl model, no-GIL/GIL = 1 + (n-1)*h, where h is
# the GIL-held fraction. Solve it at the modelled n=16 rather than invoking an
# undefined large-n limit.
def gil_held_fraction_for_gain(gain: float, n: int) -> float:
    assert gain >= 0 and n > 1
    h = gain / (n - 1)
    assert 0.0 <= h <= 1.0
    return h


implied_h = gil_held_fraction_for_gain(0.33, 16)
implied_release = 1.0 - implied_h
assert np.isclose(16 / thread_speedup(implied_release, 16), 1.33)
print(f"[3] at n=16, a 33% no-GIL gain implies {100 * implied_h:.1f}% GIL-held work; "
      f"a pipeline at f=0.50 would instead gain {100 * (16 / thread_speedup(0.50, 16) - 1):.0f}%")
assert 16 / thread_speedup(0.50, 16) > 8.0, "GIL-bound pipelines have far more to gain"

# --- 4. Small files versus shards: the metadata storm -------------------
# Reading M samples as individual objects pays a per-object latency M times.
# Packing them into shards pays it once per shard.
print()
M, SAMPLE_B = 1_000_000, 110_000
LAT_S, BW_BPS = 0.004, 1.2e9        # per-object latency, per-stream bandwidth


def read_seconds(m: int, per_shard: int) -> float:
    assert per_shard >= 1
    shards = int(np.ceil(m / per_shard))
    return shards * (LAT_S + per_shard * SAMPLE_B / BW_BPS)


for k in (1, 64, 1_000, 10_000):
    t = read_seconds(M, k)
    label = "one file per sample" if k == 1 else f"{k:,} samples/shard"
    print(f"[4] {label:22s} -> {t / 3600:7.3f} h  "
          f"(latency share {100 * (np.ceil(M / k) * LAT_S) / t:5.1f}%)")
assert read_seconds(M, 1) > 10 * read_seconds(M, 1_000), "sharding removes the latency wall"
# At one file per sample the job is latency-bound; at a large shard size it is
# bandwidth-bound and no further sharding helps.
bw_floor = M * SAMPLE_B / BW_BPS
print(f"[4] pure-bandwidth floor = {bw_floor / 3600:.3f} h; "
      f"10,000-sample shards are within {100 * (read_seconds(M, 10_000) / bw_floor - 1):.2f}% of it")
assert read_seconds(M, 10_000) < 1.01 * bw_floor, "large shards converge to the bandwidth floor"
# Adversarial: sharding is not free. A shard is the unit of random access, so
# shuffling granularity drops from M to M/K unless a shuffle buffer is used.
for k in (1, 1_000, 10_000):
    print(f"[4] {k:>6,} samples/shard -> {int(np.ceil(M / k)):>9,} independently shufflable units")
assert np.ceil(M / 10_000) < np.ceil(M / 1_000) < M, "bigger shards mean coarser shuffling"

print("\nAll assertions passed.")

Executed output:

[1] thread-pool speedup by GIL-release fraction f (Amdahl, serial = 1-f)
[1]      f |      2t      4t      8t     16t     32t   ceiling
[1]   0.50 |   1.33    1.60    1.78    1.88    1.94        2.0x
[1]   0.80 |   1.67    2.50    3.33    4.00    4.44        5.0x
[1]   0.95 |   1.90    3.48    5.93    9.14   12.55       20.0x
[1]   0.99 |   1.98    3.88    7.48   13.91   24.43      100.0x
[1]   1.00 |   2.00    4.00    8.00   16.00   32.00        infx
[1] f=0.50: going from 4 to 32 threads (8x the workers) buys 21% more throughput (1.60x -> 1.94x against a 2x ceiling)

[2] f=0.60:
[2]   n= 4: threads    181.8 items/s, processes    344.8 items/s -> processes
[2]   n=16: threads    228.6 items/s, processes   1379.3 items/s -> processes
[2] f=0.90:
[2]   n= 4: threads    307.7 items/s, processes    344.8 items/s -> processes
[2]   n=16: threads    640.0 items/s, processes   1379.3 items/s -> processes
[2] f=0.99:
[2]   n= 4: threads    388.3 items/s, processes    344.8 items/s -> threads
[2]   n=16: threads   1391.3 items/s, processes   1379.3 items/s -> threads
[2] crossover at 16 workers: threads beat processes on THROUGHPUT only once f exceeds 0.989
[2] so the case for threads is not raw throughput at moderate f: it is CPU and resident memory, since a process pool duplicates the whole pipeline per worker

[3] n= 8: with GIL  3.90x, free-threaded  8.00x (105.0% more throughput, no code change)
[3] n=16: with GIL  4.92x, free-threaded 16.00x (225.0% more throughput, no code change)
[3] at n=16, a 33% no-GIL gain implies 2.2% GIL-held work; a pipeline at f=0.50 would instead gain 750%

[4] one file per sample    ->   1.137 h  (latency share  97.8%)
[4] 64 samples/shard       ->   0.043 h  (latency share  40.5%)
[4] 1,000 samples/shard    ->   0.027 h  (latency share   4.2%)
[4] 10,000 samples/shard   ->   0.026 h  (latency share   0.4%)
[4] pure-bandwidth floor = 0.025 h; 10,000-sample shards are within 0.44% of it
[4]      1 samples/shard -> 1,000,000 independently shufflable units
[4]  1,000 samples/shard ->     1,000 independently shufflable units
[4] 10,000 samples/shard ->       100 independently shufflable units

All assertions passed.

Three things to take from that output.

The GIL-release fraction is the parameter that decides the concurrency model. Everything else is secondary. Measure it before choosing: profile how much of your per-sample wall time is spent inside functions that release the GIL. If you cannot answer that question, you cannot predict whether threads will help.

The throughput crossover in section 2 is sensitive to the assumed IPC cost, and this model's parameters are not the paper's. At 1.6 ms of serialization per sample, processes hold the throughput lead until f is very high. SPDL reports measuring a 74% throughput win in practice, which implies its comparison saw a larger per-batch process overhead than modelled here, plus the per-worker pipeline duplication this model does not price at all. Treat section 2 as a structural argument about which forces compete, not as a prediction of your numbers.

Shard size has an optimum and then a penalty. Going from 1,000 to 10,000 samples per shard buys 0.44% of remaining headroom and costs a tenfold reduction in independently shufflable units, from 1,000 down to 100. With 100 units, shard-order shuffling alone gives poor randomness and a shuffle buffer becomes mandatory rather than optional.

How to develop with it

Make the shard builder deterministic and independently verifiable. Emit a manifest containing shard checksum, byte count, sample count, and stable sample identifiers. A validation pass must detect a truncated tar member, a checksum mismatch, a duplicate identifier, and a missing identifier before training starts. Split shards by rank and worker only after the manifest is validated, then assert that one epoch covers every identifier exactly once.

Measure the concurrency seam rather than choosing it by API familiarity. Benchmark the same decode and augmentation functions in one thread, an n-thread pool, and an n-process pool. Record throughput, CPU time, resident memory, and serialization bytes. A thread backend is eligible only when native work releases the GIL or the dependency set passes under a free-threaded interpreter.

Keep acquisition, decode, and host-to-device transfer as separate stages with bounded queues. Integration tests should force each queue to fill and drain, propagate a corrupt-sample exception to the consumer, and verify that cancellation closes file descriptors and worker tasks. An unbounded prefetch queue can disguise a slow stage until it exhausts host memory.

How to run it in production

  • Target roughly 100 MB to 1 GB per shard and verify against the bandwidth floor rather than copying a number. The model shows where the curve flattens for a given sample size and per-object latency; both vary by an order of magnitude across storage backends.
  • Always pair large shards with a shuffle buffer. Shard-order shuffling is not sample shuffling. This is the most common correctness-adjacent mistake in a sharded pipeline, and it shows up as a training curve that is subtly worse for no visible reason.
  • Split shards across workers and ranks at shard granularity, so reads stay sequential within each shard.
  • Measure CPU and resident memory, not only throughput. If a process pool is at throughput parity but using 38% more CPU, that CPU is unavailable to the rest of the node, and on a GPU node the host is often the constrained resource.
  • Test the free-threaded build before committing to it. The claim is a gain with no code changes, but dependency support for free-threaded builds is uneven and a single extension that does not support it forces a fallback.
  • Keep the acquisition stage asynchronous and separate from the CPU stage. Fetching bytes is I/O-bound and does not belong in the same pool as decode.

How to maintain it

  • Re-measure the GIL-release fraction after dependency upgrades. A decoder that switches from a native path to a pure-Python fallback silently converts a thread-scaling pipeline into a serial one.
  • Re-shard when sample size changes materially. Shard size in bytes is what matters; a change in resolution or sequence length moves the optimum.
  • Watch for the metadata storm returning. New preprocessing stages that write per-sample artifacts reintroduce the small-file problem through the back door.
  • Track the storage layer's own currency. AIStore is actively developed; WebDataset moves more slowly. Both were pushed within the last six months as of this writing.

Failure modes

  • Sharding without re-checking shuffle quality. Sequential shards bound randomness to shard order plus buffer depth. A model that regresses after a storage-format migration, with throughput up, is usually a shuffling regression rather than an I/O one.
  • Expecting a thread pool to fix Python-bytecode preprocessing. Threads scale the I/O stage and any decode that releases the lock; pure-Python transforms stay serialized until you move them into a releasing extension or a free-threaded build.
  • Adding threads to a GIL-bound pipeline. Costs nothing, achieves nothing, and looks like a tuning problem rather than a structural one.
  • Sharding without a shuffle buffer. Silently degrades training quality; no error, no obvious signal.
  • Shards so large that shuffling collapses. The extreme of the previous item, visible in the model as 100 shufflable units for a million samples.
  • Process pool memory exhaustion on many-core hosts. Each worker duplicates the pipeline; the resident-set cost scales with worker count, not with data.
  • Assuming a reported benchmark transfers. The 74%, 38%, and 50 GB figures are for a specific ImageNet pipeline on the authors' hardware.
  • Free-threaded build with an unsupported extension. Falls back silently to worse behaviour or fails at import in a way that is hard to attribute.
  • Fixing the loader when storage is the bottleneck. If the job is latency-bound on small files, no amount of loader concurrency helps; fix the layout first.

Validation limits

  • No benchmark here was reproduced. This host has no GPU, and neither the ImageNet nor the AIStore comparisons were run.
  • The model's parameters (per-item cost, IPC cost, per-object latency, bandwidth) are illustrative. Substitute measurements from your own stack before using any number.
  • The AIStore and WebDataset paper dates from 2020; the storage landscape has moved, and its specific comparisons against HDFS and NFS should be read as historical rather than current.
  • Whether SPDL's advantage holds for video pipelines, which the paper notes are especially demanding, was not evaluated here.
  • The free-threaded Python result is a single reported figure on one pipeline; the model shows the gain depends entirely on the pipeline's GIL-held fraction, so it is not a transferable constant.

References

  • Hira et al., "Scalable and Performant Data Loading", arXiv 2504.20067: https://arxiv.org/abs/2504.20067
  • SPDL repository: https://github.com/facebookresearch/spdl
  • Aizman et al., "High Performance I/O For Large Scale Deep Learning", arXiv 2001.01858: https://arxiv.org/abs/2001.01858
  • AIStore repository: https://github.com/NVIDIA/aistore
  • WebDataset repository: https://github.com/webdataset/webdataset
  • CPython threading documentation (GIL, CPU-bound versus I/O-bound threading, free-threaded builds): https://docs.python.org/3.13/library/threading.html
  • PEP 703, Making the Global Interpreter Lock Optional in CPython: https://peps.python.org/pep-0703/

Related: Data-loading pipeline tuning · Storage and data platform · GPUDirect Storage · NUMA affinity and CPU pinning · NVIDIA DALI pipeline · DeepSeek 3FS filesystem

  1. CPython threading documentation, Python 3.13. "In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation)"; the GIL "limits the performance gains of threading when it comes to CPU-bound tasks"; "threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously"; and "As of Python 3.13, experimental free-threaded builds can disable the GIL, enabling true parallel execution of threads, but this feature is not available by default (see PEP 703)". ↩↩