Skip to content
Markdown

KV cache transfer with NIXL

Scope: moving KV cache blocks between prefill and decode workers and across memory or storage tiers with the NVIDIA Inference Xfer Library (NIXL). Covers connector direction, the NIXL v1.3.1 request lifecycle, failure policy, and production validation. It does not assume that every connector streams layer by layer or that every backend uses RDMA.

What it is

In disaggregated prefill/decode serving, the prefill (context) worker computes the prompt's KV cache, and a separate decode (generation) worker consumes it to produce tokens. The KV cache produced by prefill must be moved to whichever GPU runs decode. NIXL is the transfer library that performs that move.

NIXL is an open-source, point-to-point data-movement library with a modular plug-in backend architecture. It abstracts memory and storage behind a single API: callers register memory regions (GPU VRAM, CPU DRAM, or storage) with a NIXL agent as descriptors, then issue transfers. NIXL picks a backend from the memory types involved and the backends common to both agents. NIXL's own docs describe UCX and NVIDIA Magnum IO GPUDirect Storage (GDS) as the two headline backends, with "other file systems, block and object storage in development," but the live src/plugins/ tree has moved past that framing: it already ships libfabric, POSIX (io_uring/AIO), object storage (S3/S3-accelerate/S3-CRT), a multi-threaded GDS variant, Mooncake (marked preview), HF3FS, GPUNetIO, UCCL, and Azure Blob as real, substantial backends, not stubs. Concretely: a DRAM->VRAM transfer may use UCX, while a VRAM->parallel-filesystem transfer may use the GPUDirect Storage path; object storage runs through its own dedicated backend, not GDS.23

The book frames NIXL as the GPU-to-GPU transport for disaggregation: "Both roles will use NIXL and GPUDirect RDMA to transfer the KV cache blocks. NIXL abstracts transport for GPU-to-GPU data movement over NVLink and RDMA NICs. It also provides connectors for GPUDirect Storage so that KV cache pages can be read from (or written to) different storage tiers."1

For remote GPU memory, a connector can use one-sided RDMA. vLLM's default NixlConnector is pull-based (NixlConnector = NixlPullConnector): the decode worker issues a NIXL READ from prefill memory. NixlPushConnector is a separate alternative in which prefill writes into decode memory.9 Storage and non-RDMA plugins use their own transfer mechanisms.

Why use it

Decode is commonly constrained by KV-cache reads while prefill is compute-heavy. Disaggregation only pays when the KV handoff costs less than the scheduling and compute benefit. NIXL supplies asynchronous registered-memory and storage transfers through a common request API:

  • Zero-copy, no CPU in the path. Direct remote GPU reads/writes over the chosen transport avoid CPU copies. The book frames this as "while one GPU is transferring KV data, it can also service other forward-pass requests without waiting for the transfer to complete." vLLM's shipped NixlConnector does provide this overlap, but not through a per-layer streamed pipeline: its own source explicitly no-ops the connector API's layerwise hooks (wait_for_layer_load/save_kv_layer, commented "NixlConnector does not do layerwise saving"), moving KV at whole-request/block granularity once per engine step instead. The non-blocking behavior that keeps decode TPOT flat comes from that async, request-granularity transfer, not from a literal layer-by-layer overlap.19
  • Transport abstraction. One API spans NVLink/C2C/NVSwitch, InfiniBand, RoCE, and Ethernet, plus GPUDirect Storage to NVMe/PFS tiers. This is what lets Dynamo offload colder KV blocks to DRAM/NVMe instead of recomputing them.2
  • Lightweight control plane. Pre-exchanged descriptors mean a prefill request carries "just an ID for the target KV buffer." Dynamo uses etcd for worker discovery and leases; workers register memory handles so peers can fetch descriptors on first use.1

When to use it (and when not)

Needed:

  • You run disaggregated prefill/decode with prefill and decode on different GPUs/nodes, and KV must cross a network or NVLink fabric.
  • You want KV tiering (DRAM/NVMe/object) for prefix reuse beyond GPU HBM. NIXL's GDS/storage backends provide the read/write path.2

Not needed / skip the remote path:

  • Monolithic (colocated) serving, where prefill and decode share the GPU and KV never leaves local HBM.
  • Large prefix-cache hits, where most of the prompt's KV already lives on the decode worker. The book is explicit: "A large prefix hit ... means a lot of KV data would have to be transferred to a prefill worker and back, which is pointless. Hence, such requests are kept local."1
  • Short prompts, where remote-prefill overhead exceeds the compute saved. The decode-side router (should_offload_prefill) gates remote transfer on effective prompt length and prefill-queue depth, so transfer only happens when it helps. See Disaggregated Inference for the routing policy.

Architecture

flowchart LR
  C["Client request"] --> D["Decode worker and router"]
  D -->|"offload long prefill"| P["Prefill worker"]
  P --> K["Registered KV blocks"]
  D -->|"default vLLM connector: NIXL READ"| K
  K -->|"KV arrives in decode allocation"| D
  P -->|"optional push connector: NIXL WRITE"| D
  K -.->|"UCX for registered memory"| M["GPU or host memory"]
  K -.->|"GDS or storage plugin"| S["Filesystem or object tier"]
  D --> G["Continuous decode"]

The arrows distinguish connector direction from request flow. The default vLLM path pulls KV from prefill; a push connector reverses the data operation. Backend selection depends on descriptor memory types and the plugins available at both agents.

How to use it

Transfer shape: coalesce, overlap, layout-transform

Move KV in payloads large enough to amortize RDMA setup. The book's guidance: "coalesce multiple PagedAttention blocks into ~128-token payloads before RDMA (note: vLLM defaults to 16 tokens per block on CUDA)." Use pre-registered peer memory with large pinned windows to minimize re-registration churn, and overlap transfer with compute so decode never stalls.1

When prefill and decode use different tensor-parallel layouts, insert a layout-transform kernel on the receiver side, after the NIXL read and before the KV is used, to realign each block to the decode kernel's expected layout. The book notes this transform "is latency-insignificant compared to network transfer and avoids re-prefill."1

vLLM: NixlConnector

vLLM's disaggregated prefilling uses NixlConnector for fully asynchronous KV send/receive. Configure it through --kv-transfer-config; the prefill instance is the producer, the decode instance the consumer.4

# Prefill (producer)
CUDA_VISIBLE_DEVICES=0 \
VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
vllm serve Qwen/Qwen3-0.6B \
  --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail"}'

# Decode (consumer)
CUDA_VISIBLE_DEVICES=1 \
VLLM_NIXL_SIDE_CHANNEL_PORT=5601 \
vllm serve Qwen/Qwen3-0.6B \
  --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"}'

Notes from the vLLM docs: kv_role takes kv_producer or kv_consumer (kv_both is deprecated). VLLM_NIXL_SIDE_CHANNEL_PORT (default 5600) carries the descriptor handshake and must be unique per worker on a host; set VLLM_NIXL_SIDE_CHANNEL_HOST when instances span machines. Transport is tuned through UCX env vars UCX_TLS and UCX_NET_DEVICES (e.g. mlx5_0:1,mlx5_1:1). A non-default fabric is selected via kv_connector_extra_config.backends (e.g. ["LIBFABRIC"]).4

TensorRT-LLM / Dynamo: cache_transceiver_config

In Dynamo's TensorRT-LLM backend, the KV transfer backend is set on cache_transceiver_config. Valid backend values are DEFAULT, UCX, NIXL, and MPI; NIXL is the default. Set max_tokens_in_buffer to at least the maximum input sequence length across requests for best performance.6

cache_transceiver_config:
  backend: NIXL
  max_tokens_in_buffer: 8192

The TRT-LLM container also exposes env-var switches: a NIXL-built container sets TRTLLM_USE_NIXL_KVCACHE=1; to fall back to UCX, unset it and export TRTLLM_USE_UCX_KVCACHE=1. When NIXL is active, its underlying transport is chosen via TRTLLM_NIXL_KVCACHE_BACKEND (UCX default, or LIBFABRIC on v0.16.0+).76

How to develop with it

NIXL v1.3.1 core API

Most operators only touch a connector layer. Code that calls NIXL directly should pin the public C++ header to v1.3.1; the API remains fast-moving.5

  1. Backend creation. An agent enumerates available plugins with getAvailPlugins() and instantiates one with createBackend(type, params, backend_handle) (e.g. "UCX", "GDS", "LIBFABRIC"). A single agent can hold several backends at once; it picks one per transfer based on the memory types involved and what both peers support.
  2. Memory registration. registerMem(descs) registers a descriptor list (GPU VRAM, CPU DRAM, or storage regions) with the agent so backends can reference it; deregisterMem(descs) releases it. Registration makes a buffer remotely addressable; it is not itself a transfer.
  3. Metadata exchange. getLocalMD()/getLocalPartialMD() produce a metadata blob describing an agent's registered memory; a peer consumes it with loadRemoteMD(). sendLocalMD()/fetchRemoteMD() push/pull that blob over NIXL's own side channel instead of requiring the caller to move the bytes manually, and invalidateRemoteMD()/invalidateLocalMD() tear the mapping down on teardown or peer departure.
  4. Transfer request lifecycle. createXferReq(op, local_descs, remote_descs, remote_agent, req_handle) builds a request for a READ or WRITE operation without moving any data; postXferReq(req_handle) submits it to the backend; getXferStatus(req_handle) polls completion (in-progress, done, or error); releaseXferReq(req_handle) frees the handle once the caller is done with it, and must be called for every request or handles leak.
  5. Notification. A notification message may be attached through extra_params when a request is created or posted. genNotif(remote_agent, msg, ...) is a standalone control notification not bound to a transfer; getNotifs() drains received notifications.

The ownership lifecycle is createXferReq -> postXferReq -> getXferStatus -> releaseXferReq. Release every request handle. If a request is still active, releaseXferReq may cancel it or return an error when it cannot be aborted.5

This runnable standard-library model validates a strict application wrapper around that lifecycle. The wrapper deliberately forbids releasing an active transfer; NIXL itself may attempt cancellation.

from dataclasses import dataclass
from enum import Enum, auto


class State(Enum):
    CREATED = auto()
    POSTED = auto()
    DONE = auto()
    ERROR = auto()
    RELEASED = auto()


@dataclass
class Request:
    state: State = State.CREATED

    def post(self):
        if self.state is not State.CREATED:
            raise RuntimeError("post requires a newly created request")
        self.state = State.POSTED

    def observe(self, status):
        if self.state is not State.POSTED:
            raise RuntimeError("status requires a posted request")
        if status == "NIXL_IN_PROG":
            return
        self.state = State.DONE if status == "NIXL_SUCCESS" else State.ERROR

    def release(self):
        if self.state not in {State.DONE, State.ERROR}:
            raise RuntimeError("wrapper refuses to release an active request")
        self.state = State.RELEASED


req = Request()
req.post()
req.observe("NIXL_IN_PROG")
assert req.state is State.POSTED
req.observe("NIXL_SUCCESS")
req.release()
assert req.state is State.RELEASED

for invalid in (Request(), Request(State.POSTED)):
    try:
        invalid.release()
    except RuntimeError:
        pass
    else:
        raise AssertionError("early release was not rejected")

print("NIXL request lifecycle: all asserts passed")

Executed output:

NIXL request lifecycle: all asserts passed

How to run it in production

Failure policy, cleanup, and metrics

vLLM's NixlConnector exposes kv_load_failure_policy values fail and recompute. Recompute preserves request semantics by regenerating missing KV locally, but consumes decode capacity and increases latency; it is an availability-versus-tail-latency policy, not a correctness trade.4

The connector reports two kinds of observability: periodic log-line summaries (transfer counts, latency percentiles, throughput, descriptor counts per interval) and Prometheus histograms/counters covering transfer duration, posting time, bytes transferred, descriptor counts, failed transfers, and expired requests.4 Alert on the failed-transfer and expired-request counters specifically; a rising rate there is the same silent-degradation signature as any other RDMA fallback in this stack.

Bidirectional, multi-turn transfer is a distinct optimization from the single-turn pull path described above: across conversation turns, the decode worker already holds the KV cache for previously generated tokens, so a prefill worker recomputing every turn from scratch wastes that work. A stateful proxy tracks cache parameters per conversation and lets the prefill worker pull the still-resident blocks back from the decode worker instead of recomputing them, cutting repeated-turn TTFT.4

SGLang

Dynamo's disaggregation supports SGLang alongside vLLM and TRT-LLM, with NIXL moving KV directly from prefill-engine VRAM to decode-engine VRAM via RDMA; each engine has backend-specific component flags documented per backend. (Exact SGLang flag names are not reproduced here; consult the Dynamo SGLang backend docs.)8

Dynamo orchestration

Dynamo registers prefill and decode roles and routes requests through the decode side. The connector determines transfer direction: vLLM's default connector has decode read from prefill, while the push connector has prefill write to decode. Do not encode request direction as proof of RDMA operation direction.98

NIXL's own design doc is explicit that these three tiers do not all go through GDS: "if the source is DRAM and the target is VRAM, UCX might be used... if the transfer involves VRAM as the source and PFS as the backend, GPUDirect Storage APIs could be employed." GDS is GPU-to-storage-device DMA specifically; object storage runs through NIXL's separate obj backend (S3-family), not GDS.

How to maintain it

Validate that transfers are truly zero-copy and overlapped on the Nsight Systems timeline. The book's unified profiling command traces CUDA, UCX, and GPUDirect Storage, and adds NIC/IB-switch telemetry:1

IB_SWITCH_GUIDS="${IB_SWITCH_GUIDS:?set to a comma-separated GUID list}"
LAUNCH_CMD=(python -m your_package.prefill_decode)

nsys profile --trace=cuda-hw,osrt,nvtx,ucx,gds \
  --trace-fork-before-exec=true \
  --cuda-event-trace=true \
  --cuda-graph-trace=node \
  --cuda-memory-usage=true \
  --sample=cpu \
  --gpu-metrics-device=all \
  --nic-metrics=true \
  --ib-switch-metrics-device="$IB_SWITCH_GUIDS" \
  --storage-metrics --storage-devices=all \
  --gds-metrics=driver \
  -o nsys_reports/prefill_decode \
  "${LAUNCH_CMD[@]}"

Operational notes: keep peer memory pre-registered with large pinned windows to avoid re-registration churn; correlate UCX activity, GPU metrics, and IB-switch counters to confirm KV transfer overlaps decode kernels rather than blocking them. Sizing the KV payload itself follows bytes_per_token = 2 x n_layers x n_kv_heads x head_dim x bytes_per_element (the 2 x covers keys and values); GQA/MQA/MLA and FP8/FP4 KV all shrink what NIXL has to move. See KV Cache Management: PagedAttention and Prefix Caching.1

Caveat: the NIXL flags, env vars, payload sizes, and profiling command above are reproduced from the book and official vLLM/TensorRT-LLM/Dynamo docs. None were exercised on hardware in this writeup. Where the book and official docs differ, official docs win: e.g. the book calls the Dynamo+TRT-LLM NIXL path the transfer abstraction, and the TRT-LLM docs make NIXL the default cache_transceiver_config.backend (with UCX as the legacy default for the raw TRTLLM_USE_* env switches).67

Failure modes

  • Descriptor metadata is missing, stale, or invalidated while a peer still references it.
  • A transfer handle is leaked, double-posted, released before terminal status, or never polled to completion.
  • Connector direction is assumed from request flow, producing a READ/WRITE mismatch.
  • recompute hides transfer failures as prefill work on decode GPUs and pushes up tail latency.
  • A supposed zero-copy path falls back to an unintended backend or stages through host memory.
  • Payloads are too small to amortize registration, control, and transport overhead.

References

  • Chris Fregly, AI Systems Performance Engineering (O'Reilly), Chapter 17, "Scaling Disaggregated Prefill and Decode for Inference."1
  • NVIDIA Technical Blog, "Enhancing Distributed Inference Performance with the NVIDIA Inference Transfer Library." https://developer.nvidia.com/blog/enhancing-distributed-inference-performance-with-the-nvidia-inference-transfer-library/ 2
  • NIXL design doc (ai-dynamo/nixl). https://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md 3
  • vLLM docs, "NixlConnector Usage Guide." https://docs.vllm.ai/en/stable/features/nixl_connector_usage/ 4
  • TensorRT-LLM docs, "Disaggregated Serving" (cache_transceiver_config). https://nvidia.github.io/TensorRT-LLM/features/disagg-serving.html 6
  • NVIDIA Dynamo docs, "KV Cache Transfer in Disaggregated Serving" (TensorRT-LLM backend, TRTLLM_USE_NIXL_KVCACHE / TRTLLM_USE_UCX_KVCACHE). https://docs.nvidia.com/dynamo/latest/backends/trtllm/kv-cache-transfer.html 7
  • NVIDIA Dynamo docs, "Disaggregated Serving." https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving 8
  • vLLM design doc, "NIXL KV Push Connector" (pull-based default vs. push-based alternative). https://github.com/vllm-project/vllm/blob/main/docs/design/nixl_kv_push_connector.md 9
  • NIXL v1.3.1 C++ API header (ai-dynamo/nixl). https://github.com/ai-dynamo/nixl/blob/v1.3.1/src/api/cpp/nixl.h 5

Related: Disaggregated Inference · KV Cache Management: PagedAttention and Prefix Caching · NVSHMEM: GPU-Initiated Communication · Inference Parallelism: TP, PP, EP, DP for Serving · Continuous Batching and Scheduler Internals · Inference Serving and Optimization · Glossary

  1. Fregly, AI Systems Performance Engineering, Ch. 17. ↩↩↩↩↩↩↩↩↩

  2. NVIDIA, "Enhancing Distributed Inference Performance with the NVIDIA Inference Transfer Library." ↩↩↩↩

  3. ai-dynamo/nixl, docs/nixl.md. ↩↩

  4. vLLM, "NixlConnector Usage Guide": kv_load_failure_policy values fail (default, immediately fails the request) and recompute (recomputes missing blocks locally on decode, at the cost of jitter); periodic log summaries plus nine Prometheus metrics (transfer duration, posting time, bytes transferred, descriptor counts, failed transfers, expired requests); bidirectional multi-turn transfer lets a prefill worker pull KV blocks the decode worker already holds from a prior turn instead of recomputing them. ↩↩↩↩↩↩

  5. ai-dynamo/nixl v1.3.1, src/api/cpp/nixl.h at commit b20f4d0adb95f53c4a1272915ef8d86cec304584: transfer requests use createXferReq, postXferReq, getXferStatus, and releaseXferReq; transfer notifications are supplied through optional request parameters, while genNotif creates a standalone notification. https://github.com/ai-dynamo/nixl/blob/v1.3.1/src/api/cpp/nixl.h ↩↩↩

  6. TensorRT-LLM, "Disaggregated Serving." ↩↩↩↩

  7. NVIDIA Dynamo, "KV Cache Transfer in Disaggregated Serving" (TRT-LLM). ↩↩↩

  8. NVIDIA Dynamo, "Disaggregated Serving." ↩↩↩

  9. vLLM design doc, "NIXL KV Push Connector": "The default NIXL connector is pull-based: the decode (D) instance reads KV blocks from the prefill (P) instance via NIXL READ after prefill completes. NixlPushConnector adds a push-based alternative in which P writes the KV blocks directly into D's pre-allocated memory via NIXL WRITE." The connector source confirms this: NixlConnector = NixlPullConnector in connector.py; pull_worker.py issues make_prepped_xfer("READ", ...) from the decode side; wait_for_layer_load/save_kv_layer are explicit no-ops ("NixlConnector does not do layerwise saving" / "does not save explicitly"). https://github.com/vllm-project/vllm/blob/main/docs/design/nixl_kv_push_connector.md ↩↩↩↩