Skip to content
Markdown

Manifest: DRA ResourceClaim

Scope: a ResourceClaimTemplate that selects a GPU by attribute/capacity via a CEL expression (e.g. memory >= 40Gi), a Pod that consumes it through pod-level resourceClaims, the partitionable/shared-device semantics, and how to confirm the claim is allocated and the Pod runs. Pairs with NVIDIA DRA Driver for GPUs; the hub is Kubernetes & Helm: GPU Platform.

Reference templates from the upstream Kubernetes DRA API and the NVIDIA DRA driver. Pin the chart/image versions; apply via GitOps. Not hardware-tested here. Exercise on a real 1.34.2+ cluster before depending on it.

What it is

DRA (Dynamic Resource Allocation) replaces the device-plugin resources.limits["nvidia.com/gpu"] model with two K8s objects:

  • A ResourceClaim (or a ResourceClaimTemplate, which mints one claim per Pod) declares intent: "a device from class gpu.nvidia.com that satisfies this CEL filter."
  • A Pod references the claim by name; the scheduler matches the request against ResourceSlice objects the driver publishes (one per node, listing each GPU's attributes and capacity) and writes the chosen device into the claim's status.allocation.

Unlike nvidia.com/gpu: 1 (an opaque count), a claim selects on structured attributes (device.attributes, device.capacity): driver version, compute capability, product name, free memory. DRA went GA in Kubernetes 1.34; the NVIDIA driver publishes the gpu.nvidia.com and mig.nvidia.com device classes. 1 7

flowchart LR
  RST["ResourceClaimTemplate<br/>(CEL: memory >= 40Gi)"] --> RC["ResourceClaim<br/>(one per Pod)"]
  POD["Pod<br/>spec.resourceClaims"] --> RC
  SLICE["ResourceSlice<br/>(driver-published per node)"] --> SCHED["kube-scheduler"]
  RC --> SCHED
  SCHED --> ALLOC["status.allocation.devices[]<br/>driver/pool/device"]

Prerequisites

  • Kubernetes 1.34.2 or newer with the DRA feature gates and the resource.k8s.io/v1 API enabled (default on 1.34+). The DynamicResourceAllocation admission plugin must be on. 7
  • The NVIDIA DRA driver for GPUs installed and healthy (see NVIDIA DRA Driver for GPUs). It requires CDI enabled in the container runtime and an NVIDIA driver already on the node (host-installed or via the GPU Operator). 7
  • kubectl get deviceclass returns gpu.nvidia.com (and mig.nvidia.com if MIG support is enabled). 7
  • kubectl get resourceslices lists at least one slice per GPU node. Without slices, no claim can ever be satisfied. 2

The manifest

A claim for "one GPU with at least 40 GiB", consumed by a single-container Pod. selectors and allocationMode live inside exactly. 3 4

# resource.k8s.io/v1 is GA in Kubernetes 1.34+. The template mints one
# ResourceClaim per Pod that references it.
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-40g
  namespace: ml
spec:
  spec:
    devices:
      requests:
        - name: gpu                       # request name; the Pod claims it by this
          exactly:
            deviceClassName: gpu.nvidia.com
            allocationMode: ExactCount     # default; pair with count (defaults to 1)
            count: 1
            selectors:
              - cel:
                  # device.capacity[...] is a Quantity -> use compareTo()
                  expression: 'device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("40Gi")) >= 0'
---
apiVersion: v1
kind: Pod
metadata:
  name: dra-train
  namespace: ml
spec:
  restartPolicy: Never
  # Pod-level: bind the template to a local claim name.
  resourceClaims:
    - name: g
      resourceClaimTemplateName: gpu-40g
  containers:
    - name: trainer
      image: nvcr.io/nvidia/pytorch:25.05-py3
      command: ["bash", "-lc", "nvidia-smi -L && sleep 3600"]
      # Container-level: consume the claim. No nvidia.com/gpu limit here.
      resources:
        claims:
          - name: g

Tighten the selector with attributes when you need a specific SKU or driver, not just memory; combine clauses with &&: 3

            selectors:
              - cel:
                  expression: >-
                    device.attributes["gpu.nvidia.com"]["productName"] == "NVIDIA B200" &&
                    device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("180Gi")) >= 0

Attribute keys (productName, driverVersion, cudaComputeCapability, ...) are published by the driver in each ResourceSlice. Run kubectl get resourceslice -o yaml and read spec.devices[].attributes / .capacity to confirm the exact keys before pinning a selector; verify against your installed driver version rather than assuming. 7

Partitionable and shared devices

  • Shared within one Pod. List the same claim name under resources.claims in two containers; both get the identical GPU (same UUID) from one allocation. This is intra-Pod sharing, not isolation. 8
  • Shared across Pods (consumable capacity). Kubernetes 1.34 lets one physical device back multiple independent ResourceClaims, governed by driver/admin sharing policies expressed via ResourceSlice counters. 5
  • Partitionable devices (e.g. MIG). The driver publishes a parent device plus its partitions and ties them together with sharedCounters / consumesCounters on the ResourceSlice, so the scheduler never double-allocates overlapping slices. These counter fields are driver-published in the ResourceSlice, not authored in your claim. For MIG via DRA, request deviceClassName: mig.nvidia.com. 6 7

Configuration

Field (path) Object Meaning Example / note
spec.spec.devices.requests[].name ResourceClaimTemplate Request handle the Pod claims by gpu
…requests[].exactly.deviceClassName ResourceClaimTemplate Device class to draw from gpu.nvidia.com / mig.nvidia.com 7
…exactly.allocationMode ResourceClaimTemplate ExactCount (with count) or All ExactCount 4
…exactly.count ResourceClaimTemplate Devices to allocate for this request 1 4
…exactly.selectors[].cel.expression ResourceClaimTemplate CEL over device.attributes / device.capacity device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("40Gi")) >= 0 3
spec.resourceClaims[].name Pod Local name bound into the Pod g
spec.resourceClaims[].resourceClaimTemplateName Pod Template to instantiate per Pod gpu-40g 3
spec.containers[].resources.claims[].name Pod Container consumes the Pod claim g 3
status.allocation.devices[] ResourceClaim Allocated driver/pool/device/request populated by scheduler 4

Use resourceClaimTemplateName (one claim per Pod, no cleanup) for Deployments/Jobs. Use resourceClaimName only to share one explicit ResourceClaim object across Pods. 3

Apply & verify

kubectl create namespace ml --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f dra-resourceclaim.yaml
  1. The template instantiates a per-Pod claim; it appears as <pod>-<request>-… and must reach allocated: true:
kubectl get resourceclaims -n ml
# NAME              STATE                AGE
# dra-train-g-xxxxx  allocated,reserved   10s

allocated,reserved is the success signal: allocated means a device was bound, reserved means a Pod holds it. A claim stuck pending means no ResourceSlice satisfied the CEL filter (memory too high, wrong class, no slices). 3

  1. Inspect which physical GPU was bound:
kubectl get resourceclaim -n ml -o yaml | grep -A6 'allocation:'
# status.allocation.devices[]: driver: gpu.nvidia.com  pool: <node>  device: gpu-0
  1. The Pod must reach Running/Completed and see exactly the claimed GPU:
kubectl wait --for=condition=Ready pod/dra-train -n ml --timeout=120s
kubectl logs dra-train -n ml          # nvidia-smi -L lists 1 GPU with its UUID

nvidia-smi -L printing one GPU 0: NVIDIA … line confirms the device was injected via CDI. 8

Failure modes

  • Claim stuck pending. No ResourceSlice matched. The CEL threshold exceeds any device's memory, the deviceClassName is wrong, or the driver published no slices. Check kubectl get resourceslices and kubectl describe resourceclaim. 3
  • device.capacity[...] >= "40Gi" fails to compile. Capacity is a Quantity, not an int; you must use .compareTo(quantity("40Gi")) >= 0, not a bare >=. 3
  • Wrong attribute key in CEL. Keys are driver-version-specific. A typo silently matches nothing (claim stays pending). Read spec.devices[].attributes from a live ResourceSlice. 2
  • API group disabled / cluster < 1.34.2. no matches for kind "ResourceClaimTemplate" in version "resource.k8s.io/v1". Upgrade the cluster and enable the DRA feature gates and admission plugin. 7
  • Pod Pending with the claim allocated. CDI not enabled in the runtime, or the DRA driver kubelet plugin is unhealthy; the container can't get the device. Check kubectl get pods -n nvidia-dra-driver-gpu. 7
  • Two containers expecting separate GPUs but sharing one. Both referenced the same claim name; pod-level resourceClaims yields one device. Declare two claims for two GPUs. 8

References

  • Kubernetes — Dynamic Resource Allocation (concepts): https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/
  • Kubernetes — Allocate Devices to Workloads with DRA (ResourceClaimTemplate, Pod, verify): https://kubernetes.io/docs/tasks/configure-pod-container/assign-resources/allocate-devices-dra/
  • Kubernetes API — ResourceClaim v1: https://kubernetes.io/docs/reference/kubernetes-api/resource/resource-claim-v1/
  • Kubernetes 1.34 — DRA has graduated to GA: https://kubernetes.io/blog/2025/09/01/kubernetes-v1-34-dra-updates/
  • Kubernetes 1.34 — DRA Consumable Capacity (shared devices): https://kubernetes.io/blog/2025/09/18/kubernetes-v1-34-dra-consumable-capacity/
  • KEP-4815 — DRA Partitionable Devices (sharedCounters/consumesCounters): https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4815-dra-partitionable-devices/README.md
  • NVIDIA DRA Driver for GPUs (install, device classes, verify): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html
  • NVIDIA k8s-dra-driver-gpu — Validate setup for GPU allocation: https://github.com/NVIDIA/k8s-dra-driver-gpu/wiki/Validate-setup-for-GPU-allocation

Related: DRA driver (Helm) · GPU Platform hub · Kubernetes for GPUs · MIG partitioning · MIG mode manifest · Multi-tenancy · Glossary

  1. Kubernetes 1.34 — DRA has graduated to GA. https://kubernetes.io/blog/2025/09/01/kubernetes-v1-34-dra-updates/ ↩

  2. Kubernetes — Dynamic Resource Allocation (ResourceSlice, device classes). https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/ ↩↩

  3. Kubernetes — Allocate Devices to Workloads with DRA (ResourceClaimTemplate CEL, Pod resourceClaims, kubectl get resourceclaims). https://kubernetes.io/docs/tasks/configure-pod-container/assign-resources/allocate-devices-dra/ ↩↩↩↩↩↩↩↩↩

  4. Kubernetes API reference — ResourceClaim v1 (ExactDeviceRequest fields, status.allocation.devices[]). https://kubernetes.io/docs/reference/kubernetes-api/resource/resource-claim-v1/ ↩↩↩↩

  5. Kubernetes 1.34 — DRA Consumable Capacity. https://kubernetes.io/blog/2025/09/18/kubernetes-v1-34-dra-consumable-capacity/ ↩

  6. KEP-4815 — DRA Partitionable Devices. https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4815-dra-partitionable-devices/README.md ↩

  7. NVIDIA GPU Operator — NVIDIA DRA Driver for GPUs (chart 25.12.0, K8s 1.34.2+, gpu.nvidia.com / mig.nvidia.com device classes, kubectl get deviceclass/resourceslice). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/dra-intro-install.html ↩↩↩↩↩↩↩↩↩

  8. NVIDIA k8s-dra-driver-gpu — Validate setup for GPU allocation (shared GPU across containers in one Pod). https://github.com/NVIDIA/k8s-dra-driver-gpu/wiki/Validate-setup-for-GPU-allocation ↩↩↩