Markdown

Managed inference endpoints on SageMaker¶

Scope: deploying open-weight LLMs to Amazon SageMaker managed endpoints instead of a self-managed GPU cluster. Covers the four inference pathways and which ones can serve an LLM at all, the Hugging Face Deep Learning Container contract, the measured cost premium of the managed control plane over raw EC2, the production defaults (autoscaling, alarms, tags, quotas), and the blast radius of handing endpoint creation to an agent. The self-managed counterpart is inference serving and serving open-weight models; the procurement frame is cloud, neoclouds and cost.

Prices are from the AWS Price List bulk API (AmazonSageMaker and AmazonEC2 offer files), us-east-1, on-demand, Linux, shared tenancy, fetched 2026-07-13. They exclude Savings Plans and are region-dependent. Re-fetch before quoting. Container image URIs move continuously: resolve them from the AWS DLC catalog, never from memory. The boto3 and SageMaker SDK snippets below are unexecuted reference templates; the Python cost model is executed and asserted.

What it is¶

A SageMaker managed inference endpoint is a hosted model server: AWS pulls a container image, loads model weights into it on a GPU instance it owns, puts an HTTPS API in front of it, and bills per instance-hour. The operator never touches a node, a kubelet, a driver, or an ingress. What would be a Deployment, a Service, an HPA, a readiness probe, and a Prometheus rule in a self-managed cluster (vLLM inference deployment recipe) collapses into one CreateEndpoint call plus an autoscaling policy and a few CloudWatch alarms.

The model itself is unchanged. The same open-weight checkpoints served with vLLM on an owned cluster (serving open-weight models) run in the same vLLM engine inside an AWS-built container. What changes is who operates the plane underneath it, and what that costs.

SageMaker exposes four inference pathways, and they are not interchangeable for LLMs:

Pathway	Scale to zero	Serves an LLM	Use for
Real-time	Classic model-based variants: no. Inference-component endpoints: yes, but invocations fail while capacity is zero and provisioning takes minutes.	Yes	Interactive traffic; use scale-to-zero only when failed wake requests and cold starts are acceptable.
Async	Yes, genuinely. `MinCapacity=0`.	Yes	Bursty traffic, and work that outruns the 60 s real-time `InvokeEndpoint` ceiling (async allows up to one hour); batches between idle gaps.
Serverless	Yes	No. Disqualified.	Small CPU models only.
Batch transform	N/A (job-scoped)	Yes	Offline scoring over a dataset; no endpoint to leave running.

Two of those cells carry the load-bearing facts.

Serverless Inference cannot serve an LLM, and not for a soft reason. The AWS documentation lists GPUs under feature exclusions: "Some of the features currently available for SageMaker AI Real-time Inference are not supported for Serverless Inference, including GPUs, AWS marketplace model packages, private Docker registries, Multi-Model Endpoints, VPC configuration, network isolation, data capture, multiple production variants, Model Monitor, and inference pipelines." Serverless is CPU-only. Its RAM is capped between 1024 MB and 6144 MB and per-endpoint concurrency at 200, so even the memory ceiling alone excludes a 7B model, but the GPU exclusion is the hard stop. There is no LLM configuration of a serverless endpoint that works.

Async is not a cheaper hour; it is a cheaper duty cycle. The per-hour price of an async endpoint is identical to real-time. Checked against the price list for the same instance, the USE1-AsyncInf:* and USE1-Host:* rates match to the cent (ml.g5.xlarge $1.4080 both ways; ml.p5.48xlarge $63.2960 both ways). Async wins on cost only because it can scale to zero, never because the hour is discounted. Coming back off zero promptly takes a second policy: target tracking on ApproximateBacklogSizePerInstance handles scaling between min and max, and on its own it lifts the endpoint off zero only "after the number of backlog requests exceeds the target tracking value", which AWS warns "can result in long waiting times for requests in the queue". The fix is AWS's optional step-scaling policy on a HasBacklogWithoutCapacity alarm, which wakes the endpoint on the first queued request instead of on the target-th one.

Real-time scale-to-zero has a second, narrower path. AWS now supports zero instance infrastructure and zero inference-component copies for real-time endpoints that use inference components. Set managed instance scaling and copy counts to zero, then attach step scaling to NoCapacityInvocationFailures. Unlike async, real-time does not queue the wake request: invocations fail while capacity is zero, and AWS documents a several-minute provisioning delay. The classic model-based ProductionVariant path used by the template below still requires at least one initial instance and is the path priced by this page's cost model.

Why use it¶

It deletes the cluster, not the GPU bill. No driver pairing, no fabric, no node lifecycle, no k8s upgrade. For a team whose product is a model and not a cluster, the entire operational surface of cluster orchestration disappears.
Autoscaling, alarms, and rollback are configuration rather than construction. Target tracking, CloudWatch alarms, and blue/green variant shifts are API calls, not a Prometheus and Argo stack to stand up.
Weights never transit the operator's machine. With a Hub model ID in the container environment, the endpoint pulls weights from the Hub directly into the container at start. They do not pass through a laptop and do not need to be staged in S3 first. A gated model may require HUGGING_FACE_HUB_TOKEN; treat it as a secret because model environment variables are visible to principals that can read the model definition. Prefer an S3-staged model or a deployment-specific, read-only token with the smallest practical scope.
The premium is bounded and measurable. 1.15x to 1.40x over the EC2 rate for the same silicon (below). That is a number a platform team can defend or reject, not a vague "managed is expensive".

When to use it (and when not)¶

Use it when traffic is modest or bursty, when the team has no cluster and no wish to build one, when a model must ship this week, or when the workload is offline scoring (batch transform) or tolerant of queueing (async). Inference-component real-time endpoints can also scale to zero, but they fail requests during the wake-up interval instead of queueing them.

Do not use it when GPUs run hot around the clock. The managed premium is charged on every instance-hour, so at high sustained utilisation it is a permanent tax on the exact hours a reserved or owned fleet amortises best; the break-even logic is build vs rent with the premium folded in. Do not use it for multi-node inference of a model that exceeds one instance, where disaggregation and expert parallelism over NVLink (disaggregated inference) are the point. Do not use it if the deployment needs engine flags the container does not expose, and do not reach for serverless at all.

Check quota before promising an instance type. GPU endpoint quotas are per instance type, per region, and default to 0 in many accounts. Recommending a shape the account cannot launch burns a full deploy cycle on ResourceLimitExceeded:

aws service-quotas list-service-quotas --service-code sagemaker --region us-east-1 \
  --query "Quotas[?contains(QuotaName, 'for endpoint usage') && Value > \`0\`].[QuotaName, Value]" \
  --output table

Architecture¶

flowchart TB
  subgraph OPERATOR["Operator or agent"]
    SKILL["Agent skill: pathway, quota, image URI, IAM, defaults"]
    CALL["CreateModel + CreateEndpointConfig + CreateEndpoint"]
  end
  subgraph AWS["SageMaker managed plane"]
    EP["Endpoint (HTTPS)"]
    VAR["Production variant on ml.g5/g6/p4d/p5"]
    CTR["HF vLLM DLC from ECR (SM_VLLM_* env)"]
    AS["Application Auto Scaling"]
    CW["CloudWatch alarms + logs"]
  end
  HUB["Hugging Face Hub (weights)"]
  SKILL --> CALL --> EP --> VAR --> CTR
  CTR -->|"pull weights at start"| HUB
  AS -->|"InvocationsPerInstance target 20/min"| VAR
  VAR --> CW
  CLIENT["Client"] -->|"InvokeEndpoint"| EP

The execution role is what the endpoint may do at runtime. The operator's (or agent's) own credentials are what may be created. Those are different IAM surfaces, and conflating them is the sharpest risk on this page (see Blast radius).

The managed premium¶

This is the number the decision turns on: what the control plane costs over the same GPU on EC2. Both columns come from the AWS Price List bulk API on 2026-07-13 (us-east-1, on-demand, Linux, shared tenancy, CapacityStatus=Used, excluding the $0.00 Capacity Block rows). Rates are shown to four decimals; the p4d EC2 rate is $21.957642/hr exactly in the API, and the executed model below uses the unrounded value.

Instance	GPU	SageMaker Hosting $/hr	730 h/mo	EC2 equivalent $/hr	Managed premium
ml.g5.xlarge	1x A10G 24GB	1.4080	1,028	1.0060	1.40x
ml.g6.xlarge	1x L4 24GB	1.1267	822	0.8048	1.40x
ml.g6e.xlarge	1x L40S 48GB	2.6054	1,902	1.8610	1.40x
ml.g5.12xlarge	4x A10G	7.0900	5,176	5.6720	1.25x
ml.p4d.24xlarge	8x A100 40 GB (320 GB total)	25.2513	18,433	21.9576	1.15x
ml.p5.48xlarge	8x H100 80 GB (640 GB total)	63.2960	46,206	55.0400	1.15x

The premium is not a flat family rule: it shrinks as the instance grows, from 1.40x on the single-GPU G shapes to 1.25x on ml.g5.12xlarge to 1.15x on both P shapes. The convenient summary "about 1.40x on G, about 1.15x on P" is right at the ends and wrong in the middle, which is exactly why the multi-GPU G row is in the table. Read it as: 15% to 40% for the control plane, buying autoscaling, alarms, and no cluster to run.

The executed model below turns the rates into the decision. It sizes the endpoint from the autoscaling target, prices a classic model-based real-time variant (minimum one instance) against async (which reaches zero but pays a cold start on every wake), and finds the duty cycle where the two cross.

# endpoint_cost.py - executed. Classic model-based real-time (min 1 instance) vs async
# (MinCapacity=0, genuine scale-to-zero) on SageMaker Hosting.
# Prices: AWS Price List bulk API, AmazonSageMaker/AmazonEC2, us-east-1,
# on-demand, fetched 2026-07-13. No AWS calls here: pure arithmetic.
import math

MONTH_H = 730.0                 # billing month
TARGET_PER_MIN = 20.0           # SageMakerVariantInvocationsPerInstance (skill default)
MAX_CAPACITY = 4                # skill default

SM_HOSTING = {"ml.g5.xlarge": 1.4080, "ml.g6.xlarge": 1.1267,
              "ml.g6e.xlarge": 2.6054, "ml.g5.12xlarge": 7.0900,
              "ml.p4d.24xlarge": 25.2513, "ml.p5.48xlarge": 63.2960}
EC2 = {"ml.g5.xlarge": 1.0060, "ml.g6.xlarge": 0.8048, "ml.g6e.xlarge": 1.8610,
       "ml.g5.12xlarge": 5.6720, "ml.p4d.24xlarge": 21.957642,
       "ml.p5.48xlarge": 55.0400}


def instances_for_rate(rate_per_min: float, target: float = TARGET_PER_MIN) -> int:
    """Instances target-tracking settles on. Fails fast on nonsense inputs."""
    if target <= 0:
        raise ValueError(f"autoscaling target must be > 0, got {target}")
    if rate_per_min < 0:
        raise ValueError(f"request rate must be >= 0, got {rate_per_min}")
    return math.ceil(rate_per_min / target)


def billed_hours(rate_per_min: float, duty: float, bursts: int,
                 cold_start_h: float) -> tuple[float, float]:
    """(real-time, async) billed instance-hours for one month.

    The classic model-based real-time variant never scales below one instance, so it pays for idle time.
    Async scales to 0 but pays a cold start on every wake-from-zero.
    """
    if not 0.0 <= duty <= 1.0:
        raise ValueError(f"duty cycle must be in [0, 1], got {duty}")
    n = min(max(instances_for_rate(rate_per_min), 1), MAX_CAPACITY)
    active_h, idle_h = duty * MONTH_H, (1.0 - duty) * MONTH_H
    real_time = n * active_h + 1 * idle_h          # MinCapacity = 1
    asynchronous = n * active_h + n * bursts * cold_start_h   # MinCapacity = 0
    return real_time, asynchronous


def crossover_duty(rate_per_min: float, bursts: int, cold_start_h: float) -> float:
    """Duty cycle where async's cold-start waste equals real-time's idle instance."""
    n = min(max(instances_for_rate(rate_per_min), 1), MAX_CAPACITY)
    return 1.0 - (n * bursts * cold_start_h) / MONTH_H


RATE = 40.0                 # req/min while active
# BURSTS and COLD are ILLUSTRATIVE, not measured: they are the two free knobs of
# this model, so the crossover below is a range, not a constant. Swept in step 4b.
BURSTS, COLD = 600, 0.1     # 600 wakes from zero per month; 6 min cold start
INST = "ml.g5.xlarge"
price = SM_HOSTING[INST]

# 1. Idle endpoint: real-time pays a full instance-month, async pays nothing.
rt0, as0 = billed_hours(RATE, 0.0, 0, COLD)
assert rt0 == MONTH_H and as0 == 0.0, (rt0, as0)
print(f"duty=0%   real-time ${price * rt0:>9,.2f}/mo   async ${price * as0:>8,.2f}/mo")

# 2. The post's headline number is the EC2 rate, not the SageMaker one.
print(f"idle ml.g5.xlarge: SageMaker ${price * MONTH_H:,.2f}/mo "
      f"vs EC2 g5.xlarge ${EC2[INST] * MONTH_H:,.2f}/mo "
      f"(post said '$730'; understated by {price / EC2[INST]:.2f}x)")

# 3. Managed premium multiplies straight through to the bill.
for i in ("ml.g5.xlarge", "ml.g5.12xlarge", "ml.p5.48xlarge"):
    print(f"  {i:16} premium {SM_HOSTING[i] / EC2[i]:.4f}x")
assert abs(SM_HOSTING["ml.g6.xlarge"] / EC2["ml.g6.xlarge"] - 1.40) < 5e-4
assert abs(SM_HOSTING["ml.g5.12xlarge"] / EC2["ml.g5.12xlarge"] - 1.25) < 5e-4  # NOT 1.40
assert abs(SM_HOSTING["ml.p5.48xlarge"] / EC2["ml.p5.48xlarge"] - 1.15) < 5e-4

# 4. Crossover: async is NOT unconditionally cheaper once cold starts are billed.
d_star = crossover_duty(RATE, BURSTS, COLD)
print(f"\ncrossover duty cycle = {d_star:.4f}")
lo, hi = d_star - 0.05, d_star + 0.05
rt_lo, as_lo = billed_hours(RATE, lo, BURSTS, COLD)
rt_hi, as_hi = billed_hours(RATE, hi, BURSTS, COLD)
assert as_lo < rt_lo, "below crossover async must be cheaper"
assert as_hi > rt_hi, "above crossover real-time must be cheaper"
rt_x, as_x = billed_hours(RATE, d_star, BURSTS, COLD)
assert math.isclose(rt_x, as_x, rel_tol=1e-9), (rt_x, as_x)
for d in (0.10, 0.50, d_star, 0.95):
    rt, asy = billed_hours(RATE, d, BURSTS, COLD)
    win = "tie" if math.isclose(asy, rt, rel_tol=1e-9) else ("async" if asy < rt else "real-time")
    print(f"  duty={d:>6.1%}  real-time ${price * rt:>9,.2f}  "
          f"async ${price * asy:>9,.2f}  cheaper={win}")

# 4b. The crossover is only as solid as BURSTS and COLD, which are guesses. Sweep
#     them: the finding is that a crossover EXISTS, not that it sits at 83.6%.
print("\nsensitivity of the crossover to the two illustrative inputs:")
for b, c in ((100, 0.1), (600, 0.1), (2000, 0.1), (2000, 0.2)):
    d = crossover_duty(RATE, b, c)
    note = "  <- async never wins" if d <= 0.0 else ""
    print(f"  bursts={b:>4}/mo  cold={c * 60:>2.0f} min  crossover duty={d:>7.4f}{note}")

# 5. Above the crossover real-time is cheaper AND faster: async still pays a cold
#    start on the first request of every burst, so it cannot win on latency.
print(f"\nabove crossover real-time dominates: cheaper, and async still adds "
      f"{COLD * 3_600_000:,.0f} ms of cold start to the first request of each burst")

# 6. Adversarial: bad inputs must raise, not silently divide by zero.
for bad in (lambda: instances_for_rate(40.0, target=0.0),
            lambda: instances_for_rate(-1.0),
            lambda: billed_hours(40.0, 1.5, 10, COLD)):
    try:
        bad()
        raise AssertionError("expected ValueError")
    except ValueError:
        pass
print("adversarial: target=0, negative rate, duty>1 all raise ValueError")

# 7. The orphan: one forgotten p5 endpoint, 30 days, never invoked.
orphan = SM_HOSTING["ml.p5.48xlarge"] * 24 * 30
print(f"orphaned ml.p5.48xlarge, 30 days, zero requests: ${orphan:,.2f}")
assert round(orphan, 2) == 45573.12

Output:

duty=0%   real-time $ 1,027.84/mo   async $    0.00/mo
idle ml.g5.xlarge: SageMaker $1,027.84/mo vs EC2 g5.xlarge $734.38/mo (post said '$730'; understated by 1.40x)
  ml.g5.xlarge     premium 1.3996x
  ml.g5.12xlarge   premium 1.2500x
  ml.p5.48xlarge   premium 1.1500x

crossover duty cycle = 0.8356
  duty= 10.0%  real-time $ 1,130.62  async $   374.53  cheaper=async
  duty= 50.0%  real-time $ 1,541.76  async $ 1,196.80  cheaper=async
  duty= 83.6%  real-time $ 1,886.72  async $ 1,886.72  cheaper=tie
  duty= 95.0%  real-time $ 2,004.29  async $ 2,121.86  cheaper=real-time

sensitivity of the crossover to the two illustrative inputs:
  bursts= 100/mo  cold= 6 min  crossover duty= 0.9726
  bursts= 600/mo  cold= 6 min  crossover duty= 0.8356
  bursts=2000/mo  cold= 6 min  crossover duty= 0.4521
  bursts=2000/mo  cold=12 min  crossover duty=-0.0959  <- async never wins

above crossover real-time dominates: cheaper, and async still adds 360,000 ms of cold start to the first request of each burst
adversarial: target=0, negative rate, duty>1 all raise ValueError
orphaned ml.p5.48xlarge, 30 days, zero requests: $45,573.12

Three results follow under the model's stated assumptions. An idle classic model-based real-time endpoint is not free: at zero traffic it still bills a full instance-month, $1,027.84 on the smallest useful GPU shape, because this deployment path cannot drop below one instance. Async is not unconditionally cheaper: every wake from zero bills a cold start (container pull plus weight load), so above some duty cycle the cold-start waste exceeds the idle instance real-time pays for, and real-time becomes both cheaper and faster (async still owes roughly 6 minutes of first-request latency on each cold wake). A crossover exists; its position is a guess. The wake count and the cold-start duration are the two free knobs here, neither measured, and the sweep shows how much they move the answer: 0.97 at 100 wakes a month with a 6-minute cold start, 0.84 at 600 wakes, 0.45 at 2,000 wakes, and at 2,000 wakes with a 12-minute cold start async never wins at any duty cycle. Take the shape, not the 83.6%: below the crossover async dominates on cost, the more bursty the traffic the more decisively, and the crossover itself must be recomputed from a measured cold start and a measured wake count before it decides anything. The orphan is the real hazard: one forgotten ml.p5.48xlarge, never invoked, costs $45,573.12 over 30 days.

How to use it¶

Pick the container first: it is the single thing most likely to break a deployment that looked correct on paper, because a wrong or stale image produces the same opaque Failed to pass health check error as a dozen other faults.

TGI is archived. The huggingface/text-generation-inference repository is archived (last push 2026-03-21), so it is a dead end for new deployments, and the SageMaker Python SDK now auto-routes text-generation to the Hugging Face vLLM DLC (PR #5960, "change: routing logic change for Hugging Face DLCs", merged 2026-06-26). Any habit, tutorial, or model card that reaches for TGI first is out of date.

Resolve the image URI from the AWS DLC catalog, never from memory or a blog. The default repository for an LLM is huggingface-vllm, which is what the SDK's routing (PR #5960) selects. URIs look like this, and the tag moves:

763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04

The bare vllm repository is a different image: it is AWS's compatibility escape hatch, not the default. It carries a newer vLLM (vllm:0.24.0-gpu-py312-cu130-ubuntu22.04-sagemaker against huggingface-vllm's 0.21.0 in the catalog on 2026-07-13), and that version gap is the trap, because a higher number is not a reason to pick it. Reach for it only when no huggingface-vllm tag is compatible with the model or the region has none. The account ID differs in a few regions (eu-south-1 uses 692866216735), so confirm the region row in the catalog. To list the current tags directly:

aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \
  --region us-east-1 --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output text

The vLLM container is configured entirely through environment variables on the model definition, mapped to vLLM CLI flags:

Variable	Meaning
`SM_VLLM_MODEL`	Hub model ID (e.g. `Qwen/Qwen3-0.6B`), or `/opt/ml/model` when loading from S3.
`SM_VLLM_HOST`	Must be `0.0.0.0`. Otherwise vLLM binds localhost, the ping health check fails, and the container dies before emitting useful logs. The top cause of mystery failures.
`SM_VLLM_TRUST_REMOTE_CODE`	Leave unset or `false` unless an audited, revision-pinned model actually requires Hub Python code. Qwen3 is implemented in Transformers and does not require it.
`SM_VLLM_MAX_MODEL_LEN`	Max sequence length. Set it: defaults can be wrong for fine-tunes.
`SM_VLLM_GPU_MEMORY_UTILIZATION`	0.0 to 1.0; ~0.9 is reasonable.
`HUGGING_FACE_HUB_TOKEN`	Needed for gated Hub models. Use a scoped token and restrict access to `DescribeModel`; the value is stored in the model environment.

Any vLLM flag works: uppercase it, replace dashes with underscores, prepend SM_VLLM_. Embedding and reranker models use the TEI container instead, which takes HF_MODEL_ID rather than SM_VLLM_MODEL.

The image's CUDA version selects the host AMI, and getting it wrong kills the container silently. The requirement is keyed to the CUDA build in the image tag, not to the instance family. The image-selection skill's rule: vLLM images with "CUDA 13 or higher (current default: cu130) require setting InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1 on the ProductionVariant [...] Without it the container dies on startup with no CloudWatch logs ever created." Tags at cu129 or lower need no override. This applies to huggingface-vllm and to the bare vllm repo alike, since the former is layered on the latter. Note that AWS's ProductionVariant reference lists al2-ami-sagemaker-inference-gpu-3-1 as NVIDIA driver 550 / CUDA 12.4 and a separate al2023-ami-sagemaker-inference-gpu-4-1 as driver 580 / CUDA 13.0, so the pairing is not self-evident from the version numbers: take the value from the skill's lookup table (which reports cu130-on-g5 verified in June 2026) and re-verify it against the catalog at deploy time rather than reasoning it out from the AMI name.

Reference template, unexecuted, pinned:

# reference template (unexecuted). Exact API pins checked 2026-07-13:
# sagemaker==2.257.3 and boto3==1.43.46. This is the v2 `Model` API; v3 removed the
# module outright, so
# on 3.15.0 (current on 2026-07-13) `from sagemaker.model import Model` raises
# ModuleNotFoundError. The PR #5960 auto-routing is a v3 `ModelBuilder` behaviour and
# is not needed here, because this template pins the image URI explicitly.
from sagemaker.model import Model
model = Model(
    image_uri="763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm:<TAG-FROM-CATALOG>",
    role="arn:aws:iam::<ACCOUNT>:role/<EXISTING-SAGEMAKER-EXECUTION-ROLE>",
    env={
        "SM_VLLM_MODEL": "Qwen/Qwen3-0.6B",
        "SM_VLLM_HOST": "0.0.0.0",              # non-negotiable
        "SM_VLLM_TRUST_REMOTE_CODE": "false",       # enable only for audited remote code
        "SM_VLLM_MAX_MODEL_LEN": "8192",
        "SM_VLLM_GPU_MEMORY_UTILIZATION": "0.9",
    },
)
model.deploy(initial_instance_count=1, instance_type="ml.g5.xlarge",
             # Required because the tag above is cu130. Keyed to the image's CUDA
             # build, not the instance: omit it and the container dies with no logs.
             inference_ami_version="al2-ami-sagemaker-inference-gpu-3-1",
             endpoint_name="qwen3-06b", tags=[{"Key": "CreatedBy", "Value": "<owner>"}])

How to develop with it¶

Container tags, SDK routing, AMI requirements, and account quotas change independently. Keep those facts in versioned deployment automation, verify them at run time, and review changes like code.

Hugging Face publishes six Apache-2.0 skills for this deployment workflow (huggingface/skills):

Skill	Job
`hf-cloud-sagemaker-deployment-planner`	Entry point: choose the pathway, check the quota before naming an instance.
`hf-cloud-aws-context-discovery`	Read-only discovery of account, region, and existing resources.
`hf-cloud-python-env-setup`	Isolated env on Python 3.10 to 3.12; never 3.13+, where ML wheels lag.
`hf-cloud-sagemaker-iam-preflight`	Find an existing execution role; create one only if none exists.
`hf-cloud-serving-image-selection`	Container family plus the current image URI. Never default to TGI.
`hf-cloud-sagemaker-production-defaults`	Deploy, autoscaling, alarms, tags, smoke test, teardown.

The skills separate read-only discovery, IAM preflight, image selection, deployment, and teardown. Treat their lookup tables as cached operational knowledge, not authority: quota and image checks still run against the target account and region. The packaging pattern is covered in use this KB as an agent skill and skill optimization.

The baseline failure the skills are built against is worth stating precisely, because it is this KB's own thesis in miniature. An unaided coding agent, asked to deploy a model, picked TGI, failed the ping health check, bumped the TGI version, redeployed, failed again, and only pivoted to vLLM after roughly 80 minutes and four failed endpoint attempts, each billing GPU time. Its README then recommended TGI to the next reader, propagating the stale fact. The failure came from stale deployment facts: an otherwise valid workflow repeatedly selected an archived container. That is the same argument this KB makes for Xid codes and driver pairings.

What the evidence for the skills is, and is not. The published report covers two models (Qwen/Qwen3-0.6B and google/diffusiongemma-26B-A4B-it, a multimodal block-diffusion Gemma with a 128-expert, top-8 MoE), one run each, self-reported by the author, with no repeated trials and no held-out set. It is a credible practitioner report, not a benchmark, and it should not be cited as a measured effect. The repository's apps/evals-leaderboard/ is not a skill-eval harness (its collect_evals.py scrapes model-index metadata from trending Hub models), so it is not evidence either. For how one would actually know whether a skill helps, the method is a held-out selection split with a strict-improvement gate that rejects ties: see skill optimization. Anyone adopting these skills should re-verify the facts they encode rather than trust them, and the skills themselves say to check quotas and image URIs rather than guess.

The skills ship .claude-plugin/, .cursor-plugin/, and gemini-extension.json manifests, so they are packaged for several harnesses, but no harness compatibility was tested here.

How to maintain it¶

Treat the image URI as perishable. Re-resolve it from the catalog on every deploy and pin the resolved tag in the model definition. Never :latest.
Re-check the archived-upstream assumption. TGI is archived today; the SDK routes to vLLM today. Both are facts with dates, and both belong in a skill or a pinned doc that is re-verified, not in someone's habit.
Sweep for orphans by tag. The production-defaults scripts stamp a CreatedBy tag on every resource precisely so a forgotten endpoint can be found and killed. Run the sweep on a schedule; the p5 orphan above is $45,573.12 for 30 days of nothing.
Re-run the teardown path, and verify it. A teardown script that deletes the endpoint but leaves the endpoint-config and model behind is tidy but harmless; one that leaves the endpoint behind is a five-figure bug. Verify with describe-endpoint returning ValidationException, not by trusting exit code 0.
Data capture is off by default in the shipped defaults. Turning it on writes every request and response to S3, which is a privacy and cost decision, not a debugging default (security and multi-tenancy).

How to run it in production¶

The defaults below are the shipped values from the production-defaults skill's deploy.py, read from the repository rather than from the blog post:

Control	Default
Autoscaling policy	Target tracking on `SageMakerVariantInvocationsPerInstance`
Target	20 invocations/min/instance
Capacity	min 1, max 4
Cooldowns	60 s scale-out, 300 s scale-in
Alarm: latency	`ModelLatency` > 30,000 ms
Alarm: errors	`Invocation5XXErrors` > 5 in 5 min
Alarm: overhead	`OverheadLatency` > 2,000 ms
Tagging	`CreatedBy` on every resource, for orphan sweeps
Data capture	Off

The asymmetric cooldowns are correct and deliberate: scale out fast (60 s) because a backlog is user-visible, scale in slowly (300 s) because thrashing a GPU instance costs more than the idle minutes it saves. OverheadLatency is request-path service overhead: end-to-end invocation latency minus ModelLatency, reported in microseconds. It does not measure endpoint provisioning or model load for classic instance-based real-time endpoints; use endpoint state transitions, deployment events, and container logs for those. Serverless OverheadLatency is the documented exception that includes cold-start time. Alarm on both latency metrics, but interpret them accordingly (observability, SLO/SLI catalog).

Blast radius: the agent's credentials, not the execution role¶

Two IAM surfaces get conflated, and only one of them is actually controlled.

The execution role (what the endpoint may do at runtime) is well scoped. The IAM preflight skill ships a minimum-permissions.json granting exactly: s3:GetObject and s3:ListBucket on the model bucket, four ECR pull actions (GetAuthorizationToken, BatchCheckLayerAvailability, GetDownloadUrlForLayer, BatchGetImage), CloudWatch Logs on /aws/sagemaker/*, and cloudwatch:PutMetricData. That is a tight, defensible role.

The agent's own credentials (what it may create) are bounded by nothing. An agent that can call CreateEndpoint can select ml.p5.48xlarge at $63.30/hr, which is roughly $46,200/month, on a single call. The controls usually described (the agent voluntarily pausing to confirm, and a teardown script it is trusted to run) are conventions, not enforced controls. A convention is not a spend limit. The controls that actually bind:

IAM condition keys on sagemaker:InstanceTypes, allow-listing the shapes an agent may launch. This is the only control that stops the p5 call at the API.
Service quotas as a hard cap. Leaving GPU endpoint quotas at 0 and raising them deliberately, per type, is a feature and not an obstacle.
AWS Budgets with alerts and, where possible, actions.
A mandatory human approval gate before the first billable call, enforced by the harness (tool gating), not requested politely in a prompt.
CreatedBy tag sweeps to catch what still slips through.

The skill mechanism itself carries supply-chain risk: a skill is executable context that can grant itself broad tool access, and Anthropic's documentation warns to "review project skills before trusting a repository, since a skill can grant itself broad tool access". Read a third-party skill's allowed-tools and its scripts/ before installing it, with the same suspicion applied to any code that will run with cloud credentials (security and multi-tenancy, governing self-modifying agents).

Failure modes¶

Serverless chosen for an LLM. It cannot work: GPUs are an explicit feature exclusion, and RAM caps at 6144 MB. Fails at deploy, or wastes a day before it does.
TGI selected from memory or from a stale tutorial. Archived upstream; produces an opaque health-check failure that looks like a config error and invites version-bumping instead of a container change.
SM_VLLM_HOST left unset. vLLM binds localhost, the ping probe fails, the container dies before writing useful logs. Reads as a mystery.
A cu130 image deployed without InferenceAmiVersion. The worst failure on this page, because there is nothing to read: the container dies on startup and no CloudWatch log stream is ever created. An empty log group looks identical to a quota, networking, or account problem, so it routinely sends people down the wrong diagnostic path. The AMI is keyed to the image's CUDA build (cu130 or higher needs the override, cu129 and lower do not), so it changes when the tag changes, which is exactly the kind of expiring fact that belongs in a skill.
Classic real-time endpoint mistaken for free. A model-based production variant keeps at least one instance, so zero traffic still bills a full instance-month ($1,027.84 on ml.g5.xlarge in the dated example). Inference-component scale-to-zero is a different deployment path and fails wake requests while capacity is zero.
Async assumed to be a cheaper hour. It is the same hourly rate; only scale-to-zero saves money. Wake-from-zero left to target tracking alone. Without the optional HasBacklogWithoutCapacity step-scaling policy, a scaled-to-zero endpoint stays at zero until the backlog exceeds the target value, so a thin trickle of requests queues for a long time before anything starts.
Cost quoted from the EC2 price list for a SageMaker endpoint. Understates the bill by the managed premium: the widely repeated "$730/month" for an idle ml.g5.xlarge is the EC2 g5.xlarge rate ($1.006/hr x 730 h = $734). The SageMaker Hosting rate is $1.408/hr, so the real figure is $1,028, about 40% higher.
Instance recommended without a quota check. GPU endpoint quotas default to 0 in many accounts; the deploy dies on ResourceLimitExceeded.
Orphaned endpoint. No CreatedBy tag, no sweep, no budget alarm. A forgotten p5 is $45,573.12 per 30 days.
An agent with unbounded CreateEndpoint rights. One call can incur costs at a run rate of roughly $46,200 per 730-hour month. Bound it with IAM condition keys and quotas, not with a polite prompt.

References¶

Deploying models to SageMaker with agent skills (Hugging Face blog, the source practitioner report): https://huggingface.co/blog/hf-dwarez/hf-aws-sagemaker-skills
huggingface/skills (Apache-2.0; the six hf-cloud-* skills): https://github.com/huggingface/skills
AWS Deep Learning Containers, available images (resolve image URIs here): https://aws.github.io/deep-learning-containers/reference/available_images/
Hugging Face on Amazon SageMaker: https://huggingface.co/docs/sagemaker/index
SageMaker real-time endpoints: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
Scale real-time inference components to zero (failed wake invocation and several-minute provisioning delay): https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling-zero-instances.html
SageMaker endpoint invocation metrics (ModelLatency, OverheadLatency): https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipeline-logs-metrics.html
SageMaker asynchronous inference: https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html
Autoscaling an async endpoint to zero (HasBacklogWithoutCapacity): https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-autoscale.html
SageMaker Serverless Inference, incl. the GPU feature exclusion: https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html
SageMaker batch transform: https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html
SageMaker endpoint auto scaling: https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html
ProductionVariant API reference (the InferenceAmiVersion enum and its driver/CUDA versions): https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html
InvokeEndpoint API reference (the 60 s model-response ceiling): https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html
Amazon SageMaker AI pricing: https://aws.amazon.com/sagemaker/ai/pricing/
sagemaker-python-sdk PR #5960, "change: routing logic change for Hugging Face DLCs" (merged 2026-06-26): https://github.com/aws/sagemaker-python-sdk/pull/5960
huggingface/text-generation-inference (archived 2026-03-21): https://github.com/huggingface/text-generation-inference
Agent Skills standard: https://agentskills.io
Claude Code skills documentation (incl. the review-before-trusting warning): https://code.claude.com/docs/en/skills
AWS Budgets: https://aws.amazon.com/aws-cost-management/aws-budgets/