Markdown

Managed inference endpoints on SageMaker¶

Scope: deploying open-weight LLMs to Amazon SageMaker managed endpoints instead of a self-managed GPU cluster. Covers the four inference pathways and which ones can serve an LLM at all, the Hugging Face Deep Learning Container contract, the measured cost premium of the managed control plane over raw EC2, the production defaults (autoscaling, alarms, tags, quotas), and the blast radius of handing endpoint creation to an agent. The self-managed counterpart is inference serving and serving open-weight models; the procurement frame is cloud, neoclouds and cost.

Prices are from the AWS Price List bulk API (AmazonSageMaker and AmazonEC2 offer files), us-east-1, on-demand, Linux, shared tenancy, fetched 2026-07-13. They exclude Savings Plans and are region-dependent. Re-fetch before quoting. Container image URIs move continuously: resolve them from the SDK's offline registry or the AWS DLC catalog, never from memory. Verification split on this page: the Python cost model is executed and asserted; the image URI was resolved by executing sagemaker==3.16.0's bundled registry lookup locally on 2026-07-17 and cross-checked against the DLC catalog the same day; every boto3 call shape below (waiters, invocation, autoscaling, async config, deployment config, teardown) was checked against the installed botocore==1.43.50 service models locally; the rollback-alarm creation and the alarm/log-group teardown additions were re-checked on 2026-07-17 against boto3==1.34.46, the version installed in this environment, with parameter names and required-member sets pulled from operation_model rather than assumed. The AWS-side calls themselves are unexecuted: no AWS account was used, so no endpoint was created, invoked, scaled, updated, or deleted, and no alarm or log group was actually created or removed.

What it is¶

A SageMaker managed inference endpoint is a hosted model server: AWS pulls a container image, loads model weights into it on a GPU instance it owns, puts an HTTPS API in front of it, and bills per instance-hour. The operator never touches a node, a kubelet, a driver, or an ingress. What would be a Deployment, a Service, an HPA, a readiness probe, and a Prometheus rule in a self-managed cluster (vLLM inference deployment recipe) collapses into one CreateEndpoint call plus an autoscaling policy and a few CloudWatch alarms.

The model itself is unchanged. The same open-weight checkpoints served with vLLM on an owned cluster (serving open-weight models) run in the same vLLM engine inside an AWS-built container. What changes is who operates the plane underneath it, and what that costs.

SageMaker exposes four inference pathways, and they are not interchangeable for LLMs:

Pathway	Scale to zero	Serves an LLM	Use for
Real-time	Classic model-based variants: no. Inference-component endpoints: yes, but invocations fail while capacity is zero and provisioning takes minutes.	Yes	Interactive traffic; use scale-to-zero only when failed wake requests and cold starts are acceptable.
Async	Yes, genuinely. `MinCapacity=0`.	Yes	Bursty traffic, and work that outruns the 60 s real-time `InvokeEndpoint` ceiling (async allows up to one hour); batches between idle gaps.
Serverless	Yes	No. Disqualified.	Small CPU models only.
Batch transform	N/A (job-scoped)	Yes	Offline scoring over a dataset; no endpoint to leave running.

Two of those cells carry the load-bearing facts.

Serverless Inference cannot serve an LLM, and not for a soft reason. The AWS documentation lists GPUs under feature exclusions: "Some of the features currently available for SageMaker AI Real-time Inference are not supported for Serverless Inference, including GPUs, AWS marketplace model packages, private Docker registries, Multi-Model Endpoints, VPC configuration, network isolation, data capture, multiple production variants, Model Monitor, and inference pipelines." Serverless is CPU-only. Its RAM is capped between 1024 MB and 6144 MB and per-endpoint concurrency at 200, so even the memory ceiling alone excludes a 7B model, but the GPU exclusion is the hard stop. There is no LLM configuration of a serverless endpoint that works.

Async is not a cheaper hour; it is a cheaper duty cycle. The per-hour price of an async endpoint is identical to real-time. Checked against the price list for the same instance, the USE1-AsyncInf:* and USE1-Host:* rates match to the cent (ml.g5.xlarge $1.4080 both ways; ml.p5.48xlarge $63.2960 both ways). Async wins on cost only because it can scale to zero, never because the hour is discounted. Coming back off zero promptly takes a second policy: target tracking on ApproximateBacklogSizePerInstance handles scaling between min and max, and on its own it lifts the endpoint off zero only "after the number of backlog requests exceeds the target tracking value", which AWS warns "can result in long waiting times for requests in the queue". The fix is AWS's optional step-scaling policy on a HasBacklogWithoutCapacity alarm, which wakes the endpoint on the first queued request instead of on the target-th one.

Real-time scale-to-zero has a second, narrower path. AWS now supports zero instance infrastructure and zero inference-component copies for real-time endpoints that use inference components. Set managed instance scaling and copy counts to zero, then attach step scaling to NoCapacityInvocationFailures. Unlike async, real-time does not queue the wake request: invocations fail while capacity is zero, and AWS documents a several-minute provisioning delay. The classic model-based ProductionVariant path used by the template below still requires at least one initial instance and is the path priced by this page's cost model.

Why use it¶

It deletes the cluster, not the GPU bill. No driver pairing, no fabric, no node lifecycle, no k8s upgrade. For a team whose product is a model and not a cluster, the entire operational surface of cluster orchestration disappears.
Autoscaling, alarms, and rollback are configuration rather than construction. Target tracking, CloudWatch alarms, and blue/green variant shifts are API calls, not a Prometheus and Argo stack to stand up.
Weights never transit the operator's machine. With a Hub model ID in the container environment, the endpoint pulls weights from the Hub directly into the container at start. They do not pass through a laptop and do not need to be staged in S3 first. A gated model may require HUGGING_FACE_HUB_TOKEN; treat it as a secret because model environment variables are visible to principals that can read the model definition. Prefer an S3-staged model or a deployment-specific, read-only token with the smallest practical scope.
The premium is bounded and measurable. 1.15x to 1.40x over the EC2 rate for the same silicon (below). That is a number a platform team can defend or reject, not a vague "managed is expensive".

When to use it (and when not)¶

Use it when traffic is modest or bursty, when the team has no cluster and no wish to build one, when a model must ship this week, or when the workload is offline scoring (batch transform) or tolerant of queueing (async). Inference-component real-time endpoints can also scale to zero, but they fail requests during the wake-up interval instead of queueing them.

Do not use it when GPUs run hot around the clock. The managed premium is charged on every instance-hour, so at high sustained utilisation it is a permanent tax on the exact hours a reserved or owned fleet amortises best; the break-even logic is build vs rent with the premium folded in. Do not use it for multi-node inference of a model that exceeds one instance, where disaggregation and expert parallelism over NVLink (disaggregated inference) are the point. Do not use it if the deployment needs engine flags the container does not expose, and do not reach for serverless at all.

Check quota before promising an instance type. GPU endpoint quotas are per instance type, per region, and default to 0 in many accounts. Recommending a shape the account cannot launch burns a full deploy cycle on ResourceLimitExceeded:

aws service-quotas list-service-quotas --service-code sagemaker --region us-east-1 \
  --query "Quotas[?contains(QuotaName, 'for endpoint usage') && Value > \`0\`].[QuotaName, Value]" \
  --output table

Architecture¶

flowchart TB
  subgraph OPERATOR["Operator or agent"]
    SKILL["Agent skill: pathway, quota, image URI, IAM, defaults"]
    CALL["CreateModel + CreateEndpointConfig + CreateEndpoint"]
  end
  subgraph AWS["SageMaker managed plane"]
    EP["Endpoint (HTTPS)"]
    VAR["Production variant on ml.g5/g6/p4d/p5"]
    CTR["HF vLLM DLC from ECR (SM_VLLM_* env)"]
    AS["Application Auto Scaling"]
    CW["CloudWatch alarms + logs"]
  end
  HUB["Hugging Face Hub (weights)"]
  SKILL --> CALL --> EP --> VAR --> CTR
  CTR -->|"pull weights at start"| HUB
  AS -->|"InvocationsPerInstance target 20/min"| VAR
  VAR --> CW
  CLIENT["Client"] -->|"InvokeEndpoint"| EP

The execution role is what the endpoint may do at runtime. The operator's (or agent's) own credentials are what may be created. Those are different IAM surfaces, and conflating them is the sharpest risk on this page (see Blast radius).

The managed premium¶

This is the number the decision turns on: what the control plane costs over the same GPU on EC2. Both columns come from the AWS Price List bulk API on 2026-07-13 (us-east-1, on-demand, Linux, shared tenancy, CapacityStatus=Used, excluding the $0.00 Capacity Block rows). Rates are shown to four decimals; the p4d EC2 rate is $21.957642/hr exactly in the API, and the executed model below uses the unrounded value.

Instance	GPU	SageMaker Hosting $/hr	730 h/mo	EC2 equivalent $/hr	Managed premium
ml.g5.xlarge	1x A10G 24GB	1.4080	1,028	1.0060	1.40x
ml.g6.xlarge	1x L4 24GB	1.1267	822	0.8048	1.40x
ml.g6e.xlarge	1x L40S 48GB	2.6054	1,902	1.8610	1.40x
ml.g5.12xlarge	4x A10G	7.0900	5,176	5.6720	1.25x
ml.p4d.24xlarge	8x A100 40 GB (320 GB total)	25.2513	18,433	21.9576	1.15x
ml.p5.48xlarge	8x H100 80 GB (640 GB total)	63.2960	46,206	55.0400	1.15x

The premium is not a flat family rule: it shrinks as the instance grows, from 1.40x on the single-GPU G shapes to 1.25x on ml.g5.12xlarge to 1.15x on both P shapes. The convenient summary "about 1.40x on G, about 1.15x on P" is right at the ends and wrong in the middle, which is exactly why the multi-GPU G row is in the table. Read it as: 15% to 40% for the control plane, buying autoscaling, alarms, and no cluster to run.

The executed model below turns the rates into the decision. It sizes the endpoint from the autoscaling target, prices a classic model-based real-time variant (minimum one instance) against async (which reaches zero but pays a cold start on every wake), and finds the duty cycle where the two cross.

# endpoint_cost.py - executed. Classic model-based real-time (min 1 instance) vs async
# (MinCapacity=0, genuine scale-to-zero) on SageMaker Hosting.
# Prices: AWS Price List bulk API, AmazonSageMaker/AmazonEC2, us-east-1,
# on-demand, fetched 2026-07-13. No AWS calls here: pure arithmetic.
import math

MONTH_H = 730.0                 # billing month
TARGET_PER_MIN = 20.0           # SageMakerVariantInvocationsPerInstance (skill default)
MAX_CAPACITY = 4                # skill default

SM_HOSTING = {"ml.g5.xlarge": 1.4080, "ml.g6.xlarge": 1.1267,
              "ml.g6e.xlarge": 2.6054, "ml.g5.12xlarge": 7.0900,
              "ml.p4d.24xlarge": 25.2513, "ml.p5.48xlarge": 63.2960}
EC2 = {"ml.g5.xlarge": 1.0060, "ml.g6.xlarge": 0.8048, "ml.g6e.xlarge": 1.8610,
       "ml.g5.12xlarge": 5.6720, "ml.p4d.24xlarge": 21.957642,
       "ml.p5.48xlarge": 55.0400}


def instances_for_rate(rate_per_min: float, target: float = TARGET_PER_MIN) -> int:
    """Instances target-tracking settles on. Fails fast on nonsense inputs."""
    if target <= 0:
        raise ValueError(f"autoscaling target must be > 0, got {target}")
    if rate_per_min < 0:
        raise ValueError(f"request rate must be >= 0, got {rate_per_min}")
    return math.ceil(rate_per_min / target)


def billed_hours(rate_per_min: float, duty: float, bursts: int,
                 cold_start_h: float) -> tuple[float, float]:
    """(real-time, async) billed instance-hours for one month.

    The classic model-based real-time variant never scales below one instance, so it pays for idle time.
    Async scales to 0 but pays a cold start on every wake-from-zero.
    """
    if not 0.0 <= duty <= 1.0:
        raise ValueError(f"duty cycle must be in [0, 1], got {duty}")
    n = min(max(instances_for_rate(rate_per_min), 1), MAX_CAPACITY)
    active_h, idle_h = duty * MONTH_H, (1.0 - duty) * MONTH_H
    real_time = n * active_h + 1 * idle_h          # MinCapacity = 1
    asynchronous = n * active_h + n * bursts * cold_start_h   # MinCapacity = 0
    return real_time, asynchronous


def crossover_duty(rate_per_min: float, bursts: int, cold_start_h: float) -> float:
    """Duty cycle where async's cold-start waste equals real-time's idle instance."""
    n = min(max(instances_for_rate(rate_per_min), 1), MAX_CAPACITY)
    return 1.0 - (n * bursts * cold_start_h) / MONTH_H


RATE = 40.0                 # req/min while active
# BURSTS and COLD are ILLUSTRATIVE, not measured: they are the two free knobs of
# this model, so the crossover below is a range, not a constant. Swept in step 4b.
BURSTS, COLD = 600, 0.1     # 600 wakes from zero per month; 6 min cold start
INST = "ml.g5.xlarge"
price = SM_HOSTING[INST]

# 1. Idle endpoint: real-time pays a full instance-month, async pays nothing.
rt0, as0 = billed_hours(RATE, 0.0, 0, COLD)
assert rt0 == MONTH_H and as0 == 0.0, (rt0, as0)
print(f"duty=0%   real-time ${price * rt0:>9,.2f}/mo   async ${price * as0:>8,.2f}/mo")

# 2. The post's headline number is the EC2 rate, not the SageMaker one.
print(f"idle ml.g5.xlarge: SageMaker ${price * MONTH_H:,.2f}/mo "
      f"vs EC2 g5.xlarge ${EC2[INST] * MONTH_H:,.2f}/mo "
      f"(post said '$730'; understated by {price / EC2[INST]:.2f}x)")

# 3. Managed premium multiplies straight through to the bill.
for i in ("ml.g5.xlarge", "ml.g5.12xlarge", "ml.p5.48xlarge"):
    print(f"  {i:16} premium {SM_HOSTING[i] / EC2[i]:.4f}x")
assert abs(SM_HOSTING["ml.g6.xlarge"] / EC2["ml.g6.xlarge"] - 1.40) < 5e-4
assert abs(SM_HOSTING["ml.g5.12xlarge"] / EC2["ml.g5.12xlarge"] - 1.25) < 5e-4  # NOT 1.40
assert abs(SM_HOSTING["ml.p5.48xlarge"] / EC2["ml.p5.48xlarge"] - 1.15) < 5e-4

# 4. Crossover: async is NOT unconditionally cheaper once cold starts are billed.
d_star = crossover_duty(RATE, BURSTS, COLD)
print(f"\ncrossover duty cycle = {d_star:.4f}")
lo, hi = d_star - 0.05, d_star + 0.05
rt_lo, as_lo = billed_hours(RATE, lo, BURSTS, COLD)
rt_hi, as_hi = billed_hours(RATE, hi, BURSTS, COLD)
assert as_lo < rt_lo, "below crossover async must be cheaper"
assert as_hi > rt_hi, "above crossover real-time must be cheaper"
rt_x, as_x = billed_hours(RATE, d_star, BURSTS, COLD)
assert math.isclose(rt_x, as_x, rel_tol=1e-9), (rt_x, as_x)
for d in (0.10, 0.50, d_star, 0.95):
    rt, asy = billed_hours(RATE, d, BURSTS, COLD)
    win = "tie" if math.isclose(asy, rt, rel_tol=1e-9) else ("async" if asy < rt else "real-time")
    print(f"  duty={d:>6.1%}  real-time ${price * rt:>9,.2f}  "
          f"async ${price * asy:>9,.2f}  cheaper={win}")

# 4b. The crossover is only as solid as BURSTS and COLD, which are guesses. Sweep
#     them: the finding is that a crossover EXISTS, not that it sits at 83.6%.
print("\nsensitivity of the crossover to the two illustrative inputs:")
for b, c in ((100, 0.1), (600, 0.1), (2000, 0.1), (2000, 0.2)):
    d = crossover_duty(RATE, b, c)
    note = "  <- async never wins" if d <= 0.0 else ""
    print(f"  bursts={b:>4}/mo  cold={c * 60:>2.0f} min  crossover duty={d:>7.4f}{note}")

# 5. Above the crossover real-time is cheaper AND faster: async still pays a cold
#    start on the first request of every burst, so it cannot win on latency.
print(f"\nabove crossover real-time dominates: cheaper, and async still adds "
      f"{COLD * 3_600_000:,.0f} ms of cold start to the first request of each burst")

# 6. Adversarial: bad inputs must raise, not silently divide by zero.
for bad in (lambda: instances_for_rate(40.0, target=0.0),
            lambda: instances_for_rate(-1.0),
            lambda: billed_hours(40.0, 1.5, 10, COLD)):
    try:
        bad()
        raise AssertionError("expected ValueError")
    except ValueError:
        pass
print("adversarial: target=0, negative rate, duty>1 all raise ValueError")

# 7. The orphan: one forgotten p5 endpoint, 30 days, never invoked.
orphan = SM_HOSTING["ml.p5.48xlarge"] * 24 * 30
print(f"orphaned ml.p5.48xlarge, 30 days, zero requests: ${orphan:,.2f}")
assert round(orphan, 2) == 45573.12

Output:

duty=0%   real-time $ 1,027.84/mo   async $    0.00/mo
idle ml.g5.xlarge: SageMaker $1,027.84/mo vs EC2 g5.xlarge $734.38/mo (post said '$730'; understated by 1.40x)
  ml.g5.xlarge     premium 1.3996x
  ml.g5.12xlarge   premium 1.2500x
  ml.p5.48xlarge   premium 1.1500x

crossover duty cycle = 0.8356
  duty= 10.0%  real-time $ 1,130.62  async $   374.53  cheaper=async
  duty= 50.0%  real-time $ 1,541.76  async $ 1,196.80  cheaper=async
  duty= 83.6%  real-time $ 1,886.72  async $ 1,886.72  cheaper=tie
  duty= 95.0%  real-time $ 2,004.29  async $ 2,121.86  cheaper=real-time

sensitivity of the crossover to the two illustrative inputs:
  bursts= 100/mo  cold= 6 min  crossover duty= 0.9726
  bursts= 600/mo  cold= 6 min  crossover duty= 0.8356
  bursts=2000/mo  cold= 6 min  crossover duty= 0.4521
  bursts=2000/mo  cold=12 min  crossover duty=-0.0959  <- async never wins

above crossover real-time dominates: cheaper, and async still adds 360,000 ms of cold start to the first request of each burst
adversarial: target=0, negative rate, duty>1 all raise ValueError
orphaned ml.p5.48xlarge, 30 days, zero requests: $45,573.12

Three results follow under the model's stated assumptions. An idle classic model-based real-time endpoint is not free: at zero traffic it still bills a full instance-month, $1,027.84 on the smallest useful GPU shape, because this deployment path cannot drop below one instance. Async is not unconditionally cheaper: every wake from zero bills a cold start (container pull plus weight load), so above some duty cycle the cold-start waste exceeds the idle instance real-time pays for, and real-time becomes both cheaper and faster (async still owes roughly 6 minutes of first-request latency on each cold wake). A crossover exists; its position is a guess. The wake count and the cold-start duration are the two free knobs here, neither measured, and the sweep shows how much they move the answer: 0.97 at 100 wakes a month with a 6-minute cold start, 0.84 at 600 wakes, 0.45 at 2,000 wakes, and at 2,000 wakes with a 12-minute cold start async never wins at any duty cycle. Take the shape, not the 83.6%: below the crossover async dominates on cost, the more bursty the traffic the more decisively, and the crossover itself must be recomputed from a measured cold start and a measured wake count before it decides anything. The orphan is the real hazard: one forgotten ml.p5.48xlarge, never invoked, costs $45,573.12 over 30 days.

How to use it¶

Pick the container first: it is the single thing most likely to break a deployment that looked correct on paper, because a wrong or stale image produces the same opaque Failed to pass health check error as a dozen other faults.

TGI is archived. The huggingface/text-generation-inference repository is archived (last push 2026-03-21), so it is a dead end for new deployments, and the SageMaker Python SDK now auto-routes text-generation to the Hugging Face vLLM DLC (PR #5960, "change: routing logic change for Hugging Face DLCs", merged 2026-06-26). Any habit, tutorial, or model card that reaches for TGI first is out of date.

Resolve the image URI from the SDK or the AWS DLC catalog, never from memory or a blog. The default repository for an LLM is huggingface-vllm, which is what the SDK's routing (PR #5960) selects; the routing itself calls image_uris.retrieve("huggingface-vllm", ...) (sagemaker/serve/model_builder_utils.py line 706 in 3.16.0). The lookup is offline: the SDK resolves DLC URIs from a registry JSON bundled with the package (sagemaker/core/image_uri_config/huggingface-vllm.json in v3), so the current tag can be pinned without an AWS account. Executed here on 2026-07-17 with sagemaker==3.16.0:

# executed 2026-07-17, sagemaker==3.16.0, offline (bundled registry JSON, no AWS calls).
# v3 path shown; the v2 line (2.257.5, current) does NOT ship the GPU huggingface-vllm
# registry and raises FileNotFoundError for this framework (also executed).
from sagemaker.core import image_uris

uri = image_uris.retrieve(
    framework="huggingface-vllm", region="us-east-1", version="0.21.0",
    image_scope="inference", instance_type="ml.g5.xlarge",
)
print(uri)

Output:

763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04

Argument choices match this page's scenario (us-east-1, ml.g5.xlarge, inference). 0.21.0 is the newest of the three versions in the bundled registry (0.14.0, 0.17.0, 0.21.0), which pins py312 and cu130-ubuntu22.04 for it. The DLC catalog listed this exact URI on the same day, so registry and catalog agree today; the bundled registry is only as fresh as the installed SDK, so re-resolve after every SDK upgrade and cross-check the catalog on deploy day.

The bare vllm repository is a different image: it is AWS's compatibility escape hatch, not the default. It carries a newer vLLM (vllm:0.24.0-gpu-py312-cu130-ubuntu22.04-sagemaker against huggingface-vllm's 0.21.0 in the catalog on 2026-07-13), and that version gap is the trap, because a higher number is not a reason to pick it. Reach for it only when no huggingface-vllm tag is compatible with the model or the region has none. The account ID differs in a few regions (eu-south-1 uses 692866216735; the same image_uris.retrieve call with region="eu-south-1" resolved to that account, executed 2026-07-17), so confirm the region row in the catalog. To list the current tags directly:

aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \
  --region us-east-1 --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output text

The vLLM container is configured entirely through environment variables on the model definition, mapped to vLLM CLI flags:

Variable	Meaning
`SM_VLLM_MODEL`	Hub model ID (e.g. `Qwen/Qwen3-0.6B`), or `/opt/ml/model` when loading from S3.
`SM_VLLM_HOST`	Must be `0.0.0.0`. Otherwise vLLM binds localhost, the ping health check fails, and the container dies before emitting useful logs. The top cause of mystery failures.
`SM_VLLM_TRUST_REMOTE_CODE`	Leave unset or `false` unless an audited, revision-pinned model actually requires Hub Python code. Qwen3 is implemented in Transformers and does not require it.
`SM_VLLM_MAX_MODEL_LEN`	Max sequence length. Set it: defaults can be wrong for fine-tunes.
`SM_VLLM_GPU_MEMORY_UTILIZATION`	0.0 to 1.0; ~0.9 is reasonable.
`HUGGING_FACE_HUB_TOKEN`	Needed for gated Hub models. Use a scoped token and restrict access to `DescribeModel`; the value is stored in the model environment.

Any vLLM flag works: uppercase it, replace dashes with underscores, prepend SM_VLLM_. Embedding and reranker models use the TEI container instead, which takes HF_MODEL_ID rather than SM_VLLM_MODEL.

The image's CUDA version selects the host AMI, and getting it wrong kills the container silently. The requirement is keyed to the CUDA build in the image tag, not to the instance family. The image-selection skill's rule: vLLM images with "CUDA 13 or higher (current default: cu130) require setting InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1 on the ProductionVariant [...] Without it the container dies on startup with no CloudWatch logs ever created." Tags at cu129 or lower need no override. This applies to huggingface-vllm and to the bare vllm repo alike, since the former is layered on the latter. Note that AWS's ProductionVariant reference lists al2-ami-sagemaker-inference-gpu-3-1 as NVIDIA driver 550 / CUDA 12.4 and a separate al2023-ami-sagemaker-inference-gpu-4-1 as driver 580 / CUDA 13.0, so the pairing is not self-evident from the version numbers: take the value from the skill's lookup table (which reports cu130-on-g5 verified in June 2026) and re-verify it against the catalog at deploy time rather than reasoning it out from the AMI name.

Reference template, unexecuted, pinned:

# reference template (unexecuted against AWS). Pins re-checked by execution
# 2026-07-17: v2 line current is sagemaker==2.257.5 (boto3==1.43.50), and on it
# `from sagemaker.model import Model` imports, Model.__init__ accepts
# image_uri/role/env, and Model.deploy accepts inference_ami_version. On v3
# (3.16.0, current) the same import still raises ModuleNotFoundError: this is
# v2-only. The PR #5960 auto-routing is a v3 `ModelBuilder` behaviour and is not
# needed here, because this template pins the image URI explicitly.
from sagemaker.model import Model
model = Model(
    # Resolved offline by sagemaker==3.16.0 image_uris.retrieve on 2026-07-17
    # (see above); re-resolve before deploying.
    image_uri=("763104351884.dkr.ecr.us-east-1.amazonaws.com/"
               "huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04"),
    role="arn:aws:iam::<ACCOUNT>:role/<EXISTING-SAGEMAKER-EXECUTION-ROLE>",
    env={
        "SM_VLLM_MODEL": "Qwen/Qwen3-0.6B",
        "SM_VLLM_HOST": "0.0.0.0",              # non-negotiable
        "SM_VLLM_TRUST_REMOTE_CODE": "false",       # enable only for audited remote code
        "SM_VLLM_MAX_MODEL_LEN": "8192",
        "SM_VLLM_GPU_MEMORY_UTILIZATION": "0.9",
    },
)
model.deploy(initial_instance_count=1, instance_type="ml.g5.xlarge",
             # Required because the tag above is cu130. Keyed to the image's CUDA
             # build, not the instance: omit it and the container dies with no logs.
             inference_ami_version="al2-ami-sagemaker-inference-gpu-3-1",
             endpoint_name="qwen3-06b", tags=[{"Key": "CreatedBy", "Value": "<owner>"}])

The rest of the lifecycle is boto3 against five services: sagemaker and sagemaker-runtime for the endpoint, application-autoscaling for capacity, and cloudwatch plus logs for the rollback alarms and their teardown. Every call shape, waiter schedule, enum, and integer bound quoted below was verified against the installed boto3==1.43.50 service models on 2026-07-17 (the cloudwatch/logs teardown additions re-verified against boto3==1.34.46 on the same date); the calls themselves are unexecuted reference templates, since no AWS account was available. Shared clients for all the snippets:

import boto3

REGION = "us-east-1"
sm = boto3.client("sagemaker", region_name=REGION)
smr = boto3.client("sagemaker-runtime", region_name=REGION)
aas = boto3.client("application-autoscaling", region_name=REGION)
cloudwatch = boto3.client("cloudwatch", region_name=REGION)
logs = boto3.client("logs", region_name=REGION)

Wait for InService, and read the failure if there is one¶

A raw create_endpoint returns immediately; the endpoint spends minutes in Creating. The waiter is the primitive that separates a slow start from a dead one, and its error path is where the health-check failures from this page's failure-mode list surface in a script instead of in a dashboard.

# reference template (unexecuted against AWS). Waiter existence and polling
# schedule verified by executing get_waiter("endpoint_in_service") locally on
# boto3==1.43.50: polls every 30 s, up to 120 attempts (60 min ceiling), and
# exits early when EndpointStatus becomes Failed or the endpoint does not
# exist (ValidationException).
from botocore.exceptions import WaiterError


def wait_until_in_service(endpoint_name: str, max_minutes: int = 30) -> None:
    waiter = sm.get_waiter("endpoint_in_service")
    try:
        waiter.wait(EndpointName=endpoint_name,
                    WaiterConfig={"Delay": 30, "MaxAttempts": max_minutes * 2})
    except WaiterError as err:
        desc = sm.describe_endpoint(EndpointName=endpoint_name)
        raise RuntimeError(
            f"{endpoint_name}: {desc['EndpointStatus']}: "
            f"{desc.get('FailureReason', '<none recorded>')}") from err

FailureReason on describe_endpoint is where a deployment explains itself, when it explains itself at all; the cu130-without-AMI death reaches Failed with no container logs, and this wrapper turns that into a hard error with the status attached instead of a script hung for an hour. A Failed endpoint is not cleaned up for you: it, its config, and its model sit in the account until the teardown below removes them.

Invoke it¶

The transport is invoke_endpoint; the contract is the container's. The huggingface-vllm image serves vLLM's SageMaker router: GET/POST /ping for the health check and POST /invocations for inference, and the /invocations handler validates the JSON body against each enabled OpenAI schema in turn, dispatching to the first that accepts it (vLLM v0.21.0 source, vllm/entrypoints/sagemaker/api_router.py, the vLLM version inside the resolved image). A body with messages is therefore a chat completion. Two details from the same source tree: model is optional (model: str | None = None in ChatCompletionRequest), and max_tokens is deprecated there in favour of max_completion_tokens.

# reference template (unexecuted against AWS). invoke_endpoint request shape
# verified against boto3==1.43.50; payload contract verified against the vLLM
# v0.21.0 source shipped in the resolved image (schema-dispatched /invocations).
import json

resp = smr.invoke_endpoint(
    EndpointName="qwen3-06b",
    ContentType="application/json",
    Accept="application/json",
    Body=json.dumps({
        "messages": [{"role": "user", "content": "One sentence: what is a KV cache?"}],
        "max_completion_tokens": 128,
        "temperature": 0.2,
    }),
)
answer = json.loads(resp["Body"].read())["choices"][0]["message"]["content"]

The response body is an OpenAI-shape chat completion. The 60 s InvokeEndpoint ceiling from the pathway table applies here; anything slower belongs on the async path.

Autoscale the variant¶

An endpoint does not autoscale by itself; capacity lives in a different service. Two calls against application-autoscaling, not SageMaker: register the production variant as a scalable target, then attach a policy. The resource id format is endpoint/<endpoint-name>/variant/<variant-name> (AWS docs), and the v2 SDK's deploy names its variant AllTraffic unless told otherwise (the default in sagemaker==2.257.5).

# reference template (unexecuted against AWS). Shapes verified against
# boto3==1.43.50: the sagemaker ServiceNamespace, the
# sagemaker:variant:DesiredInstanceCount dimension, the
# SageMakerVariantInvocationsPerInstance predefined metric, DisableScaleIn,
# and the SuspendedState members are all present as written.
RESOURCE_ID = "endpoint/qwen3-06b/variant/AllTraffic"
DIMENSION = "sagemaker:variant:DesiredInstanceCount"

aas.register_scalable_target(
    ServiceNamespace="sagemaker",
    ResourceId=RESOURCE_ID,
    ScalableDimension=DIMENSION,
    MinCapacity=1,          # classic model-based variants cannot go below 1
    MaxCapacity=4,          # the skill default, and a per-endpoint spend cap
)
aas.put_scaling_policy(
    PolicyName="qwen3-06b-invocations-target",
    ServiceNamespace="sagemaker",
    ResourceId=RESOURCE_ID,
    ScalableDimension=DIMENSION,
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 20.0,      # invocations/min/instance, the defaults table
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance",
        },
        "ScaleOutCooldown": 60,   # out fast: a backlog is user-visible
        "ScaleInCooldown": 300,   # in slow: thrash costs more than idle minutes
    },
)

Scale-in protection has two knobs, both verified in the API shape. DisableScaleIn=True in the target-tracking configuration makes the policy scale out only, leaving scale-in to something else. SuspendedState={"DynamicScalingInSuspended": True} on register_scalable_target pauses scale-in without deleting anything, which is the right tool around a blue/green update, when metric noise on a shifting fleet could otherwise remove capacity mid-deployment; re-register with the flag off to resume. The scalable target and the policy are application-autoscaling resources, not properties of the endpoint, so the teardown below removes them explicitly and verifies, rather than assuming endpoint deletion took them along.

The async pathway, concretely¶

Async is the same model behind a different endpoint config: AsyncInferenceConfig on create_endpoint_config is the whole difference.

# reference template (unexecuted against AWS). AsyncInferenceConfig shape
# (OutputConfig.S3OutputPath / S3FailurePath / NotificationConfig,
# ClientConfig.MaxConcurrentInvocationsPerInstance) verified against
# boto3==1.43.50.
sm.create_endpoint_config(
    EndpointConfigName="qwen3-06b-async-cfg",
    ProductionVariants=[{
        "VariantName": "AllTraffic",
        "ModelName": "qwen3-06b",     # the same CreateModel as real-time
        "InstanceType": "ml.g5.xlarge",
        "InitialInstanceCount": 1,
        # Same rule as real-time: the resolved image is cu130.
        "InferenceAmiVersion": "al2-ami-sagemaker-inference-gpu-3-1",
    }],
    AsyncInferenceConfig={
        "OutputConfig": {
            "S3OutputPath": "s3://<BUCKET>/async-out/",
            "S3FailurePath": "s3://<BUCKET>/async-failures/",
            # Optional SNS instead of polling S3:
            # "NotificationConfig": {"SuccessTopic": "<ARN>", "ErrorTopic": "<ARN>"},
        },
        "ClientConfig": {"MaxConcurrentInvocationsPerInstance": 4},
    },
)
sm.create_endpoint(EndpointName="qwen3-06b-async",
                   EndpointConfigName="qwen3-06b-async-cfg",
                   Tags=[{"Key": "CreatedBy", "Value": "<owner>"}])

Invocation returns immediately; the answer lands in S3.

# reference template (unexecuted against AWS). invoke_endpoint_async shape
# verified against boto3==1.43.50; the numeric constraints in the comments are
# from the InvokeEndpointAsync API reference.
resp = smr.invoke_endpoint_async(
    EndpointName="qwen3-06b-async",
    ContentType="application/json",
    InputLocation="s3://<BUCKET>/async-in/req-0001.json",  # same JSON body as real-time
    InvocationTimeoutSeconds=3600,   # processing ceiling: default 900, max 3600
    RequestTTLSeconds=21600,         # queue-age ceiling: 60 to 21600, default 21600
)
# HTTP 202 with InferenceId, OutputLocation, and FailureLocation.

The queue is the semantics. The request payload lives in S3 (InputLocation); an inline Body exists but caps at 128,000 bytes and is mutually exclusive with InputLocation, so real LLM payloads go through S3. Accepted requests wait in an internal queue: a request older than RequestTTLSeconds expires unprocessed, and one running past InvocationTimeoutSeconds expires mid-flight, which is where the pathway table's one-hour ceiling (3,600 s against real-time's 60 s) comes from. Under overload the queue absorbs the burst and ApproximateBacklogSizePerInstance is the scaling signal; wake-from-zero needs the HasBacklogWithoutCapacity step-scaling policy described earlier, and the async register call differs from the real-time one in exactly one line: MinCapacity=0.

Blue/green update, with a rollback that actually triggers¶

An endpoint config is immutable, so a new image tag or env var means a new config plus update_endpoint. A bare update_endpoint already performs an all-at-once blue/green flip; DeploymentConfig is what buys canary or linear traffic shifting and automatic rollback.

Create the alarms before referencing them. An earlier revision of this section named AutoRollbackConfiguration alarms below without ever creating them, exactly the silent-no-op trap the prose next to it warns about. AWS/SageMaker's published per-variant metrics (EndpointName/VariantName dimensions) include Invocation5XXErrors and ModelLatency (microseconds); the two put_metric_alarm calls below create alarms under the exact names the DeploymentConfig block references, so the rollback config is wired to something real rather than an assumed-to-exist name:

# reference template (unexecuted against AWS). put_metric_alarm shape
# re-verified 2026-07-17 against the installed boto3==1.34.46 service
# model: AlarmName is the only client-side required member (operation_model
# required_members == ["AlarmName"]). Namespace/MetricName/Dimensions/
# ComparisonOperator/EvaluationPeriods/Threshold are required by the
# *service*, not the client, for the alarm to mean anything. Metric names
# and dimensions are AWS/SageMaker's documented per-variant metrics. Uses
# the shared `cloudwatch` client from the block above.
cloudwatch.put_metric_alarm(
    AlarmName="qwen3-06b-invocation-5xx",
    Namespace="AWS/SageMaker",
    MetricName="Invocation5XXErrors",
    Dimensions=[{"Name": "EndpointName", "Value": "qwen3-06b"},
                {"Name": "VariantName", "Value": "AllTraffic"}],
    Statistic="Sum",
    Period=60,
    EvaluationPeriods=3,
    Threshold=1,
    ComparisonOperator="GreaterThanOrEqualToThreshold",
    TreatMissingData="notBreaching",
)

cloudwatch.put_metric_alarm(
    AlarmName="qwen3-06b-model-latency",
    Namespace="AWS/SageMaker",
    MetricName="ModelLatency",
    Dimensions=[{"Name": "EndpointName", "Value": "qwen3-06b"},
                {"Name": "VariantName", "Value": "AllTraffic"}],
    Statistic="Average",
    Period=60,
    EvaluationPeriods=3,
    Threshold=2_000_000,  # microseconds: 2s: set from the endpoint's own measured baseline, not copied verbatim
    ComparisonOperator="GreaterThanThreshold",
    TreatMissingData="notBreaching",
)

VariantName="AllTraffic" is the default production variant name this page's earlier create_endpoint_config examples use; if a deployment names its variant differently, the alarm's dimension value must match it exactly or the alarm never receives datapoints and silently never fires, the same class of no-op as an alarm that does not exist at all.

# reference template (unexecuted against AWS). DeploymentConfig shape, the
# ALL_AT_ONCE/CANARY/LINEAR enum, CAPACITY_PERCENT/INSTANCE_COUNT, and the
# integer bounds in the comments verified against boto3==1.43.50; semantics
# from the deployment-guardrails documentation.
sm.update_endpoint(
    EndpointName="qwen3-06b",
    EndpointConfigName="qwen3-06b-cfg-v2",
    DeploymentConfig={
        "BlueGreenUpdatePolicy": {
            "TrafficRoutingConfiguration": {
                "Type": "CANARY",              # or LINEAR (LinearStepSize) / ALL_AT_ONCE
                "CanarySize": {"Type": "CAPACITY_PERCENT", "Value": 10},
                "WaitIntervalInSeconds": 600,  # canary baking period: 0 to 3600
            },
            "TerminationWaitInSeconds": 300,           # keep blue after the shift: 0 to 3600
            "MaximumExecutionTimeoutInSeconds": 3600,  # whole deployment: 600 to 28800
        },
        "AutoRollbackConfiguration": {"Alarms": [
            {"AlarmName": "qwen3-06b-invocation-5xx"},
            {"AlarmName": "qwen3-06b-model-latency"},
        ]},
    },
)

SageMaker provisions the green fleet, shifts the canary share, and watches the named CloudWatch alarms through every baking period; any alarm entering ALARM rolls all traffic back to blue. Three traps. First, rollback is only as real as the alarms: AutoRollbackConfiguration naming alarms that do not exist, or that do not measure the green fleet's failure mode, is a silent no-op, so wire it to the error and latency alarms from the production defaults table. Second, cost: AWS documents that deployments with multi-step shifting or baking periods bill both fleets for the duration of the update; only all-at-once with no baking bills one fleet, so a leisurely bake window on a p5 endpoint is priced accordingly. Third, watch describe_endpoint while it runs: EndpointStatus goes Updating then InService on success, and RollingBack (worst case UpdateRollbackFailed) when an alarm fired; all of these are in the status enum. Suspend scale-in for the duration (previous section), and note guardrails apply to real-time and async endpoints only, minus the features on AWS's exclusions list.

Teardown in dependency order, then prove the account is clean¶

Deletion order is the reverse of creation, plus the autoscaling resources that live outside SageMaker and the two CloudWatch resources the rollback section created. delete_endpoint is the call that stops the billing; the config and model are free metadata but leak confusion, a surviving scalable target is a live policy aimed at a dead resource id, and a surviving alarm or log group is a smaller but real leftover: a standard-resolution metric alarm is $0.10/month per metric (AWS CloudWatch pricing, checked 2026-07-17), and a log group with no retention policy set keeps every byte forever at standard log-storage rates.

# reference template (unexecuted against AWS). Every call shape verified
# against boto3==1.43.50, re-checked 2026-07-17 against the installed
# boto3==1.34.46 for the two new calls: cloudwatch.delete_alarms requires
# only AlarmNames (list[str]) and, per its docstring, silently skips names
# that do not exist rather than raising, so no alarm-side try/except is
# needed. logs.delete_log_group requires only logGroupName (str) and DOES
# raise logs.exceptions.ResourceNotFoundException when the group is
# missing, which is the expected case for the cu130-without-AMI failure
# mode above: that endpoint never wrote a log stream, so there is no group
# to delete. The endpoint_deleted waiter (polls every 30 s, up to 60
# attempts, succeeds when describe_endpoint raises ValidationException) and
# the rest of the deletion semantics are per the delete-resources docs.
def teardown(name: str, config: str, model: str, alarm_names: list[str]) -> None:
    resource_id = f"endpoint/{name}/variant/AllTraffic"
    for pol in aas.describe_scaling_policies(
            ServiceNamespace="sagemaker", ResourceId=resource_id)["ScalingPolicies"]:
        aas.delete_scaling_policy(
            PolicyName=pol["PolicyName"], ServiceNamespace="sagemaker",
            ResourceId=resource_id,
            ScalableDimension="sagemaker:variant:DesiredInstanceCount")
    try:
        aas.deregister_scalable_target(
            ServiceNamespace="sagemaker", ResourceId=resource_id,
            ScalableDimension="sagemaker:variant:DesiredInstanceCount")
    except aas.exceptions.ObjectNotFoundException:
        pass                                   # never registered: fine
    sm.delete_endpoint(EndpointName=name)      # billing stops with the endpoint
    sm.get_waiter("endpoint_deleted").wait(EndpointName=name)
    sm.delete_endpoint_config(EndpointConfigName=config)  # not auto-deleted
    sm.delete_model(ModelName=model)                      # not auto-deleted
    if alarm_names:
        cloudwatch.delete_alarms(AlarmNames=alarm_names)  # not auto-deleted
    try:
        logs.delete_log_group(logGroupName=f"/aws/sagemaker/Endpoints/{name}")
    except logs.exceptions.ResourceNotFoundException:
        pass                                   # no container ever logged: fine


# Call shape for this page's running example: the two alarms the rollback
# section created above.
# teardown("qwen3-06b", "qwen3-06b-cfg-v2", "qwen3-06b",
#          alarm_names=["qwen3-06b-invocation-5xx", "qwen3-06b-model-latency"])

Then verify, because trusting exit code 0 is how the $45,573 orphan happens. The CloudWatch checks belong in the same assertion as the SageMaker and autoscaling ones: an alarm with a dangling EndpointName dimension does not cost enough to notice by itself, but it is the same class of bug as the orphaned endpoint, just three orders of magnitude cheaper, and it is worth catching with the same sweep rather than a separate one someone forgets to run.

# reference template (unexecuted against AWS). list_* and
# describe_scalable_targets shapes verified against boto3==1.43.50;
# describe_alarms and describe_log_groups re-checked 2026-07-17 against
# boto3==1.34.46 (both take no required arguments; AlarmNames and
# logGroupNamePrefix are optional filters, used here to scope the check to
# this endpoint's resources).
leftovers = (
    sm.list_endpoints(NameContains="qwen3-06b")["Endpoints"]
    + sm.list_endpoint_configs(NameContains="qwen3-06b")["EndpointConfigs"]
    + sm.list_models(NameContains="qwen3-06b")["Models"]
    + aas.describe_scalable_targets(
        ServiceNamespace="sagemaker",
        ResourceIds=["endpoint/qwen3-06b/variant/AllTraffic"],
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    )["ScalableTargets"]
    + cloudwatch.describe_alarms(
        AlarmNames=["qwen3-06b-invocation-5xx", "qwen3-06b-model-latency"],
    )["MetricAlarms"]
    + logs.describe_log_groups(
        logGroupNamePrefix="/aws/sagemaker/Endpoints/qwen3-06b",
    )["logGroups"]
)
assert not leftovers, f"orphans: {leftovers}"

What deletion does not remove, per the AWS documentation: deleting a model "does not delete model artifacts, inference code, or the IAM role". Concretely, the S3 model artifacts and every async input, output, and failure payload stay in the bucket at S3 prices; the DLC image is AWS-owned, nothing to clean; the CloudWatch log groups under /aws/sagemaker/Endpoints/ and the alarms created for rollback persist until removed, which teardown() above now does with logs.delete_log_group(logGroupName=...) and cloudwatch.delete_alarms(AlarmNames=[...]). The autoscaling target and its policy are already covered by teardown()'s first two steps. The CreatedBy tag sweep from the maintenance section is the backstop for whatever this list misses; alarms and log groups do not currently take a CreatedBy tag in the templates above, so a name-prefix sweep (describe_alarms(AlarmNamePrefix=...), describe_log_groups(logGroupNamePrefix=...)) is the practical way to catch them if the endpoint name is not known.

How to develop with it¶

Container tags, SDK routing, AMI requirements, and account quotas change independently. Keep those facts in versioned deployment automation, verify them at run time, and review changes like code.

Hugging Face publishes six Apache-2.0 skills for this deployment workflow (huggingface/skills):

Skill	Job
`hf-cloud-sagemaker-deployment-planner`	Entry point: choose the pathway, check the quota before naming an instance.
`hf-cloud-aws-context-discovery`	Read-only discovery of account, region, and existing resources.
`hf-cloud-python-env-setup`	Isolated env on Python 3.10 to 3.12; never 3.13+, where ML wheels lag.
`hf-cloud-sagemaker-iam-preflight`	Find an existing execution role; create one only if none exists.
`hf-cloud-serving-image-selection`	Container family plus the current image URI. Never default to TGI.
`hf-cloud-sagemaker-production-defaults`	Deploy, autoscaling, alarms, tags, smoke test, teardown.

The skills separate read-only discovery, IAM preflight, image selection, deployment, and teardown. Treat their lookup tables as cached operational knowledge, not authority: quota and image checks still run against the target account and region. The packaging pattern is covered in use this KB as an agent skill and skill optimization.

The baseline failure the skills are built against is worth stating precisely, because it is this KB's own thesis in miniature. An unaided coding agent, asked to deploy a model, picked TGI, failed the ping health check, bumped the TGI version, redeployed, failed again, and only pivoted to vLLM after roughly 80 minutes and four failed endpoint attempts, each billing GPU time. Its README then recommended TGI to the next reader, propagating the stale fact. The failure came from stale deployment facts: an otherwise valid workflow repeatedly selected an archived container. That is the same argument this KB makes for Xid codes and driver pairings.

What the evidence for the skills is, and is not. The published report covers two models (Qwen/Qwen3-0.6B and google/diffusiongemma-26B-A4B-it, a multimodal block-diffusion Gemma with a 128-expert, top-8 MoE), one run each, self-reported by the author, with no repeated trials and no held-out set. It is a credible practitioner report, not a benchmark, and it should not be cited as a measured effect. The repository's apps/evals-leaderboard/ is not a skill-eval harness (its collect_evals.py scrapes model-index metadata from trending Hub models), so it is not evidence either. For how one would actually know whether a skill helps, the method is a held-out selection split with a strict-improvement gate that rejects ties: see skill optimization. Anyone adopting these skills should re-verify the facts they encode rather than trust them, and the skills themselves say to check quotas and image URIs rather than guess.

The skills ship .claude-plugin/, .cursor-plugin/, and gemini-extension.json manifests, so they are packaged for several harnesses, but no harness compatibility was tested here.

How to maintain it¶

Treat the image URI as perishable. Re-resolve it from the catalog on every deploy and pin the resolved tag in the model definition. Never :latest.
Re-check the archived-upstream assumption. TGI is archived today; the SDK routes to vLLM today. Both are facts with dates, and both belong in a skill or a pinned doc that is re-verified, not in someone's habit.
Sweep for orphans by tag. The production-defaults scripts stamp a CreatedBy tag on every resource precisely so a forgotten endpoint can be found and killed. Run the sweep on a schedule; the p5 orphan above is $45,573.12 for 30 days of nothing.
Re-run the teardown path, and verify it. A teardown script that deletes the endpoint but leaves the endpoint-config and model behind is tidy but harmless; one that leaves the endpoint behind is a five-figure bug. Verify with describe-endpoint returning ValidationException, not by trusting exit code 0.
Data capture is off by default in the shipped defaults. Turning it on writes every request and response to S3, which is a privacy and cost decision, not a debugging default (security and multi-tenancy).

How to run it in production¶

The defaults below are the shipped values from the production-defaults skill's deploy.py, read from the repository rather than from the blog post:

Control	Default
Autoscaling policy	Target tracking on `SageMakerVariantInvocationsPerInstance`
Target	20 invocations/min/instance
Capacity	min 1, max 4
Cooldowns	60 s scale-out, 300 s scale-in
Alarm: latency	`ModelLatency` > 30,000 ms
Alarm: errors	`Invocation5XXErrors` > 5 in 5 min
Alarm: overhead	`OverheadLatency` > 2,000 ms
Tagging	`CreatedBy` on every resource, for orphan sweeps
Data capture	Off

The asymmetric cooldowns are correct and deliberate: scale out fast (60 s) because a backlog is user-visible, scale in slowly (300 s) because thrashing a GPU instance costs more than the idle minutes it saves. OverheadLatency is request-path service overhead: end-to-end invocation latency minus ModelLatency, reported in microseconds. It does not measure endpoint provisioning or model load for classic instance-based real-time endpoints; use endpoint state transitions, deployment events, and container logs for those. Serverless OverheadLatency is the documented exception that includes cold-start time. Alarm on both latency metrics, but interpret them accordingly (observability, SLO/SLI catalog).

Blast radius: the agent's credentials, not the execution role¶

Two IAM surfaces get conflated, and only one of them is actually controlled.

The execution role (what the endpoint may do at runtime) is well scoped. The IAM preflight skill ships a minimum-permissions.json granting exactly: s3:GetObject and s3:ListBucket on the model bucket, four ECR pull actions (GetAuthorizationToken, BatchCheckLayerAvailability, GetDownloadUrlForLayer, BatchGetImage), CloudWatch Logs on /aws/sagemaker/*, and cloudwatch:PutMetricData. That is a tight, defensible role.

The agent's own credentials (what it may create) are bounded by nothing. An agent that can call CreateEndpoint can select ml.p5.48xlarge at $63.30/hr, which is roughly $46,200/month, on a single call. The controls usually described (the agent voluntarily pausing to confirm, and a teardown script it is trusted to run) are conventions, not enforced controls. A convention is not a spend limit. The controls that actually bind:

IAM condition keys on sagemaker:InstanceTypes, allow-listing the shapes an agent may launch. This is the only control that stops the p5 call at the API.
Service quotas as a hard cap. Leaving GPU endpoint quotas at 0 and raising them deliberately, per type, is a feature and not an obstacle.
AWS Budgets with alerts and, where possible, actions.
A mandatory human approval gate before the first billable call, enforced by the harness (tool gating), not requested politely in a prompt.
CreatedBy tag sweeps to catch what still slips through.

The skill mechanism itself carries supply-chain risk: a skill is executable context that can grant itself broad tool access, and Anthropic's documentation warns to "review project skills before trusting a repository, since a skill can grant itself broad tool access". Read a third-party skill's allowed-tools and its scripts/ before installing it, with the same suspicion applied to any code that will run with cloud credentials (security and multi-tenancy, governing self-modifying agents).

Failure modes¶

Serverless chosen for an LLM. It cannot work: GPUs are an explicit feature exclusion, and RAM caps at 6144 MB. Fails at deploy, or wastes a day before it does.
TGI selected from memory or from a stale tutorial. Archived upstream; produces an opaque health-check failure that looks like a config error and invites version-bumping instead of a container change.
SM_VLLM_HOST left unset. vLLM binds localhost, the ping probe fails, the container dies before writing useful logs. Reads as a mystery.
A cu130 image deployed without InferenceAmiVersion. The worst failure on this page, because there is nothing to read: the container dies on startup and no CloudWatch log stream is ever created. An empty log group looks identical to a quota, networking, or account problem, so it routinely sends people down the wrong diagnostic path. The AMI is keyed to the image's CUDA build (cu130 or higher needs the override, cu129 and lower do not), so it changes when the tag changes, which is exactly the kind of expiring fact that belongs in a skill.
Classic real-time endpoint mistaken for free. A model-based production variant keeps at least one instance, so zero traffic still bills a full instance-month ($1,027.84 on ml.g5.xlarge in the dated example). Inference-component scale-to-zero is a different deployment path and fails wake requests while capacity is zero.
Async assumed to be a cheaper hour. It is the same hourly rate; only scale-to-zero saves money. Wake-from-zero left to target tracking alone. Without the optional HasBacklogWithoutCapacity step-scaling policy, a scaled-to-zero endpoint stays at zero until the backlog exceeds the target value, so a thin trickle of requests queues for a long time before anything starts.
Cost quoted from the EC2 price list for a SageMaker endpoint. Understates the bill by the managed premium: the widely repeated "$730/month" for an idle ml.g5.xlarge is the EC2 g5.xlarge rate ($1.006/hr x 730 h = $734). The SageMaker Hosting rate is $1.408/hr, so the real figure is $1,028, about 40% higher.
Instance recommended without a quota check. GPU endpoint quotas default to 0 in many accounts; the deploy dies on ResourceLimitExceeded.
Orphaned endpoint. No CreatedBy tag, no sweep, no budget alarm. A forgotten p5 is $45,573.12 per 30 days.
An agent with unbounded CreateEndpoint rights. One call can incur costs at a run rate of roughly $46,200 per 730-hour month. Bound it with IAM condition keys and quotas, not with a polite prompt.

References¶

Deploying models to SageMaker with agent skills (Hugging Face blog, the source practitioner report): https://huggingface.co/blog/hf-dwarez/hf-aws-sagemaker-skills
huggingface/skills (Apache-2.0; the six hf-cloud-* skills): https://github.com/huggingface/skills
AWS Deep Learning Containers, available images (resolve image URIs here): https://aws.github.io/deep-learning-containers/reference/available_images/
Hugging Face on Amazon SageMaker: https://huggingface.co/docs/sagemaker/index
SageMaker real-time endpoints: https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html
Scale real-time inference components to zero (failed wake invocation and several-minute provisioning delay): https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling-zero-instances.html
SageMaker endpoint invocation metrics (ModelLatency, OverheadLatency): https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipeline-logs-metrics.html
SageMaker asynchronous inference: https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference.html
Autoscaling an async endpoint to zero (HasBacklogWithoutCapacity): https://docs.aws.amazon.com/sagemaker/latest/dg/async-inference-autoscale.html
SageMaker Serverless Inference, incl. the GPU feature exclusion: https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html
SageMaker batch transform: https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html
SageMaker endpoint auto scaling: https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html
ProductionVariant API reference (the InferenceAmiVersion enum and its driver/CUDA versions): https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html
InvokeEndpoint API reference (the 60 s model-response ceiling): https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html
InvokeEndpointAsync API reference (S3 InputLocation, 128,000-byte inline Body, TTL and timeout ranges): https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpointAsync.html
Register a variant with Application Auto Scaling (the endpoint/<name>/variant/<name> resource id): https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling-add-policy.html
RegisterScalableTarget API reference (Application Auto Scaling): https://docs.aws.amazon.com/autoscaling/application/APIReference/API_RegisterScalableTarget.html
Deployment guardrails (blue/green, canary, linear, auto-rollback): https://docs.aws.amazon.com/sagemaker/latest/dg/deployment-guardrails.html
Blue/green deployments (baking period; the both-fleets billing warning): https://docs.aws.amazon.com/sagemaker/latest/dg/deployment-guardrails-blue-green.html
UpdateEndpoint API reference (DeploymentConfig): https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateEndpoint.html
Delete endpoints and resources (what deletion does and does not remove): https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-delete-resources.html
boto3 SageMaker EndpointInService waiter reference: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker/waiter/EndpointInService.html
Amazon CloudWatch pricing (the $0.10/metric/month standard-resolution alarm rate cited in teardown): https://aws.amazon.com/cloudwatch/pricing/
DeleteAlarms API reference (Amazon CloudWatch; nonexistent alarm names are skipped, not an error): https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_DeleteAlarms.html
DeleteLogGroup API reference (Amazon CloudWatch Logs): https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DeleteLogGroup.html
vLLM v0.21.0 SageMaker router source (the /invocations schema dispatch): https://github.com/vllm-project/vllm/blob/v0.21.0/vllm/entrypoints/sagemaker/api_router.py
Amazon SageMaker AI pricing: https://aws.amazon.com/sagemaker/ai/pricing/
sagemaker-python-sdk PR #5960, "change: routing logic change for Hugging Face DLCs" (merged 2026-06-26): https://github.com/aws/sagemaker-python-sdk/pull/5960
huggingface/text-generation-inference (archived 2026-03-21): https://github.com/huggingface/text-generation-inference
Agent Skills standard: https://agentskills.io
Claude Code skills documentation (incl. the review-before-trusting warning): https://code.claude.com/docs/en/skills
AWS Budgets: https://aws.amazon.com/aws-cost-management/aws-budgets/