Reduce inference cold starts on Amazon SageMaker HyperPod with model caching

What changed

AWS launched model caching for Amazon SageMaker Inference on HyperPod, pre-loading container images and model weights onto local NVMe storage to reduce deployment cold starts.

Why you should care

Local NVMe pre-loading reduces large LLM scale-out delays from tens of minutes to seconds.

Your move

Test. Evaluate model caching on SageMaker HyperPod to verify cold start improvements for your workloads.

Evidence

ConfirmedAWS Machine Learning ↗

What to watch next

User reports and further AWS documentation on cache management overhead and storage costs at scale.

Event

release

Event date

Sep 10, 2026

Relevant to

General AI readers