Your team spent three months writing production-grade inference code. The model is tuned. The pipeline is tested. The Kubernetes manifests are ready. And now you're waiting -- not for a code review, not for a staging sign-off, but for a GPU that doesn't exist yet in your account. Meanwhile, the H100s you do have are sitting 95% idle.
That's not a hypothetical. According to Cast AI's 2026 platform telemetry across 23,000+ production clusters, average GPU utilization sits at 5%. On AKS it's 2%. On EKS, 5%. On GKE, 6%. A 20-GPU H100 cluster running at that utilization rate wastes roughly ~$168,000 per month (based on AWS p5.48xlarge on-demand pricing of ~$12.29/hr per H100). The GPUs are there. The money is leaving the account. The work isn't getting done.
This post is about why that happens, what makes the supply side worse than most teams realize, and what the actual fix looks like in a Kubernetes environment.
The Supply Crisis Is Structural, Not Temporary
The shortage of H100 and H200 hardware isn't a blip from pandemic-era logistics. It's baked into the physics of manufacturing these chips.
H100 and H200 production depends on TSMC's CoWoS (Chip on Wafer on Substrate) advanced packaging process. CoWoS capacity is constrained at the fab level -- you can't just spin up another CoWoS line in a quarter. Both H100 and H200 face HBM supply constraints, though from different generations: the H100 SXM5 (used in cloud data centers) uses HBM3; H200 uses HBM3e. All three HBM generations face tight production capacity from the same handful of manufacturers -- SK Hynix, Samsung (News - Alert), and Micron. Both constraints compound each other.
The result: H100 and H200 hardware lead times from secondary-market resellers run 36 to 52 weeks. Direct purchase through NVIDIA (News - Alert) allocates supply priority to hyperscale buyers first. NVIDIA directed roughly 60% of chip production to enterprise AI clients in Q1 2025. If you're not AWS, Google, or Microsoft (News - Alert), you're in the back of the line.
The Big-5 hyperscalers are spending roughly $725B in capex in 2026 -- up 77% from 2025. That spending secures their supply position. JP Morgan (News - Alert)'s April 2026 analysis puts the global AI compute supply-demand gap at 15-30% for 2026, persisting into 2027. More money in the market doesn't close that gap when the fab capacity ceiling is physical.
AWS raised H200 Capacity Block prices 15% in January 2026. That's the first GPU price increase in over a decade of cloud GPU availability. When the cloud providers start raising prices on reserved compute, they're signaling that demand has outrun their ability to absorb it through efficiency gains alone.
Cloud Doesn't Save You Either
The assumption that you can always get GPU capacity on demand from AWS, GCP, or Azure is wrong. It was arguably never true for high-end GPUs, and it's less true now than it was 18 months ago.
SemiAnalysis reported in April 2026 that half of GPU cloud providers are completely sold out of H100 and H200 capacity. Reserved capacity on the major hyperscalers is pre-sold 6 or more months out. When you try to provision on-demand, you're competing for whatever wasn't reserved.
On July 29, 2025, an Azure East US capacity shortfall prevented customers from creating or starting VMs. This wasn't a service outage in the traditional sense -- Azure's infrastructure was running fine. The capacity simply wasn't there. Customers with running workloads were fine. Customers trying to scale up or launch new instances were blocked.
AWS Capacity Blocks -- the mechanism for reserving GPU capacity in advance -- have become a reservation lottery. You submit a request for a future time window. If the block is available, you lock it in. If not, you wait and try again. For teams that need to scale training runs predictably, this is a real operational problem. Your deployment timeline depends on winning the lottery.
The multi-cloud instinct -- "if AWS doesn't have it, try GCP" -- is correct directionally but manually exhausting. Region-to-region price spreads are wide. Teams that provision in the most cost-favorable regions at a given time save 2-5x versus average GPU market price. But that requires real-time visibility across clouds, automated fallback logic, and workload portability. Most teams don't have that infrastructure built.
What the Data Says About Deployment Delays
In January 2025, Civo surveyed 1,367 AI/ML professionals. 84.7% of organizations reported project delays due to GPU availability. 38.5% of those organizations had delays of 3 to 6 months.
That's not a niche problem. That's the majority of teams doing serious AI work reporting that GPU access -- not model quality, not infrastructure bugs, not data pipelines -- is what's slipping their timelines.
Across enterprise deployments, the pattern is consistent: GPU availability is the single leading cause of AI project delays. It's not a procurement issue you can route around with a bigger budget. Throwing money at the problem doesn't work when the hardware doesn't exist to buy.
What actually slips in these scenarios is queue position. Your code is done. Your team is ready. You're waiting for capacity to become available. Every week that GPU access is delayed is a week of engineer time spent on standby, a week of competitive ground potentially lost, and a week of paying for infrastructure that isn't producing output.
Why Teams Hoard GPUs -- and Why It Makes Things Worse
GPU scarcity has a side effect that compounds the problem: the scarcity itself causes the waste that makes the scarcity worse.
When teams can't get GPUs reliably, they over-provision when they can. They reserve more than they need because they're afraid of being caught without capacity. Those reserved GPUs sit idle when they're not running training jobs. The team that grabbed 20 H100s because they were available is now paying for 20 H100s while using the equivalent of 1.
At 5% average utilization, 95% of paid GPU capacity is idle at any given moment across 23,000+ clusters. That idle capacity is real hardware tied up in reservations, unavailable to other buyers, and burning budget at full price. The hoarding reflex is individually rational and collectively destructive.
That 5% utilization figure isn't just inefficiency -- it's the financial logic that makes scarcity worse. The team that provisioned 20 H100s "just in case" is now sitting on ~$168,000/month in locked capacity that someone else can't access. Scale that across an organization with multiple teams doing the same thing, and you're not just wasting money -- you're holding capacity that someone else's deployment is waiting for, while your own timeline slips because their team did the same thing first.
Fewer than 2% of GPU workloads run on Spot Instances, according to Cast AI's 2026 platform telemetry. That's a striking number given that H100 Spot prices in eu-north-1 dropped 88% between January 2024 and September 2025 -- from $105.20 to $12.16 per hour. The savings are available. The infrastructure to use them safely isn't in place for most teams.
The Workaround Playbook
There's no single fix. The supply problem and the utilization problem require different interventions. Here's what actually works.
Fractional GPU: MIG vs. Time-Slicing
Most workloads don't need a full H100. Most inference jobs for models up to ~13B parameters, small fine-tuning runs, and development environments fit in a fraction of an H100. Large model inference (70B+) requires multiple full GPUs or specialized multi-node setups. Running smaller workloads on a full GPU is waste by definition.
NVIDIA's Multi-Instance GPU (MIG) on H100 creates up to 7 isolated compute instances -- with the 1g.10gb profile, that's 7 instances of 10GB GPU memory each. Each instance has dedicated memory and compute slices, with hardware-level isolation. Latency-sensitive workloads get predictable performance. MIG is the right tool when workloads need memory isolation or when you can't tolerate interference between tenants.
Important: enabling MIG or changing MIG profiles requires draining the node first -- profile changes are not hot-swappable. The GPU device plugin must be restarted after new profiles are applied; existing pods on the node must be evicted before any profile change takes effect.
In Kubernetes, both MIG and time-slicing are configured via NVIDIA's GPU Operator -- the operator manages the device plugin lifecycle, MIG profile configuration, and node labeling. Without it, manual configuration is per-node and doesn't survive pod rescheduling.
Time-slicing takes a different approach: it multiplexes GPU access across up to 48 replicas by context-switching between them. There's no memory isolation -- each replica sees the full GPU memory. This creates an OOM risk if multiple replicas attempt large memory allocations simultaneously. Set replica counts with workload memory requirements in mind, or set hard memory limits per replica at the application level. Time-slicing is appropriate for batch workloads, dev environments, and cases where you need many concurrent lightweight processes. At 48 replicas per GPU, you're turning one H100 into 48 addressable compute units in Kubernetes.
Note: 48 replicas is the theoretical upper limit -- at 80GB total VRAM, that's ~1.6GB per replica. This configuration is appropriate for embedding models, tokenizer services, and development environments. Production inference workloads typically use 4-16 replicas per GPU depending on the model's VRAM footprint.
Choosing between them depends on your workload mix. MIG for isolation and predictability. Time-slicing for density on bursty or lightweight jobs.
Multi-Cloud and Cross-Region Sourcing
No single cloud region has guaranteed GPU availability at all times. The Azure East US incident from July 2025 is a concrete example -- customers couldn't start VMs not because Azure was broken, but because the capacity ceiling was hit in that region.
Cross-region and cross-cloud sourcing spreads that risk. The price spread across regions and clouds is wide enough that where you provision matters as much as which GPU you pick. Teams provisioning in the most cost-favorable regions save 2-5x versus average GPU market price.
Spot Automation with Fallback
The 88% price drop in H100 Spot pricing in eu-north-1 between early 2024 and late 2025 represents real money left on the table by teams running everything on On-Demand. Spot is interruptible, which means you need interruption handling and fallback logic. That engineering work pays for itself quickly when the underlying compute costs drop by an order of magnitude.
For training jobs: checkpoint frequently, use Kubernetes job restart policies, and configure fallback to On-Demand when Spot isn't available. For inference: stateless replicas are inherently more tolerant of interruption. The tooling to automate this isn't complicated -- the gap is that most teams haven't built it.
Kubernetes DRA and GPU-Aware Bin-Packing
Kubernetes Dynamic Resource Allocation (DRA), available in beta from K8s 1.31+, gives schedulers structured access to GPU resource details -- memory, topology, MIG profile -- at scheduling time. This enables smarter placement decisions without custom device plugins hacked around the scheduler.
GPU-aware bin-packing consolidates workloads onto fewer nodes rather than spreading them across the cluster. Consolidation keeps more nodes fully loaded and fewer nodes half-empty. At cluster scale, that's the difference between 5% utilization and something materially higher.
How Cast AI Addresses This
Cast AI's approach to this problem has two sides: supply access and demand optimization. You need both. Solving only the supply side means you're still burning money on idle capacity. Solving only the demand side doesn't help when the GPUs you need aren't available.
On the supply side, OMNI Compute extends your existing EKS, GKE, or AKS cluster to source GPU capacity across AWS, GCP, and OCI regions without changes to your application code. Your workloads stay the same. The scheduler gains visibility into available capacity across providers and regions, and routes jobs to where capacity exists at the best current price. When one region is sold out -- as East US was in July 2025 -- workloads route to where capacity is available. The 2-5x price spread across regions becomes an active optimization, not a research project your team has to run manually.
On the demand side, Cast AI automates the fractional GPU mechanics that most teams know they should use but don't have the operational capacity to manage manually. MIG lifecycle management handles profile creation and teardown without manual intervention. Time-slicing configuration exposes up to 48 replicas per GPU to the Kubernetes scheduler. GPU-aware bin-packing places workloads to maximize utilization on the nodes you have. Spot automation with fallback logic runs workloads on interruptible instances and switches to On-Demand when Spot isn't available -- without your team building and maintaining that logic.
The result, at the top of Cast AI's dataset: a 136-node H200 fleet running at 49% utilization through fractional allocation. That's 10x the 5% average. The same cluster at 5% utilization would need roughly 10x as many nodes to do the same work -- which, in a supply-constrained market, means a very long wait or a very large budget.
Real-time cost attribution per workload and namespace makes the utilization story visible to the teams that need to act on it. When a namespace is consistently running at 3% GPU utilization, the data is there to have that conversation and right-size the allocation.
Yotpo achieved 40% cloud cost savings with Cast AI. Akamai (News - Alert) achieved 40-70% cloud savings. Those numbers come from combining supply-side sourcing with demand-side optimization -- getting better rates on the capacity you use, and using the capacity you have more fully.
What the Teams That Ship Are Actually Doing
The teams shipping on schedule aren't waiting for the market to rebalance. They're treating the supply constraint as permanent and optimizing around it -- running 49% utilization instead of 5%, sourcing capacity from wherever it's available, and using fractional allocation to squeeze 7-48 addressable compute units out of every GPU they do have. That's not a technology bet. That's operational discipline applied to a structural constraint.
The fab constraints driving this scarcity are real and multi-year. CoWoS packaging lines and HBM production capacity can't be scaled in a fiscal quarter -- no amount of hyperscaler capex changes that near-term reality. Supply will improve eventually. But "eventually" isn't a deployment timeline.
Your deployment timeline doesn't slip because your code is wrong. It slips because the GPU that would run it is either idle in someone else's over-provisioned cluster, or doesn't exist yet in your region. Both of those are solvable problems -- and the teams solving them aren't waiting in line.
See how Cast AI optimizes GPU utilization across your clusters
