Where the model behind your product should run in 2026: managed APIs vs open weights, AWS native vs Modal, RunPod and Cloud Run GPU, vLLM and SGLang, quantized CPU inference on Spot, batch pricing, prod evals, and infra as code.

Links

Try a walking desk - stay healthy & sharp while you learn & code
More OCDevel shows - this one has siblings, each on its own subject and produced the same way

The infrastructure episode, standalone: where the model behind your product runs in 2026, and what each choice costs. Managed APIs versus self-hosted open weights, AWS native (Bedrock, SageMaker) versus serverless GPU platforms (Modal, RunPod, Cloud Run GPU) versus bare GPUs, the inference servers (vLLM, SGLang, llama.cpp), quantized CPU inference on Spot, queues and batch pricing, observability and evals, and a minute on infrastructure as code. The coding-agent workflow is covered in Vibe Coding in 2026 through Agentic Software Engineering; agent architecture in AI Agents in 2026.

Calling a model vs running one

The default for a product builder is a managed API: cost is linear in usage with no floor, while a self-hosted GPU costs the same idle as busy.
Four things flip the default: sustained volume, privacy or data residency, a strict latency shape, or a model only available as open weights. Move one workload when a flip applies, not the whole stack.
The cloud-hosted middle path: Amazon Bedrock's model catalog now spans Anthropic, OpenAI (including the open-weight gpt-oss models), Meta Llama, Mistral, DeepSeek, Qwen and more, and Custom Model Import serves your own safetensors weights on demand with no instance to manage.
The framing on rent vs own is still a16z's "Navigating the High Cost of AI Compute": renting wins for almost everyone; owning pays only at large steady spend.

AWS native

Lambda and Fargate still have no GPUs; the Lambda quotas page has no accelerator dimension and the Fargate FAQ sends GPU work to EC2. Lambda offers 10 GB memory, 15-minute timeouts, 10 GB container images, and SnapStart for Python.
SageMaker AI: real-time endpoints scale to zero only via inference components; asynchronous inference queues S3 payloads up to 1 GB and scales to zero; serverless inference is CPU-only; batch transform for offline scoring.
AWS Batch GPU jobs on Spot compute environments are the cheapest bursty GPU on AWS, with no scheduler surcharge.
Aurora PostgreSQL ships pgvector 0.8.2 with HNSW, and Aurora Serverless v2 auto-pause drops compute to zero when idle. Bedrock Knowledge Bases can use that Aurora as its store. Aurora DSQL has no extensions, so no pgvector.

Serverless GPU platforms

The category: ship a container or function, pay per GPU-second, scale to zero. Differentiators are cold start and warm-pool control.
Modal: Python-native, per-second billing, T4 through B300; cold starts around a second with memory snapshots and minimum-container knobs; batch via Function.map.
RunPod Serverless: queue-based or load-balanced endpoints, FlashBoot, async run and webhooks, wide GPU list from economy cards to B200.
Cloud Run GPU: GA, NVIDIA L4 and RTX PRO 6000 Blackwell, scales to zero, instance-based billing required.
Baseten with the open-source Truss packager: operated production deployments for custom models.
Replicate joined Cloudflare in November 2025; Cloudflare Workers AI remains a curated open-model catalog, not a bring-your-own-weights host.
fal: generative media inference first, serverless GPUs on the side.
Status changes: Lambda's inference API is winding down in favor of GPU instances; Fly.io's GPU Machines appear withdrawn (docs removed, no GPU line on pricing).

Bare GPUs, neoclouds, and Spot

Hyperscaler families: AWS P6, Google Cloud GPUs, Azure ND/NC sizes. Neoclouds: CoreWeave, Nebius, Lambda, Crusoe; Vast.ai as a marketplace with interruptible pricing.
A reserved box beats serverless when the GPU is busy most of the hour.
EC2 Spot advertises up to 90 percent off on-demand, with a two-minute interruption notice. The discount is paid for with checkpointing, retry-on-reclaim, and instance-pool diversity using the SPOT_PRICE_CAPACITY_OPTIMIZED allocation strategy AWS recommends.

Inference servers

vLLM: PagedAttention, continuous batching, OpenAI- and Anthropic-compatible server, V1 engine, now under the PyTorch Foundation.
SGLang: RadixAttention prefix reuse, releases every two weeks, production use at xAI, NVIDIA, Cursor and the major clouds.
Hugging Face TGI was archived in March 2026; its notice points to vLLM and SGLang.
TensorRT-LLM for NVIDIA-only maximum throughput; NVIDIA Dynamo orchestrates disaggregated prefill/decode across engines.
llama.cpp for GGUF on CPUs, Apple Silicon and ARM with KleidiAI; Ollama now runs its own engine on GGML for multimodal models.
Also: Triton Inference Server for non-LLM models, Ray Serve for composition, OpenLLM, LMDeploy, MLC-LLM.

Quantization for CPU inference

Quantized weights cut memory and, since batch-1 decode is memory-bandwidth bound, cut time per token; integer matrix instructions (ARM dot-product and MMLA, Intel VNNI) do the multiplies.
GGUF quant naming: bits per weight, k-quant mixed precision, size variant; Q4_K_M takes Llama 3.1 8B from about 32 GB to about 5 GB.
AWS Graviton guide for llama.cpp recommends Graviton3 and later for their BF16 and MMLA instructions and publishes a prebuilt llama-server container.
vLLM's CPU backend covers x86 AVX512 and ARM. ONNX Runtime quantization for classic models, with the caveat that gains need VNNI or ARM dot-product hardware.
Practice described in the episode: a quantized Qwen3-TTS model served by a C port on Graviton Spot instances via AWS Batch, checkpointing chunks to S3.

Queues, batch, and real-time

If no human waits, queue it: SQS with Lambda needs idempotency and partial-batch failure reporting; SageMaker async, Modal map, and RunPod async run are the same shape.
Batch pricing is 50 percent off across OpenAI Batch, Anthropic Message Batches, and Bedrock batch inference, with separate rate limits and a 24-hour window.
Bedrock prompt caching bills cache reads at a large discount within a short TTL; put stable prefix content first.
Real-time paths need warm containers ("scale to zero" becomes "scale to one"), continuous batching, and streaming.

Observability and evals in prod

Trace every call with prompt, response, tokens, latency, cost. The OpenTelemetry GenAI conventions moved to their own repo; OpenLLMetry emits them.
Tools: Langfuse (open source, part of ClickHouse since January 2026), Arize Phoenix (Elastic License), LangSmith, W&B Weave, Braintrust, Helicone as a proxy.
Production evals score sampled traces with a judge model: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena found over 80 percent agreement with humans and named position, verbosity and self-enhancement bias.
Classic drift: Evidently covers tabular drift and LLM metrics; WhyLabs ceased operations but open-sourced whylogs and langkit.

Infrastructure as code

SST runs on Pulumi with Terraform providers bridged; Pulumi for languages without SST's web opinions; Terraform under BSL since 2023 with OpenTofu as the open fork. Keep ML infra in the product's repo and deploy.

Shout-outs

Fast custom silicon APIs: Groq, Cerebras. Open-model inference clouds: Together AI, Fireworks, DeepInfra. Aggregators: OpenRouter, Hugging Face Inference Providers; dedicated Inference Endpoints. AWS Trainium. SageMaker JumpStart delisted models in March 2026, so pin weights in your own bucket.

Related episodes

More Gnothi-generated shows on AI, coding, video and business: ocdevel.com/moremlg.

Patreon Unlock: Offices and Bosses S2E6: Necromancer Bandmate

Hello From The Magic Tavern

Dripfang's Backstory and Side Hustles

The AI-powered Podcast Player