What my own Serverless runs say about cold starts, FlashBoot, cost per job, and how emberserve compares with worker-vllm on the same GPU. Every number below is a row in a gold table, and every gold row traces back to a landed source file.
One dot per successful cold-labelled request. Hollow dots were answered in under s: a FlashBoot resume or a worker that was still warm. The tick is the lane's median.
From vLLM's own log lines: weights downloaded and loaded, torch.compile, CUDA-graph capture (summed over its passes — vLLM 0.30 does two), the rest of engine init, then the API server coming up. One bar per boot, sorted by total. Two boots hit a warm compile cache. The 74–81 s graph captures are the three Sep 23 boots on an unidentified 24 GB-tier GPU; the capture mode (FULL + PIECEWISE) is the same across boots. Boots whose export has no vllm serve line (the console missed it, or the log starts mid-boot) have no wall-clock bar and appear in the table only.
Qwen3-8B on an RTX 4090, full boots only (FlashBoot hits excluded). pooled rows mix every run; the cohort rows beneath split them by host state and data era, so a pooled median is never quoted alone.
Share of successful cold-labelled requests answered under the threshold, per endpoint and setting. The data cannot tell a FlashBoot resume from a worker that was still warm. failed counts the cold attempts the denominator leaves out, so a 100 % on one boot reads as 1-of-1-after-N-failures.
Median of (delay + exec) × $/hr, Qwen3-8B on an RTX 4090 at $1.10/hr, pooled. A comparison metric, not billed time: Runpod bills worker start, execution and idle per worker.
The Pulse scoring job, 50 articles per batch, cost from each backend's own price table.
Median TTFT per request rate for each emberserve Serverless sweep. ∞ = unpaced: all 200 requests queued at t=0, capped by max concurrency where the run set one (hover a point). 7b_… systems ran Qwen2.5-7B, the rest Qwen2.5-0.5B; queue endpoints add a hop and a worker pool, the load balancer talks to the server directly.
The same three images are re-pointed at each GPU pool in turn, ten cold starts per cell spread over four six-hour slots a day. A cell's median therefore mixes two things: how long the scheduler took to find a host and get the image onto it, and how long the engine took to boot once it ran. The emberserve images report that split from inside the worker; worker-vllm does not.
Rows are GPU pools, columns the image and FlashBoot setting; the number is the cell's median delay before the job ran (queue + placement + pull + boot), darker is slower, log-scaled. Hover for the spread and the number of cold starts behind it.
emberserve cells only (the handler writes a timeline). Each bar is a cell's median split: the seconds from submit until the container existed on a host — scheduling, image pull, create — and the seconds the engine then took to be ready. Baking the weights in makes the second part short and the first part long: a 27 GB image is rarely cached where the scheduler lands.
Each row joins the FlashBoot-off and FlashBoot-on medians of the same GPU and image. At a six-hour cadence there are no sub-5 s resumes; where the on-dot sits far left of the off-dot, FlashBoot kept the image warm on a host the scheduler could reuse.
One dot per gold row (cell × six-hour UTC slot): that slot's median. Hollow dots are FlashBoot-on cells. A lane whose dots spread across the axis is a pool whose availability changes with the hour.
Median of (delay + exec) × the pool's $/hr — a request-duration proxy, not billed time. It over-counts the cells whose delay is mostly placement and image pull, which Runpod does not bill, so a baked-image cell with FlashBoot off looks dearer here than it was; hover for what the cell's endpoint was actually billed in the hours it ran (shared across the cells on that image). Read it as "what the caller waited for, priced", not as the bill.
For cold-labelled requests, the time since the previous request on the same endpoint, bucketed, against the share answered under the hit threshold. The campaign's six-hour gaps and the generator's one-hour gaps both sit past the window where a worker resumes.
One row per endpoint per UTC day from the poller's snapshots: configuration at the last poll, workers seen and the most active at once, worker appearances (≈ scale-ups, derived by diffing consecutive polls), the endpoint's own job counters moved, and what Runpod billed for the day.