Many teams add AI to a product by calling a frontier model through a vendor’s API. Frontier models are the large models behind the popular chat assistants. The vendor runs the hardware. The team sends text, gets text back, and pays for every token, the small chunks of text a model reads and writes.
LiteLedger analyzes conversations between AI products and their customers, so the teams running those products can see how each customer is doing. Our analyses used to run on frontier APIs. Most now run on an open-weight model, one anyone can download, so analyzing a conversation does not mean sending it to a model vendor. The model runs on a GPU we rent by the second.
Renting by the second makes a second GPU look like free speed. Two GPUs for half an hour cost the same as one GPU for an hour, so a batch that takes one GPU an hour should take two GPUs thirty minutes. We ran into that arithmetic when we considered a batch of 500 customer analyses and asked whether a second GPU would finish it sooner for the same money.
We checked that arithmetic against measurements we had already made of our own pipeline. It holds only when every added GPU stays as busy as the first. The first limit we hit was how much work our application handed the GPU. What let us keep one GPU busy was the wait we can afford: how long a job can sit in a queue before anyone is worse off. A person waiting on a chat answer can afford a few seconds. A batch of customer analyses can afford hours.
Open models and where they run
A trained model is a large set of numbers called weights, or parameters. The makers of most frontier models do not publish theirs, so those models are available only as a service. An open-weight model’s maker publishes the weights. Many are free to download, under licenses that allow commercial use. They come in a range of sizes and variants, including versions tuned to follow instructions and compressed versions that need less GPU memory. A model has to fit in GPU memory with room left for the requests in progress, so the GPU you rent limits which models you can run.
Some vendors serve open models through a per-token API, which works like any other model API. You can also run one on your own computers, which people call local AI. Or you can rent GPUs from a cloud provider and run the model there.
A serverless GPU endpoint, where the platform starts and stops the servers for you, is the rental that works most like an API. You give the platform a container, a packaged program with everything it needs to run. Yours runs an inference server, the software that loads your model and answers requests. You send those requests to a URL the platform provides. Behind it, the platform runs workers, copies of your container on its GPUs, and bills for each second they run.
We use RunPod Serverless, which as of October 2026 offered two kinds of worker. Flex workers shut down after an idle timeout, a set time with no requests, and cost nothing for compute while they are down. The next request then waits through a cold start while a worker starts and loads the model. Active workers run around the clock, so they have no cold start, and you pay for them the whole time. The right kind depends on how steady your traffic is and how long a request can wait.
What renting the GPU changes
On our rented GPU, the conversations go to a model we chose, running in a container we configured. The GPU provider still runs that container, so the conversations pass through its systems and its data-processing terms apply. Only a model on hardware you control keeps the analysis step on your own infrastructure.
The pricing changes too. A standard API request costs the same per token whether you send one at a time or a hundred at once. Some vendors charge half as much per token for batches that can wait up to a day. A rented GPU charges the same per second whether it is working on one request or many.
Running a model yourself also means choosing and testing it. General benchmarks do not test your task. In June 2026, a 9-billion-parameter model we tried returned well-formed output in the wrong shape for our reports, with the written sections missing. We now run an instruction-tuned model of about 30 billion parameters, compressed to fit one 24 GB GPU. Compression can change how well a model does the task, so the compressed version is the one to test. Larger models take longer to load, which lengthens their cold starts.
You also set how many GPUs can run, a decision the vendor makes when you use an API.
One GPU works on many requests at once
An inference server works on many requests at once. Servers such as vLLM use continuous batching: new requests join the work already running on the GPU. The paper that introduced vLLM states the requirement directly: “High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time.”
A customer analysis makes anywhere from one model request to dozens: a full one annotates messages in small batches, summarizes each session, and then writes a report. Two settings decide how much of that work reaches the GPU together: how many analyses run at once on our application server, and how many model requests the app allows in flight, meaning sent and not yet answered.
On October 16, 2026, we ran the same eight customer analyses through one 24 GB GPU at several settings. The eight ranged from short conversation histories to long ones, and each run of all eight made 84 to 87 model requests.
| Requests in flight, at most | Analyses at once | Time to finish all eight |
|---|---|---|
| 1 | 4 | 561 s |
| 8 | 4 | 179 s, 200 s |
| 12 | 4 | 176 s, 180 s |
| 8 | 1 | 295 s |
| 8 | 2 | 215 s, 216 s |
With four analyses running, the eight reports took 561 seconds when the app allowed one request in flight and about 178 seconds when it allowed twelve. The GPU costs the same per second either way, so one request at a time used about three times as much GPU time for the same eight reports.
What per-second billing charges for
Per-second billing charges for time. The work done in that time depends, up to a point, on how many requests the GPU has in flight. Two GPUs finish a batch in half the time at the same cost only if each one does as much work per second as the single GPU did. That requires each to have as much work in flight.
Our app allows at most twelve requests in flight. If we added a second GPU and both ran, the two would share those twelve. Individual requests might finish sooner, but we did not measure whether the batch would finish sooner for the same money.
The platform also decides whether a second GPU runs at all. As of October 2026, RunPod Serverless offered two scaling rules. Queue delay adds a worker when requests have waited longer than a threshold, four seconds by default. If a worker that is already running accepts everything an app sends, requests do not wait, so that rule has nothing to respond to. While a worker is still starting, requests do wait, and the rule can add workers. Request count divides queued plus in-progress requests by a target number per worker; depending on that target, it can start a second worker that draws from the same twelve requests.
A worker that starts is billed “from when a worker starts until it fully stops.” That includes its cold start and its idle timeout.
The first limit was on our side
Our scheduler initially left capacity unused. In August 2026, our app let three analyses run at once, while the GPU accepted up to eight requests; we raised that to twelve in September. Small analyses have little to run in parallel, so three of them rarely filled eight slots; on average, 3.2 were in use. When we let eight analyses run at once, the same GPU held 7.9 requests on average. In short production measurements on August 4, finished analyses per minute rose from 2.41 to 4.26, and the median analysis took 45 seconds instead of 35.
Each analysis took longer, most likely because it shared the GPU with more of the others. More analyses finished each minute, and the price per second stayed the same.
The October runs in the table set our current limits: four analyses at once for each person, with at most twelve requests in flight. More analyses at once means more work for our application server. It runs each analysis as its own process, loading the customer’s history, building prompts, and checking and saving results between model requests, on the same machine that answers web requests. Our attempts to measure eight analyses at once with eight requests in flight did not finish cleanly, so the table stops at four.
The table also suggests a second GPU would have been less busy than the first. With four analyses running, raising the request limit from eight to twelve changed finishing time by about 6 percent, inside the spread between repeat runs. These runs cannot say why. Four analyses may seldom have had more than eight requests ready, the GPU may have been near its own limit, or the slowest analyses may have set the finish time. In each case, keeping two GPUs as busy as one would take more work in flight than four analyses supplied.
The wait you can afford
Customer analysis can wait. Someone who starts one analysis and watches its progress might wait a few minutes; a batch of hundreds can run overnight. Because the work can wait, a queue on our side holds analyses until a slot opens, and that keeps one GPU busy.
A chat answer is different. Someone is watching for the first words, and every second their request waits behind a batch, they look at an empty reply. For that work, you buy idle capacity on purpose: a worker kept running, request slots held free, or a second GPU that often has nothing to do.
Keeping a GPU full also tends to slow each job, as the August numbers suggest. That delay lands on anyone waiting for a single answer. Running both kinds of work on one GPU needs priority for the interactive requests or separate capacity for them.
When a second GPU is worth adding
Set an acceptable wait for the work, then measure the GPU you already rent. Check how many requests it has in flight and whether more requests still reduce finishing time. If they do, find out what stops you from sending more: a limit you set, your own servers, or the GPU’s memory. Consider a second GPU when the first stays full, more requests no longer help, and jobs still wait longer than you can afford.
We kept one GPU. Before we add another, our application server has to run more analyses at once, and measurements have to show that extra requests no longer improve the single GPU’s throughput. Our analyses can wait in the queue meanwhile.