Ask ten people what GPU you need for LLM work and most will just say “H200” and stop there. That’s not wrong, exactly. It’s just incomplete.
Fine-tuning a model and serving it to users are two different jobs, and they stress a GPU in different places. One needs room for gradients and optimizer states. The other needs room for a growing cache of conversation history across dozens or hundreds of concurrent requests.
Same GPU, two different bottlenecks. Get the assumption wrong and you either overpay for compute you don’t use, or hit a wall you didn’t see coming.
Here’s how to think about choosing a GPU cloud for LLM fine-tuning and inference, as two separate decisions, not one.
Fine-tuning and inference need different things from the same GPU
Fine-tuning is a memory problem you can calculate before you start. Inference is a memory problem that grows while the system is already running.
Fine-tuning holds more than just the model’s weights in memory:
- Gradients, calculated fresh for every layer, every step
- Optimizer states, which roughly double again with Adam-style optimizers
- Activations, held temporarily through each forward and backward pass
Depending on the optimizer, that overhead alone can be several times the size of the model’s weights.
Inference is lighter to start. It holds the model’s weights once, then adds a KV cache that grows as a conversation gets longer, or as more requests run at the same time. No gradients, no optimizer state, none of the training overhead.
That’s the core split, laid out side by side:
| Fine-tuning | Inference | |
| Memory driver | Weights, gradients, optimizer states, activations | Weights, plus a growing KV cache |
| Grows with | Model size, batch size, chosen optimizer | Context length, concurrent requests |
| Latency sensitivity | Low, the job just needs to finish | High, real users are waiting |
| Typical GPU count | Often multi-GPU for large models | Often single-GPU, unless serving at real scale |
Everything below follows from that one table.
What fine-tuning actually needs from a GPU cloud
Full fine-tuning needs roughly four times a model’s own memory footprint. Here’s where that multiplier actually comes from:
- Model weights: The baseline cost, unavoidable no matter how you fine-tune.
- Gradients: Roughly equal to the weights themselves.
- Optimizer states: Often double again with Adam-style optimizers.
- Activations: Held temporarily during each training step.
A 70B parameter model at 16-bit precision needs roughly 140GB just for its weights. Add the full training overhead above and you’re well past what a single 80GB H100 can hold. That’s exactly the situation the H200’s 141GB was built to solve.
LoRA and other parameter-efficient fine-tuning methods change this math substantially. You’re only training a small set of adapter weights, so the gradient and optimizer overhead drops to a fraction of full fine-tuning’s. A smaller GPU that couldn’t touch full fine-tuning on the same model suddenly becomes viable. Hugging Face’s PEFT documentation covers this math in more depth if you want to go further.
There’s a storage side too, easy to underweight. Checkpointing a large model regularly during training eats disk space fast, and losing a run to a crash without recent checkpoints is a genuinely expensive mistake.
One more thing worth flagging: if you’re fine-tuning across more than one GPU, memory stops being the only variable. How fast those GPUs sync gradients with each other becomes the next bottleneck, which is exactly where the form factor conversation below starts.
What inference actually needs from a GPU cloud
Inference lives or dies on the KV cache. That cache grows with two things:
- Context length. How long a single conversation or prompt gets.
- Concurrency. How many requests are being served at the same time.
Every token a model generates gets cached, so it doesn’t need recomputing on the next step. A single long conversation is manageable. A hundred concurrent long-context conversations is a very different memory problem, and it’s exactly the scenario where H200’s extra memory over H100 stops being a nice-to-have and starts being the reason the deployment works at all.
Long-context RAG pipelines, the kind pulling in 100k or more tokens of retrieved context per query, benefit directly from this. The retrieved context itself becomes part of what needs caching, on top of conversation history.
Multi-tenant serving has a similar shape. Running several models, or several fine-tuned variants of one model, on a single GPU means the memory has to cover every model’s KV cache at once, not just one. More memory means more tenants per GPU before you need to add hardware.
Curious what this actually costs at current rates? The H200 pricing breakdown is worth checking before you size anything.
Fine-tuning mostly doesn’t care about one thing, inference cares about a great deal: latency and throughput under load, not just whether the job eventually finishes. A fine-tuning run that takes six hours instead of four is annoying. An inference endpoint that’s too slow under peak load is a product problem, in front of real users, in real time.
SXM or NVL: which matters more for which job
Fine-tuning across multiple GPUs leans hard on NVLink bandwidth to keep gradients in sync. Inference, unless you’re running at serious multi-GPU scale, usually doesn’t need nearly as much of it.
Multi-GPU training means every GPU communicates updated gradients with every other GPU, repeatedly, throughout the run. That’s exactly what NVLink is built for, and it’s why SXM, the form factor with the most NVLink bandwidth, is worth confirming if you’re fine-tuning across more than one card.
Inference at a moderate scale, one model served to a reasonable number of concurrent users, usually fits comfortably on a single GPU. No cross-GPU communication happening, so form factor matters far less. NVL or even PCIe can be the right, cheaper call there.
Quick way to remember it:
- Multi-GPU fine-tuning → SXM, for the NVLink bandwidth
- Single-GPU inference → NVL or PCIe, cheaper, works fine
- Large-batch, multi-GPU inference → SXM again, same reason as training
We’ve broken down the full SXM vs NVL vs PCIe comparison, and the rest of the H200 datasheet, in our H200 specs guide.
Do you actually need tensor parallelism?
You need it the moment a single GPU’s memory can’t hold what you’re running, whether that’s the model itself, or the model plus everything fine-tuning adds on top. Not a moment before.
Tensor parallelism splits a model’s layers across multiple GPUs so no single card has to hold the whole thing. It’s what makes it possible to fine-tune or serve models that wouldn’t otherwise fit on one GPU.
It’s also not free. Splitting a model across GPUs means those GPUs must communicate on every forward and backward pass, which adds latency a single-GPU setup never deals with. For inference specifically, that’s why teams work hard to avoid tensor parallelism whenever a single high-memory GPU can do the job instead.
This is the practical reason H200’s extra memory over H100 matters beyond the headline spec:
- A 70B model needing tensor parallelism across two H100s to run comfortably
- Can often run on a single H200 instead
- No cross-GPU communication, no added latency, one card instead of two
There’s a related option worth knowing about too: pipeline parallelism, which splits a model by layer instead of within each layer. It solves a similar problem with different trade-offs, worth asking a provider about if tensor parallelism alone doesn’t fit your setup.
Quick check: does it fit on one GPU? If yes, stop there. If no, that’s when tensor or pipeline parallelism earns its place.
Benchmarks worth checking before you commit
A single benchmark number means almost nothing here. Three categories actually matter, and they stress a GPU in genuinely different ways:
- Long-context input: How it performs processing a large prompt, the RAG and document-analysis scenario.
- High-throughput batch inference: How many requests it serves at once before latency degrades, the production-API scenario.
- Short-context, fast output: Raw response speed for quick, simple queries, the chat-interface scenario.
A GPU that looks excellent in one category can look mediocre in another. A provider quoting one impressive number without saying which category it came from isn’t giving you a useful comparison. They’re giving you a marketing number.
Before you commit to a GPU cloud for either fine-tuning or inference, ask specifically for benchmark numbers in the category that matches what you’re actually building. If a provider can’t produce that, treat it the same way you’d treat a pricing page that can’t produce a sample invoice: as an answer in itself.
What this looks like on CloudPe
CloudPe runs H200 GPU cloud instances for the memory-heavy end of this, full fine-tuning on large models, long-context RAG, and multi-tenant serving, plus L4 instances for lighter, cost-efficient inference that doesn’t need H200-class memory.
[See current GPU configurations on CloudPe →]
Quick way to place your own project:
- Full fine-tuning, 70B+ parameters, or genuinely memory-bound inference → H200
- Smaller, already-fine-tuned model, light concurrency → L4, usually the more cost-effective call
Not everything needs to run on the biggest GPU available. The two decisions above are how you tell which situation you’re actually in, whether you need an H200 GPU cloud for inference specifically, or one built for fine-tuning at scale.
Put together, that’s the whole framework: work out your memory driver, check whether it fits on one GPU, confirm the form factor matches how you’re actually deploying, and ask for benchmarks in the category that matches your workload before you sign anything.
Frequently Asked Questions
What is the best GPU for inference?
It depends on the workload more than any single “best” GPU. A lower-cost, lower-memory GPU like the L4 is often the better choice for lighter inference. Memory-heavy GPUs like the H200 earn their cost specifically on long-context or high-concurrency inference.
Is H200 good for deep learning?
Yes, particularly for the memory-bound end of deep learning: large models, long context windows, and multi-tenant serving. For smaller models or lighter inference, the H200’s extra memory often goes unused.
Is GPU good for inference?
For anything beyond small-scale or CPU-tolerant workloads, yes. GPUs handle the parallel computation inference requires far faster than CPUs, and that gap widens as concurrency and model size grow.
Do I need the same GPU for fine-tuning and inference?
Not necessarily. Fine-tuning’s memory needs are usually higher than serving the same model afterward, especially with full fine-tuning rather than LoRA. It’s common, and often more cost-effective, to fine-tune on a higher-memory GPU and serve inference on a smaller one.
How much GPU memory does LLM fine-tuning actually need?
Roughly four times the model’s own parameter memory for full fine-tuning, once gradients, optimizer states, and activations are counted. Parameter-efficient methods like LoRA need substantially less.
Should I use LoRA or full fine-tuning?
LoRA is the practical default for most teams, since it needs far less GPU memory and still gets close to full fine-tuning’s results for many tasks. Full fine-tuning is worth the extra memory and cost when you need to adapt a model more deeply than LoRA’s smaller adapter weights allow.
What happens if I underestimate GPU memory for fine-tuning?
The job usually fails outright with an out-of-memory error partway through training, not a graceful slowdown. That’s lost GPU-hours and lost time, which is why sizing memory correctly before you start matters more here than in most infrastructure decisions.
Can I fine-tune and serve inference on the same GPU?
Technically yes, but it’s rarely the efficient choice. Fine-tuning’s higher memory needs mean you’re often paying for headroom the inference side of the job never uses. Separating the two, a higher-memory GPU for training, a smaller one for serving, is usually the more cost-effective setup.
