So you are looking for the NVIDIA H200 GPU. Mostly for AI training. If you are looking to buy an H200 GPU upfront and host it upfront let me save you some time.
You can click here and get the datasheet directly from NVIDIA. OR, you can keep reading because we have covered a lot of points here that the datasheet will not have.
Now for the ones who are looking to rent the NVIDIA H200 GPU cloud, this one is the must read. You’ve probably already shortlisted a couple of GPU cloud providers. You know broadly what it is. What you’re trying to work out now is whether the spec sheet tells you the whole story.
It doesn’t, not on its own. The H200 GPU specs are fixed. Nvidia sets them, and every provider quotes the same numbers. What actually varies, and what decides whether you get those numbers in practice, is the server running the GPU underneath them. Two providers can list identical H200 GPU specs, memory, bandwidth, every number matching to the decimal, and still deliver very different real-world performance. That difference sits entirely outside the spec sheet.
Here’s what to check before you commit to a provider based on a spec sheet alone. If you’re the one who has to defend that provider choice to your CTO or your CFO six months from now, this is the part worth getting right upfront.
H200 GPU specs at a glance
Nvidia built the H200 GPU to answer one specific bottleneck: the H100 running out of memory as models crossed 70 billion parameters and context windows stretched past 32K tokens. Every number below traces back to that one design decision.
Here’s the full datasheet: the numbers Nvidia publishes, and what each one actually changes for your workload.
| Spec | H200 | Why it matters |
|---|---|---|
| Memory | 141 GB HBM3e | How large a model fits on one GPU |
| Memory bandwidth | 4.8 TB/s | How fast data moves in and out of memory |
| NVLink bandwidth | 900 GB/s | GPU-to-GPU communication in multi-GPU setups |
| CUDA cores | 16,896 | Same as H100, no compute change |
| Tensor cores | 528, 4th generation | Same as H100, no compute change |
| TDP | 700W | Power and cooling load per GPU |
| Architecture | Hopper | Same generation as H100, not Blackwell |
Everything above the CUDA core row is where the H200 GPU differs from the H100. Everything below is identical. That’s worth sitting with for a second: you’re paying the H200 premium for memory and bandwidth, not for extra compute.
It’s a useful filter for every H200 claim you hear elsewhere. If the claim is about compute, it applies equally to the H100.
👉 Recommended read: H200 Cloud Pricing
H200 SXM vs NVL vs PCIe: which one are you actually renting?
Most spec pages quote one number for the H200 GPU cloud and stop there. In practice, the form factor changes what you’re getting, and it’s worth confirming before you sign up with a provider.
The real differences
- SXM5: the highest-bandwidth version, built for multi-GPU servers connected over NVLink. Most 8-GPU cloud instances run this.
- NVL: a dual-GPU board with NVLink bridging the pair, aimed at inference that needs more memory than one GPU without a full 8-GPU commitment.
- PCIe: the standalone card, easiest to slot into an existing server, but with lower inter-GPU bandwidth than SXM.
Which one to pick
Renting a single GPU for inference: PCIe work fine. Training, or multi-GPU inference at scale: check for NVL or SXM specifically, since that’s where the NVLink bandwidth actually gets used.
Multi-GPU rental almost always means SXM under the hood, even if the provider’s page just says “8x H200.” Single-GPU rental is where PCIe and NVL both show up, and where it’s easiest for a listing to blur the distinction.
Ask your provider which form factor they’re racking. “H200” on its own doesn’t tell you that.
Specs alone don’t tell the whole story
Here is something the datasheet won’t tell you. A Formula 1 engine bolted into a car with a weak fuel line, poor cooling, and worn tires will not perform like a Formula 1 car. The engine spec didn’t change. Everything around it did.
The H200 GPU cloud works the same way. Its 141GB of memory and 4.8TB/s of bandwidth are fixed numbers on paper. What you actually get depends on the data centre it’s running in.
Picture two providers advertising identical NVIDIA H200 GPU specs, everything matching down to the decimal. One racks it in a facility built for 3kW per rack. The other, for 10kW. Under sustained load, the first one throttles within the hour. Both spec sheets were accurate. Only one of them held up in practice.
Three things decide whether you see those numbers in practice, or something closer to the H100’s:
- Power delivery: The H200 GPU draws up to 700W. A rack that can’t sustain that draw across every GPU will throttle under load, quietly, with no alert telling you why a job slowed down.
- Cooling: Sustained training and inference push a GPU to its thermal limit for hours at a stretch. Inadequate cooling forces it to downclock to protect itself, and you lose throughput you already paid for.
- Internal networking: The 4.8TB/s figure is per GPU. The moment you scale past a single card, your real throughput is capped by whatever connects those GPUs together, not by the H200’s own spec sheet.
None of this shows up when you’re comparing spec sheets across providers. It shows up in your bill, when a job takes twice as long as it should, and in your engineering team’s time, spent debugging a performance issue that was never in their code.
Questions to ask before you rent, not after
You can’t inspect a data centre from a pricing page. But you can ask specific questions that tell you whether the H200 specs on paper will hold up under your actual workload.
- Power redundancy: Is the facility N+1 or 2N on power? A rack running H200s at 700W each can’t afford a single point of failure on the power side.
- Cooling type: Air-cooled or liquid-cooled? Dense H200 GPU deployments increasingly need liquid cooling to sustain full load without throttling.
- Network topology: InfiniBand or standard Ethernet between GPUs? InfiniBand handles NVLink-scale traffic from multi-GPU training far better than standard networking does.
- Bare metal or virtualised: Are you getting dedicated GPU access, or a shared, virtualised slice? This changes what performance you can actually count on.
Form factor, confirmed in writing. SXM, NVL, or PCIe. Don’t take “H200” as the full answer, ask which one.
None of these show up on a spec sheet comparison. All of them show up in your actual throughput.
These aren’t only engineering questions. A throttled GPU still bills you for the full hour. Silent performance loss shows up as a higher effective cost per token, not as a line item you can point to, which is exactly why it’s worth asking upfront rather than finding out three months into a contract.
Where does H200 GPU memory matter
Apart from this, the H200 specifications matter as well.
The 141GB memory pool earns its keep in a specific, common band: too big for the H100, not big enough to need a full multi-node cluster.
| Workload | Fits on H100 (80GB)? | Fits on H200 (141GB)? |
|---|---|---|
| 13B–30B parameter models, standard context | Yes | Yes, no real advantage |
| 70B parameter models, FP16, short context | Tight or no | Yes, on a single GPU |
| 70B+ models with 100K+ token context | No | Yes |
| Multi-tenant serving, several models per GPU | No | Yes, more tenants per GPU |
| Fine-tuning 70B+ models with large batch sizes | Tight or no | Yes, fewer GPUs needed |
Outside that band, the extra memory is dead weight you’re paying for. Models under 30B parameters run fine on cheaper hardware. And models that need more than one 8-GPU H200 node anyway will hit a networking ceiling before they hit a memory one, which is exactly why the data centre point above matters more as you scale up, not less.
Training workloads feel this earlier than inference does. Training keeps every GPU at sustained high utilisation for hours, which is exactly where power and cooling headroom gets tested, not just at peak, but for the full duration of the run.
H200 vs H100 specs, side by side
| Spec | H100 | H200 |
| Memory | 80 GB HBM3 | 141 GB HBM3e |
| Bandwidth | 3.35 TB/s | 4.8 TB/s |
| CUDA cores | 16,896 | 16,896 |
| TDP | 700W | 700W |
| Architecture | Hopper | Hopper |
Compute is identical. The entire case for the H200 GPU rests on the memory and bandwidth rows. If your model and context length fit comfortably inside 80GB, the H100 gets you the same compute for less.
This is also why the decision shifts so much by workload. Paying the H200 premium for a model that never leaves 80GB of memory is money spent on headroom you’ll never use. Paying it for a 100K-token context window is money spent on the one thing that actually unblocks the job.
We’ve broken down the full cost comparison, current rental rates, and the buy-versus-rent math in our H200 GPU cloud price guide.
What CloudPe actually puts around your H200
Back to the data centre point, here’s what that looks like in practice on CloudPe.
- Every H200 GPU instance runs out of a Tier-4 facility in Mumbai
- With 200+ Gbps of internal networking connecting GPUs and nodes
- Backed by a 99.9% uptime SLA.
That networking figure is the piece that decides whether a multi-GPU H200 cluster delivers combined throughput, or just adds GPUs that spend half their time waiting on each other.
Each instance comes with:
- 16 vCPUs
- 128GB RAM, and
- 250GB of NVMe storage
alongside the GPU itself.
It is available as single, 2x, 4x, and 8x configurations. Multi-GPU configurations run on dedicated infrastructure rather than a shared, virtualised slice, which answers the bare-metal-or-virtualised question from the checklist above.
If you already know your workload needs the H200’s memory and you’re deciding where to run it, this is the part worth checking with any provider, not just the per-GPU hour rate.
Before you commit
A spec sheet gets you halfway to a decision. The other half is confirming those numbers survive contact with sustained, real load.
Ask about the form factor. Ask about power and cooling. Ask about the network connecting the GPUs. Then compare providers on what’s actually left, not on a datasheet every one of them can quote identically.
Frequently Asked Questions
What are the full NVIDIA H200 GPU specs?
141GB of HBM3e memory, 4.8TB/s of bandwidth, 16,896 CUDA cores, 528 4th-generation Tensor cores, and a 700W TDP, built on the Hopper architecture.
How much VRAM does the H200 have?
141GB of HBM3e, 76% more than the H100’s 80GB.
Is the H200 the same as Blackwell?
No. The H200 is a Hopper-generation GPU, the same generation as the H100. Blackwell is the generation after it, starting with the B200.
What’s the difference between H200 SXM and NVL?
SXM is the high-bandwidth version built for multi-GPU servers connected over NVLink. NVL is a dual-GPU board aimed at inference that needs more memory than one GPU without a full 8-GPU setup.
Does the data centre affect H200 performance?
Yes. Power delivery, cooling, and internal networking all affect whether a GPU sustains its rated performance under load or throttles below it.
Is the H200 faster than the H100?
Compute is identical between the two. The H200’s advantage is memory and bandwidth, which shows up on memory-bound workloads like long-context inference, not on raw throughput.
Do I need liquid cooling for an H200 deployment?
Not always, but dense racks running H200s at full power increasingly need it to sustain rated performance. Ask your provider directly rather than assuming air cooling is enough.
Does networking matter more as I add more H200 GPUs?
Yes. Per-GPU memory bandwidth stays fixed as you scale, but the network connecting GPUs to each other becomes the actual bottleneck on multi-GPU jobs, well before the GPUs themselves run out of headroom.
