How Much VRAM Do You Need to Run an LLM on a GPU VPS in 2026

0
(0)

VRAM is the memory on the GPU, and it is usually the first limit you hit when you try to run a large language model on a GPU VPS. CPU RAM, disk and even the GPU’s raw compute matter, but if the model weights, the runtime and the KV cache do not fit in VRAM, the job will not start, or it will spill to system memory and become too slow to use.

This guide is a sizing method, not a model ranking and not a rewrite of the Ollama on a VPS guide. The same arithmetic applies whether you serve the model with Ollama, vLLM or another runtime. Treat the figures below as planning ranges. Always confirm with the runtime’s own memory report and nvidia-smi on the machine you will actually use.

What actually consumes VRAM

Three budgets share the same card:

  • Model weights. This is usually the largest block. Size scales with parameter count and the number of bytes stored for each parameter.
  • KV cache. The runtime stores attention keys and values for the current context. This grows with context length and with the number of concurrent requests.
  • Runtime and workspace. CUDA, the inference engine, temporary buffers and fragmentation all take a slice. A common planning habit is to leave a few gigabytes of headroom rather than packing the card to the last megabyte.

System RAM is still required. The operating system, the API process, tokenizers and any layers offloaded off the GPU live there. A GPU VPS with plenty of VRAM and too little RAM will still fail in messy ways. Storage matters too: model files are large, and a slow disk makes the first load painful. The filesystem comparison in ext4 vs XFS vs ZFS is useful when you dedicate a volume to models.

See also  VPS Hosting for Go Applications: Complete Guide for 2026

A simple way to estimate weight size

A reliable starting point is:

weight VRAM ≈ parameter count × bytes per parameter

Common precision and quantization sizes:

Format Approx. bytes per parameter What it means
FP16 or BF16 2 Full-precision inference for many open-weight models; highest quality, heaviest VRAM use
FP8 or INT8 / Q8 around 1 A middle ground: smaller than FP16, usually closer in quality than aggressive 4-bit quantizations
4-bit families (Q4, AWQ, GPTQ and similar) roughly 0.5 to 0.7 once runtime overhead is included The usual way to fit a larger model on a smaller card; quality depends on the method and the model

Worked examples of the weight term only, before KV cache and runtime overhead:

  • An 8 billion parameter model in FP16 is about 16 GB of weights.
  • The same 8B model in a 4-bit format is often in the 4–6 GB range for the weights alone.
  • A 70 billion parameter model in a 4-bit format is often in the mid-30 to mid-40 GB range for the weights alone.
  • The same 70B model in FP16 is about 140 GB of weights, which is a multi-GPU or very high-VRAM problem, not a single mid-size card.

Those are weights only. Add KV cache, CUDA and engine overhead, then leave headroom. If a listing is “close” to the weight size and you also want a long context or several users, it is not close enough.

KV cache, context length and concurrency

The KV cache is why two people can load the same model and see very different VRAM use. A short chat with a 4,000-token window uses far less cache than a 32,000- or 128,000-token session. Serving several users at once multiplies the cache, because each in-flight request keeps its own context.

If VRAM is tight, the levers are: a smaller or more heavily quantized model, a shorter maximum context, fewer concurrent requests, or a runtime that pages or offloads cache more efficiently. If you need long documents and several users, size the card for cache, not only for weights. This is one reason production serving stacks such as vLLM are discussed separately from single-user tools in vLLM vs Ollama for GPU inference.

See also  OpenClaw VPS Requirements in 2026: RAM, CPU, Hosting Cost and Security

Practical planning ranges

The ranges below assume a single request or light concurrency, a moderate context, and a quantized (often 4-bit) build unless stated otherwise. They are planning bands, not guarantees. A different quantization, a longer context, or a chatty multi-user API can push you into the next band.

Model class (open-weight, typical 2026 sizes) Planning band for VRAM Notes
3B–8B, 4-bit Often workable on 8 GB if context stays modest Comfortable for a personal assistant or a light internal tool; leave headroom for the OS and the API
8B, FP16 Plan around a 16–24 GB class card Weights alone are about 16 GB; cache and runtime need extra space
13B–14B, 4-bit Often in the 12–16 GB class Long context or several users may need 24 GB
30B–34B, 4-bit Often in the 20–24 GB class A common “one serious model, one card” target
70B, 4-bit Often in the 40–48 GB class, sometimes more Confirm the exact build; 70B in higher precision does not fit this band
70B+ in FP16, or 100B+ class models 80 GB class or multi-GPU Check the model card and the runtime’s tensor-parallel options before you buy

CPU-only inference is a different trade. Small quantized models can run on a regular VPS, but tokens per second drop sharply as the model grows. The comparison in GPU VPS vs CPU VPS for AI workloads covers when a CPU plan is enough and when it is not. If you are still choosing self-hosting over a hosted API, self-hosting an open-weight LLM vs using an API lays out the decision, without treating VRAM as the only cost.

How to verify the number on a real GPU VPS

Do not trust a marketing table alone. On the VPS, after the NVIDIA driver is working:

nvidia-smi
nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv

Load the exact model, quantization and context limit you plan to use, then send a prompt at the maximum context you care about, and at the concurrency you expect. Watch memory.used. If used memory sits close to the total, or if the process is killed with an out-of-memory error, step up the card, shrink the context, or pick a smaller build. Keep an eye on CPU RAM and disk at the same time; the VPS benchmark guide is a reasonable place to measure the rest of the machine, and a full disk will stop model pulls as surely as a short VRAM budget will.

See also  VPS Hosting for PHP Applications: Complete Guide for 2026

If the provider’s panel says “GPU VPS” but nvidia-smi is missing or the card is not visible inside the guest, you have a passthrough or driver problem, not a sizing problem. Fix that before you compare model charts. Shared or time-sliced GPUs can also report a large card while delivering only a fraction of it; check the allocation terms, not only the model name on the invoice.

Common sizing mistakes

Buying to the weight-size number and ignoring context is the usual one. The second is forgetting that the first user is not the same as five concurrent chats. The third is assuming every “7B” or “70B” package uses the same number of bytes; a Q4 file, a Q8 file and an FP16 file of the same model are different products. The fourth is leaving no headroom for driver updates, a second adapter, or a slightly longer prompt.

If you run agents on top of the model, tool calls and retries add more tokens and more concurrency than a single chat box. The broader host sizing notes in VPS requirements for self-hosted AI agents belong next to this VRAM math, not instead of it.

Frequently asked questions

Can I run a 70B model on a 24 GB GPU VPS?

Usually not at full precision, and often not even in 4-bit if you also want a long context. A 70B 4-bit build is commonly in the 40 GB class for weights plus cache. Some runtimes can offload layers to CPU RAM, but speed often falls off a cliff. Measure it, or pick a smaller model.

Does more system RAM make up for missing VRAM?

Only as a fallback. Offloading weights to system RAM can make a model start. It does not make GPU-speed inference appear. Use extra RAM for the OS, for CPU-offloaded layers you have already accepted will be slow, and for everything that is not the model.

Why does VRAM use go up during a long chat?

The KV cache grows with the conversation. A short question uses little cache; a long document or a long thread uses much more. Set a context cap you can afford, or size the card for the longest session you will actually allow.

Is VRAM the only GPU spec that matters?

No. Memory bandwidth, GPU generation and the driver’s compute capability affect tokens per second. But VRAM is the hard gate: if the model does not fit, faster silicon will not help until you quantize, shrink the context, or change cards.

Should I size for the biggest model I might try later?

Leave some room if your workload is still moving, but do not pay for an empty 80 GB card “just in case” unless you already know you need that class of model. It is cheaper to validate a current model on a card that fits than to guess two generations ahead.

Was this guide helpful?

Rate this guide from 1 to 5 stars.

Average rating: 0 / 5. Ratings: 0

No ratings yet. Be the first to rate this guide.

Leave a Comment