Will this model fit on your GPU? Size a local LLM on paper before you pull it

You download the biggest model your disk can hold, ask it a question, and wait. The answer arrives a word at a time, the CPU fans spin up, and ollama ps says 45%/55% CPU/GPU. The model did not fit in the card's memory, so part of it runs from system RAM, and the whole thing moves at the speed of that slow part. The opposite mistake is just as common: a tiny model that fits easily but cannot follow a two-step instruction.

Sizing is arithmetic, not guesswork. Three numbers decide whether a model fits: its parameters, its quantization, and the context you give it.

Parameters: what the "8b" means

A model's name usually carries its size: llama3.1:8b, qwen3:14b, gemma3:27b. The number is the parameter count in billions, the weights the model learned during training. More parameters generally means more knowledge and better reasoning, and always means more memory and slower answers on the same hardware.

At full 16-bit precision each parameter takes 2 bytes, so an 8-billion-parameter model is about 16 GB of weights. That is why the Ollama library lists llama3.1:8b-instruct-fp16 at 16 GB.

Mixture-of-experts models such as qwen3:30b or gpt-oss:20b hold all their parameters in memory but use only part of them for each word, so they can answer faster than a dense model of the same size. For sizing, what matters is the full size of the file you load.

Quantization: smaller numbers, smaller model

Quantization stores each weight with fewer bits. A model quantized to about 4 bits per weight is roughly a quarter the size of the 16-bit original, and in practice loses surprisingly little quality on most tasks. Ollama and llama.cpp use the GGUF file format, and the quantization is part of the tag:

• Q8_0: about 8 bits per weight. Closest to the original, twice the size of Q4.
• Q6_K, Q5_K_M: the middle ground.
• Q4_K_M: about 4 to 5 bits per weight. The common default, and what Ollama gives you for a plain tag like llama3.1:8b.
• Q3_K_M, Q2_K: smaller still, with a visible quality cost.
• fp16 or bf16: no quantization at all.

Start at Q4_K_M. Move up only if you have memory to spare and can see a difference on your own prompts. Dropping to Q3 or Q2 to squeeze in a bigger model is usually worse than running a smaller model at Q4.

Context: the memory everyone forgets

The context window is how many tokens the model can see at once: the system prompt, the conversation so far, any retrieved documents and the answer it is writing. Ollama picks a default from the card's memory; its context-length page lists 4k tokens below 24 GiB of VRAM, 32k from 24 to 48 GiB and 256k at 48 GiB or more.

Context costs memory because the model keeps a KV cache: for every token in the window it stores a key and a value vector in every layer. The size per token comes straight from the model's architecture:

KV bytes per token = 2 (key and value) x layers x KV heads x head size x bytes per number

Llama 3.1 8B's published configuration has 32 layers, 8 key-value heads and a head size of 128. With a 16-bit cache:

2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes, about 0.13 MB per token
  4,096 tokens  ->  about 0.54 GB
  8,192 tokens  ->  about 1.07 GB
 32,768 tokens  ->  about 4.29 GB
131,072 tokens  ->  about 17.2 GB

That last line is the full context Llama 3.1 supports, and its cache alone is bigger than the 16-bit model. Long context is not free.

Two settings change the picture. Ollama can store the cache in 8-bit numbers: its FAQ lists f16 as the default, q8_0 as about half the memory with a very small loss in precision, and q4_0 as smaller still with a more noticeable loss. A quantized cache needs flash attention, which Ollama enables automatically where the hardware supports it. And parallel requests multiply context: with OLLAMA_NUM_PARALLEL=4, the FAQ says required memory scales with the number of parallel requests times the context length, so four parallel 8K conversations reserve as much cache as one 32K conversation.

The estimate

Add three things:

1. Weights: the file size of the tag you pull.
2. KV cache: per-token size times context length (times parallel requests).
3. Overhead: compute buffers, the CUDA runtime and whatever else sits on the card. It varies by model, backend and driver; the examples below assume about 1 GB, and you confirm the real number on your own machine.

Llama 3.1 8B with an 8,192-token context, using the library's file sizes. These are estimates, rounded to a tenth of a gigabyte:

The same model at Q4_K_M as the context grows:

Read the tables against your card. An 8 GB card runs the Q4_K_M model comfortably at the default context and gets tight at 8K. A 12 GB card has room for 8K at Q5_K_M, or 32K with a quantized cache. A 16 GB card can hold Q8_0 at 8K. Leave a margin: if the estimate is within a gigabyte of the card's total, expect Ollama to spill part of the model to system RAM.

Check it against reality

An estimate is a starting point. Load the model with the context you plan to use, then:

ollama ps

The SIZE column is what Ollama actually allocated, the CONTEXT column is the context it used, and the PROCESSOR column must say 100% GPU. If it shows a CPU/GPU split, pick a smaller quantization, a smaller context or a smaller model, and check again.

Picking the file

The Ollama library at ollama.com/library lists every model's sizes, context window and capabilities, and each tags page lists every quantization with its file size. ollama pull qwen3:8b downloads the default quantization; ollama pull llama3.1:8b-instruct-q5_K_M asks for a specific one.

Hugging Face hosts many more GGUF files, and Ollama can run them directly:

# Q4_K_M is used by default when the repository has it
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF

# Ask for a specific quantization by its tag
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q8_0

Stick to repositories from the model's publisher or well-known quantizers, and read the model card, including the license, before you build anything on top. "Open" models come with very different licenses: as listed on Hugging Face in October 2026, Llama 3.1 uses the Llama 3.1 Community License and Gemma 3 the Gemma Terms of Use, while Qwen3, gpt-oss and Mistral 7B are Apache 2.0.

Checklist

• You know your card's memory before you pick a model.
• You start at Q4_K_M and move up only with memory to spare.
• You estimate weights + KV cache + overhead before you pull.
• You know the context you need, and that parallel requests multiply it.
• ollama ps shows the SIZE you expected and 100% GPU after loading.
• You have read the license of every model you build on.

Found this useful? Subscribe on YouTube for the daily video, or follow the RSS feed.