The pitch of AirLLM is hard to ignore: run a 70B model on a 4 GB GPU, no quantisation, no distillation. It is true. It is also the kind of true that deserves a closer look before you plan anything around it.

I tried it on a desktop with an RTX 5070 (12 GB VRAM). This post covers what AirLLM actually does, how to get it running, and where it fits, and where it does not.

How it works

A transformer is a stack of identical layers. During inference, the model only ever needs one layer at a time: compute layer 1, pass the result to layer 2, and so on. Normal inference keeps every layer in VRAM because that is fast. AirLLM keeps only one.

On first load, AirLLM splits the model into one file per layer and writes them to disk. During generation it loads a layer, runs it, frees it, and loads the next one, prefetching the following layer while the current one computes. Peak VRAM is roughly the size of one layer plus activations, which is why a 70B model fits in a few gigabytes.

The trade-off sits right there in the design: for every generated token, the entire model is read from disk.

Setting it up

A clean conda environment keeps this away from everything else:

conda create -n airllm python=3.11 -y
conda activate airllm

Install PyTorch with CUDA support from the PyTorch index, picking the CUDA build that matches your GPU and driver (check the selector on pytorch.org ). Recent GPUs need recent builds; an RTX 50-series card, for example, needs a build for CUDA 12.8 or newer:

pip install torch --index-url https://download.pytorch.org/whl/cu132
pip install numpy airllm

Verify that the GPU really computes, not just that it is detected:

python -c "import torch; print(torch.__version__, (torch.randn(1000,1000,device='cuda') @ torch.randn(1000,1000,device='cuda')).sum().item())"

A first run

Start small. A 1B model shows whether the pipeline works in minutes instead of hours:

from airllm import AutoModel

model = AutoModel.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

prompt = model.tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is the capital of the Netherlands?"}],
    tokenize=False,
    add_generation_prompt=True,
)
tokens = model.tokenizer(
    [prompt], return_tensors="pt", return_attention_mask=False,
    truncation=True, max_length=128, padding=False,
)

out = model.generate(
    tokens["input_ids"].cuda(),
    max_new_tokens=40,
    use_cache=True,
    return_dict_in_generate=True,
)
print(model.tokenizer.decode(out.sequences[0]))
<|user|>
What is the capital of the Netherlands?</s>
<|assistant|>
The capital of the Netherlands is Amsterdam.</s>

Two details matter here. Chat models expect their chat template; give them raw text and they often end the conversation before it starts. And the split layers land next to the model in the Hugging Face cache, so set HF_HOME to a fast disk with room to spare before the first run.

Switching to a big model is a one-line change to the model ID. Everything else stays the same.

The arithmetic you cannot escape

Because every token reads the full model from disk, storage throughput sets the speed limit. A back-of-the-envelope estimate:

ModelSize on disk (fp16)Read per tokenAt ~6 GB/s NVMe
8B~16 GB~16 GB~3 s/token
70B~140 GB~140 GB~25 s/token
70B, 4-bit compressed~35 GB~35 GB~6 s/token

These are lower bounds from disk speed alone; compute, PCIe transfers and Python overhead come on top, and a SATA SSD or a disk shared with other work makes it much worse. A 200-token answer from a 70B model is a matter of tens of minutes, not seconds.

AirLLM’s optional compression="4bit" or "8bit" shrinks what has to be read and is the single biggest speed lever. It depends on bitsandbytes, so check that your version supports your GPU.

Disk space adds up too: the original weights plus the split copy means roughly twice the model size. For 70B in fp16, plan on about 280 GB.

When AirLLM makes sense

  • Proving that a model runs at all. Evaluating a 70B model’s answers on hardware you already have, before renting or buying anything.
  • Offline batch jobs. Summarising or classifying a set of documents overnight, where latency does not matter and throughput is “done by morning”.
  • Quality comparisons. Running the same prompts through a small and a large model to see whether the large one is worth the cost.
  • Learning. It is a clear, readable demonstration of how layer-wise inference works.

When it does not

  • Anything interactive. Chat, coding assistants and agents need tokens per second, not seconds per token.
  • Models that already fit. With 12 GB of VRAM, a 4-bit quantised 7B–14B model runs entirely on the GPU through llama.cpp, Ollama or vLLM, at tens of tokens per second. Streaming it from disk only makes it slower.
  • Production. The project moves slowly, follows transformers changes with a delay and has no serving layer. Treat it as an experiment tool.

The realistic alternatives

If a large model needs to be fast, the options are the usual ones: quantise harder and offload part of the model to system RAM (llama.cpp does this well), or rent a GPU with enough memory for an hour. A cloud GPU with 80 GB of VRAM for an afternoon often costs less than the time spent waiting on disk-streamed inference.

Conclusion

AirLLM delivers exactly what it promises: a 70B model on a consumer GPU. What it does not promise, and cannot, is speed. It turns a memory limit into a disk-bandwidth limit. That is a good trade for experiments, evaluations and overnight batches, and a bad one for anything a person waits on.

Use it to answer “is the big model worth it?”. Then run the answer on hardware that fits.