A practical walkthrough for getting DeepSeek V4 Pro running on your own hardware, from picking the right GPU tier to squeezing real tokens-per-second out of quantized weights.
Running frontier-class open weights on your own box used to be a fantasy. In 2026, it's a Tuesday afternoon project. This guide walks you through exactly how to run DeepSeek V4 Pro locally, what hardware you actually need (not the marketing spec sheet), and what performance to expect once it's alive.
And honestly? The setup is easier than the hardware shopping.
By the end of this tutorial you'll have a working local DeepSeek V4 Pro inference server, exposed on localhost:8000 with an OpenAI-compatible API. You'll be able to hit it from Python, from your IDE, or from any tool that speaks the OpenAI SDK. No cloud bills. No rate limits. No data leaving your machine.
We'll cover single-GPU consumer setups (RTX 4090 / 5090 class), dual-GPU workstations, and a note on multi-node for the truly ambitious.
Before you touch a terminal, confirm you have:
nvidia-smi.pip, git, and reading a stack trace.If any of those are missing, fix them first. Trying to bolt CUDA on top of a broken driver install mid-tutorial is a special kind of misery.
This is where most guides hand-wave. Let's not.
DeepSeek's V-series models are Mixture-of-Experts (MoE), which means total parameter count and active parameter count are very different numbers. What you actually need to fit in VRAM is (roughly) the active experts plus KV cache plus overhead. For exact figures check the DeepSeek model card on Hugging Face once V4 Pro's weights are published, since the exact expert split and MoE routing determine your real memory footprint.
Here's a realistic sizing table based on how V3 behaved and how the community typically quantizes these models:
Important reality check: DeepSeek-V4-Pro is a 1.6T-parameter MoE model. Even at aggressive 4-bit quantization the weights alone occupy roughly 800 GB of VRAM, and the official release ships as FP4+FP8 mixed precision requiring even more. It cannot fit on a single 4090, 5090, or H100, and the official inference code uses model parallelism across 8 GPUs. For genuinely local single-workstation use, most people should target DeepSeek-V4-Flash (284B total / 13B activated) instead, which is much more tractable.
| Setup | Model | Precision | Practical Use Case | Expected tok/s (community estimates) |
|---|---|---|---|---|
| 1x RTX 4090 (24 GB) | V4-Flash + heavy expert offload | 4-bit | Experimental, low throughput | N/A |
| 1x RTX 5090 (32 GB) | V4-Flash + CPU expert offload | 4-bit | Solo hobbyist, slow context ingest | N/A |
| 2x RTX 5090 (64 GB) | V4-Flash | 4-bit | Small team, still tight | N/A |
| 1x H100/H200 80–141 GB | V4-Flash | FP8 / 4-bit | Production single-node V4-Flash | N/A |
| 8x H100 80GB (MP=8) | V4-Pro | FP4+FP8 mixed (as released) | Reference V4-Pro serving | N/A |
Those tok/s numbers are ranges reported across community benchmarks for similar-sized MoE models. Your mileage will vary with batch size, context length, and how aggressive your quantization is.
If you're on a budget and only have a single 4090, don't panic. Quantized DeepSeek runs shockingly well, and the quality drop from 4-bit is smaller than people assume for models this large.
Open a terminal and create a clean workspace:
mkdir ~/deepseek-local && cd ~/deepseek-local
python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
Don't skip the venv. Mixing PyTorch versions across projects is how weekends die.
You have three sensible options: vLLM for maximum throughput, llama.cpp for CPU/GPU hybrid on modest hardware, or Ollama for the easiest possible onboarding. This guide uses vLLM because it gives you production-grade serving with one command.
pip install vllm torch --upgrade
pip install huggingface_hub
vLLM will pull the correct CUDA-matched PyTorch build automatically. If it complains about your CUDA version, run nvcc --version and match it explicitly per the vLLM install docs.
Authenticate with Hugging Face (you'll need a free account and an access token):
huggingface-cli login
Paste your token when prompted. Then pull the model. For a 4-bit quantized version (the sweet spot for consumer GPUs), grab a community GGUF or AWQ quant. For the raw weights:
huggingface-cli download deepseek-ai/DeepSeek-V4-Pro --local-dir ./deepseek-v4-pro
Replace the repo path with whatever DeepSeek publishes at release. This download is the slowest part of the whole tutorial. Go make coffee. Or lunch, if you're on residential internet.
Once the weights are on disk, fire up vLLM:
vllm serve ./deepseek-v4-pro \
--quantization awq \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--port 8000
A few knobs worth understanding:
--quantization awq tells vLLM to use Activation-aware Weight Quantization. Swap for fp8 on H100 or bitsandbytes for 4-bit NF4.--max-model-len caps context. Lower this if you OOM on startup. 8192 is a safe default for a 24 GB card.--gpu-memory-utilization 0.92 reserves ~8% of VRAM for overhead. If your display is on the same GPU, drop this to 0.85.When you see Uvicorn running on http://0.0.0.0:8000, you're live.
From another terminal:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "./deepseek-v4-pro",
"messages": [{"role": "user", "content": "Explain MoE routing in one paragraph."}]
}'
You should get a JSON response back within a few seconds. If you want a nicer interface, point any OpenAI-compatible client (Continue.dev, Aider, Libre Chat) at http://localhost:8000/v1 and use any string as the API key.
This is the section every hardware guide skips. So let's talk numbers.
DeepSeek's V-series has been genuinely competitive on public benchmarks. For example, DeepSeek V3’s own model card reports 82.6% on HumanEval-Mul (Pass@1) for the chat model and 65.2% on the base HumanEval Pass@1 (0-shot). V4-Pro reports significantly higher HumanEval Pass@1 numbers in its model card (self-reported by DeepSeek). Cross-reference against the DeepSeek V4-Pro model card and independent leaderboards before quoting exact figures against Claude or GPT models, since vendor-reported numbers and independent reruns often differ by several points.
On local hardware, the two things you actually feel are:
Quantization impact is small but real. Community evals of 4-bit quants typically show a 1-3 point regression on MMLU-class benchmarks versus full precision. For coding and chat, most people can't tell the difference in blind tests.
A short list of things that will bite you:
--max-model-len first, then --gpu-memory-utilization, then step down a quantization level.--chat-template explicitly or use the tokenizer's default via --tokenizer-mode auto.And one non-obvious tip: if you're running the model as a shared team resource, put a reverse proxy (Caddy, nginx) in front of vLLM with basic auth. The default server has no authentication, which is fine on localhost and terrifying on a shared network.
Run this quick sanity check script:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
resp = client.chat.completions.create(
model="./deepseek-v4-pro",
messages=[{"role": "user", "content": "Write a Python function that returns the fibonacci sequence up to n."}],
max_tokens=300,
)
print(resp.choices[0].message.content)
If you get syntactically valid Python back within a few seconds, congratulations. You have a local frontier model.
Once you have inference working, the interesting problems start:
Running frontier open models locally in 2026 is genuinely pretty solid. Not gonna lie, the first time your own hardware writes a working React component in under a second, it feels a little bit like magic.
Also, cheaper than the API bill.
For **DeepSeek-V4-Pro** (1.6T parameters), no — the weights are far too large for any current Apple Silicon Mac even at 4-bit. For the smaller **DeepSeek-V4-Flash** (284B / 13B activated), 4-bit GGUF quants may be viable on very high-memory Apple Silicon (llama.cpp or LM Studio; vLLM is CUDA-only), but you will need substantial unified memory and can expect noticeably slower throughput than a discrete NVIDIA GPU. Check the exact quant sizes on Hugging Face before committing hardware.
Break-even depends heavily on your usage, your local hardware cost, and which DeepSeek model you actually run locally versus which tier you would use via the API. Per NVIDIA's official spec, the RTX 5090 has a 575 W total board power, so under sustained inference load its electricity cost alone can be meaningful at US residential rates. Price out your specific workload against DeepSeek's current published API pricing rather than relying on generic 'pays for itself in a year' rules of thumb.
Yes, when served through vLLM with the `--enable-auto-tool-choice` flag and the correct tool parser for the model. The OpenAI-compatible endpoint accepts standard `tools` parameters in chat completion requests. Confirm the exact parser name in the vLLM release notes at the time you install, since new model families sometimes require a version bump.
vLLM will preempt in-flight requests and either swap them to CPU RAM (if `--swap-space` is set) or return a 500 error. To prevent this, lower `--max-num-seqs` to cap concurrent requests and reduce `--max-model-len` so the KV cache stays bounded. Monitor with `nvidia-smi` during load testing to find your safe ceiling.
Full fine-tuning of DeepSeek-V4-Pro (1.6T parameters) requires a multi-node cluster with tens to hundreds of H100/H200-class GPUs — it is not a single-workstation task. LoRA and QLoRA on smaller DeepSeek models (V4-Flash or earlier V-series distills) are more realistic on a single H100 or A100 80GB using Unsloth or Hugging Face PEFT, and a single 24–32 GB consumer card is only viable for the smallest DeepSeek distills. Confirm the parameter count of the specific checkpoint you intend to adapt before sizing hardware.
