Can I run Mistral Large 3 on a single RTX 4090?

No. Mistral Large 3 is a 675B-parameter Mixture-of-Experts model — even in NVFP4 the weights alone are roughly 340GB, and all experts must reside in GPU memory even though only ~41B parameters are active per token. A single 24GB consumer card cannot host any usable fraction of this model. The official deployment targets are a single 8x H200/B200 node in FP8 or a single 8x H100/A100 node in NVFP4.

How long does the initial Mistral Large 3 download take?

The native FP8 weights are roughly 675GB and even on fast residential broadband will run overnight or longer. The NVFP4 build is around 340GB. Use the `--resume-download` flag with the HuggingFace CLI since large gated downloads sometimes drop mid-transfer.

Does Mistral Large 3 support function calling locally?

Yes. Mistral Large 3 has native function-calling support, and vLLM exposes it through OpenAI-compatible tool-call APIs when you launch the server with `--enable-auto-tool-choice --tool-call-parser mistral`. Without those flags, tool calls silently won't work.

What's the cheapest cloud option for running Mistral Large 3?

Spot or on-demand H100/H200 nodes on RunPod, Lambda Labs, and similar GPU clouds are the realistic budget option — the model needs a full 8-GPU node to serve at native precision. For occasional use that is far cheaper than buying hardware. For intermittent calls, hosted Mistral Large 3 endpoints on Mistral's own API or third-party serverless providers are usually cheaper than spinning up a node yourself.

Can I use Mistral Large 3 commercially when running it locally?

Yes. Mistral Large 3 is released under the Apache 2.0 license, which permits commercial use. That said, always re-read the license text on the current model card before shipping a product, since terms can be tightened on future revisions.

Run Mistral Large 3 Locally: GPU Setup & Real Benchmarks

So you want to run Mistral Large 3 on your own box. Good news: with the right quantization and a serious GPU rig, it's doable in 2026. Bad news: this model is a 675B-parameter Mixture-of-Experts (with ~41B active per token), and even quantized it does not fit on a single consumer GPU.

This guide walks through the hardware math, the install, and the throughput numbers you should expect. No fluff, no "just use the API" cop-outs.

What You'll Build

By the end of this tutorial, you'll have Mistral Large 3 running locally with one of two realistic setups:

An NVFP4 deployment on a single node of H100s or A100s (the most accessible serious-hardware option)
A native FP8 deployment on a single node of H200s or B200s (production-grade, as recommended by Mistral)

We'll also walk through what Mistral has published about the model so you know what you're getting before you commit a workstation to this.

Prerequisites

Before you start, make sure you have:

A Linux machine (Ubuntu 22.04 or 24.04 strongly recommended). Windows works via WSL2 but expect headaches.
Python 3.11 or newer
CUDA 12.4+ and matching NVIDIA drivers
Multi-hundred GB of fast SSD storage for the weights
A HuggingFace account that has accepted the Mistral Large 3 license

And patience. The download alone takes a while.

Picking the Right GPU

The official model card describes Mistral Large 3 as a granular Mixture-of-Experts with 41B active parameters and 675B total parameters, with a 256k context window and a 2.5B vision encoder bolted on. The instruct release is shipped in native FP8 format.

Hands installing a large NVIDIA GPU into a workstation motherboard

Quick VRAM math for a 675B total-parameter MoE (all experts must fit in memory even if only ~41B are active per token):

Precision	Bytes per param	Weights only	Realistic node
BF16	2	~1.35 TB	Multi-node only
FP8 (native)	1	~675 GB	1 node of 8x H200 (Mistral-recommended)
NVFP4	0.5	~340 GB	1 node of 8x H100 80GB or 8x A100 80GB

Add 10-20% on top for KV cache, especially if you plan to push toward the full 256k context. The cache scales linearly with sequence length and adds up faster than people expect.

This is not a model you run on consumer GPUs. A single RTX 4090 (24GB) cannot host even a small fraction of the active experts, and the routed-MoE structure means you cannot cheaply offload "unused" experts the way some dense models tolerate. If you want local Mistral output and don't have multi-GPU server hardware, the practical advice is to use Mistral's API or a hosted endpoint instead — or step down to Mistral Small 4, which actually fits on a single workstation GPU.

What about Apple Silicon?

Even an M3 Ultra Mac Studio (maxes out at 512GB unified memory in the current configuration) struggles here: an FP8 deployment needs ~675GB just for weights, which is more than any single Mac Studio can hold. Sub-FP8 community quants may eventually shrink this further, but at the time of writing there is no practical Mistral Large 3 setup on Apple Silicon.

Step 1: Set Up Your Python Environment

Fresh conda environment, always:

Bash

conda create -n mistral-large python=3.11 -y
conda activate mistral-large
pip install --upgrade pip

Mistral's official recommendation is vLLM. Per the model card, you need vLLM 1.12.0 or newer (which pulls in a compatible mistral_common):

Bash

pip install --upgrade "vllm>=1.12.0"

Sanity-check the install:

Bash

python -c "import mistral_common; print(mistral_common.__version__)"

Step 2: Download the Weights

Bash

huggingface-cli login
huggingface-cli download mistralai/Mistral-Large-3-675B-Instruct-2512 \
  --local-dir ./mistral-large-3 \
  --local-dir-use-symlinks False

If you're targeting H100s or A100s, grab the official NVFP4 build instead: mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4. A BF16 reference build (mistralai/Mistral-Large-3-675B-Instruct-2512-BF16) is also published if you need it.

Fair warning: large gated downloads can stall. If you hit a timeout, use --resume-download and walk away for a few hours.

Step 3: Run Inference With vLLM

For the FP8 build on a single 8x H200 node (Mistral's recommended configuration):

Bash

vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \
  --max-model-len 262144 \
  --tensor-parallel-size 8 \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --enable-auto-tool-choice \
  --tool-call-parser mistral

A few notes on those flags:

tensor-parallel-size 8 shards the model across all eight GPUs in the node.
max-model-len 262144 exposes the full 256k context. Drop it if you don't need that much; KV cache shrinks linearly.
enable-auto-tool-choice and tool-call-parser mistral are required for the model's native function-calling behavior.
For NVFP4 on H100s/A100s, swap the model ID to the -NVFP4 variant; the rest of the flags are the same.

Once the server is up, hit it with the OpenAI-compatible endpoint:

Bash

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-Large-3-675B-Instruct-2512",
    "messages": [{"role": "user", "content": "Explain MoE routing in 3 sentences."}],
    "temperature": 0.1
  }'

Mistral recommends temperature below 0.1 for daily-driver and production usage; bump it for creative tasks if you want.

Step 4: Speculative Decoding for Throughput

Mistral publishes a paired draft model at mistralai/Mistral-Large-3-675B-Instruct-2512-Eagle for use with speculative decoding. If you want maximum tokens-per-second from your node, serve the main checkpoint with the Eagle draft model attached per Mistral's vLLM instructions. This is usually a 1.5-2x throughput improvement on common tasks.

Benchmark Numbers Worth Knowing

Mistral's model card publishes head-to-head plots versus similarly-sized models but does not include a clean table of raw scores. For verified, third-party numbers you should look at the LMSYS Chatbot Arena leaderboard and curated benchmark sites — don't rely on marketing slides.

A rough qualitative picture of where Mistral Large 3 sits among open-weight competition:

Model	Architecture	License	Notes
Claude Opus 4.6	Closed	Proprietary	Closed weights, API-only
DeepSeek V3	~671B MoE	Open weights	Comparable scale, open weights
Mistral Large 3	675B MoE / 41B active	Apache 2.0	Multimodal, 256k context
Llama 4 Maverick	MoE	Llama license	Open weights

If you need apples-to-apples numbers, pull the latest from the LMSYS Arena or each model's official card.

Throughput on real hardware

Bar chart comparing Mistral Large 3 inference speed across GPU configurations

Mistral does not publish per-GPU throughput numbers, so anything specific here would be guesswork. In general:

A single 8x H200 FP8 node will produce strong single-user throughput and scale well with batching.
A single 8x H100 or A100 NVFP4 node will be noticeably slower than FP8 on H200, but workable.
Speculative decoding with the Eagle draft model is the main lever you have for higher tokens/sec.

Treat any specific tok/sec number from a forum post with suspicion until you measure it on your own hardware with your real prompts.

Common Pitfalls

A few things that will absolutely waste your evening:

Trying to run this on consumer GPUs. 675B total parameters do not fit on a 24GB or even a 48GB card, MoE or not. If you don't have multi-GPU server hardware, stop here.
Mixing CUDA versions. Recent vLLM releases want CUDA 12.4 minimum. If your driver is older, you'll get cryptic kernel errors at startup. Run nvidia-smi and check.
Forgetting the Mistral tokenizer flags. --tokenizer_mode mistral --config_format mistral --load_format mistral are not optional — leaving them off will give you broken outputs or load failures.
OOM at long context. KV cache scales with sequence length. The full 256k context can eat tens of GB by itself on top of weights. Start with a smaller --max-model-len and only push higher if you need it.
Skipping tool-call flags. Without --enable-auto-tool-choice and --tool-call-parser mistral, native function calling silently won't work.

Verifying Your Setup

Quick sanity check script. Drop this in test_inference.py:

Developer monitor displaying terminal output from a vLLM server starting up

Python

import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

prompts = [
    "Write a Python function that reverses a linked list.",
    "Summarize the plot of Crime and Punishment in 3 sentences.",
    "Solve: a train leaves Boston at 3pm going 60mph...",
]

for p in prompts:
    start = time.time()
    r = client.chat.completions.create(
        model="mistralai/Mistral-Large-3-675B-Instruct-2512",
        messages=[{"role": "user", "content": p}],
        max_tokens=300,
    )
    elapsed = time.time() - start
    tokens = r.usage.completion_tokens
    print(f"{tokens/elapsed:.1f} tok/s | {p[:40]}...")

Run it. If you're getting consistent token/sec numbers across all three prompts and the outputs make sense, you're in business.

Next Steps

Now that you have a local Mistral Large 3 running, a few directions worth exploring:

Hook it into Claude Code or Cline as a custom backend via the OpenAI-compatible endpoint. You can route specific tasks to your local model and save on API spend. (We walk through this exact swap in Ditch the API Bill: Run Claude Code on Local LLMs.)
Enable speculative decoding with the official Eagle draft model for a meaningful throughput bump on common tasks.
Fine-tune on the FP8 weights. Mistral notes FP8 can be more precise than NVFP4 for fine-tuning. Tooling like Axolotl keeps the experience reasonable if you have the hardware.

And honestly? If you don't have a multi-GPU server, the math here doesn't pencil out. Mistral Large 3 is Apache 2.0 — open weights, commercially usable — but a single 8x H200 or H100 node is the entry ticket. If you can't justify that, the Mistral API gives you the same model at full quality.

But if you have the hardware, running Mistral Large 3 locally in 2026 is straightforward and well-supported. The official tooling actually works.

Sources