TokenBench protocol

Benchmark your card

One single command. It downloads llama.cpp (official release) and the reference model (~4.9 GB, once, cached in ~/.tokenbench), runs the standard protocol, collects telemetry, then publishes the result.

Download our app

Model library, live telemetry, sustained tests with curves: download, double-click, benchmark.

⬇ Download (.exe)⬇ Portable version

Windows SmartScreen may warn on first launch (unsigned app): "More info" then "Run anyway".

Don't want to use the app? Follow these instructions

Prerequisites

Node.js LTS and an NVIDIA GPU with up-to-date drivers. That's it.

Windows: once, before the first launch

PowerShell blocks scripts by default (npx included). Allow them for your account — no admin rights needed:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

The command

Windows (PowerShell), macOS or Linux:

npx https://tokenbench.fr/tokenbench-0.4.2.tgz --token 'VOTRE-JETON'

The token is OPTIONAL: without it, the command works the same (just drop --token) and your measurement lands "pending moderation". With an API key from your account (My account → API keys), the run is published under your username. Single quotes around the key: special characters don't survive the shell without them.

Benchmark another model

Beyond the protocol, any model in the catalog (Qwen, Llama, Gemma, Mistral, DeepSeek…) can be measured: find its identifier, then run it. The measurement is automatically linked to its page on the site. Browse the catalog →

# 1. Find the identifier (and check that the model fits on your card)
npx https://tokenbench.fr/tokenbench-0.4.2.tgz --list-catalog qwen3

# 2. Download it, measure it and publish the result
npx https://tokenbench.fr/tokenbench-0.4.2.tgz --token 'VOTRE-JETON' --catalog qwen3-8b

The file (Q4_K_M quantisation) is downloaded once into ~/.tokenbench; you are offered to delete it at the end. In the app, use the “Add a model” section of the library.

Useful options

--gpu-index 1          # multi-GPU machine: benchmark the 2nd card
--ref-only             # reference model only (fast)
--list-catalog [mots]  # list the catalog models (optional filter)
--catalog <id>         # benchmark a catalog model (e.g. gemma-3-12b)
--replay result.json   # re-publish a result whose upload failed
--no-upload            # measure without publishing
--help                 # everything else

What gets measured

Generation and prompt processing speed (llama-bench, pp512/tg128, 5 repetitions), VRAM peak, temperatures and power draw (nvidia-smi, sampled during the run). Nothing is typed in by hand: everything in the database was measured.