mirror of
https://github.com/ollama/ollama.git
synced 2026-04-18 05:54:09 +02:00
79c1e93c00
New features: - Warmup phase to eliminate cold-start outliers - time-to-first-token measured in each epoch - VRAM/memory tracking to identify CPU spillover - Controlled prompt length - Defaults to 6 epochs and 200 tokens max Benchstat fixes: - ns/request instead of ns/op — non-standard unit created a separate group instead of grouping with timing metrics - Token count as the N field — benchstat interprets N as iteration count for statistical weighting, not as a token count
144 lines
4.6 KiB
Markdown
144 lines
4.6 KiB
Markdown
Ollama Benchmark Tool
|
|
---------------------
|
|
|
|
A Go-based command-line tool for benchmarking Ollama models with configurable parameters, warmup phases, TTFT tracking, VRAM monitoring, and benchstat/CSV output.
|
|
|
|
## Features
|
|
|
|
* Benchmark multiple models in a single run
|
|
* Support for both text and image prompts
|
|
* Configurable generation parameters (temperature, max tokens, seed, etc.)
|
|
* Warmup phase before timed epochs to stabilize measurements
|
|
* Time-to-first-token (TTFT) tracking per epoch
|
|
* Model metadata display (parameter size, quantization level, family)
|
|
* VRAM and CPU memory usage tracking via running process info
|
|
* Controlled prompt token length for reproducible benchmarks
|
|
* Benchstat and CSV output formats
|
|
|
|
## Building from Source
|
|
|
|
```
|
|
go build -o ollama-bench ./cmd/bench
|
|
./ollama-bench -model gemma3 -epochs 6 -format csv
|
|
```
|
|
|
|
Using Go Run (without building)
|
|
|
|
```
|
|
go run ./cmd/bench -model gemma3 -epochs 3
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Basic Example
|
|
|
|
```
|
|
./ollama-bench -model gemma3 -epochs 6
|
|
```
|
|
|
|
### Benchmark Multiple Models
|
|
|
|
```
|
|
./ollama-bench -model gemma3,gemma3n -epochs 6 -max-tokens 100 -p "Write me a short story" | tee gemma.bench
|
|
benchstat -col /name gemma.bench
|
|
```
|
|
|
|
### With Image Prompt
|
|
|
|
```
|
|
./ollama-bench -model qwen3-vl -image photo.jpg -epochs 6 -max-tokens 100 -p "Describe this image"
|
|
```
|
|
|
|
### Controlled Prompt Length
|
|
|
|
```
|
|
./ollama-bench -model gemma3 -epochs 6 -prompt-tokens 512
|
|
```
|
|
|
|
### Advanced Example
|
|
|
|
```
|
|
./ollama-bench -model llama3 -epochs 10 -temperature 0.7 -max-tokens 500 -seed 42 -warmup 2 -format csv -output results.csv
|
|
```
|
|
|
|
## Command Line Options
|
|
|
|
| Option | Description | Default |
|
|
|----------|-------------|---------|
|
|
| -model | Comma-separated list of models to benchmark | (required) |
|
|
| -epochs | Number of iterations per model | 6 |
|
|
| -max-tokens | Maximum tokens for model response | 200 |
|
|
| -temperature | Temperature parameter | 0.0 |
|
|
| -seed | Random seed | 0 (random) |
|
|
| -timeout | Timeout in seconds | 300 |
|
|
| -p | Prompt text | (default story prompt) |
|
|
| -image | Image file to include in prompt | |
|
|
| -k | Keep-alive duration in seconds | 0 |
|
|
| -format | Output format (benchstat, csv) | benchstat |
|
|
| -output | Output file for results | "" (stdout) |
|
|
| -warmup | Number of warmup requests before timing | 1 |
|
|
| -prompt-tokens | Generate prompt targeting ~N tokens (0 = use -p) | 0 |
|
|
| -v | Verbose mode | false |
|
|
| -debug | Show debug information | false |
|
|
|
|
## Output Formats
|
|
|
|
### Benchstat Format (default)
|
|
|
|
Compatible with Go's benchstat tool for statistical analysis. Uses one value/unit pair per line, standard `ns/op` for timing metrics, and `ns/token` for throughput. Each epoch produces one set of lines -- benchstat aggregates across repeated runs to compute statistics.
|
|
|
|
```
|
|
# Model: gemma3 | Params: 4.3B | Quant: Q4_K_M | Family: gemma3 | Size: 4080218931 | VRAM: 4080218931
|
|
BenchmarkModel/name=gemma3/step=prefill 1 78125.00 ns/token 12800.00 token/sec
|
|
BenchmarkModel/name=gemma3/step=generate 1 19531.25 ns/token 51200.00 token/sec
|
|
BenchmarkModel/name=gemma3/step=ttft 1 45123000 ns/op
|
|
BenchmarkModel/name=gemma3/step=load 1 1500000000 ns/op
|
|
BenchmarkModel/name=gemma3/step=total 1 2861047625 ns/op
|
|
```
|
|
|
|
Use with benchstat:
|
|
```
|
|
./ollama-bench -model gemma3 -epochs 6 > gemma3.bench
|
|
benchstat -col /step gemma3.bench
|
|
```
|
|
|
|
Compare two runs:
|
|
```
|
|
./ollama-bench -model gemma3 -epochs 6 > before.bench
|
|
# ... make changes ...
|
|
./ollama-bench -model gemma3 -epochs 6 > after.bench
|
|
benchstat before.bench after.bench
|
|
```
|
|
|
|
### CSV Format
|
|
|
|
Machine-readable comma-separated values:
|
|
|
|
```
|
|
NAME,STEP,COUNT,NS_PER_COUNT,TOKEN_PER_SEC
|
|
# Model: gemma3 | Params: 4.3B | Quant: Q4_K_M | Family: gemma3 | Size: 4080218931 | VRAM: 4080218931
|
|
gemma3,prefill,128,78125.00,12800.00
|
|
gemma3,generate,512,19531.25,51200.00
|
|
gemma3,ttft,1,45123000,0
|
|
gemma3,load,1,1500000000,0
|
|
gemma3,total,1,2861047625,0
|
|
```
|
|
|
|
## Metrics Explained
|
|
|
|
The tool reports the following metrics for each epoch:
|
|
|
|
* **prefill**: Time spent processing the prompt (ns/token)
|
|
* **generate**: Time spent generating the response (ns/token)
|
|
* **ttft**: Time to first token -- latency from request start to first response content
|
|
* **load**: Model loading time (one-time cost)
|
|
* **total**: Total request duration
|
|
|
|
Additionally, the model info comment line (displayed once per model before epochs) includes:
|
|
|
|
* **Params**: Model parameter count (e.g., 4.3B)
|
|
* **Quant**: Quantization level (e.g., Q4_K_M)
|
|
* **Family**: Model family (e.g., gemma3)
|
|
* **Size**: Total model memory in bytes
|
|
* **VRAM**: GPU memory used by the loaded model (when Size > VRAM, the difference is CPU spill)
|