For some versions of Xcode, cmake builds are failing due to header problems in
cross-compiling during the generate phase. Since generate is producing arch
independent generated output, we can skip this during cross-compiling.
* mlx: add op wrappers for Conv2d, Pad, activations, trig, and masked SDPA
Add Conv2d, flexible Pad (with axes/mode), PadConstant, Maximum,
Minimum, Softplus, ReLU, GLU, Clamp, Sin, Cos, Clip,
ScaledDotProductAttentionMasked, and RoPEWithFreqs. Refactor
RoPEWithBase to delegate to RoPEWithFreqs.
* review comments
* mlx: fix ScaledDotProductAttentionMasked to consult the mask argument
Improve the MLX model creation pipeline with several model-agnostic changes:
- Rewrite supportsVision to use vision_config instead of architecture name
- Add supportsAudio for audio encoder detection
- Add alignment checking (isAligned) for quantization group sizes
- Support per-projection mixed quantization in MoE expert packing
- Record per-tensor quant metadata in safetensors blobs
- Parse per-tensor quant metadata at model load time
- Validate quantize output is non-empty before storing
- Fix pin/unpin cleanup in expert group quantization
- Promote v_proj/k_proj/down_proj to INT8 for INT4 base quant
- Add MetalIsAvailable() utility
- Skip audio encoder tensors from quantization
* gemma4: update renderer to match new jinja template
Google has updated their jinja template for gemma4, and so this change
gives us parity with the new template. The parsing also slightly changed
upstream, so we make a small change to our parser as well.
I've also corrected a few probably existing edge cases, especially
around type unions. The upstream output format is weird (a stringified
array), but in practice the models seem to understand it well.
* gemma4: special case simple `AnyOf`s
The upstream template doesn't handle `AnyOf`s, but since in the previous
commit we saw type unions work reasonably well, I'm now treating very
simple `AnyOf`s as type unions to help in cases where they might be used
* fix lint
* gemma4: prefer empty instead of `None`
We can't currently distinguish between a result being not-present vs.
empty. The empty case seems more important (e.g., a legitimately empty
tool call)
* gemma4: be more careful for tool results with missing IDs
We were missing setting the function index for several models that can
make parallel tool calls.
In the future we may want to consider putting some sort of post-parse
hook and relieve the parsers of this duty.
Fixes: #15457
Update cloud and local model recommendations to match current
models.go: add qwen3.5:cloud and glm-5.1:cloud, replace glm-4.7-flash
with gemma4 and qwen3.5 as local options.
Add documentation for Hermes Agent by Nous Research, covering
installation, Ollama setup via custom endpoint, messaging configuration,
and recommended models.
This change fixes two issues with Modelfiles:
1. If a user uses `ollama show --modelfile` to show a safetensors based
model, the Model would leave the "FROM" field blank which won't allow
a user to recreate the model. This change adds the model's current
canonical short name to the FROM field.
2. If a user uses the `/save` command in the CLI any messages which were
saved in a previous model wouldn't get saved (only the set of messages
from the current session).
The default branch in unmarshalResponsesInputItem had two issues:
- It referenced typeField.Type instead of itemType; these differ when the
shorthand role-based format promotes an empty type to "message", meaning
an unhandled type would show the wrong value in the error string.
- It used %s formatting, so an empty type field produced the unhelpful
message "unknown input item type: " with no indication what was missing.
Fix by using itemType (the resolved value) with %q quoting, and add a
dedicated message when itemType is empty (both type and role absent):
"input item missing required 'type' field".
Tests added for the empty-type and missing-type cases.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* pull: refine safetensors pull
- Body drain in resolve() — drain response body before close so Go's HTTP
client can reuse TCP connections instead of opening a new one per blob
(1,075 extra TCP+TLS handshakes eliminated)
- Skip speed recording for tiny blobs (<100KB) — prevents
HTTP-overhead-dominated transfer times from poisoning the median, which the
stall detector uses to cancel "too slow" downloads
- Resume support for large blobs (>=64MB) — on failure, preserves partial .tmp
files; on retry, re-hashes existing datak and sends Range header to download
only remaining bytes; gracefully falls back to full download if server returns
200 instead of 206; SHA256 verification catches corrupt partials
* harden push
- Prevents killing TCP connections after every request.
- Stronger backoff to handle server back-pressure and rate limiting
- Larger buffered reads for improve safetensor upload performance
- Better error message handling from server
- Handle 201 if server says blob exists
- Fix progress reporting on already uploaded blobs
- Trace logging to help troubleshoot and tune going forward
* review comments
* review comments
In addition to strings (which we already supported), OpenResponses
supports arrays of text content, image content, or file content (see
<https://www.openresponses.org/reference#object-FunctionCallOutput-title>).
We were missing support for these arrays, which caused unmarshal errors
like
```
json: cannot unmarshal array into Go struct field ResponsesFunctionCallOutput.output of type string
```
This change adds support for text content and image content, as those
are more straightforwardly mappable to Ollama message formats (though
image and text interleaving is lost), but it's less clear what to do for
files. In the future we can partially support this by inlining
reasonably sized text files, but wanted to get this change out first.
Fixes: #15250
* create: Clean up experimental paths
This cleans up the experimental features, and adds both unit and integration test coverage to verify no regressions.
* create: preserve config and layer names when creating from safetensors models
When creating a model FROM an existing safetensors model, ModelFormat,
Capabilities, and layer Name fields were lost. ModelFormat stayed empty
because it's only set from GGML layers (which safetensors models lack),
and layer names weren't copied in parseFromModel. This caused derived
models to fail loading ("config.json not found in manifest").
* review comments
* mlx: Improve M5 performance with NAX
This modifies the Mac release to now have 2 builds of MLX for broader
compatibility while supporting the latest M5 hardware features. NAX requires
building with xcode 26.2 and targetting support only for OS v26 and up. Since
we want to support older MacOS versions as well, we now need 2 different MLX
builds and runtime detection logic to select the optimal version. The newer
build will detect NAX missing at runtime, so it is safe to run on pre M5 macs.
* mac: prevent generate on cross-compiles
For some versions of Xcode, cmake builds are failing due to header problems in
cross-compiling during the generate phase. Since generate is producing arch
independent generated output, we can skip this during cross-compiling.
The existing strict gemma4 tool parser is still the primary path, but if
this fails, we try to repair by fixing some of the most commonly seen
mistakes these models seem to make in practice.
We repair by building up a set of candidates, and use the first candidate
that parses.
Repairs cover:
- missing Gemma string delimiters
- single-quoted string values, including a dangling Gemma delimiter
- raw terminal string values (if the corresponding tool schema indicates
it should be a string)
- missing object close only after a concrete repair
Add regression coverage for malformed tool calls from issue #15315 and
focused unit tests for the individual repair helpers and candidate
pipeline.
We've observed Gemma 4 occasionally emitting extra <tool_call|> tags
after a valid tool call. We suppress leading close tags in this
immediate post-tool-call state so the extra close tags do not leak into
assistant content. The tradeoff is that if the model intentionally
begins its next content span with the literal string "<tool_call|>", we
will erroneously treat it as noise and drop it.
Replace the custom Gemma4 argument normalizer with a stricter
reference-style conversion: preserve Gemma-quoted strings, quote bare
keys, and then unmarshal the result as JSON.
This keeps quoted scalars as strings, preserves typed unquoted values,
and adds test coverage for malformed raw-quoted inputs that the
reference implementation rejects.
cublasGemmBatchedEx fails during graph capture when pool allocations
return fake pointers. This is triggered when NUM_PARALLEL is greater
than 1 for models like gemma4 that use batched matmuls. Skip it
during reservation since the memory tracking is already handled by
the pool allocations.
Fixes#15249
* model/parsers: fix gemma4 arg parsing when quoted strings contain "
Fixes: #15241
* add more tests, be careful about what we escape
We want Windows-style paths to not get misinterpreted
* fix backslash-quote case, it really should be a literal backslash
h/t to @chathaway-codes for pointing this out!
Co-Authored-By: Charles H <2773397+chathaway-codes@users.noreply.github.com>
---------
Co-authored-by: Charles H <2773397+chathaway-codes@users.noreply.github.com>
Add --num-ctx flag to set context size, and report NumCtx in model info
header. Calibrate tokens-per-word ratio during warmup using actual
tokenization metrics from the model, replacing the fixed 1.3 heuristic.
This produces more accurate prompt token counts for --prompt-tokens.
Also add fetchContextLength() to query running model context via /api/ps.
* tokenizer: add byte fallback for SentencePiece BPE encoding
When BPE merging produces tokens not in the vocabulary, fall back to
encoding each UTF-8 byte as <0xHH> byte tokens instead of silently
dropping the character. Also teach Decode to convert <0xHH> tokens
back to raw bytes.
Fixes#15229, fixes#15231
* tokenizer fixes
* bench: add prompt calibration, context size flag, and NumCtx reporting
Add --num-ctx flag to set context size, and report NumCtx in model info
header. Calibrate tokens-per-word ratio during warmup using actual
tokenization metrics from the model, replacing the fixed 1.3 heuristic.
This produces more accurate prompt token counts for --prompt-tokens.
Also add fetchContextLength() to query running model context via /api/ps.
* integration: improve vision test robustness and add thinking tests
Add skipIfNoVisionOverride() to skip vision tests when OLLAMA_TEST_MODEL
is set to a non-vision model. Add Think:false to context exhaustion test
to prevent thinking models from using all context before the test can
measure it. Add third test image (ollama homepage) and replace OCR test
with ImageDescription test using it. Relax match strings for broader
model compatibility. Add TestThinkingEnabled and TestThinkingSuppressed
to verify thinking output and channel tag handling.
* gemma4: add Gemma 4 GGML model support
Add full Gemma 4 model family support (E2B, E4B, 26B MoE, 31B Dense)
for the GGML backend including text, vision, converter, parser, and
renderer.
Text model features:
- Sliding window + full attention with per-layer patterns
- KV sharing across layers with donor map
- Per-layer embeddings (PLE) with learned projections
- MoE routing with RMSNorm + learned scale
- Proportional RoPE with freq_factors for global attention
- Final logit softcapping
Vision model features:
- SigLIP vision encoder with 2D RoPE
- ClippableLinear with input/output clamping via packed v.clamp_data
- Adaptive average pooling with nMerge kernel
- Multi-modal projection with unweighted RMSNorm
Converter:
- Safetensors to GGUF with vision tensor renaming
- Fused MoE gate_up_proj splitting
- Vision patch embedding reshape (HF to Conv2D layout)
- Packed clamp data tensor for ClippableLinear bounds
- Proportional RoPE freq_factors generation
Also includes:
- BackendGet() on ml.Tensor for reading weight tensor data
- Q6_K CUDA get_rows kernel support
- MoE-aware ffn_down quantization layer counting
- Gemma4 parser with tool calling and thinking support
- Gemma4 renderer with structured tool format
- Architecture-based auto-detection of renderer/parser/stop tokens
- Integration test gemma4 model list additions
* gemma4: add audio support with USM conformer encoder
Add audio encoding for Gemma 4 using the USM conformer architecture:
- Converter: audio tensor mapping, SSCP/conformer/embedder name replacements,
softplus repacker for per_dim_scale, F32 enforcement for conv weights
- GGML backend: Conv1DDW and PadExt tensor ops
- Audio encoder: SSCP Conv2D, 12 conformer blocks (FFW + block-local
attention with relative position embeddings + LightConv1d + FFW),
output projection, audio-to-text embedding projector
- Audio preprocessing: WAV decode, mel spectrogram, FFT (pure Go)
- Model wiring: WAV detection, audio token handling, unified PostTokenize
Correctly transcribes "why is the sky blue" from test audio.
* integration: add gemma4 audio tests including OpenAI API coverage
Test audio transcription and response via the Ollama native API, plus
two new tests exercising the OpenAI-compatible endpoints:
- /v1/audio/transcriptions (multipart form upload)
- /v1/chat/completions with input_audio content type
All tests use capability checks and skip models without audio support.
* gemma4: add OpenAI audio API support and capability detection
- Add CapabilityAudio and detect from audio.block_count in GGUF
- Add /v1/audio/transcriptions endpoint with TranscriptionMiddleware
- Add input_audio content type support in /v1/chat/completions
- Add TranscriptionRequest/Response types in openai package
* gemma4: add audio input support for run command
- /audio toggle in interactive mode for voice chat
- Platform-specific microphone recording (AVFoundation on macOS,
PulseAudio/ALSA on Linux, WASAPI on Windows)
- Space to start/stop recording, automatic chunking for long audio
* gemma4: add transcribe command (ollama transcribe MODEL)
- Interactive mode with readline prompt and slash commands
- Non-interactive mode for piped audio or record-until-Ctrl+C
- Chunked streaming transcription for long recordings
- Word-wrapped output matching run command style
* gemma4: add parser, renderer, and integration test plumbing
* gemma4: fix renderer to emit BOS token
* gemma4: add OpenAI audio transcription API and input_audio support
* gemma4: update converter for new weight drop naming
* gemma4: add per_expert_scale to MoE router and fix moe_intermediate_size config
* gemma4: rewrite renderer to match HF Jinja2 template exactly
Fix 8 bugs found by building 55 reference tests verified against the
HF Jinja2 chat template (VERIFY_JINJA2=1 shells out to Python):
- Tool responses use separate <|turn>tool turns (not inline tags)
- Tool calls emitted before content in assistant messages
- Thinking content stripped from assistant history (strip_thinking)
- User, tool, and system content trimmed (template does | trim)
- Empty system message still emits system turn (check role, not content)
- Nested object properties rendered recursively with required field
- Array items specification rendered for array-type properties
- OBJECT/ARRAY type-specific rendering comma logic matches template
Also adds Required field to api.ToolProperty for nested object schemas,
replaces old gemma4_test.go with comprehensive gemma4_reference_test.go,
and commits the Jinja2 template as testdata for verification.
* gemma4: fix MoE fused gate_up split and multiline tool-call arg parsing
- Text MoE: split `ffn_gate_up_exps` into contiguous `[gate|up]` halves instead of stride-2 slices.
- Parser: escape control characters in `<|"|>...<|"|>` string literals when converting tool-call args to JSON.
- Fixes warnings like `invalid character '\n' in string literal` for multiline tool arguments.
- Add Gemma4 parser regressions for multiline tool-call args and `gemma4ArgsToJSON`.
* cmd: simplify audio input to dropped file attachments
* gemma4: use full SWA memory for better cache reuse
* gemma4: initialize clamps after backend load
* convert: align gemma4 audio tensor renames with llama.cpp
* Remove redundant comments in gemma4 vision model
* Format Gemma4 MoE block field alignment
* use 4096 kvcache.NewSWAMemCache
* convert: support new Gemma4 audio_tower tensor naming (#15221)
Co-authored-by: jmorganca <jmorganca@gmail.com>
* fix integration test defaults for audio
* review comments and lint fixes
* remove unused audio/video files
---------
Co-authored-by: jmorganca <jmorganca@gmail.com>
Previously we were accidentally using different clients/UAs depending on
whether it was an inference call or a different call. This change makes
them consistent, other than the timeout being different.
