starred/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-04-27 19:25:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

x/imagegen: add FP4 quantization support for image generation models (#13773)

Browse Source 
Add --quantize fp4 support to ollama create for image generation models
(flux2, z-image-turbo), using MLX's affine 4-bit quantization.

Changes:
- Add fp4 to validation in CreateImageGenModel
- Add FP4 case to quantizeTensor (group_size=32, bits=4, affine mode)
- Add GetQuantization() to WeightSource interface for dynamic params
- Update LoadLinearLayer to use quantization params from model metadata
This commit is contained in:
Jeffrey Morgan
2026-01-19 00:54:54 -08:00
committed by GitHub
parent a887406c24
commit 03bf241c33
5 changed files with 43 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
x/create/client/quantize.go
View File

@@ -54,6 +54,9 @@ func quantizeTensor(r io.Reader, name, dtype string, shape []int32, quantize str
// Quantize based on quantization type
var qweight, scales, qbiases *mlx.Array
switch quantize {
case "fp4":
// affine mode: group_size=32, bits=4
qweight, scales, qbiases = mlx.Quantize(arr, 32, 4, "affine")
case "fp8":
// affine mode: group_size=32, bits=8
qweight, scales, qbiases = mlx.Quantize(arr, 32, 8, "affine")