Huawei lab open-sources MACaQ, a quantiser that outputs GGUF for llama.cpp
HUAWEI Computing Systems Lab's Apache-2.0 tool also targets vLLM, Diffusers and PyTorch. Its speed and quality figures are the authors' own.

HUAWEI Computing Systems Lab has published MACaQ, an open-source tool for quantising AI models, on GitHub. The repository was created on 10 November 2026 with a first commit titled "Initial release". The code is licensed under Apache 2.0, and the method is described in an arXiv paper submitted on 7 November.
Quantisation stores a model's numbers at lower precision so it needs less memory. MACaQ's README describes the project as "Model-agnostic quantization beyond text".
What it does
The paper is "MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations" (arXiv 2611.10649), by Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang and Lukas Cavigelli. It describes MACaQ as "an MXFP4 W4A4 post-training quantization method that treats every linear layer independently". W4A4 means both the weights and the activations are held at 4 bits.
Many earlier methods push mathematical transforms across neighbouring layers to avoid running them at inference time; MACaQ instead runs a small per-channel scale and a block-Hadamard transform online, inside each layer. According to the paper, this means "only one layer must reside in memory at a time" while a model is being quantised. A step the authors call Flip-Refinement then nudges individual 4-bit codes to neighbouring values to reduce error.
The paper lists the costs of that design:
- Because everything stays inside one layer, "cross-layer error compensation is impossible".
- The online transforms slow inference down. The paper puts that at about 2.3% for Qwen3-32B, rising to about 8% on the smallest model it tested (0.6B parameters).
- In the mixture-of-experts tests, only the expert layers were quantised.
Which runtimes it targets
The README lists four backends:
- llama.cpp: text generation, "2/3/4-bit GGUF". The full workflow uses a modified copy of llama.cpp that ships in the repository. The NOTICE file says the changes include "calibrated activation transforms". A separate mode, MACaQ-R, refines an existing quantised mixture-of-experts GGUF, and the README says its files "run with a standard llama.cpp installation". The README also says "llama.cpp also supports CPU execution".
- vLLM: text generation and serving, "W4A4 or weight-only".
- Diffusers: "image, video and audio generation, W4A4".
- PyTorch: speech generation (Qwen3-TTS and Chatterbox Turbo).
For Huawei's Ascend chips, the README links kernels submitted as pull requests to the CANN projects. It says: "Full-model execution requires integration with an Ascend inference runtime."
The lab has also posted a MACaQ-R GGUF of Qwen3.6-35B-A3B on Hugging Face. Its model card lists an Apache-2.0 licence. The NOTICE file says third-party components, model weights and datasets "retain their own terms".
The authors' numbers
Every figure below is the MACaQ authors' own; we have not tested them.
- Large models. The README says MACaQ "can quantize a 122B-parameter model in 64 minutes using just 32.5 GB of device memory". The paper says this was Qwen3.5-122B-A10B on a single 96 GB GPU. WikiText-2 perplexity was 5.14 after quantisation, against 5.06 for the full-precision model (lower is better).
- Perplexity table. The main table has eight rows: FP16, six other 4-bit methods (two QuaRot variants, FlatQuant, MR-GPTQ, and two DuQuant++ variants) and MACaQ. Its 12 columns cover six Llama and Qwen models on two test sets. In the paper's figures, MACaQ has the lowest quantised score in all 12 columns, though some gaps are small: on Qwen3.5-35B-A3B's C4 column it scores 11.21 against MR-GPTQ's 11.22.
- Generation tests. The IFEval and MATH-500 table has seven rows and four columns, and the results are mixed. MACaQ has the highest quantised score in two columns. MR-GPTQ is higher on Qwen3-32B MATH-500 (97.44% against 97.26%), and on Qwen3.5-35B-A3B IFEval MACaQ (85.42%) ranks fourth of six quantised methods, behind QuaRot (GPTQ) at 85.60%, DuQuant++ (GPTQ) at 85.59% and MR-GPTQ at 85.43%. Standard errors range from ±0.06 to ±0.40 points.
- Quantisation time. The README claims MACaQ can "match their best Llama-3-8B perplexity with 11× less quantization time", comparing against MR-GPTQ and DuQuant++.
- Images. On FLUX.2-klein-4B, the paper compares MACaQ with the model's official NVFP4 checkpoint (a three-row table). MACaQ scores closer to the full-precision output on all four fidelity metrics, and peak memory is 13.1 GB against 13.3 GB, which the paper calls a tie.
- GGUF chart. A README chart compares six Qwen3-4B GGUFs by file size and KL divergence from FP16 (lower is better). MACaQ's IQ4_XS file is 2.260 GB at 0.02804. Bartowski's Q4_K_M has the lowest divergence on the chart (0.02771) and the largest file (2.497 GB). Unsloth and Bartowski IQ4_XS files (2.271 GB) score 0.03663 and 0.03443. On Qwen3.6-35B-A3B, the README reports MACaQ-R at 0.013367 against 0.015731 for Unsloth's UD-Q4_K_XL, both at 22.36 GB.
How to try it
The README asks for Python 3.12 and "a separate environment for each backend". For llama.cpp, the install guide sets up a Python virtual environment and builds the bundled llama.cpp with CMake. There is a CPU-only option, though the guide notes that "CPU quantization takes longer". The worked example quantises Qwen3-4B to IQ4_XS using calibration text, then runs the resulting GGUF through the repository's own runtime. The llama.cpp guide says quantising Qwen3.6-35B-A3B with the full MACaQ workflow needs "about 220 GB of working disk space".
On our reading, the quickest route is the published MACaQ-R model, which the README says loads with llama cli -m in a standard llama.cpp install.
Why it matters
On our reading, the notable part is where the code points. A Huawei lab has put much of its effort into the GGUF format and the llama.cpp runtime used for local inference on ordinary GPUs and CPUs, alongside vLLM and Diffusers. Ascend support exists, but the README says it still needs integration with an Ascend runtime. The quality and speed claims are the authors' own, and we have not tested them.
Ask Relay — he reads every question himself and replies personally by email.
