quant-matmul-xpu

W4A8 and W8A8 quantized linear layers for Intel GPUs, loadable through kernels: group-quantized int4 or per-channel int8 weights against per-token int8 activations, accumulating in int32 and rescaling once. The reference baselines are an exact int32 matmul, matched at 0.0 error for W8A8, and a per-group dequantize-then-matmul, matched at 7.45e-08 for W4A8.

Quantized weights are how a model fits an integrated GPU's shared memory budget, and the decode-time cost there is weight traffic: the weight matrix is N*K bytes against M*K for activations with M small. These kernels multiply the packed weights directly, so the 4x or 8x residency reduction becomes a bandwidth reduction rather than a decode-time dequantization tax.

On an Intel Iris Xe: W8A8 beats eager float32 at every shape measured, while W4A8 wins the M=1 decode shape and loses above roughly M=4, where nibble unpacking dominates on a device with no matrix engine to absorb it.

Usage

from kernels import get_kernel

qm = get_kernel("phanerozoic/quant-matmul-xpu", version=1, trust_remote_code=True)

codes, scales = qm.quantize_act(hidden)          # per-token int8

packed = qm.pack_int4(int4_weights)              # [N, K] -> [N, K//2] uint8
out = qm.w4a8_linear(codes, scales, packed, weight_scales, group_size=128)

out = qm.w8a8_linear(codes, scales, int8_weights, weight_scales)

version selects the release branch; trust_remote_code is required by kernels for publishers without the trusted-publisher mark.

API

Method

Int4 weights are group-quantized along K in the GPTQ/AWQ-compatible symmetric layout, two values per byte with the low nibble first, stored biased by +8 so each fits an unsigned nibble; scales are [N, K / group_size]. Int8 weights are per-channel with scales [N]. Activations are per-token symmetric int8 with scales [M]. Accumulation is int32 within a group and rescaled per group for W4A8, so the product is exact up to the single float multiply that closes each group.

Weight traffic dominates at decode, so a work-group owns four output columns and walks K once, holding one accumulator per row of an M-tile: a weight byte is fetched once and reused across the tile rather than re-read per row. Each work-item takes a contiguous slice of K, and quantization-group boundaries are tracked against a running edge rather than a division per element.

Measured

Against eager float32 matmul on the same device, Intel Iris Xe (Gen12LP, 96 EUs), torch 2.13.0+xpu, 30 iterations after 5 warmup, group_size 128:

The int4 path crosses over: it wins at the M=1 decode shape and loses to eager float32 above roughly M=4. Nibble unpacking is instruction-bound and this device has no matrix engine to absorb it, so the int8 path stays ahead of int4 at every shape measured here. Choose int4 for the 8x smaller weights, not for throughput at batch. Both paths reduce weight residency against float32: 4x for int8, 8x for int4. The table is reproduced by benchmarks/bench.py.

Validation

Ten checks on Intel Iris Xe, torch 2.13.0+xpu. W8A8 matches an exact int32 matmul reference at 0.0 maximum error, both for a batched shape and for the M = 1 GEMV that decode actually issues. W4A8 matches a per-group dequantize-then-matmul reference at 7.45e-08. Activation quantization matches a reference absmax exactly for scales and bit-exactly for codes, with round-trip error verified at or below half a quantization step. Nibble packing is checked bit-exact in both directions against the source int4 values, and both paths are checked at M=12, which spans more than one M-tile.

Requirements and limits

• int8 activations, uint8-packed int4 or int8 weights, float32 scales and output.
• Intel GPU with a working Level Zero driver and a torch XPU build.
• K divisible by group_size for the int4 path.
• The int4 path is a decode kernel on this class of device; route batched shapes to int8 or to eager.
• The ops ship fake implementations, so they trace under torch.compile without breaking the graph.

References

Frantar et al., "GPTQ" (2023); Lin et al., "AWQ" (2023); symmetric group-quantized int4 storage layouts.

License

Apache-2.0.