Package smile.llm.quant

package smile.llm.quant

Weight quantization backends: FP8 / NVFP4 (primary) and Marlin (Ampere failover).

Related Packages

Package	Description
smile.llm	Large language models.

Class	Description
DenseLinearRelease	Releases placeholder dense LinearLayer shells after quantized ops are installed, so empty GPU weight storage does not inflate the KV budget.
Fp8BlockDequant	Dequantizes DeepSeek / Qwen-style fine-grained FP8 weights (weight_block_size = [128, 128] + weight_scale_inv).
Fp8BlockGemm	DeepSeek-style W8A8 block FP8 matrix multiply: activations are quantized on the fly to E4M3 with one float32 scale per 1x128 group, weights are E4M3 with one float32 scale per 128x128 tile.
Fp8BlockLinear	Block-scaled FP8 linear for DeepSeek / Qwen fine-grained FP8 checkpoints (weight_block_size=[128,128] + weight_scale_inv).
Fp8BlockMarlinLinear	Block-scaled FP8 linear for GPUs without native FP8 (Ampere, Ada): DeepSeek / Qwen fine-grained checkpoints (weight_block_size=[128,128] with float32 weight_scale_inv) keep their FP8 weights in device memory and run through the weight-only Marlin kernel with bfloat16 activations.
Fp8KvCodec	Helpers for FP8 KV store/load with per-tensor scales.
Fp8Linear	Weight-only FP8 linear layer using cuBLASLt via LibTorch _scaled_mm.
LinearOp	Shared forward interface for dense, FP8, NVFP4, and Marlin linear layers.
MarlinLinear	Marlin FP16×INT4 linear layer (Ampere/Ada failover for GPTQ/AWQ).
MarlinWeightPacker	Packs HuggingFace GPTQ / AWQ INT4 tensors into Marlin layout for MarlinLinear.
MarlinWeightPacker.Packed	Result of packing: Marlin qweight, scales, logical features.
MxFp8	MXFP8 (OCP microscaling) weights: FP8 E4M3 elements with one shared E8M0 scale per 32 consecutive elements along the reduction dimension.
MxFp8BlockLinear	Linear layer for MXFP8 weights on the native block-scaled FP8 GEMM of Hopper.
MxFp8Layer	A linear layer that keeps its weight in MXFP8 form (in a backend-specific layout) and receives it from the checkpoint after construction.
MxFp8MarlinLinear	Linear layer whose weight stays in MXFP8 (FP8 E4M3 elements, E8M0 scale per 32 elements) in the Marlin tile layout and is multiplied with bfloat16 activations by the weight-only Marlin kernel.
Nvfp4Linear	NVFP4 weight-only linear for Blackwell (sm_100+).
QuantBackendOverride	Thread-local override for smile.chat.quantization during model load.
QuantFormat	Detected checkpoint quantization format.
QuantFormatDetector	Detects HuggingFace quantization format from config.json.
QuantizedHfLoader	Loads HuggingFace GPTQ / AWQ / FP8 / NVFP4 SafeTensors into Llama linears (shard-then-pack for TP when tpSize > 1).
QuantizedQwenFp8Loader	Installs native FP8 linears into a Qwen hybrid model from HuggingFace SafeTensors.
QuantLinearFactory	Factory for LinearOp backends from checkpoint tensors + selected backend.
QuantPolicy	Resolves checkpoint QuantFormat and WeightGemmBackend for load.
QuantPolicy.Resolved	Resolved quant policy for a checkpoint load.
QuantTpSharding	Shard-then-pack helpers for tensor-parallel quantized linears.
WeightGemmBackend	Weight GEMM backend selected from checkpoint format and GPU capability.