Class QuantizedQwenFp8Loader

java.lang.Object

smile.llm.quant.QuantizedQwenFp8Loader

public final class QuantizedQwenFp8Loader
extends Object

Installs native FP8 linears into a Qwen hybrid model from HuggingFace SafeTensors.

Official Qwen3.8-FP8 uses fine-grained weight_block_size=[128,128] with weight_scale_inv. Those weights are installed as Fp8BlockLinear (LibTorch _scaled_mm_v2). Tensor-scale FP8 checkpoints still use Fp8Linear.

DeltaNet in_proj_a/in_proj_b, norms, conv, A_log, dt_bias, embeddings, lm_head, and the vision tower stay on the dense residual load path.

Method Summary

Modifier and Type	Method	Description
static void	install(QwenModel model, Path dir, Device device, int tpSize, int tpRank, ScalarType outDtype, int modelLoaderThreads)	Installs FP8 (or block-dequant dense) projections for every hybrid block on one TP rank.
static void	install(QwenModel model, Path dir, Device device, int tpSize, int tpRank, ScalarType outDtype, int modelLoaderThreads, WeightGemmBackend backend)	Same as install(QwenModel, Path, Device, int, int, ScalarType, int) with an explicit backend: WeightGemmBackend.FP8 (Hopper+, _scaled_mm_v2) or WeightGemmBackend.MARLIN_FP8 (Ampere/Ada, FP8 weights resident, weight-only Marlin GEMM).
static boolean	isInstalledProjectionKey(String hfName)	Returns whether hfName is owned by the FP8 installer (must not be force-fed into dense shells).

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

install

public static void install(QwenModel model,
 Path dir,
 Device device,
 int tpSize,
 int tpRank,
 ScalarType outDtype,
 int modelLoaderThreads)
                    throws IOException

Installs FP8 (or block-dequant dense) projections for every hybrid block on one TP rank.

Parameters:

model - target rank model (shells already on device).

dir - checkpoint directory.

device - rank device.

tpSize - tensor-parallel world size.

tpRank - this rank.

outDtype - compute dtype for dequant / Fp8Linear output.

modelLoaderThreads - shard-read concurrency (0 = auto).

Throws:

IOException - if checkpoint IO fails.

install

public static void install(QwenModel model,
 Path dir,
 Device device,
 int tpSize,
 int tpRank,
 ScalarType outDtype,
 int modelLoaderThreads,
 WeightGemmBackend backend)
                    throws IOException

Same as install(QwenModel, Path, Device, int, int, ScalarType, int) with an explicit backend: WeightGemmBackend.FP8 (Hopper+, _scaled_mm_v2) or WeightGemmBackend.MARLIN_FP8 (Ampere/Ada, FP8 weights resident, weight-only Marlin GEMM).

Parameters:

model - target rank model (shells already on device).

dir - checkpoint directory.

device - rank device.

tpSize - tensor-parallel world size.

tpRank - this rank.

outDtype - compute dtype for dequant / Fp8Linear output.

modelLoaderThreads - shard-read concurrency (0 = auto).

backend - how block-FP8 linears are multiplied.

Throws:

IOException - if checkpoint IO fails.

isInstalledProjectionKey

public static boolean isInstalledProjectionKey(String hfName)

Returns whether hfName is owned by the FP8 installer (must not be force-fed into dense shells).

Parameters:

hfName - HuggingFace weight key.

Returns:

true when the FP8 installer owns this key.