Class MiniMaxMlp

java.lang.Object

smile.llm.model.minimax.MiniMaxMlp

public class MiniMaxMlp
extends Object

Gated feed-forward with the swigluoai activation, used for the dense layers, the shared expert and every routed expert: w2(glu * (up + 1)) where glu = gate * sigmoid(alpha * gate), gate is clamped above at limit and up to [-limit, limit].

Parameters are registered as w1 (gate), w2 (down) and w3 (up) to match FeedForward.

Constructor Summary

Constructors

Constructor	Description
MiniMaxMlp(int dim, int intermediate, double alpha, double limit)	Constructor.
MiniMaxMlp(int dim, int intermediate, double alpha, double limit, MiniMaxWeightMode mode)	Constructor.

Method Summary

Modifier and Type	Method	Description
Tensor	forward(Tensor x)	Forward pass.
MemorySegment	module()	Returns the PyTorch module handle.

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

MiniMaxMlp

public MiniMaxMlp(int dim,
 int intermediate,
 double alpha,
 double limit)

Constructor.

Parameters:

dim - model dimension.

intermediate - hidden width.

alpha - sigmoid slope.

limit - clamp bound.

MiniMaxMlp

public MiniMaxMlp(int dim,
 int intermediate,
 double alpha,
 double limit,
 MiniMaxWeightMode mode)

Constructor.

Parameters:

dim - model dimension.

intermediate - hidden width.

alpha - sigmoid slope.

limit - clamp bound.

mode - weight mode; quantized layers are loaded through collect(String, Map).

Method Details

module

public MemorySegment module()

Returns the PyTorch module handle.

Returns:

module handle.

forward

public Tensor forward(Tensor x)

Forward pass.

Parameters:

x - input [..., D].

Returns:

output with the same shape.