Package smile.llm.engine

package smile.llm.engine

Continuous-batching LLM inference engine, speculative decoding, sampling, and CUDA graph execution.

Related Packages

Package	Description
smile.llm	Large language models.

Class	Description
DecodeCudaGraph	Phase 2c: optional CUDA graph capture for batch-1 decode forward.
DecodeCudaGraphLog	INFO logging for decode CUDA graph bucket lifecycle (once per bucket event).
DecodeCudaGraphSession	Per-device CUDA graph session for uniform decode (batch 1 or power-of-two B).
DecodeForwardProfile	Optional decode-forward phase timers (attn / DeltaNet / MLP / NCCL).
DecodeForwardProfile.Snapshot	Per-thread / per-rank nanosecond accumulators.
DecodeStepTiming	Per-thread decode step timing filled by Qwen and read by InferenceEngine.
GenerationHandle	Handle for an in-flight or queued GenerationRequest.
GenerationRequest	One generation job submitted to InferenceEngine.
InferenceEngine	Continuous-batching inference runtime.
ModelExecutor	Low-level execution surface used by InferenceEngine.
MultimodalPrompt	A prompt whose media (images, videos) was preprocessed by a model-specific processor.
Sampling	Shared sampling helpers for greedy / temperature + nucleus decoding.
SpeculativeDecoding	Greedy speculative accept/reject helpers (Leviathan-style prefix match).
SpeculativeDecoding.AcceptResult	Result of one speculative verify round.
TextStreamer	Coalesces generated token ids into streamed UTF-8 text chunks.
VerifyCudaGraph	Optional CUDA graph capture for the multi-token MTP window-verify forward (S = numDrafts + 1), independent of DecodeCudaGraph.
VerifyCudaGraphSession	Per-device CUDA graph session for the uniform MTP window-verify forward.