Package smile.llm.attention

package smile.llm.attention

Multi-head and paged attention kernels and backends for LLM inference.

Related Packages

Package	Description
smile.llm	Large language models.

Class	Description
Attention	Multi-head attention.
AttentionBackend	Pluggable attention kernel backends (SGLang-style names).
AttentionBackends	Process-wide attention backend selector (installed at chat-service startup).
AttentionContext	Per-call metadata for AttentionKernel.forward(Tensor, Tensor, Tensor, Tensor, AttentionContext).
AttentionKernel	Attention compute kernel (contiguous SDPA or paged FlashInfer).
FlashInferArtifacts	Resolves FlashInfer AOT / jit-cache directory for hybrid packaging: explicit dir → env / bundled default → cache-dir → optional download.
FlashInferAttentionKernel	FlashInfer paged BatchPrefill / BatchDecode backend.
FlashInferWorkspace	Opaque FlashInfer workspace / plan handle (one per device / TP rank).
RaggedContiguousAttention	Ragged contiguous self-attention over packed [N, H, D] Q/K/V tensors.
TorchNativeAttentionKernel	LibTorch scaled_dot_product_attention backend.