Edge AI
for Enterprise

SMILE is a comprehensive AI engine for the JVM. Handle everything from machine learning modeling and on-device training to ONNX graphs and local LLM inference inside your Java application—on the branch server, payment terminal, vehicle gateway, or air-gapped node.

See Serve Engine
Pure JVM CoreONNX & LibTorch0.2 GB Micro Image0.4 s Startup~80 MiB Idle RAMAir-gappedSelf-hosted

Who it is for

The people who already run the edge

Payments and branches

Authorization is already on a short clock. Score transactions in sub-millisecond latency with local TreeSHAP explanations on the server that holds the record, before cardholder data ever leaves the network.

Fleet and telematics

Driver and vehicle signals are scored—and models are retrained—directly in the cab. A summary comes back when the link is up; the raw sensor stream never rides the cellular bill.

Sovereign systems

The node keeps working with the network unplugged. A 0.2 GB native container is small enough to audit, boots in 0.4 seconds, and never fetches a weight or library from outside.

Why SMILE excels at the edge

A complete AI engine—train, score, and explain in place

  • Zero-dependency core (tiny footprint) For resource-constrained edge gateways, SMILE's core module is completely self-contained. It runs pure JVM-optimized bytecode with compact data structures—drastically outperforming Python or R in memory efficiency without requiring external native binaries.
  • Plug-and-play hardware acceleration Run pure Java on the CPU you already have, or drop in optional native dependencies when an edge node has a GPU or NPU: OpenBLAS/LAPACK for linear algebra, ONNX Runtime execution providers (CUDA, DirectML, NPU), and a LibTorch (PyTorch C++) backend for hardware-accelerated deep learning.
  • Built-in Classic ML, ONNX & LLM stack SMILE Serve hosts an OpenAI-compatible REST server and bundled React UI in a single process—serving serialized classical models (.sml decision trees, random forests, k-means, regressors), direct internal routing to .onnx models, and native local SLMs/LLMs with tokenizers, GQA, RoPE, and radix KV-caching.
  • On-device training & local adaptation Edge nodes do not just score static weights. Actively train and update Random Forests, Gradient Boosting, clustering, or anomaly detectors directly on edge data, and compute local TreeSHAP explanations right where the decision ran.
  • Explained where it ran Every classical ML decision can carry a local SHAP explanation (exact TreeSHAP for tree ensembles, LinearSHAP for linear models, and KernelSHAP for neural, SVM, discriminant, and Gaussian Process models) alongside posterior probabilities. The audit stays with the event, on the node.
  • Resilient & hot-reloadable offline Inference continues when the link is down. Updated .sml and .onnx models hot-swap atomically from disk or drain gracefully without restarting the process, backed by Kubernetes-ready health probes and unified Prometheus telemetry.

Measured runtime profile

From 80 MiB gateways to GPU clusters

SMILE Serve ships four pre-built Docker flavors (ghcr.io/haifengl/smile-serve). Every flavor serves SMILE (.sml) and ONNX (.onnx) models, the web UI, health probes, and Prometheus metrics. Numbers below were measured on x86-64 with three baseline models loaded (Random Forest, LDA, and an ONNX model), plus a fourth chat LLM where noted.

Image Flavor Runtime & Target Image Size Startup Latency Memory Consumption Capabilities
latest-micro GraalVM native executable, CPU (x86-64-v3) 0.2 GB (~205 MB) 0.4 s ~80 MiB idle SMILE (.sml) & ONNX inference, TreeSHAP & KernelSHAP, Web UI inference pages, health & metrics. No chat or database. Built for constrained edge gateways, sidecars, and scale-to-zero fleets.
latest-native GraalVM native executable, CPU (x86-64-v3) 0.4 GB (~375 MB) 0.5 s (no chat)
8 s (with Phi-4-mini)		 ~115 MiB idle (~5× less than JVM)
3 GiB (with Phi-4-mini 4.6 GB)		 Full CPU feature set: SMILE & ONNX inference + CPU LLM chat (ONNX Runtime GenAI packages), user accounts, and embedded H2 conversation history.
latest JVM (OpenJDK 25), any x86-64 CPU 0.6 GB ~3 s (no chat)
11 s (with Phi-4-mini)		 ~600 MiB idle
3.5 GiB (with Phi-4-mini 4.6 GB)		 Full CPU feature set on a JIT-compiled JVM. Runs on any x86-64 CPU (including pre-AVX2 hardware) with standard JVM diagnostics and tooling.
latest-gpu JVM + NVIDIA CUDA 13.2 16 GB ~3 s (no chat)
~25 s (Qwen3.8-27B BF16, TP=4)		 ~34 GiB per GPU (~136 GiB across 4 GPUs, Qwen3.8-27B BF16) LibTorch/CUDA LLM engine (Llama, Qwen) with CUDA graphs, FlashInfer, MTP speculative decoding (up to 2.8× single-request speedup, 906 tok/s batched), tensor parallelism, ONNX GenAI, and S3 blob storage.

Architecture comparison

SMILE vs. ONNX Runtime / LiteRT

Depending on what your edge workload does, you can run SMILE standalone or pair its native ONNX routing with classical ML and local SLMs in a single process:

Feature / Goal ONNX Runtime / LiteRT SMILE Engine
Primary Use Case Deploying heavy deep learning models (YOLO, ResNet, BERT) trained elsewhere. End-to-end edge data science: feature engineering, classical ML, analytics, ONNX routing, and local LLMs.
On-Device Training Incredibly difficult; optimized almost exclusively for static inference. Supported natively. Actively train algorithms (Random Forests, Gradient Boosting, K-Means, DBSCAN) directly on edge data.
Dependencies & Footprint Typically deployed inside a bloated Python environment alongside bulky platform-specific native binaries—resulting in multi-GB footprints, high memory overhead, and poor real-time predictability on edge hardware. Pure Java by default (or a self-contained GraalVM native binary) with deterministic latency; native extensions (OpenBLAS, ORT, LibTorch) are completely optional.
Explainability External post-processing required; no built-in feature attribution. Built-in TreeSHAP & KernelSHAP computed in-process per inference request or SSE stream, with interactive Waterfall, Force, and Decision plots in the Web UI.
Serving & Operations C++/Python library; requires custom wrapper for REST, UI, or metrics. Turnkey Quarkus server (0.2 GB micro native image, 0.4 s boot, ~80 MiB idle RAM) with OpenAI-compatible API, React UI, and Prometheus metrics.

A 30-day pilot

A few of your nodes, your models, your network

The operating system, the firmware, and the disk stay with what you already have. HALO is the inference runtime on top. We package the models you already have, put the runtime next to the traffic, and measure a live score, a local explanation, a model swap, and a run with the network unplugged. Tell us where the nodes sit and what you score today.