Class ClusterLeader

java.lang.Object
smile.llm.cluster.ClusterLeader
All Implemented Interfaces:
AutoCloseable
public final class ClusterLeader extends Object implements AutoCloseable
The control side of a multi-node model: the leader process owns the server and the engine, and every call that changes model state (bind, prefill, decode step, finish...) is mirrored to the follower processes, which hold the other ranks and execute the identical call so that the NCCL collectives of all ranks line up.

Wire format: leader to follower [int length][byte op][payload]; follower to leader [byte status][long seq][int value] (status 0 ok, otherwise followed by a UTF error message). Messages of one connection are numbered 1, 2, ... in send order; the follower acknowledges each after executing it.

If a follower fails or disconnects, the collectives of the other ranks would hang forever, so by default the leader halts the JVM (SMILE_CLUSTER_FAIL_FAST=0 only logs).

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static interface 
    ClusterLeader.PayloadWriter
    Writes the arguments of a message.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    OP_SHUTDOWN
    Op code of the shutdown message.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    awaitAck(long n)
    Blocks until every follower has executed message n; throws if one failed.
    void
    checkHealthy()
    Throws if a follower has failed.
    void
    close()
     
    int
    followers()
    Returns the number of followers.
    long
    send(int op, ClusterLeader.PayloadWriter payload)
    Sends one message to every follower (they execute it concurrently with the leader).
    static void
    setFailFast(boolean enabled)
    Whether a failed follower halts the JVM (the default: the other ranks would hang in a collective forever).
    static ClusterLeader
    start(int port, int followers, int leaderRanks, long timeoutMs, Map<String,String> params)
    Waits for the followers to connect, assigns ranks (the leader owns ranks 0 .. leaderRanks - 1, then the followers in nodeIndex order) and sends each follower its welcome: world size, first rank, the NCCL unique id and params.
    static ClusterLeader
    start(int port, int followers, int leaderRanks, long timeoutMs, Map<String,String> params, byte[] uniqueId)
    As start(int, int, int, long, Map) with an explicit NCCL unique id (tests, or a launcher that made it).
    byte[]
    uniqueId()
    Returns the NCCL unique id shared with the followers.
    int
    valueOf(int index, long n)
    The value follower index returned with message n (call after awaitAck(long)).
    int
    worldSize()
    Returns the ranks of the whole communicator.

    Methods inherited from class Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • OP_SHUTDOWN

      public static final int OP_SHUTDOWN
      Op code of the shutdown message.
      See Also:

  • Method Details

    • setFailFast

      public static void setFailFast(boolean enabled)
      Whether a failed follower halts the JVM (the default: the other ranks would hang in a collective forever).
      Parameters:
      enabled - false only logs, for tests.

    • start

      public static ClusterLeader start(int port, int followers, int leaderRanks, long timeoutMs, Map<String,String> params) throws IOException
      Waits for the followers to connect, assigns ranks (the leader owns ranks 0 .. leaderRanks - 1, then the followers in nodeIndex order) and sends each follower its welcome: world size, first rank, the NCCL unique id and params.
      Parameters:
      port - TCP port to listen on.
      followers - number of follower processes to wait for.
      leaderRanks - ranks (GPUs) of the leader process.
      timeoutMs - how long to wait for all followers.
      params - settings every process must share (batch size, memory fraction...).
      Returns:
      the connected cluster.
      Throws:
      IOException - if a follower does not show up or the handshake fails.

    • start

      public static ClusterLeader start(int port, int followers, int leaderRanks, long timeoutMs, Map<String,String> params, byte[] uniqueId) throws IOException
      As start(int, int, int, long, Map) with an explicit NCCL unique id (tests, or a launcher that made it).
      Parameters:
      port - TCP port to listen on.
      followers - number of follower processes to wait for.
      leaderRanks - ranks (GPUs) of the leader process.
      timeoutMs - how long to wait for all followers.
      params - settings every process must share.
      uniqueId - the 128-byte NCCL unique id, or null to create one.
      Returns:
      the connected cluster.
      Throws:
      IOException - if a follower does not show up or the handshake fails.

    • worldSize

      public int worldSize()
      Returns the ranks of the whole communicator.
      Returns:
      ranks of the whole communicator.

    • uniqueId

      public byte[] uniqueId()
      Returns the NCCL unique id shared with the followers.
      Returns:
      the NCCL unique id shared with the followers.

    • followers

      public int followers()
      Returns the number of followers.
      Returns:
      the number of followers.

    • send

      public long send(int op, ClusterLeader.PayloadWriter payload)
      Sends one message to every follower (they execute it concurrently with the leader).
      Parameters:
      op - op code (> 0); the follower's handler decides what it means.
      payload - writes the arguments, or null for none.
      Returns:
      the message number, for awaitAck(long) and valueOf(int, long).

    • awaitAck

      public void awaitAck(long n)
      Blocks until every follower has executed message n; throws if one failed.
      Parameters:
      n - the message number from send(int, ClusterLeader.PayloadWriter).

    • valueOf

      public int valueOf(int index, long n)
      The value follower index returned with message n (call after awaitAck(long)).
      Parameters:
      index - follower, in rank order.
      n - message number.
      Returns:
      the value, or -1 when it returned none.

    • checkHealthy

      public void checkHealthy()
      Throws if a follower has failed.

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable