Gensyn 백서 전문

PyTorch module from a given set of operations. #### Python API ```python from gensyn_mjolnir import convert, CompileOptions # Convert an ONNX model to a PyTorch module with reproducible kernels module = convert( "model/model.onnx", options=CompileOptions(requires_reproducibility=True) ) ``` #### Convert() `convert()` is the core function of the Gensyn Compiler. It takes an ONNX model and converts it into a PyTorch module that can be used for inference. When `requires_reproducibility` is enabled (which it is by default), the compiler replaces standard PyTorch operations with RepOp kernels that guarantee bitwise-identical results across hardware. It has two parameters: * `onnx_model_or_path`: A `str`, `Path` or `ModelProto` type. This is either a file path to an ONNX model on a disk or an in-memory `ModelProto` object. * `options` (`CompileOptions`) is the configuration for the compilation process, which you can read more about below. It defaults to reproducible mode. ```python def convert( onnx_model_or_path: str | Path | ModelProto, options: CompileOptions = CompileOptions(), ) -> torch.nn.Module ``` #### CompileOptions `CompileOptions` controls how the compiler processes the model. In most cases the defaults are what you want, which corresponds to reproducible mode with symlinked tensors and a temporary artifacts directory. The fields are: * `artifacts_dir`: always `str` or `None`. This is the directory where the compiler writes intermediate artifacts. If not set, a temporary directory is used (preserved when the `MJOLNIR_DEBUG` environment variable is set, which is useful for inspecting compiler output during debugging). * `colocate_tensors`: A `bool` that is `False` by default. When set to `True` it copies external tensor files into the artifacts directory. When `False`, it creates symbolic links instead. {% hint style="info" %} *Symlinking* is faster and saves disk space, but copying may be needed if you plan to move the artifacts directory to another location. {% endhint %} * `requires_reproducibility`: Also a `bool` but set to `True` by default. When `True`, the compiler replaces standard PyTorch operations with RepOp kernels for cross-hardware reproducibility. When `False`, it uses standard PyTorch kernels which are faster but not reproducible across different hardware. ```python @dataclass(frozen=True, slots=True, kw_only=True) class CompileOptions: artifacts_dir: str | None = None # Directory for compiler artifacts. Uses a temp directory if not specified # (preserved when MJOLNIR_DEBUG env var is set). colocate_tensors: bool = False # When True, copies external tensor files into the artifacts directory. # When False, creates symbolic links instead. requires_reproducibility: bool = True # When True, compiles with RepOp kernels for cross-hardware reproducibility. # When False, uses standard PyTorch kernels. ``` ### Inference Sessions REE v0.3.0 introduces `InferenceSession`, an SDK abstraction for managing the lifecycle of an inference workflow. An inference session is responsible for setup, execution, and teardown. This lets applications reuse session state across multiple inference calls within the same container instance instead of repeating setup steps for every generation. This is useful for: * Running multiple prompts against the same prepared environment. * Building multi-turn inference workflows. * Implementing tool-call loops where the model may request an external tool before producing a final answer. * Avoiding repeated compiler initialization where session reuse is supported. ### RepOps RepOps (Reproducible Operators) are purpose-built GPU kernels that guarantee bitwise-identical outputs regardless of hardware architecture. They cover the full set of operators needed for neural network inference and training. For a standalone demo of RepOp kernels, see the [RepOps Demo repository](https://github.com/gensyn-ai/repops-demo). #### How RepOps Achieve Cross-Hardware Reproducibility * **Fixed reduction ordering:** Every kernel accumulates values in a single canonical order. The reduction tile size is fixed across all GPU architectures. All accumulation is in FP32 using fused multiply-add instructions. * **Correctly rounded transcendentals:** Custom implementations of `exp`, `sin`, `tanh`, etc. that produce identical results on every CUDA-capable GPU. * **Extended-precision arithmetic:** Operations like the error function (used in GELU) use extended-precision fixed-point arithmetic for cross-hardware consistency. * **Architecture-adaptive output tiling:** Kernels adapt output tile dimensions to different GPU architectures (using available shared memory), but never change the reduction dimension, so reproducibility is preserved. REE `v0.6.0` adds reproducible int8 attention, fused gathered-log-probability, and MoE expert GEMM kernels. It also improves reproducible attention, matmul, softmax, and RMSNorm performance. {% hint style="warning" %} The `v0.6.0` 3D RMSNorm kernel changes bit-level output. Some receipts generated with `v0.5.0` may not re-verify with `v0.6.0`. See [Receipts](/tech/ree/receipts) for compatibility guidance. {% endhint %} REE `v0.7.0` and `v0.8.0` add reproducible attention, stochastic rounding, and grouped expert matmul kernels. They also speed up CUDA inference with optimized decode attention, GEMVs, GEMM, and fused RMSNorm. More operators now match CPU results byte-for-byte on Apple silicon. ### Pipeline Parallelism Large models often exceed the memory of a single GPU. Pipeline parallelism splits the model's layers into sequential *partitions* (also called stages), each placed on a different GPU. {% hint style="danger" %} Pipeline Parallelism is only possible on multi-GPU devices. Using the flag to enable this mechanism on a single-GPU host will result in a failed run. {% endhint %} A forward pass walks the input through partition 1, then passes its activations to partition 2, and so on, with each GPU holding only its own slice of the weights. This trades a single large memory footprint for several smaller ones, making it possible to run models that would otherwise be impossible to load. {% hint style="info" %} Pipeline parallelism is orthogonal to the operation set. You can combine `--n-partitions` with `default`, `deterministic`, or `reproducible` mode. Only `reproducible` mode guarantees bitwise identity across different hardware. {% endhint %} REE exposes this through the `--n-partitions` flag, which controls how many partitions the model is divided into. On a host with enough aggregate GPU memory, this lets REE run models up to 72B parameters while preserving the reproducibility guarantees provided by [RepOps](#repops). Partition boundaries are deterministic for a given model and partition count, so splitting a model does not introduce new sources of numerical drift: the same `--n-partitions` value on different supported hardware produces bitwise-identical output, and cross-partition-count runs of the same model also match when using `--operation-set reproducible`. ### Container Details | Property | Value | | ---------------- | ------------------------- | | REE Image | `gensynai/ree:v0.8.0` | | REE Version | `0.8.0` | | SDK Version | `gensyn-sdk 0.1.0` | | Compiler Version | `gensyn-compiler 0.1.0` | | Base OS | Ubuntu 24.04.1 LTS | | Python | `3.11.15` | | PyTorch | `2.12.1` | | Transformers | `5.10.4` | | urllib3 | `2.7.0` | | ONNX | `1.21.0` | | Entrypoint | `/runtime/bin/gensyn-sdk` | | User | `gensyn` (non-root) | | Working Dir | `/home/gensyn` | #### Inference Sessions `InferenceSession` is a stateful SDK wrapper that loads a prepared task directory once and can run multiple completions against it. At initialization, the session reads prepare metadata, loads the tokenizer and generation config, configures the selected operation set, selects the device, and builds the inference backend. Calls to `complete()` reuse that state. `InferenceSession.complete()` supports either a plain `prompt` or chat-style `messages`. Tool definitions can be supplied only with `messages`, and are forwarded into the tokenizer's chat template when supported. ### Interactive Mode To explore REE's components directly (SDK, Compiler), start the container in interactive mode: ```bash docker run -it --entrypoint bash -v ~/.cache:/home/gensyn/.cache gensynai/ree:v0.8.0 ``` From inside the container, you can run `gensyn-sdk` commands directly and inspect intermediate artifacts. # Troubleshooting Common errors, their causes, and how to fix them. ## Common Errors Quick fixes for Docker, CLI, and compilation issues you might hit while using REE. | Error | Cause | Fix | | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PermissionError: [Errno 13] Permission denied: '/gensyn'` | Container runs as non-root `gensyn` user; can't write to root-owned paths | Use `/tmp/` paths for ephemeral runs, or mount a volume with `-v` | | `one of the arguments --tasks-root --task-dir is required` | Missing required output directory argument | Add `--task-dir /tmp/task` or `--tasks-root /tmp/tasks` | | `one of the arguments --prompt-text --prompt-file is required` | Missing prompt input | Add `--prompt-text "your prompt"` or `--prompt-file path.jsonl` | | `argument command: invalid choice: 'bash'` | Trying to launch a shell but entrypoint is locked to `gensyn-sdk` | Use `--entrypoint bash` to override: `docker run -it --entrypoint bash ree` | | Gibberish / nonsensical output | Using `hf-internal-testing/tiny-random-LlamaForCausalLM` which has random untrained weights | Expected behavior for test models; use a real model for meaningful output | | Shell hangs after pasting command | Trailing `\\` on the last line of a command | Remove the backslash from the final line | | `--n-partitions` run fails on a single-GPU machine | Pipeline parallelism requires multiple GPUs; a partition count above `1` can't be satisfied with only one device. | Omit `--n-partitions` or set it to `1`. Single-GPU hosts can only run models that fit in one GPU's memory. | | A `v0.5.0` receipt fails `verify` with `v0.6.0` | The `v0.6.0` 3D RMSNorm kernel produces different bit-level output. | Generate new receipts with `v0.8.0`. | | A `v0.6.0` receipt fails `verify` with `v0.7.0` or later | `v0.7.0` changed tokenization for some models (Transformers 5.x) and how some kernels handle signed zero and subnormal inputs. | Verify the legacy receipt with a compatible `v0.6.0` environment. Generate new receipts with `v0.8.0`. See Receipts. | | A `v0.7.0` receipt fails `verify` with `v0.8.0` | `v0.8.0` changed CUDA decode and speculative-verify attention for contexts longer than 256 tokens, and int8 Hadamard GEMM results on CUDA and Metal when dequantization overflows. | Verify the legacy receipt with a compatible `v0.7.0` environment. Generate new receipts with `v0.8.0`. See Receipts. | | `ree.sh` hangs or fails while installing `acl` on Linux | The `needrestart` prompt, a failed apt list refresh, or another process holding apt's package-list lock. | Pull the latest `ree` repo. Current `ree.sh` retries apt list refreshes, waits up to ten minutes for the package-list lock to clear, and doesn't hang on the `needrestart` prompt. | | `ValueError: 'tools' can only be provided with 'messages'.` | Tool definitions were passed with a plain prompt string in `InferenceSession.complete()`. | Use `messages=[...]` instead of `prompt=...` when providing tools. | | Tool definitions appear to be ignored | The selected tokenizer may not have a chat template, or the model may not be trained to use tools. | Use a chat/instruct model with a tool-aware chat template and verify the model emits the expected tool-call format. | | No parsed tool call is returned | REE provides basic tool-definition support, not a full tool executor or agent loop. | Parse the model output in your application, execute tools yourself, and pass follow-up context back into the session. | ### **Tool-Call Workflows** REE does not execute external tools. If a model emits tool-call-shaped output, your application must parse it, run the tool, and continue the conversation. For reproducible workflows, record the exact tool output passed back to the model. Live APIs, web pages, databases, and search results may change between runs. | Error | Cause | Fix | | ----------------------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `ValueError: 'tools' can only be provided with 'messages'.` | `tools` passed with `prompt=` | Use `messages=[...]` | | No parsed tool call in SDK result | REE does not execute or return structured tool calls | Parse `result.text`, run tools in application code | | `enable_thinking` seems ignored | Plain `prompt` or unsupported tokenizer | Use `messages` and a thinking-capable model such as `Qwen3` | | Cannot set thinking on/off from TUI | `enable_thinking` is SDK-only | Use `InferenceSession.complete(..., enable_thinking=False)` or short-circuit flags on CLI | ### Compiler Trace Warnings When running REE, you may see trace warnings and verbose compiler output in your terminal. These are expected and can be safely ignored. They originate from the ONNX export and MLIR compilation stages. #### `--tasks-root` vs. `--task-dir` If you see errors about missing artifacts, make sure you're using consistent location flags. When using `--tasks-root`, the task directory is automatically derived from the model name. When using `--task-dir`, you must point to the same directory across all operations. ### CUDA Not Available If you're running on a machine with a GPU but REE doesn't detect it, ensure you're passing the `--gpus all` flag to Docker: ```bash docker run --gpus all -v ~/.cache:/home/gensyn/.cache gensynai/ree:v0.8.0 run-all \\ --tasks-root /home/gensyn/.cache/gensyn \\ --model-name <model> \\ --prompt-text "..." \\ --operation-set reproducible ``` {% hint style="info" %} Use `--cpu-only` to explicitly force CPU execution when GPU is not available or not desired. {% endhint %} ### Out-of-Memory (OOM) Issues If you are using Docker Desktop, you may need to [adjust the memory limit.](https://docs.docker.com/desktop/settings-and-maintenance/settings/#advanced) Otherwise, you may attempt to run larger models (models with a higher parameter count) and encounter a failure during the model loading or checkpoint 'sharding' phase. This typically shows up as a `run:failed` status with exit code 137, and the logs will show the process dying partway through *"Loading checkpoint shards."* ### NaN Errors & Crashes with Certain Models Some FP16 models, particularly certain Qwen 2.5 Instruct variants, may produce NaN (Not a Number) errors and crash when run in `default` or `deterministic` mode. This is a numerical stability issue: attention score calculations can overflow the FP16 value range during inference. If you encounter this, try switching to `reproducible` mode (`--operation-set reproducible`), which handles these edge cases more gracefully. Note that even in `reproducible` mode, some affected models *may still produce degraded output quality* (repetitive text or unexpected tokens). {% hint style="info" %} This is a known limitation related to the ONNX export pipeline's use of FP16 precision and is being actively addressed in future releases. {% endhint %} # Agent eXchange Layer (AXL) What AXL is, what it does, and what you can build with it. ## What is AXL? Agent eXchange Layer (AXL) is a peer-to-peer network node built by Gensyn.  It offers an encrypted, decentralized communication layer for applications, allowing AI agents, ML pipelines, distributed computing, and more to exchange data directly between machines *without* a central server. Fundamentally, it works like this: you run the node on your machine where it handles all peer-to-peer transport, encryption, and routing. The node exposes a local HTTP bridge as an application interface, compatible with whatever you're building.  ### Features AXL is designed to stay out of your way. It runs without root access, works behind NATs, and exposes a plain HTTP interface so any language can use it. * **No TUN required:** Runs entirely in userspace using gVisor's network stack. No root privileges, no system-level network configuration. * **No port forwarding needed:** Connects outbound to peers and receives data over the same encrypted tunnel, so standard nodes work behind NATs and firewalls without any extra configuration. If you're bootstrapping a new network from scratch, at least one node needs to be publicly reachable with an exposed port.  {% hint style="info" %} Running a public node on an existing network is also helpful since it adds to the overall robustness of the mesh. {% endhint %} * **Simple local interface:** Your application talks to `localhost:9002`. Any language that can make HTTP requests can use AXL. * **End-to-end encrypted:** All traffic between nodes is encrypted at two layers: TLS for the direct peering link, and Yggdrasil's end-to-end encryption for the full path. Intermediate routing nodes cannot read your messages. * **Application-agnostic:** The node doesn't care what you send. You could send JSON, protobuf, raw bytes, or tensors. * **Protocol support:** AXL features built-in support for [MCP](https://modelcontextprotocol.io/) (Model Context Protocol) and [A2A](https://github.com/google/A2A) (Agent-to-Agent) for structured request/response communication between agents. ### At a Glance Getting two machines talking takes four steps and no infrastructure. 1. You build and run the AXL node binary on your machine. 2. The node connects to the Yggdrasil[^1] mesh network and gets a public key (your identity). 3. You share your public key with another person. They share theirs with you. 4. Your applications communicate through their local nodes. The nodes handle everything else. It doesn't require any servers, cloud accounts, or DNS. It's just two machines (or more) running nodes that communicate directly over the mesh. ``` Your Machine Their Machine ┌──────────────────────┐ ┌──────────────────────┐ │ [Your App] │ │ [Their App] │ │ ↕ HTTP │ │ ↕ HTTP │ │ [AXL node :9002] │ ◄── mesh ──► │ [AXL node :9002] │ └──────────────────────┘ └──────────────────────┘ ``` ### Philosophy AXL enforces a clean separation between the **\[1]** network layer (the node) and the **\[2]** application layer (your code). The node is a *pipe* insofar as it moves bytes between peers, but it has no opinion about what those bytes mean.  This separation means: * You can build any application on top of AXL without modifying the node or worrying about transport protocols. * Multiple applications can share the same node. * The network handles encryption, routing, and peer discovery. Your application handles business logic. {% hint style="info" %} AXL is permissionless. Anyone can run a node or spin up their own private network. There are no gatekeepers. {% endhint %} ### What You Can Build Because AXL is just a communication layer, what you build on top is up to you.  A few [examples](/tech/agent-exchange-layer/examples-and-building#built-in-examples) ship with the repo: * **AI agent collaboration:** Agents on different machines sharing research signals over MCP * **Distributed ML inference:** Tensor exchange between nodes using `msgpack` * **GossipSub:** Pub/sub message propagation across the mesh * **Convergecast:** Tr