bitpro가 원문에서 글자만 모은 내용이에요. 표·그림·수식은 빠지거나 깨질 수 있어요. 전체 16쪽 중 8쪽
generate problems, solve them, and evaluate solutions, all within the same peer-to-peer network.
#### What Makes CodeZero Different
CodeZero introduces a new task domain focused on programming challenges evaluated by model-based reward functions. 
This represents a shift from the previous **Reasoning Gym** environment, which focused on math and logic tasks verified by symbolic correctness checks.
In CodeZero, models participate as **Proposers**, **Solvers**, and **Evaluators**, each playing a distinct role in the collective learning process. This multi-agent architecture enables dynamic difficulty adjustment, peer-to-peer knowledge sharing, and continuous improvement through reinforcement learning.
### Roles
In CodeZero, **\[1]** Proposers, **\[2]** Solvers, and **\[3]** Evaluators collaborate to create, address, and review programming challenges, forming an ecosystem where models instruct, critique, and enhance one another.
* **Proposers:** Generate coding problems and unit tests, adjusting difficulty dynamically based on solver performance. Proposers create challenges that adapt to the swarm's current capabilities, ensuring continuous learning opportunities.
* **Solvers:** Attempt coding challenges, learn locally through RL, and share rollouts with peers. Solvers exchange solutions to promote diversity and accelerate collective learning across the network.
* **Evaluators:** Frozen models that assess correctness and assign rewards. Evaluators use rule-based assessment to score submissions without executing code, ensuring safety and scalability.
#### Training Loop
The CodeZero training cycle follows a structured progression:
{% stepper %}
{% step %}
### Question Generation
Proposers create coding tasks and tests, drawing from their learned patterns and difficulty adjustment logic.
{% endstep %}
{% step %}
### Sampling
Solvers draw tasks from proposers or from small external datasets (MBPP, CodeContests) for fallback stability.
{% endstep %}
{% step %}
### Rollout Sharing
Solvers exchange solutions with peers to promote diversity and accelerate learning across the swarm.
{% endstep %}
{% step %}
### Evaluation
Evaluators score rollouts using a frozen model (no code execution) to assess structure, formatting, and predicted correctness.
{% endstep %}
{% step %}
### Reward Assignment
Scoring combines structure, formatting, and predicted correctness into a composite reward signal.
{% endstep %}
{% step %}
### Difficulty Adjustment
Proposers adjust challenge levels based on solver success rates, maintaining an optimal learning curve.
{% endstep %}
{% step %}
### Policy Update
Solvers optimize locally via GRPO (Group Relative Policy Optimization), incorporating feedback from the swarm's collective experience.
{% endstep %}
{% endstepper %}
### Technical Details
This section outlines the datasets, model architectures, metrics, and optimization strategies that enable CodeZero to learn safely and autonomously across a decentralized network of peers.
#### Datasets
CodeZero uses two primary datasets for fallback stability and baseline challenges:
* **MBPP** **(Mostly Basic Python Problems)**: A curated set of Python programming problems with test cases.
* **CodeContests:** A collection of competitive programming challenges used for additional task diversity.
These datasets provide a stable foundation when proposer-generated tasks need supplementation.
#### Models
CodeZero employs the Qwen model family across different roles:
* **Qwen 2.5 Coder (0.5B and 1.5B):** Used for Solvers, enabling efficient local learning and rollout generation.
* **Qwen 3 (4B):** Used for Proposers and Evaluators, providing stronger generation and assessment capabilities.
#### Metrics
CodeZero tracks performance using two key metrics which provide complementary views of model performance and learning progress.
* **average\@k:** Average performance across `k` attempts, measuring consistency.
* **pass\@k:** Probability of at least one correct solution in `k` attempts, measuring capability.
#### Evaluation Safety
CodeZero uses a **rule-based evaluator** that avoids code execution. Instead, evaluators assess submissions based on:
* Code structure and formatting
* Predicted correctness (via frozen model inference)
* Adherence to problem requirements
#### Dynamic Difficulty
Proposers maintain a **5-level difficulty system** with thresholds for adjustment:
* Difficulty levels adapt based on solver success rates
* Thresholds trigger automatic adjustment to maintain optimal challenge levels
* This ensures the swarm continuously faces appropriately challenging tasks
#### Policy Optimization
Solvers use **GRPO (Group Relative Policy Optimization)** for local policy updates:
* Incorporates feedback from the swarm's collective experience
* Enables efficient learning from peer rollouts
* Maintains local autonomy while benefiting from network-wide signals
### Integration with RL Swarm
CodeZero runs on the same RL Swarm infrastructure as previous environments:
* **Same network:** Uses the existing peer-to-peer gossip protocol
* **Same identity:** Node identities and `swarm.pem` files work identically
* **Same setup:** No changes required to node installation or configuration
### Next Steps
If you're ready to test out RL Swarm in the latest environmental iteration, CodeZero, check out the [Getting Started](/testnet/rl-swarm/getting-started) guide or learn about [node management](/testnet/rl-swarm/node-management) and [troubleshooting](/testnet/rl-swarm/troubleshooting) steps. 
# Legacy Environments
An archive of legacy RL Swarm environments, including Reasoning Gym, which has been replaced by CodeZero.
## Overview
Before CodeZero, RL Swarm used an early research environment called **Reasoning Gym**.
Reasoning Gym focused on math and logic tasks verified by symbolic correctness checks. It provided a foundation for distributed reinforcement learning research and demonstrated the viability of peer-to-peer RL training.
#### Deprecation
{% hint style="danger" %}
This environment is now **deprecated and archived**.
{% endhint %}
This environment has been deprecated and replaced by CodeZero.
All current nodes now run CodeZero automatically, and no manual migration is required beyond a simple `git pull` command to update. Existing nodes retain their same network, identity, and connection structure during the transition.
#### Archived Resources
* [Reasoning Gym Repository](https://github.com/gensyn-ai/reasoning-gym): Original implementation and research codebase
#### Migration Notes
If you have documentation or scripts referencing Reasoning Gym:
* **Node setup:** No changes required. CodeZero uses the same installation process.
* **Identity files:** `swarm.pem` files remain compatible.
* **Network connection:** Same peer-to-peer protocol and gossip mechanism.
The transition from Reasoning Gym to CodeZero is transparent to end users, as both environments run on the same RL Swarm infrastructure.
{% hint style="success" %}
RL Swarm's architecture is designed to support multiple environments. 
As new environments are developed, they will be documented here alongside CodeZero.
{% endhint %}
# Getting Started
Browse resources for getting started with RL Swarm.
{% embed url="<https://github.com/gensyn-ai/rl-swarm>" %}
## Overview
RL Swarm is a peer-to-peer reinforcement-learning swarm you can run on a laptop or a GPU server. This page is an orientation & navigation hub. 
Start by picking your OS below and follow the platform guide. If something breaks, start at [Troubleshooting](/testnet/rl-swarm/troubleshooting) or join the [Gensyn Discord](https://discord.com/invite/gensyn) for help.
### Before you Begin
RL Swarm is available across several environments: [Windows (via WSL 2)](/testnet/rl-swarm/getting-started/windows-wsl-2), [Linux (Ubuntu 22.04+)](/testnet/rl-swarm/getting-started/linux), and [macOS (Intel and Apple Silicon)](/testnet/rl-swarm/getting-started/macos). 
### System Requirements
* A 64-bit arm64 or x86 CPU with at least 32 GB RAM, or an officially supported NVIDIA GPU (3090, 4090, 5090, A100, H100)
* Between **Python 3.10** and **Python 3.13**
* [Docker](https://www.docker.com/) and [Git](https://git-scm.com/install/) installed & configured
* Stable internet connection
{% hint style="danger" %}
Python 3.14 is incompatible with RL Swarm.
{% endhint %}
#### Accounts & Access
* **Hugging Face account:** You’ll need a Write-access API token to participate in the swarm.
* **Gensyn Testnet account:** Automatically created when you first log in through your browser during setup.
### Supported Platforms & Installation Paths
<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Windows (WSL 2)</strong></td><td>Run RL Swarm via Docker inside WSL 2 for a consistent Linux-based environment.</td><td><a href="/testnet/rl-swarm/getting-started/windows-wsl-2">Windows (WSL 2)</a></td></tr><tr><td><strong>Linux (Ubuntu 22.04+)</strong></td><td>Recommended for most users. Run RL Swarm directly or inside Docker.</td><td><a href="/testnet/rl-swarm/getting-started/linux">Linux</a></td></tr><tr><td><strong>macOS (Intel & Apple Silicon)</strong></td><td>Best for M1/M2/M3 users with Apple Silicon. <em>Supports CPU-only training.</em></td><td><a href="/testnet/blockassist/getting-started/macos">macOS</a></td></tr></tbody></table>
#### FAQ
<details>
<summary>Can I run RL Swarm on my laptop?</summary>
Yes. If you don’t have a GPU, you’ll run in CPU mode. It will be slower, but you’ll still participate in the swarm.
</details>
<details>
<summary>What happens when I log in?</summary>
You’ll create an on-chain identity (via Alchemy) and a local `swarm.pem` file that identifies your peer.
</details>
<details>
<summary>What if I want to switch machines?</summary>
Copy your `swarm.pem` file to the new machine. That preserves your peer identity and animal name.
</details>
<details>
<summary>What if I encounter errors or crashes?</summary>
Check the [Troubleshooting](/testnet/rl-swarm/troubleshooting) guide first, or visit our [Discord](https://discord.com/invite/gensyn) for help.
</details>
#### Resources
If you need additional set-up or post-launch support, you can [open a ticket](https://github.com/gensyn-ai/rl-swarm/issues) or [visit our Discord.](https://discord.com/invite/gensyn)
| [Troubleshooting Guide](/testnet/rl-swarm/troubleshooting) | Common setup issues and fixes. |
| ---------------------------------------------------------- | ------------------------------------------------------------------ |
| [Gensyn Discord](https://discord.gg/gensyn) | Ask questions, report issues, or chat with the team. |
| [Gensyn Blog](https://gensyn.ai/blog) | Learn more about the AI Prediction Market and current experiments. |
| [Gensyn Testnet Dashboard](https://dashboard.gensyn.ai) | Track your peer’s training and on-chain activity. |
# Windows (WSL 2)
Spin up your node and participate in the swarm in a Windows (WSL 2) environment.
{% hint style="success" %}
**Updating RL Swarm to the CodeZero Environment**
Run `git pull`, then restart your swarm.
1. **Docker users:** Run `docker-compose run --rm --build -Pit swarm-cpu` or `docker-compose run --rm --build -Pit swarm-gpu`  
2. **Script users:** Rebuild your venv by running: `rm -rf .venv && python -m venv .venv && source .venv/bin/activate`
{% endhint %}
## Overview
This guide walks you through setting up RL Swarm on Windows using WSL 2 (Windows Subsystem for Linux). 
This allows you to run a native Linux environment compatible with Docker and Python, ensuring consistent results across platforms.
### Prerequisites
Make sure your system meets the minimum requirements and that you also have any additional dependencies installed. 
* A WSL 2 environment
* A 64-bit arm64 or x86 CPU with at least 32 GB RAM, or an officially supported NVIDIA GPU (3090, 4090, 5090, A100, H100)
* Python 3.10+
* [Docker](https://www.docker.com/) installed and configured
* Stable internet connection
* [Git](https://git-scm.com/install/) installed
### Installing WSL 2
1. Open **PowerShell** as Administrator.
2. Install WSL using the following command:
```bash
wsl --install
```
3. Restart your computer when prompted.
4. After reboot, open the Microsoft Store and install Ubuntu.
5. Launch Ubuntu and set up a username and password when prompted.
### Installing Dependencies
Once your Ubuntu environment is installed and open, install the required system dependencies.
Run the following commands one at a time to install Python, Docker, Git, and supporting packages:
```bash
sudo apt update
```
```bash
sudo apt install -y python3 python3-venv python3-pip curl wget git docker.io build-essential
```
#### Docker
Make sure you have Docker installed and the Docker daemon is running on your machine. To do that, follow [these instructions](https://docs.docker.com/get-started/get-docker/) according to your OS. Make sure you allot sufficient memory to the Docker containers. 
For example, if you are using Docker Desktop, this can be done by going to **Docker Desktop Settings > Resources > Advanced > Memory Limit**, and increasing it to the *maximum* possible value.
If you installed Docker via the command line, you can start it and spin up containers by running:
```bash
sudo service docker start
```
#### Clone the RL Swarm Repository
1. Navigate to your home directory in your WSL 2 environment and clone the RL Swarm GitHub repository using this command:
```bash
git clone https://github.com/gensyn-ai/rl-swarm.git
```
2. Then move into the project folder:
```bash
cd rl-swarm
```
{% hint style="info" %}
WSL paths differ from Windows (e.g., /home/user vs C:\Users\\). Make sure to double-check all filepaths if you're not copy-pasting from this guide.
{% endhint %}
#### Run RL Swarm
Depending on your hardware, you can run RL Swarm in either **CPU** or **GPU** mode.
{% tabs %}
{% tab title="CPU-Only" %}
For CPU-only setup (the default on most Windows machines):
```bash
docker compose run –rm –build -Pit swarm-cpu
```
{% endtab %}
{% tab title="GPU-Only" %}
For GPU-enabled setup (requires WSL and an NVIDIA GPU):
```bash
docker compose run –rm –build -Pit swarm-gpu
```
{% hint style="warning" %}
GPU support requires NVIDIA drivers and WSL integration enabled in Docker Desktop.
{% endhint %}
{% endtab %}
{% endtabs %}
<div data-with-frame="true"><figure><img src="https://1034405018-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FIcazOdplbOmP4R0T7sG8%2Fuploads%2FJylnfEDauK6g57P2KET2%2Frl_swarm_initial_spin_up.png?alt=media&token=fb157527-d254-4b2d-a9b2-c056e2ffdb1d" alt=""><figcaption></figcaption></figure></div>
{% hint style="warning" %}
If you encounter an error saying “`docker-compose: command not found`”, use “`docker compose`” (without the hyphen) instead.
{% endhint %}
#### Log into RL Swarm
When you start RL Swarm, it will open a browser window automatically pointing to `http://localhost:3000.`
{% hint style="info" %}
If the browser does not open automatically, navigate to that address manually.
{% endhint %}
You will see the RL Swarm login screen powered by Alchemy. From here, you can log in using your preferred method such as Google or email.
<div data-with-frame="true"><figure><img src="https://1034405018-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FIcazOdplbOmP4R0T7sG8%2Fuploads%2F0lf6tpSRK93cS66x44WY%2Fgensyn_login_screen.png?alt=media&token=61cb7cac-ef06-4cc1-88d8-ed651988ac40" alt=""><figcaption></figcaption></figure></div>
After login, a `swarm.pem` file will be created in your repository folder. This identifies your peer on the Gensyn Testnet.
***
### Joining the new RL-Swarm Environment (CodeZero)
#### Updating via git pull
First, start by running `git pull` to fetch the latest changes to RL Swarm. 
#### Launching after Updating
There are two launch paths.
1. **Launching with Docker:** If you are using Docker, you can restart your swarm after updating via `git pull` by using one of the two following commands:
```bash
docker-compose run --rm --build -Pit swarm-cpu
docker-compose run --rm --build -Pit swarm-gpu
```
2. **Launching with Shell Script:** If you are using the `run_rl_swarm.sh` script, you must remove your old virtual environment and create a fresh one:
```bash
rm -rf .venv
python -m venv .venv
source .venv/bin/activate
/run_rl_swarm.sh
```
{% hint style="success" %}
In both cases, restart your swarm after updating.
{% endhint %}
***
#### Huggingface
If you would like to upload your model to Hugging Face, enter your Hugging Face access token when prompted. You can generate one from your Hugging Face account, under [Access Tokens](https://huggingface.co/docs/hub/en/security-tokens).
#### Verify your Node
Once you are logged in, your node will begin training automatically.
You can verify that your peer has successfully connected by visiting the [Gensyn Testnet Dashboard.](https://dashboard.gensyn.ai/) Your peer should appear in the active swarm list, and you can monitor training progress in real time.
<div data-with-frame="true"><figure><img src="https://1034405018-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FIcazOdplbOmP4R0T7sG8%2Fuploads%2FDWHs4gYzjRqDnRQyjz8z%2Fimage.png?alt=media&token=5f706d42-3174-4469-9fef-1ee3c17fda80" alt=""><figcaption></figcaption></figure></div>
#### Optional: Experimental Mode (No Docker)
If you want to experiment with the [GenRL](https://github.com/gensyn-ai/genrl) library or the [configurable parameters](https://github.com/gensyn-ai/rl-swarm/blob/main/rgym_exp/config/rg-swarm.yaml), we recommend you run RL Swarm via shell script:
```
python3 -m venv .venv
source .venv/bin/activate
./run_rl_swarm.sh
```
{% hint style="info" %}
This method gives you access to GenRL’s configuration parameters and experimental features.
{% endhint %}
To learn more about experimental mode, check out our [getting started guide](https://github.com/gensyn-ai/genrl/blob/main/getting_started.ipynb) on Github.
### Troubleshooting
Refer to the multi-platform [RL Swarm Troubleshooting guide](/testnet/rl-swarm/troubleshooting) for unblocking information and fixes to common set-up issues.
{% hint style="success" %}
If you need additional support, you can [open a ticket](https://github.com/gensyn-ai/rl-swarm/issues) or [visit our Discord.](https://discord.com/invite/gensyn)
{% endhint %}
# Linux
Spin up your node and participate in the swarm in a Linux (Ubuntu 22.04+) environment.
{% hint style="success" %}
**Updating RL Swarm to the CodeZero Environment**
Run `git pull`, then restart your swarm.
1. **Docker users:** Run `docker-compose run --rm --build -Pit swarm-cpu` or `docker-compose run --rm --build -Pit swarm-gpu`
2. **Script users:** Rebuild your venv by running: `rm -rf .venv && python -m venv .venv && source .venv/bin/activate`
{% endhint %}
## Overview
This guide walks you through setting up RL Swarm on a Linux machine.
Linux provides the most stable and performant environment for RL Swarm, especially for users running NVIDIA GPUs. You can run RL Swarm via Docker for simplicity or directly through Python for more advanced experimentation.
### Prerequisites
Make sure your system meets the minimum requirements and that you also have any additional dependencies installed. 
* Ubuntu 22.04+
* A 64-bit arm64 or x86 CPU with at least 32 GB RAM, or an officially supported NVIDIA GPU (3090, 4090, 5090, A100, H100)
* Python 3.10+
* [Docker](https://www.docker.com/) installed and configured
* Stable internet connection
* [Git](https://git-scm.com/install/) installed
### Installing Dependencies
First, update your package lists and install all required dependencies:
```ba프로젝트로 돌아가기
