How to Add a New Benchmark

This section describes how to add a new benchmark to XuanCe. A benchmark in XuanCe is defined by one algorithm, one environment scenario, and multiple random seeds.

Step 1: Choose the Benchmark Task

Determine the target environment and scenario. For example: - Environment: Atari - Scenario: Breakout-v5

Create the corresponding directory if it does not exist:

Step 2: Create an Algorithm-Specific Directory

Under the scenario directory, create a subdirectory for the algorithm:

For example, for PPO:

Step 3: Prepare the Algorithm Configuration (Optional)

If the algorithm requires a specific configuration file, place it in the algorithm directory:

benchmarks/Atari/Breakout-v5/ppo/ppo_atari.yaml

This configuration file defines hyperparameters and environment-specific settings used by the benchmark.

Step 4: Write the Benchmark Script

Create a benchmark script named:

For example:

Each benchmark script should: - Call the shared train.py script - Run multiple random seeds (default: 5) - Clearly indicate the start and end of each seed - Not duplicate algorithm or environment information already printed by XuanCe

Example structure:

Step 5: (Optional) Add the Benchmark to a Suite Script

If you want the new benchmark to be included in a benchmark suite, edit the suite script under the scenario directory:

Add the new benchmark script to the list:

Step 6: Run and Verify

Run the benchmark script:

Verify that: - All seeds run sequentially - Each seed prints clear START / END markers - Results are saved under the correct directory structure - The benchmark can be reproduced by re-running the script

Design Principles

When adding a new benchmark, please follow these principles:

  • One script = one benchmark

  • Benchmark scripts are the source of truth

  • Do not hard-code absolute paths

  • Do not duplicate logging already handled by XuanCe

  • Prefer clarity and reproducibility over convenience