Setup and Run

• Python 3.12 or newer.
• Docker with a running daemon. Agents and all validation gates run inside per-repository images.
• git to clone the target repositories at their pinned commits.
• An OpenRouter API key.

Clone the repository and install it into a virtual environment.

git clone https://github.com/sweduel/SWE-Duel.git
cd SWE-Duel
python3.12 -m venv venv
./venv/bin/pip install -e ".[dev]"     # or: make install
source venv/bin/activate               # puts the swe-duel-* commands on PATH

Set your OpenRouter API key

export SWE_DUEL_OPENROUTER_API_KEY="sk-or-..."

What lives where

Clone every repository listed in config/repos/ at its pinned commit, then build the swe-duel-base, swe-duel-mock, and swe-duel-<repo> images. A full build takes a while; use --only to start with a subset.

swe-duel setup repos                       # → ./repos/<name> at pinned commits
swe-duel setup docker                      # all images (add --force to rebuild)

# or start small:
swe-duel setup repos  --only flask jinja chi
swe-duel setup docker --only flask jinja chi

swe-duel doctor command test-runs the arena's real code paths (install, config, Docker daemon, images, clones, an offline test run, the deterministic scorer, and a replay of sealed gate-regression fixtures) to ensure that the environment is ready to start generating tasks and running tournaments.

swe-duel doctor                     # host + Docker stages, no API spend
swe-duel doctor --all-repos         # replay every repo's sealed gate fixture (slower)
swe-duel doctor --live              # + one tiny OpenRouter call per configured model
swe-duel doctor --live --smoke-agent glm-5.3-flash   # + one real agent session

Each of the swe-duel doctor ... command fails fast on a missing Docker daemon, a missing swe-duel-<repo> image, or a missing clone, and suggest the exact swe-duel setup … command that fixes it.

A participant is the 4-tuple (model, harness, reasoning effort, provider). Thus, the same model under a different harness, at a different effort, or served by a different OpenRouter provider counts as a distinct participant.

Register models you want to evaluate in config/models.yaml

models:
  deepseek-v4-flash-0731:                  # nickname displayed on the CLI
    model_id: "deepseek/deepseek-v4-flash-0731"
    temperature: 0.2
    reasoning_efforts: ["max", "high", "low"]       # will become selectable on CLI
    providers: ["deepseek", "wafer/fast", "fireworks"]   # will become selectable on CLI
    provider_rate_limits:                  # requests/minute, shared by all workers
      "deepseek": 60
      "wafer/fast": 60
      "fireworks": 40
      "default": 6                         # auto-route / unlisted providers

No per-token prices are statically configured in the config/models.yaml. SWE-Duel tracks cost by reading from the per-response metadata for accurate OpenRouter cost reports.

You may add additional OpenRouter providers, and probe their rate limits with real API calls:

swe-duel-probe-rate-limits --models deepseek-v4-flash-0731 --providers fireworks

SWE-Duel Harnesses

In order for participant to enter a tournament or run a match against another participant, it should generate Red tasks (interchangeably, challenges) for every repository involved in the match or tournament. Admitted tasks are stored under data/challenge_bank/.

# interactive wizard (needs a TTY)
swe-duel-generate \
  --repos jwt helmet java-html-sanitizer cjson flask jinja expressjs java-jwt libexpat chi sqlalchemy simdjson jjwt node-jsonwebtoken csrf \
  --target-per-repo 1  

• Task generation's --target-per-repo argument maps to the tournament's --turns_per_player argument (match.turns_per_player in arena.yaml, default 3). Each match draws that many challenges per repository per side, in deterministic bank order.
• Slots are deterministic (1..target). Re-running skips every slot that already has a record (successfully admitted, or failed after challenge_bank.max_generation_attempts attempts), so the command is resume-safe and never re-spends on a finished/exhuasted slot.
• A slot that does not store an admitted challenge, but is exhausted due to reaching challenge_bank.max_generation_attempts failed attempts will not manifest as error during a match or tournament. Rather, in a match, the opponent is awarded an auto-win that Blue turn.

The round-robin tournament schedules a match between every two participants in the tournament. A single match spans all selected repositories: the two participants alternate Red and Blue for turns_per_player challenges per repository per side. Blue defenses are cached in data/defenses/ per (challenge, Blue participant), so later tournaments reuse the cached records to save costs.

# interactive wizard (needs a TTY)
swe-duel-tournament-rr \
  --repos jwt helmet java-html-sanitizer cjson flask jinja expressjs java-jwt libexpat chi sqlalchemy simdjson jjwt node-jsonwebtoken csrf \
  --turns-per-player 1

On start the console UI lists existing round_robin_state_*.json files for the same repository set and turns-per-player configuration, so you can resume a tournament or start a new one. Commands for navigating the console UI:

State persists to data/tournaments/round_robin_state_<tournament_id>.json and match records to data/matches/, so an interrupted tournament resumes exactly where it stopped. After resuming, the console also offers Add a new participant: newcomers are onboarded with ceil(log2 N)+1 matchups against the incumbents via the active sampling algorithm, instead of a matching against the entire incumbent pool.

Once the matches of the tournament are played, export the tournament's standings from ./data/.

ls data/tournaments/                 # round_robin_state_<tournament_id>.json
swe-duel-rankings <tournament_id>
# Will output to ./rankings/rankings_<tournament_id>.json   (override with --rankings-dir)

The export is self-contained:

• rankings: pinned identity (model, harness, effort, provider), points, matches played, match-level Bradley–Terry and Elo with ranks, TrueSkill μ/σ, and 1,000-draw match-level bootstrap uncertainty (Elo/BT confidence intervals and bootstrap rank distributions).
• head_to_head: W/L/D per pair, with match ids.
• matches: the raw claimed match list (endpoints, outcome, timestamp, repositories).
• repos + targets_per_repo: the exact arena configurations the tournament ran.