Regenerable Software Lab

An experimental benchmark for evaluating whether AI coding agents can repeatedly generate software from a durable, implementation-independent verification bundle.

GitHub Read the Spec

What

Most AI coding benchmarks evaluate issue resolution against an existing repository. Regenerable Software Lab treats source code as a replaceable candidate implementation and measures whether agents can satisfy progressively stronger verification profiles from the same specification bundle.

The first benchmark is a small HTTP order-pricing API. Multiple coding models and agent harnesses receive the same specification and must produce an implementation that passes three verification profiles spanning public tests, hidden behavioral tests, mutation testing, and operational constraints.

How It Works

SPEC + Benchmarksrsl CLI (6 commands)Runner (orchestration)Agent WorkspaceHarness adapter → LLM → codeEvaluator9-stage verification pipelineReports + Traces + Metrics

Verification Profiles

Profile A — Basic

Install, Build, Lint, Typecheck, Public Tests, Contract Validation

Implemented

Profile B — Behavioral

Hidden Tests, Property Tests, Mutation Testing

Implemented

Profile C — Operational

Dependency Policy, Secret Scanning, Performance Budgets

In Progress

CLI Commands

rsl [command]
run — Run a single benchmark against a candidate
verify — Run verification against a workspace
compare — Compare two benchmark run results
experiment — Run an experiment with multiple configurations
report — Display or export a benchmark run report
trace — Inspect trace events from a benchmark run

Harness Adapters

Codex CLI
@rsl/harness-codex
AgentHarness
Factory Droid
@rsl/harness-droid
AgentHarness
Claude Code
@rsl/harness-claude-code
AgentHarness
Generic CLI
@rsl/harness-generic-cli
AgentHarness
Fake (CI)
@rsl/harness-fake
AgentHarness

Quickstart

Terminal
$ git clone https://github.com/rmax-ai/regenerable-software-lab.git
$ cd regenerable-software-lab
$ pnpm install && pnpm build
$ pnpm test
✓ 168 tests passed (packages)
$ cd benchmarks/order-pricing/reference-impl
$ pnpm install && pnpm build && pnpm test
✓ 136 tests passed (reference impl)

Tech Stack

TypeScript 5.9Node.js >= 24pnpm (monorepo)FastifyZodVitestStryker Mutator

Packages

benchmark-core
Shared types, config parsing, schemas
runner
Run lifecycle orchestration, budgets
evaluator
9-stage verification pipeline
trace
Normalized JSON Lines event collection
metrics
Metric computation and aggregation
policies
Dependency allowlist, network/filesystem policies
reporting
Markdown, JSON, CSV report generation
harness-adapters
AgentHarness interface + 5 adapters