Regenerable Software Lab

An experimental benchmark for evaluating whether AI coding agents can repeatedly generate software from a durable, implementation-independent verification bundle.

Phase 4: runner, CLI, harness adapters, and evaluator are operational. Profiles A and B are implemented. Profile C is partial.

GitHub Read the Spec Setup Guide

What

Most AI coding benchmarks evaluate issue resolution against an existing repository. Regenerable Software Lab treats source code as a replaceable candidate implementation and measures whether agents can satisfy progressively stronger verification profiles from the same specification bundle.

The first benchmark is a small HTTP order-pricing API. Multiple coding models and agent harnesses receive the same specification and must produce an implementation that passes verification profiles spanning public tests, hidden behavioral tests, mutation testing, and (partially) operational constraints.

How It Works

SPEC + Benchmarksrsl CLI (6 commands)Runner (orchestration)Agent WorkspaceHarness adapter -> model -> codeEvaluator9-stage verification pipelineReports + Traces + Metrics

Verification Profiles

Profile A - Basic

Install, Build, Lint, Typecheck, Public Tests, Contract Validation

Evaluator stages 1-6

Implemented

Profile B - Behavioral

Hidden Tests, Property Tests, Mutation Testing

Evaluator stages 7-9; local oracle passes against reference-impl

Implemented

Profile C - Operational

Dependency Policy, Secret Scanning, Performance Budgets

Partial; not a complete operational gate yet

In Progress

CLI Commands

node apps/cli/dist/main.js [command]
run - Run a single benchmark against a candidate
verify - Run verification against a workspace
compare - Compare two benchmark run results
experiment - Run an experiment with multiple configurations
report - Display or export a benchmark run report
trace - Inspect trace events from a benchmark run

After pnpm build, invoke the compiled CLI entrypoint directly. Named harnesses use --harness-id (for example fake, codex, droid).

Harness Adapters

Codex CLI
@rsl/harness-codex
AgentHarness
Factory Droid
@rsl/harness-droid
AgentHarness
Claude Code
@rsl/harness-claude-code
AgentHarness
Generic CLI
@rsl/harness-generic-cli
AgentHarness
Fake (CI)
@rsl/harness-fake
AgentHarness

Quickstart

Terminal
$ git clone https://github.com/rmax-ai/regenerable-software-lab.git
$ cd regenerable-software-lab
$ pnpm install && pnpm build
$ pnpm test
$ pnpm --filter reference-impl test
$ pnpm --dir benchmarks/order-pricing/hidden install --ignore-workspace
$ pnpm test:hidden && pnpm test:property
$ node apps/cli/dist/main.js run order-pricing --harness-id fake --profile basic
  • pnpm test runs workspace package tests (many packages still stub).
  • pnpm --filter reference-impl test runs the public order-pricing suite.
  • Hidden/property suites need a one-time install under benchmarks/order-pricing/hidden.
  • Fake-harness smoke runs exercise the runner path without real model calls.

Tech Stack

TypeScript 5.xNode.js >= 24pnpm (monorepo)FastifyZodVitestStryker Mutator

Packages

benchmark-core
Shared types, config parsing, schemas
runner
Run lifecycle orchestration, budgets
evaluator
9-stage verification pipeline
trace
Normalized JSON Lines event collection
metrics
Metric computation and aggregation
policies
Dependency allowlist, network/filesystem policies
reporting
Markdown, JSON, CSV report generation
harness-adapters
AgentHarness interface + 5 adapters