Skip to main content

Publication state: results unpublished.

Harbor v4 AI coding agent benchmark

Harbor v4 is Lamina’s product-shaping benchmark for AI coding agents. It compares matched workflows on the same public tasks, budget, runtime, and behavior-only measurement.

Each task starts from a fuzzy founder brief. Agents must shape a thin in-memory product with actor-scoped behavior, not just surface UI.

Inspect methodologyView frozen source (opens in new tab)
Benchmark
harbor-v4
Matched arms
4 workflows
Public tasks
4 scenarios
Methodology locked
July 22, 2026

Benchmark methodology

Harbor v4 compares matched AI coding agent workflows on the same public task corpus. Each arm receives the same total agent-time budget; workflow steps differ by arm design, and the release declares the controls used to keep the comparison interpretable.

Checklist agentshape_build → verify_fix · 4,200 seconds (70 minutes)

Normal coding agent with a generic product checklist before building. No Lamina skills.

shape_build
Agent 2100s · verifier 60s
verify_fix
Agent 2100s · verifier 60s
Direct agentshape_build → verify_fix · 4,200 seconds (70 minutes)

Normal coding agent without Lamina skills. Implements from the founder brief with no plan-first or checklist workflow.

shape_build
Agent 2100s · verifier 60s
verify_fix
Agent 2100s · verifier 60s
Lamina agentlamina_init → lamina_design → implement → lamina_verify → fix · 4,200 seconds (70 minutes)

Full Lamina loop: init, design with persona panel, implement, verify with UI walkthrough, and fix.

lamina_init
Agent 600s · verifier 60s
lamina_design
Agent 1100s · verifier 60s
implement
Agent 800s · verifier 60s
lamina_verify
Agent 1100s · verifier 90s
fix
Agent 600s · verifier 60s
Plan-first agentshape_build → verify_fix · 4,200 seconds (70 minutes)

Normal coding agent using a plan-first workflow. Writes a short plan, then implements. No Lamina skills.

shape_build
Agent 2100s · verifier 60s
verify_fix
Agent 2100s · verifier 60s

Runtime held constant

Harness
claude-code
Model
sonnet
Attempts per arm
2
Budget per arm
4,200 seconds (70 minutes)

Fairness and treatment controls

Behavior-only scoring
Final reward derives from behavior_pass_rate against golden sequences. Agents never receive graded expect substrings.
Lamina treatment validity gates
The lamina arm must invoke required slash commands, persona-panel subagents, and UI walkthrough capture or the trial is marked invalid_treatment.
Matched agent budget
Each arm receives 4,200 seconds (70 minutes) of agent time across its workflow steps.
Structural self-check
Coding steps require a structural self-check that validates action mutability without revealing golden expects.
Thin-slice delivery posture
Products ship as in-memory reducers with HTML UI — no external services — while still modeling authority, lifecycle, and actor-scoped projections.

Public benchmark task corpus

Each scenario begins with a public fuzzy founder brief. Open a task to read its premise, risk class, and commit-pinned brief; grading inputs stay outside the public release artifact.

A tiny household listgreenfield low risk · spark stage

I want a pleasant little list for one person to capture a few things, mark them done, and clear completed items. Keep it simple and friendly. Please shape the product and build the next coherent version.

benchmarks/corpus/control-simple-list/brief.mdRead public brief (opens in new tab)
Shared care coordinationgreenfield high risk · shape stage

I want a small product that helps a family coordinate care for an older relative. People should know what was done and what still needs attention without turning the experience into project-management software. Please shape the product and build the next coherent version.

benchmarks/corpus/pilot-care-circle/brief.mdRead public brief (opens in new tab)
Borrowing things among neighborsgreenfield multi actor · shape stage

I want a friendly way for a small neighborhood group to lend useful things to one another. People should know who has what and whether it is safe to lend again, without making it feel like a logistics system. Please shape the product and build the next coherent version.

benchmarks/corpus/pilot-loan-library/brief.mdRead public brief (opens in new tab)
Lightweight document reviewgreenfield privacy · shape stage

I want a small product where someone can invite a trusted person to review one document and leave useful comments. It should feel safe and focused rather than like giving away access to a whole workspace. Please shape the product and build the next coherent version.

benchmarks/corpus/pilot-review-room/brief.mdRead public brief (opens in new tab)

Benchmark limitations

The running release supports inspecting the methodology, not making a product-performance claim.

  • Running state shows methodology and corpus only. It does not establish a product claim.

  • Scores and aggregate comparisons require a published release artifact approved in a website pull request.

  • Task briefs describe public scenarios; golden behavior sequences stay in the benchmark repository.

Benchmark FAQ

Direct answers about what Harbor v4 measures, what is public, and what remains unavailable before results are published.

What is the Harbor v4 AI coding agent benchmark?
Harbor v4 is Lamina’s product-shaping benchmark for AI coding agents. It evaluates behavior integrity of a thin product shaped from a fuzzy founder brief under matched arms across 4 matched workflows and 4 public tasks, using behavior pass rate as the declared measurement.
Are Harbor v4 benchmark results published?
The methodology is locked and public. Results are not published until a frozen release passes review; no interim scores, trends, or winners appear here. This page publishes the locked methodology and public corpus, but no aggregate rates, per-task outcomes, trends, or winner comparisons until a frozen release is approved through review.
How are the benchmark arms kept comparable?
Every arm receives 2 attempts on the same 4-task corpus and 4,200 seconds (70 minutes) of agent time. The release also declares 5 fairness and treatment controls, with all public methodology links pinned to one source commit.
Where can the benchmark methodology be verified?
The methodology is linked to aryaniyaps/lamina at commit e851d03. The public release records protocol SHA-256 0fa45f7bbfd20cb5f2a1aaf09d2e24d034d757cf6532ba39d69771f75be23412, and the Sources section links each declared corpus, grader, and protocol artifact to that commit.

Benchmark sources and provenance

Every public methodology link resolves to the recorded Git commit. The protocol fingerprint covers the corpus, task briefs, grader, generator, validator, and generated Harbor tasks.

Source repository

aryaniyaps/lamina (opens in new tab)

Source commit

e851d037199b189a9b98b7d807bbcfd427bf0c5c

Protocol SHA-256

0fa45f7bbfd20cb5f2a1aaf09d2e24d034d757cf6532ba39d69771f75be23412

Release generated July 22, 2026 · harbor-v4-2026-07

Public artifacts

Open the exact corpus, grader, and protocol inputs used by this release.

Read benchmark documentation