Independent agent evaluation
Measure whether a fresh coding agent can build with the toolkit from its public documentation, without relying on a previous conversation. This is a repeatable evaluation protocol, not a certification or permission to create agents, publish work or change existing applications.
01Isolate the run
Obtain authorization for independent runs. Give each agent one brief below, access to the checked-out toolkit, the canonical reference index and the boilerplate setup. Do not supply undocumented API hints or another agent's implementation. Use a fresh conversation for each brief and a temporary prototype directory; keep production apps, shared toolkit sources and the lockfile read-only during generation. Avoid package installation, network services and credentials.
Record the toolkit version, source revision and uncommitted changes, the agent's reported model/settings when available, start and finish times, original brief and allowed sources. Do not claim identical starting conditions if runs use different source snapshots. Let the agent report uncertainty instead of quietly providing implementation hints.
02Briefs
Editorial
Build a one-page exhibition announcement for an English-speaking audience, with a title, date and location, introductory text, three reading sections, a quotation with attribution and a contact link. Include one deliberately long title and a long source URL. Use a coherent heading hierarchy, shared page frame, readable text and metadata. The page must work on desktop and around 390px in both themes, support the skip link and preserve the theme after reload. Use fictional text and identify it as prototype content; do not invent artwork provenance or remote images.
Collection
Build a collection browser with six fictional entries, search and a native category selector. Each entry has a title, year, category, short description and a destination. Provide a filtered result count, no-results recovery and a simulated refresh that can succeed, fail, be retried or be cancelled. Preserve filters and useful content through transitions. Include long titles and verify an embedded narrow container as well as a narrow viewport. Keep all data local and identify simulated requests. Do not add settings that an audience cannot use meaningfully.
Conversation
Build a local conversation prototype with an empty state, suggested questions, user and assistant messages, a composer and a simulated delayed response. Support sending, pending, stop, a recoverable error and retry without duplicating the user message or losing the draft. Translate visible and accessible control labels together into French. Keep older messages readable and avoid moving focus on each generated fragment. The prototype must use no network request, model credentials or persistence service and must identify its simulated responses.
03Evaluate the first submission
Preserve the first submission before correcting it. Record initial failures separately from failures remaining after revision; a successful final build does not mean the first attempt was autonomous. Do not silently revise the brief to match the implementation.
| Measure | Record |
|---|---|
| Invented APIs | Unknown imports, props, methods, classes or tokens; list exact names and occurrences. |
| Unnecessary duplication | Shared components or visual values recreated locally without a concrete constraint. |
| Missing behavior | Applicable states, transitions, recovery paths, semantic landmarks or translated labels omitted. |
| Verification | Source diagnostics, build result, browser scenarios, console errors and screenshots actually reviewed. |
| Human intervention | Number of prompts and manual edits; distinguish brief clarification, missing documentation and implementation mistakes. |
| Completion | Time, remaining issues and whether the brief is satisfied without undisclosed scaffolding. |
Use the state contract, exact component references and accessibility checklist as review criteria. Test desktop and narrow widths in both themes, keyboard behavior, long content, overflow, reduced motion and reload persistence. A type check does not validate every JavaScript spread, CSS string or accessible interaction. A simulated response is not backend integration evidence.
04Review and improve
A satisfactory submission uses only documented public APIs, avoids unexplained shared-style overrides, satisfies applicable states and passes its source, build and rendered checks. Record manual screen-reader or browser-zoom review separately as performed, failed or not performed; an agent cannot infer a pass from ARIA markup.
Classify each failure before changing the toolkit: unclear brief, hard-to-find guidance, ambiguous reference, missing reusable behavior, generation mistake or verification limitation. Change canonical documentation or a public API only when evidence justifies it, following the maintenance process. Repeat a brief in another fresh run after a documentation change to check that the improvement actually helps.
Keep run evidence with the evaluation work, not in this reusable guide. Report the date, toolkit version, first-pass findings, corrections and remaining limitations. A protocol being present is not evidence that independent runs have been performed.