We simplified our backlog screen, and users would have approved the wrong thing

We redesigned aictrl's Backlog seven times in two days. AI agent testers and a simulated user built on Jev, TypeSafe's System One model, checked each version. No real users took part: novice agent testers stood in for first-time users, and the title says what those users would have done. This is the story of four moments where their evidence overturned what we believed, and of the places where it fell short.

We cut the Backlog landing screen from 37 controls to 16. It looked calmer. Then we ran the next test. Both novice agent testers approved a security review that nobody had asked about. Their task was to start a piece of work, and neither did.

The simplification caused the harm. Measurement caught it in one round. This post covers that round and three more like it, where a test result overturned what we believed about a screen.

Every number about our Backlog comes from our own measurement between 24 and 26 September 2026. Outside studies are cited where they appear, mostly in Part 5. The testers were AI agents and a simulated user, not people. We come back to what that means in the limits section.

The setup: a design loop with evidence at every step

Before this work, Epics, Planner and Chat were three separate places in aictrl. You could not start a workflow from an issue. A run did not record which issue it was for. Our goal was to make the Backlog the control room: the one place where you start agent work and follow it.

Two kinds of testers

AI agent testers drive the page through a Playwright harness. The harness exposes only the accessibility tree, which is what a screen reader gets. Haiku plays the novice, in a text variant and a vision variant. Sonnet is the control. A "hurried" persona answers first-glance questions.

We read success from the page state, never from the tester's own report.

The testers do tasks rather than judge screens. When Baymard Institute tested ChatGPT-4 on UX audits, 19.9% of its suggestions were accurate and 80.1% were false positives (Baymard, 2023). An off-the-shelf agent judging usability did worse than chance (AUC 0.436) until it was trained for the job (Gao et al., 2026). So our agents do the task, and the page state says whether they succeeded.

A task model with one number per variant

We wrote 22 tasks across 4 roles: engineering lead, product owner, reviewer and admin. Each task has a weight (frequency × criticality), a success check and a must-not check. The must-not check is how we see harm, such as an approval nobody asked for.

Each variant gets one score:

We accept a variant only if J improves and no criticality-3 task gets worse. A better average cannot hide a broken critical task.

A simulated user built on Jev

An agent round takes about an hour. We wanted something faster for the first look, so we built a simulated user. Code does the perceiving. A scanner sees only what is above the fold: the top 14 controls by visual weight. A reader sees the whole page.

Jev makes one snap judgement per screen: which control next, am I done, can I answer. We sample many walks per task under four conditions: focused, laptop (1366×768), interrupted and paraphrased. Judgements are cached per screen, so a run over the task model takes minutes. One task costs about 100–200 Jev calls.

Seven versions in two days

Each version answered a finding from the one before. Reviews shaped the early versions. Tests shaped the later ones.

The explorer has four views. Screen marks what changed. For v5, v5.1 and v6, Tester clicks traces where agent testers clicked, and Simulated first click shows where Jev expected a first-time user to click first. Compare puts a version and the one before it under a slider, or, for those three, shows the tested before and after side by side.

Density fell hard at v5. Then it climbed back as each fix added something a tester had needed. That climb is the subject of the limits section.

Four moments where the evidence changed the design

1. We simplified the screen, and novice testers approved the wrong thing

v5 applied four rules: one word per status, one verb, one place per job and one attention list. The attention list was Needs you. It opened on approvals and hid the work that was ready to start. Both Haiku novices approved a security review. Neither started the issue they were asked to start.

v5.1 put ready work back inside Needs you. Buttons now say what they launch. Blocked issues say what they wait on. Approvals ask for confirmation. All 3 novice testers started the right issue.

A cleaner screen is not a safer one. The first thing on screen is what a hurried first-time user would act on.

2. On a laptop, the first screen showed only approvals, and the simulator saw it first

On a 1366×768 laptop, the first screen of v6 showed only approvals. The simulator flagged this before any agent run did.

We tested one change: move Ready to start above approvals. Over 24 walks per arm, success rose from 0.04 to 0.96 and harm fell from 0.67 to 0. The 90% confidence interval of the difference was +0.83 to +1.00.

Then we checked it with hurried agent testers on a laptop viewport. Before the change, 1 of 3 started the right issue. After it, 3 of 3 did. Wrong approvals fell from 2 to 0.

3. A confirm beat removing the button

A tester triaging the backlog approved a gate nobody had asked for. We had two ideas.

• Idea A: remove Approve from lists. The simulator found no clear difference. We did not take it forward.
• Idea B: a one-line confirm when you approve from a list. Harm fell from 0.13 to 0, with a 90% confidence interval of the difference from −0.25 to −0.04. Success did not change. The cost was one extra step.

Agent testers agreed with Idea B. Unasked approvals fell from 1 of 7 to 0 of 5. The one mistaken click was cancelled at the confirm.

The null result for Idea A mattered as much as the win for Idea B. It kept a change with no measured gain out of the design.

4. Placeholder controls fail tasks

v6 was the first version built from the real product CSS and components. We ran the full role model against it. Success was 83%. After seven fixes it was 96%, and J fell from 4.27 to 2.47.

Several failures traced back to a stub or an icon-only control. The table shows six of the fixes and what they did. The seventh lets chat answer “who is working on X”.

Where the simulator is right, and where it isn't

We checked the simulator against 72 agent sessions.

What it reproduced

• v5, task T1. It predicted 0 before the fix and 1.0 after. The agents did the same: novices approved a gate, then 3 of 3 succeeded.
• L8, who is working on what. The scanner scored 0 and the reader scored 1.0. That matches vision testers failing and text testers passing.
• P6, the icon-only status control. It scored 0, matching the vision tester's miss.

What it missed, or flagged wrongly

• P1, making an issue ready through chat. It scored 0 while agents got 3 of 3. The simulator has no multi-turn chat.
• P5, an interpretive progress answer. It scored 0.13 while agents got 2 of 2.
• R3, sending a review back. It scored near 0 because simulated users clicked Approve.
• P2, splitting an issue. It did not reproduce the agents' premature stop.

What had no effect

Distracting, irrelevant text in the prompt changed nothing. The fold is what hides things, not a cut to the top few controls.

What the field has found

Our runs cover one product area over three days. Larger studies point the same way.

Simulated users are too cooperative

In one study, 49.0% of a GPT-4o simulated user's turns were polite, against 15.3% for humans (Zhou et al., 2026). The authors call simulators “excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity.” In another, agent success rates moved by up to 9 percentage points depending on which LLM played the user (Seshadri et al., 2026).

First-click predictions are shaky

Kuric and colleagues compared GPT's predictions with 12 first-click tests and 3,431 real participants (Kuric et al., 2026). The predictions differed significantly in 53% of tasks. Personas and chain-of-thought “fail to create sensible fidelity improvements apart from inflating believability.”

Our Jev simulator makes the same kind of prediction. Against 72 agent sessions it caught the misclick and the fold problems, but missed multi-turn chat and interpretive answers. So we use it only to rank layout bets, then confirm with agent testers and, finally, people.

Agent testers find real problems, but not all of them

In UXCascade, UX professionals working from simulated agents found about as many issues as those working from a report written by people: 2.875 against 2.625 each, not a significant difference (Holter et al., 2026). The human report was stronger on discovery and interaction issues: 5 of 7 against the agents' 2 of 7. WebProber found 29 usability issues across 120 academic personal websites, many missed by traditional tools (Ye et al., 2025). At Userbrain, 5 of 5 synthetic users completed a task that 3 of 5 real users completed (Rössler, 2026). The simulation overestimated success.

Where UX researchers land

Nielsen Norman Group sees synthetic users as preparation for research with people, not a replacement: “A synthetic user can be useful if researchers treat the output as a hypothesis to guide future research.” (Rosala and Moran, 2024). Raluca Budiu found that “the standard deviation in the synthetic data was consistently lower than the standard deviation in the human data.” (Budiu, 2025). Less spread means fewer edge cases. A MeasuringU review of experiments with synthetic users counted 9 encouraging findings and 14 discouraging ones (Lewis and Sauro, 2026). UX researchers rated UXAgent's simulated sessions 3.4 out of 5 for helpfulness and 3 for realism (Lu et al., 2025).

Honest limits, and the detail that came back

Agents are not people

Agent testers are patient. They read text rather than layout. They never see colour. We treat their results as a check on clarity and flow. Real users give the final answer.

Our testers read the accessibility tree, which catches unnamed or mislabelled controls. That is no substitute for testing with people who use screen readers.

The case against synthetic testers

Erika Hall argues that talking to real people is not optional: “It is unethical, indefensible, and also unnecessary, to create a product or service or policy that affects other people, without having conversations with representatives of those populations.” (Erika Hall, LinkedIn)

Vitaly Friedman puts it as a trade: “I would always choose one day with a real customer instead of one hour with 1,000 synthetic users pretending to be humans.” (Friedman, 2025)

We agree on the final answer. Our agents check flows between rounds with people, not instead of them.

Detail crept back

Since v5, controls went from 16 to 28 (+75%) and words from 114 to 214 (+88%). Each addition had a measured reason. The screen is still below v4's 37 controls and 299 words. The open question is whether 214 words is right. The next test is the real page, with people.

Known open issues

• Vision testers still miss content below the fold (L8).
• The mock's chat understands only a narrow set of phrases.

What it costs

A 6-task baseline is 18 tester runs, mostly Haiku. A 20-task round is about 70. The simulator screens a hypothesis in minutes.

What we would do again

1. Write tasks with state checks before you design, not after. A success check and a must-not check per task. Read results from the page, not from the tester.
2. Simplify, then test with a hurried novice on a laptop screen. That is where our cleanest version failed its testers.
3. Use simulators to rank ideas, agents to check flows, and people for the final answer. The studies in Part 5 back this, not just our runs. Each step is slower and closer to the truth than the one before.

What would you test first on your own product?

Sources and method

• aictrl internal measurement, 24–26 Sep 2026 — AI agent tester runs (Playwright harness, accessibility tree only), success and must-not checks read from page state; 22 tasks across 4 roles.
• aictrl internal measurement, 24–26 Sep 2026 — simulated-user walks (scanner and reader perception in code, one Jev judgement per screen), validated against 72 agent sessions.
• aictrl internal measurement, 24–26 Sep 2026 — landing-screen density (visible controls and words in main, 1440×900, default scenario).
• TypeSafe — Jev, TypeSafe's System One model, used for the simulated user's per-screen judgements.
• Baymard Institute, Testing ChatGPT-4 for ‘UX Audits’ Shows an 80% Error Rate, Baymard Institute, 18 Oct 2023.
• Gao, Tong, Vempati, Reinecke, Shapiro, Zhang, Wu, Training Computer Use Agents to Assess the Usability of Graphical User Interfaces, arXiv 2604.26020, Apr 2026.
• Zhou, Sun, Ma, Xie, Liu, Du, Welleck, Yang, Neubig, Wu, Sap, Mind the Sim2Real Gap in User Simulation for Agentic Tasks, arXiv 2603.11245, Mar 2026, revised Jul 2026.
• Seshadri, Cahyawijaya, Odumakinde, Singh, Goldfarb-Tarrant, Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations, arXiv 2601.17087, Jan 2026.
• Eduard Kuric, Peter Demcak, Matus Krajcovic, What Would GPT Click: Practical Effects of Human-AI Behavioral Misalignment and the Cost of Synthetic Participants in User Experience, arXiv 2605.18302, May 2026.
• Holter, Koh, Dogan, Chan, UXCascade: Scalable Usability Testing with Simulated User Agents, arXiv 2601.15777, Jan 2026, revised Aug 2026.
• Ye, Yu, Xu, Peng, Yu, AI Agents for Web Testing: A Case Study in the Wild, arXiv 2509.05197 (introduces WebProber), Sep 2025.
• Stefan Rössler, We Ran an Experiment with Synthetic Users, Userbrain, 28 Apr 2026.
• Maria Rosala and Kate Moran, Synthetic Users: If, When, and How to Use AI-Generated ‘Research’, Nielsen Norman Group, 21 Jun 2024.
• Raluca Budiu, Evaluating AI-Simulated Behavior: Insights from Three Studies on Digital Twins and Synthetic Users, Nielsen Norman Group, 15 Aug 2025.
• Jim Lewis and Jeff Sauro, A Review of Experiments with Synthetic Users, MeasuringU, 14 Apr 2026.
• Lu et al., UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design, CHI EA ’25, arXiv 2502.12561, 2025.
• Erika Hall, Post on synthetic users, LinkedIn.
• Vitaly Friedman, How To Argue Against AI-First Research, Smashing Magazine, 28 Mar 2025.

Related Articles