The fix is not another larger dashboard. It is a cross-layer evidence record that joins the model’s action, the harness’s result and the system’s observed state. Each automated change should carry the original target, the proposed target, the evidence used to make the substitution, the confidence score, the resulting assertion and an explicit human-review status.
For an evaluation or procurement conversation, I would ask five questions. First, can the tool distinguish a successful repair from a successful execution against the wrong target? Second, does it measure false-heals on adversarially perturbed locators rather than only measuring whether a test reruns? Third, can an engineer reproduce the decision from an audit record? Fourth, does the tool abstain when evidence is weak, or does it optimize for a green build? Fifth, can its events be correlated with the application’s runtime traces and the model’s tool calls?
I would also require a staged operating mode. Autonomous repair can propose a change, but high-impact changes should enter assisted triage until the organization has evidence that the repair preserves meaning. This is similar in spirit to chaos engineering’s emphasis on disciplined, observable experiments: the system should reveal its failure modes under controlled conditions before it is trusted in an uncontrolled one.
