A correct diagnosis still needs a separate permission check
Radar’s Kubernetes study is useful evidence about diagnosis. Its single-trial design and read-only setup also show why a good answer cannot authorize a production repair.
Radar’s Kubernetes study is useful evidence about diagnosis. Its single-trial design and read-only setup also show why a good answer cannot authorize a production repair.
Ready is a limited observation
A Kubernetes pod can report Ready while an application remains wrong. Radar’s published AI SRE study separates 14 such faults from obvious crashes. That distinction interests me more than a single headline score: the tool has to connect a symptom to configuration evidence instead of stopping at a green status. This article is a reading of the vendor’s study, not a test on a production cluster.
What the study actually measured
The September 20 study, updated October 1, reports 44 correct diagnoses out of 50 for Claude Code with Radar MCP 1.13.1, compared with 41 for kubectl-ai. Each fault had one counted trial. The publisher explicitly says small accuracy gaps can move between runs. Five products used Claude Sonnet 5 on Bedrock; AWS DevOps Agent used an undisclosed managed model. These are product configurations, not a controlled comparison of model intelligence.
The awkward conditions belong beside the scores
Faults were injected minutes after deployment, which the authors say favors recent-change correlation. Two products also saw extra chaos-engineering namespaces. Radar reports an average 79 seconds and$0.27 per fault for its tested configuration; the study uses different billing units across products and notes caching differences. I would not apply those prices or timings to another cluster. The 14 Ready-but-broken cases were grouped after the runs, another reason to inspect individual faults before generalizing.
My proposed investigation contract
For an authorized staging trial, I would record the workload, symptom window, permitted reads and forbidden changes before connecting an assistant. The evidence record should include resource identity, observation time, relevant configuration and a falsifiable explanation. A diagnosis that names a deployment but cannot identify the changed field is a lead for further work. It should remain possible to say that the available evidence is insufficient.
Keep repair authority outside the evidence record
The benchmark was read-only for everyone. It does not demonstrate safe autonomous remediation, a rollback strategy or permission to modify a cluster. My proposed review record below separates the diagnostic claim from a human-approved change. It is explanatory JSON, not a tested access-control implementation. An executor must enforce its own scope and refuse missing authorization.
{
"scope": "authorized staging investigation",
"evidence": {
"resource": "example deployment",
"observed_at": "record actual timestamp",
"changed_field": "record exact field or unknown"
},
"diagnosis": "hypothesis requiring verification",
"repair": {
"authorized": false,
"proposed_diff": null,
"rollback": null
}
}A trial that can disprove the benefit
I would repeat a small fixed set of staging faults, including a case where the most recent change is innocent. Hold account permissions and model settings constant; record wrong answers and refusals as well as successful diagnoses. Measure time to a verified explanation separately from time to a restored service. If useful context merely makes a confident wrong story faster, the trial should expose that. I have not run this trial or independently reproduced the study.