TeamStation AI / Research / Agentic AI Research / Agentic AI SDLC: test the skills your team actually ships
A coding agent's test score only helps if your team knows what actually ran. Check the installed skill, available tools, permissions and working directory.
A coding agent's test score only helps if your team knows what actually ran.
A coding agent's test score only helps if your team knows what actually ran. The same skill and model can produce different results when the app or tool access changes. For CTOs and CIOs, that's a release risk worth checking before the next rollout. IMO, the useful question is pretty simple: can the agent finish this job in the setup our team really uses?
In an agentic AI SDLC, a skill is part of the software your team depends on. Give it clear inputs, useful tests, an owner and a version you can roll back to. Keep the checks close to the work, bc nobody wants another dashboard that looks busy while the PR is still stuck.
What should an agent skill eval check?
An eval is a repeatable test of the agent's behavior and result. Start with the job you need done, the changes you're allowing and the evidence that would prove success. Write that down before tweaking the prompt again.
OpenAI's January 22 guide, Testing Agent Skills Systematically with Evals, separates outcome, process, style and efficiency checks. It suggests starting with 10 to 20 focused prompts for a skill, including requests that should trigger it and nearby requests that shouldn't. That's a manageable starting point, not proof that every possible failure is covered.
For a repo maintenance skill, I'd include a normal change, an existing dirty working tree, an unavailable dependency and a request outside the skill's job. Make the expected result clear for each. If the request only asks for an explanation, a skill that edits files has missed the point even when the code it writes is fine.
Keep the first test set small enough that an engineer can explain every failure. Add cases when you find a real miss. Don't let a bigger test count hide weak checks.
Why test inside the actual app?
OpenAI and deepsense.ai's September 23 plugin evaluation walkthrough compares direct tool tests, a standalone model loop and a product harness. The harness controls parts of the prompt and which tools the model can reach. The authors show why a result from one setup doesn't automatically carry over to another.
Their example uses a small synthetic dataset. They also point out that the harness test can still miss behavior specific to the target client. So even a stronger test setup has a boundary. It's worth writing that boundary beside the score.
My practical take: run the cheap tool checks during development, then run representative tasks through the app your team will use. Check the installed skill, available tools, permissions and working directory. Save those details with the result. If the agent can't reach a required tool, record a runtime failure so the next engineer knows where to look.
This gives the release owner a useful split. The tool might work when called directly, while the app never exposes it to the agent. Changing the skill, prompt or model won't necessarily fix that connection. Trace the failed step first.
Check the result, then inspect how it got there
Anthropic's January 9 agent evaluation guide distinguishes the agent's recorded interaction from the final state of its environment. It also separates capability tests, which explore harder tasks, from regression tests, which protect behavior that already works.
For coding agents, the guide describes using software tests alongside checks of the recorded run. The checks can answer different questions. Does the change work? Did the agent respect the task? Did it create extra work or skip a necessary step?
Here's the release check I'd use for a small app change: verify the requested behavior, run the relevant existing tests, inspect the diff and open the affected screen. Give each check a clear result. A screenshot can show a layout problem, while a unit test can catch a logic bug. Pick the evidence that matches the claim.
Keep the same version of the release checks while comparing a repair with the earlier result. Add new cases as you find failures, then give that test set a new version. When a requirement really changes, record who changed it and why, then keep the old result. Quietly moving the pass line makes the next comparison useless.
Also record how many attempts the job needed. If a task succeeds only after repeated retries, that matters when you're planning cost and delivery time. Keep the failed attempts in the record, bc the team needs the whole picture.
Who owns the skill after the demo?
I'd give one engineer ownership of each skill's instructions, examples and supported use cases. An evaluation engineer or QA lead can own the test cases and expected results. Platform engineering owns the runtime and tool connections. The release owner decides whether the checked change is ready for its intended users.
These are jobs to cover. One person may cover several, and the titles can stay as they are. What matters is that a failed test has somewhere specific to go. Otherwise, ppl end up passing the same screenshot around and calling it a model problem.
Context matters here too. If the skill keeps missing product rules or pulling stale repo notes, TeamStation's context engineer role page describes work around that gap. The agentic engineering concepts article gives teams shared language for the broader workflow. Use the terms to make ownership clearer, then tie them to a real task.
For distributed engineering teams, I'd put the skill version and failing case in the handoff. The next engineer should be able to reproduce the issue without sitting through the original demo. That's a concrete piece of software quality work you can ask for when discussing TeamStation's engineering services.
A heads-up for teams using OpenAI Evals
There is a current deadline to check. As verified on October 14, OpenAI's deprecation notice says existing Evals become read-only on October 31, with the Evals dashboard and API scheduled to shut down on November 30, 2026.
OpenAI's Promptfoo migration guide describes manually recreating prompts, providers, cases and grading rules. It warns that recreated graders may produce different scores. Give that migration an owner and compare results before relying on the new setup for release decisions.
What can a CTO ask for this week?
Pick one skill your engineers already use. Ask for the exact version, its intended job, a small set of meaningful tests and a run through the actual app. Then review one failure with the engineer who owns the relevant step.
You don't need a huge new program to start. You need a result you can inspect, a failure the team can reproduce and a clear owner for the repair. Get that working for one skill, then expand with evidence.
