TeamStation AI / Research / Evaluation Research / Constraint Shift Test for Engineering Judgment
Use a constraint shift test to see whether an engineer understands the live system or is repeating a polished pattern before production.
A practical Cognitive Fidelity method for checking whether an engineer can update a system model when one production constraint changes.
Production errors often start as map errors because the engineer has one picture of the system in their head while the live system carries a different set of dependencies, limits, failure paths, and business rules, so the code can appear clean while those two maps quietly move apart.
That is the operating problem behind Cognitive Fidelity. In plain English, cognitive fidelity is the match between an engineer's mental model and the part of the real system they are trying to change.
We need a way to see that match before the work reaches production, and one useful method is the constraint shift test.
Short answer
Give the engineer a realistic system problem. Let them explain the model. Then change one important constraint and ask what changes next, what stays stable, and why.
Make the database read only, cut the latency budget, remove an integration, move a data boundary, or double one failure cost, then ask the engineer to update the model without losing the original business goal.
If the explanation updates in the right places, the engineer is showing evidence that they understand the mechanism. If the answer stays polished but no longer fits the new state, the first response may have been pattern recall dressed up as judgment.
A magic personality score would overstate what one exercise can prove; the constraint shift is a controlled engineering work sample that creates an evidence object a CTO, CIO, or technical evaluator can inspect.
The science is about the map
Let the engineer's current mental model be M and the relevant system state be S.
The goal is not to prove that every detail in M equals every detail in S because no human model can hold the whole production estate; the useful question is narrower: does M preserve the relationships that matter for the decision in front of the team?
Now change the system by one bounded constraint, which we can call delta S. A strong engineering model should produce a related update, delta M. The engineer should be able to trace where the change travels and where it stops.
That is the core test:
Base evidenceM ≈ S
Constraint response\Delta S \Rightarrow \Delta M
The symbols are simple on purpose because we are not publishing a clinical formula or pretending one prompt can measure a human being; we are checking whether the causal map moves when the real conditions move.
Software already uses the same scientific habit: change one variable, hold the rest steady, observe the response, preserve the evidence, and do not confuse a confident story with a tested model.
Why the first answer is weak evidence
A normal technical interview rewards the first answer too much because the candidate has seen common architecture questions, AI can generate fluent explanations, and interview prep can teach a clean script, yet none of those conditions proves the person can update the system model when the assumptions break.
The first answer still matters, but it is only a baseline. The second answer, after the constraint changes, tells us more about the structure underneath it.
Here is a simple analogy: a person can memorize a route across town without understanding the city, but closing one bridge exposes the difference fast because the person with a real map can reroute while the person with a memorized sequence keeps driving toward the closed bridge, and engineering judgment works the same way.
How to run the test
The evaluator needs a small, role-specific problem that stays close to the work the engineer would actually own instead of turning the exercise into trivia.
Run the protocol in six parts:
- Define the base system. Give the engineer enough context to reason, including users, data, dependencies, limits, and the outcome that matters.
- Capture the first model. Ask for the architecture, key assumptions, likely failure paths, and the decision they would make.
- Shift one constraint. Change one condition only. Keep the shift clear enough that the evaluator can trace its blast radius.
- Ask for the causal update. What changes now? What remains true? Which dependency moved? Which risk became more important?
- Ask for evidence. What would the engineer inspect, measure, log, or test before acting?
- Preserve the result. Keep the diagram, assumptions, predicted failure modes, confidence level, and revised decision as one evaluation packet.
The point is not to trap the person; the point is to see how they repair the model when the evidence changes.
Four useful constraint shifts
Different roles need different tests, but the following four shifts work across many engineering systems and expose different parts of the causal map.
1. The database becomes read only
Start with a service that writes user state to a primary database. Then make the database read only for thirty minutes.
A strong answer should move beyond "show an error" and inspect write paths, queues, retries, idempotency, user feedback, recovery order, and the risk of a retry storm after the database returns.
The key evidence is not the number of tools they name. It is whether they can trace state through the system.
2. The latency budget is cut
Start with an API that has a 500 millisecond response target. Cut the target to 150 milliseconds.
The engineer should not promise a faster server and call it done; they should question network hops, query shape, serialization, cache behavior, synchronous dependencies, tail latency, and what the user actually needs in the first response.
The latency shift exposes whether the person sees a system or just a component.
3. A third-party integration disappears
Start with a workflow that depends on an external provider. Remove that provider with no replacement date.
Watch for boundary awareness by asking whether the engineer can separate core business logic from the external contract, define a degraded mode, protect data integrity, and show where human approval is still required.
AI engineering has the same dependency problem because models, vector stores, observability services, and agent tools can all fail; a real production design needs an answer for what happens when one of them goes dark.
4. A data boundary moves
Start with one allowed data path. Then add a residency, privacy, or customer-isolation rule that blocks part of that path.
The answer should update identity, storage, logging, access, model context, support workflow, and audit evidence where relevant while naming what remains unknown, because fake certainty is a risk signal.
What the evaluator should watch
At TeamStation, the evidence connects to the wider Axiom Cortex engineer vetting system. The constraint shift does not replace the full evaluation. It gives us a controlled way to observe parts of engineering judgment.
Useful signals include:
- Architectural instinct. Does the engineer see boundaries, dependencies, blast radius, and future pain?
- Problem-solving agility. Can they update the plan when the condition changes without losing the goal?
- Learning orientation. Do they use new evidence to repair the model, or defend the first answer?
- Collaborative mindset. Can they explain the updated model so another engineer can act on it?
- Uncertainty calibration. Can they separate what they know, what they infer, and what they still need to inspect?
These are evidence lanes, not protected-trait judgments. The test is about the work, the system, and the reasoning shown in the work sample.
Where AI changes the interview
AI makes syntax cheap. A candidate can arrive with a clean answer, polished language, and a familiar architecture diagram. That is why the Turing Trap matters.
The answer can sound senior without carrying a senior model underneath it, so constraint shifts make the conversation harder to fake by forcing the person to maintain a causal map across a new condition. AI can still help when AI use belongs in the role, but the evaluator should inspect how the person checks the output, corrects weak assumptions, and owns the final decision.
The goal is not human versus AI. The goal is accountable engineering judgment inside an AI-assisted loop.
Why it matters for distributed LATAM teams
Distributed work gives a weak assumption more places to travel before another person has enough context to challenge it.
One engineer writes the ticket, another interprets it later, and a third reviews the change across a time-zone handoff, so the bad map may already be code, review debt, or an incident by the time it becomes visible.
That is why we treat LATAM and nearshore delivery as the application layer, not the science itself. The same cognitive fidelity test applies anywhere. In a distributed team, the evidence packet matters more because the reasoning has to survive async handoffs.
The engineer should leave a model another person can inspect, the system should preserve the assumptions, and engineering telemetry should show what happened after the decision entered the delivery loop.
A work sample connects to a governed Distributed Engineering Operating System when evaluation creates the first evidence and delivery telemetry tests whether that evidence holds after launch.
What the test cannot prove
One simulation cannot prove production performance because the scenario can be too narrow, the prompt can favor one background, the evaluator can overvalue communication style, a candidate can get lucky, and a strong engineer can misunderstand weak context.
So the test needs boundaries:
- Use more than one role-specific scenario.
- Give every candidate equivalent decision context.
- Score the causal update, not accent, slang, charisma, or presentation polish.
- Combine the result with code, architecture, collaboration, and delivery evidence.
- Keep a human accountable for the decision.
- Validate the pre-hire signal against post-launch telemetry instead of declaring victory at the interview.
The Mathematical Validation doctrine exists because form noise should not hide technical content, and technical fluency should not be allowed to fake real structure.
The operating decision
For a CTO, the constraint shift test answers a practical question: can the engineer keep the system model intact when production changes the deal?
For a CIO, it answers another one: can the reasoning leave enough evidence for governance, audit, security, and accountable approval?
For TeamStation, it is one part of a larger chain. Define the work. Map the required capability. Evaluate the engineer against the real loop. Launch inside governed controls. Watch the telemetry. Repair the model when evidence changes.
No single score owns that decision because the value comes from the full chain of evidence.
That is the lesson behind Cognitive Fidelity. Do not only ask whether the first answer sounds right. Change one constraint and see whether the map still works.
Related TeamStation research
