For example, a candidate’s Python worker sample demonstrates causal debugging of duplicate processing, but they are applying for a Java service role. If that role requires independent transaction work immediately, the sample leaves a hiring-critical question unanswered.
The choice is whether to test that requirement before hiring or support the candidate in learning it. An unfamiliar stack is neither proof of weak engineering nor a guarantee of easy adaptability.
Missing evidence is a question to resolve, not a score of zero.
Define what must be true at entry
Before reviewing the candidate’s preferred technologies, write down the capabilities the job requires. Separate outcomes from tools, while preserving tool-specific requirements that genuinely matter.
“Knows Java” is too broad to assess consistently. “Can identify a transaction boundary that permits a partial write” gives the reviewer something observable to check.
Classify each requirement:
- Required at entry: The candidate must demonstrate it before a hiring decision because the initial work depends on it.
- Learnable with available support: The team can provide a defined learning period, review, and access to someone who knows the system.
- Incidental: Familiarity would be convenient but is not necessary for the work.
Do this before comparing candidates. Otherwise, a familiar framework can become an unstated advantage, while an unfamiliar one attracts extra scrutiny.
Be concrete about support. Immediate, independent incident response requires different preparation from work with paired onboarding. General engineering experience does not erase that difference.
Use a two-column evidence map
The evidence map separates what the work supports from what the role still requires. Record observations, not assumptions, and leave a cell blank when there is no evidence or no relevant unresolved question.
For the Python-to-Java case:
| Role requirement and demonstrated transferable evidence | Unresolved stack-specific question |
|---|---|
| Diagnose failures: The candidate connects duplicate processing to a retry path, shows relevant logs, and explains how they ruled out another cause. | Can they trace retries and exception handling in the target service’s execution model? |
| Protect data integrity: Their design identifies the invariant that each operation must have one durable effect and explains the limits of their safeguard. | Do they understand the target application’s transaction boundaries and concurrency behavior? |
| Test risky behavior: A regression test reproduces the original failure and distinguishes the fix from the previous implementation. | |
| Deliver changes safely: Their work includes a rollout plan, a rollback condition, and a check for the intended outcome. | Can they interpret the target system’s deployment and health signals? |
The testing row stays blank unless fluency with the target project’s test tools is explicitly required at entry. If it is, ask whether the candidate can choose an appropriate test boundary using those tools.
Evidence travels only as far as the claim it supports.
A strong test in one ecosystem supports test-design judgment. Fluency with another ecosystem’s test runner, dependency injection, or integration setup needs separate evidence when the role requires it. Likewise, a described fix without an explanation of its mechanism leaves the causal reasoning unresolved.
Check explanations and individual contribution
Use consistent anchors across candidates, without averaging unrelated capabilities into one score:
- Failure diagnosis: Distinguish naming a symptom or fix from explaining the causal path, comparing plausible causes, and identifying disconfirming evidence.
- Testing assumptions: Distinguish showing passing tests from connecting a test to a risk or invariant, explaining its limits, and choosing another check.
- Design constraints: Distinguish naming technologies from explaining constraints, rejected options, and when a decision should change.
- Delivery judgment: Distinguish describing implementation from explaining rollout checks, recovery options, failure signals, and consequences for affected users.
Check individual contribution rather than inferring it from a polished repository. Request a walkthrough of one decision and supporting artifacts: a sanitized diff, test, design note, or reconstructed example. Do not ask candidates to disclose proprietary code or customer data.
For a senior role, extend the walkthrough beyond the function under review. Ask what the candidate decided, what others decided, and which outcomes they could observe, especially for system-wide work such as a migration.
Choose a follow-up that changes the decision
I would not add a broad coding exercise simply because the sample uses another language. Choose the highest-impact unresolved entry requirement and ask:
Would a satisfactory answer to this question change the hiring decision?
If not, the task is unlikely to justify the candidate’s time.
For the Java role, specify whether the requirement is reasoning about transactions or verifying actual transaction behavior. That determines the fixture.
A small Java project with a mocked database interface can test reasoning about a supplied model, but not real commit, rollback, or concurrency semantics. If those semantics are required at entry, use a representative database-backed fixture and specify the behavior to assess, such as rollback after a partial write or duplicate effects under concurrent retries.
Build the bounded follow-up around that choice:
- Supply the relevant project structure and transaction documentation.
- Show a retry path that can produce an unintended duplicate effect.
- Ask the candidate to trace the behavior, propose a change, and describe a regression test.
- Set a clear time box, such as 25 minutes, and state whether working code is required.
Write the criteria first: recognizing the failure path, protecting the invariant, choosing a meaningful test, and identifying remaining uncertainty. Allow documentation if that reflects the job; assess unaided recall only when it is an explicit requirement.
A short exercise does not establish sustained production ownership. Assess that through the work sample and discussion of broader responsibility.
Keep strengths and entry requirements separate
Strong debugging evidence should not cancel an unresolved requirement to make safe changes independently on day one.
For the reasoning-focused follow-up, the decision note could read:
The candidate demonstrated causal debugging and risk-based testing in their previous stack. The follow-up established reasoning about the transaction issue in the supplied model. Framework configuration remains unverified, but the role includes paired onboarding for that work.
If a required capability remains unsupported, record that result without turning it into a judgment about the candidate’s entire engineering ability. If the gap is learnable, name the support needed rather than assuming adaptability will cover it.
Spend five minutes choosing the next check
Take one open role and one work sample. Map the three requirements most consequential to the hiring decision, then select the unresolved entry requirement whose answer would matter most. This is a review-planning exercise, not validation of the candidate’s claims.
Finish with two written items: one unresolved entry requirement and one bounded follow-up that could resolve it.