Imagine a candidate asking to replace a 30-minute video call about a database migration with a written migration plan. If the assessment measures migration reasoning, either format can show sequencing, failure handling, and recovery planning. If it measures live incident coordination, writing alone leaves part of the capability untested.
The hiring decision turns on which evidence the format preserves, not whether it resembles the original interview. Keep the capability and scoring criteria fixed while changing conditions that are not essential to the job.
Separate job requirements from assessment habits
A countdown, camera requirement, unfamiliar editor, or uninterrupted presentation might introduce a demand beyond the engineering capability being assessed. Before requiring one, identify what evidence it adds.
For every constraint, complete this sentence:
This condition is necessary because the role requires ___, and without it we cannot observe ___.
“We have always used a live interview” does not complete that sentence. Neither does “It is easier to schedule.” Those are operational preferences, not evidence that a condition is essential.
Some constraints do have a defensible purpose:
- Writing: A role that requires usable design documents needs evidence of written communication. A spoken explanation can supplement that evidence but cannot fully replace it.
- Real-time interaction: A role involving incident coordination can justify observing how a candidate exchanges information as a situation changes.
- Time-sensitive decisions: A bounded exercise can assess prioritization under a deadline when that is an actual job requirement. The deadline still needs a rationale beyond making the assessment harder.
Preserve the narrow requirement, not every detail of the original interview. Observing real-time coordination, for example, does not automatically require video or typing while speaking.
Record capabilities and acceptable formats before scoring
Build a capability-to-format table before reviewing candidates. Each row represents a capability to score separately, not an interchangeable way to assess every capability. Use only the rows relevant to the role, and adapt the alternatives below.
For each selected row, record why the role needs the capability, justify any essential constraint, and specify accepted formats and required evidence. Define both the “meets expectations” anchor and what would fall short. The anchor stays unchanged when the format changes.
| Capability being assessed | Is the original constraint essential? | Potential evidence and format alternatives | “Meets expectations” anchor |
|---|---|---|---|
| Reasoning about a migration | A live spoken explanation is unnecessary unless interaction itself is being assessed. | Written plan, annotated diagram with explanatory text, or recorded or live explanation covering the same prompt. | Identifies relevant dependencies, proposes a coherent sequence, and explains how to detect failure and recover safely. |
| Debugging from incomplete evidence | A timer is unnecessary when the target is diagnostic reasoning rather than response speed. | Asynchronous analysis of a fixed log bundle, or a scheduled session with equivalent evidence and prompts. | Separates observations from hypotheses, chooses a discriminating next check, and revises the diagnosis when evidence conflicts. |
| Producing a usable design document | Written evidence is essential if writing is the capability. A particular editor or live drafting session usually is not. | Accessible document format, candidate-selected editor, and an agreed submission window. | Makes the proposal, constraints, alternatives, and unresolved decisions understandable without additional oral explanation. |
| Coordinating during a changing incident | Real-time exchange is essential if coordination is the target. Video need not be. | A scheduled voice or text simulation using a fictional service, with equivalent events and response opportunities. | Communicates current impact, assigns or requests clear actions, updates others when evidence changes, and confirms shared understanding. |
Before using an alternative with candidates, run a small calibration trial. Have reviewers independently score sample responses in each format, compare scores and cited evidence, and resolve disagreements about the anchors. Revise unclear prompts or anchors. These alternatives are starting points: reviewer agreement checks scoring consistency but does not by itself establish equal difficulty or comparable assessment validity across formats. If the team cannot explain what must remain constant, it is not ready to score consistently.
Apply the same rubric without rewarding the medium
For example, consider a task to add a required field to a service while older application instances remain active. Candidates may submit a written plan or give a recorded explanation. Both receive the same system description and assumptions, including that older instances tolerate an added optional field.
Migration reasoning is the capability; the rubric scores three dimensions of it separately, each from 0 to 2:
| Dimension | 0: Missing or unsafe | 1: Partial evidence | 2: Meets the standard |
|---|---|---|---|
| Compatibility | Ignores older application instances. | Notices compatibility but leaves the deployment sequence unresolved. | Explains a sequence that keeps the stated old and new versions compatible. |
| Verification | Provides no meaningful validation. | Suggests checks without explaining what their results establish. | Specifies checks tied to the migration’s main failure risks and explains how results affect the next step. |
| Recovery | Provides no response to failure. | Names rollback without addressing changed data or intermediate states. | Explains a viable recovery or roll-forward path and identifies its limits. |
Compare two excerpts using only the compatibility dimension.
Confident recorded explanation, transcribed:
The compatibility risk is straightforward. Add the field as optional, deploy the new version so it writes the field and tolerates missing values on reads, then backfill. Once no missing values remain, enforce the required-field constraint.
Score: 1 for compatibility. The response recognizes compatibility and protects new readers, but never establishes that old writers have been retired before enforcement. A successful backfill check does not prevent an active old writer from subsequently attempting a write without the field. Confident delivery cannot supply the missing deployment step.
Plain written plan:
Add field as optional. New instances populate it and tolerate missing values on reads. Retire all old writers. Backfill missing values. Verify none remain. Then enforce the required-field constraint.
Score: 2 for compatibility. The plan preserves old writers while they remain active and delays enforcement until they are gone and the backfill is verified. Its terse presentation does not reduce that evidence. Neither excerpt alone establishes the full verification or recovery score.
A polished presentation does not compensate for an unsafe migration sequence.
Do not quietly add “executive presence,” presentation design, or typing speed to this rubric. If one represents a genuine role requirement, define and assess the relevant capability separately.
Also decide how clarification works. An interviewer who supplies hints changes the evidence available to that candidate. Define permitted prompts, record substantive assistance, and provide an equivalent clarification route for asynchronous submissions.
Keep asynchronous work bounded
Removing a timer can replace a short exercise with an open-ended production assignment. Extra polish can then obscure whether the required reasoning is present.
For asynchronous work, specify:
- Scope: Required decisions or artifacts, plus explicit non-goals.
- Effort: Expected working time and a concrete output boundary.
- Resources: Allowed documentation, external tools, or assistance, and how their use should be disclosed.
- Clarification: Where candidates ask questions and how material answers are shared.
- Completion: A submission window and a statement that extra features or visual polish earn no additional credit unless assessed.
For example, request no more than two pages covering the migration sequence, verification plan, and recovery approach, returned within three days, with an expected effort budget of 30 minutes. Require no working deployment, and award no extra credit for features or visual polish. Treat 30 minutes as a workload estimate, not a measured cutoff: do not collect actual working time or penalize candidates who report exceeding it. Reviewers should score the requested evidence within the output boundary, not infer speed from submission timing. If candidates report needing longer, review the exercise’s scope and estimate.
Use fictional systems and synthetic data; no live credentials or customer data are needed to demonstrate this reasoning.
A submission deadline and a test of working speed are different requirements.
Offer a process that does not depend on personal disclosure
Tell candidates what the exercise measures and offer a clear contact for format or access requests. Ask what adjustment would let them demonstrate the capability, rather than requesting a diagnosis or medical history as part of the assessment conversation.
Keep adjustment discussions separate from scoring notes. Reviewers need the agreed conditions and permitted assistance, not private explanations for the request. Do not score the request itself as evidence about motivation, reliability, or communication skill.
This is assessment-design guidance, not legal advice. Formal accommodation obligations and documentation processes require appropriate local advice.
Try a five-minute constraint review
Take one existing work sample and underline every requirement about time, medium, tools, or interaction. Choose one and fill in a table row: the job capability, whether the constraint is essential, an alternative format, and the scoring anchor that stays unchanged.
If you cannot identify the capability, remove or justify the constraint before using it again. If the alternative omits essential evidence, add a targeted component rather than rejecting the entire adjustment.
This review can expose irrelevant restrictions. It is a design check, subject to the calibration and validity limits above, not proof that two formats are comparable.