Resolve hiring disagreements before averaging scores

For example, two reviewers give the same engineering work sample reliability scores of 2 and 4 on a four-point rubric. Averaging them to 3 can conceal three different kinds of disagreement: a missed safeguard, missing evidence, or different expectations of the role.

Each needs a different remedy. Before combining scores, identify the disagreement, then choose a correction, a bounded follow-up, or an explicit decision about the hiring bar.

Averaging scores preserves the arithmetic, not the reasoning.

Start with independent judgments, not a group score

Before the debrief, ask each reviewer to record:

• The criterion and role-level expectation being assessed.
• An initial rating, including “insufficient evidence” where appropriate.
• A specific artifact reference, such as a file, test, design section, or interview answer.
• An explanation connecting that evidence to the rating.
• Any uncertainty that could change the decision.

Keep these judgments separate until everyone has submitted them. This preserves what each reviewer concluded before hearing a more senior colleague’s interpretation.

For the reliability assessment, a useful explanation would be:

The duplicate-delivery test reuses the same job identifier and verifies one recorded result. That supports the stated requirement to handle repeated delivery, but it does not show what happens when the external operation succeeds and its response is lost.

“Strong engineer” and “not senior enough” are conclusions without a traceable basis. Neither gives another reviewer much to inspect.

Treat “insufficient evidence” separately from a low rating. A missing demonstration and a demonstrated weakness call for different next steps.

Use a disagreement worksheet

Use this remedy table for each disputed criterion, rather than debating the candidate as a whole. It keeps a disagreement about failure handling from becoming a vague argument about overall quality.

Start with observations, then examine evidence sufficiency and the hiring standard. These categories can overlap; they are prompts, not a rigid sequence. If clarifying the role standard changes which evidence matters, return to the artifact.

Keep a compact record alongside the table:

Criterion and expected role level:
A / B: initial rating, artifact reference, reasoning
Disagreement type; smallest resolution question; next action:
Decision: final rationale, remaining uncertainty, owner, dissent

Work through the split before revising either score

Continue with the background-job worker behind those reliability scores. It uses a mock external service. The role includes owning a service’s reliability, but the rubric says little about broader organizational responsibility.

One reviewer says the tests cover duplicate delivery. Another says every test uses a different job identifier. Inspect the referenced test together. If it reuses an identifier and asserts a single recorded result, the submission itself resolves that factual dispute. Record the corrected observation against the opening 2 and 4, preserving both initial ratings for later reconsideration.

Both reviewers then agree that the sample does not demonstrate a lost-response case: the external operation succeeds, but the worker receives no confirmation. That gap matters to the stated service-reliability criterion. Request one targeted clarification:

Using the existing mock service, walk through what happens if the operation succeeds but its response is lost. Point to the relevant code and explain what additional evidence you would need to establish safe retry behavior.

This assesses reasoning about the failure mode, not prior recognition or demonstrated implementation correctness. Use a consistent time allowance and assessment rule for equivalent gaps. Keep the task to this explanation, with no new implementation or repeated follow-ups seeking a preferred answer. In the worksheet supporting the opening scores, record the bounded clarification request as the next action, leaving the ratings unresolved.

A separate disagreement concerns scope. One reviewer expects a cross-team incident process and a migration plan covering other services; the other considers those outside the role. The role owner should determine whether cross-team coordination was an established expectation and where it belongs in the rubric, rather than asking the candidate to settle an internal role-definition dispute.

If the expectation was never defined or communicated through the assessment, do not quietly add it as a candidate-specific hurdle. Clarify the assessment for future use and document how the current decision handles the omission. Against the same 2 and 4, record the disputed expectation and send it to the role owner before reconsidering the scores.

Make the decision without requiring identical ratings

After resolving what can be resolved, ask each reviewer to update their judgment independently. A changed rating should identify the evidence or clarified standard that changed it.

The decision owner can then record:

• Which role requirements the evidence supports.
• Which essential requirements remain unsupported.
• Whether the remaining uncertainty is acceptable for this role.
• Why any unresolved dissent did or did not change the decision.

I would use scores to summarize judgments only after this discussion, not to settle it. A numerical average cannot tell you whether two reviewers assessed the same requirement.

Try the worksheet on one disputed criterion

Spend five minutes with a previous work-sample disagreement. Using an authorized, redacted record, identify the criterion, cite the evidence, and classify the disagreement before discussing the candidate’s overall suitability.

If the original independent judgments are missing, mark them as unavailable. Record reviewers’ current interpretations separately rather than reconstructing their initial ratings or reasoning.

The immediate result should be one next action, not necessarily a hiring decision. This exercise can expose an unclear standard; it cannot repair one without an accountable role owner.