For example, one candidate reports an 80% latency reduction from an uncontrolled before-and-after comparison; another reports 15% under matched test conditions. The smaller result deserves closer attention because the candidate documents the comparison, not simply because the conditions match.
| Candidate A | Candidate B | |
|---|---|---|
| Headline claim | “Reduced latency by 80%.” | “Reduced p95 latency by 15% for the checkout endpoint.” |
| Baseline and result | 1,000 ms to 200 ms | Median run-level p95: 400 ms to 340 ms |
| Measurement | One run before and one run after | Four paired comparisons, eight runs total; each configuration ran for 30 minutes per pair, alternating which configuration ran first |
| Comparison conditions | Different request mixes; cache state unknown | Same request mix, dataset, concurrency, and warm-cache setup |
| Other changes | Database upgrade happened during the same period | Only the query change differed between test configurations |
| Tradeoffs checked | Not documented | Error rate and database CPU checked; production behavior still unverified |
A smaller improvement can support a stronger conclusion
Candidate A reports the larger percentage. Candidate B provides a more inspectable comparison: the observation count and execution order are explicit, and the workload conditions are matched. The table omits variation across pairs, so readers cannot judge how consistent the measured improvement was.
The arithmetic behind both percentages can be correct while the conclusions differ substantially. Candidate A's evidence does not isolate the query change from the database upgrade or workload differences. Candidate B's evidence supports a narrower statement about a particular endpoint under specified test conditions.
A large improvement can be real and still be poorly explained.
I would give Candidate B's result more weight as evidence of measurement discipline because of the documented method, not the smaller percentage. I would ask Candidate A for the missing context before assigning credit for the reduction. Neither result establishes hiring priority without considering the metric's relevance and the candidate's contribution.
Use a claim audit before assigning credit
Copy this claim audit into an interview document or portfolio review. Complete it for one consequential claim rather than demanding exhaustive documentation for every bullet point.
| Check | Evidence to record | Follow-up when incomplete |
|---|---|---|
| Baseline | Starting value, metric definition, affected system, why the baseline was representative, and why the metric mattered to users or the system | “Was the starting point normal operation, an incident, or a deliberately poor configuration, and why did this metric matter?” |
| Measurement | Data source, calculation, before-and-after windows, observation count, execution order, and variation between observations | “How was this measured, and how stable was the result across runs or days?” |
| Comparison conditions | Request mix, traffic volume, dataset, hardware, cache state, and other conditions that affect the metric | “What changed about the workload or environment between measurements?” |
| Alternative explanations | Concurrent deployments, infrastructure changes, seasonal effects, and the candidate's own contribution | “What else could explain the difference, and which decisions or implementation work were yours?” |
| Tradeoffs | Effects on reliability, resource use, operating cost, maintenance, and affected users | “Did the improvement move cost or failure risk somewhere else?” |
These checks expose different weaknesses. A missing baseline makes the percentage hard to interpret. A changed workload weakens the comparison. An unexamined cost increase can turn a local improvement into a questionable system decision.
Measurement quality and attribution are separate questions.
A dashboard can show a genuine improvement without establishing which change caused it or who deserves credit. A well-measured reduction in an irrelevant metric is limited evidence of engineering impact, though it may still demonstrate useful technical skills.
Rate the claim, not the person
End the audit with one of three evidence ratings:
- Supported: The evidence supports the claim at its stated scope. Important comparison conditions and tradeoffs are documented. A descriptive claim can be supported without proving causation.
- Qualified: The evidence supports a narrower claim. An improvement might be demonstrated in a test environment while the portfolio describes it as a production result.
- Unresolved: Missing information prevents a useful conclusion. Record the specific follow-up needed to assess it.
These labels classify support, not hiring priority. A supported descriptive claim need not outweigh a qualified causal claim. Record metric relevance and individual contribution beside each rating.
Missing evidence does not establish weak engineering ability. A candidate may lack access to historical dashboards, face confidentiality restrictions, or have inherited an imperfect measurement setup. Their explanation of the gap can still reveal whether they understand what the number shows.
For fairer comparisons, use the same worksheet and follow-up time for each candidate. Accept redacted artifacts, synthetic examples, or a clearly labeled reconstruction when original material cannot be shared. Do not reward disclosure of confidential employer data.
Evaluate the explanation separately from the documentation: can the candidate identify a confounder, choose a better comparison, and describe what they would measure next?
Help candidates make narrower, defensible claims
A useful portfolio statement connects the action, observation, scope, and limitation. It need not imply experimental certainty.
Candidate A could replace:
Reduced latency by 80% through query optimization.
With:
Implemented query optimizations during a period when measured latency fell from 1,000 ms to 200 ms. A database upgrade and request-mix changes occurred in the same period, so the available measurements do not isolate the effect of my changes.
That revision gives up an unsupported causal claim while preserving the observed result and the candidate's contribution. The next interview question can address the actual work: which query plans changed, why those changes were chosen, and how regressions were checked.
Candidate B could write:
Changed the checkout query, reducing median run-level p95 latency from 400 ms to 340 ms across four matched test pairs with alternating execution order. Error rate and database CPU showed no observed increase. Production impact was not measured.
Neither statement needs extra decimal places. Precision in wording matters more than precision the measurement cannot justify.
Review one claim before the next interview
Choose one portfolio bullet and spend five minutes filling in the worksheet. Mark missing information as unknown rather than guessing. Assign an evidence rating and record the metric's relevance and the candidate's contribution. If you are preparing your own work sample, use the review to rewrite the claim at the scope your evidence supports.
Finish by writing the single follow-up question most likely to change your assessment. A desk review cannot authenticate private records or replace a technical discussion.