Sourcing criteria consistency and job relevance

Two researchers can follow the same search request yet produce very different pools because words such as "senior," "strategic," or "strong communicator" lack operational meaning. Consistency work should make criteria inspectable without turning a nuanced role into a hidden keyword formula.

Job relevance comes before agreement

The Uniform Guidelines address evidence for selection procedures and the treatment of adverse impact. EEOC guidance similarly warns that a facially neutral practice may be unlawful if it disproportionately excludes a protected group and is not job related and consistent with business necessity. A team should therefore ask why a criterion belongs before asking whether sourcers apply it consistently.

OPM's assessment resources support analysis of work and competencies. Its structured-interview material illustrates the value of common questions and rating standards, even though sourcing is an earlier stage. CIPD's selection factsheet also emphasizes choosing methods appropriate to the role. None of these sources turns a convenient proxy, such as a particular school or employer brand, into proof of capability.

Write a criteria dictionary that can survive examples

Each entry needs more than a label.

Dictionary field	What to record
Work requirement	Task, outcome, or competency established for the role
Observable evidence	Information a permitted source can directly show
Inclusion example	A profile excerpt that satisfies the definition and why
Counterexample	Similar wording that does not satisfy it and why
Unknown state	Missing or ambiguous evidence that must not be scored negatively
Owner and version	Approver, effective date, and change history

Consider "experience leading distributed incident response." An inclusion example might show accountable coordination of a named incident across locations. Merely working for a remote company is a counterexample because it does not show incident leadership. No public evidence should remain unknown rather than become a rejection.

Avoid degree, years-of-experience, title, location, or continuous-employment rules unless the hiring owner has documented their relationship to the work and reviewed alternatives. Apparently precise filters can be poor measures of the intended competency.

Run a calibration that can reveal ambiguity

Create a small set of fictional or properly de-identified records. Include close cases and matched examples that vary only on information outside the approved criteria. Ask reviewers to classify each record independently and to cite the dictionary entry they used.

Do not start by forcing consensus. Examine the reason for every disagreement:

The criterion may contain undefined terms.
The source may not support the claimed fact.
Two entries may overlap.
A reviewer may be using an unapproved proxy.
An accessibility or accommodation issue may have been mistaken for missing capability.
The role itself may have changed since approval.

Resolve the definition with the accountable owner, publish a new version, and identify which active records require reevaluation. Keep the earlier decision history rather than silently rewriting it.

Distinguish three kinds of consistency

Interpretive consistency means reviewers understand a term similarly. Procedural consistency means comparable records receive the same approved steps. Decision consistency means evidence is translated into the same threshold. A team may have one without the others. Measuring only final agreement can conceal offsetting mistakes.

Consistency also cannot establish fairness. Reviewers might uniformly apply a criterion that is poorly related to work or functions as a proxy. EEOC's discussion of algorithmic tools reinforces that software-assisted decisions remain subject to Title VII principles. Automated queries should therefore use the same approved dictionary, change control, and impact review as manual searches.

Accessibility and accommodation boundaries

ADA.gov explains employment protections for qualified people with disabilities. DOL accommodation resources and the Job Accommodation Network provide practical hiring and accommodation information. These sources caution against treating the way evidence is presented as identical to the ability to do the work.

Sourcers should not diagnose disability or decide accommodation questions from profiles. If a criterion assumes one way of communicating, traveling, or performing a task, ask whether the role requires that method or the underlying outcome. Route accommodation requests and legal questions to authorized specialists.

Monitoring after launch

Sample decisions by criterion version, source, role, and reviewer. Look for unexplained differences, frequent unknowns treated as failures, and criteria that produce little direct evidence. An adverse-impact review requires appropriate data, definitions, and qualified oversight. A simple agreement score is not a substitute.

Do not publish a universal target for reviewer agreement from this evidence set. The appropriate diagnostic depends on category prevalence, sample design, decision stakes, and the measurement method. The useful output is a traceable correction: revised wording, added examples, retraining, removed criterion, or specialist review.

Limitations

This framework does not validate criteria or determine compliance. De-identified exercises may not reproduce live search conditions. Small operational samples can miss rare patterns, and demographic data may be unavailable or inappropriate for front-line access. US federal resources do not answer all state, national, or sector rules. Professional guidance is not law.

Consistent sourcing cannot correct exclusion already present in a platform's population or ranking. It also cannot establish candidate interest, eligibility, or eventual job performance. No causal or performance claim is made here.

Methodology and dated sources

The editorial team reviewed the following items on 2026-10-13. We used selection and job-analysis sources for relevance and procedure, EEOC materials for discrimination risk, and disability resources for accommodation context. Dates and versions are publisher supplied where available. The sources do not share data suitable for a combined numerical estimate.

EEOC, Employment Tests and Selection Procedures (2007)
NIST, AI Risk Management Framework (2023)
Uniform Guidelines on Employee Selection Procedures (current eCFR, accessed 2026-10-13)
OPM, Structured Interviews (accessed 2026-10-13)
OPM, Assessment and Selection (accessed 2026-10-13)
CIPD, Selection Methods (2024)
CIPD, Recruitment Overview (2024)
DOL, Disability Resources (accessed 2026-10-13)
EEOC, Disability Resources (updated 2024)
OPM, Disability Employment (accessed 2026-10-13)

Frequently asked questions

Who should approve a sourcing criterion?

The hiring owner should establish the work requirement with appropriate HR, assessment, accessibility, or legal input. A sourcing operator should not create a decisive threshold alone.

Should disagreement always be resolved by majority vote?

No. A majority can share the same unsupported interpretation. Return to work evidence, the dictionary, source quality, and the accountable approver.

Does equal treatment mean identical treatment in every situation?

Not necessarily. Approved accommodations and materially different evidence may require a different process while preserving the same job-related standard. Route those cases appropriately.

Use a free recruiting call to map criteria, calibration, and escalation responsibilities while keeping selection authority with your hiring leaders.