Define the annotation task precisely
Specify label classes, unit boundaries, allowed overlaps and the treatment of empty or unclear examples. Confirm permitted access to the source material and choose an export format that preserves item identifiers and label versions.
Calibrate before scaling
Use agreed gold samples and independent review on a representative subset. Record disagreements separately from clerical errors, and have an authorized reviewer settle ambiguous cases before updating the guideline and affected records.
Assess label complexity
Pilot effort varies with object density, text length, domain knowledge and review requirements. Agree acceptance by task type, critical error rules and revision handling; pricing cannot be inferred from raw file count alone.
A label dictionary must define what counts and what does not
Specify the annotation unit before listing classes. For a support-message dataset, the unit might be the full customer turn, a sentence or a span; those choices produce different labels. State whether multiple labels can coexist, how quoted text is treated and what to do when the message lacks context. A label name such as refund is insufficient without operational boundaries.
For each class, include a positive example, a nearby negative and an ambiguous example. Define an uncertainty route rather than forcing uncertain items into the nearest class. Preserve source IDs and guideline versions so a later policy decision can identify affected records without rereading the whole dataset.
Illustrative example: labeling customer intent at message level
In this hypothetical task, the client wants one primary intent per complete customer message: refund request, delivery-status request or other. Guideline G1 says to label the action explicitly requested, not every topic mentioned. The operator receives four message IDs and uses an uncertainty flag when the requested action is unclear. Two reviewers disagree on M04, so it is held for the client’s label owner rather than settled by majority guess.
The owner adjudicates M04 as a refund request because “that” refers to a refund in the permitted preceding turn. G2 adds a context rule and a reviewed example. The operator relabels the affected item, checks other flagged G1 items and exports message_id, primary_intent, guideline_version and adjudication_id in the agreed format. Acceptance requires valid IDs, allowed labels, documented adjudication and no unresolved forced labels; this tiny example does not establish model quality.
| ID / text | Dictionary treatment | Released label / evidence |
|---|---|---|
| M01: “Please refund this order.” | Positive refund example: explicit requested action | refund_request / G2 |
| M02: “Where is my parcel?” | Negative refund example: asks for location, no refund | delivery_status / G2 |
| M03: “The refund arrived, thank you.” | Near negative: mentions refund without requesting it | other / G2 |
| M04: “Yes, please do that.” | Ambiguous alone; preceding turn asks about refund | refund_request / G2 / adjudication A4 |
Separate disagreement, instruction gaps and execution defects
Two reviewers can agree on the wrong interpretation, so agreement is not a replacement for reference decisions. Use a client-approved reference set to test rule application, and independent overlap to discover inconsistent interpretation. Record which classes and edge conditions cause differences instead of reporting a single percentage without its sample size.
A missed explicit rule is an execution defect; a case the guideline does not cover is an instruction gap. Route them differently. The first needs correction and targeted feedback; the second needs adjudication, a versioned rule change and a decision about previously released items. Do not silently edit reference labels to make agreement improve.
Verify exports against the intended downstream use
For classification, validate allowed values, required IDs and one-row-per-unit rules. For spans or image regions, also agree coordinate conventions, text offsets and overlap rules. Export success means the consumer can associate the label with the intended original item; a visually convincing review screen does not prove that mapping.
Deliver rejected or unresolved items in a separate register with their reasons. If the dataset is split for training and evaluation, preserve the client’s split assignments and prevent accidental duplicate items crossing them. The client’s model owner decides whether the released labels meet the downstream experimental purpose.