ML Annotation QA Engineer

About the Role

We are looking for an ML Annotation QA Engineer to own the quality of annotated data across our computer vision and machine learning programs. This role is responsible for the judgment-heavy analysis that cannot be reliably outsourced, for the decision rules behind it, and for turning annotation output into an ongoing read on how our systems are actually performing in the field.

What You’ll Do

• Own the judgment-heavy quality analysis on annotated data that cannot be reliably outsourced — working the daily review queue and producing verdicts and root cause in house.
• Own, version, and refine the verdict taxonomy, decision rules, and quality guidelines for the categories you cover.
• Build and maintain performance trackers over annotated data — error rates by facility, site, equipment, and data format over time, against an agreed baseline.
• Detect anomalies against that baseline and flag them the day they appear rather than weeks later.
• Run root cause analysis on flagged anomalies, distinguishing annotation error from model or system error from genuine degradation in the field.
• Report findings to engineering and ML with reproducible evidence and stated confidence, fast enough that the issue is still observable.
• Identify systematic failure patterns rather than one-off misses, and maintain a documented pattern library others can use.
• Query and analyse annotation data directly with Python and SQL to test hypotheses, without waiting on extracts from anyone.
• Feed annotation-quality findings back as concrete SOP and instruction changes when the root cause is labeling rather than system behaviour.
• Specify annotation tool improvements — what the tool should surface so analysis stops requiring manual work — and validate the fixes.
• Stand up quality analysis and reporting for new annotation programs as they come online.
• Track work across Jira and contribute to pre-release validation for the behaviours you cover.

Required Technical Skills

• BS in Computer Science/Engineering, Electrical Engineering, or equivalent experience
• Experience working with annotated datasets for CV/ML, including assessing label quality
• Strong understanding of statistics, and able to work with data
• Root cause analysis — generating competing hypotheses and naming the evidence that separates them
• Familiarity with Python and SQL, or equivalent, for querying and analysing data independently
• Writing quality guidelines, decision rules, and labeling taxonomies
• Excellent documentation, communication, and collaboration skills

Required

• BS in Computer Science/Engineering, Electrical Engineering, or equivalent experience
• 2–5 years of experience in ML QA, annotation quality, data quality, or analytics at an AI/ML company
• Hands-on experience with annotated ML datasets, including assessing label quality
• Demonstrated experience finding, diagnosing, and reporting data anomalies to a technical audience
• Able to get to a defensible answer from unfamiliar data without someone preparing it first
• Understanding of data privacy and confidentiality requirements when working with customer operational data

Required Technologies

• Python
• SQL, or an equivalent query language
• Jira