Senior AI Evals Engineer

About the Role:

The Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar agentic frameworks. This role bridges AI development and reliable deployment, ensuring agents are safe, ethical, effective, and meet high-quality performance standards. A significant focus is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement. The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams. The role is 70% Automation and 30% Manual Testing, and sits within the Applied AI: Test Engineering team as a key technical authority.

Key Responsibilities:

• Design and build agent evaluation frameworks, including end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate
• Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions
• Build golden datasets and test harnesses, curating domain-specific evaluation sets for Rubrik's product verticals
• Establish eval-as-a-service, building a centralized evaluation platform that IT team members can self-serve
• Automate regression and drift detection with CI/CD-integrated eval pipelines
• Define eval taxonomies and rubrics for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate, policy adherence, error recovery)
• Instrument production observability with real-time monitoring of agent quality
• Benchmark and select models by running structured cost, quality, and latency comparisons
• Analyze and interpret evaluation results, providing actionable quality and safety reports
• Define end-to-end QA strategies covering functional, integration, regression, and user acceptance testing for conversational and goal-oriented AI agents
• Develop and maintain test artifacts such as test plans, test cases, test scenarios, and automated test suites
• Identify, document, prioritize, and track defects and alignment failures using Jira, working with Applied AI Engineers on root causes
• Evaluate new technologies and patterns, run proofs-of-concept, and influence the platform roadmap
• Partner with policy and legal teams to cover regulatory compliance, fairness objectives, and AI governance requirements
• Provide technical leadership to engineers, including design reviews, code reviews, pair programming, and mentoring
• Collaborate cross-functionally to embed eval-driven quality gates into the agent development lifecycle

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
• 8+ years of overall software test engineering experience, with 5+ years focused on test engineering, QA automation, or developer tooling
• 2+ years working with ML/AI systems, including testing or evaluating models in production environments
• 2+ years as a senior or lead, owning evaluation strategy, quality frameworks, and technical decisions for complex AI agent systems
• 1+ years designing or operating evaluation frameworks for generative AI / LLM agents, including benchmarking, red-teaming, and production quality monitoring

Technical Skills:

• Deep hands-on experience with LLM evaluation using frameworks such as LangSmith, DeepEval, LLM as a Judge, Promptfoo, RAGAS, Hugging Face evaluation tooling, and custom eval harnesses
• Strong understanding of agentic AI patterns, including tool use, function calling, multi-step planning, ReAct/CoT reasoning, and RAG
• Production experience with CI-integrated evaluation pipelines
• Proficiency in Python, including pytest, async patterns, and data processing at scale
• Familiarity with Google Cloud Platform or other cloud services relevant to AI/ML, and integrating testing into MLOps workflows
• Statistical literacy for designing experiments and interpreting eval metrics
• Enterprise product sensibility, including understanding of how AI quality affects customer trust in a B2B context
• Excellent communication with engineering leadership and non-technical audiences
• Hands-on experience with CI/CD pipelines and Git-based workflows, including automated testing, static analysis, and deployment automation

Tech Stack:

• Languages: Python (Must), TypeScript, Java (Optional)
• Test Automation: Selenium, Playwright
• LLM Providers: Anthropic Claude, OpenAI, Azure OpenAI, open-source models
• Eval Tooling: DeepEval (Must), LLM as a Judge (Must), Promptfoo, RAGAS, LangSmith, custom harnesses
• Agent Evals Frameworks/Libraries: LangGraph, Claude Agent SDK, custom orchestrators
• Infrastructure: DevOps (Basic), Kubernetes, CI/CD (Jenkins/GitHub Actions), observability stacks (Datadog/Grafana)

Preferred Qualifications:

• Fundamental understanding of how LLMs work, including RAG, prompting, and alignment challenges
• Experience with data analysis and visualization tools (e.g., Pandas, SQL) for evaluation and safety results
• Familiarity with AI ethics principles and responsible AI industry standards
• Experience with tools that test model responses to ambiguous, misleading, or malformed inputs
• Experience testing complex APIs, especially for tool-integration and function-calling scenarios
• Experience with prompt injection testing (Must) and data poisoning is a plus
• Certifications in AI/ML, cloud platforms (GCP, AWS), or software testing methodologies

Soft Skills:

• Ability to lead without direct authority, influencing cross-functional teams through clarity and data
• Experience working in agile environments with product owners, scrum teams, and distributed stakeholders
• Excellent communication and documentation skills
• Strong architectural thinking and problem-solving skills

Nice to have:

• Experience evaluating agents in cybersecurity, data protection, or infrastructure domains
• Familiarity with Rubrik's product portfolio or similar enterprise data management platforms
• Hands-on experience with human-in-the-loop eval workflows
• Experience with model fine-tuning evaluation
• Background in building internal developer platforms or tooling teams
• Contributions to open-source eval frameworks or published work on AI evaluation methodology
• Experience designing adversarial evaluation suites for jailbreak resistance, data leakage prevention, and boundary enforcement