Software Engineering Director, Agentic Evaluations

About The Role

As a Software Engineering Director, Agentic Evaluations, your responsibilities will center around delivery of useful and credible evaluations of agents offered by software vendors. You’ll enhance the system that performs validation of agent performance, and lead the core team behind it. You will leverage “agent-first” ways of working, applying agentic engineering techniques to deliver your work to production.

Detailed Responsibilities:

Build and enhance agent evaluations that run against live agents [80%]:

• Help scope out the feasibility and effort involved in evaluating an agent offered by a software vendor, identifying the most practical path to reaching a credible evaluation
• Own the full stack of the core evaluation system, from admin & API surface to eval workflow dispatch
• Design evaluation system primitives that generalize across software verticals, so onboarding a new category benefits from reuse
• Distill the appropriate and repeatable processes in onboarding and maintaining integrations into repeatable AI skills or agents to increase evaluation velocity
• Keep tabs on emerging frameworks and techniques for agent evaluation which offer improvements to evaluations, and provide industry-accepted means to distribute eval results
• Collaborate with data science peers in their pursuit to develop rich and proprietary benchmarks

Lead a group of engineers and be a source of wider influence [20%]:

• You will facilitate the growth and mentorship of a core team of engineers, and be a source of technical leadership and subject matter expertise in evals for them
• You will also help socialize the use of evals in agent-oriented products built by the wider organization.