AI Evals Engineer
Hyderabad, India · Hybrid · Full-time
- Posted 6d ago
- From MSD’s careers page
- Location
- Hyderabad, India
- Work mode
- Hybrid
- Type
- Full-time
- Level
- Senior
- Experience
- 7+ years
- Department
- Software Development
Apply on MSD’s site
Opens the listing on jobs.msd.com
Let the right jobs find you
In your inbox every Wednesday and SaturdayPersonalised suggestions from verified career pages, matched to your role, location, level and skills.
About the role
The Opportunity
- Based in Hyderabad, join a global healthcare biopharma company with a 130-year legacy of success backed by ethical integrity, forward momentum, and an inspiring mission in global healthcare.
- Be part of an organisation driven by digital technology and data-backed approaches that support a diversified portfolio of prescription medicines, vaccines, and animal health products.
- Drive innovation and execution excellence. Be part of a team with passion for using data, analytics, and insights to drive decision-making, and which creates custom software to tackle some of the world's greatest health threats.
Role Overview
It is a Senior AI Evals Engineer position in our Hyderabad Tech Center. You will have the opportunity to work on global teams to identify and tackle the biggest opportunities and challenges at the intersection of healthcare, information and technology.
What will you do in this role
- Lead the design, build, and execution of evaluations of AI systems across the stack, from foundation and fine-tuned models to retrieval-augmented generation (RAG) pipelines and agentic applications, assessing quality, accuracy, robustness, and safety.
- Define and develop evaluation datasets, test suites, metrics, and scoring rubrics, including automated and LLM-as-a-judge approaches, and establish rigorous methods to validate them against human judgement.
- Own the evaluation strategy and architecture: select, adapt, and extend established evaluation frameworks and benchmarks, and build scalable, reusable evaluation pipelines and tooling.
- Analyze and interpret evaluation results, identify failure modes, and communicate findings and recommendations to engineering teams, senior stakeholders, and leadership.
- Partner with AI engineering, product, and risk and governance teams to set acceptance criteria, evaluation standards, and best practices for AI use-cases across the company.
- Provide technical leadership and mentorship to AI evaluation engineers, review their work, and lead internal communities of practice.
- Proactively scan and keep up to date with AI evaluation research, frameworks, and best practices, and drive adoption of new evaluation methods where they add value.
What should you have
- Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Machine Learning, Data Science, Engineering, or a related science or technology discipline.
- 7 to 11 years of experience in fields related to AI and ML evaluations, testing and benchmarking.
Required Skills
- Quantitative, scientific and technology background such as Mathematics, Statistics, Analytics, Physics, or Bioinformatics.
- 6+ years of experience working on AI and ML projects, including hands-on experience designing and leading evaluations of AI systems such as models, RAG pipelines, and agents.
- Deep knowledge of AI evaluation frameworks and approaches, including benchmark-based evaluation, human evaluation, LLM-as-a-judge, RAG metrics, agent task-completion and trajectory evaluation, and safety or red-team testing.
- Strong hands-on Python development experience, including the scientific computing stack and evaluation tooling, and experience building production-grade evaluation pipelines.
- Experience with AI testing and evaluation platforms is a plus.
- Experience in qualitative and quantitative research methods is a plus.
- Familiarity with cloud-based infrastructure and services (AWS or MS Azure) is a plus.
- Domain knowledge in pharmaceutical business areas such as research, clinical, manufacturing, or commercial is a plus.
- Ability to communicate technical viewpoints to senior stakeholders with succinctness and clarity, and to influence decisions.
- Experience providing technical leadership, mentoring engineers, and driving technical direction on cross-functional projects.
- Proficiency in GenAI, NLP, and ML frameworks.
What we offer
- Exciting work in a great team, global projects, international environment.
- Opportunity to learn and grow professionally within the company globally.
- Hybrid working model, flexible role pattern.
- Internal reward system plus referral program.
- Competitive salary, incentive pay, and more.
Skills they ask for
Pick one to see other roles that ask for it.
About MSD
Medicines and healthcare productsMSD is a healthcare company that researches and develops medicines and vaccines for people and animals.
See all 5 roles at MSDMore roles at MSD
See all 5- Software Engineering ManagerHyderabad · Senior · HybridSoftware Development · Senior · HybridHyderabad, India3d
- AI Engineering SpecialistHyderabad · Senior · HybridSoftware Development · Senior · HybridHyderabad, India4d
- Data Products DirectorHyderabad · Director · HybridData and Analytics · Director · HybridHyderabad, India1w
- Process Automation SpecialistHyderabad · On-siteEngineering · On-siteHyderabad, India1w
Share this role
Let the right jobs find you
In your inbox every Wednesday and SaturdayPersonalised suggestions from verified career pages, matched to your role, location, level and skills.