Senior Machine Learning Engineer , LLM Evaluations

About UsHippocratic AI is the leading generative AI company in healthcare. We have the only system that can have safe, autonomous, clinical conversations with patients. We have trained our own LLMs as part of our Polaris constellation, resulting in a system with over 99.9% accuracy.Why Join Our TeamReinvent healthcare with AI that puts safety first. We’re building the world’s first healthcare‑only, safety‑focused LLM — a breakthrough platform designed to transform patient outcomes at a global scale. This is category with the people shaping the future. Hippocratic AI was co‑founded by CEO Munjal Shah and a team of physicians, hospital leaders, AI pioneers, and researchers from institutions like El Camino Health, Johns Hopkins, Washington University in St. Louis, Stanford, Google, Meta, Microsoft, and NVIDIA.Backed by the world’s leading healthcare and AI investors. We recently raised a $126M Series C at a $3.5B valuation, led by Avenir Growth, bringing total funding to $404M with participation from CapitalG, General Catalyst, a16z, Kleiner Perkins, Premji Invest, UHS, Cincinnati Children’s, WellSpan Health, John Doerr, Rick Klausner, and alongside the best in healthcare and AI. Join experts who’ve spent their careers improving care, advancing science, and building world‑changing technologies — ensuring our platform is powerful, trusted, and truly transformative.Location RequirementWe believe the best ideas happen together. To support fast collaboration and a strong team culture, this role is expected to be in our Palo Alto office five days a week, unless otherwise specified.About the RoleEvaluation is the bottleneck in healthcare AI — you can't ship what you can't measure. You'll build the systems that tell us whether our models are safe, accurate, and ready for real patients: evaluation frameworks, synthetic data pipelines, automated benchmarks, and LLM-as-judge systems. This is a high-leverage engineering role where your work directly gates what goes to production.What You’ll DoDesign and build evaluation frameworks for LLM safety, clinical accuracy, and conversational qualityDevelop synthetic data generation pipelines to stress-test models across diverse clinical scenariosBuild automated and human-in-the-loop evaluation pipelines at scaleCreate benchmarks, metrics, and LLM-as-judge systems for healthcare tasks and conversational experienceAnalyze failure modes and translate findings into actionable model improvements by collaborating with the LLM post-training teamCollaborate with research, engineering, and clinical teams to define and raise the quality barWhat You BringMust-Have:MS or PhD in CS or related field5+ years in ML engineering, evaluation systems, or applied MLStrong software engineering skills — Python, PyTorch, and production-quality codeHands-on experience with LLM evaluation, benchmarking, or synthetic data generationComfort building robust data analysis and evaluation infrastructure, not just running experimentsExperience with UI/UX and front-end development toolkits such as Streamlit, Gradio, React, etc.Nice-to-Have:Experience in healthcare AIExperience with RL/RLVR/RLHF or safety evaluationPlease be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.