AI Quality and Evaluation, Lead

About this role

Not quite right? These jobs in Pune match the same skills — apply to a few to improve your chances.

Senior Software Engineer

QAD, Inc.

HybridPune

Apply

Senior Software Engineer (5 to 8 years)

QAD, Inc.

In OfficePune

Apply

Principal Software Engineer (8-12 years)

QAD, Inc.

In OfficePune

Apply

Senior Principal Software Engineer (12+ years)

QAD, Inc.

In OfficePune

Apply

Skills you'll use

What you'll do

• Build and own a shared evaluation framework across the product organization: golden datasets, LLM-judge rubrics, and code-based checks. Measure not just response quality but the quality of the decisions the AI products produce — did the recommendation actually serve the customer outcome the product was built for?
• Own the technical criteria for stage-gate release reviews: define what evaluation evidence a product must produce to clear each gate, and each capability-tier progression. You don't chair the gates; the Head does. But a gate cannot pass without your evidence.
• Own decision auditability: every AI-driven recommendation must be logged with the context considered, the rationale, and the outcome — in a way that's faithful, end-to-end, and useful for both customer trust and continuous improvement of the product.
• Own production drift detection: monitor evaluation-score regression, human-override-rate increase, and exception-rate spikes. Treat these as leading indicators of customer issues and surface findings before they show up as support escalations. When a product regresses, you trigger a capability-tier review.
• Define blast-radius and rollback requirements with engineering, and gate releases on thresholds in CI.
• Partner with each product team's AI lead to translate "what good looks like" into rubrics — breaking quality into independent dimensions (correctness, constraint compliance, decision quality, latency, tone) rather than one blended score.
• Run a regular evaluation-review cadence across teams, surface regressions early, and build the organization's shared vocabulary for what product quality means in this domain.
• 5-8 years in product ops, AI/ML product, data, or quality engineering, with hands-on AI evaluation experience on production LLM systems.

Who can apply

• Strong fluency with agent architecture: traces and spans, tool calls, RAG/groundedness, LLM-as-judge, offline vs. online evaluations, drift monitoring.
• Practical technical skills — SQL plus Python or JavaScript; comfort with evaluation/observability platforms and Git.
• A discovery mindset toward evaluation: failure data and edge cases are the core of the job, not a postmortem activity.
• Bias toward decision quality over surface quality. We are not optimizing for hallucination rate alone — we're measuring whether the AI's recommendation actually served the customer's need.
• Your health and well being are important to us at QAD. We provide programs that help you strike a healthy work-life balance.
• Opportunity to join a growing business, launching into its next phase of expansion and transformation.
• Collaborative culture of smart and hard-working people who support one another to get the job done.
• An atmosphere of growth and opportunity, where idea-sharing is always prioritized over level or hierarchy.

About QAD, Inc.

Searching for “QAD, Inc. jobs” or “jobs at QAD, Inc.”? You’re in the right place.