Senior AI Engineer 

Job Summary 

Key Responsibilities 

• Design, develop, and deploy scalable AI/ML and Agentic AI solutionsfor enterprise applications. 

• Build and integrate LLM-powered applications, AI agents, APIs, and intelligent automation solutions. 

• Develop solutions using Azure AI Agents, Azure AI Services, Azure Machine Learning, and Microsoft Fabric. 

• Design and implement AgentOps pipelines for data preparation, model development, evaluation, deployment, and monitoring. 

• Work with Large Language Models (LLMs), prompt engineering, RAG, embeddings, vector search, and AI agents. 

• Develop robust and scalable applications using Python and C#/.NET. 

• Integrate AI capabilities with enterprise applications through REST APIs and cloud-native services. 

• Work with Microsoft Fabric for data engineering, analytics, and AI/ML use cases. 

• Implement model and application optimization techniques to improve performance, scalability, accuracy, and reliability. 

• Collaborate with data engineers, software engineers, solution architects, and business stakeholders to translate requirements into technical solutions. 

• Participate in technical design, architecture discussions, code reviews, testing, and production deployments. 

• Implement CI/CD, version control, and deployment workflows using Git/GitHub and Azure DevOps. 

• Troubleshoot production issues and provide technical solutions for AI/ML applications. 

• Follow secure coding, cloud security, data governance, and enterprise development best practices. 

• Mentor junior AI engineers and contribute to technical knowledge sharing within the team. 

• Stay updated with emerging developments in Generative AI, Microsoft AI technologies, LLMs, and Machine Learning. 

Qualifications 

• Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, Information Technology, Engineering, or a related field. 

• 5+ years of professional experiencein AI/ML, Machine Learning Engineering, Data Science, Software Engineering, or a related role. 

• Strong hands-on experience in developing and deploying production-grade AI/ML solutions. 

• Strong programming experience in Python. 

• Good hands-on experience with C#/.NETand enterprise application development. 

• Strong understanding of Machine Learning concepts, model evaluation, optimization, and deployment. 

• Experience working with cloud-based AI and data platforms, preferably within the Microsoft Azure ecosystem. 

• Strong analytical, problem-solving, communication, and collaboration skills. 

Required Technical Skills 

Microsoft & Azure Technologies 

• Microsoft Azure 

• Azure OpenAI Service 

• Azure AI Services 

• Azure Machine Learning 

• Microsoft Fabric 

• Azure Functions and other Azure cloud services 

• Azure DevOps 

AI / Machine Learning 

• Machine Learning and Deep Learning fundamentals 

• Generative AI and Large Language Models (LLMs) 

• Prompt Engineering 

• Retrieval-Augmented Generation (RAG) 

• Embeddings and Vector Search 

• AI Agents / Agentic AI 

• Model evaluation and optimization 

• AI/ML deployment and monitoring 

Programming & Development 

• Python – Strong proficiency 

• C#/.NET 

• SQL 

• REST APIs 

• Git/GitHub 

• Software development and debugging practices 

Data & Cloud 

• Microsoft Fabric 

• Data processing and data pipelines 

• Cloud architecture and services 

• Data integration and API development 

Preferred Skills 

• Azure Agent Framework 

• LangChain / LangGraph 

• Azure AI Search 

• Azure Data Factory 

• Databricks 

• Docker and containerization 

• Kubernetes 

• MLOps / LLMOps 

• CI/CD pipelines 

• Power BI 

• Experience building enterprise AI agents and copilots 

• Experience with vector databases and semantic search 

• Knowledge of responsible AI, AI security, and governance 

About the Role 

What You Will Do 

Own the evaluation process 

• Build and maintain evaluation suites for the full agent stack. 

• Test prompt behavior, tool selection, tool calls, multi-step plans, memory use, error recovery, and task results. 

• Define quality metrics for each capability. Examples: task success rate, trajectory correctness, tool-call accuracy, latency, and cost. 

• Build and maintain golden datasets, regression suites, and adversarial test sets. Put these datasets under version control. 

Build evaluation infrastructure 

• Build automated evaluation pipelines. Connect them to CI/CD. 

• Make sure each change to a model, prompt, tool, or the harness must pass evaluation before release. 

• Use more than one grading method: rule-based checks, exact match, semantic match, human review, and LLM-as-judge. 

• Calibrate LLM judges against human labels. Monitor the judges for drift. 

• Add tracing and logs to agent runs. Record each step, each tool call, and each cost. Make failures easy to diagnose. 

Establish and standardize KPIs and metrics 

• Define the KPI set for agent quality. Include accuracy, faithfulness, answer relevance, context precision, context recall, tool-call correctness, hallucination rate, latency, and cost. 

• Write standard definitions for each metric. Write standard rubrics and report formats. 

• Make sure teams can compare results across models, prompts, and releases. 

• Extend evaluation to more than one modality: text, image, audio, and structured documents. Select the correct grading method for each modality. 

Analyze failures and drive fixes 

• Do structured error analysis. Group failures by type. 

• Find the source of each failure: the model, the harness, or a tool. 

• Set sample sizes, pass thresholds, and confidence bounds. Agent runs are not deterministic. Plan for variance. 

• Work with engineers to reproduce failures and to verify fixes. 

Protect the evaluation results 

• Find and prevent test contamination, overfitting to benchmarks, and metric gaming. 

• Keep a mix of offline evaluations, staged evaluations, and production monitoring. 

• Make sure evaluation results predict production behavior. 

Set the quality bar 

• Define release criteria for agent changes. Approve or block releases with data. 

• Write clear documentation for the evaluation methods. 

• Report results to engineering and to leadership. 

What We Require 

AI and agent evaluation (4+ years) 

• 4 or more years of hands-on work in evaluation of LLM systems and agents. 

• Hands-on skill with RAGAs, Promptfoo, and DeepEval. Skill with Braintrust, LangSmith, W&B Weave, or OpenAI Evals is a plus. 

• Proof that you can create KPIs and metrics from zero: select the metrics, set the baselines, set the thresholds, and get agreement from stakeholders. 

• Proof that you can standardize evaluation across teams: shared metric definitions, reusable templates, versioned datasets, and one report format. 

• Experience with evaluation in more than one modality: text plus image, audio, video, or structured documents. 

• Knowledge of the agent evaluation problem space: trajectory evaluation, outcome evaluation, tool-use correctness, multi-turn behavior, non-determinism, and prompt sensitivity. 

• Experience with LLM-as-judge design: rubric design, calibration against human labels, and bias control. 

• Knowledge of statistics for evaluation: sampling, significance tests, inter-rater agreement, and variance across runs. 

• Strong Python skills. Experience with test harnesses, data pipelines, or evaluation tools that you built. 

• Skill with tracing and observability for agent systems. Example: OpenTelemetry-style traces and structured logs of tool calls. 

Core software QA 

• Knowledge of QA fundamentals: test plans, test-case design, functional tests, regression tests, integration tests, and defect management. 

• Hands-on skill with test automation frameworks. Examples: pytest, Playwright, Selenium. 

• Hands-on skill with API tests. Examples: Postman, REST-assured. 

• Experience with test suites in CI/CD pipelines. Examples: GitHub Actions, Jenkins, GitLab CI. 

• Skill with bug tracking and test management tools. Examples: Jira, TestRail, Xray. 

• Ability to write clear defect reports that engineers can reproduce. 

• Knowledge of non-functional tests: performance, load, and reliability. 

What Is a Plus 

• Experience with red-teaming, safety evaluations, or adversarial tests of AI systems. 

• Experience with human annotation work: labeling guidelines, calibration sessions, and inter-annotator agreement. 

• Background in CI/CD, infrastructure-as-code, or platform engineering. 

• Contributions to open-source evaluation tools. Published work on evaluation methods. 

What Success Looks Like 

• 90 days: You know the current agent harness. You found the gaps in the current evaluations. A baseline regression suite runs on each change. 

• 6 months: Evaluation results are trusted release gates. Failure types are tracked and decrease. Engineers come to you before they ship. 

• 12 months: Other teams use the evaluation platform on their own. Offline results predict production quality. Our quality bar is a competitive advantage. 

Why This Role Matters