I'm a Machine Learning Engineer working on LLM evaluation, agentic systems, AI infrastructure, and production-grade GenAI workflows.

Currently, I work at Microsoft Core AI, where I build evaluation frameworks for conversational AI systems, including multi-turn agent assessment, LLM-as-a-Judge pipelines, rubric-based scoring, and scalable AI quality measurement.

Previously, I worked at AWS on SageMaker AI, Bedrock, Ground Truth, human-in-the-loop systems, RLHF workflows, and agent-guided model customization.

• Multi-turn LLM evaluation systems
• Agentic orchestration frameworks
• LLM-as-a-Judge and rubric-based evaluators
• Human-in-the-loop and RLHF pipelines
• RAG and synthetic user simulation workflows
• MLOps infrastructure for AI/ML systems
• Applied ML systems across NLP, CV, and ranking

I'm especially interested in building AI systems that are not only powerful, but also measurable, debuggable, and reliable in production.

My current work focuses on:

• Evaluating multi-turn AI agents beyond single-response scoring
• Designing rubric-based and failure-mode-based evaluation systems
• Building synthetic user simulations for scalable agent testing
• Improving agreement between automated evaluators and human judgment
• Creating infrastructure that helps teams deploy, monitor, and improve AI workflows

Some directions I'm exploring:

• Agent reliability: evaluating whether agents can stay coherent, useful, and task-aligned across long multi-turn interactions
• Failure-mode discovery: automatically identifying recurring agent failures instead of relying only on predefined test cases
• Self-improving evaluators: building evaluators that learn from past failures and improve their diagnostic coverage over time
• Human-aligned scoring: closing the gap between automated evaluation and expert human judgment
• Production AI observability: connecting evaluation, monitoring, user feedback, and deployment pipelines into one reliability loop
• Model customization workflows: making fine-tuning, dataset validation, and model evaluation easier through agent-guided systems

Long term, I want to work on systems that make AI agents easier to trust, debug, and improve.

Languages
Python · Java · C++ · SQL · Bash · JavaScript/TypeScript

AI / ML
PyTorch · TensorFlow · Hugging Face · LangChain · OpenCV · RAG · LLM-as-a-Judge · RLHF · SFT · DPO · RLVR

Cloud & Infrastructure
AWS · Azure AI Foundry · SageMaker · Bedrock · Docker · Kubernetes · CI/CD · Spark · Kafka · Glue · Athena

Systems
Distributed orchestration · Event-driven pipelines · MLOps · Data validation · Model evaluation · Agent workflows

• Machine Learning Engineer 2 at Microsoft Core AI
• Former Software Development Engineer at AWS
• MIT graduate researcher in machine learning and applied AI
• Georgia Tech Computer Engineering graduate

• LinkedIn: linkedin.com/in/rayan-khoury-863ab61a5
• GitHub: github.com/rayankhouryy