From Prototype to Production: Week 3 of the Agentic AI Summit
Week 3 of the Agentic AI Summit brought the series to a powerful close, shifting from building and deploying agents to evaluating, governing, and scaling them in production environments. Leaders from Google, Monte Carlo, Databricks, and the open-source ecosystem shared their hard-won insights on ensuring reliability, compliance, and continuous improvement for agentic systems. From Vertex AI Agent Engine to governance frameworks, from evaluation pipelines to multi-agent collaboration, this final week delivered the advanced tools and mindsets needed to operationalize agentic AI at scale.
You can also read the week 1 and week 2 recaps.
Day 1: Wednesday, July 30th
Deploying to Vertex AI Agent Engine & A2A
Ivan Nardini and Annie Wang, Developer Relations Engineers at Google Cloud
The speakers demonstrated how to build, deploy, and scale production-grade AI agents using Google’s agentic stack. They built a grounded agent with the Agent Development Kit (ADK) and Model Context Protocol (MCP), then deployed it on the Vertex AI Agent Engine. Finally, they enabled multi-agent collaboration through the Agent2Agent (A2A) protocol, showing the full lifecycle from local prototype to production-scale collaboration.
Building Reliable Multi-Agent Systems in the Enterprise: From Construction to Evaluation
Josh Reini, Developer Advocate at Snowflake
John showed how to design, deploy, and evaluate multi-agent systems using Snowflake Cortex. He built data agents that connected to structured and unstructured enterprise sources, performing multi-step operations with Cortex Analyst and Cortex Search. He then demonstrated how to instrument agents with inline evaluation, detect failure modes, and refine plans using TruLens and Cortex eval APIs. The session provided a repeatable framework for deploying trustworthy, production-ready agentic systems.
Building Behavioral Conversational Agents with Agent Flows and Agency
Micheal Lanham, Principle AI Engineer at Brilliant Harvest
Micheal taught participants how to design behavior-driven conversational agents using Agent Flows. He demonstrated how agents can use the Model Context Protocol (MCP) to access tools and memory, coordinate with other agents, and manage real-time interactions. The session included hands-on implementation, debugging, and evaluation techniques, giving attendees a framework for building adaptable, multi-agent conversational systems.
Agentic Workflows for Graph RAG: Evaluating & Benchmarking Results
David Hughes, Principal Data & AI Solution Architect at Enterprise Knowledge
David addressed the challenges of evaluating agentic Graph RAG systems, explaining why traditional LLM metrics fall short. He demonstrated how to design evaluation frameworks that measure retrieval quality, reasoning consistency, and decision reliability. Using OPIK, David showed how to benchmark full agent decision sequences and monitor production systems for performance degradation, ensuring Graph RAG workflows remain reliable and effective at scale.
Day 2: Thursday, July 31st
Governing AI Agents in the Enterprise
Amber Roberts, Staff Technical Marketing Manager at Databricks
Amber demonstrated how to build governance-aware AI agents using Databricks’ enterprise platform, focusing on HR scenarios with strict compliance requirements. She showed how to implement Unity Catalog row-level security, column masking, and data classification to protect PII. Participants built agents with embedded governance controls, validated them through MLflow and AI Gateway, and deployed them with shared policies — providing a framework for secure, compliant agent deployment in regulated industries.
Build Agentic Applications with Google Gemini
Philipp Schmid, AI Developer Experience at Google DeepMind
Philipp guided participants through building agentic AI systems with Google Gemini 2.5 Pro, starting from simple chatbot concepts and moving toward multi-step, tool-integrated workflows. He demonstrated how to use Google AI Studio and the Gemini API to prototype agents, leverage function calling and structured outputs, and design orchestrated workflows. The session bridged foundational theory with hands-on implementation, giving attendees practical experience in creating enterprise-ready Gemini-powered agents.
Build & Optimize Agentic AI Systems: From Rapid Prototyping to Research and Production
Chi Wang, Founder of AutoGen (Now AG2) and Senior Staff Research Scientist at Google DeepMind
Boris Bolliet, Head of Agentic AI at Infosys-Cambridge AI Centre
The speakers introduced AG2, an open-source AgentOS for building AI agents with high levels of autonomy. They demonstrated how to design and control agents using AG2 and Waldiez, a drag-and-drop UI for workflow creation. Through examples from academic research automation and a BetterFutureLabs production case study, they showed how AG2 enables rapid prototyping, deep analysis, and real-world decision-making in dynamic environments.
Agent Evaluation in 2025
Greg Loughnane, Co-Founder & CEO of AI Makerspace
Chris Alexiuk, Co-Founder & CTO of AI Makerspace and DL at NVIDIA
The speakers explored best practices for evaluating agentic LLM workflows using the RAGAS framework. They covered metrics like Topic Adherence, Tool Call Accuracy, and Agent Goal Accuracy, demonstrating how to assess outputs quantitatively and avoid hallucinations. Using tools such as RAGAS, LangChain, LangGraph, and LangSmith, they showed how engineers can instrument production-grade agents with robust evaluation methods.
Conclusion
With Week 3, the Agentic AI Summit completed its journey from foundational concepts to advanced enterprise deployment. Across three weeks, attendees learned how to design, deploy, evaluate, govern, and scale agentic AI systems that work in the real world. The final week underscored that the future of agentic AI isn’t just about capability — it’s about trust, governance, and continuous improvement. As the summit closes, the path forward is clear: building collaborative, trustworthy, and production-ready agents that can scale across industries.
If you missed your chance to attend the Agentic AI Summit live, then you can still watch it on-demand at any time for $399.99.
For an even better value, sign up for a bootcamp or VIP ODSC West 2025 pass and get access to the Agentic AI Summit and the entire AI+ Training library included! That’s hands-on live training during ODSC West and a year of on-demand training together. Use this link for an additional 10% off.
