Replay: Rubber Duck Thursdays: Building Agents with Copilot

Introduction and Audience Engagement

• The stream begins with a greeting and a discussion about the current time in London, which is 11:30 AM, and mentions viewers from various countries including Australia, Italy, India, Nigeria, the Netherlands, Turkey, the United States, Greece, Germany, and the UK. 10s
• The host expresses enthusiasm for discussing the topic of building AI agents and interacts with viewers by asking about their current projects. 42s

Host's Setup and Viewer Projects

• The host shares their preference for using Visual Studio Code (VS Code) for coding and displays it on the screen. 2m6s
• Viewers share their projects, including Enrique building an AI OCR app, James working on a marketing engine and a compliance application for pest control, Zara developing an EHR platform, and Moan working on AI agents with Python. The host emphasizes the effectiveness of niche AI agents. 2m6s

Initial Discussion on AI Agent Projects

• The discussion begins with mentions of various projects, including building a PC system and an app with Django, a framework that is well-liked for its capabilities, depending on the specific project requirements 10s.
• Ditri Gabriel raises a question about building AI agents with Copilot, specifically regarding the recommended way to enforce a zero-trust security context, as AI agents can generate vulnerabilities such as silently mounting authenticated upload routes 2m6s.

Security and Compliance in AI Agent Development

• The importance of security and compliance in AI development is highlighted, with guardrails being a crucial aspect to consider when building agents to prevent unintended actions and ensure user safety 2m6s.
• A workshop was conducted with the Claude team, where participants learned how to build AI agents using Claude models, and the materials from this workshop are available for download on GitHub 4m30s.
• The partnership between Microsoft, GitHub, and Anthropic is mentioned, with many people using Claude models in these platforms, and there is a growing need to move beyond experimental agent building to shipping them into production while ensuring user safety 6m20s.
• The challenge of keeping users safe and preventing agents from performing unwanted actions is a key concern, with many people seeking the best ways to achieve this and ensure their agents are secure and reliable 8m30s.

Workshops and Frameworks for AI Agent Development

• A link to a GitHub repository for a workshop on building AI agents using the Microsoft agent framework is shared. The workshop was created by a coworker named Hank and involves building an AI agent from scratch. 10s
• A custom AI agent project is demonstrated, which includes adding content safety middleware to ensure the agent does not perform undesirable actions. This middleware acts as a filter around the language model, allowing the user to define specific filters and their strictness. 1m0s
• The process of building an AI agent using Langchain and connecting it to a deployment, such as GBT 5 Mini, is explained. The middleware is used to prevent the agent from responding inappropriately. 2m6s

Building a Custom AI Agent with Middleware

• An example is provided where a simple agent is run using a Python script, but there are issues with activating the correct environment for the agent to function properly. 3m0s
• There is an interaction with the audience, where feedback and questions are addressed, and participants from different locations, such as Germany, are acknowledged. 4m0s
• Gabriel expressed gratitude for addressing a question about security context, which is a significant challenge for developers. Gabriel shared security prompts in different languages to help prevent vulnerable code. 10s

Technical Challenges and Solutions in Agent Development

• There was a discussion about whether to rely on model providers' inbuilt safety features or to add custom middleware and policy layers when building agents. It was suggested that using platforms like Azure for enterprise projects is beneficial due to their built-in content safety features. 42s
• There were technical difficulties with screen sharing, which were eventually resolved, allowing for a demonstration of the process of building and running different agents. 2m6s
• The process involved navigating directories and setting up the correct virtual environment to install necessary dependencies and middleware for the agents. There was a focus on ensuring the correct virtual environment was activated to run the required commands successfully. 2m6s

Implementation Details and Code Setup

• A lang chain agent is being built by importing several modules, including create_agent from Langchain, chat_open_AAI for model definition, and a middleware function for content safety guardrails. The agent is given an endpoint and uses the GPT5 mini model for demonstration purposes. Content safety guardrails are set by defining categories to filter and setting a threshold for filtering strictness 10s.
• There are technical difficulties encountered while attempting to run the agent, including issues with the code and missing installations. Attempts are made to run the agent without middleware to troubleshoot the problems 2m6s.
• The stream experiences multiple technical issues, including audio outages and incorrect screen sharing, which prevent the demonstration of code and explanation of building AI agents and content safety moderation. Apologies are made for the disruptions, and it is noted that the stream has been challenging due to these issues 2m6s.

Workshop Overview and Deployment Process

• The workshop at the Code with Claude conference was about teaching people how to build with Microsoft Agent Framework and Claude models, and a GitHub repository is available with the manual for using Foundry with Claude 10s.
• The Claude models are available on Azure due to a partnership between Microsoft and the Claude team, and they can be used with Co-pilot to build agents 2m6s.
• The workshop provides a full walkthrough of how to deploy a model in Foundry, including choosing which model to use, such as the Sonnet 4.6 model, and then testing it out 4m30s.
• The goal of the workshop is to show people how to use Microsoft Agent Framework and Claude models to create an agent, and the workshop includes a practical example where participants can order a cupcake using an MCP server 6m40s.

Content Safety and Middleware Integration

• Content safety is an important consideration when building agents, and middleware can be used to filter out malicious prompts and other unwanted content 10m30s.
• A GitHub repository is available with the code and manual for the workshop, and participants can try out the workshop and provide feedback 12m10s.

Building with Copilot and Frameworks

• The discussion is about building agents with Copilot and there is an opportunity to ask questions about the topic, specifically about MTP or building agents in general 10s.
• A question was raised about why use a framework instead of building from scratch, and it was suggested that building from scratch is not necessary and that frameworks are there to speed up the development process 2m6s.
• The reliability of MCP servers was questioned, but it was stated that they are generally reliable, and any issues are often due to too much context being sent to the LLM, which can be handled with proper context engineering and the use of middleware 2m6s.

Middleware and Context Management

• The use of middleware can help to summarize context and prevent context overload, which can slow down agents, and this is a common issue with all agents, not just those using MCP 2m6s.
• A comment was made about the GitHub mobile app lacking important functionalities and needing to be improved, but this was not directly related to building agents with Copilot 2m6s.

MCP Server and Context Provisioning

• The context for an agent using MCP comes from the MCP server, which can be a custom-built server, such as the cupcake server example, and provides the necessary context for the agent to function 10s.
• A Python script is used to build agents with pre-designed prompts or tools that return specific information, and the server creator determines the information provided to the agent. The context depends on the design of the MCP server. 10s

Middleware Functionality and Content Filtering

• Middleware functions are added to agents, running every time a query is received by the LLM. Content safety middleware filters prompts, preventing those that violate content safety requirements from reaching the LLM. This acts as a filter rather than a sandbox. 42s

Stream Logistics and Technical Issues

• The discussion included a mention of the LinkedIn app, with a participant named Alex joining towards the end of a challenging stream. The stream was noted to be disorganized, with technical issues such as a non-functioning microphone and screen sharing problems. 2m6s
• A walkthrough repository link was shared for setting up an agent with Claude, requiring a special account to access Claude models from the Azure marketplace. 2m6s

Rubber Ducking and Code Review Processes

• The concept of "rubber ducking" involves using a second model from a different AI family to review agents. The discussion touched on when to manually trigger critique versus relying on automatic checkpoints, with a preference for trusting automatic rubber ducking using Copilot CLI. 2m6s
• An automatic review process is in place where a model receives an immediate review when there are significant code changes, potentially involving another model family for review. This process typically works well with the automatic system. 10s
• The command-line interface (CLI) is generally trusted to determine when to involve a second model family, although manual triggering is possible, including using a fleet option for automatic execution. 10s
• Rubber ducking is recommended for making significant changes, while it is not as beneficial for smaller changes. 42s

Closing Remarks and Audience Interaction

• The session concluded with an apology for any disorganization and an invitation for questions in the chat, with a promise of improved sessions in the future. Participants are encouraged to try out Copilot and the workshop. 2m6s