Rubber Duck Thursdays: Building Agents with Copilot

Introduction and Audience Participation

• The stream begins with a greeting and a mention of the current time in London, which is 11:30 AM. Participants from various countries, including Australia, Italy, India, Nigeria, the Netherlands, Turkey, the United States, Greece, Germany, and the UK, are acknowledged. 42s
• The topic of the stream is building AI agents, a subject of frequent discussion. The host expresses enthusiasm for the topic and interacts with viewers, asking about their current projects. 1m30s

Participants' Projects and Tools

• The host shares their screen to display VS Code, expressing a preference for this coding environment. Participants are encouraged to share what they are building. 2m6s
• Enrique is working on an AI OCR app, James is developing a marketing engine, and Zara is building an EHR platform. The host inquires about the nature of an EHR platform. 3m0s
• Mohan is working on AI agents using Python, and the host, identifying as a Python enthusiast, expresses support for Python projects. James is also creating a compliance application for pest control, and the host comments on the effectiveness of niche AI agents. 4m0s
• Sergio mentions building out a Lido beach, which is noted as an interesting project. 4m30s
• Vivek is building a PC system from scratch, and Dimitri is building an app with Django, a framework that is well-liked for its capabilities, depending on the project 10s.

Security and Guardrails in AI Agents

• Gabriel is working on building AI agents with Copilot, which is a significant advancement in terms of speed, but he is concerned about enforcing a zero-trust security context to prevent vulnerabilities 2m6s.
• Security is a major issue with AI, including compliance, and guardrails are essential to prevent agents from doing things that are not intended, with many people having found that security is a big concern 2m6s.
• A workshop was held with the Claude team, where participants learned how to build AI agents with Claude models, and the presentation from the workshop will be shared later, possibly on GitHub 4m30s.
• The partnership between Microsoft, GitHub, and Anthropic has led to increased use of Claude models, and people are now looking to move beyond just building AI agents for fun and into production, where security and safety are crucial 6m20s.
• The goal is to ensure that AI agents are not doing things that are not wanted, and understanding how to build agents with guardrails is essential to keeping users safe 8m40s.

Workshops and Frameworks for Building AI Agents

• A link to a GitHub repository for a workshop on building AI agents from scratch using Microsoft's agent framework will be shared, and the workshop is based on an Anthropic workshop that was recently given, showcasing how to build an AI agent using the framework 10s.
• The agent being built has content safety middleware added to it, which acts as a filter around the Large Language Model (LLM) to detect and prevent certain types of responses, and this middleware can be defined to filter for specific things and set to different levels of strictness 2m6s.
• When building an AI agent, especially with frameworks like LangChain or Microsoft's framework, it's possible to connect to a deployment of an LLM, such as GPT 5 mini, and add the content safety middleware to it, allowing for more control over the agent's responses 2m6s.
• An example of a simpler agent built with LangChain will be shown, demonstrating how the agent can be run and interacted with, and the importance of activating the correct environment for the agent to function properly 4m30s.

Audience Interaction and Technical Setup

• The conversation will also involve interacting with the audience, discussing how people are handling certain topics, and responding to questions and comments from viewers, including those from different locations such as Germany and New York 8m30s.
• Gabriel highlighted that security context is a significant challenge for developers, particularly in containing agents and preventing vulnerable code. Gabriel shared security prompts in different languages to assist with this issue 10s.
• Sagar inquired about whether to rely on model providers' built-in safety features or to add custom middleware and policy layers. It was suggested that for enterprise applications, using platforms like Azure with built-in content safety is beneficial, but custom middleware can also be integrated 42s.
• There were technical difficulties with screen sharing, which were eventually resolved, allowing for the demonstration of coding activities and the setup of virtual environments 2m6s.

Technical Setup and Environment Configuration

• The process involved navigating directories and attempting to activate the correct virtual environment to install necessary dependencies for building agents. There was a focus on ensuring the correct environment was activated to demonstrate middleware installation 2m6s.
• The installation process included using commands like pip install and UV sync to set up the Python environment and dependencies required for the project 2m6s.

Building an AI Agent with LangChain and Middleware

• The process of building a LangChain agent involves importing several modules, including the create agent module from LangChain, chat open AI, and middleware, which allows for the definition of a model and the implementation of content safety guardrails 10s.
• The agent is defined by giving it an endpoint, such as an Azure endpoint, and specifying the model to be used, in this case, GPT-5 mini, as well as setting up content safety guardrails to filter out negative words and setting a threshold for strictness 42s.
• The middleware function is a special feature that can be imported from LangChain or used outside of it, and it enables the definition of categories to filter for and the setting of a threshold for strictness 1m6s.

Technical Challenges and Debugging

• The agent is then started, but technical issues arise, including the realization that a necessary module, prompt-to-you, has not been installed, and the decision is made to try running the agent without middleware 2m6s.
• Due to ongoing technical issues, including audio and screen sharing problems, the decision is made to seek help from Copilot to debug the code and get the agent running 4m30s.
• The technical issues persist, including audio loss and screen sharing problems, causing disruptions to the stream and making it difficult to demonstrate the building of the AI agent and the use of content safety for moderation 6m0s.

Stream Quality and Technical Issues

• The stream is experiencing technical difficulties, with the screen not displaying properly, and the quality of the stream is poor, with the host apologizing for the issues 10s.

Workshop Overview and Microsoft-Claude Partnership

• The host was speaking at the "Code with Claude" conference the previous day, where they gave a workshop on building with Microsoft Agent Framework and Claude models, and they have a fork of the workshop repository on their own GitHub 2m6s.
• Microsoft has created a partnership with the Claude team, making the Claude models available on Azure, and allowing users to build agents with Copilot and plug them into Foundry 5m42s.
• The workshop provides a full walk-through of how to deploy a model in Foundry, including choosing which model to use, such as Sonnet 4.6, and how to get started with the Microsoft Agent Framework 8m15s.
• The workshop also covers how to use middleware to filter out malicious prompts and ensure content safety when building agents 15m30s.

Practical Example: Cupcake Ordering with MCP

• The host recommends trying out the workshop, which is available on GitHub, and allows users to test out building an agent with Microsoft Agent Framework and Claude models, including a fun example of ordering a cupcake using an MCP server 12m0s.
• The workshop demonstrated how to connect to a cupcake MCP server, which provides the necessary context for ordering a cupcake through an agent. The process involves greeting the agent, receiving a response, and then ordering the cupcake via the agent using the defined MCP tool and URL. 10s

Frameworks vs. Building from Scratch

• There is a discussion on whether to use frameworks or build from scratch when developing agents. It is suggested that using frameworks can speed up the process by allowing developers to plug in pre-built code, rather than building components like an MCP client and server from scratch. 1m6s
• MCP servers are considered reliable, and issues with them are often due to context overload rather than the servers themselves. Proper context engineering and using middleware to summarize context can help manage this. Agents and LLMs perform better without context overload, and tools like GitHub Copilot automatically compact context to improve performance. 2m6s

MCP Servers and Context Management

• The context for agents using MCP comes from the MCP server. In the example provided, a Microsoft team built an MCP server specifically for a cupcake ordering service. 3m6s
• The agent is built using a Python script that utilizes pre-designed prompts or tools to retrieve specific information, and the server creator decides what information is shared with the agent 10s.

Middleware and Content Safety

• Middleware is a function added to the agent that runs every time the LLM receives a query, and content safety middleware filters out bad content from prompts before they reach the LLM 2m6s.
• The content safety filter stops queries that violate requirements at the middleware level, preventing them from reaching the LLM, and this is considered a useful option 2m6s.
• Middleware is likened to a filter, rather than a sandbox, as it does not create a container but instead filters out unwanted content 8m30s.

Accessing Claude Models and Rubber Ducking

• To build an agent with Claude, a special type of account and access to Claude models from the Azure marketplace are required, and a walk-through repository is available to guide users through the process 10m40s.
• Rubber ducking involves bringing in a second model from a different AI family to review agents, and the decision to manually trigger critique versus trusting automatic checkpoints depends on the specific use case 12m10s.
• The automatic rubber ducking feature in Copilot CLI can be trusted to review agents, but there may be cases where manual triggering or skipping the critique is necessary 12m10s.
• The automatic review process involves the model receiving an immediate review when there are sufficient code changes, and if necessary, a second model family is brought in for review 0s.
• The CLI is typically trusted to recognize when a second model family should be brought in, but it is also possible to manually trigger this or use the fleet option to automate the process 10s.
• Rubber ducking is recommended for larger code changes, as it can be beneficial in these situations, but it is not as necessary for smaller changes 1m30s.

Conclusion and Next Steps

• The stream was concluded due to time constraints, with an apology for the disorganized nature of the discussion and an invitation for viewers to leave questions in the chat for later response 2m6s.
• The next stream is anticipated to be an improvement, with the host having had more rest and preparation, and viewers are encouraged to try out Co-pilot and the workshop until then 2m40s.