Open Source Friday: Mozilla AI | Project cq

Introduction to Peter Wilson and Project CQ

• Peter Wilson, a staff engineer at Mozilla AI, is introduced as the guest to discuss Project CQ, and he shares a brief introduction about himself, including his background in software engineering, which spans over 20 years, and his experience working at Hashi Corp before joining Mozilla AI 2m6s.
• Peter Wilson's journey into open source began when he joined Hashi Corp, which had a strong focus on open-source principles and worked on various open-source projects, including libraries in the Go ecosystem, and he started contributing to open-source projects in a more significant way during this time 4m42s.
• The discussion touches on Peter Wilson's first open-source contribution, and although he doesn't recall the specifics, he mentions that it was likely a small documentation pull request, and he is encouraged to dig up more information about his early contributions 6m15s.
• The conversation shifts to Project CQ, and Peter Wilson explains that the project can be thought of as a "Stack Overflow for agents," making it easier to understand, and he is about to delve into the problem space and the motivations behind building Project CQ 9m30s.

The Problem Space and Motivation Behind Project CQ

• The idea of Project CQ is to address the problems that arise when using coding agents, such as the accumulation of huge files and the expectation that the language model can focus on all the information, which does not always work 10s.
• The project aims to enable coding agents to share what they have learned, so they do not always have to encounter the same problems, and to make the information more accessible on demand 2m6s.

Project CQ's Plugin and Functionality

• A plugin has been created as part of the project, which works with open code, Claude, or some IDEs, and installs into cursor and wind surf, allowing it to query a local SQL-like database or a remote server to save information 4m6s.
• The plugin has a skill that tells it how to perform its tasks and runs an MCP server underneath the hood, and before starting a task, it queries CQ to see if it knows anything about the task beforehand 5m10s.
• The project is designed to store general "gotchas" or common problems that developers encounter, such as issues with open-source libraries or documentation, and proposes solutions when it finds them 6m20s.

Project CQ's Governance and Remote Configuration

• The project has an approval process for remote configurations, and once information is approved, it becomes available to anyone else who queries it 7m30s.
• The project has been built over the course of about two months, and it includes a Go SDK, a Python SDK, a CLI written in Go, a plugin, and an open-source server that can be hosted locally or among a team 9m10s.

Project CQ's Development and Team Structure

• The project currently has a small team of people working on it, with one person spending most of their time on it, and a few others contributing to different aspects of the project, as well as some open-source contributions 11m20s.
• A hosted version of the project is being built and is expected to be officially released soon 12m40s.

Project CQ's Architecture and Knowledge Unit Schema

• The project aims to provide a service where people can have their own private namespace and push their own content, with the goal of making it free for users, and features will be built on top of this service 10s.
• The architecture of the project involves a memory system with a published schema in the open source repository, which defines a "knowledge unit" that contains information about a proposed solution, including applicable domains, frameworks, and languages 2m6s.
• The knowledge unit is proposed by an agent, validated, and then confirmed if it works, with the goal of building a signal over time of how often something gets confirmed and by how many different types of agents 2m6s.
• The schema is intentionally light and flexible, with the idea of building features in the backend and then proposing them to the schema once they are proven to be useful, and there are conversations about creating a working group to standardize this type of system 4m30s.

Scoring, Accuracy, and Standardization of Knowledge Units

• The scoring and accuracy of the memory units are calculated in the backend, leaving room for flexibility in implementation, and there are plans to potentially create a standard around this type of system through a working group 6m40s.
• The goal is to strike a balance between a protocol and a platform, without forcing too much on users, and to gain insight into what is useful and not useful for people, which will drive the development of the project 10s.

Use Cases and Traction of Project CQ

• Initial traction and usage are mostly found in areas where there are "gotchas", such as dealing with code review, interacting with GitHub's API, and refactoring, which are the glue between things where friction normally occurs 2m6s.
• The concept of a unit of memory was chosen as the thing to solve specifically because the training data that models use is often slightly behind, and this approach aims to provide the smallest amount of information to make the user happy when using these tools daily 4m42s.

Integration and SDKs for Project CQ

• The project has had conversations with folks who are doing different things, and there are SDKs available, such as Go and Python SDKs, that allow people to easily hook into the system and build it into their apps, enabling them to emit prompts and handle calls to save data 8m10s.
• The project is designed to be flexible and allow users to integrate it with other open-source memory frameworks, and it can be used in various ways, such as in a coding CLI type tool or built into an app with an agentic loop 10m0s.
• The integration of a product into another product being built is desired, as it could enhance the process and allow for quicker emission of content, with the Command Line Interface (CLI) being available for use and capable of performing tasks similar to the SDK 10s.

Local-First Design and Shared Memory Concepts

• The framework for the product started with a local-first approach, which was a natural progression rather than a planned strategy, and the goal was to fix a specific issue and then expand it to be shared across teams or workflows 2m6s.
• The local-first mindset is becoming increasingly popular, as it allows for faster and easier building, as well as reducing the need for inference when testing applications, and the idea of shared memory and applications is powerful and important 4m6s.
• The concept of shared memory and scaling it to become shared is an interesting construct, and the product being discussed is related to other projects, such as LLaMA and libraries in Python, which are designed to allow people to work without being tied to specific things 6m6s.

Project CQ Demo and Plugin Installation

• A demo of the product was planned, but it was decided to pause and walk through it instead, with the possibility of showcasing it at a later time, and the product is part of a larger effort to create tools that can be used by people without requiring significant expertise 8m6s.
• The demo of Mozilla AI's Project CQ is being presented, and it starts with installing a plugin from the cloud plug-in marketplace, specifically the Mozilla CQ plugin, which is already installed on disk 2m6s.
• The plugin installation is followed by setting environment variables for the agent to connect to a remote, including a CQ address and an API key, which can be found in the GitHub repo's readme 4m30s.
• The presenter then tests the plugin by asking it to add a GitHub action CI workflow, and the plugin requests permission to use the CQ skill and query, which is granted 6m15s.

Plugin Limitations and Real-World Examples

• The plugin is then expected to generate code for the GitHub action, but it may not always produce the most up-to-date or accurate results due to the rapid pace of changes in the field and the training data not keeping up 8m40s.
• The presenter mentions that they have seen the plugin confidently generate code that uses outdated versions of certain actions, such as the checkout action, which can be problematic 10m20s.
• A colleague, Devday, is mentioned as having a good example of the plugin's limitations, where it struggled to update its own configuration for an MCP server, taking several attempts to get it right 12m10s.

Executing Commands and Reviewing Plugin Output

• The presenter decides to let the plugin run and generate code, and then review the results, acknowledging that it may not always produce perfect outcomes 14m0s.
• The discussion begins with an examination of a diff, and the decision to run a command, with the expectation of seeing what it recommends and when to run certain things 10s.
• The command is executed, and it is noted that the versions of GitHub actions used are out of date, with the current version being 6.0.2, whereas the used version is 4 2m6s.
• A message is displayed indicating that the versions of GitHub actions used are a major version out of date, and a suggestion is made to check the latest release, which is version 6 4m6s.
• The command is updated to use a more up-to-date version, and it is decided to let it run, with the option to use a plugin, and a live demo is performed 6m6s.

System Interactions and Knowledge Unit Proposals

• The demo is successful, and it is noted that everything lines up as expected, although there is a minor issue that is not fully understood 8m6s.
• The CQ status is checked, and it is found that the knowledge unit is stored locally, with an API key that is not useful, causing it to fall back to an offline local cache 10m6s.
• The CQ status is reset, and a new session is started, with the expectation that it will not make any mistakes, and the command is run again to query CQ 12m6s.
• The process involves a query finding information to save time, and it has already found some relevant data, such as versions six and eight, to verify the current major version before checking the actual latest versions 10s.
• The system is interacting with GitHub and Anthropic, and it is generating an API key in the background, which will be explained later, and it is also syncing and running various tasks 2m6s.
• The system is learning and confirming information, and it has learned something new, which is interesting, and it is going to propose a new knowledge unit, and it should give a little summary 4m30s.
• The key action that proposes a new knowledge unit is based on the skill, which is the wording of the skill that leaves it up to the interpretation of the Large Language Model (LM), and there are examples in the open-source repo of when to propose something, such as discovering undocumented API behavior or finding an unobvious workaround 8m40s.

Vibe Check and Content Validation Process

• The system has a section in the skill that provides criteria and examples of when to propose a new knowledge unit, and there is also a blog post that was recently published with a co-contributor that provides more information on this topic 12m10s.
• A blog post was written by Daniel and Lauren Mushro to integrate a "vibe check" into Project CQ, which is a five-stage check to ensure that proposed content does not include personally identifiable information (PII) and can be found on Mozilla's blog 10s.
• The vibe check is used to validate content and flag any incorrect or stale information, which affects the confidence scoring and relevance scoring, and this process can be reviewed and approved by a human before it is shown to other agents 2m6s.

Remote Configuration and Human-in-the-Loop Review

• When Project CQ is configured with a remote setup, it allows for a human-in-the-loop stage to vet content, and this setup can be used to remove or update content in the future, with the goal of having different heuristics to determine when content needs to be re-reviewed 4m30s.
• The Project CQ system can be set up to work with a remote server, which allows for a review process and the ability to approve or reject content before it is shared with other agents, and this process can be managed through a hosted platform 6m10s.

Plugins and CQ Reflect Feature

• The system uses plugins, including the CQ plugin and an MCP server, which can be installed and configured to work with the Project CQ system, and these plugins can be used to update and manage content 8m20s.
• The CQ reflect feature is a skill-driven function that allows users to review their session, identify problems, and propose solutions, which can be edited or selected for further action 10s.

Model Performance and Skill Adjustments

• The effectiveness of different models with CQ is uncertain, as it seems that models like Claude, Codeex, and Gemini have moments of clarity and confusion, making it difficult to determine if one model is better than another 2m6s.
• The skill used with CQ has undergone tweaks to improve its performance, including adjustments to when it should be triggered, and it appears to be working better now, especially with Open Code and Codeex 4m6s.

Remote CQ Instance and Review Process

• The CQ remote instance can be accessed by logging in with GitHub, allowing users to view pending reviews, known as KUs, and see the recommended actions for each item 8m6s.
• The inbox-type view in the CQ remote instance provides a clear overview of the items under review, including explanations of the issues and proposed actions, which can be confirmed or flagged for further review 10m6s.
• The human in the loop stage is a crucial part of the process, where individuals review and approve content to ensure it does not contain personal data or information that could trick a language model, and this stage is necessary before confirming and making the content available to agents 10s.

Governance, Policies, and Multi-Tenancy

• Governance and policies for access to certain information are currently being developed, with a mini internal road map in place, and the goal is to introduce multi-tenency for organizations, which will involve more policy-related aspects 2m6s.
• The approval process currently allows agents with access to read approved content, and a commons name space will be introduced, which will be available to every agent and will combine relevant knowledge from various sources 2m6s.
• The system will eventually include nomination and graduation processes, and currently, agents can query content using a valid API key, allowing them to access approved content 4m30s.

API Key Management and Access Control

• API keys can be created and managed, with options to set a time-to-live (TTL) before the key is revoked, and keys can be revoked at any time to prevent agents from accessing content 6m40s.
• The management of API keys and agent access is a platform-level feature, allowing users to delegate authority to agents and manage their access to content 8m20s.

Engineering Team Management and Security

• The discussion revolves around the management of engineering teams, particularly in the context of DevOps teams, and the need for control over access to certain knowledge units, with security notes and recommendations to be published soon 10s.
• The project involves a double-blind review process, where team members are nominated as reviewers, and automated guardrail pipelines are being implemented to vet certain things beforehand, with the goal of validating shared context and managing access 42s.

Developer Flow and Context Capture

• The demo showcases the ability to capture developer intent with agent reasoning and control, logging the context of changes and updates to the knowledge unit, which is an interesting aspect of the new developer flow 2m6s.
• The new developer flow involves not just the code itself, but also the attached information, such as the intent and context, which needs to be encapsulated in a block before being pushed into the agent 2m6s.

Project Roadmap and Future Developments

• The project, referred to as "cq", involves querying and managing knowledge units, with a command-line interface that allows users to interact with the system, and a full-fledged roadmap is being planned, with exciting developments on the horizon 4m30s.
• The team is working on unlocking memory with these agents, and there are plans to publish more information and recommendations soon, with a focus on security, access control, and automated pipelines 6m10s.

Mozilla AI's Principles and Platform Design

• Mozilla AI is focused on principles and values such as choice and privacy, and is working on a platform where users own the knowledge units (KUs) they create, can export them, and have control over their private namespace 10s.
• The platform is designed to allow users to opt-in to features, and the team is considering how to provide functionality for deduplication of KUs, such as merging and combining similar KUs, and generating new ones that supersede existing ones 2m6s.
• The team is exploring features such as cross-namespace scanning, where they could identify similar KUs across different namespaces and create a single, superseding KU that can be shared in the commons, making it available to everyone 4m30s.
• The platform currently uses SQLite for local storage and Postgress for the open-source server, and the team is testing and implementing various technologies, including semantic search, to support the platform's functionality and scalability 6m10s.
• The goal is to provide a platform that allows users to distill relevant knowledge and share it with others, while maintaining control over their private data and namespace, and the team is working to develop features that balance user needs with the principles of choice and privacy 8m0s.

Long-Term Vision and Knowledge Commons

• The project's goal is to create a commons of knowledge that is available to everyone, and it will be interesting to see how knowledge comes and goes over time, and what sort of knowledge becomes relevant, with potential applications in academic fields 10s.
• The project's underlying concepts are related to cognition, memory, and psychology, and understanding how memory is managed and how the brain works, which could lead to interesting discoveries 1m42s.

Community Involvement and Contribution

• Community engagement is crucial for the project, and people can get involved by contributing to the open-source repository, which has a development guide and a contributing guide to help get started, and by filing issues or helping with development 4m6s.
• There are many ways to contribute to the project, including working on GitHub issues, deploying self-hosted versions, and helping with specific tasks such as PostgreSQL integration, and contributors can open RFCs in the discussions part of the repository to collaborate with the team 6m6s.
• People can find the project repository and blog post, and the team can be reached through various social media platforms, including Mastodon and LinkedIn, under the handle Pitky22, and the team is open to feedback and collaboration 10m6s.

Closing Thoughts and Encouragement for Feedback

• The project is led by a team of people, and anyone who gets in touch with the team will be able to provide useful information and support, and the team is grateful for any involvement, whether it's contributing to the project or just using it and providing feedback 12m6s.
• The goal is to inspire people to try new things and be more open to AI-assisted engineering, considering both its pros and cons, and to encourage feedback on the project 10s.
• The discussion aims to educate and inform people about the project, and it is hoped that viewers will learn something new and gain a better understanding of what knowledge will look like on agents 2m6s.
• The conversation has been interesting and exciting, and the participants are grateful for the opportunity to talk about the project and share their ideas with the audience, with plans to continue the discussion in future episodes of Open Source Fridays 2m6s.