Now Anyone Can Code: How AI Agents Can Build Your Whole App

Coming up 0s

• The Mac brought personal computing to the masses in 1984, and now in 2024, personal software is available, allowing users to orchestrate a giant army of agents 8s.
• This new ability is compared to Mickey Mouse's magical experience in Fantasia, where he learns to control a menagerie of objects, illustrating the potential of personal software to build whatever users want 22s.
• With personal software, users can bring their ideas to life quickly, as seen in an example where someone built their 15-year-old idea in just 15 minutes and recorded their emotional reaction 34s.

Intro 47s

• The hosts of the video are Gary, Jared Harge, and Diana, who have collectively funded companies worth hundreds of billions of dollars. 49s
• They are joined by Amjad, one of their best alumni, who has just launched a product called Repet Agent. 1m1s
• Repet Agent is currently in Early Access, meaning it is still in the beta stage of development, and while it has generated excitement, it still contains many bugs. 1m13s
• Despite its early stage, a live demo of Repet Agent will be shown, and Amjad plans to build an application using the product during the demonstration. 1m27s

Making an app with Replit 1m29s

• A personal app is being created to track morning mood correlated with activities done the previous day, such as coffee consumption, alcohol intake, and exercise, with the goal of logging mood and activities in the morning and sending the data to an agent 1m30s.
• The app is built using a chat interface where the agent reads the message and thinks about the next steps, similar to a multiplayer experience on Replit 2m6s.
• The agent creates a plan for the app, suggesting features such as visualization, reminders, and a database connection, and picks a tech stack including Flask, vanilla JS, and Postgres 2m28s.
• The progress pane shows the AI installing packages, writing code, and building a database connection, making it easy for new software engineers to get started without worrying about dependencies and packages 2m58s.
• The app is built with a backend, Postgres, and can be deployed, allowing users to log their mood, view history, and rate the app 3m41s.
• The agent tests the app on its own, taking a screenshot and using computer vision to check if something is presented, but also asks for human testing and QA 4m3s.
• The models used are multimodal, including Claude Sonnet 3.5, GP4, and in-house models like a fast embedding model and retrieval system, which are important for making the agents work 4m23s.
• The retrieval system is a key part of the agent's functionality, allowing it to find the right places to edit in the code, and is a notable achievement 4m57s.
• The agent can create data and deploy the app, making it possible to go from an idea to a deployed web app that anyone can access in a short amount of time 5m56s.
• The idea of personal software is exciting, allowing anyone to create apps that are tailored to their individual needs 6m19s.

Feel the AGI, personal software era 6m23s

• The concept of personal software has emerged in 2024, similar to how Mac brought personal computing to the masses, with the potential to revolutionize the way people interact with technology 6m23s.
• Karpathy tweeted about Repet Agent, describing it as a "feel the AGI moment," which refers to the experience of using artificial general intelligence (AGI) to build software 6m31s.
• Using Repet Agent, a Hacker News clone was created, and the experience felt like having a development partner, with the AI agent asking questions and making design decisions on its own 6m41s.
• The AI agent demonstrated good intuition about what to build and how to design it, such as creating a slider bar with emojis without being explicitly instructed to do so 6m50s.
• The AI agent got stuck at one point, but was able to ask for help and continue working on the project after receiving guidance 7m15s.
• The experience of working with the AI agent felt like collaborating with a developer, and it is suggested that having different modes or personalities for the AI agent could be useful, such as a "grumpy programmer" or an "over engineer" mode 7m37s.
• The idea of having a toggle to switch between different modes or personalities for the AI agent is proposed, but it is unclear if this feature is currently functional 7m54s.

Having AI code the way humans do 8m7s

• AI programmers do not possess super intelligence that can build an entire app perfectly from start to finish without making mistakes; instead, they code in a way similar to humans, writing code, trying it out, and fixing bugs as needed 8m7s.
• The design decision behind AI programmers is to have them act as coworkers, allowing users to close the AI's code and fix it themselves if needed 8m28s.
• The goal is for non-coders to learn a little bit of coding along the way as they work with the AI agent, similar to how people in the past learned to code by making small edits to their Myspace page or other online projects 8m46s.
• There is a need to revive the incremental learning scale, where people can learn to code through fun, side projects, rather than requiring a computer science degree or boot camp 9m6s.
• Fully automated software engineering agents are still far from being developed, and people should still learn how to code, although they will have to do less coding and focus more on reading and debugging code 9m29s.
• AI agents can get users fairly far in the coding process, but sometimes they will get stuck, and users will need to go into the code and figure it out themselves 9m41s.

You should still learn to code! 9m51s

• Many young people, such as freshmen, believe that with the advancement of technology, they no longer need to study how to code, but this is not true, and knowing how to code is more important and powerful than ever before 9m51s.
• Having the ability to code will allow individuals to orchestrate and leverage the power of AI agents, making them more powerful and able to build whatever they want, whenever they want 10m16s.
• The return on learning to code is increasing rapidly, with the usefulness of knowing how to code doubling every six months, similar to Moore's Law 10m46s.
• In 2020, knowing how to code was not that useful, as individuals would still get blocked by deployment and configuration issues, but with the help of chat in 2023, knowing how to code a little bit can get individuals fairly far 11m1s.
• By 2024, knowing how to code a little bit is massive leverage, thanks to the availability of AI agents and tools like Cursor, which can help individuals extend their abilities and get super far 11m20s.
• Programmers are on a massive trajectory of increased power, with their abilities and leverage continuing to grow rapidly 11m36s.

The underlying tech 11m42s

• The technology behind the system is a multi-agent system with a core React-like loop, utilizing a Chain of Thought type prompting that has been around for a couple of years, and most agents are built on this concept 11m45s.
• The system is also a multi-agent system, providing it with a ton of tools using tool calling, and these tools are the same ones exposed to people, requiring careful consideration of how to expose these tools and how the agent sees them 12m9s.
• The edit tool returns errors from the language server, a Python language server, which provides feedback to the agent as it codes, similar to how a human coder would receive feedback 12m29s.
• The agent gets feedback from the language server for any action, allowing it to react to that feedback, and it has access to various tools such as package management, editing, deployment, and database management 12m44s.
• To prevent the agent from going off the rails, there are mechanisms in place, including a reflection loop that constantly evaluates whether the agent is doing the right thing, and the use of L chain tools like Lang graph and Lsmith 13m22s.
• Lang graph is a tool that allows for building agent dags, and Lsmith is used for debugging, providing a way to visualize the graph and look at the traces for dags 13m29s.
• Retrieval is crucial for the system, requiring a neuro-symbolic approach that can do RAG-style embeddings retrieval and look up functions and symbols inside the code 14m5s.
• Even with large context windows, specialized tools will still be needed for lookups, and context management is necessary to prevent the model from biasing towards certain information 14m33s.
• The agent uses a memory bank to store information from each step, and it needs to be able to pick the right memories and put them in context for each subsequent step 15m9s.
• Memory management is crucial when working with AI agents, as it's essential to pick the right memories for the right tasks and not put the entire memory in context, which can be fragile and not great at following instructions 15m44s.
• The idea of situational awareness and the sci-fi argument that AGI will kill us tomorrow is rebutted by the fact that simply scaling up parameters and using more GPUs will not work, and there is utility in having agents work together and being smart about intermediate representation 16m6s.
• Building a system with AI agents can be humbling and sets expectations about AI progress in a different way, as the systems are fragile and not great at following instructions, and people often talk about the hallucination problem, but the bigger problem is getting them to follow orders 16m52s.
• The challenge of getting AI agents to follow instructions and do the right thing is significant, and it's hard to get them to actually do what is intended 17m12s.

The path to AGI 17m19s

• The path to Artificial General Intelligence (AGI) may lead to "functional AGI," which involves automating economically useful tasks, and this goal is considered fairly within reach as it can be approached as a brute force problem 17m20s.
• Achieving functional AGI might require building and fine-tuning orchestrations of groups of agents for each task, similar to what has been done for programming, and eventually combining them into one model 17m45s.
• The history of machine learning has shown that systems are created and grown around models, but eventually, the model replaces those systems, and hopefully, this will lead to an end-to-end machine learning system that can perform various tasks 18m0s.
• The example of Tesla's development is mentioned, where they moved from using logic and other systems to end-to-end training, and it is expected that eventually, a similar end-to-end system will be developed for other tasks 18m17s.
• However, functional AGI would not be considered "true AGI" because it would not be able to handle tasks outside of its training data, and true AGI would require efficient learning and the ability to navigate new environments 18m32s.
• True AGI would need to be able to learn skills required to navigate new environments with no prior information, and current systems, such as LMS, are not efficient learners and require additional layers, such as symbolic representation, to work effectively 18m50s.
• The use of symbolic representation and classical computer science concepts, such as backtracking and Turing completeness, are specialized and not generalized, and while incredibly useful, they are not a sign of true AGI 19m20s.

What users made with Replit 19m41s

• People have already created impressive and interesting projects with a new tool, including a personal app that allows users to put memories on a map and attach files and audio files, which was built in 15 minutes by someone who had the idea for 15 years but didn't have the tools to build it 19m53s.
• Another user, Meck, built a Stripe coupon tool in 5-10 minutes, which would be difficult to build with no-code tools and would likely require using multiple tools like Bubble and Zapier 20m36s.
• The new tool is seeing a lot of traction, which is a challenge for no-code tools that often start as no-code but then find that users want to build more complex projects that push the limits of what the tool can do 21m23s.
• The new tool allows no-code users to switch to coding by initially using prompts and then gradually becoming programmers by editing the code 21m44s.
• The tool can be used to build complex projects, such as a recruiting CRM with role-based permissions, which would normally be a $10,000 a month Enterprise feature 22m8s.
• The tool has been used to build projects much faster than traditional methods, including generating an app in 10 minutes that took 18 months to build and building an app in an hour that took a year to build 22m42s.
• The current AI system can save millions of dollars in human hours, but it's still in its early stages and can't be applied to existing coding stacks yet 23m7s.
• A retrieval system has been built to index codebases quickly and provide intelligence about the codebase, with features like summaries of files and projects created using large language models (LLMs) 23m24s.
• The next step is to add more autonomy to the system, allowing it to work in the background, fork projects, and send pull requests or report problems when encountered 23m54s.
• The vision for the system includes a bounties program where users can submit problems or projects they want to build, and the community can help fix them for a price 24m21s.
• The system can also summon a human expert, known as a "bounty hunter," to help with problems the AI agent can't solve, using a real-time market to find an expert for a set price 24m53s.
• The idea is to create a human-machine symbiosis, where humans and AI agents work together as part of a greater intelligence orchestration system, with humans being another agent in the system 25m18s.
• This approach is inspired by the concept of human-machine symbiosis, which emphasizes the importance of computers being extensions of humans rather than competitors 25m25s.
• The ultimate goal is to create a system where humans and AI agents collaborate seamlessly, with humans being able to prompt the agent or edit the code themselves 25m11s.

Challenges in resetting the org 25m56s

• A company had a significant moment earlier in the year with a demo that impressed many, which was the result of hard work on remaking the way software is deployed and written. 25m57s
• The company had previously raised a large round of funding and felt the need to grow, but this led to a layoff and a reset of the organization. 26m18s
• The company was initially very lean, with only four or five employees for many years, despite having millions of users. 26m52s
• The decision to grow and hire more people, including executives, led to a more complex management structure, which ultimately became miserable and unproductive. 27m1s
• The company has since flattened its organization, eliminating multiple layers and meetings, and now focuses on only a few key projects. 27m50s
• The founder is involved in all of these projects and believes that the company has become more productive as a result of getting smaller. 27m55s
• The founder notes that the temptation to add more bureaucracy and management layers can be strong, especially when there are many ideas and resources available. 28m33s
• The company is trying to stay disciplined and focused on a few key projects, rather than trying to do too many things at once. 28m55s
• The concept of the "compound startup" is mentioned, where multiple product lines are treated as separate startups, each with their own governance and decision-making processes. 29m7s
• Parker Conrad, the founder of Rippling, has a unique hiring tactic where he hires former founders and puts them in charge of a product line, which has worked well for the company, but may be challenging for others to replicate due to the difficulty in hiring high-quality former founders unless the company is already successful or has a top-tier recruiter 29m25s.
• Parker Conrad also emphasizes the importance of staying connected to customers by answering customer support tickets, which provides a direct line of information on what's really going on with the customer 30m10s.
• The development of an AI agent involved building a new technology that the team wasn't used to working on, and it required a big effort to pull it off organizationally 30m33s.
• The AI agent was built by a task force consisting of people from different teams, including the IDE team, devx team, uxx and design team, and the AI team, which was at the center and connected to all the other teams 31m13s.
• The task force was organized similarly to a Cara diagram, with the AI team as the kernel OS and the other teams creating tools that connected to it 31m34s.
• The product team worked on the entry points and structure of the AI agent, which was a challenging task that required frequent meetings and rapid progress 31m55s.
• The development process involved regular meetings, including a war room meeting on Mondays and an agent salon on Fridays, where the team would review progress, prioritize tasks, and make changes to the product 32m5s.
• Doing a "run" with the AI agent meant literally testing it and reviewing its performance to identify what was working and what was broken 32m36s.
• The team went through the product, identified where it broke, and determined the priorities to fix the issues that arose during the process 32m40s.
• Each team member built their own agent, with some teams requiring this due to the specific needs of their tasks, such as the ID team creating a screenshot agent 32m47s.
• The ID team developed the screenshot agent, which utilized AI to analyze screenshots, generate thoughts, and return them to the main manager agent 32m56s.
• The package management team built a text stack setup type of configuration, which was a unique and effective approach 33m7s.
• The overall structure and organization of the teams and their agents worked out surprisingly well, with the AI acting as the central user 33m16s.
• The success of this approach is attributed to its similarity to how teams worked in the past, with the AI now taking on the role of the central user 33m22s.

Future plans 33m29s

• The next big leap forward for the AI agent is reliability, ensuring it doesn't break or spin, and expanding it to support any stack the user wants 33m35s.
• Currently, the agent doesn't listen to user requirements for the stack, but the goal is to accept user requirements and support various stacks, including Python 33m46s.
• The agent's UI is being improved to make it more user-friendly, with the possibility of interacting with the AI agent through drawing and voice commands 34m26s.
• Future plans include allowing users to draw on a canvas to communicate with the AI agent, making it possible to express ideas more creatively 34m57s.
• The iPad app will also be improved to make it more fun and creative, allowing users to hand-sketch UI mockups and have the agent implement them 35m7s.
• Simpler agentic tools will be added, allowing more advanced users to have more control over the code they're writing, including single-step or single-action agents 35m24s.
• These single-action agents will allow users to review and accept or reject changes before they are implemented, giving them more agency over the code 35m42s.
• The AI agent is still in beta, and users are advised to be cautious when using it, but the goal is to make it more reliable and user-friendly in the future 36m7s.

Outro 36m12s

• To test the AI agent, users can sign up for the core plan on Repet, but it's expensive and not free 36m12s.
• Once signed up, users can find the module on the homepage that says "what do you want to build today" and start working with the agents 36m27s.
• To get started, users should have an idea in mind, write a couple of sentences, and keep it simple, without making it too complicated or technical 36m34s.
• Working with the agent should be pretty intuitive, and users can share their projects to get feedback and support 36m47s.
• The community is encouraged to share their projects built with the agent, and the team is happy to reshare and retweet them 36m49s.
• The video concludes with a mention of "feeding the AGI" and a promise to see the viewers next week 36m53s.