Why this CEO thinks video games make better training data than the internet | Equity Podcast

Company Overview and Funding

• General Intuition, a New York-based startup, has closed a $320 million dollar round led by Costanoa Ventures with backing from notable investors such as Jeff Bezos and Eric Schmidt, and researchers at MIT and Google DeepMind, giving the company a $2.3 billion valuation 10s.
• The company's CEO, Pim de Wit, sought out investors who were directionally aligned with the research and had the luxury of Costanoa betting a large portion of their fund, making it easier to close the round as the models kept getting better 2m6s.

Data Strategy and Model Training Approach

• General Intuition is building a proprietary data set, and the company spun out from Metal, with a focus on training large language models using data from video games, which provides a more comprehensive understanding of the world by including spatial and temporal dynamics 4m42s.
• Traditional large language models are trained on text data from the internet, which can lead to a limited understanding of the world, as text removes information about space and time, whereas video games provide a more dense and dynamic source of information 6m10s.
• Video games offer a unique combination of information density, discourse, and spatial and temporal dynamics, making them a more suitable source of training data for AI models, allowing them to develop a more nuanced understanding of the world 8m20s.

Advantages of Video Game Data

• The company has a unique data set with diversity and scale, allowing for internet-scale pre-training with trillions of tokens, which is a bet on the next scale of pre-training, similar to early LLMs, and this data set is based on video game clips with hundreds of millions of hours of data, including video and action data 10s.
• The models trained on this data are good at controlling behavior in simulation, spatial-temporal reasoning, and understanding space and time, and they have the potential to predict text and do science, but the extent of these capabilities is still unknown 2m6s.
• Text is always biased by the author's perception, whereas the models trained on video game data are trained in a neutral, unbiased way, resulting in a different type of intelligence that is unlike LLMs and can be both good and bad, as it makes interpretation harder in some cases 4m42s.

Pre-training Methodology and Data Sources

• The company's pre-training method uses data from Metal TV, which has hundreds of millions of hours of video game clips, including action data such as button presses and timing, and this unique data set allows for pre-training in a different way 6m15s.
• The models can be fine-tuned with very little data, such as 8 minutes of real-world robotics post-training data, and can navigate complex environments, including indoor and outdoor spaces, with dynamic objects and people, which is a significant surprise and a potential breakthrough 10m30s.

Generalization and Model Performance

• The concept of generalization and in-distribution is relevant to various models, including those used for navigation and large language models (LLMs), with some actions working better than others in these contexts 10s.
• Before LLMs, custom models like BERT were used for specialized tasks such as moderation, safety, and data filtering, but they required a lot more data to be used in intelligent chatbots 42s.
• There are companies focusing on specialized work for individual embodiments, environments, and robots, but a general pre-trained base model is expected to emerge, where the model's generalization is the product, allowing for reduced need for large amounts of real-world data 1m6s.
• The amount of data needed can vary depending on the task, such as navigation, which may only require a few minutes of data, but other tasks like robotic arms may need more data 1m30s.

Company Origins and Vision

• The company General Intuition was born out of rejecting an acquisition offer, and its name reflects the focus on intuition as the next big leap after reasoning in world models, as discussed with Vinod Khosla 2m6s.
• The company's team, including co-founders who are authors of notable papers like Diamond, Delta Iris, and Iris, is seen as a key factor in its potential as a generational company rather than an M&A target 3m30s.

Founding Team and Research Background

• The co-founders have a strong background in world models, with their work on papers like Diamond, which was a NeurIPS spotlight paper in 2024, and their creation of foundational world model architecture papers 4m0s.
• The creation of something demonstrably great, such as world models, can be achieved by individuals who are still in the early stages of their careers, and they can make significant contributions to their field, even with limited resources, as evidenced by the development of a real-time world model running in Counter-Strike on a basic GPU with only 100 hours of data 10s.
• Delivering a state-of-the-art paper in a constrained environment, such as with limited compute available, can be a significant achievement, and it requires a deep understanding of the world and the ability to work with people who share similar values and missions 42s.

Collaboration and Talent Attraction

• Having a good understanding of the world and being able to work with people who share similar values and missions is more important than the specific location where the work is being done, and this can help attract top talent, even in a competitive market where companies like OpenAI are poaching from Google 2m6s.
• The availability of high-quality data is a key factor in attracting top researchers and driving innovation, as every researcher wants to do their best work and publish frontier papers, and the best work is always going to come from where the data is 4m10s.

Data as a Strategic Asset

• Certain consumer products, such as video games, can suddenly become AI infrastructure, and the indicators of this are the diversity of environments in which the product is being used, such as 2D interfaces, 3D games, and different types of games 6m30s.
• Other fields, such as biology, healthcare, and physics, may also have deep buckets of data that can be leveraged to drive innovation, and companies like ASML are already working on developing specialized models built on top of generalization capabilities 10m20s.

Data Generalization and Model Development

• The question of whether a data set is general or specialized is important, as it determines whether a harness approach or a fine-tuning approach is needed, and this can have significant implications for the development of AI models 12m40s.
• The process of training models involves pre-training for specific economic problems or fine-tuning multimodal models to beat benchmarks, which helps learn how to process and apply the data, and this journey can lead to unexpected outcomes, such as developing skills that can be applied to various jobs 10s.

Challenges in Learning from Video Data

• The difference between learning from video and learning from other datasets, like those found on the internet, is that video can provide action information from pixels, but this information may not always be accurate or relevant, especially in edge cases where fully accurate long-horizon data is necessary 2m6s.
• Labs often argue that action information can be obtained from pixels in videos, but this information may not be sufficient for models to perform well in edge cases, and using inferred data can lead to errors accumulating over time, whereas models trained on ground truth data can outperform those trained on inferred data 2m6s.
• The separation of actions from the environment is crucial for models to learn effectively, and this separation allows models to scale differently and have different downstream applications, which is why models trained from scratch on ground truth data can perform better than those trained on inferred data 4m30s.

World Model Architectures and Capabilities

• The way Genie and other world models generate agents and world models differently, with Genie's model generating both the world and the agent, whereas other models, like the one being discussed, generate the world model and the agent separately, which can affect the model's performance and ability to internalize physics and collisions in different environments 6m20s.
• Genie's approach to world models is broader, allowing it to predict unfolding dynamics over time in pixel space, starting with an image and predicting the correct dynamics, but this broad approach can also lead to models encountering environments where they have not internalized the physics and collisions, making it more challenging to achieve accurate predictions 8m40s.
• Genie has internalized physics reasonably well by utilizing physics engines, and this internalization is important for training models in predictable environments 10s.

Ground Truth vs. Inferred Data

• The difference between ground truth and inferred data is significant, as ground truth provides intuitive and predictable environments, which is essential for training models 42s.

Ethics and Alignment in AI Development

• The discussion around ethics and aligning with ethics is crucial, particularly in the context of defense and lethal autonomy, with the goal of not harming humans and making architectural and technical decisions to achieve this 2m6s.
• The use of violent clips in training models is a complex issue, and the approach is to use these examples to negatively reward the models, which helps to prevent harmful behavior 2m6s.
• The idea of pitching defense work as a way to prevent harm, rather than causing it, is an important consideration, with the goal of training models that do not know how to kill people 4m30s.
• The importance of evaluating decisions and hiring good people who can assess the impact of models on national security and democratic values is emphasized, with the goal of advancing these values without becoming part of an escalatory system 6m15s.
• The challenge of navigating the complexities of war and national security is acknowledged, with the recognition that democracies may require the use of general capabilities for defense, but this should be done in a way that prioritizes democratic values and does not contribute to escalation 8m45s.

Responsibility and Democratic Values

• The belief that one should not be more important than the democracies they serve is firmly held, and making one's stance clear on these issues is essential 10m50s.
• The development of models should prioritize actions that do not harm humans, and it is crucial to consider the potential impact of these models on the economy and job market 10s.

Economic Impact and Job Creation

• A company has launched a marketplace called Nerve, which aims to provide opportunities for gamers, including data labeling and teleoperations jobs, in an effort to create jobs in the new economy 2m6s.
• The approach to addressing job loss due to AI involves putting effort into creating tools that generate jobs, rather than just hiring economists to study the issue, and this can help absorb the shockwave of job displacement 2m6s.
• The company is using its understanding of human actions to create suitable jobs for people, and this is being done through Nerve, with General Intuition being the first customer, and jobs such as data labeling and teleoperations are being created 2m6s.
• The creation of these jobs can lead to a feedback loop, where data is collected on human actions and teleoperated machines, which can further inform job creation and improvement 2m6s.

Company Contact and Media Presence

• The company's founder, Pim de Wit, can be connected with on Twitter or X, and listeners can also find the Equity podcast on various platforms, including X, LinkedIn, and YouTube 2m6s.