Stanford CS153 Frontier Systems | Jensen Huang from NVIDIA on the Compute Behind Intelligence

The Evolution of Computing and AI

• The current era is considered a great time to be in computer science because computing is being reinvented for the first time in about 60 years, with the traditional computing model having remained largely the same since the IBM System 360 2m6s.
• The fundamental part of computer science has changed with the way software is written, processed, and applied, making everything fundamentally different, and this change is largely driven by the shift from pre-recorded content to generated content in real-time 4m30s.
• The new computing model allows for contextually consistent and relevant content that can respond to user intentions, and this has significant implications for how software is developed, organized, and deployed 5m20s.
• The methodology, tools, and approach to software coding have completely changed, and this change affects every layer of the stack, from computer systems and networks to storage, software stacks, and cloud services 6m10s.
• The emergence of deep learning and artificial intelligence has unlocked new applications, such as self-driving cars, which were previously considered impossible, and this has significant implications for various industries and fields 8m30s.
• The shift to AI-driven computing raises important questions about what it means to be a software engineer, how to organize a company, and how to architect computers for the age of AI, and these questions will have significant implications for the future of computing 10m0s.
• The journey of change in the field of computer science began about 15 years ago, with the realization that everything has changed, and this is the first generation of AI becoming useful 10s.
• Generative AI has not only made it possible to do image generation, text summarization, and translation, but it has also enabled AI to think and reason, with the ability to generate thoughts in the form of images and reason with them 1m42s.
• The question of how to train and fine-tune AI to reason step by step and teach it to do so at a large scale in a semi-supervised way is an engineering problem that needed to be solved, and the moment GPT happened, it was clear that thinking was just around the corner 2m6s.
• The idea that after GPT, agentic systems would emerge was fairly easy to predict, and now the question is what's next in a world where computers are not just responsive to what you ask them to do, but are continuously running 3m30s.
• The emergence of agentic systems raises questions about the future of cloud services, personal computers, and other systems, and presents a great opportunity to rethink all of these areas 4m30s.
• The field of computer science has changed, and everything about every field of science has changed because of the advancements in AI and computing, making it an exciting time to be in school and explore these new developments 5m40s.

Co-Design and Specialized Computing

• Codeesign is an interesting concept that involves abstracting computing so that different fields can work together, and it is related to the work of John Hennessy and the concept of risk 7m10s.
• The concept of harmonious co-design between compilers and microprocessor architectures is crucial to avoid creating a microprocessor that is difficult to compile, and a simpler instruction set can expose simplicity to compilers, allowing them to generate better code 10s.
• In the post-world of general-purpose computing, not all problems can be solved by a general-purpose instrument, and some extreme problems, such as computer graphics, molecular dynamics, and deep learning, require specialized systems due to their computational intensity 2m6s.
• Nvidia is a computer systems company that co-designs across various components, including CPUs, GPUs, networking, switches, and storage, to optimize performance, and this approach has led to significant advancements in computing power 4m42s.
• Moore's Law, which stated that computing power would double every 18 months, has slowed down in recent years, and while traditional microprocessor scaling might have resulted in a 10x increase in performance over 10 years, Nvidia's co-design approach has achieved a 1 millionx increase in performance over the same period 6m10s.

AI in Education and Learning

• The significant increase in computing power has enabled AI researchers to process vast amounts of data, and the ability to perform computations at such high speeds has created new opportunities and changed the way people approach computing, with applications such as taking all of the world's data and giving it to the computer 10m0s.
• The evolution of education in response to the industry is a crucial question, and it is suggested that AI should be part of the curriculum, not just for learning about AI, but also for using AI in the curriculum, as it can help keep up with the vast amount of information and knowledge being generated in real-time 10s.
• Traditional textbooks have limitations, as they take a significant amount of effort to create and update, and it is not possible for universities to keep up with the pace of information generation using pre-recorded textbooks alone, making a union of traditional and AI-based learning necessary 2m6s.
• AI can be a powerful tool for learning, as it can read papers, summarize them, and even interact with the user as a dedicated researcher, allowing for a more efficient and effective learning experience, and it is hoped that curriculums will become tightly integrated with AI in the future 4m30s.
• While AI is changing the way we learn and work, it is still essential to appreciate the first principles and fundamental methodologies, as they do not change, and understanding where we came from is crucial, even as we exhaust traditional design optimizations and move towards new technologies 6m20s.
• The use of AI in education can provide a similar experience to having "feet on both sides" of theory and practice, allowing students to learn from both the first principles and real-world applications, making learning more effective and enjoyable 8m10s.

Open Source and Proprietary AI Models

• The question of open source versus closed proprietary software is also important, and it is noted that Nvidia uses a significant amount of open source and anthropic AI tokens, with all engineers being agentically supported, highlighting the company's intentions and commitment to open source 10m40s.
• The recommendation is to use open AI and anthropic because they are useful, work well, and are constantly improving, with large language models being the technology inside and cloud code providing a harness around it that is also getting better all the time 10s.
• Language models are important because they represent the codification of intelligence, and automating ourselves, especially in important parts, requires learning the representation, meaning, and structure of information, which is possible due to the structure in biological and physical systems 1m20s.
• The goal is to learn higher-level representations of various domains, including chemicals, proteins, genes, physics, and robotics, and then generate and manipulate them, just like language models can learn and generate human language 2m6s.
• Different domains have fundamentally different structures and dimensionalities, requiring new strategies for training models, and this is why new approaches are needed for domains such as biology, autonomous vehicles, and climate science 3m30s.
• NVIDIA has dedicated itself to pioneering foundation models in several domains, including Neutron's language, Bioneo for biology, Alpamo for autonomous vehicles, Groot for humanoid articulation and general robotics, and climate science, in order to provide scientists with the necessary scale and technology 4m40s.
• As a result of this effort, NVIDIA has activated healthcare, life sciences, and is working with every single self-driving car company in the world, demonstrating the potential impact of these foundation models 6m10s.

Domain-Specific AI Models and Applications

• The development of foundation models is crucial for expanding AI and democratizing its capabilities, allowing the entire industry to flourish, and enabling the creation of various applications such as robotics 10s.
• Language models are being developed for two main reasons: to support languages with smaller scales, such as Swedish, that may not be a high priority for other companies, and to fuse language models with domain-specific models, allowing for more human-like reasoning and decision-making 42s.
• The fusion of language models with domain-specific models, such as Elpio, which combines a language model with a world model, enables the detection of objects like cars and roads, and allows for more efficient and safe decision-making, as seen in self-driving car models like Alpha Mayo 2m6s.

Security and Transparency in AI

• Open models are essential for creating safe and secure AI systems, as they allow for transparency and accountability, enabling researchers to interrogate and understand the decision-making processes of the models, and making it possible to defend against potential threats and vulnerabilities 4m30s.
• Transparent systems enable the creation of more effective cyber security measures, such as swarms of cheap AIs that can systematically surround and counter potential threats, rather than relying on a continuous cycle of escalating model development 6m40s.
• Neimotron Nano is being utilized for cyber security purposes due to its speed and cost-effectiveness, allowing it to be trained to detect cyber attacks and deployed in large quantities 10s.

Scaling and Utilization of Compute Resources

• The topic of open scaling was discussed, including bottlenecks such as data and compute, and the concept of coalition scaling, which was announced at GTC, aims to address these issues 2m6s.
• The utilization of compute resources is a significant concern, with a memo from XI revealing that their Memphis cluster pool is running at 11% MFU utilization, resulting in a substantial amount of unutilized MFU flops 2m6s.
• MFU, or Model Flops Utilization, refers to the percentage of flops consumed while doing work, and it is a metric that can be misleading, as high MFU may not always be desirable 4m42s.
• Overprovisioning of resources such as flops, memory bandwidth, and network capacity is necessary to avoid bottlenecks, but this can lead to idle resources when they are not needed, highlighting the challenge of optimizing utilization 6m10s.
• The cost of flops is not the primary concern, as the price of H100s is increasing due to factors such as bandwidth, architecture, and other features, rather than just the flops themselves 8m40s.
• Compute should not be considered a non-scarce resource, and teams should focus on optimizing utilization and addressing bottlenecks to improve overall efficiency 10m20s.

Measuring Intelligence and Performance

• When evaluating the performance of a system, the traditional measure of horsepower or flops is no longer sufficient, and a more comprehensive measure, such as tokens per watt, is needed to accurately assess intelligence 10s.
• The output of tokens can be considered a unit of intelligence, and the goal is to achieve a high number of tokens per watt, which is more important than just having a high number of flops 2m6s.
• However, not all tokens are equal, and the value of different types of tokens can vary, making it challenging to develop a standard measure of intelligence 4m30s.
• To address this challenge, it is essential to decide what evaluation metrics to use, as optimizing for the wrong metrics can lead to suboptimal performance, and a more nuanced approach is needed to account for different types of intelligence 6m15s.
• The problem of designing an index of different intelligences is complex, as various labs and research teams may have different evaluation metrics, but they all use the same underlying platform, such as NVIDIA chips 8m40s.
• The key to solving this problem is to strike a balance between being good at multiple domains and avoiding being too general-purpose, which requires a combination of vision, strategy, trial and error, and personal judgment 12m10s.
• Ultimately, the goal is to create a system that can excel in multiple areas, while also being mindful of the need to focus on specific domains to achieve sufficient funding for research and development 14m50s.

Advancements in AI Hardware and Systems

• The development of Hopper was driven by the need for pre-training, which required building larger systems, even larger than the largest scientific supercomputers in the world, with the goal of designing systems that could be multi-billion dollars 42s.
• The focus then shifted to inference, and a system called MVLink72 was created to address the high memory bandwidth requirements for token generation, leading to the development of the world's first rack scale computer, called Grace Blackwell MVLink72, which achieved a 50 times speedup over the previous generation 2m6s.
• The success of Grace Blackwell led to the development of Vera Rubin, which is designed for agents and focuses on the compute pattern and processing pattern of agents, requiring long-term memory to be stored and directly communicated with the GPU 4m30s.
• The design of Vera Rubin involves storage connected to the fabric, the use of CPUs, and the need for extremely low latency in CPUs to support the multi-billion dollar AI system, as the CPU is responsible for running tools while the GPU supercomputer waits for the instruction to be executed 6m10s.
• The development of these systems has been driven by the goal of achieving significant improvements in performance, with the aim of supporting the growth of AI and its applications, and the creation of new architectures and technologies to support this growth 10s.
• Vera is a CPU designed for single-threaded, multiple core code, and it is the most performant for the current generation, with the goal of solving computing pattern problems intuitively by thinking about what the computing pattern is and how it differs from the past 10s.

The Future of AI and Energy Demands

• The introduction of agents, which can be thought of as modules or sub-modules, will lead to systems of agents and sub-agents, creating a swarm of agents that will require a specific type of computer to manifest, potentially what Flyman is about 1m5s.
• One of the bottlenecks in computing is energy, and to address this, efforts can be made to improve energy efficiency, such as improving tokens per watt, which has already been increased by 50x, and will need to continue to be improved by significant factors 2m6s.
• To prepare for the enormous amount of energy needed for computing, which is likely to be a thousand times more than currently available, the ecosystem needs to be inspired and educated to get ready for this challenge, and computers of the future will be generative and continuous, requiring a different approach to thinking about energy needs 3m20s.
• The future of computing will involve generative computing in a continuous way, compared to pre-recorded, retrieval-based computing, and this will require a new way of thinking about the amount of energy necessary, taking into account that computers will be contextually aware and always generating 4m40s.
• The amount of compute and energy needed is extremely high, and it is necessary to explain this to people in a way that is easy to understand, using common sense and observable indicators to demonstrate that this is happening 10s.

Sustainable Energy and Market Forces

• There are various sources of energy, but due to concerns about the cost of sustainable energy, there has been underinvestment in this area, although market forces are now strong enough to make it a viable option without government subsidies 42s.
• This is considered the best time in history to invest in sustainable energy, as market forces are powerful and can drive the upgrade of the grid and the addition of sustainable energy sources 2m6s.
• In terms of education, there is an opportunity to provide information to a wide audience, including investors and capital allocators, who are interested in learning about sustainable energy and how to allocate resources effectively 4m6s.

Career Advice and Personal Development

• When it comes to choosing a career, the advice to follow one's passion may not be realistic for everyone, as many people do not know what they are passionate about, and it is more important to do one's best in any given job, regardless of whether it is a passion or not 6m6s.
• Even CEOs, who may claim to love their jobs, likely only enjoy a small percentage of their work and have to suffer through the rest, doing it to the best of their ability nonetheless 8m6s.
• Resilience is developed through suffering and struggling, which is essential for personal growth and being able to handle tough situations when needed, and it is advised to seek not just joy, but also some pain and suffering to build this resilience 10s.
• The importance of resilience is highlighted as a crucial character trait that can only be developed by going through struggles and hardships multiple times, making it a valuable asset for individuals to possess 1m42s.

Lighthearted and Personal Anecdotes

• A conversation about favorite food orders at Denny's takes place, with mentions of the fried chicken, Superbird, and a custom grill ham and cheese with tomato and mustard, showcasing a lighthearted and personal side 4m6s.

Nvidia's Role and Ethical Considerations

• The discussion shifts to the topic of Nvidia chips and their usage, with the clarification that Nvidia makes GPUs used for various purposes such as video games, medical imaging, and other applications, emphasizing the company's role in different industries 8m42s.
• The comparison of Nvidia GPUs to atomic bombs is deemed stupid and ridiculous, highlighting the significant difference between the two, with Nvidia GPUs being widely used and advocated for, whereas atomic bombs are not 10m6s.
• The idea that American companies should not compete in foreign countries because they will lose anyway is not a viable philosophy, as it is through competition that markets are served and companies are enhanced 10s.
• The notion of depriving certain countries of general-purpose computing, which Nvidia is a part of, does not make sense, as it would allow one or two companies to benefit at the expense of others, and the American technology industry is a national treasure that should not be compromised 2m6s.
• If the American technology industry were to give up its position in the global market due to policy decisions, it would lead to a decline in the industry, similar to what happened in the telecommunications sector, where America lost its fundamental technology due to policy decisions 4m30s.

AI Singularity and Misconceptions

• The concept of an AI singularity that will come suddenly and be incredibly powerful is not supported, as it is not true that the technology will become infinitely powerful and take over the world, and it is irresponsible to spread such science fiction fantasies as fact 6m40s.

Access to Compute and National Priorities

• The goal should be to create a future where everyone benefits from AI, and it is important to be rational optimists who believe in the potential of technology to improve society, rather than spreading fear and misinformation 10m20s.
• There is a problem with reasoning by analogy, and the lack of access to compute for independent teams, startups, and universities in America is a significant issue that needs to be addressed 12m10s.
• The priority for scarce resources, such as computer chips, should be given to America first, but this is not currently happening, and the reason for this is unclear 10s.
• There is a misconception that people are placing orders for chips and not receiving them, but the reality is that the system is not built to deliver massive scale compute, and research departments at universities like Stanford are not equipped to handle this 2m6s.
• The fundamental problem is that the system is no longer centralized, and each department at Stanford raises its own funding and has its own grants, which are not enough to support large-scale computing, and the university does not have a budget for a billion-dollar compute 2m6s.
• The lack of a budget for a billion-dollar compute is seen as Stanford's fault, and by acknowledging this, the university can empower itself to solve the problem by changing the way it does budgeting and computing 4m30s.

Solutions for University-Level Compute Access

• One possible solution is for Stanford to build a campus-wide supercomputer that everyone can share, or to contract with someone else to provide this service, which would require a significant investment of around a billion dollars 6m20s.
• With Stanford's $40 billion endowment, it would be possible to allocate a billion dollars to provide every student and researcher with access to AI supercomputers, but this would require planning and coordination 8m10s.

Leadership and Strategic Vision

• As a CEO, one has to conceive of the intersection between vision, strategy, and execution, and being surrounded by amazing computer scientists can make a vision more ambitious and realizable 10s.
• The fun part of being a CEO is constantly updating one's view of the future, vision, and role in it, which is a highly imaginative, strategic, and complicated process with no right answer 42s.
• However, with the power of being a CEO comes the responsibility for the well-being of the team members who have joined to help create the future, which can be a deeply vulnerable and humiliating experience, especially during difficult times 2m6s.
• In the early days of Nvidia, one of the biggest mistakes made was the completely wrong architecture and technology used in the first generation of products, including using curved surfaces instead of triangles and no Z-buffer, which led to tremendous bad choices 4m30s.
• Despite the technical bad choices, Nvidia was able to transform and become the only remaining company in its field, which taught a valuable lesson about the importance of strategy, conserving resources, and applying resources, and the experience of deep failure led to a significant learning experience about strategic thinking and maneuvering 6m40s.
• The experience of nearly going out of business multiple times and being on the verge of failure was embarrassing, humiliating, and hard, but it ultimately contributed to the development of strategic genius moves and a deeper understanding of the importance of strategy in business 8m10s.

Strategic Mistakes and Business Lessons

• A mistake was made when the company shifted its resources to build mobile devices, as the amount of value that could be delivered was probably marginal, and the company was eventually locked out of the market during the 3G to 4G transition, with Qualcomm becoming the leader in that space 10s.
• The company's mobile device business grew to a billion dollars but then went back to zero after being shut out, and the expertise gained in extreme low power and energy efficiency was later shifted to the field of robotics 2m6s.
• The expertise and teams built up during the mobile device era were helpful in developing new technologies, such as the chip called Thor, which is a descendant of the chip used in mobile devices 4m30s.
• Strategic mistakes can occur when forecasting is not precise enough, and it is essential to update priors and develop a forecasting mechanism to give confidence in the direction being taken, even when the future is not entirely clear 6m40s.

Forecasting and Building Mental Models

• To navigate uncertain situations, it is crucial to observe, reason about things from first principles, and ask questions about what is happening next and what it means for the field, as seen in the example of deep learning, computer vision, and AlexNet 8m10s.
• The development of AlexNet, a neuronet network model, was a big deal as it significantly improved computer vision capabilities, and it is essential to consider what else can be solved using similar approaches and what it means for computers and computing 10m20s.
• To create a mental model about the future of computing, one must keep asking questions and iterating until reaching first principles, considering examples such as self-driving cars and robotics, and then build up a mental model of the future from there 0s.
• The process involves reasoning through the differences between processing neuronet networks and traditional computing methods, such as floating-point numbers and integers, and considering how large models will become and what computers will look like in the future 42s.
• When building a mental model of the future, it is essential to be aware that one's predictions may not be entirely correct, and it is helpful to categorize potential outcomes into things that will likely happen, things that will absolutely happen, and things that may happen 1m30s.
• The skill of building successful companies involves moving in a chosen direction, which requires energy, time, and money, and it is crucial to consider the opportunity cost of pursuing a strategy and try to reduce it while increasing optionality 2m6s.
• The goal is to make the journey pay for itself, as everyone will be seeking more resources, and this requires continuous thinking and planning to navigate the challenges and uncertainties of the future 3m30s.