How GitHub Copilot helped SAP reimagine the developer experience

Introduction by Sumeet Shetty and Tobias Schimmer, outlining the session and its goals. 0s

• Sumeet Shetty and Tobias Schimmer are presenting at GitHub Universe, discussing how GitHub Copilot is helping SAP reimagine the developer experience 11s.
• Sumeet Shetty is the head of tools India at SAP, and Tobias Schimmer is the head of developer experience 28s.
• The presentation will cover SAP's point of view on developer experience, their AI programming journey, key learnings, and the impact of GitHub Copilot on SAP 47s.
• The AI programming journey at SAP has lasted over a year and a half 59s.
• The presentation will also share key takeaways on the impact of GitHub Copilot at SAP 1m14s.
• Sumeet Shetty and Tobias Schimmer have known each other for over a decade, and Sumeet initially reached out to Tobias for research support in evaluating GitHub Copilot 1m29s.
• Tobias was asked to help understand how GitHub Copilot resonates with people and its impact to make a better decision for SAP 1m49s.

Developer Experience at SAP 1m56s

• Developer experience at SAP is focused on three key aspects: efficiency, effectiveness, and creating a motivating and retaining team, with the foundation being good practices such as DevOps, Agile, and Lean 2m17s.
• The productivity equation consists of efficiency, effectiveness, and creating a motivating team, with effectiveness meaning addressing the right markets, talking to the right customers, and solving problems for the right end-users 2m20s.
• The mission for developer experience is to deliberately put humans at the center, ensuring a perfect flow for individuals and teams, and guaranteeing the value flow of entire product units 3m11s.
• The developer experience function is human-centered, taking the perspective of one developer with a certain problem to solve, and is data-driven, with two aspects: data about products and data about the process 5m5s.
• Data about products includes understanding how a product is being adopted, customer satisfaction, and usage patterns, while data about the process includes measuring Dora 4 and flow metrics 5m40s.
• The goal is to provide teams with a way of data-driven continuous process improvement and to leverage the power of AI along the software development life cycle where it makes sense 6m2s.
• Generative AI was the starting point for reimagining what developer experience is and will be at SAP, with the aim of creating a better experience for developers 6m17s.
• SAP has over 100,000 employees and rarely builds one product with one team, so ensuring the value flow of entire product units is crucial 3m44s.
• Cognitive load is also an important consideration, with technical cognitive flow and domain knowledge being key factors, as developers need to have specific knowledge in their minds to build software for specific industries 4m27s.
• The developer experience team aims to stay connected to customers and end-users, with customer feedback being essential, not just from a technical perspective but also from an actual customer feedback perspective 4m4s.

AI Pair Programming at SAP 6m30s

• Generative artificial intelligence is central to developer experience at SAP, with AI pair programming being the number one use case for enhancing developer experience using generative artificial intelligence 6m36s.
• AI pair programming involves programming with the support of an assistant powered by artificial intelligence 6m56s.
• SAP evaluated various AI pair programming tools, including GitHub Copilot, Safai, Tab9, Codium, Ghost Writer, Gemini Code Assist, Q Developer, Code GPT, Cod, and Cursor, before selecting GitHub Copilot as their AI pair programming tool of choice 7m43s.
• The evaluation was based on a list of desired capabilities for an ideal AI pair programming tool, which included intelligent autocomplete suggestions, generating code from scratch using prompts, and generating unit tests 8m1s.
• GitHub Copilot provides additional capabilities, such as finding and fixing bugs, reviewing code, refactoring code, translating code from one programming language to another, generating developer documentation, and answering questions across multiple programming languages 9m7s.
• GitHub Copilot brings all these capabilities to SAP developers, enhancing their experience and providing a huge value proposition 9m53s.
• The journey of AI pair programming at SAP began with an evaluation of tools in March 2023 and involved working with various corporate functions, including AI ethics, legal, data protection, and procurement 10m25s.
• A group of 500 pilot users, called early adopters, tested GitHub Copilot for three months, and the collected data led to a decision to roll out the tool across the organization 11m20s.
• The rollout of GitHub Copilot started in November 2023, and the journey involved gathering data through systems and people, with a focus on human centricity 12m9s.

Research strategy & using survey to gather data 12m33s

• The research strategy involved using Telemetry data from GitHub Copilot and enriching it with a comprehensive set of questions in a survey to gather experience data from developers and understand their perceptions of productivity, ease of use, and willingness to promote the tool 13m12s.
• The survey aimed to gather data on how developers perceive GitHub Copilot, including its effectiveness, ease of use, and willingness to promote it to colleagues, as well as understanding the challenges they face while using the tool 13m25s.
• A welcome survey was conducted to understand the demographics of the 500+ participants, including their tenure, location, tools used, and technology stack, followed by regular two-weekly check-ins to understand specific use cases and challenges 13m47s.
• The data gathered from the surveys and Telemetry data was shared with GitHub to provide feedback and improve the tool 14m20s.
• A comprehensive survey was conducted after three months, and a closing survey was conducted a couple of weeks later to gather additional information and address any remaining questions 14m23s.
• The research was not conducted alone, but in collaboration with research partners, including the Hasso Plattner Institute in Germany, the University of California, Irvine, and the University of Mannheim, among others 14m51s.
• The research team also included a staff researcher hired specifically to study GitHub Copilot and other development tools, and collaborations with other universities, including the Technical University of Munich, to study AI use cases in software development 15m32s.

Results and feedback from the first phase of GitHub Copilot implementation. 15m59s

• The early adopters of GitHub Copilot, consisting of 500 developers, found the tool to be easy to use, successfully drawing context from their prompts and code, making them feel more productive, and increasing the time for value-adding tasks 16m7s.
• The use of GitHub Copilot also made software development easier and faster, allowing developers to write more code, deliver more product features, and reduce the effort for repetitive tasks 16m29s.
• A survey of the 500 early adopters showed that 87% wanted to continue using GitHub Copilot at the end of the early adoption period 16m47s.
• A key summary of the early adoption phase revealed that 93% of developers found GitHub Copilot easy to use, 84% said it made software development faster, 81% felt more productive, 88% said time for repetitive tasks decreased, and 87% wanted to continue using it 17m13s.
• The introduction of Copilot Chat led to an increase in most of these numbers, with 84% of developers saying they were able to learn from the suggestions provided by GitHub Copilot, 78% saying they could write more code, 80% feeling more productive, and 71% saying they could write code with better quality 17m34s.
• The percentage of people who wanted to continue using GitHub Copilot increased from 87% to 93% after the introduction of Copilot Chat 18m0s.
• A larger survey of around 10,000 developers confirmed the results of the early adoption phase, with 93% saying GitHub Copilot was easy to use, 81% agreeing that it made software development faster, 79% feeling more productive, and 85% saying repetitive tasks got faster 18m28s.
• The larger survey also showed that 88% of developers, who use multiple programming languages, technology stacks, and code editors, would love to continue using GitHub Copilot 19m1s.

Learnings from our journey 19m10s

• Research with Microsoft found that GitHub Copilot's value proposition was challenged, but SAP's own research yielded similar or better numbers, which was the first surprise 19m27s.
• The introduction of GitHub Copilot Chat led to a step change, with numbers showing increased learning and willingness to use GitHub Copilot, translating to a net promoter score of 77 19m50s.
• Scaling out to 10,000 people still showed the same numbers, with 88% willing to continue using GitHub Copilot, translating to a net promoter score of 66 20m15s.
• A closing survey found that developers were using their time to tackle technical debt, with 40% of responses related to this, which is a top productivity blocker in annual developer experience surveys 21m0s.
• The survey also found that introducing AI programming with GitHub Copilot led to cultural changes, including enhanced developer experience, productivity, code quality, and learning, as well as the ability to focus on creative and value-adding work 21m40s.
• Potential negative consequences of relying on AI-generated code include over-reliance, decreased programming skills, code review skills, and motivation to learn new languages and technologies, as well as a decrease in human-centered programming 22m18s.
• A quote from Fabian Hala, a developer with a strong data science background, stated that GitHub Copilot makes it easier to be a full-stack developer proficient in multiple languages 23m11s.
• A developer at SAP, who prefers the GitHub Copilot metaphor, uses it for smooth and fast coding with Java, and was able to provide a quote and picture within five minutes of being asked, showcasing his satisfaction with the tool 23m34s.
• Surui Brahma, a Java developer at SAP in Bangalore, uses GitHub Copilot for its ability to intelligently complete her code, generate code from scratch, generate unit tests, and find and fix bugs, and she loves using it with Java, a language she is an expert in 23m51s.
• Surui Brahma also used GitHub Copilot to work with Go, a new programming language to her, and was astonished by how easily she was able to get started with it, describing the experience as "magical" 24m21s.
• Max Milon Teitz, who works with Kubernetes and Go in Berlin, uses GitHub Copilot to develop cloud-native applications better and faster, and it significantly lowered his entry barrier to getting started with Go, a programming language he was unfamiliar with 24m57s.
• As of October last year, SAP had 500 early adopters of GitHub Copilot, but today they have over 18,000 developers onboarded to the platform, which is a significant milestone worth celebrating 25m33s.
• SAP took a phased approach to the global rollout of GitHub Copilot, starting with 500 developers in October, then 3,000 by the end of the year, and scaling rapidly since January 2024, reaching their target of 18,000 developers in September 2024 25m55s.
• Scaling in phases allowed SAP to learn from each phase, apply those learnings to the next phase, and regroup and react to feedback from previous rounds, giving them options and time to reflect on what went well and what did not 26m40s.

Impact of GitHub Copilot 27m4s

• GitHub Copilot has provided 23.6 million AI-powered suggestions to developers at SAP between January and September 2024, with 6 million suggestions accepted and 9.3 million lines of code incorporated into SAP products and solutions 27m11s.
• The average code acceptance rate is 25.6%, calculated as the ratio of suggestions accepted to suggestions provided, with a significant increase in suggestions provided and accepted over time 27m39s.
• The number of suggestions provided increased consistently as more developers were added, with a significant increase in suggestions accepted by developers, from 1 million in January to 3.5 million in August 28m6s.
• The number of lines of code accepted also increased, from 300,000 to 350,000 in January to 1 to 1.25 million in August and September 28m35s.
• The code acceptance rate initially hovered between 20% to 22% but increased to around 24% after the introduction of chat and has since averaged around 30% for the last three months 29m35s.
• SAP provides multiple resources to onboard developers to GitHub Copilot, including a SharePoint site with an overview, FAQs, setup instructions, and dos and don'ts for responsible use 30m6s.
• A GitHub Copilot Community space allows users to share and collaborate, and a Q&A Channel provides support from experts within SAP and GitHub 30m40s.
• A series of expert talks has been added as an onboarding resource, featuring experts in AI programming and GitHub Copilot discussing various topics related to AI programming 31m10s.
• These onboarding resources have been successful, with the SharePoint site viewed over 115,000 times and the channels being actively used 31m30s.
• The SAP community and Q&A Channel have over 18,000 members each, with expert talks having more than 10,000 participants in just a few months 31m37s.
• SAP's adoption of GitHub Copilot began with an unusual start when a colleague from Buer was contacted on LinkedIn, leading to connections with other colleagues, including Tim Mina Zagen, who is at Universe 32m2s.
• SAP is now sharing their learnings from the 1.5-year journey with customers, partners, and friends, including companies like Daimler Truck, Bosch, Raj da Telecom, and BASF 32m29s.
• An important moment in SAP's adoption of GitHub Copilot was when they reached 18,000 developers on boarded 32m54s.
• Another significant moment was when Gartner released their first-ever Magic Quadrant for AI assistance, rating GitHub as a leader, which validated SAP's research and decision to roll out Copilot at scale 33m0s.

Key Takeaways 33m31s

• SAP, the world's largest provider of Enterprise application software, rolled out AI pair programming with GitHub Copilot to over 18,000 developers, focusing on three key qualities: speed, efficiency, and security 33m31s.
• The adoption of GitHub Copilot at SAP helped create trust in the ease of use and output of generative artificial intelligence, and also allowed the company to gain empathy for corporate users of their software 33m56s.
• Research indicates that GitHub Copilot significantly enhances developer experience at SAP, which was a highlight of the last year and helped the company start reimagining the developer experience 34m32s.
• GitHub Copilot brings fun, creativity, and flow back into programming, as stated by GitHub CEO Thomas Dohmke, which aligns with SAP's experience and research findings 34m57s.
• SAP successfully scaled GitHub Copilot adoption from 500 to over 18,000 developers, with support from the GitHub Copilot adoption team and GitHub leadership 35m32s.
• SAP is proud to share their story of AI pair programming with GitHub Copilot at the 10th anniversary of GitHub Universe, and is open to sharing additional details and receiving feedback 35m49s.