Thriving through change: A decade of engineering evolution at Spotify

Introduction

• Spotify operates on a large scale with over 600 million users in 180 countries, making it a truly global platform, and has over 2,800 engineers working across 500 teams to deliver exciting and innovative features 1m4s.
• The engineering team makes almost 3,000 production deployments every day, which translates to two changes every minute, 365 days a year, without disrupting the music experience 1m41s.
• Spotify has grown from music to podcasts and audiobooks, catering to unique tastes of its users, and has released audiobooks in multiple countries 2m6s.
• The company has over 190,000 GitHub repositories spread across 2,800 GitHub organizations, and 8.8 million lines of code are managed by automatic bots making automated changes 2m37s.
• Spotify's path to its current scale involved a huge technological and cultural transformation over the last 10 years, which will be discussed in the presentation 2m49s.
• The presentation will cover the early days at Spotify, cultural shifts, technological shifts, fragmentation, and AI investments, as well as exciting prospects for the future 3m22s.

Spotify's Culture and Collaboration with GitHub

• Spotify's culture is built around five core values: innovative, playful, collaborative, passionate, and sincere, with a focus on collaboration and adaptability to change 3m56s.
• The company's long-standing collaboration with GitHub is an example of its adaptability to technological shifts, and its technical strategy is defined by its ability to react to these changes 4m26s.
• Software engineering is a team sport that involves people, tools, and processes, and building large-scale applications requires collaboration and defining processes to work together effectively 4m45s.

Early Days and Server DB

• In the early 2010s, Spotify was a small company with 23 SREs managing data centers and available in only 28 markets, with servers named after Greek mythology and provisioning of new hardware coordinated through spreadsheets 5m9s.
• To improve the process, a system called Server DB was built to automate the process of importing and exporting server information, giving human names and purposes to servers, and tracking hardware 5m40s.
• At that time, there were no fancy interfaces, and everything was driven by a command-line interface 6m11s.

Migration to GitHub

• In 2014, Spotify was using Gerrit to manage its Git repositories, but the industry was shifting towards GitHub, which was becoming the industry standard for version control systems 6m26s.
• Spotify's primary goals for evolving its version control system were to increase code quality through collaborative code review and ensure that every commit was backed by continuous delivery 6m56s.
• A kickoff pilot was done in late 2014, and user surveys showed that engineers wanted to use GitHub, which they were already familiar with from open-source projects 7m36s.
• In May 2015, Spotify successfully shut down its Gerrit instance and fully migrated to GitHub Enterprise Server, allowing engineers to release more often and create more features 8m1s.

Service DB and Growth

• As the company grew, the number of components and services increased, and Spotify needed a better way to keep track of them, leading to the creation of a new service called Service DB 8m32s.
• Service DB was linked to Server DB, allowing engineers to browse services and figure out where they were deployed, and providing a way for engineers to work together more effectively 8m44s.

Transition to the Cloud and Backstage

• As Spotify grew, its engineers transitioned from physical hardware to the cloud, opening up opportunities for automation, and a single click of a button could handle tasks that previously took weeks or months with multiple tickets between developers and operations 9m13s.
• The platform organizations invented a new set of tools for developers, but they were scattered and difficult to find, leading to an explosion of new services and hypergrowth, with the number of components growing exponentially over time 9m54s.
• The discoverability problem led to the creation of Backstage, a unified experience that started as a small pilot project and became the most widely used developer tool inside Spotify 10m27s.
• Backstage aimed to provide more information about services, such as whether they were running in production, who owned them, and who to page when something went wrong 10m47s.

Backstage Features and Development

• The software catalog, a searchable and browsable list of all software inside Spotify, was created to keep metadata about every piece of software, including ownership and relationships to other software 11m3s.
• The catalog was stored alongside the source code in GitHub, distributing the responsibility of keeping the metadata up to date to the teams managing the source code 11m21s.
• Technical documentation was also added, built on top of markdown files stored in the git repository and linked to Backstage, providing a dedicated space for each service in the catalog 11m49s.
• Each service had an overview page with general information, browsing documentation, and additional fields for build history, capacity management, deploy status, and traffic routing 12m0s.
• The system was linked together with GitHub at the heart, containing all the data in source repositories, and an API for configurability using pull requests 12m39s.

Addressing Developer Experience Challenges

• The focus on solving discoverability problems over the years led to the creation of a comprehensive system, and the discussion will now be handed over to Sanjana to talk about other developer experience problems, such as fragmentation 12m53s.
• Spotify started investing in developer experience about eight years ago due to troubling trends, including slower development and shipping of new features, and harder onboarding of new developers 13m4s.
• Spotify experienced years of hyper-growth, with the number of developers quadrupling every year, resulting in a huge and chaotic organization where it was hard to keep track of who owned what and affecting developer productivity 13m24s.
• To address this issue, an intense look was taken at the day-to-day life of developers to understand what was making their lives difficult and find a solution to unblock faster development 13m59s.
• Two major problems were identified: it was hard to do things the right way fast due to a scattered software ecosystem, and the burden and maintenance toil that came with hyper-growth, including managing bills, deployments, security audits, and upstream dependency breaking changes 14m16s.
• A developer portal was created to make it easier to find things, but it was clear that more needed to be done to centralize and manage the software life cycle and take away infrastructure pain from developers 15m28s.

Tech Health Strategy and Golden Paths

• The software ecosystem was extremely fragmented, with every team building and deploying services differently, making it hard to automate 15m48s.
• A Tech Health strategy was cultivated to enable developers to become productive and efficient, including the creation of "Golden paths," a gold standard of how to develop software at Spotify, which documented best practices and were used as tutorials for new hires 16m13s.
• The first Golden path was created for backend development and covered how to set up an environment and deploy a backend application, initially 21 pages long but later condensed to cover more ground 16m37s.
• "Golden Technologies" were introduced, a standard set of languages, infrastructure, and frameworks expected to be used for all production components at Spotify, allowing developers to focus on end-user features 17m23s.
• The concept of "Golden state" was also introduced, the desired state of where Spotify wants all its software components to be, a target state in which the tech stack can be easily managed and kept up to date 17m48s.
• To keep up with the ever-evolving technology landscape, a simple yet effective tech strategy called "Golden Path" using "Golden Technologies" to reach the "Golden State" was implemented 18m12s.
• To make it easier to adhere to best practices, software templates were created, allowing developers to spin up their own services with just a click of a button in Backstage, setting up their entire service with all configurations and features already set up and ready to go 18m30s.
• This standardization allowed for the automation of some infrastructure toil, making it easier to create new software and ensuring that it adhered to current standards by default 19m19s.

Fleet Management and Automation

• A dependency graph of all services in production at Spotify was used to visualize the complexity of the system, with every edge being a dependency and every node being a service 19m50s.
• To automate away infrastructure toil, a new program called Fleet Management was introduced, which is split into three pieces: making sure everything is versioned and codified, making it easy to make changes at the click of a button, and doing this at scale across hundreds of thousands of repositories 20m20s.
• Fleet Management was able to address all three parts of the problem, resulting in huge savings in infrastructure toil and allowing developers to focus on building end-user features 21m12s.
• An example of the impact of Fleet Management is the resolution of the Log4j vulnerability, where 80% of the fleet was secured in just 9 hours, and the entire software ecosystem was completely secure within 8 days 21m56s.

Measuring the Impact and Results

• The effects of these systems were measured, and the change was astounding, with a significant reduction in the time for a new hire to create their 10th pull request 22m33s.
• New hires at Spotify can now contribute to the code in 20 days, down from 60 days initially, due to improvements in finding information, documentation, and standardizing technology, resulting in a significant return on investment that compounds every year as new developers are hired 22m56s.
• Existing engineers have seen a 25% reduction in high-urgency incidents and a 56% reduction in local build times, while Fleet Management has enabled faster feature shipping, reducing the time from 200 days to less than a week 23m46s.
• The reliance on bots and tight integration with GitHub has enabled zero-touch code changes, with bots creating and committing three times more code changes than humans, leading to faster feature delivery to end-users 24m29s.
• Developers who frequently use golden paths, Fleet Management, and other preset tools built into Backstage are 2.3 times more active in GitHub, make twice the number of code changes with 177% less cycle time, and deploy twice as often with deployments staying deployed three times as long 24m59s.

AI Adoption and Feature Launches

• The adoption of AI tools, such as Co-Pilot, has increased from 13% to 56% among engineers, resulting in a 43 percentage point increase, with Co-Pilot being used in daily workflows 27m25s.
• Spotify's investment in developer experience has enabled the launch of features such as Audiobooks and AI DJ, with GitHub being at the center of these changes 26m42s.

Streamlining GitHub Co-Pilot Access

• Spotify operates on a large scale and has a small team managing GitHub, so they needed to automate and streamline the process of granting users access to GitHub Co-Pilot, a tool that assists with coding tasks, to avoid a manual and time-consuming process 27m42s.
• To achieve this, Spotify worked with GitHub to design a streamlined process that allows developers to opt-in to internal tools using Backstage with a single click, supported by their underlying identity provider system 28m5s.
• GitHub makes it easy to link the identity provider system into GitHub organizations, enabling users to have automatic access to Co-Pilot when added to a GitHub organization 28m31s.
• By combining these two systems, Spotify sent out an announcement to their mailing group, allowing engineers to self-onboard at their own pace, following documentation written in Tech dos, and granting them automatic access to Co-Pilot 28m50s.
• Over a thousand engineers signed up and gained access to Co-Pilot within the first two weeks, with almost no handholding or marketing required 29m9s.
• Spotify thanks GitHub Professional Services for helping them build a streamlined process that was easy to follow 29m25s.

Spotify Portal for Backstage and Future Outlook

• The company has been working towards improving their developer experience over the last 10 years, continually evolving and innovating 29m49s.
• Spotify has recently launched Spotify Portal for Backstage, a cloud offering that packages up their learning and experience, allowing users to sign up and get started easily 30m13s.
• This cloud offering comes with integrations to GitHub APIs built-in, ensuring developers can focus on delivering end-user features without worrying about configurations, updates, or maintenance 30m34s.
• The portal is accessible in the cloud, and users can learn more by visiting backstage.spotify.com 30m59s.
• Looking to the future, Spotify is excited to evaluate GitHub Enterprise Cloud with data residency to bring new features to developers in real-time, improving their productivity and happiness 31m19s.
• Some benefits Spotify is interested in include real-time access to newly released features, zero downtime for upgrades, and isolated company space compared to shared space on github.com 31m39s.

Final Thoughts and Conclusion

• Spotify has undergone significant engineering evolution over the past decade, with a focus on proper planning, testing, and security measures before implementing changes 32m10s.
• The company encourages developers to use GitHub APIs to build, integrate, and customize workflows to improve productivity and efficiency 32m36s.
• Spotify's experience with GitHub has been positive, with flexible APIs and workflows helping the company's developers to be more productive and efficient 33m6s.
• The company's biggest takeaway is to encourage developers to think outside the box and focus on end-user features to improve productivity and efficiency 32m58s.
• Spotify's 10-year developer experience was condensed into a 40-minute talk, providing a brief glimpse into the company's evolution and experience with GitHub 33m18s.
• The company invites attendees to chat with them in the speaker section and wishes everyone a great rest of the GitHub universe 33m40s.