Evolving Trainline Architecture for Scale, Reliability and Productivity

Introduction and Overview of Trainline

• The presentation will cover lessons learned from scaling Trainline's architecture, including handling more traffic, enabling multiple engineers to work on the architecture simultaneously, and scaling the efficiency of the platform for cost-effectiveness and growth 38s.
• Trainline is Europe's number one rail digital platform, retailing rail tickets for users worldwide and providing services throughout the rail journey, including platform information, disruption assistance, and compensation for delays 1m42s.
• The company provides its services through its B2C brand, Trainline.com, as well as a white-label solution to partners in the carrier space and the wider travel ecosystem 2m24s.
• Trainline is a public company, well-established, and profitable, with a significant size of business, although specific numbers will be discussed later in the presentation 3m35s.
• The presentation will also cover the business lens of productivity, team impact, cost efficiency, and financial business impact, in addition to actual traffic growth and handling more business 1m16s.
• The speaker will discuss how Trainline has made its architecture possible for more engineers to work on it simultaneously, allowing for a faster pace of growth and innovation 56s.
• The presentation will last around 25-30 minutes, with time for questions at the end, and attendees are encouraged to note down any questions that arise during the presentation 3m19s.
• Trainline has a significant scale in terms of technical impact, with around five billion net ticket sales and 350 searches per second for journeys and origin-destination pairs across 3.8 million monthly unique routes 3m45s.
• The company has around 500 people in its tech and product organization, with the majority being tech professionals 4m28s.
• Trainline has real-time information on the location of each live train in Europe, which presents a large problem space in terms of data and actions required when trains are delayed, canceled, or changed 4m46s.
• The company has over 270 API integrations with individual rail and bus carriers, with zero standardization in this space, resulting in high maintenance costs and non-trivial integration challenges 6m27s.
• The lack of standardization in rail APIs is in contrast to the airline industry, which has standardized APIs through systems like Amadeos 6m48s.
• The complexity of the problem space was not immediately apparent, but it became clear over time that certain problems, such as the aggregation of supply, were harder than initially thought 6m13s.
• The rail industry faces a problem with inconsistent and disintegrated APIs, which have different access patterns and limitations, such as look-to-book ratios and rate limits, making it challenging to aggregate supply from various rail companies 7m17s.
• Europe has 100 times more train stations than airports, adding to the complexity and scale of the problem, particularly when it comes to handling journey searches across multiple APIs 8m11s.

Challenges in the Rail Industry

• The aggregation of supply in the rail industry is unique, but the problem of handling transactions over a finite inventory is not, and is similar to the classic "Ticket Master problem" 8m34s.
• Selling seats on unique trains with limited inventory is more complex than selling digital products, as it requires checking inventory and handling transactions reliably and quickly 9m3s.
• The company currently handles around 1300 transactions per minute at peak times, which has grown from 800 in the past three years, partly due to the recovery of rail travel and the company's growth in Europe 9m33s.
• The speed at which people expect to receive their tickets, literally within a second, adds to the complexity of the problem, requiring instant fulfillment 10m33s.
• When buying rail travel, most people purchase tickets a couple of months or weeks in advance, but about 60% of tickets bought on Trainline are purchased on the day, often just before boarding the train, requiring instant processing and industry-level standard interactions with processes to validate the ticket at barrier gates 10m48s.
• The expectations for processing and validating tickets are high, involving complex interactions with industry-level processes to ensure the ticket is recognized as valid at barrier gates 11m32s.

Lessons Learned from Scaling

• The talk will cover three lessons learned from scaling, focusing on team and productivity, cost efficiency, and scaling with growth in traffic and achieving higher reliability 11m45s.
• The first lesson will focus on the impact of architecture on team productivity, highlighting how it can both enable and hinder progress as team sizes change 11m51s.
• The second lesson will discuss cost efficiency and scaling the efficiency of the platform 12m5s.
• The third lesson will cover scaling with growth in traffic, achieving higher reliability, and dealing with availability issues 12m10s.

Team Productivity and Organizational Structure

• When the speaker joined Trainline in July 2021, the company had around 350 engineers organized in a cluster model, with teams focused on specific parts of the technical stack, such as Android, iOS, web, and backend 12m49s.
• However, this organizational structure led to low team productivity, as most projects required collaboration between at least five and often up to 10 different teams, resulting in complex project management and delays 13m43s.
• The previous team structure was slow and not suitable for the scale, leading to a massive reorganization in January 2022, where the team was restructured into a platform and verticals model, with platform owning the technical stack and verticals having people with different skill sets, shaped around clear ownership of product and business goals 14m26s.
• The platform and verticals model improved alignment of the team to goals, but came with challenges like tension between platform and vertical teams, where vertical teams wanted to deliver quickly and platform teams wanted to ensure proper implementation and refactoring 15m14s.
• The tension between platform and vertical teams is an embedded function of the model, but can be frustrating for people on both sides, and is a good thing to have, but sometimes needs to be managed 15m46s.
• Recently, the team reorganized again, reducing the ownership of the platform to very core services and moving 50% of the tech surface to the verticals, which were renamed diagonals, to get the best parts of both models and streamline the process 15m58s.
• The new model aims to remove some of the tension between platform and verticals, and the team is trying to find the right balance between the two 16m33s.
• The key question in all three models is who owns each part of the technical surface, who is in charge of sustain work, mandatory technical upgrades, and driving the technical strategy and vision for that part of the codebase 16m57s.
• The team needs to determine who effectively does the core work to sustain and drive the technical roadmap, in addition to who builds all the features 17m26s.
• The architectural implications of these changes are relevant to the audience, and the team is trying to find the right balance between different models to achieve their goals 16m46s.
• The team's goals and alignment can be defined by answering two questions: which product or business or tech goal is the team on the hook for, and what are the key performance indicators (KPIs) for that goal, with different teams having different answers to these questions 17m37s.
• The three key performance indicators (KPIs) are A (alignment of engineering investment to business goals), P (productivity), and Q (quality of technical work), which translate to the risk of adding technical debt or making the platform worse or better 18m16s.
• In the past, clusters were used, but they were poor for alignment, as engineers didn't care about product or business goals and only focused on their part of the code, resulting in poor end-to-end productivity and good quality due to people working in a restrained small part of the technology surface 18m35s.
• When the team moved to a platform of verticals, alignment became super crisp and clear, with engineers on the hook for specific business goals, resulting in perfect alignment, better productivity, and good quality, but with some tension between verticals and the platform team 19m7s.
• The current model has removed the platform team's policing of contributions, slightly diluting the clarity of alignment, but enabling productivity, and it's essential to note that different models work better at different times for a company, and shifting the model can bring a different balance and kind of productivity 20m5s.

Architectural Implications and Ownership

• Customer and business needs do not respect architectural boundaries, and it is essential to acknowledge this fact when designing a platform 20m54s.
• Even with a well-designed platform, business priorities and needs can change, requiring the architecture to evolve accordingly 21m29s.
• Business strategy, technology ownership, and organizational structure will also change over time, and it is crucial to adapt to these changes 21m39s.
• Conway's Law states that technology ends up getting the shape of the organization that makes it, and how teams are organized affects the technology built 22m18s.
• There is no perfect reverse Conway maneuver, making it challenging to change the technical architecture designed by a certain organizational structure 22m39s.
• It is essential to build technology and architectures with the fact of ownership transfers and external contributions in mind 23m7s.
• Enforcing consistency is crucial, and leaders should establish a company-wide approach to technology, rather than allowing individual preferences to dictate the way things are done 23m34s.
• This approach is necessary because team members and structures are likely to change over time, and a consistent approach ensures that technology can be maintained and updated efficiently 23m46s.
• To achieve scale, reliability, and productivity, it is essential to enforce consistency within an organization by using as few languages and technologies as possible, even if it means sacrificing individual autonomy, to make it easier to transfer ownership and allow for external contributions 24m8s.
• Consistency is key, as it allows for the transfer of "Lego blocks" rather than custom-built items, making it easier to reassemble them in different ways 24m19s.
• Trainline has been around for 20 years and still has technology in production that was written 15 years ago, highlighting the importance of keeping things consistent with few languages and technologies 25m3s.

Cost Efficiency and Optimization

• Production costs are a significant concern, with Trainline's AWS bill accounting for about 25% of their overall software engineer compensation bill 26m11s.
• The cost of the platform was growing faster than traffic, prompting the need to take a goal to run things in an efficient way and make sure the organization is disciplined and running a tight ship 27m14s.
• The goal is not to make drastic cuts but to ensure the organization is running efficiently and making the most of its resources 27m20s.
• The goal was to drive down the annual run rate of production costs by 10% to reduce the entire annual bill. 27m26s
• The team has a massive surface area with over 700 microservices and more than 100 databases, making it challenging to identify areas for cost reduction. 27m43s
• To achieve the goal, the team considered various levers, including cleaning up unneeded data, consolidating non-production environments, reviewing old low-value services, and right-sizing the platform. 28m25s
• They also looked at data retention policies and reviewed architectural choices, such as the use of cloud functions and lambdas, to determine if they were efficient. 28m55s
• The team decided to delegate the problem to individual teams that own parts of the technology stack, tasking each team with driving 10% of the bill down for their area. 29m58s
• The plan was to use attribution to track what's driving the costs and hold each team accountable for reducing their costs. 30m6s
• A goal was set for each team to reduce their part of the bill by 10%, but this led to some teams taking risks and doing things that ultimately caused problems, such as outages, due to underprovisioning of services 30m39s.
• Instead of reviewing architecture choices and making changes, most teams simply scaled down their services, which sometimes worked but also led to outages 31m27s.
• In a three-month period, there were four outages caused by underprovisioning, which occurred when the platform hit peak traffic, usually on Tuesdays 31m42s.
• The goal of reducing costs by 10% was achieved, but it was not the best value for money, as engineers spent more time than necessary, and there were unintended consequences, such as outages 32m28s.
• Cost management is important for the long-term efficiency of the platform, but understanding where to make cuts in a large, fragmented microservice-based system requires centralized thinking and cannot be delegated to individual teams 32m55s.
• Predicting which cost-reducing efforts are worth it can be tricky, especially for those without a full understanding of the system, and blindly pushing down cost-saving goals to individual teams can lead to more problems than solutions 33m21s.
• A centralized task force that works with individual teams to evaluate where investments to save costs are worth it would be a better approach than delegating cost-saving goals to teams 33m42s.

Scaling for Growth and Reliability

• The third lesson learned is about scaling for growth in traffic and reliability, which includes managing cost system cost savings efforts centrally and avoiding fully delegating, as it can backfire 33m55s.
• The speaker briefly covers three big bouts of outages, which could each be a talk on their own, and asks the audience to keep the information confidential 34m26s.
• The first outage occurred in October 2021, when Trainline went down for four hours one day and two hours the next day, due to the platform struggling to handle the sudden increase in traffic after the COVID-19 pandemic 34m53s.
• The cause of the outage was the contention in terms of database connections, as many new microservices had been added, each maintaining connections to the database, leading to a bottleneck in the relational databases 36m16s.
• The relational databases were hosted on a single machine, which couldn't keep up with the many connections, but the issue was eventually tweaked and tuned, and the platform survived the period 36m43s.
• A year later, in October, another outage occurred, also related to the database, but this time involving old Oracle databases 37m14s.
• The company experienced a significant outage due to a gradual increase in load on the orders database, which was caused by the addition of features related to the journey experience, such as following a journey and receiving notifications about platform changes, over the course of a year 37m46s.
• The company's observability was primarily focused on transactional flows, and as a result, the increase in load on the orders database went unnoticed until it caused an outage 38m24s.
• The company had to implement database-related fixes to resolve the issue and prevent similar outages in the future 38m55s.
• The company recently experienced a series of DDoS attacks, which may have been attributed to nation-state actors, and had to tighten up its DDoS protections and make other changes to mitigate the attacks 39m4s.
• Despite initial concerns that the platform was being DDoS attacked again, it was discovered that the issue was actually caused by sloppy retry strategies throughout the stack, which allowed small issues to snowball into larger problems and eventually bring the platform down 39m56s.
• The company did not have a coordinated retry strategy, which contributed to the problem, and everything from client-side retries to backend service retries ended up creating a 10x load that brought the platform down 40m25s.
• The architectural lesson learned from past experiences is that none of the issues were caused by a single team, change, or regression, but rather by a buildup of small problems over time, making it difficult to predict and detect bottlenecks in a large microservice-based system 40m51s.
• Predicting bottlenecks in such systems is challenging due to the complexity and spread of microservices, with each team chasing their own goals and contributing to the overall problem, often resulting in a tragedy of the commons 41m40s.
• The best approach to handling this issue is to regularly review longer-term traffic mix or load changes, such as reviewing changes in critical databases or services over a period of six months, to identify potential bottlenecks and guide teams accordingly 42m15s.
• Service fleet coordination is critical in guiding teams and ensuring a strong architecture function or principal engineering function to oversee the big picture and prevent issues from arising due to individual team ownership and decisions 42m53s.
• Observing over longer terms and coordinating microservice leads are essential lessons learned from past experiences, with consistency being key to productivity in the long run, especially for startups or seed-stage companies 43m30s.
• When building a business where technology should survive for five to 15 years, it is essential to insist on consistency and manage system cost-saving efforts centrally, even if engineers may not love it, to avoid losing the wider context and to coordinate microservice fleets to avoid outages 43m52s.

Knowledge Transfer and Service Ownership

• Building an architecture towards changing the structure in people can be challenging, as it goes against optimizing for knowledge and change management in the near term, but it is crucial to balance short, medium, and long-term goals 45m3s.
• The "build it, you own it" strategy is effective for the first six to 12 months of a service, but after that, the person who built it is likely to move on, and the service needs to be handed over to others, requiring a transition period to ensure adoption and knowledge transfer 45m39s.
• The platform verticals model can be used to facilitate this transition, where a vertical builds a new service and is on the hook for it for six to nine months, with the relevant parts of the platform advising through that period 46m17s.
• The goal is to create "Lego blocks" that anyone can pick up and take care of, even if they are not an expert, to enable an agile organization that can focus on the most important things 47m10s.
• The transition period can be challenging, and it is essential to pull teams back from the mindset of having a single person who knows the service, as this can cause contention and make it difficult to touch the service 46m49s.
• The "bus factor of one" is a common problem, where only one person knows the service, and it is essential to avoid this by creating a culture of knowledge sharing and transfer 46m56s.

Microservices, Consistency, and Technology Adoption

• Microservices were initially adopted for team autonomy, allowing different languages to be used, but now consistency is considered key, raising questions about its future and potential downsides 47m41s.
• Having multiple languages for front-end development, such as Android and iOS, and different back-end languages, like .NET and Ruby, can lead to fragmentation of skill sets within an organization 48m14s.
• This fragmentation can make it challenging to assemble cross-functional teams to deliver simple features, requiring a large number of people with different skill sets 48m44s.
• The complexity of native platforms like Android and iOS, each with their own languages, contributes to this challenge 49m11s.
• To address this, it's essential to strike a balance between allowing innovation and trying new things, while also having a path for making successful new technologies official and widely adopted 49m30s.
• Allowing everyone to choose their own technologies without a clear path for adoption can lead to unmanageability, especially as the business grows 49m44s.
• A successful business needs a clear strategy for managing technology adoption to avoid complications in the long run 49m55s.