Why We Deprecated Google Analytics (And Built a System 3x Cheaper)

Introduction and Motivation for Internal Tracking System

• Delivery Hero, a food delivery company with a central office in Germany and multiple local brands worldwide, deprecated Google Analytics and built an internal tracking user tracking service, with the company's engineering manager, Alina, overseeing the project 10s.
• The decision to deprecate Google Analytics was made due to several reasons, including the need to migrate from Universal Analytics, limitations in providing real-time data, and reaching the limit of definable events in Google Analytics, as well as GDPR concerns and cost considerations 4m42s.

Architecture and Components of the Internal Tracking System

• The internal tracking system, similar to Google Analytics, consists of a mobile SDK and a front-end SDK in TypeScript to collect data, which is then sent to an API and streamed to consumers and data storage, using Google infrastructure such as BigQuery, Pub/Sub, and Google Analytics alternatives 8m30s.
• The system has two types of consumers: real-time consumers that use Pub/Sub and others that consume data from BigQuery storage, with the initial architecture being very simple, containing only an API and two processors, but allowing for scalability and causing no major problems 10m50s.

Project Overview and Initial Rollout

• The project involved an MVP rollout, introducing the problem, architecture, testing, and rollout process, as well as further development, including optimizations such as cost optimization, solving problems, and introducing data validation to improve data quality 2m6s.
• The internal tool has provided several benefits, including unlimited event types, real-time data, and improved cost efficiency, with the cost being three times cheaper than Google Analytics, and has also addressed GDPR concerns by storing sensitive data in the company's own infrastructure 6m30s.

Performance and Cost Improvements

• The system has many additional services, including the first one API and processors, to solve problems like reliability and data validation, and it has a data validation service, curation jobs, and more SDKs to serve data producers and consumers 10s.
• After the rollout, there was a little bit of success, with an order match rate, a data quality metric, increasing by 6% from 85% with Google Analytics, which translates to a lot of money in terms of billable data, and the cost was improved by 25%, with twice the load and zero incidents of data loss 1m20s.
• The first requirement was for purchase tracking to track at least equal data as Google Analytics, and after the rollout, the system started with four brands and was able to track more data than Google Analytics, by fixing the SDK and building a reliable infrastructure 2m40s.

Metrics and Validation of System Performance

• The order match rate metric was defined by comparing orders from the back end of the food delivery system to orders that arrived with user tracking, which is a good metric because it is predictable and has a source of truth 4m10s.
• The second challenge was to not pay more for the internal tool than for Google Analytics, and a simple metric, cost per message, was defined to compare costs, and after optimizing, the system was paying 25% less after one year 5m30s.

Testing and Load Handling

• Load testing with real data helped the system to survive peak loads, such as a public holiday, and testing with three times more data than the peak load ensured that the system could handle it 7m20s.
• The system also used a pattern of testing for every hour, similar to testing for big software pieces like SDKs or Google Tag Manager removal, to ensure that the system was working correctly 9m10s.

Challenges with Google Tag Manager and SDK Rollout

• Deprecating Google Analytics also meant deprecating Google Tag Manager, and testing this removal involved doubling the pipeline, which was expensive due to the increased load and data received, resulting in higher costs 10s.
• The process of rolling out a new application version with a new SDK, which sent data twice, was complex and time-consuming, and some users were still using the old version with the testing setup at least half a year later, incurring additional expenses 1m5s.

SDK Development and Implementation Decisions

• The decision to use an internal SDK instead of the open-source Snowplow SDK was made to have control over the development and implementation of features, as relying on an external SDK would have led to slower development and potential maintenance issues 2m6s.
• The rollout of the new system involved testing and duplicating the second pipeline of Google Tag Manager, and once the new system was proven to collect at least the same amount of data, the second pipeline was duplicated, and later application versions continued with the internal SDK without Google Tag Manager 3m30s.

Lessons Learned and Best Practices

• The key takeaways from the experience include the importance of careful KPI selection to track the impact of changes, the value of load testing and parallel testing, and the introduction of a progressive rollout strategy for the backend and SDK 5m40s.
• Lessons learned from the experience highlight the need for adding logging everywhere when testing and experimenting, as the lack of logging made it difficult to debug data loss issues 8m10s.
• The system initially lacked alerting, which led to data loss incidents, and better observability was needed to address these issues, with monitoring and alerting at key points in the system 10s.

System Resilience and Reliability Improvements

• Progressive rollout and chaos testing were introduced to prevent data loss incidents, such as the one caused by an SDK rollout, and to test the system's resilience in case of infrastructure outages like the GCP outage 42s.
• After optimizing the system and solving reliability problems, the results showed an improvement in data receiving from 85% to 97%, and the system became three times cheaper than Google Analytics 2m6s.
• The system was able to handle 10 times more load than during the testing period against Google Analytics, and further improvements were made to address data completeness and accuracy issues 2m6s.

Data Governance and Accuracy Enhancements

• Data governance problems, such as null values and inconsistent data formats, were addressed by introducing metrics to measure data completeness and by implementing code-generated event models to ensure accurate data 2m6s.
• Back-end reliability was improved by solving problems such as data loss on pod restarts, and cost improvements were made by optimizing metrics and implementing SDK improvements 2m6s.
• The introduction of code-generated event models helped to improve data accuracy by allowing developers to receive errors at compile time, ensuring that data is sent in the correct format 2m6s.

Further System Optimization and Cost Reduction

• The system was improved by using code generation from schemas to automate data governance, which helped to solve the problem of having global requirements for events and local properties needed only for local brands, and this approach also helped with data governance by storing event models in one place 10s.
• The reliability of the system was improved by making every request synchronous, which increased latency but prevented data loss, and the introduction of gRPC and a doubled flow also contributed to the improvement 2m6s.
• Cost improvements were made by introducing data archival, which reduced costs by storing less frequently accessed data in cheaper storage, and by storing JSON fields as JSON instead of text, which lowered costs by 20% 2m6s.
• The system's cost was further reduced by using on-demand nodes instead of standard nodes, and by using cheaper storage options while maintaining reliability 2m6s.

SDK Improvements and Data Handling

• The SDK was improved by introducing a queue that stores events before they are sent to the API, and a work manager that reads from the queue and sends events in batches, which helps to prevent data loss and ensures that the system works in a non-blocking manner 2m6s.
• Monitoring was introduced to track the queue and prevent data loss incidents, and event prioritization was also introduced to ensure that more important data, such as billable data, is prioritized over less important data 2m6s.

Data Prioritization and Validation

• The importance of data is categorized into layers, with some data being more important than others, and exponential back off is used as an industry standard for sending data 10s.
• Data validation is crucial and has changed the support line, as most errors are now caught during development time, and developers are considered stakeholders who use the SDK and need to produce better data 2m6s.
• The product was initially built for product analysts, but now it also considers developers, who use the product and need to produce high-quality data, and this change has helped a lot 2m6s.

Architecture Design and MVP Implementation

• A simplistic but scalable architecture was built, which saved a lot of time and helped to avoid potential issues, and an MVP was created and tested before being rolled out gradually 4m30s.
• The system has undergone further improvements, including the addition of features like data validation, event forwarding to third parties, and serving as a source for marketing campaigns on platforms like Facebook 6m10s.

SDK Optimization and Scalability

• The SDKs were refactored to be as small as possible, which was important for some markets with limited phone memory, and code generation was optimized to handle many event types while keeping the code small 8m40s.
• Measuring data completeness and improving cost are ongoing processes, and having an in-house solution provides full control over implementation, priorities, and cost improvement, which also makes compliance officers happy 10m50s.

Expanding Use Cases and Reuse of the System

• The in-house tool is being reused for more use cases, such as marketing and storing application metrics data, like Firebase, which allows for similar storage to serve additional use cases 12m30s.
• Developments are being made around experimentation data, which is very data-intensive, and the mobile SDK is being refactored to improve data collection by 1 or 2%, which could result in significant cost savings 10s.

Data Visualization and Dashboard Management

• The analytics dashboard is being managed using Looker Studio, which serves its purpose despite being described as ugly, allowing for data visualization, filtering, and drawing conclusions 2m6s.

Data Loss Prevention and Synchronous Processing

• The issue of data loss when pods die has been solved by making the data collection process synchronous, so if a pod dies, it returns a 500 error and the client resends the data 4m14s.

Application of Tracking Across Different Platforms

• The system is being used to track data for various applications, including mobile and web apps, as well as internal tools, such as the application used by deliverers and the help page 5m23s.
• The tracking requirements for internal applications differ from those for mobile applications, with a focus on user journey and event validation, and a new pipeline may be built to skip validation for certain use cases 7m14s.

Alternatives Considered Before Internal Development

• Before building their own system, alternatives considered included Google Analytics 4 (GA4) and the Snowplow SDK, as well as other mobile SDKs, but GA4 and Snowplow were the main options 9m30s.