Optimizing Java Applications on Kubernetes: beyond the Basics

Introduction and Overview of Java at Microsoft

• The speaker works for Microsoft, specifically in the Java Engineering Team, and has previously worked for Oracle, and is a Java developer who uses a Mac. 38s
• Microsoft has its own JDK and helps internal teams optimize Java workloads, with examples including Azure, Minecraft, and LinkedIn. 57s
• Java is widely used within Microsoft, including in Azure's control plane, Minecraft servers, and LinkedIn's production environment, with hundreds of thousands of JVM instances running. 1m22s
• The talk will cover four main topics: optimizing Java workloads on Kubernetes, container size and startup time, JVM defaults and ergonomics, and Kubernetes' impact on Java workloads, as well as a concept called AB performance testing in production. 3m5s
• The talk is not an advanced JVM tuning class, and attendees are encouraged to seek out other resources for learning about JVM tuning, garbage collection logs, and other related topics. 3m50s
• The talk aims to provide pointers and opportunities for attendees to learn more about optimizing Java workloads on Kubernetes. 4m28s

Optimizing Container Image Size

• Reducing the size of container images is a topic of interest, with the goal of improving startup time and overall performance. 4m41s
• Reducing the size of a container image is important, but it's not the most critical factor, especially when running infrastructure on a data center with high-speed internet, as the download speed may not be significantly impacted by the image size 4m52s.
• However, reducing the size of an image is crucial for security reasons, as it minimizes the attack surface area by reducing the number of components shipped in the image, making it easier to patch and update, and decreasing dependencies in production 5m40s.
• Removing unnecessary dependencies can also make the image easier to audit, which is essential for supply chain security and component governance 6m16s.
• To reduce the size of an image, there are three main areas to focus on: the base image layer, the application layer, and the JVM runtime 6m33s.
• For the base image layer, using slim versions of Linux distributions, such as Alpine, or building a custom Linux base image can be effective 6m52s.
• For the application layer, only including necessary dependencies and breaking down the layer into separate layers for dependencies and application code can help reduce the image size 7m12s.
• Using caching can also help, as it allows for reusing the dependencies layer if it hasn't changed, and only rebuilding the application layer 7m43s.
• Running as a non-root user and using a JVM runtime that can shrink down to only include necessary components can also help reduce the image size 7m58s.
• The JDK project's module mechanism can be used to create a JVM runtime with only the necessary modules for the application 8m16s.
• Building a native image can also be an option for further reducing the image size 8m27s.
• Examples of size differences between images include Ubuntu, dban, and Alpine, with Alpine being a smaller option, but requiring consideration of its musl libc library and potential compatibility issues 8m34s.
• The JDK is compatible with Alpine, but there may be issues, so it's essential to test and ensure compatibility, and also note that commercial support from cloud vendors for Alpine might be limited 9m6s.
• A classic Docker file for a Spring application can be improved by using a custom user instead of running as root, and by separating dependencies into different layers to optimize build and image download 9m30s.
• Using the Spring Boot Maven plugin can automate the process of building an optimized Docker image 10m22s.
• Creating a custom Java runtime with only the necessary bits can significantly reduce the JVM size, from 334 megabytes to 57 megabytes, and using GraalVM native image can reduce it further to less than 10 megabytes 10m27s.

Improving Startup Time

• The JDK's class data sharing feature can improve startup time by half, and future projects like Project Loom and Project Crack aim to further improve startup time 11m4s.
• Project Loom, led by Oracle, and Project Crack, led by AO Systems, are working on checkpoint restore technology, which can significantly improve startup time, but requires framework, library, and runtime support 11m34s.

JVM Defaults and Ergonomics

• The JDK has default settings that can be optimized for better performance, and understanding these defaults is essential for optimizing Java applications 13m1s.
• Java runtime stack has defaults that tend to be conservative and work for most applications, but these defaults can be optimized for specific use cases 13m9s.
• JVM ergonomics can affect how the JVM runs, and environment settings can impact the JVM's behavior, such as the number of processors available 14m12s.
• When running a Java application in Docker, the JVM may not see all available processors, depending on the Docker configuration 14m29s.
• If a container is set to use a non-integer number of CPUs (e.g., 1.2 CPUs or 1200m in Kubernetes), the JVM will round up to the nearest whole number of processors 15m27s.
• Memory settings can be tricky, and the JVM's garbage collector and heap size can be affected by the amount of memory available 15m58s.
• The JVM's garbage collector can be selected based on the amount of memory available, and the heap size can be set to a percentage of the total memory 16m30s.
• If the JVM is not properly tuned, it can result in a bad heap configuration, leading to performance issues 17m7s.
• The G1 garbage collector is selected when the JVM has 2 CPUs and 1792 megabytes of memory 18m24s.
• The JVM's default heap size is 50% for environments with less than 256 megabytes, and then it remains stable at 127 megabytes up to 512 megabytes, after which it is set to 25% of the available memory 19m40s.
• The JVM's default settings were designed for shared environments, but in the container world, the JVM needs to be informed manually about the available resources 20m20s.
• There are ongoing projects, involving companies like Microsoft, Google, and Oracle, to enhance the ergonomics and defaults of the JVM for container environments 20m40s.
• Simply packaging a Java application as a JAR can result in wasted resources, and proper configuration is necessary to optimize performance 20m51s.
• There are various garbage collectors in the JVM, including the ZGC, Shenandoah, and G1, and understanding their differences is important for optimization 21m0s.
• The Absen GC is a garbage collector that does not collect anything, making it useful for benchmarking applications but not suitable for production environments 21m8s.
• When running applications in the cloud, it's essential to consider that certain areas of memory, such as the Met space or code cache, will consume the same amount of memory regardless of the heap size 21m40s.
• Configuring the JVM involves setting the heap size to 75% of the memory limit, and using memory calculators can simplify the process 22m20s.
• Build packs, such as those provided by the Paketo project, can offer optimizations for containers and include memory calculators for building Java workloads 22m29s.

Optimizing Java Workloads on Kubernetes

• Java applications on Kubernetes can be optimized by understanding JVM tuning, as setting xmx is not always necessary and can be calculated automatically with tools like Peto build packs 22m50s.
• Horizontal Pod Autoscaler (HPA) is a common scaling solution, but it can be expensive and may not be the most effective approach, as it involves adding more computing power rather than optimizing resource usage 23m30s.
• Vertical Pod Autoscaler (VPA) is an alternative scaling solution that allows pods to increase their resource allocation without restarting the container, but it requires the runtime to understand and take advantage of the additional resources 26m3s.
• The JVM currently lacks the capability to effectively utilize VPA, but this is being worked on, and other runtimes like Rust may offer better performance in some cases 26m35s.
• A company in Latin America migrated from Java to Rust to improve performance, but this could have been achieved through JVM tuning and resource redistribution, highlighting the importance of understanding JVM behavior 24m56s.
• Google's Cube startup CPU boost is a feature that allows containers to access additional resources for a limited time, and can be used on Azure, offering a potential solution for optimizing JVM performance 26m51s.
• Java applications on Kubernetes can initially require more CPU and memory to start up, but CPU usage can be reduced and stabilized after the JVM has completed its initial work, such as JIT compilation and optimization, allowing for more efficient resource allocation 27m16s.
• The main issues people face when running Java applications on Kubernetes include limited memory and CPU throttling, which can cause latency and impact application performance 28m1s.
• CPU throttling occurs when an application is given a limited amount of CPU time, but the JVM and other runtime components require additional CPU resources, leading to delays and increased latency 28m50s.
• Setting a CPU limit can impact the JVM's performance, as it only allows the application to access a certain amount of CPU time within a specified period, and any additional CPU requests will be delayed until the next period 29m45s.
• The JVM's garbage collector also requires CPU time to perform its tasks, which can further reduce the available CPU time for the application and increase latency 30m56s.
• Understanding CPU throttling and its impact on the JVM is crucial to optimizing Java application performance on Kubernetes and ensuring that the application has sufficient resources to perform its tasks efficiently 29m28s.
• The JVM's active processor count flag can be used to specify the number of processors available to the JVM, which can be different from the actual CPU limit, and this can be useful for IO-bound applications 32m0s.
• Most microservices on Kubernetes are IO-bound, as they involve network requests and responses, and the active processor count flag can be used to optimize the JVM's thread pool size for such applications 32m40s.
• Microsoft has provided recommendations for JVM settings on Kubernetes based on CPU limits, and these can be used as a starting point for optimizing JVM performance 32m59s.
• When optimizing JVM performance, it's essential to have a clear goal in mind, such as throughput, latency, or cost, and to use the appropriate garbage collector for that goal 33m31s.
• Resource distribution can significantly impact JVM performance, and reducing the number of replicas while increasing CPU and memory allocation can lead to improved performance and cost savings 34m14s.
• A benchmark study showed that reducing replicas from six to two while increasing CPU and memory allocation improved throughput and latency while reducing costs 35m41s.
• Resource redistribution can be applied to any language and is a viable strategy for optimizing performance and reducing costs in Kubernetes clusters 36m18s.
• To optimize Java applications on Kubernetes, one approach is to merge a few pods to improve performance on those nodes, and then apply the rollout to more nodes with only one replica per node, which can be achieved by writing a Kubernetes operator 36m38s.
• Another approach is to increase the node pool to taller VMs, increase the resource limits of the pods, and have only three replicas, which can provide spare resources for more workloads while keeping the cost the same 37m20s.
• This approach allows for resiliency and provides the ability to do more with the same resources, making it beneficial from a cloud vendor perspective 37m48s.

A/B Performance Testing in Production

• The concept of A/B performance testing can be applied to production performance, where a load balancer routes loads to different instances of the application configured differently, such as using different garbage collectors or resource limits 38m13s.
• This approach can be used to test different configurations, such as smaller JVMs with more replicas or taller JVMs with lesser replicas, and can be easily implemented on Kubernetes 39m4s.
• An example of A/B performance testing is using NGINX to route traffic to different instances of the application, and using round-robin or least connection patterns to distribute the load 39m11s.
• Another scenario is to use different garbage collector configurations and tuning, such as using ergonomics default JVM, G1GC, and parallel GC, and configuring the deployment to use least connection or round-robin 39m41s.
• By combining these approaches, it is possible to achieve optimal performance and resource utilization on Kubernetes, as shown in the Azure dashboard 40m7s.
• A demonstration is shown using a CPU-bound emulation, specifically a prime factor test, to compare the performance of different deployments in a Kubernetes cluster 40m34s.
• The test is run on different topologies, including 2x2 and 2x4, and the results are displayed in a dashboard, showing live metrics such as request rate and CPU usage 42m30s.
• The demonstration highlights the ability to compare the performance of different deployments in production, including different garbage collectors, JVM tuning flags, and parameters, without affecting the application's functionality 43m43s.
• The test is run on a container in a cluster, and the results are displayed in a dashboard, allowing for real-time comparison of the performance of different deployments 41m41s.
• The demonstration also shows how to define roles for different deployments using environment variable names, which can be used to differentiate between deployments in the dashboard 43m4s.
• The importance of testing in production is emphasized, as it allows for a more accurate evaluation of performance under real-world conditions, which may not be replicable in a lab environment 44m6s.
• The demonstration is run on a live cluster, and the cost of running the demo is mentioned, highlighting the importance of efficient resource usage 44m18s.

Key Takeaways and Future Directions

• The main takeaways for optimizing Java applications on Kubernetes include reducing the size of container images, focusing primarily on security rather than size, and addressing potential issues with loading images into nodes. 45m37s
• Startup time can be optimized by utilizing JVM features such as Class Data Sharing, which is available in Java 11, 17, and 21, and evaluating Project Crack and Project Loom for modernization. 46m18s
• Understanding the runtime defaults and capabilities of the JVM is crucial, and observing memory, CPU, garbage collection, and J compilation in production can help identify areas for improvement. 46m39s
• It is essential to understand the impact of resource constraints on the runtime stack and ensure sufficient resources are allocated for proper behavior. 46m59s
• Horizontal scaling is not a silver bullet, and vertical scaling should also be considered to optimize performance. 47m16s
• Performance 20 in production is expected to be a significant focus area, and utilizing staging environments for testing can be beneficial. 47m22s
• Microsoft is researching the addition of CRA support in the OpenJDK distribution, and while there are some complexities, frameworks like Spring have implemented checkpoint restore flows. 48m3s
• Framework teams are required to build capabilities into the application framework for certain features, and efforts are being made to explore this possibility 49m10s.
• JSR (Java Specification Request) does not specifically focus on enhancing the JVM for performance issues, but there are ongoing projects by Google, Microsoft, and Oracle to make the JVM heap dynamic, allowing it to grow and shrink as needed 49m29s.
• A dynamic JVM heap would enable in-place scaling and vertical scaling capabilities to be taken advantage of by the JVM 49m51s.
• Oracle is working on the ZGC (Z Garbage Collector) to address issues with garbage collectors defining memory areas and managing objects 50m5s.
• Oracle is also working on adaptable heap sizing for the ZGC, while Google has done work on the G1 GC, and Microsoft is exploring serial GC for dynamic heap management 50m17s.