70,000 Lines of Code, 7 Languages: How to Scale via Rust Core

Architectural Motivation and Goals

• The Temporal SDK team utilizes an architectural pattern featuring a shared core written in Rust, with language-specific layers built on top of it 5s.
• A single local activity state machine in Temporal is represented by approximately 800 lines of code 25s.
• The complete set of state machines for the platform comprises roughly 7,000 lines of code, while the entire repository contains approximately 70,000 lines of code 45s.
• Because this complex logic must reside on the client side within the SDKs rather than on the server, writing the code separately for each of the seven supported languages is considered impractical 55s.
• Temporal provides SDKs in multiple languages to meet users where they are, avoiding the limitations associated with proprietary graph-based domain-specific languages 1m20s.

Platform Reliability and Maintenance

• Temporal functions as a platform for durable execution, taking user-written code and applying various resilience guarantees 1m45s.
• The SDKs are designed to be "fat" clients because performing remote procedure calls for every minor piece of logic would not scale 2m5s.
• Reliability is the primary business priority, as any bugs in the SDKs would result in a loss of customer trust 2m25s.
• Consistency across different language implementations is a critical requirement to prevent divergent behaviors 2m35s.
• Maintainability is a key goal for the SDK team, which has grown from four or five members to ten 2m45s.

Development Strategy and Language Selection

• The organization currently supports seven programming languages, with a team of ten developers managing the codebase across these languages 0s.
• A primary goal for the development team is to scale the codebase to accommodate a larger number of contributors 25s.
• The team prioritizes providing an idiomatic experience for users, ensuring that SDKs feel natural and utilize patterns familiar to developers within their respective languages rather than simply piping data through a Rust core 35s.
• Performance is a significant consideration because library code running on client-side infrastructure impacts the cloud costs incurred by users 1m5s.
• Writing and maintaining the same logic multiple times across seven different languages was determined to be an unviable strategy, necessitating a shared code solution 1m25s.
• Rust was selected as the shared language due to its safety, speed, expressiveness, and portability across various platforms, including Linux, macOS, and Windows, as well as ARM and x64 architectures 1m45s.
• Rust was chosen over C because of the desire to avoid the complexities of writing in C, and over Zig due to Rust's focus on memory safety, expressiveness, and the strength of its community 2m15s.
• C compatibility is a critical requirement for the architecture, as the C Foreign Function Interface (FFI) is the established standard for cross-language memory boundaries, particularly since Rust lacks a stable binary interface for exporting functions directly 2m5s.

Core and Bridge Layer Architecture

• The adopted architecture involves a core component that interacts with a service, such as a Temporal worker, though this pattern is applicable to any system requiring similar deployment structures 2m55s.
• The software architecture is organized into three distinct layers: a Rust core containing shared logic, a bridge layer, and language-specific SDKs 0s.
• The Rust core serves as the foundation, while the bridge layer acts as the glue connecting the core to the language-specific SDKs 0s.
• Language-specific SDKs are the primary interface for developers, who ideally should not need to interact with or be aware of the underlying bridge or core layers 0s.
• Each language typically has its own bridge, though Swift and C# share a single bridge implementation 0s.
• Bridge layers are written in Rust and are defined as crates located within the respective language-specific repositories 35s.
• All bridge layers communicate with the Rust core using C Foreign Function Interface (CFFI) 35s.
• While some languages utilize helper libraries to abstract the CFFI calls, others lack such tools, requiring the direct exposure and manual calling of C-compatible functions 35s.
• Bridge layers are intended to be thin, as they serve only as mechanical conduits between the core logic and the user-facing SDKs, though they remain non-trivial to implement 1m5s.

Language-Specific Integration Libraries

• Python utilizes the PyO3 library, and Node-based TypeScript/JavaScript utilizes the Neon library to facilitate Rust integration 1m25s.
• Ruby uses the Magnus library, which provides some assistance but offers fewer features than PyO3 or Neon 1m25s.
• For Swift and .NET, no sufficiently mature helper libraries were available, necessitating the manual, handwritten creation of C-exported representations 1m25s.
• Apple authored a Swift integration tool that utilizes the existing Rust core, representing an external contribution to the project 1m25s.

Type Conversion and Memory Management Challenges

• Type conversion is a primary challenge when scaling via a Rust core, as data must be represented in both Rust and the target language layer, requiring a bridge that can become quite verbose 0s.
• Asynchronous programming presents integration difficulties because Rust futures, TypeScript promises, Python futures, and Ruby fibers do not map directly to one another, necessitating custom bridging solutions 23s.
• Memory management can be simplified by using existing helpers, but writing C-style code manually requires the use of unsafe Rust, which is complex and challenging 42s.
• While writing unsafe Rust is difficult, it offers the advantage of restricting potential issues to the bridge layer rather than the entire codebase 1m3s.

Interface Definition Languages and Serialization

• Interface Definition Languages (IDLs) like Protobuf can automate code generation for type conversions, though they often fail to provide ergonomic types that are pleasant for developers to work with 1m15s.
• Developers often deal with multiple type representations, including ergonomic types for logic, IDL-generated types for data transmission, and handwritten types for specific bridge scenarios, all of which require significant time and effort to convert between 1m45s.
• Protobuf is utilized within Temporal specifically because the service communicates via gRPC, making the reuse of existing Protobuf definitions more practical than rewriting them 2m18s.
• Rust’s macro system provides an effective technique for reducing the verbosity of type conversion code by automating repetitive conversion tasks 2m45s.
• Instead of manually writing repetitive code to map fields between types, macros can be utilized to automatically generate the necessary code for all types. 0s

Bridge Interface Design and Performance Optimization

• It is advisable to avoid using complex Rust types, such as enumerations with generics or intricate type bounds, within a bridge interface. 12s
• Complicated types can lead to non-obvious issues when passed across a C bridge, making it more practical to use simpler representations and perform conversions on either side of the bridge. 12s
• Protobuf is categorized as a serializing Interface Definition Language (IDL) because it requires serialization into a wire format, which incurs a performance cost. 42s
• While it is intuitive to assume that creating a JavaScript object directly from Rust using Node's C APIs is faster than serializing to JSON and deserializing, Node's object creation APIs can be unexpectedly slow. 42s
• JSON serialization in the V8 engine is highly optimized, leading to scenarios where serializing to JSON might theoretically outperform direct object creation. 1m25s
• Benchmarks conducted on a specific use case showed that direct object creation remained faster than JSON serialization, though the performance advantage diminished as the complexity and number of fields in the object increased. 1m45s
• Because performance results can vary based on the specific use case, it is essential to measure performance rather than relying on assumptions. 2m15s
• While serializing data within the same process may seem wasteful or pointless on a principled level, the practical benefits of using an IDL—such as avoiding the manual creation of numerous types—may outweigh the minor CPU cost of serialization. 2m15s

Asynchronous Programming and Threading

• Adopting a Rust core architecture is primarily beneficial when performing tasks such as network calls, disk I/O, or asynchronous operations that utilize Rust futures. 0s
• Integrating Rust with other languages requires bridging the gap between Rust futures and the specific task, promise, or async representations used by the host language. 15s
• Helper libraries like PyO3 and Neon provide built-in mechanisms to handle the synchronization between Rust and host language async concepts. 35s
• When helper libraries are unavailable, developers may implement callbacks, where a Rust function accepts a pointer to a function from the host language, executes a Rust future, and then invokes the callback with the result. 50s
• Callback implementations can be complicated by language-specific constraints, such as single-threaded event loops or global interpreter locks (GIL) found in environments like JavaScript (V8), Python, and Ruby. 1m15s
• In languages like Ruby, which utilize a Global VM Lock (GVL), callbacks cannot be invoked from arbitrary threads created by a Rust executor like Tokio because the host language requires that code execution occur only on threads it has initialized. 1m35s
• To address thread-ownership issues, developers can use an event loop approach where callbacks are replaced by a mechanism that pushes events into a queue, which the host language then processes during its own execution loop. 2m5s
• Executing callback fulfillment requests in Ruby requires a single thread that spins and pulls requests from a queue to ensure they are processed within the Ruby environment 0s.

Memory Management and Safety in Bridges

• Integrating Rust with other languages varies in difficulty, with some integrations like PyO3 for Python being significantly simpler than others 7s.
• Memory management in cross-language bridges often necessitates writing unsafe code if helper libraries are unavailable 25s.
• Even highly skilled engineers are prone to making errors in memory management, making it essential to keep bridge code and logic as thin as possible to minimize the risk of mistakes 35s.
• When memory is allocated in Rust and passed to another language, it must be converted into a raw pointer, passed across the bridge, reconstituted into its original form (such as a box or an arc), and then freed 1m15s.
• When dealing with garbage-collected languages, memory must be protected from collection before being passed to Rust, and the host language must be notified to resume garbage collection once the memory is returned 1m35s.

Extensibility and Behavior Injection

• A significant architectural limitation involves injecting behavior into the core layer, such as when users want to introspect or modify specific RPC calls 2m0s.
• Adding specific options to accommodate individual user requests for behavior modification can lead to bloated code that provides no value to the majority of users 2m15s.
• It is recommended to plan for generic mechanisms from the beginning to allow for future introspection or behavior modification without needing to implement highly specific, one-off solutions 2m25s.
• Designers should incorporate generic callbacks into their architecture to allow for user introspection and behavior modification, rather than relying on specific, difficult-to-manage configuration knobs 0s.
• Ideally, side effects like gRPC calls should be handled by the language layer rather than the core layer, or at least routed through a callback that allows the language layer to execute the call 15s.

Distribution and Operational Challenges

• Shipping native code across different languages presents varying levels of difficulty, with the Python ecosystem being well-equipped for native extensions due to its data science requirements 42s.
• NPM package management does not support platform-specific binaries, forcing users to download binaries for every architecture whenever they take a dependency 1m5s.
• Java and Go SDKs currently exist independently of the Rust core because they were developed before the Rust project began 1m20s.
• Porting Java and Go SDKs to the Rust core is hindered by operational concerns, as users in these ecosystems often dislike running native extensions 1m32s.
• Using Cgo in Go requires users to modify their build processes and set specific build flags, which creates an undesirable burden on the end user 1m45s.

WebAssembly for Portable Execution

• WebAssembly is a promising technology for shipping portable code because it functions as a fast, portable bytecode that eliminates the need for platform-specific compilation 2m15s.
• WebAssembly offers security benefits, such as the ability to constrain compute and restrict side effects, making it suitable for running untrusted code 2m45s.
• WebAssembly extends beyond its original conceptualization as a faster version of JavaScript for web browsers, as it can be executed in contexts outside of the browser 0s.
• Using WebAssembly allows for the distribution of a single piece of bytecode rather than shipping multiple blobs for various platforms and architectures 15s.
• Rust provides strong support for targeting WebAssembly, requiring only the selection of a specific compiler back end 25s.
• The ability of WebAssembly to perform operating system-level tasks, such as disk access, depends on the specific implementation of the WebAssembly virtual machine 35s.
• Pure WebAssembly virtual machine interpreters, such as Chickory for Java and Wasmtime for Go, allow for the avoidance of native extensions because they are written in the host language 48s.
• While pure WebAssembly interpreters are effective for compute-heavy tasks, their capability to handle operating system-level functions like network or file system access varies 1m5s.
• WebAssembly enables the potential for dynamic updates, where core logic can be updated by pushing a new WebAssembly module without requiring users to redeploy the entire application 1m15s.

Advanced Optimization and Future Design

• Utilizing an Interface Definition Language (IDL) that avoids serialization, such as FlatBuffers or Cap'n Proto, can improve performance by allowing multiple languages to access the same physical memory layout 1m45s.
• Serialization can be a significant performance bottleneck, as evidenced by Temporal's core spending approximately 90% of its time on serialization processes 2m15s.
• Routing side effects—including network calls, logging, and metrics—back through the language layer provides users with customization hooks and centralizes functionality 2m30s.
• Utilizing WebAssembly for a Rust core architecture eliminates concerns regarding operating system calls, as these calls are handled by the language layer rather than the core logic 0s.

Summary of Architectural Benefits and Recommendations

• Implementing a Rust core is considered a worthwhile investment that significantly reduces the need to rewrite redundant code 25s.
• The adoption of a Rust core architecture is estimated to cut the development time for new languages in half 38s.
• Ongoing maintenance benefits include the ability to write logic once in the Rust core while still providing tailored, user-friendly APIs at the language layer 48s.
• The reduction in the total amount of code leads to a decrease in the number of bugs, although this improvement is difficult to quantify precisely 1m3s.
• Rust is recommended as a suitable language for projects that require the development of shared logic 1m16s.
• Developers adopting this architecture are encouraged to use available helper libraries to manage complex tasks 1m23s.
• Code generation is recommended to assist with development, and developers who are not constrained by specific requirements should consider using options that do not require serialization 1m28s.
• It is important to prioritize creating an idiomatic, high-quality experience for users at the language layer rather than relying solely on auto-generated code that may feel unrefined 1m43s.
• Planning for hooks—specific points where custom behavior can be injected—is a challenging but necessary aspect of design that requires significant forethought 2m0s.