GORD at Jane Street handles trading data normalization and client connectivity improvements via transitioning to Kafka.
Scalable architecture with parallel pipelines enables efficient message throughput handling during peak trading periods.
Enhancements in engineering processes, like in-code and performance tests, ensure system reliability and efficient maintenance.
Deep dives
GORD System Overview and Legacy Challenges
GORD, a legacy system at Jane Street, manages trading data normalization, position calculations, trade settlement, and regulatory compliance. The challenges of engineering GORD involve its outdated design requiring continuous improvements and adaptations. The system's key tasks include processing diverse trading data sources and ensuring data accuracy and consistency across different exchanges and asset classes.
Performance Requirements and Scalability of GORD
GORD's core requirements include high performance during peak trading periods, scaling two to five times higher during busiest days, with an increasing demand for performance over the years. The system's architecture allows for horizontal scalability using parallel pipelines to handle increasing message throughput efficiently.
Transition to Kafka and Data Normalization Library
Recent improvements to GORD involve transitioning client connections from TCP IP to Kafka, enhancing scalability and reducing the burden of per-client data buffering. The implementation of a data normalization library facilitates seamless data exchange by providing structured transaction types and simplifying the data-handling process, moving towards a more structured and typed system. These changes aim to streamline data flows and enhance system reliability and flexibility.
Optimizing Data Handling with Monitoring Streams
The introduction of a monitoring stream allowed for a flexible way to manage data that wasn't directly related to the core activities of monitoring bookings and positions. This approach of normalizing data proved to be more valuable beyond its initial purpose, catering to various use cases. By segmenting the information into different streams, users could selectively subscribe to specific data subsets, enhancing the efficiency and usability of the system.
Enhancing Engineering Processes for System Changes
Improvements in engineering processes were crucial to manage the system's growth and complexity, ensuring reliability and efficiency in making changes. The implementation of in-code tests and performance tests transformed the testing procedures and enhanced confidence in making system modifications. Transitioning from manual testing to automated in-unit tests streamlined the validation process, optimizing the system maintenance and development cycle.
Ella Ehrlich has been a developer at Jane Street for close to a decade. During much of that time, she’s worked on Gord, one of Jane Street’s oldest and most critical systems, which is responsible for normalizing and distributing the firm’s trading data. Ella and Ron talk about how to grow and modernize a legacy system without compromising uptime, why game developers are the “musicians of software,” and some of the work Jane Street has done to try to hire a more diverse set of software engineers.
You can find the transcript for this episode on our website.
Some links to topics that came up in the discussion:
EG, The League of Legends team that Ella is a huge fan of.
Apache Kafka, the message bus that Gord migrated to.
Some of the varioussourcesofsymbology you have to deal with when normalizing trading data. (Really, there are too many sources to list here!)
A list of Jane Street’s recruiting Programs and Events, including INSIGHT, which focuses on women, and IN FOCUS, which focuses on historically underrepresented ethnic or racial minorities.
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
