Foundational Data Engineering At Two Sigma
Data Engineering Podcast
00:00
The Journey into Data Management Complexity
This chapter recounts the speaker's ten-year journey into data management, focusing on the difficulties encountered in ensuring high-quality research data from complex trading systems. It explores the transition to DAG orchestration systems like Airflow and the significant improvements this brought to their data management practices.
Transcript
Play full episode
Transcript
Episode notes
Summary
In this episode of the Data Engineering Podcast Effie Baram, a leader in foundational data engineering at Two Sigma, talks about the complexities and innovations in data engineering within the finance sector. She discusses the critical role of data at Two Sigma, balancing data quality with delivery speed, and the socio-technical challenges of building a foundational data platform that supports research and operational needs while maintaining regulatory compliance and data quality. Effie also shares insights into treating data as code, leveraging modern data warehouses, and the evolving role of data engineers in a rapidly changing technological landscape.
Announcements
Parting Question
In this episode of the Data Engineering Podcast Effie Baram, a leader in foundational data engineering at Two Sigma, talks about the complexities and innovations in data engineering within the finance sector. She discusses the critical role of data at Two Sigma, balancing data quality with delivery speed, and the socio-technical challenges of building a foundational data platform that supports research and operational needs while maintaining regulatory compliance and data quality. Effie also shares insights into treating data as code, leveraging modern data warehouses, and the evolving role of data engineers in a rapidly changing technological landscape.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
- This episode is brought to you by Coresignal, your go-to source for high-quality public web data to power best-in-class AI products. Instead of spending time collecting, cleaning, and enriching data in-house, use ready-made multi-source B2B data that can be smoothly integrated into your systems via APIs or as datasets. With over 3 billion data records from 15+ online sources, Coresignal delivers high-quality data on companies, employees, and jobs. It is powering decision-making for more than 700 companies across AI, investment, HR tech, sales tech, and market intelligence industries. A founding member of the Ethical Web Data Collection Initiative, Coresignal stands out not only for its data quality but also for its commitment to responsible data collection practices. Recognized as the top data provider by Datarade for two consecutive years, Coresignal is the go-to partner for those who need fresh, accurate, and ethically sourced B2B data at scale. Discover how Coresignal's data can enhance your AI platforms. Visit dataengineeringpodcast.com/coresignal to start your free 14-day trial.
- Your host is Tobias Macey and today I'm interviewing Effie Baram about data engineering in the finance sector
- Introduction
- How did you get involved in the area of data management?
- Can you start by outlining the role of data in the context of Two Sigma?
- What are some of the key characteristics of the types of data sources that you work with?
- Your role is leading "foundational data engineering" at Two Sigma. Can you unpack that title and how it shapes the ways that you think about what you build?
- How does the concept of "foundational data" influence the ways that the business thinks about the organizational patterns around data?
- Given the regulatory environment around finance, how does that impact the ways that you think about the "what" and "how" of the data that you deliver to data consumers?
- Being the foundational team for data use at Two Sigma, how have you approached the design and architecture of your technical systems?
- How do you think about the boundaries between your responsibilities and the rest of the organization?
- What are the design patterns that you have found most helpful in empowering data consumers to build on top of your work?
- What are some of the elements of sociotechnical friction that have been most challenging to address?
- What are the most interesting, innovative, or unexpected ways that you have seen the ideas around "foundational data" applied in your organization?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working with financial data?
- When is a foundational data team the wrong approach?
- What do you have planned for the future of your platform design?
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
- Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
- 2Sigma
- Reliability Engineering
- SLA == Service-Level Agreement
- Airflow
- Parquet File Format
- BigQuery
- Snowflake
- dbt
- Gemini Assist
- MCP == Model Context Protocol
- dtrace
The AI-powered Podcast Player
Save insights by tapping your headphones, chat with episodes, discover the best highlights - and more!
