Foundational Data Engineering At Two Sigma

86 snips

Jul 6, 2025

Effie Baram, a leader in foundational data engineering at Two Sigma, shares her insights into the pivotal role of data in finance. She delves into the complexities of maintaining data quality while ensuring quick delivery, navigating the socio-technical challenges of building a robust data platform. Effie discusses treating data as code, leveraging modern data warehouses, and the evolution of data engineering roles. She emphasizes the transition from fixed schemas to flexible structures, underscoring the importance of collaboration and regulatory compliance in a rapidly changing landscape.

Ask episode

AI Snips

Chapters

Transcript

Episode notes

ANECDOTE

Adopting DAGs Improved Data Management

Effie Baram's team struggled with intertwined business and infrastructure logic and unreliable cron jobs delivering complex data sets.
Adopting DAG orchestration with Airflow significantly improved data management and separation of concerns.

INSIGHT

Data as Core Product at Two Sigma

Two Sigma views data as a core product and focuses on foundational data for research.
Data is procured from vendors and internally produced, targeting various business needs with different latency and quality requirements.

INSIGHT

Balancing Quality and Latency

Foundational data for research has higher accuracy needs but can tolerate more latency than production data.
Balancing speed of delivery and depth of data richness is essential to meet varied user needs.

Get the Snipd Podcast app to discover more snips from this episode

Get the app

Summary
In this episode of the Data Engineering Podcast Effie Baram, a leader in foundational data engineering at Two Sigma, talks about the complexities and innovations in data engineering within the finance sector. She discusses the critical role of data at Two Sigma, balancing data quality with delivery speed, and the socio-technical challenges of building a foundational data platform that supports research and operational needs while maintaining regulatory compliance and data quality. Effie also shares insights into treating data as code, leveraging modern data warehouses, and the evolving role of data engineers in a rapidly changing technological landscape.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
This episode is brought to you by Coresignal, your go-to source for high-quality public web data to power best-in-class AI products. Instead of spending time collecting, cleaning, and enriching data in-house, use ready-made multi-source B2B data that can be smoothly integrated into your systems via APIs or as datasets. With over 3 billion data records from 15+ online sources, Coresignal delivers high-quality data on companies, employees, and jobs. It is powering decision-making for more than 700 companies across AI, investment, HR tech, sales tech, and market intelligence industries. A founding member of the Ethical Web Data Collection Initiative, Coresignal stands out not only for its data quality but also for its commitment to responsible data collection practices. Recognized as the top data provider by Datarade for two consecutive years, Coresignal is the go-to partner for those who need fresh, accurate, and ethically sourced B2B data at scale. Discover how Coresignal's data can enhance your AI platforms. Visit dataengineeringpodcast.com/coresignal to start your free 14-day trial.
Your host is Tobias Macey and today I'm interviewing Effie Baram about data engineering in the finance sector

Interview

Introduction
How did you get involved in the area of data management?
Can you start by outlining the role of data in the context of Two Sigma?
What are some of the key characteristics of the types of data sources that you work with?
Your role is leading "foundational data engineering" at Two Sigma. Can you unpack that title and how it shapes the ways that you think about what you build?
- How does the concept of "foundational data" influence the ways that the business thinks about the organizational patterns around data?
Given the regulatory environment around finance, how does that impact the ways that you think about the "what" and "how" of the data that you deliver to data consumers?
Being the foundational team for data use at Two Sigma, how have you approached the design and architecture of your technical systems?
- How do you think about the boundaries between your responsibilities and the rest of the organization?
What are the design patterns that you have found most helpful in empowering data consumers to build on top of your work?
What are some of the elements of sociotechnical friction that have been most challenging to address?
What are the most interesting, innovative, or unexpected ways that you have seen the ideas around "foundational data" applied in your organization?
What are the most interesting, unexpected, or challenging lessons that you have learned while working with financial data?
When is a foundational data team the wrong approach?
What do you have planned for the future of your platform design?

Contact Info

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA