Data Engineering Podcast

Data Migration Strategies For Large Scale Systems

18 snips
May 27, 2024
Experienced data engineer Sriram Panyam shares insights on managing data migration projects in high traffic environments. Topics include strategies for maintaining data consistency, pitfalls to avoid, and involving application teams for organizational alignment. The episode explores challenges in data management and the importance of architectural soundness.
Ask episode
AI Snips
Chapters
Transcript
Episode notes
ANECDOTE

LinkedIn Migration

  • At LinkedIn, Sriram Panyam migrated 100 billion edges in the social actions platform.
  • This involved moving from one Espresso store to another, requiring a two-part approach with snapshots and CDC.
ADVICE

Data Migration Planning

  • Understand the broader impact of the data and its ecosystem.
  • Consider customer experience, business use cases, and stakeholder priorities during data migrations.
INSIGHT

Tooling Trade-offs

  • Many data migration tools offer various functionalities and trade-offs.
  • Understanding these trade-offs is crucial for choosing the right tool and avoiding surprises.
Get the Snipd Podcast app to discover more snips from this episode
Get the app