MLOps.community Streaming Ecosystem Complexities and Cost Management // Rohit Agrawal // #302
6 snips
Apr 4, 2025 Rohit Agarwal, Director of Engineering at Tecton, shares his expertise in streaming data management and ML challenges. He discusses the complexity of navigating both real-time and batch data systems. Rohit highlights the financial implications of tool fragmentation, the evolution of managed services, and the importance of collaboration among data teams. He also explores the emerging trend of Bring Your Own Cloud solutions for enhanced data security. Lastly, he touches on simplifying data processing paradigms and the future of data storage technologies.
AI Snips
Chapters
Transcript
Episode notes
Cometeer Coffee
- Rohit Agarwal started using Cometeer coffee during COVID-19.
- He continues to use it due to its simplicity and ease of preparation.
Streaming Ecosystem Challenges
- Companies struggle to use real-time data effectively, despite having streaming solutions like Kafka or Kinesis.
- The streaming ecosystem is fragmented, requiring diverse tools and expertise, making it complex and costly, especially for smaller teams.
Balancing Freshness and Cost
- Balance data freshness with cost; higher freshness typically means higher costs.
- Seek systems that offer adjustable trade-offs between these factors, allowing flexibility for various use cases.