Unlocking The Value Of Data Across The Organization Through User Friendly Data Tools With Prophecy
Data Engineering Podcast
00:00
Spark - What Are Some of the Most Interesting Use Cases?
Prophecy is a new tool that lets data engineers transform their own data. It's being used by companies to speed up production and reduce time to value. The company has seen some unexpected use cases for it, such as the ability to build pipe lines from scratch with no previous knowledge of spark. They're also seeing users using li custom blocks for doing against all kinds of things.
Transcript
Play full episode
Transcript
Episode notes
Summary
The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction. In this episode he explores the tension between code-first and no-code utilities and how he is working to balance the strengths without falling prey to their shortcomings.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
- So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at dataengineeringpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
- Your host is Tobias Macey and today I’m interviewing Raj Bains about how improving the user experience for data tools can make your work as a data engineer better and easier
Interview
- Introduction
- How did you get involved in the area of data management?
- What are the broad categories of data tool designs that are available currently and how does that impact what is possible with them?
- What are the points of friction that are introduced by the tools?
- Can you share some of the types of workarounds or wasted effort that are made necessary by those design elements?
- What are the core design principles that you have built into Prophecy to address these shortcomings?
- How do those user experience changes improve the quality and speed of work for data engineers?
- How has the Prophecy platform changed since we last spoke almost a year ago?
- What are the tradeoffs of low code systems for productivity vs. flexibility and creativity?
- What are the most interesting, innovative, or unexpected approaches to developer experience that you have seen for data tools?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on user experience optimization for data tooling at Prophecy?
- When is it more important to optimize for computational efficiency over developer productivity?
- What do you have planned for the future of Prophecy?
Contact Info
- @_raj_bains on Twitter
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
- Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
- Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
- If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
- To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
- Prophecy
- CUDA
- Clustrix
- Hortonworks
- Apache Hive
- Compilerworks
- Airflow
- Databricks
- Fivetran
- Airbyte
- Streamsets
- Change Data Capture
- Apache Pig
- Spark
- Scala
- Ab Initio
- Type 2 Slowly Changing Dimensions
- AWS Deequ
- Matillion
- Prophecy SaaS
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Remember Everything You Learn from Podcasts
Save insights instantly, chat with episodes, and build lasting knowledge - all powered by AI.
