The podcast explores the evolution of data cleansing and transformation processes at Microsoft, emphasizing the importance of non-coding solutions and discussing various interfaces and tools. They delve into the evolution of Power Query and its current state in Microsoft Fabric, and compare Microsoft tools such as DTS, SSIS, Power Query, and data factory, expressing a preference for data factory and fabric. They also discuss the potential of delivering analytics through a SaaS platform as a game changer and mention upcoming webcasts.
Low-code and no-code interfaces have become essential in data cleansing and transformation, enabling users to perform tasks without writing code.
Microsoft is infusing AI capabilities into data integration tools, enhancing productivity and providing guidance to users.
Deep dives
The Journey into Data and Tech
We Schonk, a product leader at Microsoft, shares his journey into the data and tech field. He started with a PhD in query processing in databases and joined Microsoft to work on SQL Server Integration Services (SSI). Over time, he became involved in the development of Azure Data Factory and Power Query. These products targeted different users, from citizen developers to professional developers, and received positive feedback. Eventually, the data integration efforts of both products merged into one big Microsoft Data Integration team.
Evolution of Data Cleansing and Transformation
Data cleansing and transformation have evolved over the years, focusing on enabling users to perform tasks without writing a single line of code. Low code and no code interfaces have become vital, allowing users to remove columns, change data types, and more. Products like Azure Data Factory and Power BI have been converging towards intuitive drag-and-drop interfaces that simplify data integration processes. These efforts aim to provide a consistent and user-friendly experience, emphasizing serverless, no code, and low code solutions.
AI Integration in Data Integration Tools
Microsoft has been infusing AI capabilities into its data integration tools. AI-powered features are already present in Power Query and Dataflow Generation 2, enabling tasks like webpage data extraction and text file pattern recognition. A co-pilot feature is being developed, which will allow users to describe their tasks using natural language and let the AI generate the necessary queries, pipelines, and activities. This AI-powered co-pilot will enhance productivity and provide guidance to users, making data integration tasks more efficient and streamlined.
On today's episode of Insights Tomorrow, Patrick LeBlanc and Partner Director of Products with the Cloud and AI organization at Microsoft, Wee Hyong, discuss his journey into the tech and data field, the importance of infusing AI into products to provide value differentiation, and how technology has evolved over the years to offer no-code and low-code interfaces for data integration.
In this episode you’ll learn:
How data integration has evolved at Microsoft
Potential AI capabilities coming to Data Factory and Power Query
The evolution of Power Query and how it has become a powerful ETL tool
Some questions we ask:
When did you transition into AI, and what led you to that field?
How has the process of data integration evolved at Microsoft?
Can you explain modern data and the infusion of data integration technologies?
Guest bio
Wee Hyong is a highly accomplished leader in Data and AI, boasting a stellar track record of steering multi-million dollar enterprises to enduring dominance within their respective industries. Throughout his illustrious career, Wee Hyong has adorned numerous hats, having excelled as a developer, a seasoned product manager, a data scientist, an innovative researcher, and a trusted technical advisor. Furthermore, he has penned over 10 authoritative books on Data and AI. These diverse and rich professional experiences have endowed him with a distinct set of stills, enabling him to conquer some of the world's most intricate and challenging technological enigmas.