Explicit Measures Podcast

265: Mailbag Hot Takes

Nov 7, 2023

The podcast covers topics such as optimizing data warehouses and reports, using Power BI reports server for cloud-averse companies, and developing an organizational policy for Power BI workspace creation. They also discuss new functions for table creation in a lake house, challenges of bringing custom code and libraries into Microsoft fabric, optimizing partition strategy and file size in Microsoft Databricks, and moving data to the cloud with Power BI.

58:52

Creator website

AI Summary

AI Chapters

Episode notes

Podcast summary created with Snipd AI

Quick takeaways

Data warehouses and reports should be optimized to only pull new data instead of performing full refreshes.

For companies hesitant to load data to the cloud, Power BI Reports Server offers a solution for internal use.

Deep dives

Optimal architecture for data set and report development in the fabric era

The optimal architecture for data set and report development in the fabric era is still similar to the pre-fabric era. The pattern of collecting data sources into data flows, creating shared data sets, and building reports on top of them remains effective. However, with the introduction of fabric, there are now more options and tools available for data engineering and processing. The use of Spark notebooks, data flows Gen2, Azure Data Factory, and other services can enhance the development process. While Power BI desktop remains a vital tool, the Power BI service is expected to take over in the future, offering a more streamlined and up-to-date experience. It's important to consider the specific needs and requirements of your organization when choosing the optimal architecture for data set and report development.

Introduction

4min

New Function for Table Creation in a Lake House and Optimizing Tables

3min

Discussion on Microsoft fabric and bringing your own library

3min

Optimizing Partition Strategy and File Size in Microsoft Databricks

13min

Moving Data to the Cloud with Power BI

25min

Exploring New Tools and Challenges in Data Engineering

10min

We had a ton of mailbags we needed to get off our chests! Quick Shots on the following:

Can Data Warehouses and Reports Only Pull New Data Instead of Full Refreshes? Thanks for your show. I have been enjoying it for a few months now. I'd like to say I'm new to Data work, but I have been working as an analyst for a few years now.

I have observed that my data warehouse goes out and grabs a massive chunk of data from source databases, and brings it in to the DW and overwrites 99% of the exact same data that is already in the warehouse. Then my on-prem power bi reports do the same thing. Bring in a mass of data from the DW, 99% of it was brought in yesterday. This feels wrong. Is there anyway to just grab the *new* data? Is this the way it has to be, or is it a bad *data smell*? Being relatively new, I don't know how, but think there must be a better way.

Request for Tips and Tricks on Power BI Reports Server for Cloud-Averse Companies

Thanks for your great efforts, I have been listening to your podcast for more than 9 months now.

I wish if you could cover support for PBI Reports Server (Tweeks, tricks, hints, ... etc).

My company is hesitant to load its data to the cloud, we installed PBI RS and currently use it for our internal use.

This is a data culture thing

Developing an Organizational Policy for Power BI Workspace Creation to Support Self-Service BI

Topic: Power BI (Now Fabric) Workspace creation policy for self service Power BI

What should be the organisational policy for creating Power BI workspaces in order to support Self Service Power BI within the organisation?

The factors that affect this are

1) To prevent the explosion of number of workspaces

1.1) Is it a problem if number of workspaces increase exponentially? Why is it a problem? We don’t worry about the explosion of number of child folders in a organisational shared drive, so why is this a problem in Power BI workspaces?

2) Only 1 App can be published in a workspace, therefore, even a single team will need multiple workspaces contributing to the explosion of number of workspaces

3) Who should be allowed to create workspaces?

3.1) Does IT create workspaces when requested by end user teams?

4) Should there be a naming standard/pattern for workspaces?

4.1) Why?

4.2) If so, can you please suggest some naming strategies?

Thank you for your podcast. I enjoy listening to it.

Get in touch:

Send in your questions or topics you want us to discuss by tweeting to @PowerBITips with the hashtag #empMailbag or submit on the PowerBI.tips Podcast Page.

Visit PowerBI.tips: https://powerbi.tips/

Watch the episodes live every Tuesday and Thursday morning at 730am CST on YouTube: https://www.youtube.com/powerbitips

Subscribe on Spotify: https://open.spotify.com/show/230fp78XmHHRXTiYICRLVv

Subscribe on Apple: https://podcasts.apple.com/us/podcast/explicit-measures-podcast/id1568944083‎

Check Out Community Jam: https://jam.powerbi.tips

Follow Mike: https://www.linkedin.com/in/michaelcarlo/

Follow Seth: https://www.linkedin.com/in/seth-bauer/

Follow Tommy: https://www.linkedin.com/in/tommypuglia/