Jukka Niiranen’s Post

View profile for Jukka Niiranen, graphic

The Original Power Platform Advisor. 11x Microsoft MVP. Low-code 4 life.

Dataflows - the least governable service in #PowerPlatform? Considering how easy it is to create a standard Dataflow to schedule the retrieval of external data and push it into Dataverse, it is truly frightning how little admin capabilities there exists for the feature. In fact, you can see all them in the image below. You can open an environment, go to "All Dataflows" view and then search by owner name - assuming that you know that there is something to search for. How could you as an admin learn that there in fact are any Dataflows running in the environment? The only way seems to be to open the default solution and check if any components of type Dataflow exist in there. If they do, then you'll need to make note of the owner's name and go back to the "All Dataflows" view. How do you find out what the Dataflows are actully doing? You don't - unless you are ready to forcefully change the ownership of the Dataflow to your account. There appears to be no way to read what the Dataflow does, or even see the run log to check if/when it is running and what erros it might be generating. Only the Maker can tell what's happening to the Dataflow. What if they are not currently / any longer available? Good luck. The fact that there are no admin APIs / PowerShell cmdlets to query for Dataflows in the tenant / inside environments means that these will all fly under the radar of Power Platform admins. You won't see them in CoE Starter Kit reports, nor anywhere else. You won't know they exist, until the impact of these Dataflows forces you to investigate. What if the Dataflows are syncing data from your ERP system into your Dataverse environment? What if there are now millions and millions of rows consuming capacity as a result of some poor choices in the solution architecture? Can you create guardrails to say from which data sources your Makers are allowed to pull data into Dataverse? I don't think so. At least the Power Platform DLP policies aren't going to affect these Dataflows, since they are running on the Power Query connector architecture rather than the Power Apps / Automate connectors that the Power Platform environment specific DLP policies are controlling. It really looks like there's only a single admin feature for standard Dataflows: "change owner". All the governance fundamentals around that are missing. Or have I missed something? Have you discovered / implemented some admin capabilities to make standard Power Platform Dataflows visible and controllable to the admins?

  • No alternative text description for this image
Alex T.

Power Platform Solution Architect at Nestlé

1y

You haven't missed anything 🙂 Very good analysis. Worth noting is also that if a dataflow remains orphan, it is impossible to find it by doing a search in the UI since the UI uses Graph to resolve the user account query. Since the user account is gone, you can't actually use the email to search since there is no account that would match the email address in Graph. The UI does not honor any of the Dataverse or AAD elevated security roles to display all dataflows. To govern dataflows you need to load them somewhere from each instance directly from the dataflow table. The only control you could add is reactive, meaning that you can break the ownership at the time of creation and have the user certify it somehow. To understand what each dataflow does you would need to read the mashup document and parse it somehow. To understand which dataflow is live you would need to look at the refresh history table. Aside from this, dataflows are also available in Teams environments, however if you look at the new feature that is supposed to remove unsed environments, you will notice that the feature doesn't take dataflow runs into consideration when determining if a Teams environment is in use or not.

Thomas Sandsør

Head of Dynamics 365 & Power Platform @ Point Taken AS | Microsoft MCT | Microsoft MVP

1y

Great article! Not being able to run as application user, not being able to choose Insert/update/upsert, not having the option to choose matching criteria (only KEY) are things that make it hard for me to use. The Key part is personal preference when I am used to SSIS Kingswaysof. The more picky part of me doesn't like that it can create data structure with prefix of field names that is not a part of solution😀

Fwiw, the only way to see and monitor dataflows I found, is to query the relevant tables per environment (msdyn_dataflows and msdyn_dataflowrefreshhistories). And use PBI, Excel to view (using possibly a meta dataflow gather the records all environments😁).

Maciej Frydrych

IT Product manager @ Totalizator Sportowy | Digital Workplace

1y

That's terrible and what the heck. So you need to modify dataflows and replace user account to another service account ? after creating Dataflow ?

Jordi Montaña Vázquez

Entrepreneur, Author of #FakeXrmEasy, Founder of dynamicsvalue.com, 2 x (former) Microsoft MVP

1y

Pushing data straight into prod.... What could go wrong?? 😁

Thank you for sharing. Manageability is not the priority for ms I guess. First they empower every person, somewhat later organisational objectives are considered. So it seems to me at least.

Has anything changed since your post? Anything on the MS roadmap to have beter management of dataflows in dataverse environments? Running dataflows as systemusers is definitly needed!

Like
Reply
Saad NADIR

Microsoft Dynamics 365 CRM Expert

1y

One of the weirdest thing i encountered is that when i deploy a solution that contains a dataflow using devops pipline with power platform build tools extension, when i use a service principal as a method of authentication, the owner of the dataflow will be the service principal and will never be displayed in make.powerapps.com …. So i had to switch my authentication method to an office 365 account when importing solutions …. Also there is no support of the deployment settings file for the moment so that we can link connection references with connections the way we do in power automate. Last, i had trouble deploying solutions thats contains dataflows when i package them in package deployment, a microsoft plugin is triggered and causing timeout when exceeding 2mn during the import of solution that contains dataflows. Have a look at https://github.com/microsoft/powerplatform-build-tools/issues/178

David Wyatt

Lead Power Platform Developer | Power Platform Administrator | Intelligent Automation Developer | Low Code & Power Platform Blogger

1y

Great analysis, I was probably like most people excited with its potential, without realising what exposures it creates

Mariette Jansen

D365 Sales & Marketing Consultant: I am a connector, someone who will built the bridge between IT and end users. As a catalyst, I can bring ideas and people together. When I haven't got any blue, I use red. - Picasso

2mo
Like
Reply
See more comments

To view or add a comment, sign in

Explore topics