Measure Power Query Performance using Diagnostics Tool in Power BI

Overview:

In this article, I'll walk through on how we can measure performance of Power Query Report using the Diagnose Tool in Power BI. Microsoft has just launched this new feature in Power BI - December 2019 update.

Real-Life Use-case and Scenario:

There are the scenarios or requirements where we need to apply many operations (e.g., Change the data types, Create calculated columns, Perform some M functions etc.) within Power Query Editor.

Sometime, the performed operations take time to load the data, resulting degradation in performance.

To improve the performance, we need to identify which part of the query is taking higher amount of time. Based on that analysis, we can enhance the formula to improve Power Query performance.

So, let’s see how we can achieve this!!

Step 1:

Please make sure that Power BI Desktop is updated with the latest version (December 2019 or later).

In Power BI Desktop tool, From File, select Options and Settings and click on Options.

This will open a Preview Feature. Make sure that the feature - "Query Diagnostics" is activated.

Step 2:
Open the Power BI report for which we are facing performance issue.

From Home, select Transform data. This will open the Query Editor window.

Here, we have four steps in our query editor window.

Now, let’s check which step is taking more time to execute.

Step 3:

Go to Tools Menu. We have following 2 Group Options here.

• Step Diagnostics
• Session Diagnostics

Step Diagnostics:

Here, we can measure the performance of an individual step. Suppose, if our Power Query has 15 steps and we only wish to measure the performance of step #12, at that time this one is best suitable option.

Session Diagnostics:

With this option, we can measure the performance of the entire report covering all the steps of Power Query.

Here, we will perform both the actions and try to analyze the data.

Step 4:

Let’s first perform “Session Diagnostics”.

Click on “Start Diagnostics”.

Now, Click on Refresh All data.

Once data is refreshed, click on “Stop Diagnostics”.

Step 5:

When we stop diagnostics, this will add two new tables as a part of Query Diagnostics.

One represents Overall diagnostics and the other represents Detailed diagnostics. Let’s click on detailed diagnostics.

Here, we can see all the steps listed with the time taken for each execution.

From the result, we can easily find out which step is taking the more time.

Also, if we have used any calculated column or condition-based formula, this will also show how much time is taken by that process.

Step 6:

Let’s say if we wish to measure the performance of the specific step, then select the appropriate step from the Query Window, then click Diagnostic Step.

This will add following step(s) which will represent the diagnostics data.

Conclusion:

This is how, we can easily measure the performance of Power Query. Isn’t it amazing?

Happy Reporting!! 

If you have any questions you can reach out our SharePoint Consulting team here.

Combine data from multiple SharePoint Lists using Power Query in MS Excel - Part 1

Have you ever received a requirement to get data from SharePoint list and you wanted to modify it, but, you have trouble with the same?

Well, Microsoft Excel has been evolving on a regular basis, making it a powerful tool. But are we aware of the powerful things, that we don't use on a daily basis? Here, is one such function i.e. Power Query.

What is Power Query?
Power Query is an add-in for business intelligence available in Microsoft Excel and now in Power BI. Similar to SQL query, it gives you the functionality to modify or create new model of the data by combining various data sources like relational databases, Excel, text and XML files, OData feeds, web pages, SharePoint, Hadoop file (HDFS), Exchange, SAP HANA, etc. and utilize it for further processes. 

Example:
Here, in this example, we would be getting the data of 2 SharePoint lists, and combining those, to make a new model for our use. 


Please Note: This blog provides information on the basics like using the Power Query, creating functions, and using those functions to model data for further use.

In this example, we have 2 SharePoint Lists - Subject Name and Students. 

• Subject Name list is the master list of the subjects. 
• Students list contains the data of students of different grades and Subject Name they excel at. 

Requirement:
We need to get list of students those who are Primary school i.e. Grades <=7, and High School i.e. Grades > 7 in Excel.

Implementation : Connecting SharePoint List with Excel and transforming data.

So, we begin with connecting SharePoint online data with Excel:

• In the Data Tab, from the ribbon > Get Data. In this you would find various data connectors, and we would be going for SharePoint Online List. So, under the "Get Data menu > From Online Services > From SharePoint Online List.

• Now, in the open SharePoint List dialog box, we would enter the Site Collection/Subsite URL of List1.

• This would ask us to authenticate the details, and ask us for the credentials.

• On Clicking Sign In, this would open Office 365 login page pop-up dialog box. Enter the account details. On authentication, it would show that you have signed in and Click on Connect. 

• On Clicking Sign In button, this would open Office 365 login page pop-up dialog box. Enter the account details.

• On authentication, it would show that you have signed in and Click on Connect. This would open up the total list and libraries in the site. 

• Here, select the List1 from which the data is to be fetched. In our case, we would be using Subject Names list. On selection, on the right, you would see the preview of the data in a table. 

Please note, that while fetching the data, it brings all the data in the list, including the hidden and back-end columns, not generally used and is recommended not to be used.

• Now, there are 2 options, Load and Edit.

1. Load: This would fetch the information as it is. Then, the changes would be made in the Excel file, but every-time, the data would fetch all the information. This would being unwanted data, and specially, when the data volume is very high, this is not recommended.
2. Edit: The recommended approach, if the data is too large, to have a cleanup performed. This would open up our table, where we can make changes in the model  

• So, we would be click on Edit to edit the fields, to change the model and get limited data only. This would open up Power Query Editor window. 

• We will remove the unwanted columns, by selecting the columns and then right-clicking the column to remove the columns. 

• We have removed the unwanted columns and our List 1 is ready. Now, we have just the Subject Name and Description. This would look like something as below.

• Now, we would be adding another list here, in the same Power Query Editor window. We can add our 2nd list, by clicking New Source > Online Services < SharePoint Online List. Alternatively, we can right-click in the Queries section and select New Query > Online Services > SharePoint Online List. This would open the same dialog box to enter the SharePoint Site URL.

• We will add Students which contains the Student Names, their grades and the subject name (lookup of the Subject Name). We will clean it by removing unwanted columns, and we would be having just Student Name, Grades and Subject Name. Now, as this is a lookup, we would be seeing "Record". So, we would be expanding it and selecting the Title column i.e. - Subject Name. To expand, we will click the expand button besides the column Subject Name and select Title column.

• So, in this part, we have added the data source list, and modified the data as per our requirement.

In the next part, we would creating function in Power Query Editor, to get the students per the grade, and and use those for different purpose. 

Use SharePoint data in Microsoft Excel using Power Query - Part 2

#PowerQuery #MSExcel #Excel #Query #SharePoint #Office365 #SharePointOnline #SharePointList #PowerBI #MicrosoftSharePoint

If you have any questions you can reach out our SharePoint Consulting team here.