When working with different sources, the data is often not well-formatted or has some kind of error. A common example is a CSV or Excel file containing a value that should be a number but in fact is a non-numeric character, causing an error when importing this record.
We can shorten review times by having a good monitoring system and being able to see which records have not been inserted into the project.
Power BI already has a system to save the records that have an error, but it cannot save more than one error detail per record. Saving the three parameters of the Error object (the Reason, the Message, and the Detail) in a table would allow us to identify the error easily.
1. Model Creation
In order to separate the errors from the other records, it is good practice to create 2 new tables. We will save the errors in one table, and all the good records in the other.
First, we must create two references to the table, instead of duplicating it (Duplicate will duplicate the code of the query, while Query Reference will only refer the result of the query).
In Power Query Editor, we select the source table, and we go to Home>Manage>Reference.
Figure 1: Options table
Once the model has this structure, we can direct the errors to the table called Error Logs, leaving the Fact table error-free. The following image shows the final result.
Figure 2: Model Diagram
2. Find Records with Errors and Save Them
By default, when saving errors, Power BI saves the records that have them, but we do not have access to these errors. If a record contains more than one error, searching for them through the tables is tedious and time-consuming. To solve this problem, we decided to write the following code in Power Query M.
The main purpose of the code is to search every record and find out if it has an error. If it does, the record is saved in a list that will be saved in a table later.
1. let
2. Source = Original,
3. KeptErrors = Table.SelectRowsWithErrors(Source, Table.ColumnNames(Source)),
4. isError = (value) => value[HasError],
5. getErrorList = (row) => List.Select(Record.FieldNames(row), (column) => isError(try Record.Field(row, column))),
6. ErrorList = Table.AddColumn(KeptErrors, "Errors", (row) => Record.SelectFields(row, getErrorList(row))),
7. FinalTable = Table.TransformColumns(ErrorList, {"Errors", (column) => Record.ToTable(column)}),
8. ExpandedErrors = Table.ExpandTableColumn(FinalTable, "Errors", {"Name", "Value"}, {"Column name", "Error"}),
9. Details = Table.AddColumn(ExpandedErrors, "CHECK ERRORS", each try [Error]),
10. ExpandedCheckedErrors = Table.ExpandRecordColumn(Details, "CHECK ERRORS", {"Error"}, {"Error Details"}),
11. ExpandedErrorDetails = Table.ExpandRecordColumn(ExpandedCheckedErrors, "Error Details", {"Reason", "Message", "Detail"}, {"Reason", "Message", "Detail"}),
12. RemovedColumns = Table.RemoveColumns(ExpandedErrorDetails,{"Error"})
13. in
14. RemovedColumns
|
Figure 3: Power Query Code
The following images show two records with typing errors. It is difficult to know the reason for the error, but thanks to the code shown above, the errors are now saved in a new record, making them easy to identify.
Figure 4: Errors in Records
Figure 5: Reasons for errors
By storing the errors in a record in the database, they can be displayed in a Power BI visual. You can save time by tracking the reason for the error, and quickly identify what and where the error is.
Figure 6: Display of errors
3. Roles for the Errors Page
Dashboards are often accessed by different profiles: users who only have to consult the data and users with privileges to administer the dashboard (admins). In order to manage the Error Logs page correctly, only certain users should be able to access it.
To start, we set the page to “hidden”.
Figure 7: Hide page
Then we create a table to store all the users who will have access to the Error Logs page. With the following DAX expression, you can filter the users who have access.
1. Button Action Error Logs = 2. IF (CONTAINS (Admins, Admins[User Email], USERNAME()), 3. "Error Logs", 4. "Landing Page" 5. ) |
Figure 8: Button action message
You can also display a message to keep users informed with a tooltip.
1. Button Tooltip Error Logs = 2. IF (CONTAINS (Admins, Admins[User Email], USERNAME()), 3. "Press to go Error Logs Page", 4. "You don’t have privileges to access Error Logs Page" 5. ) |
Figure 9: Button Message Tooltip
With the DAX created above, the fields are filled in so that the button has the desired behavior.
Figure 10: Button configuration
Conclusion
When working with data sources that cannot be rigorously verified, this method allows us to observe the errors and remove them from the final dashboard. This helps to ensure that the dashboard itself will never fail, and that we will have full observability and traceability of what data has an error and why.
Additionally, with role management, you can deny access to error pages for clients who should not see such information.
In conclusion, Power BI, together with Power Query M, provides us with a very powerful and versatile tool to deal with errors and manage a dashboard correctly.
At ClearPeaks, we have a team of certified experts with in-depth experience. Simply contact us, and we will help you with your Power BI and Microsoft needs!
The post How to Handle Data Errors in Power BI appeared first on ClearPeaks.