Selenium Reporting With CSharp Automation Framework
Proper Selenium Reporting is essential to be able to realize one of the biggest benefit in test automation i.e quick feedback and faster analysis on issues found. We are able to collate results from different test execution sessions, over wide range of time. To get this benefit, we will have the capability included in our Test Automation framework. The report should also cater for different targeted audience of our automation tests execution result.
In the case of Framework Development with C# and selenium we don’t have a wide range of option, however, we do have extent reports. It is highly customisable, and produces beautiful Html reports using the xml generated by the unit testing framework, NUnit in our case.
- 5 Items we are going to consider and include in our reports
- Integrate and customize relevant tools for selenium reporting
- Overview of styling and other configurations
Pre-requisite and Tools:
- Previous Solution from here
- Extent Reports
5 Items we are going to consider and include in our reports:
1. Consolidated report section showing results of test execution. Result could be simply Number of passed, failed, or ignored tests, Total number of tests, start time, end time, total time to run suite.
2. For non-technical stake holders, we might include a graphical representation of test results in the form of a pie chart, or a bar chart.
3. Failed tests summary including, specific step that failed, Error message suggesting reason for failure, severity of failed tests
4. Test Execution summary including, Environment test ran, Screenshot of failure (Not going to cover yet)
5. Label Showing type of test executed (Regression, Smoke test)
6. Beauty – we want it to look good. And possibly customized to include company’s branding colours, images, logo etc.
A good tool to achieve the above is Extent report library by Anshoo Arora. The good thing is that out of the box, we are able to generate good looking reports with useful information.
Note, I have decided to exclude the documentation project so that this post could be stand alone, and so that the completed solution can be ran without having to worry about installing sandcastle on the local machine. Grab the previous solution from this post.
This time we will be adding this new library into our .common and .web projects. The reason is because, when it comes to a reporting module, it is not platform specific, we can reuse it across multiple test projects, mobile or web.
Let’s flip to Visual Studio and add the Extent report library via nuget package.
Integrate and customize relevant tools for selenium reporting
Adding relevant Library and Initializing the Report
- Right click on the solution
- Select Manage nuget packages for solution
- Search for extent report. At time of writing latest-stable version is v2.41.0
- Select the .common and .web projects and click install. This should add the RelevantCodes.ExtentReports reference to the projects.
- In the. common project we will add a folder called Reporting and add 2 Classes ReportingManager.cs and ReportingTasks.cs
- The ReportingManager.cs class will provide the instance of the extent report. It is a singleton class which will ensure it is only a single instance of the ExtentReport class is created.
- In the ReportingManager.cs class (probably badly named) we select a location to save the generated report. I have hard-coded the location to be in the bin folder and it will be named TestResults.html
- The ReportingTask.cs will encapsulate the tasks to be executed. In this class we need information about the test context during test execution, hence, we will reference NUnit in the .common project as well.
- Since we have NUnit within the solution, right Click on solution and select manage nuget package for solution.
- Select the project and click install
- We can now reference NUnit.Framework and NUnit.Framework.Interfaces in the class.
- The constructor of this class takes an instance of Extent Report. This will be provided by the ReportingManager class.
- We declare 3 Methods, InitializeTest(), FinalizeTest(), CleanUpReporting(). The later will finalize reporting by calling Extent.close()
Now in the .Web Project, we can setup reporting on the web project. We can later include new projects, and reuse the reporting classes and functions from the .common project.
Now to use our new functionality
- In the TestBase Class, lets declare a private static instance variable of type ReportingTasks.
- In the Suite setup method (BeginExecution()), we create an instance of the ExetentReports and use this to initialize the ReportingTask field.
- In the Test Setup method (TestSetup()) we will setup reporting tasks for each tests.
- In the Test Cleanup method (CleanUpTest()) we will finalized reporting tasks for each tests.
- Finally, To finalize the suite Execution in the suite cleanup method (ExitExecution()) we will clean up resources.
Now we are all setup. Start up our AUT, by running instant WordPress. Refer to this post for the AUT we have been using. Download latest solution from this post if you haven’t got the latest solution.
Simply run the tests we already have in our .Tests project.
In the bin/Debug folder of the Test project, open the Testresult.html file.
Select the failed test, as you can see it displays the failure message we included in the test, expected and actual result.
There is more. A lot more.
We can customize the report even further to display our own logo.
We can add a lot more info that the company or organization might find important.
This report could be included as artefacts on a CI system e.g. teamCity, and get emailed to relevant team distribution list.
For reporting I think the depth of details are enough. When it comes to detailed information about test run, I personally prefer to defer this to the logging mechanism, which we will cover later. This will capture information on various exceptions, details of errors, etc.
Overview of Styling and other configurations
To achieve this, we will need to include an xml file. This could be placed anywhere, but I prefer it goes to the test script project. This will contain project specific customisations.
As demonstration, we will just change the colour of the header.
- Call the LoadConfig() method of the ExtentReport class.
- Right click on the .Tests project, and add a new xml file called extent-config.xml
- Set the location of the config file in the Test Base, BeginExecution() method dynamically as shown.
- The Actual xml content, you can get from Extent report website
As you can see the file shows a lot of configuration option. Modify as you will like.
In the styles section, we will add the css as shown just to change the colour.
If you are not comfortable with CSS, there are a number of tutorials to get you going with simple styling. or take doughnuts to work and ask the developers to help out.
Let’s re-run the tests.
A good thing to not is that, we havent made any changes to out test scripts. All the logic for the reporting are in the framework.
Click Here to Download the Complete Working Solution.