I am trying to get bytecode coverage analysis using a code coverage tool (like Emma or Jacoco) after testing with a GUI based functional testing tool (like HP QuickTest Pro or Selenium).
Anyone who has done this could please give me an idea to start this project?
user2110844user2110844
1 Answer
I am doing this now. My approach is to use JaCoCo ant tasks to instrument the binary byte-code files, and use a specific CLASSPATH to execute the instrumented binaries from an ant build.xml from Jenkins.
The reason for doing the code coverage from byte-code is that there is an existing set-up that runs test scripts for a large application using HP QuickTest Pro . I would imagine that the test coverage is in the single digits, but we need an empirical baseline to demonstrate the possible improvements in code coverage from doing unit tests during a build.
rayray
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Not the answer you're looking for? Browse other questions tagged seleniumcode-coveragefunctional-testingemmajacoco or ask your own question.
-->To determine what proportion of your project's code is actually being tested by coded tests such as unit tests, you can use the code coverage feature of Visual Studio. To guard effectively against bugs, your tests should exercise or 'cover' a large proportion of your code.
Code coverage analysis can be applied to both managed (CLI) and unmanaged (native) code.
Code coverage is an option when you run test methods using Test Explorer. The results table shows the percentage of the code that was run in each assembly, class, and method. In addition, the source editor shows you which code has been tested.
Requirements
The code coverage feature is available only in Visual Studio Enterprise edition.
To analyze code coverage on unit tests in Test Explorer
- On the Test menu, choose Analyze Code Coverage.
- To see which lines have been run, choose Show Code Coverage Coloring.To alter the colors, or to use bold face, choose Tools > Options > Environment > Fonts and Colors > Show settings for: Text Editor. Under Display Items, adjust the Coverage items.
- If the results show low coverage, investigate which parts of the code are not being exercised, and write more tests to cover them. Development teams typically aim for about 80% code coverage. In some situations, lower coverage is acceptable. For example, lower coverage is acceptable where some code is generated from a standard template.
Tip
- make sure that compiler optimization is turned off
- if you are working with unmanaged (native) code, use a debug build
- make sure that you are generating .pdb (symbol) files for each assembly.
If you don't get the results you expect, see Troubleshoot code coverage. Don't forget to run code coverage again after updating your code. Coverage results and code coloring are not automatically updated after you modify your code or when you run tests.
Report in blocks or lines
Code coverage is counted in blocks. A block is a piece of code with exactly one entry and exit point. If the program's control flow passes through a block during a test run, that block is counted as covered. The number of times the block is used has no effect on the result.
You can also have the results displayed in terms of lines by choosing Add/Remove Columns in the table header. If the test run exercised all the code blocks in any line of code, it is counted as one line. Where a line contains some code blocks that were exercised and some that were not, that is counted as a partial line.
Some users prefer a count of lines because the percentages correspond more closely to the size of the fragments that you see in the source code. A long block of calculation would count as a single block even if it occupies many lines.
Manage code coverage results
The Code Coverage Results window usually shows the result of the most recent run. The results will vary if you change your test data, or if you run only some of your tests each time.
The code coverage window can also be used to view previous results, or results obtained on other computers.
You can merge the results of several runs, for example from runs that use different test data.
- To view a previous set of results, select it from the drop-down menu. The menu shows a temporary list that is cleared when you open a new solution.
- To view results from a previous session, choose Import Code Coverage Results, navigate to the TestResults folder in your solution, and import a .coverage file.The coverage coloring might be incorrect if the source code has changed since the .coverage file was generated.
- To make results readable as text, choose Export Code Coverage Results. This generates a readable .coveragexml file, which you could process with other tools or send easily in mail.
- To send results to someone else, send either a .coverage file or an exported .coveragexml file. They can then import the file. If they have the same version of the source code, they can see coverage coloring.
Merge results from different runs
In some situations, different blocks in your code will be used depending on the test data. Therefore, you might want to combine the results from different test runs.
For example, suppose that when you run a test with input '2', you find that 50% of a particular function is covered. When you run the test a second time with the input '-2', you see in the coverage coloring view that the other 50% of the function is covered. Now you merge the results from the two test runs, and the report and coverage coloring view show that 100% of the function was covered.
Use Merge Code Coverage Results to do this. You can choose any combination of recent runs or imported results. If you want to combine exported results, you must import them first.
Use Export Code Coverage Results to save the results of a merge operation.
Limitations in merging
- If you merge coverage data from different versions of the code, the results are shown separately, but they are not combined. To get fully combined results, use the same build of the code, changing only the test data.
- If you merge a results file that has been exported and then imported, you can only view the results by lines, not by blocks. Use the Add/Remove Columns command to show the line data.
- If you merge results from tests of an ASP.NET project, the results for the separate tests are displayed, but not combined. This applies only to the ASP.NET artifacts themselves: results for any other assemblies will be combined.
Exclude elements from the code coverage results
You might want to exclude specific elements in your code from the coverage scores, for example if the code is generated from a text template. Add the System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute attribute to any of the following code elements: class, struct, method, property, property setter or getter, event.
Tip
Excluding a class does not exclude its derived classes.
For example:
Exclude elements in Native C++ code
To exclude unmanaged (native) elements in C++ code:
Use the following macros:
ExcludeFromCodeCoverage(
ExclusionName, L'
FunctionName');
ExcludeSourceFromCodeCoverage(
ExclusionName, L'
SourceFilePath');
- ExclusionName is any unique name.
- FunctionName is a fully qualified function name. It may contain wildcards. For example, to exclude all the functions of a class, write
MyNamespace::MyClass::*
- SourceFilePath is the local or UNC path of a .cpp file. It may contain wildcards. The following example excludes all files in a particular directory:
MyComputerSourceUnitTests*.cpp
#include <CodeCoverageCodeCoverage.h>
- Place calls to the exclusion macros in the global namespace, not within any namespace or class.
- You can place the exclusions either in the unit test code file or the application code file.
- The exclusions must be compiled as unmanaged (native) code, either by setting the compiler option or by using
#pragma managed(off)
.
Note
To exclude functions in C++/CLI code, apply the attribute
[System::Diagnostics::CodeAnalysis::ExcludeFromCodeCoverage]
to the function. This is the same as for C#.Include or exclude additional elements
Code coverage analysis is performed only on assemblies that are loaded and for which a .pdb file is available in the same directory as the .dll or .exe file. Therefore in some circumstances, you can extend the set of assemblies that is included by getting copies of the appropriate .pdb files.
You can exercise more control over which assemblies and elements are selected for code coverage analysis by writing a .runsettings file. For example, you can exclude assemblies of particular kinds without having to add attributes to their classes. For more information, see Customize code coverage analysis.
Analyze code coverage in Azure Pipelines
When you check in your code, your tests run on the build server along with tests from other team members. It's useful to analyze code coverage in Azure Pipelines to get the most up-to-date and comprehensive picture of coverage in the whole project. It also includes automated system tests and other coded tests that you don't usually run on the development machines. For more information, see Run unit tests with your builds.
Analyze code coverage from the command line
To run tests from the command line, use vstest.console.exe. Code coverage is an option of the vstest.console.exe utility.
- Launch the Developer Command Prompt for Visual Studio:In the Windows Start menu, choose Visual Studio 2017 > Developer Command Prompt for VS 2017.In the Windows Start menu, choose Visual Studio 2019 > Developer Command Prompt for VS 2019.
- At the command prompt, run the following command:
For more information, see VSTest.Console.exe command-line options.
Troubleshoot
If you do not see code coverage results, the Troubleshoot code coverage article might help you.
See also
Calculating percentages can be an easy task. There are numerous percentage calculators online that can help with task by simply searching for “percentage calculator.” However, there may be a time when (however, unlikely it sounds) you may need to be able to calculate percentages without any digital assistance.
Before you can calculate a percentage, you should first understand exactly what a percentage is.
The word percentage comes from the word percent. If you split the word percent into its root words, you see “per” and “cent.” Cent is an old European word with French, Latin, and Italian origins meaning “hundred”. So, percent is translated directly to “per hundred.” If you have 87 percent, you literally have 87 per 100. If it snowed 13 times in the last 100 days, it snowed 13 percent of the time.
The numbers that you will be converting into percentages can be given to you in 2 different formats, decimal and fraction. Decimal format is easier to calculate into a percentage. Converting a decimal to a percentage is as simple as multiplying it by 100. To convert .87 to a percent, simply multiple .87 by 100.
.87 × 100=87
Thus, resulting in 87 percent.
Percent is often abbreviated with the % symbol. Presenting your answer as 87% or 87 percent is acceptable.
If you are given a fraction, convert it to a percentage by dividing the top number by the bottom number. If you are given 13/100, you would divide 13 by 100.
13 ÷ 100 = .13
Then, follow the steps above for converting a decimal to a percent.
.13 × 100 = 13
Thus getting 13%.
The more difficult task comes when you need to know a percentage when you are given numbers that don’t fit so neatly into 100.
Most of the time, you will be given a percentage of a given number. For example, you may know that 40 percent of your paycheck will go to taxes and you want to find out how much money that is. To calculate the percentage of a specific number, you first convert the percentage number to a decimal.
This process is the reverse of what you did earlier. You divide your percentage by 100. So, 40% would be 40 divided by 100 or .40.
40 ÷ 100 = .40
Once you have the decimal version of your percentage, simply multiply it by the given number. In this case, the amount of your paycheck. If your paycheck is $750, you would multiply 750 by .40.
750 × .40 = 300
Your answer would be 300. You are paying $300 in taxes.
Let’s try another example. You need to save 25 percent of your paycheck for the next 6 months to pay for an upcoming vacation. If your paycheck is $1500, how much should you save?
Start by converting 25 percent to a decimal.
25 ÷ 100 = .25
Now, multiply the decimal by the amount of your paycheck, or 1500.
1500 × .25 = 375
You need to save $375 from each paycheck.
Understanding the SOAP Protocol
Before we create an SOAPUI Test case, let us understand basics about the SOAP Protocol. This will help you use SOAP UI to test SOAP requests and response effectively.
SOAP stands for Simple Object Access Protocol. Below are the properties of a SOAP Protocol.
- It is an XML-based protocol for communicating between two different systems.
- It is a platform and language independent. Hence, a system developed using Java can communicate with a system developed in.NET.
- SOAP requests/response are transported via HTTP.
In this tutorial we will learn
Learn the SOAP Message FORMAT
A SOAP message is an ordinary XML document containing the following elements. Message can be either a request message or a response message.
After setting up the workspace which we had performed in the last tutorial, we have to create projects, test suites, test cases in order to test a given web service. Let us understand the steps involved in doing the same.
Create a Project
Step 1: Now depending upon the project, we need to import SOAP/REST protocol. We will create a new SOAP Project.
Step 2: We will make use following SOAP request
- Enter the Project Name
- Enter the path of the WSDL request. In this case http://www.webservicex.net/CurrencyConvertor.asmx?WSDL
- Click OK
Note:
- Upon checking – Create sample request for all operation' creates a sample request for all the available operations in the given WSDL. As soon as you enter the WSDL address, this option is checked automatically. You may uncheck it.
- Upon checking – Create, a Test Suite for the imported WSDL' creates a test suite within the project for the imported WSDL.
- Upon Checking – Relative Path' enables the user to save all the files relative to the project file.
Step 3: Upon creating the SOAP project with the above-said WSDL, we will be able to see that there are two operations that will be imported into the project.
Step 4)
- Expand the first request and double click on the 'Request1'. It will display the SOAP request in the XML format.
- Enter the From Currency and To Currency
- Click on the submit button
- Response XML will be displayed right side pane.
You may wonder why create Test Cases? When you can directly test WebService here…
Well, you can send a request for one operation. What about others? How many combinations of FOREX conversion you can do using this operation. You have to edit the request for each and every combination.
For example: If you want to convert from GPB to INR instead of USD to INR… You need to edit the operation again. So, one has to create a test suite/cases to have all possible scenarios tested without having to directly edit the operation itself.
Creating Test Suite
Step 1: Within the project, testers can create a test suite by performing a right click on the root of the project.
Step 2: We need to enter the name of the test suite and press OK.
Step 3: The created test suite is displayed the navigator pane as shown below.
Step 4: The test Suite window opens in the Right Pane. As we have just created there are NO test cases. Hence all options are disabled.
Creating Test Case
Step 1: Within a test suite, we can create multiple tests by performing right click on the 'test suite' and choosing 'New Test Case'.
Step 2: Specify the name of the Test Case and click 'OK'.
Step 3: The created test case has zero steps as shown below.
Note: We can see that the test case is added with zero test steps for all kinds of tests available. Upon adding the test steps the numbers in the bracket would change automatically. The functional test step
should go into 'Test Steps' while a performance test step should go into 'Load Test' and a security test step should go into 'security Tests'.
Step 4: We can insert a variety of test steps by performing a right click on test steps and selecting an appropriate test step as shown below. So if you were to test a REST WebService, you would select the REST Test Request.
Test Step Insert
Now let us add a test step to validate the imported SOAP request.
Step 1: Add a new step 'SOAP Test Request' as shown below.
Step 2: Enter the step name and click OK.
Step 3: Upon clicking 'OK', a dialog pops up to select the operation to invoke. All the operations are listed, and user can select the operation that they would like to invoke.
- There are two operations that will be listed. Both the Operations are the same except the SOAP version used.CurrencyConvertorSoap – uses SOAP version 1.1 where as,CurrencyConvertorSoap12 – uses SOAP version 1.2
- The Version does not matter for us in this context. Hence you can select the one of your choice.
Step 4: Whileaddinga test case, we can add standard assertions. Assertions also called as checkpoints/validation points which we will be dealing in detail in the next tutorial.
We can add following checkpoints/assertions while creating test case. Let us create a test case with the default option which means creating test step WITHOUT any of the below validation points
- Verifies if the response message is SOAP, upon executing the test.
- Verifies if the response schema is valid.
- Verifies if the SOAP response contains FAULT.
Step 5: Upon creating the test case, the request XML is shown below. The structure of the XML is explained within the below snapshot.
Step 6: The test step count is now incremented to one as we have just added one test step. Similarly upon adding load and security tests step, the corresponding number would be automatically incremented based on the number of steps added.
Send Request Manually & Reading Response
Step 1: We would like to convert the currency from USD to INR.
- FromCurrency – USD
- ToCurrency – INR
Next,
- We need to enter these inputs in place of the question mark which will be sent as request XML.
- After inputting those values into the corresponding XML tags, click 'submit request' button to check the response.
Step 2: Upon submitting a request the web service request is processed by the webserver and sends back a response as shown below.
By reading the response, we are able to conclude that 1 unit of USD = 63.525 units of INR.
Understanding the Soap Response & Log Panels
As explained at the beginning of this tutorial the SOAP messages are transported via HTTP protocol. Let us take a look at the RAW messages. This will help us learn how the SOAP request and response were transported by HTTP.
Step 1: Click 'RAW' Tabin both SOAP-UI request Window.
- The Request is posted to the webserver. Hence, the POST method of Http is used.
- The SOAP Request is transported in the body of the Http message as shown below
Step 2: Nowclick 'RAW' Tabin SOAP-UI Response Window to understand how the response is sent via HTTP.
- After processing the request, the Http response code (200) is shown which means it is a success. The webserver has processed it successfully.
- The SOAP response is sent back to the client as part of the body of the HTTP message.
A Quick snapshot of the Http Response codes for easy understanding and debugging. The below table will help you to trouble shoot based on the HTTP code received from the webserver.
How To Calculate Test Value
Step 3: Let us understand the other information that are displayed in the test case window.
- Represent NO header in the request that is being sent
- Represents NO attachments in the request that is being sent to the web server.
- Represents 12 header information and the same are displayed upon clicking on it.
- Represents that there are no attachments from the response message.
LOGS PANE:
Logs pane has the complete information regarding the transaction between the client and the server. Users will be able to see the tabs of the Log pane as shown below. We will discuss the most commonly used log panes when working with SOAP-UI.
SoapUI Log – Displays the response information from webserver. The same information is stored in soapui.log file of the SOAP-UI installed folder under 'bin' directory.
Http Log – Displays all the HTTP packet transfer. All the information in 'RAW' is shown in HTTP log.
Error Log – Error log displays all the errors that we have encountered during the entire project session. The same information is available in 'soapui-errors.log' present in the 'bin' directory of the SOAP UI installed location.
Memory Log – This tab monitors the memory consumption and displays it in the form of the chart as shown below. It is really helpful when there is a memory intensive operation is performed.
Automation Test Coverage
Now that we have created test suite, test case, test step and got a response, next step is to validate the response. We will deal with types of assertions in the next tutorial.