- Basic concepts
- Quick guide for Jira Administrator
- Quick guide for QA manager
- Quick guide for testers
- Demo environment
- Reporting & tracking
- Reporting and tracking overview
- Requirement Test Coverage Report
- Requirement Traceability Report
- Requirement Test Execution Report
- Test Plan Execution Report
- Test Plan Iterations Report
- Test Execution Report
- Test information panel
- Issue panels
- Export test issues
- Dashboard Gadgets
- Administration & Configuration
- Projects' organization
- Create project
- Project configuration
- TestFLO settings
- TestFLO Permissions
- Custom fields
- Advanced workflow configuration
- Panel Filter Result
- Import/Export TCT
- Test Automation configuration
- Upgrade reports
- Upgrade guide
Test Automation project configuration
The Test Automation tab appears in the project configuration when Jira Administrator adds a CI Server into the CI Servers configuration and sets the given project. On this page, you can decide if you want to create Test Case Templates along with the creation of Test Cases and set statuses mapping for Cucumber test results.
Configuration in the Test Automation tab is available to users with permissions:
- Jira System Administrator
- Jira Administrator
- Project Administrator
To get to the Test Automation tab:
- Click on Projects in the main bar in Jira and select a project from the project list where you run the automated tests.
- Go to Project Settings and click on Test Management from the left side menu.
- Select the Test Automation tab.
View of the Test Automation tab.
Creating Test Case Templates from test results
This option gives you the possibility of enabling the creation of Test Case Templates in the selected project. The Test Case Templates are created together with Test Cases which are based on automated tests. The benefits of this approach are:
- Created Test Case Templates can be reused - you can only add selected Test Case Templates to the Test Plan so you can limit which test results are to be imported.
- Test Case Templates and their executions in Test Cases can be linked with Requirements and tracked in TestFLO Reports.
To enable Test Case Template creation it is required to enable the Test Repository module in Test Management in a project where you want to store the Test Case Templates.
To enable the creation of Test Case Templates select the Create in current project or Create in another project. Available options:
- Don’t create - Test Case Templates will not be created. This option is marked as default.
- Create in current project - Test Case Templates will be created in the current project.
- Create in another project - Test Case Templates will be created in selected the project from the list.
After you set the creation, when you run automated tests, the Test Case Templates will be created along with the Test Cases. If the Test Case Template has already been created from the source Test Case, they will not be created again if the Run automated tests operation is used again.
CI job custom parameters
This option is associated with the Run automated tests operation on the Test Plan and is used to send additional parameters to a build on the CI Server. By setting additional parameters you can:
- add more information to a given build,
- make the script execution on the CI Server side dependent on the needs of a given team or users, version, component, or other factors,
- select for rerun only those Test Cases which you are interested in (without importing all results of automated tests).
In this option, you can manually set the additional parameters whose values TestFLO will send to the CI Server during the Run automated tests operation on the Test Plan.
Go to the Running tests with additional parameters configured page and see how using parameters looks in action.
To add parameters:
- Click the Add button.
- In the Display name provide a clearer parameter name that the user will see when using the Run automated tests operation.
- In the Parameter name provide the exact technical parameter name.
In order to be able to transfer parameter values to the Jenkins CI Server, each parameter configured here must also be added to the given job in Jenkins. In the case of the Bamboo CI Server, there is no need for additional configuration.
- In the Field type select type of the parameter. You can choose from three types:
- Text field - plain text field
- Select field - a field with a selection list
- Jira field - field from Jira. Supported fields: Number field, Text field (single line), Radio buttons, Select list (single choice and multiple choices)
- In the Field mode decide in which mode the parameter will be used:
- Editable - allows users to edit the value of a parameter when using the Run automated tests operation
- Read-only - displays the default value of a parameter when using the Run automated tests operation
- Hidden - the set parameter value will not be displayed when using the Run automated tests operation. This is useful for purely technical parameters.
- In the Value(s) enter the default value (for Editable field mode) or the target value (Read-only and Hidden field mode).
Fill in the values for the given field types:
- Text field - fill the field with a text value.
- Select field - define the options that can be selected from the list. To configure options, click the pencil next to the field. On the Manage select field options screen, add a Label and Value for the option. Click the Add button. Add as many options as you need and click Save.
- Jira field - select a Jira field from the list. Remember that this must be a field that is configured for the Test Plan.
- Save the changes by clicking the Checkmark.
- To add more parameters, repeat steps 1 through 7.
You have added a parameter/s. All parameters that you configured with the Editable option will appear for users to change when using the Run automated tests operation. Parameters that you have configured as Read-only will appear to users but they will not be able to change them. The Hidden option will not display the set parameters at all.
After configuring the parameters, you can manage them:
- Use drag & drop to set the order in which the parameters are displayed.
- To edit, click the Pencil icon.
- To delete, click on the Bin icon.
Selected Test Cases Parameter
This parameter is responsible for the Select Test Cases to run option when using the Run automated tests operation. With this option, you can select from the existing Test Cases only those that you want to run again within the same or a new iteration.
The parameter stores the key values of selected Test Cases by the user during the Run automated tests operation. These values are the connection keys between each Test Case and the test on the CI Server, which are stored in the Automation Test Key field on the Test Case.
Important information regarding Maven:
- to execute selected Test Cases in the Run automated tests dialog, the maven-surefire-plugin version must be at least 2.22.2 (in pom.xml related to the code of your automated tests).
- if the Bamboo agent executing the tests is installed on Windows, then the maven task does not support spaces in testflo.automationTestKeys variable when used in goal. I this particular case this limits the usage of tests to those where names do not contain spaces (some programming languages like Kotlin allow spaces in function names).
There is a limitation of sending testflo.automationTestKeys parameter value to the Bamboo CI Server (the parameter name may vary depending on your settings):
- If the parameter value does exceed 4000 characters then TestFLO will send the parameter and divide them into parts (each part of the sent parameters will contain 4000 characters).
- Each parameter will have an index starting from 1. For example, testflo.automationTestKeys1, testflo.automationTestKeys2, testflo.automationTestKeys3, etc.
- Additionally, a parameter named testflo.automationTestKeys-paramsNumber will be sent to show the number of all testflo.automationTestKeys parameters.
If you want to use the transferred parameters in your script you will have to combine them into one parameter on the Bamboo CI Server side.
To configure the Selected Test Cases Parameter option:
- Click slider next to Selected Test Cases Parameter.
- Optional: In the Parameter name you can change the name of the parameter that stores connection keys.
- Configure a string parameter on the Jenkins CI Server for the jobs on which you want to use the Select Test Cases to run option. The parameter name must be the same as in the Parameter name field.
For the Bamboo CI Server you do not need to configure this parameter additionally.
The configuration is complete. From now on, an additional option Select Test Cases to run will appear on the Run automated tests operation dialog, where you can decide which Test Cases to run again in the same or a new iteration.
Go to the Running tests with Selected Test Cases Parameter configured page and see how this parameter is used in action.
Cucumber step status mapping
In this option, you can map the statuses that TestFLO receives in the Cucumber test results file to the statuses in the Steps field on the Test Case. There are three statuses from Cucumber to map Passed, Failed, and Skipped. You can assign each of these statuses a corresponding status in the Steps field. If you do not assign any status then when you import the test results TestFLO will assign the default steps status.
NUnit step status mapping
With this option, you can map the statuses that TestFLO receives in the test results file of the parameterized NUnit tests to the statuses in the Steps field in Test Case. There are three statuses from NUnit tests to map Passed, Failed, and Skipped. You can assign each of these statuses a corresponding status in the Steps field. If you do not assign any status then when you import the test results TestFLO will assign the default steps status.