Many of our clients have been asking us about the differences between TestFLO and Zephyr. That’s why we decided to create this comparison to provide them with an objective answer to their questions.
We don’t want to say that one solution is simply better than the other. We believe that people who are looking for a solution should have full access to relevant information. Only then they can pick the right tool for their particular problem.
What follows by no means exhausts the topic. That’s why we encourage you to share your experience and opinions in the comments section.
Are you using Jira to manage projects at your organization?
If so, there’s something you should know: you could be using the same instance for test management as well.
Transforming your Jira into a testing environment is quick and easy thanks to the wide range of available plugins that offer testing functionalities.
Testing your software in Jira is a smart move. You won’t need to create a new testing infrastructure or implement new tools. Your team knows JIRA very well, so there’s no need for additional staff training either.
Have a look here to learn more about using Jira for test management.
So how do you start testing in Jira? First things first, you need to pick the right plugin.
In this article I compare our add-on TestFLO with the leading testing solution on the Atlassian marketplace, Zephyr, to help you decide what features you need to optimize test management at your organization.
Once you install and launch Zephyr, the tool will be instantly ready to use in all your projects. It can’t get easier than that.
Launching TestFLO is a bit more complex, but you’ll be able to configure the plugin to be available for a given project.
That’s something you set up in the configuration. All you need to do is open your Administration dashboard, click on the list of your projects, and choose Test Management from the left-hand menu.
At this point, you can launch three testing modules:
- Test Repository
- Test Execution
You can decide whether all modules will go into one project, or divide these modules between projects. Each module also includes a number of options that can be enabled selectively or as a group.
Zephyr offers relatively modest configuration options. Users can configure test completion or steps statuses. The plugin offers functionalities like link management, but its global configuration is rather simple.
In TestFLO, users can configure the plugin in two ways – at the project level (as described above) and at the global level.
In this configuration users will find 5 categories that relate to different TestFLO elements:
- Global Configuration
- Test Cases
- Test Case Templates
- Custom Fields
TestFLO configuration is based on JIRA – our fields are JIRA custom fields, so users can modify the plugin at the level of JIRA mechanisms.
Zephyr is very easy to use. The plugin doesn’t require the user to do anything more than installation. The intuitive structure and clear naming help users to quickly make sense of the plugin and start using it without the need to consult the documentation. Zephyr’s entry threshold is relatively low.
TestFLO requires a bit more from its users. The configuration of the plugin is a more complex, but the standard setup allows to quickly start working with the plugin. TestFLO’s entry threshold is medium.
Zephyr uses its own objects, but its interface fits JIRA seamlessly. Users generally find it easy to navigate the plugin.
TestFLO uses JIRA elements, so it naturally fits every JIRA instance. That’s why users can start working with TestFLO without any problems.
In Zephyr, users can develop a new test by creating a special issue type called Test. This issue type includes test steps and a table that represents the completion of these steps.
The plugin also features Test Details History section where all changes applied to tests are logged. Tests are created for a particular project and can only be used in that project.
In TestFLO, users can start testing by creating an issue type called Test Case Template (TCT). That issue type allows to define test steps. The steps are fully customizable and users can easily create test views that match their needs.
The plugin features a list of all test created from that TCT, as well as a an information panel that displays the completion states of these tests.
Test planning in Zephyr is based on their custom object – Test Cycles. A test cycle is a collection of tests to be executed that can only be created for a given version (Fix Version).
Just like tests, test cycles are created for specific projects – however, users can easily move a Test Cycle from one version to another.
Additionally, every project contains an ad hoc test execution mode created for each version – if users want to quickly execute a single test, the test will go straight to the ad hoc.
In TestFLO, test planning is based on the creation of an issue type called Test Plan (TP). A Test Plan includes a dedicated panel in the place of Sub-tasks where users can see all Tests Cases to be executed.
Users can create Test Cases are part of the Test Plan, but also as part of a single issue – they’ll simply look like Sub-tasks.
Have a look here to see how to create a test plan in TestFLO.
In Zephyr, adding a test to the Test Cycle is enough to make sure that it’s executed together with the rest of the cycle. That’s another Zephyr-specific element that users won’t find in JIRA.
All test executions are visible on the issue screen. Tests can be executed at the level of detailed execution – that’s where users can set test step statuses, as well as add comments, defects and attachments. This feature also record test execution history.
Another way for executing tests in Zephyr is the quick execution at the test or test cycle level, as well as in Execution Navigator — users can do that by simply changing the status of a test.
In TestFLO, test execution is based on Test Case sub-tasks that are created for particular Test Plans. Users can create them at the level of Test Case Template or Test Plan. With TestFLO, users can easily mark the status of each test step included in a test.
The status of the entire test is defined on the basis of issue status through workflow transitions. That’s where users can also link a defect to a test and check a smart information panel that displays test step completion progress in their Test Plan.
In Zephyr, each test step is based on three standard columns:
- Test Step
- Test Data
- Expected Result
As test steps are added, new data will appear in this table. Users can delete, clone, or change the order of test steps through Drag & Drop, or customize their content. As the test is executed, users can can change the status of each step, add a comment, attachment, or defect.
TestFLO is more flexible in test step creation. To start, every test step includes three standard columns:
- Expected Result
Users can configure the table – for example, the names and size of columns – by configuring the field Steps. Test steps can be imported from CSV in editing mode. Moreover, users can group together and delete test steps, as well as change their order through Drag & Drop.
All test steps are presented in an understandable table where each line corresponds to the instructions for test execution. Users will find an additional Status column in Test Case sub-tasks.
In Zephyr, users can search for tests just like for other issues – through the Issue Navigator. Zephyr also offers Execution Navigator based on their custom language ZQL, a solution similar to the Issue Navigator where instead of searching for issues, users can search for executions.
In TestFLO, users can find their tests using the Issue Navigator that has been enriched with a few new JQL functions.
Zephyr offers limited requirement management options. Users can basically link other issues to test issue types – and that functionality is used in Traceability Reports.
In TestFLO, users can connect issues that were defined in configuration as requirements with test issue types (TCT, TC, TP) or defects (through the Requirement field).
Each connected element is visible in the right panel at the level of requirement where users can also create TCT and TP that will be automatically connected to that requirement.
In Zephyr, defects can be created at the test execution level or particular test steps. Users can also link defects that they’ve created before.
In TestFLO, users can create a defect at the level of sub-task type Test Case. In that case, the defect will be automatically connected to that TC. If the TC has an attached requirement, the defect will also be attached to that requirement.
Test Import and Export
Users can import tests to Zephyr from an Excel file or XML (TestLink format). To do that, they’ll need to use an external tool called Zephyr for JIRA Test Importer Utility.
Tests can be exported to CSV, Excel, HTML or XML files. A test cycle can be exported to CSV format as well.
In TestFLO, users can import tests from CSV files. Test steps can also be imported to previously created issues. A TCT can be exported to a CSV file.
Zephyr offers a relatively simple reporting feature, but it does the job. A special page sums up the information about test executed in a project and offers a list of test cycles together with their executions.
Users can employ JIRA gadgets for reporting of testing information as well. On top of that, Zephyr offers 5 gadgets for test cycle reporting, test executions, or defects from JIRA dashboard screen. In Traceability Reports section users will find 2 report types.
TestFLO uses dashboards with JIRA gadgets, as well as the Issue Navigator and issue screens. Thanks to Panel Filter Results, users can list issues selectively. There’s no dedicated reporting feature available to TestFLO users yet.
Zephyr offers a dedicated Bamboo plugin that allows for configuring test plans in Bamboo to create test issue types in JIRA. These tests will have executions created for them.
Launching the plan once more, users will see their tests and executions added to the previously chosen test cycle. Users can create a new cycle for each build.
To automate testing in TestFLO, users need to install a special plugin – TestFLO Automation. The plugin cooperates with Jenkins and Bamboo. It’s based on a post-function in the test plan workflow transition that gives test results from the defined build plan. Each test creates a sub-task.
Zephyr’s documentation is simple and concise, but it doesn’t always include updated information related to the newest versions of the plugin.
Users will also find plenty of practical information in TestFLO’s documentation which is updated every time a new version is released.
Thanks to these dedicated plugins, JIRA users can easily transform their instances into powerful environments for test management and execution.
Want to learn more about testing in JIRA? Have a look at this article my colleague wrote about the top 5 test management tools for JIRA teams.
If you’ve got any questions about testing in JIRA, post them in the comments section – I’ll be happy to help you out.