If you ask your testers about the challenges they have to deal with every day, the majority of them will admit that race against time is one of the most problematic. At some point, the software development team finishes coding, and the release date is coming closer and closer. Managers are watching the testers’ back, requiring to predict and prevent all the possible bugs, but silently hoping they won’t find anything to fix. Fortunately, there are some tools which support the collaboration between developers and testers. One of them is Jira, a well-known project management software. The Atlassian suite upgrades workflow organization and makes it more efficient. But we can’t forget that testing teams have their own, special needs. Can Jira meet the expectations of even the most demanding projects? It turns out that indeed, it can! You’ll be surprised when you discover the miracles Jira Software with the right extension can do for your test management.
Why test inside Jira?
The first answer which comes to mind is: because the time of your team members matters to you. It would be a waste not to benefit from the solution which can easily speed up testing-relation actions by structuring and prioritizing tasks. Besides, embedded testing will always impact the final product in a good way. Jira along with a dedicated test management app allows each stakeholder to monitor the process step by step, keep track of implemented changes, and see the reports anytime it’s necessary. The last example confirms a little bonus point: in addition to preventing bugs or duplicating work, testing in Jira also supports avoiding these famous meetings that should’ve been an e-mail.
Tests performed in Jira not only economize your team’s time but also your company’s money. As the majority of people working in the software development industry already knows Jira, you won’t have to provide any additional training before the actual work starts. Jira apps are simply cheaper than external testing tools, and they still offer all the functionalities or even more. Considering the prices and the wide range of features, test management inside Jira seems to be the right way to go. Here’s a quick summary of what Atlassian product can do to improve your testing teams’ efficiency.
Preparation of the competitors
Requirements and Test Management for Jira (RTM) by Deviniti and Test Management for Jira (TM4J) by Adaptavist are two opponents in a race where the goal is elimination of bugs from your software. Each app has strong and weak points. The key to your success is choosing the right one, and that’s what we’re here to help you with. But first, before we start going through the pros and cons, it’s a good idea to analyze your individual needs. There are some factors which we believe should be taken into consideration first:
- the size of your team;
- the number of requirements;
- the general complexity of the project;
- the risk of unexpected changes;
- the project management approach.
Gathering this kind of information is definitely worth your while, as it will help determine which conditions will be the best for your project and which features you really want. For example, if you have less than 100 people in the team, we recommend you using Jira Cloud instead of Server. Not only because it’s cheaper, but also more suited for smaller companies. The number of requirements constitutes an important question as well. If your project requires many of them, you’ll need a tool which provides its users with embedded requirements management. With this in mind, even if unexpected changes occur, you and your team will be well prepared.
If you already got your starting point specified, we can now move to the actual comparison. We’ll look at the most common testing challenges and see how the tools selected for our comparison can help deal with them.
Ready, set, go!
Stage 1: Requirements management
The testing process should be treated as a whole. It’s not possible to execute complete functional tests without well-managed requirements. The more complex the project, the more important this initial stage is. It leads to the conclusion that integrating requirements into the process is a good way to track the relations between them and the other testing objects. Requirements and Test Management for Jira (RTM) makes it possible. Thanks to our app, you can see at a glance if the requirements are safely covered by test cases throughout the process and make sure nothing’s omitted. Additionally, RTM for Jira lets us observe which requirements generate most bugs during the execution.
There are different types of requirements, and structuring them well is related directly to the stakeholders’ and customers’ expectations. The best way to do this is through Jira issue types. With RTM, you can not only categorize your requirements but also organize them in a transparent tree with folders and subfolders. On the other hand, TM4J doesn’t provide its users with dedicated issue types nor any kind of structure for requirements, which makes effective planning of the further steps practically impossible.
RTM has four dedicated issue types for requirements to choose from
Stage 2: Intuitive process
This circuit starts well for both our runners. After a quick installation, typical for the majority of Jira testing apps, here comes the time for setting up Permissions. At this stage, Adaptavist’s and Deviniti’s tools give us many customization possibilities. TM4J offers even a built-in permissions module, which we can divide the roles and define who’ll be able to implement what changes throughout your project development, regarding separately each object. It’s also the only Jira built-in test management app on the market which provides the possibility of versioning test cases and creating large libraries for the issues ordered by labels. But this is where its advantages end, as later on, the testing process in Adaptavist’s tool gets more complicated. The test cases can be attached directly to the executions, which undermines the sense of creating test plans and inevitably brings chaos into the process.
Most of the objects in TM4J by Adaptavist are displayed as a list, which may not pass the test when there are many elements. Image source: Software Testing News
RTM for Jira guides us through our testing from end to end. We start by analyzing requirements, then move on to creating test cases and assign them relatedly. Prepared and organized objects are ready to be scheduled into test plans. This way the process gets maximally standardized, and we know exactly which steps or objects caused potential defects. It’s much easier to fix something when we know exactly where it comes from. What’s more, our app ensures all the stakeholders are up-to-date, as each stage and change is easy to track thanks to a high-grained relations view.
Thanks to well-described relations in RTM for Jira, your QA team will know exactly where the defects come from and be able to fix them right away
Stage 3: Transparent relations between the objects
As RTM for Jira is originally made to bring the whole software project into a single tool, it enables full traceability. And we’re not talking only about requirements and test cases. As the successive testing stages result from each other, it’s important that everyone involved in the process could keep up with the dependencies. Of course, by everyone involved we mean not only testers, but also analysts, software developers, product managers, and designers. The main goal of RTM is to provide its users with all the benefits of embedded testing, so it gives a wide range of relations tracking options. We can observe direct and indirect links between the elements on each issue screen on the Relations tab. There are also two transparent reports which display connections in an understandable way: the Traceability Matrix and the Requirement Coverage. The first one allows verifying correlations between any two baselined issue types using many-to-many relationship comparison. The second is generated as a table and focuses on requirements and their coverage by test cases, test plans, executions, and defects if they occur during the process.
Requirement Coverage Report in RTM allows tracking the relations between requirements and testing objects
This time Adaptavist’s TM4J falls behind quite significantly. Inside the app, requirements can’t be directly connected to other testing objects, as they’re basically separated from the process. This is the main reason why their relations with related test cases can be set only through web links. Adaptavist’s test management tool for Jira offers the Traceability and Coverage reports too, but their design is rather simple and seems incomplete.
TM4J’s Coverage Report shows only basic information. Image source: Adaptavist App Documentation
Stage 4: Automated testing support
TM4J takes the lead. The tool makes it possible to enter the code through built-in REST API. It also integrates with CI tools like Jenkins or Bamboo, supports writing tests in Cucumber and enables importing data from JUnit XML. This is a strong point of Adaptavist’s app, as automated testing is getting very popular these days, and more and more teams decide on facilitating their work with it.
TM4J enables adding scripts through REST API. Image source: Adaptavist App Documentation
As to RTM for Jira, it focuses mainly on manual testing which is why it doesn’t allow to run scripts, nor integrates with automation servers. The good news is we’re currently working on it, and you can expect automation features to be added in the near future.
Stage 5: Customizable test results reports
RTM for Jira is back on track! Even though most of the testing tools include built-in test reports, they usually present a completely different approach. Requirements and Test Management for Jira features four fully flexible reports, all ready to be exported to a .pdf or a .csv file. Two of them were described in the paragraph concerning relations. The other two are Test Execution and Test Case Execution reports. The first one shows the overall progress of our Test Plans, and the second goes in-depth for each test case. Both can be set up with filters, so we can select the data only for a particular Project, Test Plan, Execution, Environment, TCE Created Date, or TCE Assignee. The reports are generated as transparent charts with detailed tables below. With RTM, the risk of misunderstanding between the stakeholders decreases to a minimum!
Transparent test result reports show the progress of testing and are ready to export to a .pdf or .csv file
Reporting in Adaptavist’s TM4J is possible, but not simple and definitely not intuitive. The tool provides various kinds of the Test Execution report. Instead of customizing our results with filters, we’re obliged to choose their display per parameter as separate reports. The forms of generated reports are also hard to read and will require an additional explanation from one group of team members to another.
In Adaptavist’s tool, there are many versions of the Test Execution report instead of filters. Image source: Adaptavist App Documentation
And the winner is…
In this race, Requirements and Test for Jira (RTM) evidently emerges as a winner. From the very beginning, almost to the end, our app leaves its opponent far behind. It tripped only when it came to automated testing support, which is still yet to come, but in the end, it came back in glory. All the other features, like requirements management, reports, and relations-tracking solutions support your testing at a high level. What’s more, RTM for Jira is now available for Jira Server as well as for Jira Cloud, and it looks identical on both hostings, so in case of a need for migration, it can be performed smoothly.
TM4J tries to catch up with the flexible Permissions settings and the automated testing support, but for most teams it still won’t be enough. Although Adaptavist’s app has all the functionalities necessary to perform your test management inside Jira, the clarity of the whole process leaves a lot to be desired.
If you’ve decided to test our app, RTM for Jira, take a free 30-day trial from the Atlassian Marketplace. You can also book a live demo via Calendly to see the app in action. We also invite you to read more on bringing test management process inside Jira on Deviniti blog:
- RTM vs Zephyr – choose your fighter for testing in Jira
- 7 ways to structure software requirements and test cases in Jira
- Why you should bring Requirements and Test Management right inside your Jira
Kasia Kornaga is an Atlassian Apps Content Specialist, responsible for writing about requirements and test management on the Deviniti blog. She tries to discover and understand the mysteries of software testing process in order to share the knowledge with the readers interested in the subject. As an SEO enthusiast, most of all she values helpful, unique content where users can find answers to their questions, so she probably knows what you will look for before you even start. When she’s not writing, you can find her at the theatre, at home with a book or passing on city streets on her bike.