Tools for Automated Testing: Katalon Studio

Katalon Studio is an Integrated Development Environment (IDE) that facilitates the execution of Selenium tests through a user-friendly interface. Even as someone without prior experience in automated testing, I found it straightforward to run tests with just a single click.

Idea 1: Using Data Attributes for Element Identification

To identify elements effectively, automated tests typically rely on various methods such as XPATH, CSS selectors, and element attributes. However, since these elements can change with updates to the product or UI, I opted to use data attributes for most test elements to ensure stability.

Idea 2: Integration with Xray

To manage test cases and results, we implemented Xray, a tool that allows us to automatically synchronize Katalon execution results with Xray. Each Katalon test case is assigned an Xray test case ID, enabling seamless reporting of test outcomes via Xray's API.

The Struggles of Automated Testing

Although the initial phase of automation went smoothly for about six months, I soon encountered several challenges.

Testing Across Multiple Browsers

Our automated tests were initially developed only for Chrome. However, as the number of test cases grew, we were also asked to conduct tests on Firefox and Edge. While Katalon allows for browser selection during test execution, running tests designed for Chrome on Firefox led to a significant number of failures—approximately half of the test cases. The discrepancies in browser behavior necessitated unique implementations for each browser, complicating test management.

Impact of Product Upgrades on Automated Tests

When a new version of the product was released, automated tests often failed due to specification changes or behavioral differences. Consequently, I had to revise the automated tests each time a new version was rolled out. To manage this, I created a structured directory for each product version to maintain previous tests.

Test Code Management Challenges

By the time I transitioned away from this project, I had successfully automated 15 out of the 27 test cases. However, the sheer volume of test cases, combined with the challenges previously mentioned, made management quite difficult. Minor changes in the test code before release often led to complex debugging processes to determine whether the issue stemmed from the product or the test code itself.

As I developed tests with intricate functions, I recognized the importance of minimizing changes. I aimed to consolidate duplicate code to simplify future modifications, but I should have been more strategic about which elements should remain distinct.

Katalon's Quirks

Katalon is user-friendly, but it comes with its own set of peculiarities that differ from traditional Selenium usage. For instance, issues arose when using wait functions for specific elements, leading to unexpected errors. Additionally, while Katalon provides features for handling Shadow DOM, I often found it more effective to revert to plain Selenium for these cases.

Data Attributes and Collaboration

While data attributes proved beneficial for element identification, I faced challenges when new features lacked these attributes. This underscored the need for closer collaboration with the product development team to ensure that data attributes were consistently implemented.

The Final Outcomes

Ultimately, I managed to automate 15 of the 27 test cases, representing 40-50% of the total. Automated testing was successfully carried out across Chrome, Firefox, and Edge (Chromium). However, the total execution time for tests exceeded three hours, primarily due to serial execution. This could be significantly improved by parallelizing the tests.

Reflecting on My Experiences

Throughout this journey, I learned that both manual and automated testing possess unique advantages and disadvantages. I realized that not all tests should be automated; complex behaviors are often better suited for manual testing, while simpler tests can benefit from automation.

The key takeaway is that the stability of automated tests should take precedence over merely increasing the number of tests. Given the propensity for automated tests to change, careful creation and maintenance are crucial.

In Conclusion

After a year of hands-on experience in automated testing, I regret that I left the process with some instability and incompleteness during my internship. Nevertheless, I hope to contribute positively to the team through this article.

Stackademic

Thank you for reading this account of my experiences. If you found this useful, please consider showing your support by clapping and following the writer! 👏 You can also connect with us on Twitter (X), LinkedIn, and YouTube. To learn more about our mission to democratize free programming education worldwide, visit Stackademic.com.