Testing Guidewire: Smart Code Analysis with HealthCheck

Test code often plays second fiddle to production code. Yet, the quality of our tests directly affects the reliability and maintainability of our entire codebase. In a previous article, we explored the importance of code quality and the aspect of designing and establishing a clean, reusable architecture. Now, we turn our attention to an equally critical aspect: monitoring test code quality, especially when working with less experienced teams.

In this article, we will discuss code quality by examining a past project where Kimputing was contracted to provide test automation for a mid-sized insurance carrier. This project gave rise to the CenterTest HealthCheck product, code-named Project Eir after the Norse goddess of healing. Project Eir aimed to address a critical challenge in our industry: how to maintain high test code quality when test automation framework users may lack extensive coding experience. We recognized that while CenterTest excels at simplifying test development and execution, the real challenge lays in empowering less experienced users to write high-quality tests.

The need for Project Eir arose from observing that even the best frameworks and tools can’t compensate for a lack of testing and coding expertise. We realized that improving test code quality required more than just providing robust frameworks; it demanded a holistic approach to monitor, report and coach on test code quality that considered the varying skill levels of our framework users.

In the following sections, we’ll explore the specific challenges we faced with inexperienced framework users, the innovative approach we developed to support them, and the transformative results we achieved. Whether you’re a testing professional or a developer looking to improve your team’s test code quality, our story offers valuable insights into revolutionizing the test code quality process for users of all skill levels.

By implementing Project Eir, we’ve changed the approach to test automation and quality. While other frameworks merely provide tools for test automation and leave quality concerns to the user, Kimputing offers a comprehensive solution. We provide CenterTest for efficient test automation, coupled with our HealthCheck feature to actively support and enhance code quality.

Our approach doesn’t just end with automating tests; it actively guides users towards best practices, ensuring that automation efforts translate into genuine quality improvements.

This not only enhances the reliability of software but also fosters a culture of quality that permeates throughout the entire development lifecycle. With Kimputing’s solution, clients don’t just get a tool – they get a partner in building a robust, efficient, and high-quality testing process.

The Challenges

Our path to improve our client’s test code quality, specifically for No Name Insurance Company, revealed numerous challenges that extended beyond simple tooling issues. Despite the implementation of CenterTest as their primary test automation framework, several obstacles hindered the achievement of high-quality, maintainable test code:

• Skill Gap: The client’s team, while enthusiastic, lacked the necessary testing expertise. The company had mistakenly assumed that basic programming training would be sufficient for writing good test code.

• Reluctant Transitions: Experienced developers showed reluctance in moving to testing roles, further widening the skills gap.

• Inconsistent Practices: The absence of standardized guidelines led to a mix of coding styles and approaches across teams and projects, complicating maintenance and knowledge sharing.

• Tool Limitations: Existing static analysis tools like SonarCloud and IDE problem checkers, while useful for production code, fell short in addressing test-specific issues such as improper assertion statements, hard-coded test data or brittle Selenium selectors.

• Cultural and Organizational Hurdles: The challenges weren’t purely technical but also involved cultural and organizational aspects of the client’s development process.

These multifaceted challenges highlighted the need for a comprehensive, tailored solution that could address both technical and non-technical aspects of test code quality. This realization became the catalyst for developing Project Eir.

Our Approach

1. Parsing Java Code

The foundation of our approach was to create a detailed representation of the code structure. We developed a set of Data Transfer Object (DTO) classes to mirror the existing codebase. These DTOs captured all the essential elements of the Java code, including classes, methods, fields, annotations, and their interrelationships.

With the DTO structure in place, we then turned to JavaParser, a powerful library for analyzing Java code. JavaParser parses the existing codebase and extracts all the information to populate the DTOs. Here’s a simplified example of how we use JavaParser:

This creates a rich, interconnected representation of the entire codebase, both the basic structure and detailed information like method bodies, annotations, and even comments.

This comprehensive model lays the groundwork for sophisticated analysis. We can now easily traverse the relationships between different code elements, understand inheritance patterns, track method calls, and much more. This deep understanding of the codebase forms the foundation for the CenterTest HealthCheck process by enabling custom validations and report on the consistency and quality of our users’ test code.

2. Custom Validation Methods

The CenterTest HealthCheck process utilizes custom validations, each able to address specific aspects of test code quality, to effectively support multiple customers with varying requirements.

They ranged from basic code statistics to complex behavioral analysis.

These custom rules encode Kimputing’s best practices gleaned from over 30 years of test automation experience and 20 years testing Guidewire InsuranceSuite. This ensures that HealthCheck aligns perfectly with our recommended testing approach and technology stack.

3. Dynamic Rule Sets with HOCON

Flexibility is key to the long-term success of any quality control tool. We chose HOCON (Human-Optimized Config Object Notation) to create a dynamic, adaptable rule set system. HOCON offers a more readable format and sophisticated structure compared to simple properties or JSON files, making our configurations easier to understand and maintain.

In our implementation, we annotate each rule in the code with an identifier and description.

We then use a ruleset.conf file to combine different rules into sets:

This approach lets us group rules easily by type or for specific purposes. For instance, we can create different rule sets for pull request checks, general code health assessments, or focused refactoring efforts. The flexibility of HOCON allows us to adapt our rule sets quickly as our needs grow.

By using HOCON, we have created a system that is both powerful and accessible to teams who might be uncomfortable with more complex configuration formats. This accessibility has been crucial in getting widespread adoption and input on our quality standards across the development team.

4. Git Handling for Continuous Checks

Git integration transformed the tool from a post-commit code checker into a real-time quality control system, actively monitoring and improving code throughout the development process. We use the org.eclipse.jgit library to interact with our Git repositories, allowing us to:

• Clone repositories

• Analyze lists of commits

• Compare diffs between two commits

• Evaluate commits within a date range

• Assess the first commits in a month

This Git integration enables various modes of operation:

• Diff analysis for pull request checks

• Monthly analysis to evaluate code quality trends over time

• Analysis of specific commit ranges for targeted quality assessments

By integrating these checks into our CI/CD pipeline, we have created a safety net that catches quality issues before they impact our main codebase.

Results and Benefits

The impact of HealthCheck has been transformative. At the end of each analysis, a comprehensive report is generated using AsciiDoc, which is then converted to both HTML and PDF formats. These reports provide invaluable insights into code quality improvements and serve as a basis for the ongoing refinement of our development processes.

We’ve seen substantial reductions in code review time, as many issues are now caught and fixed before the pull request stage. New team members onboard faster, learning coding standards through immediate, context-specific feedback. Perhaps most importantly the increased confidence in the test suite allows you to speed up your release cycles while maintaining high-quality standards.

Even more?

While HealthCheck was conceived to address challenges in test code quality, particularly for less experienced framework users, the approach we developed has far-reaching implications beyond this specific context. The methods and tools we developed offer a robust framework for analyzing and improving code quality that can be applied to any Java codebase.

Whether you are working with production code, test suites, or ancillary systems, the principles behind Project Eir can be adapted to suit your needs. Organizations facing various code-related challenges can benefit from our approach:

• Teams dealing with legacy systems can use similar parsing and analysis techniques to understand and improve complex, long-standing codebases.

• Companies looking to enforce consistent coding standards across diverse teams can adapt our dynamic ruleset approach.

• Projects aiming to improve overall code quality, regardless of the developers’ experience levels, can implement custom validation methods tailored to their specific needs.

• Managers seeking to enhance development practices and code reliability can use our continuous integration approach with Git as a model.

The beauty of our solution lies in its flexibility and adaptability. While we focused on test code, the core ideas – parsing code into a comprehensible structure, applying custom validations, using dynamic rulesets, and integrating with version control – are universal in their application to Java development.

By sharing our journey, we hope to inspire and equip other teams and organizations to elevate their code quality and development practices. Whether you are a testing professional, a Java developer, or a software architect, the lessons learned from Project Eir can provide valuable insights into creating more robust, maintainable, and high-quality code across all aspects of your Java projects.

Arek Frankowski
Senior Software Architect

---

Special thanks to the Kimputing team: Raivis Sarkovskis, Kim Filiatrault, and Duncan Guignet for their contributions to this article.

---

Watch or read our other posts at Kimputing Blogs. You’ll find everything from Automated testing to CenterTest, Guidewire knowledge to general interest. We’re trying to help share our knowledge from decades of experience.