George Vegelien · Carolin Brandt · Bas Graaf · Arie van Deursen

This paper is a collaboration with Exact that resulted from George’s master thesis. We looked into the test flakiness in one of Exact’s systems, highlighting how important meaningful measurement and communication both with the engineers and with management is to address test flakiness.

Abstract

Flaky tests—tests that pass or fail unpredictably even without code changes—undermine the speed and trustworthiness of modern, database-heavy software systems. This study investigates the underlying causes of flakiness at Exact, a large-scale industrial system in which shared database states and resource contention introduced frequent test instability. By repeatedly rerunning the same commit, we pinpointed recurring problem areas and implemented three targeted solutions: (1) minimizing redundant background database tasks, and two strategies for cleaning up “dirty” tests: (2) explicitly disposing of test data and (3) disabling polluting tests with a database sanity check. Together, these interventions raised Exact’s chance of a passing pipeline run from 27% to 95% and boosted their monthly release rate from 60% to 96%. Our findings confirm that rich, systematic measurement and well-prioritized fixes can significantly reduce flakiness in industrial-scale software.

Carolin Brandt · Aurora Ramírez

This paper introduces a novel way to look at code coverage in software testing: can we predict it based on the code? Our idea is to learn from existing code coverage in open source projects what characterizes code that developers find it worth to cover with tests.

Abstract

To measure and improve the strength of test suites, software projects and their developers commonly use code coverage and aim for a threshold of around 80%. But what is the 80% of the source code that should be covered? To prepare for the development of new, more refined code coverage criteria, we introduce a novel predictive problem in software testing: whether a code line is, or should be, covered by the test suite. In this short paper, we propose the collection of coverage information, source code metrics, and abstract syntax tree data and explore whether they are relevant to predict whether a code line is exercised by the test suite or not. We present a preliminary experiment using four machine learning (ML) algorithms and an open source Java project. We observe that ML classifiers can achieve high accuracy (up to 90%) on this novel predictive problem. We also apply an explainable method to better understand the characteristics of code lines that make them more “appealing” to be covered. Our work opens a research line worth to investigate further, where the focus of the prediction is the code to be tested. Our innovative approach contrasts with most predictive problems in software testing, which aim to predict the test case failure probability.

Ali Khatami · Carolin Brandt · Andy Zaidman

Software quality is crucial, but previous research showed that developers often lack awareness about QA practices in their projects. We aimed to address this gap by creating a dashboard that provides insights into various QA activities.

In this work:

• We designed RepoInsights, a dashboard that gives an overview of QA practices like testing, code reviews, automated workflows, and project guidelines.
• We conducted interviews with 14 software engineers from open-source, industry, and academia to evaluate the tool.
• We asked participants to assess real projects using RepoInsights and provide feedback on its usefulness.

Key Findings

• Most participants found value in having a centralized overview of QA metrics across their projects.
• The dashboard helped engineers reflect on QA practices by providing meaningful metrics and details about project activities.
• Participants expressed interest in using such a tool in the future, especially for periodic project health checks.
• Engineers wanted more granular, contextualized information tailored to each project’s specific needs and practices.
• There’s a need for actionable insights derived from the data, not just raw metrics.

Recommendations for Improvement

Based on participant feedback, future QA analytics tools should:

1. Provide highly detailed, project-specific information
2. Offer meaningful comparisons between similar projects
3. Include more context about each project’s unique situation
4. Generate actionable insights and improvement suggestions
5. Allow customization of displayed information
6. Integrate seamlessly into existing developer workflows

Why It Matters

This research lays the groundwork for better QA analytics tools. By improving developers’ awareness and understanding of QA practices, we can potentially enhance software quality across the industry.

Carolin Brandt · Ali Khatami · Mairieli Wessel · Andy Zaidman

In this paper we conduct a large-scale open source contribution study, submitting pull requests with amplified tests to Java projects on GitHub. We analyze the feedback from the maintainers, as well as the preparations we needed to do to prepare the amplified tests for the pull requests. Based on this we collect guidelines on what developers can expect to change in amplified tests before including them in their test suite. Part of this are very helpful changes that are based on the developer’s understanding of the code base and their software project, leading us to call for more support for developers to understanding amplified tests instead of focusing on further automation.

Abstract

Test amplification makes systematic changes to existing, manually written tests to provide tests complementary to an automated test suite. We consider developer-centric test amplification, where the developer explores, judges and edits the amplified tests before adding them to their maintained test suite. However, it is as yet unclear which kind of selection and editing steps developers take before including an amplified test into the test suite. In this paper we conduct an open source contribution study, amplifying tests of open source Java projects from GitHub. We report which deficiencies we observe in the amplified tests while manually filtering and editing them to open 39 pull requests with amplified tests. We present a detailed analysis of the maintainer’s feedback regarding proposed changes, requested information, and expressed judgment. Our observations provide a basis for practitioners to take an informed decision on whether to adopt developer-centric test amplification. As several of the edits we observe are based on the developer’s understanding of the amplified test, we conjecture that developer-centric test amplification should invest in supporting the developer to understand the amplified tests.

Keywords

Software Testing · Automatic Test Generation · Developer-Centric Test Amplification

Carolin Brandt · Marco Castelluccio · Christian Holler · Jason Kratzer · Andy Zaidman · Alberto Bacchelli

This paper is a collaboration with Mozilla, where we try out whether we can use fuzzers to automatically generate partial input-tests that developers then complete into full functional tests with assertions. Inspired by the feedback from developers on the initial Bugzilla reports we submitted, we then dive deeper into which kind of coverage gaps are actually relevant for developers to close.

Abstract

Can fuzzers generate partial tests that developers find useful enough to complete into functional tests (e.g., by adding assertions)? To address this question, we develop a prototype within the Mozilla ecosystem and open 13 bug reports proposing partial generated tests for currently uncovered code. We found that the majority of the reactions focus on whether the targeted coverage gap is actually worth testing. To investigate further which coverage gaps developers find relevant to close, we design an automated filter to exclude irrelevant coverage gaps before generating tests. From conversations with 13 developers about whether the remaining coverage gaps are worth closing when a partially generated test is available, we learn that the filtering indeed removes clearly non- test-worthy gaps. The developers propose a variety of additional strategies to address the coverage gaps and how to make fuzz tests and reports more useful for developers.

Khalid El Haji · Carolin Brandt · Andy Zaidman

This paper is based on Khalid’s Master Thesis, where he investigated how usable tests generated by GitHub Copilot are. We observe that the usability is low, meaning that the developers need to make changes to most generated tests. Less changes are needed when there are already tests existing in the same class and the generated test closely mimics the structure of these existing tests.

Abstract

Writing unit tests is a crucial task in software development, but it is also recognized as a time-consuming and tedious task. As such, numerous test generation approaches have been proposed and investigated. However, most of these test generation tools produce tests that are typically difficult to understand. Recently, Large Language Models (LLMs) have shown promising results in generating source code and supporting software engineering tasks. As such, we investigate the usability of tests generated by GitHub Copilot, a proprietary closed-source code generation tool that uses an LLM. We evaluate GitHub Copilot’s test generation abilities both within and without an existing test suite, and we study the impact of different code commenting strategies on test generations. Our investigation evaluates the usability of 290 tests generated by GitHub Copilot for 53 sampled tests from open source projects. Our findings highlight that within an existing test suite, approximately 45.28% of the tests generated by Copilot are passing tests; 54.72% of generated tests are failing, broken, or empty tests. Furthermore, if we generate tests using Copilot without an existing test suite in place, we observe that 92.45% of the tests are failing, broken, or empty tests. Additionally, we study how test method comments influence the usability of test generations.

Carolin Brandt · Danyao Wang · Andy Zaidman

In this paper we compare a user-driven guided approach of test amplification to the original open test amplification approach. While the user guidance makes it easier for the developers to understand the amplified tests, and it fits better into their typical workflow of writing tests in conjunction with production code, there are also trade-offs. For example, that now we create the expectation of being able to generate tests for the selected branch, while it can be computationally expensive to find the right branch conditions and initialize the objects under test accordingly.

Abstract

Test amplification generates new tests by mutating existing, developer-written tests and keeping those tests that improve the coverage of the test suite. Current amplification tools focus on starting from a specific test and propose coverage improvements all over a software project, requiring considerable effort from the software engineer to understand and evaluate the different tests when deciding whether to include a test in the maintained test suite. In this paper, we propose a novel approach that lets the developer take charge and guide the test amplification process towards a specific branch they would like to test in a control flow graph visualization. We evaluate whether simple modifications to the automatic process that incorporate the guidance make the test amplification more effective at covering targeted branches. In a user study and semi-structured interviews we compare our user-guided test amplification approach to the state-of-the-art open test amplification approach. While our participants prefer the guided approach, we uncover several trade-offs that influence which approach is the better choice, largely depending on the use case of the developer.

Keywords

Software Testing · Test Amplification · Automated Test Code Modification · User-centric Design · Human-Automation Interaction

Carolin Brandt · Andy Zaidman

In this paper we took a look at how we should build visualizations that let developers compare one test case to the rest of a test suite.

The paper was accepted at VISSOFT’22, the 10th IEEE Working Conference on Software Visualization.

Short video introducing our visualization: https://youtu.be/R5KDR_viFRA
Artifact with our implementation, replication documentation and the full codes elicited during the study: https://zenodo.org/record/7022610 We received the award for the best VISSOFT’22 artifact at ICSME’s ROSE festival.

Abstract

Developer testing, the practice of software engineers programmatically checking that their own components behave as they expect, has become the norm in today’s software projects. With the constantly growing size and complexity of software projects and with the rise of automated test generation tools, understanding a test case is becoming more and more important compared to writing test cases from scratch. This holds especially in the area of developer-centric test amplification, where a tool automatically generates new test cases to improve a developer-maintained test suite. To investigate how visualization can help developers understand and judge test cases, we present the TestImpactGraph, a visualization of the call tree and coverage impact of a JUnit test case proposed for amplification. It empowers the developer to drill down into the behavior of a test case, as well as providing them a clear view on how the proposed test case contributes to the coverage of the overall test suite. In a think-aloud study we investigate which information developers seek from the TestImpactGraph, how its features can support them in accessing this information, and observations regarding the coverage impact of test cases. We infer ten actionable recommendations on how developer tests can be visualized to help developers understand their behavior and impact.

Keywords

Software Testing · Test Amplification · Test Review · Test Visualization · Test Understanding

Carolin Brandt · Andy Zaidman

This paper explores which factors make developer-centric test amplification—where generated test cases are taken over into the maintained test suite—
a useful tool for developers.
Short answer: consider and design the interaction of developer & test amplification tool.
Details: have a look at the paper for those 😉!

Abstract

Automatically generating test cases for software has been an active research topic for many years. While current tools can generate powerful regression or crash-reproducing test cases, these are often kept separately from the maintained test suite. In this paper, we leverage the developer’s familiarity with test cases amplified from existing, manually written developer tests. Starting from issues reported by developers in previous studies, we investigate what aspects are important to design a developer-centric test amplification approach, that provides test cases that are taken over by developers into their test suite. We conduct 16 semi-structured interviews with software developers supported by our prototypical designs of a developer-centric test amplification approach and a corresponding test exploration tool. We extend the test amplification tool DSpot, generating test cases that are easier to understand. Our IntelliJ plugin TestCube empowers developers to explore amplified test cases from their familiar environment. From our interviews, we gather 52 observations that we summarize into 23 result categories and give two key recommendations on how future tool designers can make their tools better suited for developer-centric test amplification.

Keywords

Software Testing · Test Amplification · Test Exploration · Test Generation · Developer-Centric Design

Casper Boone · Carolin Brandt · Andy Zaidman

This paper discusses a plugin that brings information from the CI to the developer’s IDE to help them fix failing tests faster. It is based on Casper Boone’s excellent Master Thesis, who I had the pleasure to supervise in 2020/21.

The paper was accepted at ICPC’22, the 30th IEEE/ACM International Conference on Program Comprehension (DOI).

Abstract

The most common reason for Continuous Integration (CI) builds to break is failing tests. When a build breaks, a developer often has to scroll through hundreds to thousands of log lines to find which test is failing and why. Finding the issue is a tedious process that relies on a developer’s experience and increases the cost of software testing. We investigate how presenting different kinds of contextual information about CI builds in the Integrated Development Environment (IDE) impacts the time developers take to fix a broken build. Our IntelliJ plugin TestAxis surfaces additional information such as a unique view of the code under test that was changed leading up to the build failure. We conduct a user experiment and show that TestAxis helps developers fix failing tests 13.4% to 48.6% faster. The participants found the features of TestAxis useful and would incorporate it in their development workflow to save time. With TestAxis we set an important step towards removing the need to manually inspect build logs and bringing CI build results to the IDE, ultimately saving developers time.

Keywords

Software Testing · Continuous Integration · Developer Assistance · IDE Plugin · User Experiment

Wessel Oosterbroek · Carolin Brandt · Andy Zaidman

This paper proposes minimization approach for amplified test cases based on first drastically then step by step more conservatively removing statements while checking that the improvement in mutation score remains the same.

The paper was accepted at the NIER track of SCAM’21, the IEEE International Working Conference on Source Code Analysis and Manipulation.

Abstract

Test amplification generates new tests by modifying existing, manually written tests. Up until now, this process preserves statements that were relevant for the original test case but are no longer needed for the behavior of the new test case. These unnecessary statements impact the readability of the tests in question. As a part of the effort to make amplified test cases more readable, we investigate dynamic slicing, taint analysis and static analysis as approaches to remove redundant statements. We design and evaluate a static analysis approach that we implemented as part of the test amplification tool DSpot. Our empirical evaluation on 274 amplified test cases shows that the implemented approach works well: while being rudimentary, it is able to remove a significant portion of the redundant statements in the amplified test cases. While the removal of the statements themselves is fast, verifying that the tests still work as intended through mutation testing is still resource-intensive.

Nienke Nijkamp · Carolin Brandt · Andy Zaidman

This paper proposes a novel naming approach for amplified test cases based on the methods where the tests provides additional coverage compared to the existing test suite.

The paper was accepted at the NIER track of SCAM’21, the IEEE International Working Conference on Source Code Analysis and Manipulation.

Abstract

Test amplification generates new test cases that improve the coverage of an existing test suite. To convince developers to integrate these new test cases into their test suite, it is crucial to convey the behavior and the improvement in coverage that the amplified test case provides. In this paper, we present NATIC, an approach to generate names for amplified test cases based on the methods they additionally cover, compared to the existing test suite. In a survey among 16 participants with a background in Computer Science, we show that the test names generated by NATIC are valued similarly to names written by experts. According to the participants, the names generated by NATIC outperform expert-written names with respect to informing about coverage improvement, but lack in conveying a test’s behavior. Finally, we discuss how a restriction to two mentioned methods per name would improve the understandability of the test names generated by NATIC.

Carolin Brandt · Annibale Panichella · Andy Zaidman · Moritz Beller

A dataset of continuous integration log files from a variety of 80 GitHub repositories and the chunk of the file that states why the build failed.

The paper was accepted at the dataset track of MSR’20, the 17th IEEE/ACM International Conference on Mining Software Repositories.

Abstract

Build logs are textual by-products that a software build process creates, often as part of its Continuous Integration (CI) pipeline. Build logs are a paramount source of information for developers when debugging into and understanding a build failure. Recently, attempts to partly automate this time-consuming, purely manual activity have come up, such as rule- or information-retrieval-based techniques.

We believe that having a common data set to compare different build log analysis techniques will advance the research area. It will ultimately increase our understanding of CI build failures. In this paper, we present LogChunks, a collection of 797 annotated Travis CI build logs from 80 GitHub repositories in 29 programming languages. For each build log, LogChunks contains a manually labeled log part (chunk) describing why the build failed. We externally validated the data set with the developers who caused the original build failure.

The width and depth of the LogChunks data set are intended to make it the default benchmark for automated build log analysis techniques.

Keywords

Continuous Integration · Build Log Analysis · Build Failure · Chunk Retrieval