Carolin Brandt · Marco Castelluccio · Christian Holler · Jason Kratzer · Andy Zaidman · Alberto Bacchelli

This paper is a collaboration with Mozilla, where we try out whether we can use fuzzers to automatically generate partial input-tests that developers then complete into full functional tests with assertions. Inspired by the feedback from developers on the initial Bugzilla reports we submitted, we then dive deeper into which kind of coverage gaps are actually relevant for developers to close.

Abstract

Can fuzzers generate partial tests that developers find useful enough to complete into functional tests (e.g., by adding assertions)? To address this question, we develop a prototype within the Mozilla ecosystem and open 13 bug reports proposing partial generated tests for currently uncovered code. We found that the majority of the reactions focus on whether the targeted coverage gap is actually worth testing. To investigate further which coverage gaps developers find relevant to close, we design an automated filter to exclude irrelevant coverage gaps before generating tests. From conversations with 13 developers about whether the remaining coverage gaps are worth closing when a partially generated test is available, we learn that the filtering indeed removes clearly non- test-worthy gaps. The developers propose a variety of additional strategies to address the coverage gaps and how to make fuzz tests and reports more useful for developers.

Carolin Brandt · Ali Khatami · Mairieli Wessel · Andy Zaidman

In this paper we conduct a large-scale open source contribution study, submitting pull requests with amplified tests to Java projects on GitHub. We analyze the feedback from the maintainers, as well as the preparations we needed to do to prepare the amplified tests for the pull requests. Based on this we collect guidelines on what developers can expect to change in amplified tests before including them in their test suite. Part of this are very helpful changes that are based on the developer’s understanding of the code base and their software project, leading us to call for more support for developers to understanding amplified tests instead of focusing on further automation.

Abstract

Test amplification makes systematic changes to existing, manually written tests to provide tests complementary to an automated test suite. We consider developer-centric test amplification, where the developer explores, judges and edits the amplified tests before adding them to their maintained test suite. However, it is as yet unclear which kind of selection and editing steps developers take before including an amplified test into the test suite. In this paper we conduct an open source contribution study, amplifying tests of open source Java projects from GitHub. We report which deficiencies we observe in the amplified tests while manually filtering and editing them to open 39 pull requests with amplified tests. We present a detailed analysis of the maintainer’s feedback regarding proposed changes, requested information, and expressed judgment. Our observations provide a basis for practitioners to take an informed decision on whether to adopt developer-centric test amplification. As several of the edits we observe are based on the developer’s understanding of the amplified test, we conjecture that developer-centric test amplification should invest in supporting the developer to understand the amplified tests.

Keywords

Software Testing · Automatic Test Generation · Developer-Centric Test Amplification

Khalid El Haji · Carolin Brandt · Andy Zaidman

This paper is based on Khalid’s Master Thesis, where he investigated how usable tests generated by GitHub Copilot are. We observe that the usability is low, meaning that the developers need to make changes to most generated tests. Less changes are needed when there are already tests existing in the same class and the generated test closely mimics the structure of these existing tests.

Abstract

Writing unit tests is a crucial task in software development, but it is also recognized as a time-consuming and tedious task. As such, numerous test generation approaches have been proposed and investigated. However, most of these test generation tools produce tests that are typically difficult to understand. Recently, Large Language Models (LLMs) have shown promising results in generating source code and supporting software engineering tasks. As such, we investigate the usability of tests generated by GitHub Copilot, a proprietary closed-source code generation tool that uses an LLM. We evaluate GitHub Copilot’s test generation abilities both within and without an existing test suite, and we study the impact of different code commenting strategies on test generations. Our investigation evaluates the usability of 290 tests generated by GitHub Copilot for 53 sampled tests from open source projects. Our findings highlight that within an existing test suite, approximately 45.28% of the tests generated by Copilot are passing tests; 54.72% of generated tests are failing, broken, or empty tests. Furthermore, if we generate tests using Copilot without an existing test suite in place, we observe that 92.45% of the tests are failing, broken, or empty tests. Additionally, we study how test method comments influence the usability of test generations.

Carolin Brandt · Danyao Wang · Andy Zaidman

In this paper we compare a user-driven guided approach of test amplification to the original open test amplification approach. While the user guidance makes it easier for the developers to understand the amplified tests, and it fits better into their typical workflow of writing tests in conjunction with production code, there are also trade-offs. For example, that now we create the expectation of being able to generate tests for the selected branch, while it can be computationally expensive to find the right branch conditions and initialize the objects under test accordingly.

Abstract

Test amplification generates new tests by mutating existing, developer-written tests and keeping those tests that improve the coverage of the test suite. Current amplification tools focus on starting from a specific test and propose coverage improvements all over a software project, requiring considerable effort from the software engineer to understand and evaluate the different tests when deciding whether to include a test in the maintained test suite. In this paper, we propose a novel approach that lets the developer take charge and guide the test amplification process towards a specific branch they would like to test in a control flow graph visualization. We evaluate whether simple modifications to the automatic process that incorporate the guidance make the test amplification more effective at covering targeted branches. In a user study and semi-structured interviews we compare our user-guided test amplification approach to the state-of-the-art open test amplification approach. While our participants prefer the guided approach, we uncover several trade-offs that influence which approach is the better choice, largely depending on the use case of the developer.

Keywords

Software Testing · Test Amplification · Automated Test Code Modification · User-centric Design · Human-Automation Interaction

Carolin Brandt · Andy Zaidman

This paper explores which factors make developer-centric test amplification—where generated test cases are taken over into the maintained test suite—
a useful tool for developers.
Short answer: consider and design the interaction of developer & test amplification tool.
Details: have a look at the paper for those 😉!

Abstract

Automatically generating test cases for software has been an active research topic for many years. While current tools can generate powerful regression or crash-reproducing test cases, these are often kept separately from the maintained test suite. In this paper, we leverage the developer’s familiarity with test cases amplified from existing, manually written developer tests. Starting from issues reported by developers in previous studies, we investigate what aspects are important to design a developer-centric test amplification approach, that provides test cases that are taken over by developers into their test suite. We conduct 16 semi-structured interviews with software developers supported by our prototypical designs of a developer-centric test amplification approach and a corresponding test exploration tool. We extend the test amplification tool DSpot, generating test cases that are easier to understand. Our IntelliJ plugin TestCube empowers developers to explore amplified test cases from their familiar environment. From our interviews, we gather 52 observations that we summarize into 23 result categories and give two key recommendations on how future tool designers can make their tools better suited for developer-centric test amplification.

Keywords

Software Testing · Test Amplification · Test Exploration · Test Generation · Developer-Centric Design

Carolin Brandt · Andy Zaidman

In this paper we took a look at how we should build visualizations that let developers compare one test case to the rest of a test suite.

The paper was accepted at VISSOFT’22, the 10th IEEE Working Conference on Software Visualization.

Short video introducing our visualization: https://youtu.be/R5KDR_viFRA
Artifact with our implementation, replication documentation and the full codes elicited during the study: https://zenodo.org/record/7022610 We received the award for the best VISSOFT’22 artifact at ICSME’s ROSE festival.

Abstract

Developer testing, the practice of software engineers programmatically checking that their own components behave as they expect, has become the norm in today’s software projects. With the constantly growing size and complexity of software projects and with the rise of automated test generation tools, understanding a test case is becoming more and more important compared to writing test cases from scratch. This holds especially in the area of developer-centric test amplification, where a tool automatically generates new test cases to improve a developer-maintained test suite. To investigate how visualization can help developers understand and judge test cases, we present the TestImpactGraph, a visualization of the call tree and coverage impact of a JUnit test case proposed for amplification. It empowers the developer to drill down into the behavior of a test case, as well as providing them a clear view on how the proposed test case contributes to the coverage of the overall test suite. In a think-aloud study we investigate which information developers seek from the TestImpactGraph, how its features can support them in accessing this information, and observations regarding the coverage impact of test cases. We infer ten actionable recommendations on how developer tests can be visualized to help developers understand their behavior and impact.

Keywords

Software Testing · Test Amplification · Test Review · Test Visualization · Test Understanding

Casper Boone · Carolin Brandt · Andy Zaidman

This paper discusses a plugin that brings information from the CI to the developer’s IDE to help them fix failing tests faster. It is based on Casper Boone’s excellent Master Thesis, who I had the pleasure to supervise in 2020/21.

The paper was accepted at ICPC’22, the 30th IEEE/ACM International Conference on Program Comprehension (DOI).

Abstract

The most common reason for Continuous Integration (CI) builds to break is failing tests. When a build breaks, a developer often has to scroll through hundreds to thousands of log lines to find which test is failing and why. Finding the issue is a tedious process that relies on a developer’s experience and increases the cost of software testing. We investigate how presenting different kinds of contextual information about CI builds in the Integrated Development Environment (IDE) impacts the time developers take to fix a broken build. Our IntelliJ plugin TestAxis surfaces additional information such as a unique view of the code under test that was changed leading up to the build failure. We conduct a user experiment and show that TestAxis helps developers fix failing tests 13.4% to 48.6% faster. The participants found the features of TestAxis useful and would incorporate it in their development workflow to save time. With TestAxis we set an important step towards removing the need to manually inspect build logs and bringing CI build results to the IDE, ultimately saving developers time.

Keywords

Software Testing · Continuous Integration · Developer Assistance · IDE Plugin · User Experiment

Carolin Brandt · Annibale Panichella · Andy Zaidman · Moritz Beller

A dataset of continuous integration log files from a variety of 80 GitHub repositories and the chunk of the file that states why the build failed.

The paper was accepted at the dataset track of MSR’20, the 17th IEEE/ACM International Conference on Mining Software Repositories.

Abstract

Build logs are textual by-products that a software build process creates, often as part of its Continuous Integration (CI) pipeline. Build logs are a paramount source of information for developers when debugging into and understanding a build failure. Recently, attempts to partly automate this time-consuming, purely manual activity have come up, such as rule- or information-retrieval-based techniques.

We believe that having a common data set to compare different build log analysis techniques will advance the research area. It will ultimately increase our understanding of CI build failures. In this paper, we present LogChunks, a collection of 797 annotated Travis CI build logs from 80 GitHub repositories in 29 programming languages. For each build log, LogChunks contains a manually labeled log part (chunk) describing why the build failed. We externally validated the data set with the developers who caused the original build failure.

The width and depth of the LogChunks data set are intended to make it the default benchmark for automated build log analysis techniques.

Keywords

Continuous Integration · Build Log Analysis · Build Failure · Chunk Retrieval

Nienke Nijkamp · Carolin Brandt · Andy Zaidman

This paper proposes a novel naming approach for amplified test cases based on the methods where the tests provides additional coverage compared to the existing test suite.

The paper was accepted at the NIER track of SCAM’21, the IEEE International Working Conference on Source Code Analysis and Manipulation.

Abstract

Test amplification generates new test cases that improve the coverage of an existing test suite. To convince developers to integrate these new test cases into their test suite, it is crucial to convey the behavior and the improvement in coverage that the amplified test case provides. In this paper, we present NATIC, an approach to generate names for amplified test cases based on the methods they additionally cover, compared to the existing test suite. In a survey among 16 participants with a background in Computer Science, we show that the test names generated by NATIC are valued similarly to names written by experts. According to the participants, the names generated by NATIC outperform expert-written names with respect to informing about coverage improvement, but lack in conveying a test’s behavior. Finally, we discuss how a restriction to two mentioned methods per name would improve the understandability of the test names generated by NATIC.

Wessel Oosterbroek · Carolin Brandt · Andy Zaidman

This paper proposes minimization approach for amplified test cases based on first drastically then step by step more conservatively removing statements while checking that the improvement in mutation score remains the same.

The paper was accepted at the NIER track of SCAM’21, the IEEE International Working Conference on Source Code Analysis and Manipulation.

Abstract

Test amplification generates new tests by modifying existing, manually written tests. Up until now, this process preserves statements that were relevant for the original test case but are no longer needed for the behavior of the new test case. These unnecessary statements impact the readability of the tests in question. As a part of the effort to make amplified test cases more readable, we investigate dynamic slicing, taint analysis and static analysis as approaches to remove redundant statements. We design and evaluate a static analysis approach that we implemented as part of the test amplification tool DSpot. Our empirical evaluation on 274 amplified test cases shows that the implemented approach works well: while being rudimentary, it is able to remove a significant portion of the redundant statements in the amplified test cases. While the removal of the statements themselves is fast, verifying that the tests still work as intended through mutation testing is still resource-intensive.