Adding a bug

In this section, we provide instructions for adding a new historical bug to the BugZoo platform.

Note

Before adding a bug, you must first create a source that will provide it. Note that a source may provide multiple bugs and tools. Typically, a source is used to provide a dataset of bugs.

Creating a manifest file

To begin with, we need to create a YAML-formatted manifest file to provide the bug. Alternatively, we can extend an existing manifest file with the necessary information for the bug. For the rest of this example, we will create a new manifest file, named gcd.bugzoo.yml. The top of the file should specify the version of the BugZoo Manifest File Format that was used to write the file. For this example, we will be using version 1.0:

version: '1.0'

Writing a Dockerfile

Next, we need to write a Dockerfile that will be used to provide a Docker container image for the buggy program. The Docker container for the buggy program is responsible for providing a minimal execution environment for building the program and running its tests. For general tips on writing Dockerfiles, see https://docs.docker.com/get-started/part2/#dockerfile and https://docs.docker.com/develop/develop-images/dockerfile_best-practices/.

For the remainder of this guide, we will add the gcd bug from the GenProg TSE 2012 dataset to BugZoo. To begin with, we place all of the necessary files for the bug scenario somewhere in our source directory (so that BugZoo can find them), as shown below.

$ ls -R gcd
gcd:
Dockerfile  gcd.bugzoo.yml  gcd.c  test  test.sh

gcd/test:
output.0.55  output.1071.1029  output.112.135  output.16.4  output.16777216.512  output.310.55  output.315.831  output.513332.91583315  output.555.666  output.678.987  output.8767.65

The contents of the directory are as follows:

  • gcd.bugzoo.yml: a manifest file that registers the bug (and its associated Docker container) with BugZoo.

  • Dockerfile: used to provide a Docker container for interacting with the program under test.

  • gcd.c: source code for the program under test,

  • test.sh: a simple test harness

  • test: supplementary files used by test.sh.

Below is a simplified example of a Dockerfile that is used to provide a Docker container for the gcd bug scenario.

FROM ubuntu:16.04

# install dependencies
# sudo and gcovr are required by BugZoo
RUN apt-get update \
 && apt-get install -y --no-install-recommends \
      gcc \
      sudo \
      gcovr \
      libc6-dev

# create a directory to hold the program under test (PUT)
WORKDIR /experiment

# add the source code for the PUT
ADD gcd.c /experiment/source/

# add the test harness and supplementary files
ADD test.sh /experiment
ADD test /experiment/test

Note

In practice, it’s common to add a docker user with sudo privileges to the container to prevent the buggy program from misbehaving. A more detailed example of a Dockerfile for the gcd scenario can be found at https://github.com/ChrisTimperley/genprog-tse-2012-bugs/blob/master/gcd/Dockerfile.

Registering a blueprint

Now that we have written a Dockerfile for our bug scenario, we need to register a blueprint for constructing the Docker image for our scenario. A blueprint is used to provide instructions for building a named Docker image. For this example, we will name the Docker image gcd.

To add a blueprint for the bug scenario, we need to create a blueprints section in the manifest file (if one doesn’t already exist), before adding an entry for the gcd blueprint to it, as shown below. For more details on the properties that can be used to describe a (more advanced) blueprint please refer to the manifest file format documentation.

version: '1.0'

blueprints:
  - tag: gcd
    file: Dockerfile

The file property of the blueprint gives BugZoo the path of the file that should be used to build the Docker container image, relative to the location of the manifest file.

Note

When writing your own blueprints, it’s good practice to use a fully qualified tag for your Docker image. For example, squareslab/tse2012:gcd is tag used by the manifest file in the TSE 2012 dataset repository.

Registering a bug

Finally, we need to add the bug scenario to the manifest file. To do so, first we must create a bugs section in the file. Next, we add an entry to the bugs section for the gcd bug scenario, shown below. Details about each of the properties used to describe bugs can be found in the Manifest File Format description.

version: '1.0'

blueprints:
  - tag: gcd
    file: Dockerfile

bugs:
  - name: tse2012:gcd
    dataset: tse2012
    program: gcd
    image: gcd
    languages:
    - c
    source-location: /experiment/source
    compiler:
      time-limit: 20
      type: simple
      context: /experiment/source
      command: |
        gcc gcd.c -o gcd
      command_with_instrumentation: |
        gcc gcd.c -o gcd --coverage
    coverage:
      files-to-instrument:
        - gcd.c
    test-harness:
      context: /experiment
      command: ./test.sh __ID__
      time-limit: 5
      tests:
        - name: p1
        - name: p2
        - name: p3
        - name: p4
        - name: p5
        - name: p6
        - name: p7
        - name: p8
        - name: p9
        - name: p10
        - name: n1