Obtaining Coverage

BugZoo’s Python API provides a number of uniform, language-independent interfaces for collecting line coverage information for individual test cases, entire test suites, and arbitrary executions.

Obtaining Coverage for Bugs

Complete test suite coverage information for a given bug can be obtained in the form of a TestSuiteCoverage object by using client.bugs.coverage(), as shown below. client.bugs.coverage() will attempt to load precomputed coverage data for the given bug from its on-disk cache; if no coverage data can be found, it will be recomputed from scratch and saved to disk for future requests.

bug = client.bugs['manybugs:python:69223-69224']
coverage = client.bugs.coverage(bug)

Coverage information for a particular test is represented as a TestCoverage object, and can be retrieved from a TestSuiteCoverage object by supplying the name of a test to its [] operator:

coverage_for_suite = client.bugs.coverage(bug)
coverage_for_test = coverage_for_suite["p1"]

TestSuiteCoverage also provides an iterator over the names of the tests included within its report:

for test_name in coverage_for_suite:
  coverage_for_test = coverage_for_suite[test_name]

Discuss test outcomes and file lines.

Obtaining Coverage for Containers

Like client.bugs.coverage(), client.containers.coverage() is used to compute coverage information for an entire test suite. (Note that, unlike client.bugs.coverage(), client.containers.coverage() does not cache coverage information to disk as containers are ephemeral.)

coverage_for_suite = client.containers.coverage(container)

Coverage information for individual test executions (and executions of arbitrary shell commands) can also be obtained by first instrumenting the the container, then executing the test (or shell command) for which coverage shoud be collected, before finally extracting a report of the source code lines that were executed:

client.containers.prepare_for_coverage(container)
outcome = client.containers.test(container, test)
lines = client.containers.read_coverage(container)
client.containers.cleanup_coverage(container)
coverage = TestCoverage(test.name, outcome, lines)

Computing Fault Spectra using Coverage

Fault spectra, represented as Spectra objects, are used to provide a concise summary of the number of passing and failing tests that touch (or do not touch) each line in the program. Each entry, or row, in a fault spectra is encoded as a LineSpectra object, and represents a line in the program, and is represented by four numbers: ep, np, ef, and nf.

API Reference

class FileLine[source]

Represents a one-indexed line within a specific file.

num → int

The one-indexed number of the line.

filename → str

The name of the file to which the line belongs.

class FileLineSet[source]

A set of file lines.

__contains__(elem: object) → bool[source]

Determines whether this set contains a given element.

__iter__() → Iterator[bugzoo.core.fileline.FileLine][source]

Returns an iterator over the lines contained in this set.

class TestCoverage[source]

Provides complete line coverage information for all files and across all tests within a given project.

class TestSuiteCoverage[source]

Holds coverage information for all tests belonging to a particular program version.

__getitem__(name: str) → bugzoo.core.coverage.TestCoverage[source]

Retrieves coverage information for a given test case.

Parameters

name – the name of the test case.

Raises

KeyError – if there is no coverage information for the given test case.

class Spectra[source]

Contains a summary of the number of passing and failing tests that cover each line in a given project.

__iter__() → Iterator[bugzoo.core.fileline.FileLine][source]

Returns an iterator over the source code lines that are represented in this spectra.

__getitem__(line: bugzoo.core.fileline.FileLine) → bugzoo.core.spectra.LineSpectra[source]

Retrieves the spectra information for a given line.

class LineSpectra[source]

Summarises the coverage information for a single line in the program in terms of the number of passing and failing tests that do and do not cover it, respectively.

ep → int
np → int
ef → int
nf → int