Running Tests

BugZoo’s Python API provides an interface for executing the individual test cases belonging to the test suite for a particular bug. The tests property of a Bug provides access to its TestSuite. The code snippet below provides an example of executing each of the tests contained within a TestSuite. The outcome of a test execution is provided as a TestOutcome object.

outcomes = {}  # type: Dict[TestCase, TestOutcome]
for test in snapshot.tests:
  outcomes[test] = client.containers.test(container, test)

Individual tests can also be fetched using TestSuite’s [] operator:

client.containers.test(container, snapshot.tests["test-foo"])

API Reference

class TestCase

Describes an individual test case for a particular snapshot.

name → str

The unique name of the test case.

oracle → TestCaseOracle

The oracle used by this test case.

time_limit → int

The maximum number of seconds that an execution of this test case is allowed before it is terminated and considered a failure.

command → str

The shell command that is used to execute this test case.

context → str

The absolute path of the directory in which the shell command for this test case should be executed.

expected_outcome → Optional[bool]

An optional flag that specifies whether or not an execution of this test case is expected to succeed for its corresponding bug. Used to indicate test cases that are expected to fail. Set to True if the test case is expected to pass, False if it is expected, and None if its expected outcome is unknown.

kill_after → int

Upon reaching the time limit, the process inside the container that is responsible for a particular test execution is sent to a SIGTERM signal to instruct it to terminate cleanly. kill_after specifes the number of seconds that may pass before a SIGKILL signal is sent to the process. Note that SIGTERM is used to ensure that any coverage information is flushed to disk before the command is terminated.

class TestCaseOracle

Used to determine whether the outcome of a test case execution should be considered as a success or failure.

check(response: bugzoo.cmd.ExecResponse) → bool

Determines whether the raw command output from a test execution satisfies this oracle.

class TestOutcome

Describes the outcome of a test execution.

passed → bool

A flag indicating whether or not the test execution succeeded (as determined by its oracle).

duration → float

The duration of the test execution, measured in seconds.

class TestSuite

Describes the test suite for a particular snapshot. Test suites are composed of a set of uniquely named individual test cases.

__iter__() → Iterator[bugzoo.core.test.TestCase]

Returns an iterator over the test cases contained within this test suite.

__getitem__(name: str) → bugzoo.core.test.TestCase

Attempts to fetch a test case from this test suite by its name.

Raises

KeyError – if no test case with the given name belongs to this test suite.