TestIt

Just TestIt!

TestIt is a python package designed to automate full-system integration testing using a Software-Based Self-Test (SBST) approach. While formal verification methods are highly effective for targeting individual components, their complexity grows exponentially with the size of the System-Under-Test (SUT), making them impractical for testing large scale systems, like MCUs. Furthermore, they do not test interactions between the hardware platform and the software stack, which is essential for validating real-world functionality. Last but not least, they are limited to simulated environments, with all their limitations.

TestIt tackles these limitations and aims at bridging the gap between formal verification and real-world applications.

In short, what can TestIt do?

How can TestIt do all of this?

With just three requirements:

How can you use TestIt?

Again, it’s super simple! Just follow this documentation and you’ll find plenty of useful tips to integrate TestIt into your project. If you have any questions or suggestions, don’t hesitate to reach out to Tommaso Terzano at tommaso.terzano@gmail.com. He’s always happy to help clear up any doubts you might have.

⚠️ Take every “must do” with the utmost seriousness, as not following those advices will break TestIt. In Italy we say “Uomo avvisato mezzo salvato“…

Installation

Simply run the following command in your bash terminal:

pip install testitpy

That’s it!

Common Q&A

Structure of a TestIt environment

Figure 1: Structure of a TestIt environment

In order to run TestIt and take full advantage of its features, it’s important to get to know how it works. This section answers many of the questions you might have.

Functional Description

This section covers every aspect and feature of TestIt. If the previous section didn’t answer all your doubts, this one will. Let’s get technical!

Requirements

This section covers all the requirements that TestIt needs to run a successful integration test.

Configure the testing environment: config.test

This file is at heart a JSON database, which contains all the information that TestIt needs to operate, about both your workflow and the tests you want to run. Specifically, there are three fields that you need to fill out:

Define the golden functions: testit_golden.py

This file is quite simply a collection of all the functions that your test camapign will need to generate reference values. The functions declared in file will be dynamically imported by TestIt at execution time, just by looking for the name included in the goldenResultFunction field in config.test. You can include any package you need in this file, so be free to experiment with it and taylor this Python module to your needs.

The only critical aspect you need to take into account is the nature of the arguments and return value of the function, which must be:

This section will guide you through a couple of example functions, a simple one and a more complex one, that you can use as a reference to build your own function.

The first one is the workhorse of machine learning, the matrix multiplication. When calling the command testit setup, TestIt will include this very function in the template testit_golden.py that it will generate. Let’s analyze this function!

def matrix_multiply(inputs: list, parameters: list) -> list:
    """
    Multiplies two NumPy matrices and returns the result.

    :param inputs: Randomly generated datasets (List of Numpy Arrays)
    :param parameters: Parameters (Dict)
    :return: Resulting Golden Datasets (List of Numpy Arrays)
    """
    matrix_A = inputs[0]
    matrix_B = inputs[1]

    # Check if multiplication is valid
    if matrix_A.shape[1] != matrix_B.shape[0]:  
        raise ValueError("Matrix dimensions do not match for multiplication!")

    result = np.matmul(matrix_A, matrix_A)

    return [result]

This simple function extracts two NumPy matrices from the list that TestIt generates, checks if they have compatible shapes, and multiplies them. If the dimensions don’t match, it raises an error to keep you informed. Notice how it returns the result as a single-element list, this is exactly what TestIt expects from a golden function.

The following function is a more complex example of what it’s possible to achieve with TestIt. It performs the im2col reshaping transformation, which enables to compute convolutions as a single matrix multipication. Here it is:

def im2col_function(inputs, parameters):
    inputs = np.array(inputs)

    # Extract parameters
    batch_size = next(item['value'] for item in parameters if item['name'] == 'BATCH') 
    channels = next(item['value'] for item in parameters if item['name'] == 'CH')
    image_height = next(item['value'] for item in parameters if item['name'] == 'IH')
    image_width = next(item['value'] for item in parameters if item['name'] == 'IW')
    top_pad = next(item['value'] for item in parameters if item['name'] == 'TOP_PAD')
    bottom_pad = next(item['value'] for item in parameters if item['name'] == 'BOTTOM_PAD')
    left_pad = next(item['value'] for item in parameters if item['name'] == 'LEFT_PAD')
    right_pad = next(item['value'] for item in parameters if item['name'] == 'RIGHT_PAD')
    stride_d1 = next(item['value'] for item in parameters if item['name'] == 'STRIDE_D1')
    stride_d2 = next(item['value'] for item in parameters if item['name'] == 'STRIDE_D2')
    (filter_height, filter_width) = (next(item['value'] for item in parameters if item['name'] == 'FH'), next(item['value'] for item in parameters if item['name'] == 'FW'))
    kernel_size = (filter_height, filter_width)

    # Convert the input array into a PyTorch tensor with the correct shape
    input_tensor = torch.tensor(inputs).view(batch_size, channels, image_height, image_width)

    dilation = 1
    # Ensure kernel_size, stride, padding, and dilation are tuples
    if isinstance(kernel_size, int):
        kernel_size = (kernel_size, kernel_size)
    if isinstance(dilation, int):
        dilation = (dilation, dilation)

    # Adjust padding format for F.pad (expects pad_left, pad_right, pad_top, pad_bottom)
    padding_format = (left_pad, right_pad, top_pad, bottom_pad)

    # Apply zero padding
    padded_input = F.pad(input_tensor, padding_format, "constant", 0)

    # Unfold the padded input tensor
    unfolded = padded_input.unfold(2, kernel_size[0], stride_d2).unfold(3, kernel_size[1], stride_d1)
    unfolded = unfolded.permute(0, 2, 3, 1, 4, 5)

    # Reshape to get the 2D tensor where each row is a flattened receptive field
    channel_dim = padded_input.size(1)
    unfolded_tensor = unfolded.contiguous().view(-1, channel_dim * kernel_size[0] * kernel_size[1]).t()

    # Convert the PyTorch tensor to a NumPy array and then to a list (simple array)
    unfolded_array = unfolded_tensor.numpy()
    unfolded_array = [unfolded_array.reshape(*unfolded_array.shape)]

    return unfolded_array

The interesting thing about this example is how it’s possible to use parameters generated by TestIt to perform complex computations. In this case, TestIt generates a multitude of parameters, that define the input tensor shape (BATCH, CH, IW and IH) and the filter shape (FW and FH), as well as padding parameters and strides. Since parameters are passed as a list of dictonaries, it’s necessary to find the correct dictonary, i.e. the parameter wanted, and then extract its value. This is the best way we found, but you’re welcome to try and find better ways. For example, you could write a simple function, getParameter(), in the same testit_golden.py or even in a separate module.

Commands

TestIt provides three main commands to access its functionalities.

Setup the environment

testit setup

This is the first command you’ll want to run. It does exactly what it says—it sets up your TestIt environment. Running this command generates template versions of testit_golden.py and config.test, based on the targets and examples in this documentation.

You don’t have to place these files in the root of your RTL project, but keep in mind that all subsequent TestIt commands must be executed in the same directory where you first ran testit setup. That’s where TestIt expects to find its configuration files.

Run the testing campaign

testit run [flags]

This is the core command of TestIt, the one that actually executes the testing campaign. As mentioned earlier, you must run it in the directory where testit_golden.py and config.test are stored, which doesn’t necessarily have to be the root of your RTL project.

You can customize the test campaign with the following flags:

testit run --nobuild

If you have already built your simulation or FPGA model and want to skip the build step, this flag will prevent TestIt from rebuilding it.

testit run --sweep

This enables sweep mode, which overrides the iterations parameter and instead tests all possible parameter combinations for each individual test. This mode ensures a more comprehensive testing effort and is particularly useful for performance characterization. Be aware that this may generate a large number of iterations. The command line interface will always display the current iteration count and provide a dynamic estimate of the total test duration.

testit run --mammamia

Use this flag to let your test campaign cook like an Italian nonna would.

Generate a test report

testit report [flags]

This command generates a report of your most recent test campaign, displaying it in your terminal and saving it as report.rpt in the directory specified in config.test.

⚠️ Every time you run testit run, the previous test campaign data is erased! If you need to keep it, make sure to back it up somewhere else.

There are a couple of options to customize your report:

testit report --sort_key [key_name]

This sorts the test results based on any of the output tags defined in config.test. If you want to sort the results in descending order, add --descending to the command. Otherwise, sorting will be in ascending order.

This site is open source. Improve this page.