Generation

Contract Test

Contract Testing

This guide explains how to generate contract tests using the Skyramp CLI. Throughout this guide, we will demonstrate key capabilities using Skyramp’s Demo Shop API, a simple e-commerce API for product and order management. Learn more about the Demo Shop API.

If you haven’t already installed Skyramp, please refer to the instructions in the Installation Guide.

Overview

A contract test asserts that inter-application messages conform to a shared understanding that is documented in a contract. In this context, every contract is between a client (consumer) and a server (provider). The consumer side is interested in making sure that the client can handle an expected response correctly. The provider side is interested in making sure that the server response matches an expected response.

Today, Skyramp generates provider-side contract tests from request and response data supplied by the user. Generation of consumer-side contract tests is coming soon!

Generate a contract test for REST APIs

This section explains how you can use Skyramp to generate a contract test for a specific method of a REST API.

To reliably generate test cases, we require at least one of the following inputs:

An OpenAPI schema (JSON or YAML file)
Sample request and response data (JSON blob or JSON file)

In this guide, we'll use the OpenAPI schema approach. If you want more control over the generated body values, you can also do test generation from sample data.

Generate Contract Test for a Single Method

To create a contract test for a single method, specify the method you want to test against in the command. In this example, we are using the https://demoshop.skyramp.dev/api as the URL to our service. When testing your service, replace it with the URL to the endpoint you want to test.

You can find the used API specification here.

Contract Testing

If you haven’t already installed Skyramp, please refer to the instructions in the Installation Guide.

Overview

Today, Skyramp generates provider-side contract tests from request and response data supplied by the user. Generation of consumer-side contract tests is coming soon!

Generate a contract test for REST APIs

This section explains how you can use Skyramp to generate a contract test for a specific method of a REST API.

To reliably generate test cases, we require at least one of the following inputs:

An OpenAPI schema (JSON or YAML file)
Sample request and response data (JSON blob or JSON file)

In this guide, we'll use the OpenAPI schema approach. If you want more control over the generated body values, you can also do test generation from sample data.

Generate Contract Test for a Single Method

You can find the used API specification here.

Contract Testing

If you haven’t already installed Skyramp, please refer to the instructions in the Installation Guide.

Overview

Today, Skyramp generates provider-side contract tests from request and response data supplied by the user. Generation of consumer-side contract tests is coming soon!

Generate a contract test for REST APIs

This section explains how you can use Skyramp to generate a contract test for a specific method of a REST API.

To reliably generate test cases, we require at least one of the following inputs:

An OpenAPI schema (JSON or YAML file)
Sample request and response data (JSON blob or JSON file)

In this guide, we'll use the OpenAPI schema approach. If you want more control over the generated body values, you can also do test generation from sample data.

Generate Contract Test for a Single Method

You can find the used API specification here.

Python

Java

Typescript

skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema

This command generates one file:

products_POST_contract_test.py.

The content of the generated test is explained here.

Explanation of Command

https://demoshop.skyramp.dev/api/v1/products: Defines the URL to the endpoint we aim to test.
-X: Specifies the API method to test.
--language: Specifies the test output language. For contract testing, we currently support Python, Typescript, and Java.
--framework: Specify the test execution framework of choice.
--api-schema: Points to the OpenAPI schema used to generate the test. We also support sample data as an input for contract test generation.

Adjustments

These flags customize the test generation. Additional flags are explained here.

--asserts: An assert is a way to tell Skyramp what return values the test should check for. Skyramp evaluates the endpoint to determine what values are expected, and then the test checks to see whether they are returned. For more information about asserts, see the Examples page.
- Specifying all will return an assertion for every API response key available in the endpoint.
- Specifying none will not return any assertions.
- You can also specify the number of return values you wish. The default is 3, which will return the HTTP response code and the first three response body values that occur after that.
--response-status-code: Specify the expected status code. By default, Skyramp either asserts against the defined status code in the API specification or defaults to 20X.
--output: Specify the name of the generated test file.
--output-dir: Specify the directory to store the generated test file.

Execute the Contract Test

You can execute the generated tests without any additional adjustments to the code. However, based on the application you want to test, you can pass your authentication token to Skyramp Tests via an environment variable.

Set environment variable for authentication (if applicable)

Skyramp’s sample application doesn't require any authentication.

Ensure proper authentication for test execution. To test against an application that does require authentication, pass your token using an environment variable. By default, Skyramp expects a Bearer Token but we support additional authentication methods. If your API does not require any authentication, you can skip this step and just run the test.

export SKYRAMP_TEST_TOKEN=$your_auth_token

Run the test

Run the test using Pytest. If you don’t have Pytest, install it with pip by running the following command in your terminal:

# Prerequisites 
pip3 install pytest

# Execution of contract test for products/POST 
python3 -m

Review Test Results

We are using Pytest’s default test output in this guide, printing a line for each test that is being run and listing all failures at the end. You can adjust the output behavior following this documentation.

Successful test

Using the single POST method example (default asserts) of the /v1/products endpoint, the test passes.

Test failure

For this section, we intentionally changed the expected response body in the products_POST_contract_test.py test file.

Generated

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }'''

Manual change

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0,
            "extra_value": "invalid_field"
        }''

This results in a test failure given the expected response body does not match the actual response body anymore. The expected output looks as follows:

Next Steps

Congratulations, you have just generated your first contract test! To learn more about how to adjust the test file, please go to the Test File Anatomy page.

Related topics

Python

Java

Typescript

skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema

This command generates one file:

products_POST_contract_test.py.

The content of the generated test is explained here.

Explanation of Command

https://demoshop.skyramp.dev/api/v1/products: Defines the URL to the endpoint we aim to test.
-X: Specifies the API method to test.
--language: Specifies the test output language. For contract testing, we currently support Python, Typescript, and Java.
--framework: Specify the test execution framework of choice.
--api-schema: Points to the OpenAPI schema used to generate the test. We also support sample data as an input for contract test generation.

Adjustments

These flags customize the test generation. Additional flags are explained here.

--asserts: An assert is a way to tell Skyramp what return values the test should check for. Skyramp evaluates the endpoint to determine what values are expected, and then the test checks to see whether they are returned. For more information about asserts, see the Examples page.
- Specifying all will return an assertion for every API response key available in the endpoint.
- Specifying none will not return any assertions.
- You can also specify the number of return values you wish. The default is 3, which will return the HTTP response code and the first three response body values that occur after that.
--response-status-code: Specify the expected status code. By default, Skyramp either asserts against the defined status code in the API specification or defaults to 20X.
--output: Specify the name of the generated test file.
--output-dir: Specify the directory to store the generated test file.

Execute the Contract Test

Set environment variable for authentication (if applicable)

Skyramp’s sample application doesn't require any authentication.

export SKYRAMP_TEST_TOKEN=$your_auth_token

Run the test

Run the test using Pytest. If you don’t have Pytest, install it with pip by running the following command in your terminal:

# Prerequisites 
pip3 install pytest

# Execution of contract test for products/POST 
python3 -m

Review Test Results

Successful test

Using the single POST method example (default asserts) of the /v1/products endpoint, the test passes.

Test failure

For this section, we intentionally changed the expected response body in the products_POST_contract_test.py test file.

Generated

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }'''

Manual change

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0,
            "extra_value": "invalid_field"
        }''

This results in a test failure given the expected response body does not match the actual response body anymore. The expected output looks as follows:

Next Steps

Congratulations, you have just generated your first contract test! To learn more about how to adjust the test file, please go to the Test File Anatomy page.

Related topics

Python

Java

Typescript

skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema

This command generates one file:

products_POST_contract_test.py.

The content of the generated test is explained here.

Explanation of Command

https://demoshop.skyramp.dev/api/v1/products: Defines the URL to the endpoint we aim to test.
-X: Specifies the API method to test.
--language: Specifies the test output language. For contract testing, we currently support Python, Typescript, and Java.
--framework: Specify the test execution framework of choice.
--api-schema: Points to the OpenAPI schema used to generate the test. We also support sample data as an input for contract test generation.

Adjustments

These flags customize the test generation. Additional flags are explained here.

--asserts: An assert is a way to tell Skyramp what return values the test should check for. Skyramp evaluates the endpoint to determine what values are expected, and then the test checks to see whether they are returned. For more information about asserts, see the Examples page.
- Specifying all will return an assertion for every API response key available in the endpoint.
- Specifying none will not return any assertions.
- You can also specify the number of return values you wish. The default is 3, which will return the HTTP response code and the first three response body values that occur after that.
--response-status-code: Specify the expected status code. By default, Skyramp either asserts against the defined status code in the API specification or defaults to 20X.
--output: Specify the name of the generated test file.
--output-dir: Specify the directory to store the generated test file.

Execute the Contract Test

Set environment variable for authentication (if applicable)

Skyramp’s sample application doesn't require any authentication.

export SKYRAMP_TEST_TOKEN=$your_auth_token

Run the test

Run the test using Pytest. If you don’t have Pytest, install it with pip by running the following command in your terminal:

# Prerequisites 
pip3 install pytest

# Execution of contract test for products/POST 
python3 -m

Review Test Results

Successful test

Using the single POST method example (default asserts) of the /v1/products endpoint, the test passes.

Test failure

For this section, we intentionally changed the expected response body in the products_POST_contract_test.py test file.

Generated

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }'''

Manual change

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0,
            "extra_value": "invalid_field"
        }''

This results in a test failure given the expected response body does not match the actual response body anymore. The expected output looks as follows:

Next Steps

Congratulations, you have just generated your first contract test! To learn more about how to adjust the test file, please go to the Test File Anatomy page.