Generation

Getting Started

Contract Test

Contract Testing

This guide describes generating contract tests using the Skyramp CLI. Throughout this guide, we are using Skyramp’s Demo Shop API as an example REST API. You can find all relevant information on the Demo Shop here.

If you haven’t already installed Skyramp, follow the instructions here.

1. Overview

A contract test is a specific set of test communication that ensures a service is properly communicating with another service. A contract test asserts an expected condition, specifies valid responses, and evaluates based on whether the responses are returned for that condition.

2. Generate Contract Test for REST APIs

This section explains how you can use Skyramp to generate contract tests for REST APIs. You can generate a contract tests for:

a specific method of an endpoint or
all methods of an endpoint.

To reliably generate test cases, we require at least one of the following inputs:

OpenAPI schema (required if generating for all methods of an endpoint)
- Can be provided as JSON or YAML file
Sample request and response data
- Can be provided as JSON blob or JSON file

We are focusing on using an API schema as input, however, we also support test generation from sample data - this allows you to have more control over the generated body values.

Contract Testing

If you haven’t already installed Skyramp, follow the instructions here.

1. Overview

2. Generate Contract Test for REST APIs

This section explains how you can use Skyramp to generate contract tests for REST APIs. You can generate a contract tests for:

a specific method of an endpoint or
all methods of an endpoint.

To reliably generate test cases, we require at least one of the following inputs:

OpenAPI schema (required if generating for all methods of an endpoint)
- Can be provided as JSON or YAML file
Sample request and response data
- Can be provided as JSON blob or JSON file

We are focusing on using an API schema as input, however, we also support test generation from sample data - this allows you to have more control over the generated body values.

Contract Testing

If you haven’t already installed Skyramp, follow the instructions here.

1. Overview

2. Generate Contract Test for REST APIs

This section explains how you can use Skyramp to generate contract tests for REST APIs. You can generate a contract tests for:

a specific method of an endpoint or
all methods of an endpoint.

To reliably generate test cases, we require at least one of the following inputs:

OpenAPI schema (required if generating for all methods of an endpoint)
- Can be provided as JSON or YAML file
Sample request and response data
- Can be provided as JSON blob or JSON file

We are focusing on using an API schema as input, however, we also support test generation from sample data - this allows you to have more control over the generated body values.

Python

Java

Typescript

2.1 Single Method

To create a contract test for a single method, specify the method you want to test against in the command. In this example, we are using the https://demoshop.skyramp.dev/api/v1/products as the URL to our service. When testing your service, replace it with the URL to the endpoint you want to test.

You can find the used API specification here.

skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema openapi.json

This command generates a contract test file: products_POST_contract_test.py.
The content of this file is explained in Section 4 below.

Adjustments

Below are a few flags to customize the test generation. Additional flags are explained here.

--asserts: Specify Skyramp’s behavior to generate body value asserts [all, none] (default=3). Based on this flag Skyramp either generates asserts against no, three, or all response body values.
--response-status-code: Allows you to specify the expected status code. By default, Skyramp either takes the the provided status code in the API spec or for sample data assumes 20X.
--output: Allows you to specify the name of the generated test file.
--output-dir: Allows you to specify the directory to store the generated test file.

3. Execute Contract Test

You can execute the generated tests without any additional adjustments to the code.

3.1 Set env variable for authentication (if needed)

Skyramp’s sample application (link) doesn't require authentication, however, as most applications do, you can quickly pass your token via an environment variable. By default, Skyramp expects a Bearer Token but we support additional authentication methods.

export SKYRAMP_TEST_TOKEN=$your_auth_token

3.2 Run the test

Now you can run the tests using pytest.

# Prerequisites 
pip install pytest

# Execution of contract test for products/POST 
python3 -m pytest products_POST_contract_test.py

3.3 Results

We are using Pytest’s default test output in this guide, printing a line for each test that is being run and listing all failures at the end. You can adjust the output behavior following this documentation.

3.3.1 Successful Test

Using the single POST method example (default asserts) of the /v1/products endpoint, the test passes out of the box.

3.3.2 Test Failure

For this section, we intentionally changed the expected response body in the products_POST_contract_test.py test file.

Generated

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }''

Manual change

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0,
            "extra_value": "invalid_field"
        }''

This results in a test failure given the expected response body does not match the actual response body anymore. The expected output looks the following:

Congratulations, you have now successfully created a contract test, executed it against a REST API, and received results!

4. Skyramp Test File

This section explains the key elements of the generated test files. This will enable you to make adjustments when needed quickly.

At the top of each file, we show when the test was generated and what command was used
Below, we import all relevant libraries and specify the URL for all test requests
We define a function per method that is tested. It consists of:
- Invocation of Skyramp Client
- Definition of the authentication header
- Definition of the request body
- Definition of the expected response body
- Creation of the request
- Generated Assertions
  - Status code assertion
  - Schema check
  - Body value checks (first 3 values)

4.1 Single Method

# Generated by Skyramp v0.5.5 on 2025-02-27 16:28:07.011532 -0500 EST m=+0.192097209
# Command: skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
# 		--api-schema openapi.json \
# 		--framework pytest \
# 		--language python \
# 		--method POST \

# Import of required libraries
import skyramp
import os
import time
# URL for test requests
URL = "https://demoshop.skyramp.dev"

# contract test for /api/v1/products POST
def test_api_v_1_products_post():
    # Invocation of Skyramp Client
    client = skyramp.Client()
    # Definition of authentication header
    headers = {}
    if os.getenv("SKYRAMP_TEST_TOKEN") is not None:
        headers["Authorization"] = "Bearer " + os.getenv("SKYRAMP_TEST_TOKEN")

    # Request Body
    api_v1_products_POST_request_body = r'''{
            "category": "string",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0
        }'''
    
    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:28:07-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }'''
    
    # Execute Request
    api_v1_products_POST_response = client.send_request(
        url=URL,
        path="/api/v1/products",
        method="POST",
        body=api_v1_products_POST_request_body,
        headers=headers
    )
    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None

if __name__ == "__main__":
    test_api_v_1_products_post()

4.1.1 Changing asserts

By default, we generate value asserts only on the first 3 body values - this prevents the test code from becoming too verbose. Generated body value asserts validate if the field returned a value but does not do any matching. For more complex and/or additional asserts, you can adjust the code as follows:

Default Generated Asserts (lines 55-60)

    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None

Manually Adjusted Asserts

Following the same principle you can simply adjust and add more assertions to the generated code. Additional details on defining assertions can be found in Pytest's Documentation.

    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None
    # Manual addition of asserts 
    assert skyramp.get_response_value(api_v1_products_POST_response, "image_url") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "in_stock") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "name")=="string"
    assert skyramp.get_response_value(api_v1_products_POST_response, "price")==0
    assert skyramp.get_response_value(api_v1_products_POST_response, "product_id") is not None

Python

Java

Typescript

2.1 Single Method

You can find the used API specification here.

skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema openapi.json

This command generates a contract test file: products_POST_contract_test.py.
The content of this file is explained in Section 4 below.

Adjustments

Below are a few flags to customize the test generation. Additional flags are explained here.

--asserts: Specify Skyramp’s behavior to generate body value asserts [all, none] (default=3). Based on this flag Skyramp either generates asserts against no, three, or all response body values.
--response-status-code: Allows you to specify the expected status code. By default, Skyramp either takes the the provided status code in the API spec or for sample data assumes 20X.
--output: Allows you to specify the name of the generated test file.
--output-dir: Allows you to specify the directory to store the generated test file.

3. Execute Contract Test

You can execute the generated tests without any additional adjustments to the code.

3.1 Set env variable for authentication (if needed)

export SKYRAMP_TEST_TOKEN=$your_auth_token

3.2 Run the test

Now you can run the tests using pytest.

# Prerequisites 
pip install pytest

# Execution of contract test for products/POST 
python3 -m pytest products_POST_contract_test.py

3.3 Results

3.3.1 Successful Test

Using the single POST method example (default asserts) of the /v1/products endpoint, the test passes out of the box.

3.3.2 Test Failure

For this section, we intentionally changed the expected response body in the products_POST_contract_test.py test file.

Generated

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }''

Manual change

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0,
            "extra_value": "invalid_field"
        }''

This results in a test failure given the expected response body does not match the actual response body anymore. The expected output looks the following:

Congratulations, you have now successfully created a contract test, executed it against a REST API, and received results!

4. Skyramp Test File

This section explains the key elements of the generated test files. This will enable you to make adjustments when needed quickly.

At the top of each file, we show when the test was generated and what command was used
Below, we import all relevant libraries and specify the URL for all test requests
We define a function per method that is tested. It consists of:
- Invocation of Skyramp Client
- Definition of the authentication header
- Definition of the request body
- Definition of the expected response body
- Creation of the request
- Generated Assertions
  - Status code assertion
  - Schema check
  - Body value checks (first 3 values)

4.1 Single Method

# Generated by Skyramp v0.5.5 on 2025-02-27 16:28:07.011532 -0500 EST m=+0.192097209
# Command: skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
# 		--api-schema openapi.json \
# 		--framework pytest \
# 		--language python \
# 		--method POST \

# Import of required libraries
import skyramp
import os
import time
# URL for test requests
URL = "https://demoshop.skyramp.dev"

# contract test for /api/v1/products POST
def test_api_v_1_products_post():
    # Invocation of Skyramp Client
    client = skyramp.Client()
    # Definition of authentication header
    headers = {}
    if os.getenv("SKYRAMP_TEST_TOKEN") is not None:
        headers["Authorization"] = "Bearer " + os.getenv("SKYRAMP_TEST_TOKEN")

    # Request Body
    api_v1_products_POST_request_body = r'''{
            "category": "string",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0
        }'''
    
    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:28:07-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }'''
    
    # Execute Request
    api_v1_products_POST_response = client.send_request(
        url=URL,
        path="/api/v1/products",
        method="POST",
        body=api_v1_products_POST_request_body,
        headers=headers
    )
    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None

if __name__ == "__main__":
    test_api_v_1_products_post()

4.1.1 Changing asserts

Default Generated Asserts (lines 55-60)

    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None

Manually Adjusted Asserts

Following the same principle you can simply adjust and add more assertions to the generated code. Additional details on defining assertions can be found in Pytest's Documentation.

    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None
    # Manual addition of asserts 
    assert skyramp.get_response_value(api_v1_products_POST_response, "image_url") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "in_stock") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "name")=="string"
    assert skyramp.get_response_value(api_v1_products_POST_response, "price")==0
    assert skyramp.get_response_value(api_v1_products_POST_response, "product_id") is not None

Python

Java

Typescript

2.1 Single Method

You can find the used API specification here.

skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema openapi.json

This command generates a contract test file: products_POST_contract_test.py.
The content of this file is explained in Section 4 below.

Adjustments

Below are a few flags to customize the test generation. Additional flags are explained here.

--asserts: Specify Skyramp’s behavior to generate body value asserts [all, none] (default=3). Based on this flag Skyramp either generates asserts against no, three, or all response body values.
--response-status-code: Allows you to specify the expected status code. By default, Skyramp either takes the the provided status code in the API spec or for sample data assumes 20X.
--output: Allows you to specify the name of the generated test file.
--output-dir: Allows you to specify the directory to store the generated test file.

3. Execute Contract Test

You can execute the generated tests without any additional adjustments to the code.

3.1 Set env variable for authentication (if needed)

export SKYRAMP_TEST_TOKEN=$your_auth_token

3.2 Run the test

Now you can run the tests using pytest.

# Prerequisites 
pip install pytest

# Execution of contract test for products/POST 
python3 -m pytest products_POST_contract_test.py

3.3 Results

3.3.1 Successful Test

Using the single POST method example (default asserts) of the /v1/products endpoint, the test passes out of the box.

3.3.2 Test Failure

For this section, we intentionally changed the expected response body in the products_POST_contract_test.py test file.

Generated

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }''

Manual change

    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:32:14-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0,
            "extra_value": "invalid_field"
        }''

This results in a test failure given the expected response body does not match the actual response body anymore. The expected output looks the following:

Congratulations, you have now successfully created a contract test, executed it against a REST API, and received results!

4. Skyramp Test File

This section explains the key elements of the generated test files. This will enable you to make adjustments when needed quickly.

At the top of each file, we show when the test was generated and what command was used
Below, we import all relevant libraries and specify the URL for all test requests
We define a function per method that is tested. It consists of:
- Invocation of Skyramp Client
- Definition of the authentication header
- Definition of the request body
- Definition of the expected response body
- Creation of the request
- Generated Assertions
  - Status code assertion
  - Schema check
  - Body value checks (first 3 values)

4.1 Single Method

# Generated by Skyramp v0.5.5 on 2025-02-27 16:28:07.011532 -0500 EST m=+0.192097209
# Command: skyramp generate contract rest https://demoshop.skyramp.dev/api/v1/products \
# 		--api-schema openapi.json \
# 		--framework pytest \
# 		--language python \
# 		--method POST \

# Import of required libraries
import skyramp
import os
import time
# URL for test requests
URL = "https://demoshop.skyramp.dev"

# contract test for /api/v1/products POST
def test_api_v_1_products_post():
    # Invocation of Skyramp Client
    client = skyramp.Client()
    # Definition of authentication header
    headers = {}
    if os.getenv("SKYRAMP_TEST_TOKEN") is not None:
        headers["Authorization"] = "Bearer " + os.getenv("SKYRAMP_TEST_TOKEN")

    # Request Body
    api_v1_products_POST_request_body = r'''{
            "category": "string",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0
        }'''
    
    # Expected Response Body
    expected_api_v1_products_POST_response_body = r'''{
            "category": "string",
            "created_at": "2025-02-27T16:28:07-05:00",
            "description": "string",
            "image_url": "string",
            "in_stock": false,
            "name": "string",
            "price": 0,
            "product_id": 0
        }'''
    
    # Execute Request
    api_v1_products_POST_response = client.send_request(
        url=URL,
        path="/api/v1/products",
        method="POST",
        body=api_v1_products_POST_request_body,
        headers=headers
    )
    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None

if __name__ == "__main__":
    test_api_v_1_products_post()

4.1.1 Changing asserts

Default Generated Asserts (lines 55-60)

    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None

Manually Adjusted Asserts

Following the same principle you can simply adjust and add more assertions to the generated code. Additional details on defining assertions can be found in Pytest's Documentation.

    # Generated Assertions
    assert api_v1_products_POST_response.status_code == 201
    assert skyramp.check_schema(api_v1_products_POST_response, expected_api_v1_products_POST_response_body)
    assert skyramp.get_response_value(api_v1_products_POST_response, "category") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "created_at") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "description") is not None
    # Manual addition of asserts 
    assert skyramp.get_response_value(api_v1_products_POST_response, "image_url") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "in_stock") is not None
    assert skyramp.get_response_value(api_v1_products_POST_response, "name")=="string"
    assert skyramp.get_response_value(api_v1_products_POST_response, "price")==0
    assert skyramp.get_response_value(api_v1_products_POST_response, "product_id") is not None