E2E Testing

Introduction

This guide explains how to generate E2E tests for complex scenarios with Skyramp using traces. Throughout this guide, we will demonstrate key capabilities using Skyramp’s Demo Shop, a simple e-commerce application for product and order management. Learn more about the Demo Shop.

Prerequisites

• Skyramp CLI installed

• Relevant libraries installed (ie Python, TypeScript libraries)

• Docker installed and running (for trace collection only)

Refer to the Installation Guide if you haven't installed Skyramp yet.

Overview

An end-to-end (E2E) test validates an application's entire workflow from start to finish, simulating real-world user scenarios. It ensures that all integrated components, including front-end, back-end, and external systems, work together seamlessly to deliver the expected functionality and performance. 

To generate E2E tests that span across multiple endpoints or services, Skyramp requires a trace to generate the test case reliably.

What is a trace?

A trace is a log of a set of actions executed in a particular window or scenario in your system. Traces can be used to simulate real-world scenarios of product usage, and those traces can in turn be used to generate tests that give you confidence that said scenario will be resilient in production traffic.

Skyramp can function as a “Man-in-the-Middle” proxy that monitors your traffic and allows you to simply record the scenario you want to test. Skyramp can collect traffic from 3 different sources:

• HTTP traffic from a browser UI

• Backend traffic through the CLI

• UI traces through the Playwright recorder

By following these simple steps, you'll be able to collect a trace that you can directly supply for the Skyramp test generation.

Start Trace Collection

Want to skip straight to test generation? Download the following Skyramp trace and Playwright trace and skip to “Generate E2E Test from Trace”

To start collecting traces, you need to run the following command. This command will spawn a new shell configured to collect all relevant backend network traffic as well as a Playwright Browser and Inspector. All visited web links traffic will also be part of the collected trace. The first time you run this command, it might take a while to bring up Skyramp.

skyramp generate trace \
--playwright \
--include "https://demoshop.skyramp.dev/api/*" \
--output e2e_test_trace.json \
--playwright-output

Explanation of Command

• --playwright: Enable Playwright to capture UI activities (default: false)

• --include: Specify a comma-separated list of URLs for Skyramp to collect traces for. Skyramp will only collect traces for those domains and exclude all others.

• --output: Specify the name of the generated Skyramp trace file (default: skyramp_traces.json)

• --playwright-output: Specify the name of the generated Playwright trace file (default: skyramp_playwright.zip)

Adjustments:

• --exclude: Specify a comma-separated list of URLs for which you do not want Skyramp to collect traces. This flag takes precedence over --include if a URL is specified in both. --exclude will only exclude the domains specified and keep all other domains in the trace.

• --docker-network: Specify the name (string) of your Docker network.

• --docker-skyramp-port: Specify a port number for Skyramp (default=35142).

A Note on Including/Excluding URLs

When using the --include or --exclude flag, URLs need to be specified either as an exact path to the endpoint or in combination with a wildcard:

• demoshop.skyramp.dev/api/v1/products will only filter for POST and GET of the products endpoint.

• demoshop.skyramp.dev/products* will filter for all products endpoints + any nested endpoints.

• demoshop.skyramp.dev/api/v1/products/* will filter only for any nested endpoints that require a product ID. It will not collect the traces of POST and GET products.

For endpoints that require path parameters, please replace the path parameter with the wildcard:

• Convert demoshop.skyramp.dev/products/{product_id}/reviews to demoshop.skyramp.dev/products/*/reviews

Record Scenario

We now leverage the freshly spawned shell and Playwright Browser to collect the traces for our test scenario. Here is a quick summary of the scenario:

1. Fetch session ID (via UI)

2. Clear state (via cURL)

3. Create new product (via cURL)

4. Find product in product list (via UI)

5. Create new order containing the new product (via UI)

6. View newly created order (via UI)

7. Delete newly created order (via cURL)

8. Delete newly created product (via cURL)Let's get started!

1. Setup session (via UI)

• To start, in the newly spawned Chromium browser, go to https://stg.demoshop.skyramp.dev/products . This will open the Demo Shop UI.

• At the top of the page, you will see a section called Session ID with three words. Click on the Edit button, and fill in the session ID with any string (for this example, we will use skyramp-e2e-demo)

  

• Save the session ID by clicking on the green checkmark.

• Next, click “Clear State” to reset and begin your session recording.

• This session ID will be used as your “Authorization” token for every API call made to the Demo Shop. You can save your session ID in an environment variable for re-use across the different network requests from the terminal:

  export SKYRAMP_TEST_TOKEN=skyramp-e2e-demo

2. Create new product (via cURL)

• Now, let’s create a new product with the Demo Shop API

  curl -X POST https://demoshop.skyramp.dev/api/v1/products \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $SKYRAMP_TEST_TOKEN" \
    -k \
    -d '{
      "name": "SkyPad Pro",
      "description": "High-performance tablet powered by Skyramp",
      "price": 999.99,
      "image_url": "https://shorturl.at/sZeO8",
      "category": "Tablets",
      "in_stock": true
    }'

3. Find product in product list (via UI)

• Refresh your Chromium browser. This will cause your newly created product to appear on the Product Catalog.

  

• 
  

3. Update the product price

• Click on the “View Details” button in the “SkyPad Pro” tile.

  

• Click on “Edit Product”

• In the “Price” field, enter “1999.99” and click “Save Product”

  

4. Create new order containing the new product (via cURL)

• From the original POST products API call, copy the product_id field that is returned in the response (product_id should be 24 if this entire recorded flow is done from a cleared session)

• We will use this product_id to create a new order:

  curl -X POST https://demoshop.skyramp.dev/api/v1/orders \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $SKYRAMP_TEST_TOKEN" \
    -k \
    -d '{
      "customer_email": "sahil@skyramp.dev",
      "items": [
        {
          "product_id": 24,
          "quantity": 2
        }
      ]
    }''

5. Find order in order list (via UI)

• Click on the Orders tab in the navigation bar. Find your newly created order on the list.

  

  
  

6. Assert the total price of the order (via UI)

• Click on “View Details” in the newly created order.

• Assert that the price of the order appears correctly

  • Click on the button

  • Click on the total price of the order

  • This will ask you to generate an assertion that the element contains the text “$3999.98”. Click the checkmark.

    

• 
  

7. Cancel newly created order (via UI)

• From the order details page, click on “Cancel Order”

  

7. Delete newly created product (via UI)

• Navigate to the Product Catalog

• Click on the product you created during this session

• Click on Delete Product

  

End Trace Collection

Once the scenario is completed, you can end the trace collection in the spawned shell with Ctrl + D. You will see for which endpoints Skyramp was also able to collect backend traces. Note that if the --include flag was not used to filter for specific domains, this might be a very long list.

Skyramp creates two trace files: e2e_test_trace.json and e2e_test_playwright.zip. The first file only contains the backend traces that were collected, while the zip file includes all collected traces from Playwright. Both traces will be used for the generation of the E2E test.

You can find sample traces for the test scenario here:

• Skyramp Trace

• Playwright Trace