In software development, especially with microservices, ensuring seamless integration between components is crucial for delivering high-quality applications. One approach I really like, to tame this complexity, is contract testing.

Contract testing is a powerful technique that focuses on verifying interactions between software components early and in a controlled environment. In this post, I want to show why I think contract testing can often reduce the amount of integration testing.

Contract Testing

A contract, in this context, is a scenario that describes an interaction between two components. A very simple contract could describe a call to a REST API, and its response, but they can describe more complex scenarios.

Contract testing consists in testing both components against the contract and specifications. Crucially, there are two different tests: not only is Service-1 tested against the contract, but Service-2 is tested against the specifications. Contract testing doesn’t involve both components at the same time.

Contrary to unit testing with a mock, contract tests are bi-directional, verifying both requirements and implementation during build time. Also contrary to integration testing, we don’t need to tackle the preparation of dependencies in an integration testing environment. Contract testing can be run in an isolated way whenever there is an update, even locally. For these reasons, I consider contract testing as both easy and useful.

Challenges in Integration Testing

As the number of services grow, the number of interactions between them, that integration tests are traditionally responsible for testing, grows, and the challenges of interaction testing become more apparent:

• Integration tests are late
• Integration tests require a lot of context
• Integration tests are expensive

Let me elaborate.

Integration tests are late

Integration testing is conducted after several stages in the pipeline, including static checks, code building, unit tests, reviews, and deployment to the test environment. Providing feedback on integration issues after all these steps requires repeating the entire process multiple times. Consequently, running integration tests can result in delayed feedback. While this approach may seem reasonable due to the structured process, it can become a significant bottleneck in the pipeline for large projects.

Integration tests require a lot of context

Integration testing environments encompass the integration of all necessary cloud resources, effectively creating production-like settings. This approach is valuable as it provides comprehensive feedback on the overall system’s performance. However, evaluating the impact of a single commit within such an environment is often slow and inefficient. Maintaining these environments is challenging, and any issues or failures in dependent services can lead to false-positive results.

Another significant challenge is flakiness, which frequently arises from improperly managed test data that might be used by one of the dependent services. Managing this data is complex due to the numerous dependencies involved in creating and manipulating it.

Integration tests are expensive

Challenges in integration testing makes it expensive to maintain and run tests. Building a complex production-like testing environment requires all dependent services and databases to be updated and functioning. Running a single check requires to go all the way down to the network. Imagine how hard, slow and expensive it is to run integration tests given the following network traffic:

The microservice architectural pattern has led to a notable increase of such complex networks. For this reason, the shift-right strategy of doing integration tests late in the pipeline became less relevant, and microservice pioneers like Netflix or Amazon have been advocating for a shift-left strategy such as contract testing to test their massive networks.

The first thing you observe from the image is that we have a lot of integrations between the microservices. With hundreds of microservices, the number of integrations between them becomes too many. Consider two services, which have one interaction. With three services, it can be up three, then six, ten, fifteen, twenty-one, and so on. This increases drastically as the number reaches hundreds. For example, 100 services have 4950 potential integrations, and 500 services have 124750. This is based on the $n$-choose-$k$ formula where $n$ is the number of microservices, and $k=2$ (as we’re counting pairs of services that can interact bidirectionally):

This calculates the maximum number of integrations with $n$ services. Asymptotically, the number of interactions grows as the square of the number of services. It is not realistic to say we will have that many of integrations but it gives an idea of how fast the number of interactions grows. On the other hand, one interaction can involve many API calls, each requiring tests.

Let’s give a real-world example. Say we have 10 microservices and the integration between the microservices are shown in the table:

Total Integration Points = $\sum$ (Number of integrations for each microservice)
Total Integration Points = 3 + 3 + 3 + 3 + 3 + 3 + 3 + 3 + 3 + 3 = 30

Again each interaction can require many tests.

Netflix, Google, and Amazon are pioneers in microservices testing. Netflix publicly shared their experience, showing how their testing evolved. Netflix and Spotify have also changed the traditional test pyramid, turning it into a test diamond/honeycomb. While unit tests are still important, the focus has shifted to writing more integration tests rather than extensive unit tests. To learn more about Spotify’s test pyramid transformation for microservice testing, read this post.

Consumer-Driven Contract Testing

In the terminology of contract testing, a consumer is a client of the API under test while a provider is a service that exposes the API. The most common architecture for contract-testing is consumer-driven contract testing, where the contract is defined in the consumer component, and shared with the provider. The converse, provider-driven contract testing is mostly useful when you have a public API and want to share contracts with unknown consumers. I’ll be focusing on consumer-driven contract testing.

Imagine the following scenario:

• Order Service (Consumer): Responsible for managing orders and inventory.
• Inventory Service (Provider): Maintains the inventory levels of products. The Order Service needs to check the availability of products in the Inventory Service before processing an order.

The Order Service needs to check the availability of products in the Inventory Service before processing an order. The Order Service sets the expectation as the consumer by defining this expectation in a specification which produces a contract document. The contract is then stored by a special service, called the broker, which makes the contract available to the provider for its own tests.

We’ll use Pact to ensure that the Inventory Service meets the expectations of the Order Service. Pact is the most popular contract-testing framework and supports many languages. Another popular contract-testing framework is Spring Cloud Contract which supports JVM based applications.

Consumer Implementation in Python (Order Service):

# test_order_service.py
import unittest
from pact import Consumer, Provider
import requests

class OrderServicePactTest(unittest.TestCase):
    def setUp(self):
        # create a `pact` object by defining the consumer and the provider
        self.pact = Consumer('OrderService').has_pact_with(Provider('InventoryService'), pact_specification_version="3.0.0")
        self.pact.start_service()
        self.addCleanup(self.pact.stop_service)
        self.base_url = 'http://localhost:1234'

    def test_order_service(self):
        # simple order object that should be the response
        expected = {
            'product_id': '123',
            'available': True
        }

        # setting the specification
        (self.pact
         .given('Product 123 exists')                                # set a precondition
         .upon_receiving('a request to check product availability')  # this is the name of the interaction aka test case, `description` of the interaction in the contract
         .with_request('get', '/inventory/123')                      # request detail
         .will_respond_with(200, body=expected))                     # response detail

        # running the specification
        with self.pact:
            result = requests.get(f'{self.base_url}/inventory/123')

        self.assertEqual(result.json(), expected)

if __name__ == '__main__':
    unittest.main()

Let’s run the test on the consumer side (Order Service):

python -m unittest test_order_service.py

Upon running the command above, the specification: “a request to check product availability”, in the Pact consumer test, is turned into an interaction in a document. This document is the contract between OrderService and InventoryService which is generated by Pact in a JSON file (e.g. orderservice-inventoryservice.json):

{
  "provider": {
    "name": "InventoryService"
  },
  "consumer": {
    "name": "OrderService"
  },
  "interactions": [
    {
      "description": "a request to check product availability",
      "request": {
        "method": "GET",
        "path": "/inventory/123",
        "headers": {}
      },
      "response": {
        "status": 200,
        "headers": {},
        "body": {
          "available": true
        }
      }
    }
  ],
  "metadata": {
    "pactSpecification": {
      "version": "3.0.0"
    }
  }
}

Notice how this test can easily be run locally on your machine. It can just as easily run in CI, even if the inventory service is in another repository, since no actual inventory service is required to run the test. It doesn’t require a complex setup or configuration, and runs on the local network loop, which is very fast.

When the contract is ready, we can publish it to the Pact broker which is a service for holding all the contracts.

export PACT_BROKER_BASE_URL=<patc-broker-url>
export PACT_BROKER_USERNAME=<username>
export PACT_BROKER_PASSWORD=<password>

pact-broker publish ./pacts/orderservice-inventoryservice.json \
    --consumer-app-version consumer-version \
    --broker-base-url $PACT_BROKER_BASE_URL \
    --broker-username $PACT_BROKER_USERNAME \
    --broker-password $PACT_BROKER_PASSWORD \
    --tag dev

You can either run the Pact broker locally or use the cloud services provided by Pact. Either way, you’ll have to set up a few environment variables for the Pact CLI to connect to the service. To run the Pact broker locally, you should select an database adapter such as sqlite or postgres and then run the Docker command.

docker run -d --name pact-broker -p 9292:9292 \
  -e PACT_BROKER_DATABASE_ADAPTER=sqlite \
  -e PACT_BROKER_DATABASE_NAME=/var/pact_broker/db.sqlite3 \
  -v $(pwd)/pact_broker:/var/pact_broker \
  pactfoundation/pact-broker

Provider Implementation in Python (Inventory Service):

# test_inventory_service.py
import unittest
from pact import Verifier

class InventoryServicePactTest(unittest.TestCase):
    def test_inventory_service(self):
        # define the verifier by defining the `provider` which will be used to get all the contracts
        # whose provider are set to `InventoryService` so that to run all the verification tests
        verifier = Verifier(provider='InventoryService', provider_base_url='http://localhost:8000')
        pact_broker_url = '<pact-broker-url>'
        broker_username = '<username>'
        broker_password = '<password>'

        # `verify_with_broker` connects to the pact broker and pulls all the related contract and does the verification
        verifier.verify_with_broker(
            broker_url=pact_broker_url,
            broker_username=broker_username,
            broker_password=broker_password,
            publish_version='1.0.0',
            provider_tags=['master']
        )

if __name__ == '__main__':
    unittest.main()

First, we should run the provider service (inventory service):

python inventory_service.py

Then, let’s run the provider verification test for the inventory service:

python -m unittest test_inventory_service.py

Here again, the test was easy to run, and doesn’t require any knowledge of an actual implementation of the consumer.

What happens if there’s a mistake in the provider’s implementation, and it doesn’t actually satisfy the contract? In this case, Pact would respond with an error looking something like this.

    >       assert resp.status_code == 200, resp.text
    E       AssertionError: Actual interactions do not match expected interactions for mock MockService.
    E
    E       Missing requests:
    E           GET /inventory/123
    E
    E       See pact-mock-service.log for details.

    venv/lib/python3.10/site-packages/pact/pact.py:209: AssertionError

After the error is fixed, you can check the status, for instance, in Pact’s UI (pactflow):

Conclusion

Contract testing addresses the challenges inherent in integration testing. By shifting integration testing to an earlier stage in the development process, it eliminates the need for maintaining complex integration testing environments, such as data preparation and deployment. Additionally, contract testing can be executed in isolation whenever a service changes, removing the necessity for integrating all services into a single running environment.

Contract testing doesn’t eliminate the need for integration tests altogether, such as testing end-to-end scenarios as system tests. But many integration tests can be replaced by contract tests, such as interactions between microservices. As a consequence we can have much fewer tests that depend on a complex, slow, unreliable network environment, rendering the whole process much faster.