Consumer Driven Contract Testing

We all have used some REST API where we did not care about most of the response fields. We might care about the name of the repos of a Github user but not about his email or followers. On the other side an API provider such as Github might be scared to remove fields from a response. They might have no idea if anyone, and if then who, uses the provided fields.

There are APIs around which, if we wanted to display relevant data to our users, we had to do three requests. Each of the request might be depending on the previous request. We might have to fetch a user by id, then all his friends, to then finally get the birthday of the friends.

Right now services are being built with beautiful APIs. Some might be using HATEOAS, just to then have the frontend developers ask “Can’t we just do everything with GET requests?”.

One way to deal with the issues above is to employ a technique called Consumer Driven Contract Testing. The consumers, e.g. the teams working on an iOS or a React app, specify what data they require from the API. These data requirements will be provided as a contract. Put into words the contract might define the following:

If I, the consumer A, call the api endpoint /users/{userId} with a specific userId then I expect the api to return the address of the user.

Another consumer might state

If I, the consumer B, call the api endpoint /users/{userId} with a specific userId then I expect the api to return the birthday of the user.

To allow the enforcement verification of these contracts the sentences above have to be put into code. After all, we probably dont want ambiguous contracts.

Showtime

The following example use Spring Cloud Contract

As displayed above, we have two consumers of the API. One consumer is interested in the address of a user. The other consumer is interested in the birthday of the user. Both consumers will write a contract each. They specify the attributes they care about and provide it to the producer to fulfill.

Contract 1: Provide the address of a user

org.springframework.cloud.contract.spec.Contract.make {
    request {
        method GET()

        url("/users/1")
    }

    response {
        status 200
        body("""
            {
                "id": 1,
                "address": "Somestreet 123 in City"
            }
        """)
    }
}

Contract 2: Provide the birthday of a user

org.springframework.cloud.contract.spec.Contract.make {
    request {
        method GET()

        url("/users/2")
    }

    response {
        status 200
        body("""
            {
                "id": 2,
                "birthday": "1995-08-15"
            }
        """)
    }
}

We explicitly went for two contracts to display the fact that lots of services have multiple consumers.

Of course these contracts are not just blindly accepted and fulfilled by the producers. A nice side effect of these contracts is to start a conversation between consumers and producers. The conversation should result in a proper API that fits both sides. What often emerges is the Backend For Frontends Pattern.

Consumer side

The contracts will be used to generate stub files for the consumers. These stub files could be plain JSON files. They can be interpreted by specific HTTP-servers to make sure if the HTTP-Server is called with specific values the response described in the contract is provided.

Thus a consumer calling /users/2 on the HTTP-Server will receive the birthday field and when calling /users/1 will receive the address field. It’s important that the requests differ in the url or query parameters. The stub-server needs a way to decide which response to provide.

As a result the consumer is able to write HTTP clients and json parsers to work with the stubbed responses.

It’s important the stubs the consumer is using do not get out of sync with the stubs generated by the contract. This is often achieved by having the CDCT library download the newest stubs during a test run. Of course the contract should not change after the fact. That’s where the consumer driven part shines. The provider should not change the contracts. But if he does having up to date stubs at least serves as an early warning.

Provider side

On the provider side the contracts are used to generate tests. With these tests the provider can make sure his API abides by the contract. Any breakage of API should lead to a failing test.

The generated tests might look as follows for the first contract

assertThat(response.statusCode()).isEqualTo(200);
DocumentContext parsedJson =
	JsonPath.parse(response.getBody().asString());
assertThatJson(parsedJson).field("address")
	.isEqualTo("Somestreet 123 in City");

Let’s say a malicious provider removes fields from the response. Of course the generated tests will fail and he knows something went wrong. But as soon as he also removes the field from the contract there will be no more test verifying whether the value is present or not. The contract was broken.

But, if the consumer is always using the latest stub files he will notice the stubs suddenly lack a field he is expecting. Appropriate measures can then be taken. Such as notifying the provider team about the issue at hand.

Issues

Some users don’t like how the tests ignore unknown fields. Our API could return

{
  "address": "Somestreet 123 in City",
  "birthday": "1995-08-15",
  "foo": "bar"
}

and the tests on the provider side would not care at all if the response contains "foo":"bar". Nor should it. The reason is Postel’s law, often stated as:

Be conservative in what you send, be liberal in what you accept

Martin Fowler states it as:

My recommendation is to be as tolerant as possible when reading data from a service. If you’re consuming an XML file, then only take the elements you need, ignore anything you don’t.

Consumers following Postel’s law would be considered tolerant readers and support the evolution of services without breaking.

Additionally we should remember the contracts are not the schema or documentation of the API. We have better tools to achieve that. Swagger comes to mind. Although you might use them to document which parts of the API are in use.

Another thing which leads to pain is asserting the explicit values when testing the parsers and HTTP-Clients.

The consumers should not have a test which attempts the following

assertThat(response.address).isEqualTo("Somestreet 123 in City");

The above would lead to a failing test if the provider would ever change the value to Somestreet 123 in Town although the contract was not broken. It’s still returning an address.

Having the following test instead would greatly improve the stability.

assertThat(response.address).isNotBlank();

Summary

In one of my projects were extensively using CDCT. We use it both to keep our backends and clients in sync and also to keep the interaction of the different backends (Messaging and REST) in sync. I would not want to miss it outside of a microservices environment. One or more consumers of an API would already warrant the technique.

We can also use CDCT to write integration tests for third party API