Query Batching

Learn how query batching can help preserve data consistency of responses, and learn how to configure the GraphOS Router to receive query batches.

About query batching

Modern applications often require several requests to render a single page. This is usually the result of a component-based architecture where individual micro-frontends (MFE) make requests separately to fetch data relevant to them.

Query batching makes response data consistent for MFEs making multiple requests per page. It primarily preserves data consistency rather than improve performance.

Preserving data consistency

When the underlying data is changing rapidly, the separate requests of an application may result in responses with inconsistent data. Given two requests for a single page, the values returned in their responses may be different because the data's been updated or subgraph servers briefly had different values.

To prevent this inconsistency from happening, clients can bundle multiple requests together into a batch so routers or servers can produce responses with consistent data. MFE-based UIs usually batch multiple client operations, issued close together, into a single HTTP request. Both Apollo Client and Apollo Server support this.

Router batching support

The router's batching support is provided by two sets of functionality:

• client batching

• subgraph batching

With client batching, the router accepts batched requests from a client and processes each request of a batch separately. Consequently, the router doesn't present requests to subgraphs in batch form, so subgraphs must process the requests of a batch individually.

With subgraph batching, the router analyzes input client batch requests and issues batch requests to subgraphs. Subgraph batching is an extension to client batching and requires participating subgraphs to support batching requests. See the examples below to see illustrations of how this works in practice.

The GraphOS Router supports client and subgraph query batching.

The GraphOS Router must be configured to receive query batches, otherwise it rejects them. When processing a batch, the router deserializes and processes each operation of a batch independently. It responds to the client only after all operations of the batch have been completed. Each operation executes concurrently with respect to other operations in the batch.

Client batching support

If you’re using Apollo Client, you can leverage the built-in support for batching to reduce the number of individual operations sent to the router.

Once configured, Apollo Client automatically combines multiple operations into a single HTTP request. The number of operations within a batch is client-configurable, including the maximum number in a batch and the maximum duration to wait for operations to accumulate before sending the batch.

Configure client query batching

Both the GraphOS Router and client need to be configured to support query batching.

Configure router

Client query batching

By default, receiving client query batches is not enabled in the GraphOS Router.

To enable query batching, set the following fields in your router.yaml configuration file:

1batching:
2  enabled: true
3  mode: batch_http_link

Subgraph query batching

If client query batching is enabled, and the router's subgraphs support query batching, then subgraph query batching can be enabled by setting the following fields in your router.yaml configuration file:

1batching:
2  enabled: true
3  mode: batch_http_link
4  subgraph:
5    # Enable batching on all subgraphs
6    all:
7      enabled: true

1batching:
2  enabled: true
3  mode: batch_http_link
4  subgraph:
5    # Disable batching on all subgraphs
6    all:
7      enabled: false
8    # Configure(over-ride) batching support per subgraph
9    subgraphs:
10      subgraph_1:
11        enabled: true
12      subgraph_2:
13        enabled: true

Example: simple subgraph batching

This example shows how the router can batch subgraph requests in the most efficient scenario, where the queries of a batch don't have required fetch constraints.

Assume the federated graph contains three subgraphs: accounts, products, and reviews.

The input client query to the federated graph:

1[
2    {"query":"query MeQuery1 {\n  me {\n    id\n }\n}"}
3    {"query":"query MeQuery2 {\n  me {\n    name\n }\n}"},
4    {"query":"query MeQuery3 {\n  me {\n    id\n }\n}"}
5    {"query":"query MeQuery4 {\n  me {\n    name\n }\n}"},
6    {"query":"query MeQuery5 {\n  me {\n    id\n }\n}"}
7    {"query":"query MeQuery6 {\n  me {\n    name\n }\n}"},
8    {"query":"query MeQuery7 {\n  me {\n    id\n }\n}"}
9    {"query":"query MeQuery8 {\n  me {\n    name\n }\n}"},
10    {"query":"query MeQuery9 {\n  me {\n    id\n }\n}"}
11    {"query":"query MeQuery10 {\n  me {\n    name\n }\n}"},
12    {"query":"query MeQuery11 {\n  me {\n    id\n }\n}"}
13    {"query":"query MeQuery12 {\n  me {\n    name\n }\n}"},
14    {"query":"query MeQuery13 {\n  me {\n    id\n }\n}"}
15    {"query":"query MeQuery14 {\n  me {\n    name\n }\n}"},
16    {"query":"query MeQuery15 {\n  me {\n    id\n }\n}"}
17]

From the input query, the router generates a set of subgraph queries:

1"query MeQuery1__accounts__0{me{id}}",
2"query MeQuery2__accounts__0{me{name}}",
3"query MeQuery3__accounts__0{me{id}}",
4"query MeQuery4__accounts__0{me{name}}",
5"query MeQuery5__accounts__0{me{id}}",
6"query MeQuery6__accounts__0{me{name}}",
7"query MeQuery7__accounts__0{me{id}}",
8"query MeQuery8__accounts__0{me{name}}",
9"query MeQuery9__accounts__0{me{id}}",
10"query MeQuery10__accounts__0{me{name}}",
11"query MeQuery11__accounts__0{me{id}}",
12"query MeQuery12__accounts__0{me{name}}",
13"query MeQuery13__accounts__0{me{id}}",
14"query MeQuery14__accounts__0{me{name}}",
15"query MeQuery15__accounts__0{me{id}}",

All of the queries can be combined into a single batch. So instead of 15 (non-batch) subgraph fetches, the router only has to make one fetch.

Example: complex subgraph batching

This example shows how the router might batch subgraph requests for a graph, where the client batch contains a query for an entity.

Assume the federated graph contains three subgraphs: accounts, products, and reviews.

The input client query to the federated graph:

1[
2    {"query":"query MeQuery1 {\n  me {\n    id\n }\n}"},
3    {"query":"query MeQuery2 {\n  me {\n    reviews {\n      body\n    }\n  }\n}"},
4    {"query":"query MeQuery3 {\n  topProducts {\n    upc\n    reviews {\n      author {\n        name\n      }\n    }\n  }\n  me {\n    name\n  }\n}"},
5    {"query":"query MeQuery4 {\n  me {\n    name\n }\n}"},
6    {"query":"query MeQuery5 {\n  me {\n    id\n }\n}"}
7]

From the input query, the router generates a set of subgraph queries:

1"query MeQuery1__accounts__0{me{id}}",
2"query MeQuery2__accounts__0{me{__typename id}}",
3"query MeQuery3__products__0{topProducts{__typename upc}}",
4"query MeQuery3__accounts__3{me{name}}",
5"query MeQuery4__accounts__0{me{name}}",
6"query MeQuery5__accounts__0{me{id}}",
7"query MeQuery2__reviews__1($representations:[_Any!]!){_entities(representations:$representations){...on User{reviews{body}}}}",
8"query MeQuery3__reviews__1($representations:[_Any!]!){_entities(representations:$representations){...on Product{reviews{author{__typename id}}}}}",
9"query MeQuery3__accounts__2($representations:[_Any!]!){_entities(representations:$representations){...on User{name}}}",

The first six queries can be combined into two batches—one for accounts and one for products. They must be fetched before the final three queries can be executed individually.

Overall, without subgraph batching, the router would make nine fetches in total across the three subgraphs, but with subgraph batching, that total is reduced to five fetches.

Configure client

To enable batching in an Apollo client, configure BatchHttpLink. For details on implementing BatchHttpLink, see batching operations.

Configuration compatibility

If the router receives a query batch from a client, and batching is not enabled, the router sends a BATCHING_NOT_ENABLED error to the client.

Query batching and GraphOS plans

When processing batched operations, the router counts each entry in the batch as a distinct billable operation that counts towards your graph usage. Sending a batch with N operations is counted the same as sending N requests of the same non-batched operation.

Note that GraphOS plans only track router operations, so configuring subgraph batching does not impact your graph usage count.

Metrics for query batching

Metrics in the GraphOS Router for query batching:

The subgraph attribute is optional. If the attribute isn't present, the metric identifies batches received from clients. If the attribute is present, the metric identifies batches sent to a particular subgraph.

Query batch formats

Request format

A query batch is an array of operations.

1[
2query MyFirstQuery {
3  me {
4    id
5  }
6},
7query MySecondQuery {
8  me {
9    name
10  }
11}
12]

Response format

Responses are provided in JSON array, with the order of responses matching the order of operations in the query batch.

1[
2  {"data":{"me":{"id":"1"}}},
3  {"data":{"me":{"name":"Ada Lovelace"}}}
4]

Error handling for query batching

Batch error

If a batch of queries cannot be processed, the entire batch fails.

For example, this batch request is invalid because it has two commas to separate the constituent queries:

1[
2query MyFirstQuery {
3  me {
4    id
5  }
6},,
7query MySecondQuery {
8  me {
9    name
10  }
11}
12]

As a result, the router returns an invalid batch error:

1{"errors":
2  [
3    {"message":"Invalid GraphQL request","extensions":{"details":"failed to deserialize the request body into JSON: expected value at line 1 column 54","code":"INVALID_GRAPHQL_REQUEST"}}
4  ]
5}

Excessive queries batch error

If the number of queries provided exceeds the maximum batch size, the entire batch fails.

For example, this configuration sets a batch size limit of 2, but three queries are provided:

1batching:
2  enabled: true
3  mode: batch_http_link
4  maximum_size: 2

1[
2  query MyFirstQuery {
3    me {
4      id
5    }
6  },
7  query MySecondQuery {
8    me {
9      name
10    }
11  },
12  query MyThirdQuery {
13    me {
14      name
15    }
16  }
17]

As a result, the router returns an error:

1{
2  "errors": [
3    {
4      "message": "Invalid GraphQL request",
5      "extensions": {
6        "details": "Batch limits exceeded: you provided a batch with 3 entries, but the configured maximum router batch size is 2",
7        "code": "BATCH_LIMIT_EXCEEDED"
8      }
9    }
10  ]
11}

Individual query error

If a single query in a batch cannot be processed, this results in an individual error.

For example, the query MyFirstQuery is accessing a field that doesn't exist, while the rest of the batch query is valid.

1[
2query MyFirstQuery {
3  me {
4    thisfielddoesnotexist
5  }
6},
7query MySecondQuery {
8  me {
9    name
10  }
11}
12]

As a result, an error is returned for the individual invalid query and the other (valid) query returns a response.

1[
2  {"errors":
3    [
4      {"message":"cannot query field 'thisfielddoesnotexist' on type 'User'",
5       "extensions":{"type":"User","field":"thisfielddoesnotexist","code":"INVALID_FIELD"}
6      }
7    ]
8  },
9  {"data":{"me":{"name":"Ada Lovelace"}}}
10]

Known limitations

Unsupported query modes

When batching is enabled, any batch operation that results in a stream of responses is unsupported, including:

• @defer

• subscriptions