Validating schema changes

Check if schema changes are safe or breaking by comparing against live server traffic

There are many types of schema changes that can be potentially breaking to clients, like removing a field, if made without special consideration. For safety, some organizations take the approach of never making these types of changes, but this leads to an ever-growing schema and reduced API flexibility over time. In reality, making these types of changes to a schema can be very safe as long as you have tools in place to ensure that no queries are broken in the process.

Apollo provides a tool to protect for exactly this scenario called schema validation.

Note: Schema validation is an Apollo Platform feature available on the Team and Enterprise plans of Apollo Graph Manager.

How it works

Schema validation is run through the Apollo CLI by executing the apollo service:check command. Apollo will generate a diff between your local schema and your most recently registered schema, then validate that the changes are safe by checking if any queries actively running against your graph will be affected.

Here's how it works:

  1. You run apollo service:check locally or in CI. The proposed schema is sent to Graph Manager's schema registry.
  2. Graph Manager creates a diff between the local schema and the most recently published schema in the registry.
  3. Graph Manager fetches a list of all operations sent to your graph in the last day (time window is configurable).
  4. Graph Manager walks through the schema diff change-by-change and compares against the operation list to see if the changes will affect the behavior of any operations.
  5. Graph Manager returns the schema diff and indicates any breaking changes found.
  6. The CLI prints the output of this check with a link to view more details in the Graph Manager UI.

Breaking change detection

Not all schema changes are potentially breaking. Some changes, like adding a field, will always be safe and never cause unexpected behavior for active clients. Other changes, like removing a field or changing a return type, can potentially affect the behavior of clients making queries that use those fields. These are what we consider potentially breaking changes.

If schema validation detects that a proposed schema has a potentially breaking change, the apollo service:check command will return a non-0 exit code. Apollo schema validation will detect breaking changes according to the following rules:


Each of these change types removes a schema element. If an element of your graph is being actively used by an operation and it is removed, your GraphQL layer will start returning errors to the dependent operations.

  • FIELD_REMOVED: Field used by at least one operation was removed
  • TYPE_REMOVED: Type(scalar, object) used by at least one operation was removed
  • ARG_REMOVED: Argument was removed from a field used by at least one operation
  • TYPE_REMOVED_FROM_UNION: Type was removed from a union used by at least one operation
  • INPUT_FIELD_REMOVED: Field removed from an input type referenced by an argument on a field used by at least one operation
  • VALUE_REMOVED_FROM_ENUM: A value removed from an enum used by at least one operation
  • TYPE_REMOVED_FROM_INTERFACE: An object removed from an interface used by at least one operation

Required arguments

Each of these changes adds a required input to a schema element. If an operation is actively using an element of your graph and doesn't update itself to add the new required input argument, the GraphQL layer will start returning an error to the operation.

  • REQUIRED_ARG_ADDED: Non-nullable argument added to field used by at least one operation
  • NON_NULL_INPUT_FIELD_ADDED: Non-null field added to an input object used by at least one operation

In-place updates

Each of these changes updates an existing schema element. If an operation is activley using an element of your graph and that element is updated, the operation could start receiving an error from the GraphQL layer or, in some cases, an unexpected result.

Note: In some cases, these changes are compatible with the client at runtime, such as a type rename or an object to interface conversion with the same fields. Schema validation still marks these breaking changes because validation does not have enough information to ensure safety and these changes deserve extra scrutiny, such as their impact on type generation.

  • FIELD_CHANGED_TYPE: Field used by at least one operation changed return type
  • INPUT_FIELD_CHANGED_TYPE: Field in input object changed type and is referenced by argument on field used by at least one operation
  • TYPE_CHANGED_KIND: Type used by at least one operation changed, ex: scalar to object or enum to union
  • ARG_CHANGED_TYPE: Argument changed type on field used by at least one operation

Default arguments

These changes update the default value for an argument. If an operation is using an element of your graph and does not specify a value for this argument, the operation could experience unexpected results when the schema is updated if it was relying on the original default value.

  • ARG_DEFAULT_VALUE_CHANGE: Default value added or changed for argument on a field used by at least one operation

Non-breaking changes

These are change types detected ny the apollo service:check command, but they are "safe" and will always be compatible with all exisitng client usage of the graph. They will not affect the behavior of any clients if deployed.

  • Optional arguments
    • OPTIONAL_ARG_ADDED Nullable argument added to a field
    • NULLABLE_FIELD_ADDED_TO_INPUT_OBJECT Nullable field added to an input object
  • Additions
    • FIELD_ADDED Field added to a type
    • TYPE_ADDED Type added to the schema
    • VALUE_ADDED_TO_ENUM Value added to an enum. If clients contain a switch case on the enum and do not include the `default`, this change could cause unexpected behavior
    • TYPE_ADDED_TO_UNION Type added to a union used by at least one operation
    • TYPE_ADDED_TO_INTERFACE Interface added to an object used by at least one operation
  • Deprecations
    • FIELD_DEPRECATED Field deprecated
    • FIELD_DEPRECATION_REMOVED Field no longer deprecated
    • FIELD_DEPRECATED_REASON_CHANGE Reason for deprecation changed
    • ENUM_DEPRECATED Enum deprecated
    • ENUM_DEPRECATION_REMOVED Enum no longer deprecated
    • ENUM_DEPRECATED_REASON_CHANGE Reason for enum deprecation changed

Validation response

Running a schema validation check is as simple as running apollo service:check on the command line from within a service repository that is configured to be an Apollo project.

Running apollo service:check will output the diff of all schema changes found and highlight changes determined to be breaking. Here's an example:

$ npx apollo service:check --tag=prod
  ✔ Loading Apollo Project
  ✔ Validated local schema against tag prod on service engine
  ✔ Compared 8 schema changes against 110 operations over the last 24 hours
  ✖ Found 3 breaking changes and 5 compatible changes
    → breaking changes found

FAIL    ARG_REMOVED                `ServiceMutation.checkSchema` arg `gitContext` was removed
FAIL    FIELD_REMOVED              `Schema.fieldCount` was removed
FAIL    FIELD_REMOVED              `Schema.typeCount` was removed

PASS    FIELD_ADDED                `SchemaTag.schemaRepoID` was added
PASS    FIELD_CHANGED_TYPE         `ServiceMutation.uploadPartialSchema` changed type from `UploadPartialSchemaResponse!` to `CompositionResult!`
PASS    FIELD_DEPRECATION_REMOVED  `IntrospectionSchema.fieldCount` is no longer deprecated
PASS    FIELD_DEPRECATION_REMOVED  `IntrospectionSchema.typeCount` is no longer deprecated
PASS    TYPE_REMOVED               `UploadPartialSchemaResponse` removed

View full details at: https://engine.apollographql.com/service/example-1234/check/<DETAILS>

Each change to the schema will be labeled with PASS or FAIL and a URL with full details on the changes and their impact on clients and operations will be generated. Following the URL will take you to Graph Manager:

Service check page in the Graph Manager UI

Note: If you have installed schema validation checks on your GitHub PRs, the "Details" link in your GitHub checks will take you to the same details link in this output.

A failed apollo service:check command will exit with a non-0 exit code and fail CI checks. There are many cases where it is safe to make a potentially breaking change, as long as the change is made intentionally with an understanding of its impact.

Since breaking changes are detected using live traffic, your service will need active metrics for the change algorithm to detect failures. If there are no metrics associated with your service, all changes will be labeled as a PASS as opposed to a FAIL.

Set up schema validation

To set up schema validation, you will need to be both actively sending traces and registering schemas to Apollo:

  1. Set up trace reporting to Apollo Graph Manager
  2. Set up schema registration in your continuous delivery pipeline

Then, you will need to configure your project for the apollo service:check command:

Note: If you have set up one of Apollo's workflows previously, your project may already have its .env file and apollo.config.js file configured.

Once you've got these set up, running your schema check is as simple as running:

$ npm install apollo
$ npx apollo service:check

The command can be placed in any continuous integration pipeline. To surface results, apollo emits an exit code and integrates with GitHub statuses. The time window of live traffic that the check command validates against can be configured to any range within your data retention window.

Note: The Apollo CLI will be looking in your Apollo config for a location from which to fetch your local schema and using your ENGINE_API_KEY to authenticate its requests with the Graph Manager service.

Run validation on each commit

We highly recommended that you add validation to your continuous integration workflow (e.g. Jenkins, CircleCI, etc.). In doing so, you can detect potential problems automatically and display the results of checks directly on pull requests.

Here's a example of how to add a schema validation check to CircleCI:

version: 2

      - image: circleci/node:8

      - checkout

      - run: npm install

      # Start the GraphQL server.  If a different command is used to
      # start the server, use it in place of `npm start` here.
      - run:
          name: Starting server
          command: npm start
          background: true

      # make sure the server has enough time to start up before running
      # commands against it
      - run: sleep 5

      # This will authenticate using the `ENGINE_API_KEY` environment
      # variable. Configure your endpoint's location in your Apollo config.
      - run: npx apollo service:check

Note: If you're using GitHub status checks, we recommend ignoring the exit code of the apollo service:check command so your continuous integration can complete without failing early. This can be done by appending || echo 'validation failed' to the command call.

GitHub integration

GitHub Status View

Like most tools, schema validation is best used when it is integrated directly into the rest of your workflow. If you're using GitHub, you can install the Apollo Engine GitHub app. This will enable Apollo's systems to send a webhook back to your project on each apollo service:check, providing built-in pass/fail status checks on your pull requests.

To install the Apollo Engine integration on GitHub, go to https://github.com/apps/apollo-engine, click the Configure button, and select the appropriate GitHub profile or organization.

Posting a comment to your PRs

For teams using GitHub Enterprise, Bitbucket, and other source control tools, we recommend setting up your CI to post a comment on your PRs with the results of schema validation. Surfacing schema diffs and breaking changes directly in your PR will speed up your review workflow by saving you the time of searching your CI logs to check why validation didn't pass.

The CLI supports passing a --markdown flag to apollo service:check, which outputs the results of schema validation in a markdown format specifically. This markdown can be piped directly into a comment to your source control tool, like in this example of posting a comment with the results of schema validation to GitHub.

The output of apollo service:check --markdown looks like this:

### Apollo Service Check

🔄 Validated your local schema against schema tag `staging` on service `engine`.
🔢 Compared **18 schema changes** against **100 operations** seen over the **last 24 hours**.
❌ Found **7 breaking changes** that would affect **3 operations** across **2 clients**

🔗 [View your service check details](https://engine.apollographql.com/service/engine/checks?...).

Multiple environments

Product cycles move fast and it's common for schemas to be slightly different across environments as changes make their way through your system. To support this, schemas pushed to the registry can be associated with specific variants of your graph (also referred to tags).

Variants mostly commonly represent environments and can also indicate branches or future schemas. Passing the --tag=<VARIANT> flag to apollo service:check specifies which schema variant to compara against, such as prod or staging. It's common to run checks against multple different graph variants in the same continuous integration pipeline to ensure that all important deployments are accounted for. Running service:check against multiple variants will result in status checks similar to:

multiple service checks

Adjusting validation parameters

Depending on the requirements of your application, you may want to configure the timeframe to validate operations against. You can do so by providing a validationPeriod flag to the CLI. The timeframe will always end at "now", and go back in time by the amount specified.

apollo service:check --validationPeriod=P2W

Note: Valid durations are represented in ISO 8601. It can also be provided as a number in seconds, i.e. 86400 for a single day.

Two other parameters for customizing the results of service:check are threshold values. For example, you may wish to drop support for an old version of an app in order to remove some deprecated fields. Using these parameters, you can decide what amount of breakage is acceptable before shipping any breaking changes.

  • queryCountThreshold - This flag will only validate the schema against operations that have been executed at least the specified number of times within the provided duration.
  • queryCountThresholdPercentage - Similar to queryCountThreshold, but expressed as a percentage of all operation volume.

Note: these flags are compatible with each other. In the case that both are provided, an operation must meet or exceed both thresholds.

Here's an example of how to run a service:check with custom thresholds set:

npx apollo service:check \
# Validate the schema against operations that have run in the last 5 days
--validationPeriod=P5D \
# Only validate against operations that have run at least 5 times during the 5 day duration
--queryCountThreshold=5 \
# Only validate against operations that account for at least 3% of total operation volume

If you have any requests for other filtering or threshold mechanisms, please get in touch with us on the apollo-tooling repository.

Edit on GitHub