/
Launch Graph Manager

Schema validation

Ensure the safety of changes to your data graph


Certain changes to your GraphQL schema (such as removing a field or type) can potentially break one of your application's clients. Apollo Graph Manager provides schema validation to help you identify breaking changes before you make them, and to help you identify when a change won't break anything.

Schema validation is a paid feature that is available as part of Graph Manager Team and Enterprise plans.

Prerequisites

To enable schema validation for your data graph, first do the following:

  1. Set up trace reporting to Apollo Graph Manager from your GraphQL server.
  2. Set up schema registration in your continuous delivery pipeline.
  3. Install the Apollo CLI in your development environment if you haven't yet.

After you complete the above, you can perform a test validation by running the following command from your project's root directory:

# Omit 'npx' if you installed the Apollo CLI globally
$ npx apollo service:check

If everything is set up correctly, the command's output looks similar to the output shown in The validation response.

The validation process

When you validate your schema with the apollo service:check command:

  1. Graph Manager generates a diff between your local schema and the schema that you most recently registered.
  2. It uses this diff to determine whether the changes affect any operations that have been executed against your graph recently (you can customize this time window).
  3. Graph Manager returns the diff and a list of any breaking changes found.
  4. The Apollo CLI prints the result of the validation and returns a non-zero exit code if at least one breaking change is found.

The validation response

Running apollo service:check outputs the diff of all detected schema changes and highlights breaking changes:

$ npx apollo service:check
  ✔ Loading Apollo Project
  ✔ Validated local schema against tag prod on service engine
  ✔ Compared 8 schema changes against 110 operations over the last 24 hours
  ✖ Found 3 breaking changes and 5 compatible changes
    → breaking changes found

FAIL    ARG_REMOVED                `ServiceMutation.checkSchema` arg `gitContext` was removed
FAIL    FIELD_REMOVED              `Schema.fieldCount` was removed
FAIL    FIELD_REMOVED              `Schema.typeCount` was removed

PASS    FIELD_ADDED                `SchemaTag.schemaRepoID` was added
PASS    FIELD_CHANGED_TYPE         `ServiceMutation.uploadPartialSchema` changed type from `UploadPartialSchemaResponse!` to `CompositionResult!`
PASS    FIELD_DEPRECATION_REMOVED  `IntrospectionSchema.fieldCount` is no longer deprecated
PASS    FIELD_DEPRECATION_REMOVED  `IntrospectionSchema.typeCount` is no longer deprecated
PASS    TYPE_REMOVED               `UploadPartialSchemaResponse` removed

View full details at: https://engine.apollographql.com/service/example-1234/check/<DETAILS>

Each change to the schema is labeled with either PASS or FAIL.

Note: Because breaking changes are detected by analyzing recent operations, your GraphQL server must be pushing analytics to Graph Manager for schema validation to work. If there are no operation metrics to compare against, all schema changes are labeled with PASS.

The CLI output also includes a Graph Manager URL that provides full details on the changes and their impact on existing clients and operations:

Service check page in the Graph Manager UI

Note: If you've installed schema validation checks on your GitHub PRs, the "Details" link in your GitHub checks will take you to the same details link in this output.

Validating a federated schema

If your data graph uses Apollo Federation, the schema validation process differs slightly.

For details, see Validating changes to the graph.

Using with continuous integration

Schema validation is especially useful when you add it to your continuous integration pipeline (such as Jenkins or CircleCI). By doing so, you can obtain validation results and display them directly on your team's pull requests.

We recommend defining a separate CI job for each variant of your schema (production, staging, etc.) you want to validate your changes against. The apollo service:check command returns a non-zero exit code when it detects a breaking change, meaning the job will fail when validation fails.

Example configuration

The following config defines a schema validation job for a CircleCI pipeline. Your config's syntax varies depending on your CI tool, but the job's steps are the same.

config.yml
version: 2

jobs:

  # ...other jobs...

  # Define a separate job for each environment you validate against.
  validate_against_staging:
    docker:
      - image: circleci/node:8

    steps:
      - checkout

      - run: npm install

      # Start the GraphQL server. If a different command is used to
      # start the server, use it in place of `npm start` here.
      - run:
          name: Starting server
          command: npm start
          background: true

      # Make sure the server has enough time to start up before running
      # commands against it.
      - run: sleep 5

      # This command authenticates using the `ENGINE_API_KEY` environment
      # variable. Specify your GraphQL endpoint's URL in your Apollo config.
      - run: npx apollo service:check --tag=staging

Integrating with GitHub

If you're using GitHub, you can install the Apollo Engine GitHub app. This app enables Graph Manager to send a webhook back to your GitHub project on each call to apollo service:check, providing built-in pass/fail status checks on your pull requests:

GitHub Status View

Integrating with other version control services

If you're using GitHub Enterprise, Bitbucket, or another version control service, we recommend setting up your CI tool to post a comment on each pull request with the results of schema validation. By surfacing schema diffs and breaking changes directly in your PR, you can avoid having to search your CI logs to check why validation failed.

The apollo service:check supports a --markdown flag, which outputs the results of schema validation in markdown format. You can pipe this output directly into a comment to your source control tool, like in this example of posting a comment with the results of schema validation to GitHub.

The output of apollo service:check --markdown looks like this:

### Apollo Service Check

🔄 Validated your local schema against schema tag `staging` on service `engine`.
🔢 Compared **18 schema changes** against **100 operations** seen over the **last 24 hours**.
❌ Found **7 breaking changes** that would affect **3 operations** across **2 clients**

🔗 [View your service check details](https://engine.apollographql.com/service/engine/checks?...).

Validating against multiple environments

You might want to validate your schema changes against multiple environments, such as production, staging, and beta. Each of these environments might have a slightly different schema to support features that are experimental or in active development. In Apollo Graph Manager, these schemas are represented as variants of a single data graph.

You can validate your schema against a particular variant by including the --tag flag in your call to apollo service:check, like so:

$ apollo service:check --tag=staging

If you want to validate your schema against multiple variants, call apollo service:check once for each variant. Running service:check against multiple variants results in status checks similar to the following:

multiple service checks

Customizing validation

You can customize the logic that schema validation uses to classify a schema change as either breaking or non-breaking.

For example, you can provide the --validationPeriod option to the apollo service:check command to specify how far back in time Graph Manager should look when determining the compatibility of past operations with the changes to your schema (by default, the command uses the last 7 days of operations).

This command checks your schema changes against the past two weeks of operations:

apollo service:check --validationPeriod=P2W

Valid durations are represented in ISO 8601 format. You can also provide a number in seconds (e.g., 86400 for a single day).

Note: If you specify a validationPeriod that exceeds your organization's operation retention period, the service:check command fails with an error.

Threshold values

You can also customize the behavior of schema validation by specifying threshold values to the service:check command. These values tell schema validation to ignore all historical operations that do not exceed the threshold(s) you specify.

For example, you might want to discontinue support for an old version of your client that uses deprecated fields. You can set threshold values to determine when an acceptably small number of users are using the outdated client, thus reducing the impact of discontinuing support.

Provide threshold values with the following flags:

  • --queryCountThreshold - Validate your schema only against operations that have been executed at least the specified number of times within the specified duration.
  • --queryCountThresholdPercentage - Validate your schema only against operations that account for at least the specified percentage of all operations against the graph within the specified duration. For example, specify 3 to set this threshold to 3%.

Note: You can provide values for both of these flags. If you do, an operation must meet or exceed both thresholds for schema validation to include it.

Here's an example of running service:check with threshold values:

npx apollo service:check \
# Validate the schema against operations that have run in the last 5 days
--validationPeriod=P5D \
# Only validate against operations that have run at least 5 times during the 5 day duration
--queryCountThreshold=5 \
# Only validate against operations that account for at least 3% of total operation volume
--queryCountThresholdPercentage=3

If you have any requests for other filtering or threshold mechanisms, please get in touch with us on the apollo-tooling repository.

Types of schema changes

Not every change to a schema is a potentially breaking change. Additive changes (such as adding a field to a type) are typically safe and do not affect active clients. Deletions and modifications (such as removing a field or changing a return type), however, can break clients that use affected types and fields.

Potentially breaking changes

Removals

These changes remove a schema element. If a removed element is actively being used by an operation, your data graph will return an error to affected clients.

  • FIELD_REMOVED: Field used by at least one operation was removed
  • TYPE_REMOVED: Type(scalar, object) used by at least one operation was removed
  • ARG_REMOVED: Argument was removed from a field used by at least one operation
  • TYPE_REMOVED_FROM_UNION: Type was removed from a union used by at least one operation
  • INPUT_FIELD_REMOVED: Field removed from an input type referenced by an argument on a field used by at least one operation
  • VALUE_REMOVED_FROM_ENUM: A value removed from an enum used by at least one operation
  • TYPE_REMOVED_FROM_INTERFACE: An object removed from an interface used by at least one operation

Addition of required arguments

These changes add a required input to a schema element. If an operation is actively using an element of your graph and doesn't add the new required input argument, the data graph will return an error to affected clients.

  • REQUIRED_ARG_ADDED: Non-nullable argument added to field used by at least one operation
  • NON_NULL_INPUT_FIELD_ADDED: Non-null field added to an input object used by at least one operation

In-place updates

These changes update an existing schema element. If an operation is actively using an element that is updated, the operation might start receiving an error from your data graph. It also might receive an unexpected result.

Note: In some cases, in-place updates are compatible with affected clients at runtime (such as a type rename or a conversion from an object to an interface that uses the same fields). However, schema validation still marks these as breaking changes, because validation does not have enough information to ensure that they are safe.

  • FIELD_CHANGED_TYPE: Field used by at least one operation changed return type
  • INPUT_FIELD_CHANGED_TYPE: Field in input object changed type and is referenced by argument on field used by at least one operation
  • TYPE_CHANGED_KIND: Type used by at least one operation changed, ex: scalar to object or enum to union
  • ARG_CHANGED_TYPE: Argument changed type on field used by at least one operation

Default arguments

These changes update the default value for an argument. If an operation is using an element of your graph and does not specify a value for this argument, the operation might get an unexpected result when the schema is updated if it was relying on the original default value.

  • ARG_DEFAULT_VALUE_CHANGE: Default value added or changed for argument on a field used by at least one operation

Non-breaking changes

These changes are detected by the apollo service:check command, but they are "safe." They will never affect the behavior of any existing clients if deployed.

  • Optional arguments
    • OPTIONAL_ARG_ADDED Nullable argument added to a field
    • NULLABLE_FIELD_ADDED_TO_INPUT_OBJECT Nullable field added to an input object
  • Additions
    • FIELD_ADDED Field added to a type
    • TYPE_ADDED Type added to the schema
    • VALUE_ADDED_TO_ENUM Value added to an enum. If clients contain a switch case on the enum and do not include the `default`, this change could cause unexpected behavior
    • TYPE_ADDED_TO_UNION Type added to a union used by at least one operation
    • TYPE_ADDED_TO_INTERFACE Interface added to an object used by at least one operation
  • Deprecations
    • FIELD_DEPRECATED Field deprecated
    • FIELD_DEPRECATION_REMOVED Field no longer deprecated
    • FIELD_DEPRECATED_REASON_CHANGE Reason for deprecation changed
    • ENUM_DEPRECATED Enum deprecated
    • ENUM_DEPRECATION_REMOVED Enum no longer deprecated
    • ENUM_DEPRECATED_REASON_CHANGE Reason for enum deprecation changed
Edit on GitHub