Validating schema changes

Check if proposed schema changes are safe or breaking by comparing against live server traffic

As GraphQL scales within an organization, it becomes harder to evolve the schema while guaranteeing that no query or client will ever break from a change. Some organizations take the approach of just never making schema changes that might be breaking; however, managing an only-ever-growing schema is unnecessarily difficult for most teams. It can actually be very safe to evolve the schema through field removals and return type changes if you have the right tools to guarantee that no such change will ever break an active query.

As such, schema change validation is one of the cornerstones of the Apollo Platform and we’ve built a set of tools to make the workflow possible.

Note: Schema validation is an Apollo Platform feature available on the Team and Enterprise plans. To get started with the Apollo Platform, begin with the documentation. If you already have an Engine account, upgrade to a Team plan here.

How it works

The schema validation mechanism utilizes both Apollo’s schema registry and Apollo’s trace warehouse. The schema registry is used to identify a “schema diff” with changes between schema versions. The trace warehouse is used to identify which clients and which operations are using which fields in the schema in real time. We compare each change in the schema diff against the live usage data to determine if that change will be a “breaking change” for any clients.

Here’s how it works:

  1. You run apollo service:check locally or in CI. The proposed schema is sent to Engine’s schema registry.
  2. Engine creates a diff between the local schema and the most recently published schema in the registry.
  3. Engine fetches a list of all operations sent to your service in the last day (time window is configurable).
  4. Engine walks through the schema diff change-by-change and compares against the operation list to see if the changes will affect the behavior of any operations.
  5. Engine will return the schema diff and indicate any breaking changes found.
  6. The CLI will print the output of this check with a link to view more details in the Engine UI.

Breaking change detection

Engine’s cloud service uses an algorithm to detect breaking changes in a schema diff. It follows the following rules to determine which potentially breaking change types should actually fail the apollo service:check command and return a non-0 exit code.

  • Removals
    • FIELD_REMOVED A field referenced by at least one operation was removed
    • TYPE_REMOVED A referenced type(scalar, object) was removed
    • ARG_REMOVED A referenced argument was removed
    • TYPE_REMOVED_FROM_UNION A type in a union used by at least one operation was removed
    • INPUT_FIELD_REMOVED A field in an input type used by at least one operation was removed
    • VALUE_REMOVED_FROM_ENUM A value in an enum used by at least one operation was removed
    • TYPE_REMOVED_FROM_INTERFACE An object used by at least one operation was removed from an interface
  • Required arguments
    • REQUIRED_ARG_ADDED Non-nullable argument added to field used by at least one operation
    • NON_NULL_INPUT_FIELD_ADDED Non-null field added to an input object used by at least one operation
  • In-place updates
    • FIELD_CHANGED_TYPE Field used by at least one operation changed return type
    • INPUT_FIELD_CHANGED_TYPE Field in input object referenced in field argument used by at least one operation changed type
    • TYPE_CHANGED_KIND Type used by at least one operation changed, ex: scalar to object or enum to union
    • ARG_CHANGED_TYPE Argument used by at least one operation changed a type
  • Type extensions
    • TYPE_ADDED_TO_UNION New type added to a union used by at least one operation
    • TYPE_ADDED_TO_INTERFACE New interface added to an object used by at least one operation
  • Optional arguments
    • ARG_DEFAULT_VALUE_CHANGE Default value added or changed for argument on a field that is used by at least one operation

Note: This is not an exhaustive list of all possible schema change types, just breaking change types. Visit the graphql package’s repository for more details on schema changes types.

A failed apollo schema:check command will exit with a non-0 exit code and fail CI checks on purpose! There are actually many cases where it is safe to make a potentially breaking change, as long as the change is made intentionally.

Since breaking changes are detected using live traffic, your service will need active metrics for the change algorithm to detect failures. If there are no metrics associated with your service, all changes will be assigned the NOTICE severity as opposed to the FAILURE severity.

Validation output

Running a schema validation check is as simple as running apollo service:check on the command line from within a service repository that has been configured to be an Apollo project.

Note: Skip ahead to the setup section for details on how to configure your project for schema change validation.

Running the apollo service:check command will output the diff of all schema changes found and highlight changes determined to be breaking as FAILURE. Here’s an example:

~example$ apollo schema:check
  ✔ Loading Apollo Project
  ✔ Checking service for changes


Change   Code           Description
───────  ─────────────  ──────────────────────────────────
FAILURE  FIELD_REMOVED  `User.name` was removed
NOTICE   FIELD_ADDED    `User.friends` was added


View full details at: https://engine.apollographql.com/service/example-1234/checks?<DETAILS>

If there are any changes to the schema, FAILURE or NOTICE, a URL to Engine will be generated with details showing which clients and operations are affected by the changes specifically:

Schema checks page in the Engine UI

The Service Check page in Engine will have full details on the changes in the diff and which clients are affected by the changes, if any.

Note: If you set up your checks on GitHub, the “Details” link in your checks will take you to this special URL as well.

Set up schema validation

You will need to be actively sending traces to the Apollo trace warehouse and registering schemas to the Apollo schema registry to properly use schema validation. Follow these guides to set those up:

  1. Set up trace reporting to Apollo Engine (either through Apollo Server 2+ or the Engine proxy).
  2. Set up schema registration in your continuous delivery pipeline.

For the apollo schema:check command to be configured properly, you will also need:

  1. A .env file with an ENGINE_API_KEY.
  2. An apollo.config.js file with a service configured.

If you have set up schema registration, your project may already have its .env file and apollo.config.js file configured. Once you’ve got these set up, running your schema check is as simple as running:

apollo service:check

The command can be placed in any continuous integration pipeline. To surface results, apollo emits an exit code and integrates with GitHub statuses. The check command validates against traffic from the past day by default, but this time window can be configured to be a longer range.

Note: The Apollo CLI will be looking in your Apollo config for a location from which to fetch your local schema and using your ENGINE_API_KEY to authenticate its requests with the Engine service.

Run validation on each commit

We highly recommended that you add validation to your continuous integration workflow (e.g. Jenkins, CircleCI, etc.). In doing so, you can detect potential problems automatically and display the results of checks directly on pull requests.

Here’s a example of how to add a schema validation check to CircleCI:

version: 2

jobs:
  build:
    docker:
      - image: circleci/node:8

    steps:
      - checkout

      - run: npm install

      # Start the GraphQL server.  If a different command is used to
      # start the server, use it in place of `npm start` here.
      - run:
          name: Starting server
          command: npm start
          background: true

      # make sure the server has enough time to start up before running
      # commands against it
      - run: sleep 5

      # This will authenticate using the `ENGINE_API_KEY` environment
      # variable. Configure your endpoint's location in your Apollo config.
      - run: npx apollo service:check

Note: With a GitHub status check, to allow continuous integration to complete without failing early, ignore the exit code of the apollo service:check command. The exit code can be ignored by appending || echo 'validation failed' to the command call.

GitHub integration

GitHub Status View

Like most tools, schema validation is best used when it is integrated directly into the rest of your workflow. If you’re using GitHub, you can install the Apollo Engine GitHub app. This will enable Apollo’s systems to send a webhook back to your project on each apollo schema:check, providing built-in pass/fail status checks on your pull requests.

Go to https://github.com/apps/apollo-engine and click the Configure button to install the Apollo Engine integration on the appropriate GitHub profile or organization.

Multiple environments

Product cycles move fast, and it’s common for a schemas to be slightly different across environments as changes make their way through your system. To accommodate for this, schemas can be registered under specific “schema tags , and checks can be performed against specific “schema tags”.

schema registry allows each schema to be registered under a “schema tag”. Tags are mostly commonly used to represent environments, but can also be used to represent things like branches and future schemas. Passing the --tag flag to apollo service:check specifies which schema to compare against, such as prod or staging. It’s common to run checks against multiple different schema tags during continuous integration to ensure that all important deployments are accounted for. Checking multiple tags will result in check statuses similar to:

multiple service checks

Adjusting validation parameters

Depending on the requirements of your application, you may want to configure the timeframe to validate operations against. You can do so by providing a validationPeriod flag to the CLI. The timeframe will always end at “now”, and go back in time by the amount specified.

apollo service:check --validationPeriod=P2W

Valid durations are represented in ISO 8601. It can also be provided as a number in seconds, i.e. 86400 for a single day.

Two other parameters for customizing the results of service:check are threshold values. For example, you may wish to drop support for an old version of an app in order to remove some deprecated fields. Using these parameters, you can decide what amount of breakage is acceptable before shipping any breaking changes.

  • queryCountThreshold - This flag will only validate the schema against operations that have been executed at least the specified number of times within the provided duration.
  • queryCountThresholdPercentage - Similar to queryCountThreshold, but expressed as a percentage of all operation volume.

    Note: these flags are compatible with each other. In the case that both are provided, an operation must meet or exceed both thresholds.

If you have requests for other filtering or threshold mechanisms, we’d love to hear them! Please feel free to submit a feature request or PR to the apollo-tooling repo.

apollo service:check \
# Validate the schema against operations that have run in the last 5 days
--validationPeriod=P5D \
# Only validate against operations that have run at least 5 times during the 5 day duration
--queryCountThreshold=5 \
# Only validate against operations that account for at least 3% of total operation volume
--queryCountThresholdPercentage=3
Edit on GitHub
// search box