Validating schema changes
As GraphQL scales within an organization, it becomes harder to evolve the schema while guaranteeing that no query or client will ever break from a change. Some organizations take the approach of just never making schema changes that might be breaking; however, managing an only-ever-growing schema is unnecessarily difficult for most teams. It can actually be very safe to evolve the schema through field removals and return type changes if you have the right tools to guarantee that no such change will ever break an active query.
As such, schema change validation is one of the cornerstones of the Apollo Platform and we’ve built a set of tools to make the workflow possible.
Note: Schema validation is an Apollo Platform feature available on the Team and Enterprise plans. To get started with the Apollo Platform, begin with the documentation. If you already have an Engine account, upgrade to a Team plan here.
How it works
The schema validation mechanism utilizes both Apollo’s schema registry and Apollo’s trace warehouse. The schema registry is used to identify a “schema diff” with changes between schema versions. The trace warehouse is used to identify which clients and which operations are using which fields in the schema in real time. We compare each change in the schema diff against the live usage data to determine if that change will be a “breaking change” for any clients.
Here’s how it works:
- You run
apollo service:checklocally or in CI. The proposed schema is sent to Engine’s schema registry.
- Engine creates a diff between the local schema and the most recently published schema in the registry.
- Engine fetches a list of all operations sent to your service in the last day (time window is configurable).
- Engine walks through the schema diff change-by-change and compares against the operation list to see if the changes will affect the behavior of any operations.
- Engine will return the schema diff and indicate any breaking changes found.
- The CLI will print the output of this check with a link to view more details in the Engine UI.
Breaking change detection
Engine’s cloud service uses an algorithm to detect breaking changes in a schema diff. It follows the following rules to determine which potentially breaking change types should actually fail the
apollo service:check command and return a non-0 exit code.
FIELD_REMOVEDA field referenced by at least one operation was removed
TYPE_REMOVEDA referenced type(scalar, object) was removed
ARG_REMOVEDA referenced argument was removed
TYPE_REMOVED_FROM_UNIONA type in a union used by at least one operation was removed
INPUT_FIELD_REMOVEDA field in an input type used by at least one operation was removed
VALUE_REMOVED_FROM_ENUMA value in an enum used by at least one operation was removed
TYPE_REMOVED_FROM_INTERFACEAn object used by at least one operation was removed from an interface
- Required arguments
REQUIRED_ARG_ADDEDNon-nullable argument added to field used by at least one operation
NON_NULL_INPUT_FIELD_ADDEDNon-null field added to an input object used by at least one operation
- In-place updates
FIELD_CHANGED_TYPEField used by at least one operation changed return type
INPUT_FIELD_CHANGED_TYPEField in input object referenced in field argument used by at least one operation changed type
TYPE_CHANGED_KINDType used by at least one operation changed, ex: scalar to object or enum to union
ARG_CHANGED_TYPEArgument used by at least one operation changed a type
- Type extensions
TYPE_ADDED_TO_UNIONNew type added to a union used by at least one operation
TYPE_ADDED_TO_INTERFACENew interface added to an object used by at least one operation
- Optional arguments
ARG_DEFAULT_VALUE_CHANGEDefault value added or changed for argument on a field that is used by at least one operation
Note: This is not an exhaustive list of all possible schema change types, just breaking change types. Visit the
graphqlpackage’s repository for more details on schema changes types.
apollo schema:check command will exit with a non-0 exit code and fail CI checks on purpose! There are actually many cases where it is safe to make a potentially breaking change, as long as the change is made intentionally.
Since breaking changes are detected using live traffic, your service will need active metrics for the change algorithm to detect failures. If there are no metrics associated with your service, all changes will be assigned the
NOTICE severity as opposed to the
Running a schema validation check is as simple as running
apollo service:check on the command line from within a service repository that has been configured to be an Apollo project.
Note: Skip ahead to the setup section for details on how to configure your project for schema change validation.
apollo service:check command will output the diff of all schema changes found and highlight changes determined to be breaking as
FAILURE. Here’s an example:
~example$ apollo schema:check ✔ Loading Apollo Project ✔ Checking service for changes Change Code Description ─────── ───────────── ────────────────────────────────── FAILURE FIELD_REMOVED `User.name` was removed NOTICE FIELD_ADDED `User.friends` was added View full details at: https://engine.apollographql.com/service/example-1234/checks?<DETAILS>
If there are any changes to the schema,
NOTICE, a URL to Engine will be generated with details showing which clients and operations are affected by the changes specifically:
The Service Check page in Engine will have full details on the changes in the diff and which clients are affected by the changes, if any.
Note: If you set up your checks on GitHub, the “Details” link in your checks will take you to this special URL as well.
Set up schema validation
You will need to be actively sending traces to the Apollo trace warehouse and registering schemas to the Apollo schema registry to properly use schema validation. Follow these guides to set those up:
- Set up trace reporting to Apollo Engine (either through Apollo Server 2+ or the Engine proxy).
- Set up schema registration in your continuous delivery pipeline.
apollo schema:check command to be configured properly, you will also need:
If you have set up schema registration, your project may already have its
.env file and
apollo.config.js file configured. Once you’ve got these set up, running your schema check is as simple as running:
The command can be placed in any continuous integration pipeline. To surface results,
apollo emits an exit code and integrates with GitHub statuses. The check command validates against traffic from the past day by default, but this time window can be configured to be a longer range.
Note: The Apollo CLI will be looking in your Apollo config for a location from which to fetch your local schema and using your ENGINE_API_KEY to authenticate its requests with the Engine service.
Run validation on each commit
We highly recommended that you add validation to your continuous integration workflow (e.g. Jenkins, CircleCI, etc.). In doing so, you can detect potential problems automatically and display the results of checks directly on pull requests.
Here’s a example of how to add a schema validation check to CircleCI:
version: 2 jobs: build: docker: - image: circleci/node:8 steps: - checkout - run: npm install # Start the GraphQL server. If a different command is used to # start the server, use it in place of `npm start` here. - run: name: Starting server command: npm start background: true # make sure the server has enough time to start up before running # commands against it - run: sleep 5 # This will authenticate using the `ENGINE_API_KEY` environment # variable. Configure your endpoint's location in your Apollo config. - run: npx apollo service:check
Note: With a GitHub status check, to allow continuous integration to complete without failing early, ignore the exit code of the
apollo service:checkcommand. The exit code can be ignored by appending
|| echo 'validation failed'to the command call.
Like most tools, schema validation is best used when it is integrated directly into the rest of your workflow. If you’re using GitHub, you can install the Apollo Engine GitHub app. This will enable Apollo’s systems to send a webhook back to your project on each
apollo schema:check, providing built-in pass/fail status checks on your pull requests.
Go to https://github.com/apps/apollo-engine and click the
Configure button to install the Apollo Engine integration on the appropriate GitHub profile or organization.
Product cycles move fast, and it’s common for a schemas to be slightly different across environments as changes make their way through your system. To accommodate for this, schemas can be registered under specific “schema tags , and checks can be performed against specific “schema tags”.
schema registry allows each schema to be registered under a “schema tag”. Tags are mostly commonly used to represent environments, but can also be used to represent things like branches and future schemas. Passing the
--tag flag to
apollo service:check specifies which schema to compare against, such as
staging. It’s common to run checks against multiple different schema tags during continuous integration to ensure that all important deployments are accounted for. Checking multiple tags will result in check statuses similar to:
Adjusting validation parameters
Depending on the requirements of your application, you may want to configure the timeframe to validate operations against. You can do so by providing a
validationPeriod flag to the CLI. The timeframe will always end at “now”, and go back in time by the amount specified.
apollo service:check --validationPeriod=P2W
Valid durations are represented in ISO 8601. It can also be provided as a number in seconds, i.e. 86400 for a single day.
Two other parameters for customizing the results of
service:check are threshold values. For example, you may wish to drop support for an old version of an app in order to remove some deprecated fields. Using these parameters, you can decide what amount of breakage is acceptable before shipping any breaking changes.
queryCountThreshold- This flag will only validate the schema against operations that have been executed at least the specified number of times within the provided duration.
queryCountThresholdPercentage- Similar to
queryCountThreshold, but expressed as a percentage of all operation volume.
Note: these flags are compatible with each other. In the case that both are provided, an operation must meet or exceed both thresholds.
apollo service:check \ # Validate the schema against operations that have run in the last 5 days --validationPeriod=P5D \ # Only validate against operations that have run at least 5 times during the 5 day duration --queryCountThreshold=5 \ # Only validate against operations that account for at least 3% of total operation volume --queryCountThresholdPercentage=3