Validating schema changes
Check if schema changes are safe or breaking by comparing against live server traffic
There are many types of schema changes that can be potentially breaking to clients, like removing a field, if made without special consideration. For safety, some organizations take the approach of never making these types of changes, but this leads to an ever-growing schema and reduced API flexibility over time. In reality, making these types of changes to a schema can be very safe as long as you have tools in place to ensure that no queries are broken in the process.
Apollo provides a tool to protect for exactly this scenario called schema validation.
Schema validation is run through the Apollo CLI by executing the
apollo service:check command. Apollo will generate a diff between your local schema and your most recently registered schema, then validate that the changes are safe by checking if any queries actively running against your graph will be affected.
Here's how it works:
- You run
apollo service:checklocally or in CI. The proposed schema is sent to Graph Manager's schema registry.
- Graph Manager creates a diff between the local schema and the most recently published schema in the registry.
- Graph Manager fetches a list of all operations sent to your graph in the last day (time window is configurable).
- Graph Manager walks through the schema diff change-by-change and compares against the operation list to see if the changes will affect the behavior of any operations.
- Graph Manager returns the schema diff and indicates any breaking changes found.
- The CLI prints the output of this check with a link to view more details in the Graph Manager UI.
Not all schema changes are potentially breaking. Some changes, like adding a field, will always be safe and never cause unexpected behavior for active clients. Other changes, like removing a field or changing a return type, can potentially affect the behavior of clients making queries that use those fields. These are what we consider potentially breaking changes.
If schema validation detects that a proposed schema has a potentially breaking change, the
apollo service:check command will return a non-0 exit code. Apollo schema validation will detect breaking changes according to the following rules:
Each of these change types removes a schema element. If an element of your graph is being actively used by an operation and it is removed, your GraphQL layer will start returning errors to the dependent operations.
FIELD_REMOVED: Field used by at least one operation was removed
TYPE_REMOVED: Type(scalar, object) used by at least one operation was removed
ARG_REMOVED: Argument was removed from a field used by at least one operation
TYPE_REMOVED_FROM_UNION: Type was removed from a union used by at least one operation
INPUT_FIELD_REMOVED: Field removed from an input type referenced by an argument on a field used by at least one operation
VALUE_REMOVED_FROM_ENUM: A value removed from an enum used by at least one operation
TYPE_REMOVED_FROM_INTERFACE: An object removed from an interface used by at least one operation
Each of these changes adds a required input to a schema element. If an operation is actively using an element of your graph and doesn't update itself to add the new required input argument, the GraphQL layer will start returning an error to the operation.
REQUIRED_ARG_ADDED: Non-nullable argument added to field used by at least one operation
NON_NULL_INPUT_FIELD_ADDED: Non-null field added to an input object used by at least one operation
Each of these changes updates an existing schema element. If an operation is activley using an element of your graph and that element is updated, the operation could start receiving an error from the GraphQL layer or, in some cases, an unexpected result.
Note: In some cases, these changes are compatible with the client at runtime, such as a type rename or an object to interface conversion with the same fields. Schema validation still marks these breaking changes because validation does not have enough information to ensure safety and these changes deserve extra scrutiny, such as their impact on type generation.
FIELD_CHANGED_TYPE: Field used by at least one operation changed return type
INPUT_FIELD_CHANGED_TYPE: Field in input object changed type and is referenced by argument on field used by at least one operation
TYPE_CHANGED_KIND: Type used by at least one operation changed, ex: scalar to object or enum to union
ARG_CHANGED_TYPE: Argument changed type on field used by at least one operation
These changes update the default value for an argument. If an operation is using an element of your graph and does not specify a value for this argument, the operation could experience unexpected results when the schema is updated if it was relying on the original default value.
ARG_DEFAULT_VALUE_CHANGE: Default value added or changed for argument on a field used by at least one operation
These are change types detected ny the
apollo service:check command, but they are "safe" and will always be compatible with all exisitng client usage of the graph. They will not affect the behavior of any clients if deployed.
- Optional arguments
OPTIONAL_ARG_ADDEDNullable argument added to a field
NULLABLE_FIELD_ADDED_TO_INPUT_OBJECTNullable field added to an input object
FIELD_ADDEDField added to a type
TYPE_ADDEDType added to the schema
VALUE_ADDED_TO_ENUMValue added to an enum. If clients contain a switch case on the enum and do not include the `default`, this change could cause unexpected behavior
TYPE_ADDED_TO_UNIONType added to a union used by at least one operation
TYPE_ADDED_TO_INTERFACEInterface added to an object used by at least one operation
FIELD_DEPRECATION_REMOVEDField no longer deprecated
FIELD_DEPRECATED_REASON_CHANGEReason for deprecation changed
ENUM_DEPRECATION_REMOVEDEnum no longer deprecated
ENUM_DEPRECATED_REASON_CHANGEReason for enum deprecation changed
Running a schema validation check is as simple as running
apollo service:check on the command line from within a service repository that is configured to be an Apollo project.
apollo service:check will output the diff of all schema changes found and highlight changes determined to be breaking. Here's an example:
$ npx apollo service:check --tag=prod ✔ Loading Apollo Project ✔ Validated local schema against tag prod on service engine ✔ Compared 8 schema changes against 110 operations over the last 24 hours ✖ Found 3 breaking changes and 5 compatible changes → breaking changes found FAIL ARG_REMOVED `ServiceMutation.checkSchema` arg `gitContext` was removed FAIL FIELD_REMOVED `Schema.fieldCount` was removed FAIL FIELD_REMOVED `Schema.typeCount` was removed PASS FIELD_ADDED `SchemaTag.schemaRepoID` was added PASS FIELD_CHANGED_TYPE `ServiceMutation.uploadPartialSchema` changed type from `UploadPartialSchemaResponse!` to `CompositionResult!` PASS FIELD_DEPRECATION_REMOVED `IntrospectionSchema.fieldCount` is no longer deprecated PASS FIELD_DEPRECATION_REMOVED `IntrospectionSchema.typeCount` is no longer deprecated PASS TYPE_REMOVED `UploadPartialSchemaResponse` removed View full details at: https://engine.apollographql.com/service/example-1234/check/<DETAILS>
Each change to the schema will be labeled with
FAIL and a URL with full details on the changes and their impact on clients and operations will be generated. Following the URL will take you to Graph Manager:
Note: If you have installed schema validation checks on your GitHub PRs, the "Details" link in your GitHub checks will take you to the same details link in this output.
apollo service:check command will exit with a non-0 exit code and fail CI checks. There are many cases where it is safe to make a potentially breaking change, as long as the change is made intentionally with an understanding of its impact.
Since breaking changes are detected using live traffic, your service will need active metrics for the change algorithm to detect failures. If there are no metrics associated with your service, all changes will be labeled as a
PASS as opposed to a
To set up schema validation, you will need to be both actively sending traces and registering schemas to Apollo:
- Set up trace reporting to Apollo Graph Manager
- Set up schema registration in your continuous delivery pipeline
Then, you will need to configure your project for the
apollo service:check command:
Note: If you have set up one of Apollo's workflows previously, your project may already have its
Once you've got these set up, running your schema check is as simple as running:
$ npm install apollo $ npx apollo service:check
The command can be placed in any continuous integration pipeline. To surface results,
apollo emits an exit code and integrates with GitHub statuses. The time window of live traffic that the check command validates against can be configured to any range within your data retention window.
Note: The Apollo CLI will be looking in your Apollo config for a location from which to fetch your local schema and using your ENGINE_API_KEY to authenticate its requests with the Graph Manager service.
We highly recommended that you add validation to your continuous integration workflow (e.g. Jenkins, CircleCI, etc.). In doing so, you can detect potential problems automatically and display the results of checks directly on pull requests.
Here's a example of how to add a schema validation check to CircleCI:
version: 2 jobs: build: docker: - image: circleci/node:8 steps: - checkout - run: npm install # Start the GraphQL server. If a different command is used to # start the server, use it in place of `npm start` here. - run: name: Starting server command: npm start background: true # make sure the server has enough time to start up before running # commands against it - run: sleep 5 # This will authenticate using the `ENGINE_API_KEY` environment # variable. Configure your endpoint's location in your Apollo config. - run: npx apollo service:check
Note: If you're using GitHub status checks, we recommend ignoring the exit code of the
apollo service:checkcommand so your continuous integration can complete without failing early. This can be done by appending
|| echo 'validation failed'to the command call.
Like most tools, schema validation is best used when it is integrated directly into the rest of your workflow. If you're using GitHub, you can install the Apollo Engine GitHub app. This will enable Apollo's systems to send a webhook back to your project on each
apollo service:check, providing built-in pass/fail status checks on your pull requests.
To install the Apollo Engine integration on GitHub, go to https://github.com/apps/apollo-engine, click the
Configure button, and select the appropriate GitHub profile or organization.
For teams using GitHub Enterprise, Bitbucket, and other source control tools, we recommend setting up your CI to post a comment on your PRs with the results of schema validation. Surfacing schema diffs and breaking changes directly in your PR will speed up your review workflow by saving you the time of searching your CI logs to check why validation didn't pass.
The CLI supports passing a
--markdown flag to
apollo service:check, which outputs the results of schema validation in a markdown format specifically. This markdown can be piped directly into a comment to your source control tool, like in this example of posting a comment with the results of schema validation to GitHub.
The output of
apollo service:check --markdown looks like this:
### Apollo Service Check 🔄 Validated your local schema against schema tag `staging` on service `engine`. 🔢 Compared **18 schema changes** against **100 operations** seen over the **last 24 hours**. ❌ Found **7 breaking changes** that would affect **3 operations** across **2 clients** 🔗 [View your service check details](https://engine.apollographql.com/service/engine/checks?...).
Product cycles move fast and it's common for schemas to be slightly different across environments as changes make their way through your system. To support this, schemas pushed to the registry can be associated with specific variants of your graph (also referred to tags).
Variants mostly commonly represent environments and can also indicate branches or future schemas. Passing the
--tag=<VARIANT> flag to
apollo service:check specifies which schema variant to compara against, such as
staging. It's common to run checks against multple different graph variants in the same continuous integration pipeline to ensure that all important deployments are accounted for. Running
service:check against multiple variants will result in status checks similar to:
Depending on the requirements of your application, you may want to configure the timeframe to validate operations against. You can do so by providing a
validationPeriod flag to the CLI. The timeframe will always end at "now", and go back in time by the amount specified.
apollo service:check --validationPeriod=P2W
Note: Valid durations are represented in ISO 8601. It can also be provided as a number in seconds, i.e. 86400 for a single day.
Two other parameters for customizing the results of
service:check are threshold values. For example, you may wish to drop support for an old version of an app in order to remove some deprecated fields. Using these parameters, you can decide what amount of breakage is acceptable before shipping any breaking changes.
queryCountThreshold- This flag will only validate the schema against operations that have been executed at least the specified number of times within the provided duration.
queryCountThresholdPercentage- Similar to
queryCountThreshold, but expressed as a percentage of all operation volume.
Note: these flags are compatible with each other. In the case that both are provided, an operation must meet or exceed both thresholds.
Here's an example of how to run a
service:check with custom thresholds set:
npx apollo service:check \ # Validate the schema against operations that have run in the last 5 days --validationPeriod=P5D \ # Only validate against operations that have run at least 5 times during the 5 day duration --queryCountThreshold=5 \ # Only validate against operations that account for at least 3% of total operation volume --queryCountThresholdPercentage=3
If you have any requests for other filtering or threshold mechanisms, please get in touch with us on the apollo-tooling repository.