apollo-smith

apollo-smith

A test case generator for GraphQL language.

About

The goal of apollo-smith is to generate valid GraphQL documents by sampling from all available possibilities of GraphQL grammar.

We've written apollo-smith to use in fuzzing, but you may wish to use it for anything that requires GraphQL document generation.

apollo-smith is inspired by bytecodealliance's wasm-smith crate, and the article written by Nick Fitzgerald on writing test case generators in Rust.

This is still a work in progress, for outstanding issues, checkout out the apollo-smith label in our issue tracker.

Rust versions

apollo-smith is tested on the latest stable version of Rust. Older version may or may not be compatible.

Using apollo-smith with cargo fuzz

Define a new target with cargo fuzz,

$ cargo fuzz add my_apollo_smith_fuzz_target

and add apollo-smith to your Cargo.toml:

## fuzz/Cargo.toml

[dependencies]
apollo-smith = "0.16.0"

It can then be used in a fuzz_target along with the arbitrary crate,

// fuzz/fuzz_targets/my_apollo_smith_fuzz_target.rs

#![no_main]

use libfuzzer_sys::fuzz_target;
use arbitrary::Unstructured;
use apollo_smith::DocumentBuilder;

fuzz_target!(|input: &[u8]| {
    let mut u = Unstructured::new(input);
    let document = DocumentBuilder::new(&mut u).build()?;
    let document_str = String::from(document);


});

and fuzzed with the following command:

$ cargo +nightly fuzz run my_apollo_smith_fuzz_target

Using apollo-smith with apollo-parser

You can use apollo-parser to generate valid operations in apollo-smith.

use std::fs;

use apollo_parser::Parser;
use apollo_smith::{Document, DocumentBuilder};

use libfuzzer_sys::arbitrary::{Result, Unstructured};

/// This generate an arbitrary valid GraphQL operation
pub fn generate_valid_operation(input: &[u8]) -> Result<String> {
    let parser = Parser::new(&fs::read_to_string("supergraph.graphql").expect("cannot read file"));

    let tree = parser.parse();
    if tree.errors().next().is_some() {
        panic!("cannot parse the graphql file");
    }

    let mut u = Unstructured::new(input);

    // Convert `apollo_parser::Document` into `apollo_smith::Document`.
    let apollo_smith_doc = Document::try_from(tree.document()).unwrap();

    // Create a `DocumentBuilder` given an existing document to match a schema.
    let mut gql_doc = DocumentBuilder::with_document(&mut u, apollo_smith_doc)?;
    let operation_def = gql_doc.operation_definition()?.unwrap();

    Ok(operation_def.into())
}

Generating responses using apollo-smith with apollo-compiler

If you have a GraphQL operation in the form of an ExecutableDocument and its accompanying Schema, you can generate a response matching the shape of the operation with apollo_smith::ResponseBuilder.

ResponseBuilder is generic over its randomness source via the RandomProvider trait. This allows it to be used with arbitrary::Unstructured for fuzz testing, with RandProvider for standard random generation, or with any custom implementation.

Using Unstructured (for fuzz testing)

use apollo_compiler::validation::Valid;
use apollo_compiler::ExecutableDocument;
use apollo_compiler::Schema;
use apollo_smith::{ResponseBuilder, ResponseError};
use arbitrary::Unstructured;
use rand::RngExt as _;
use serde_json_bytes::Value;

pub fn generate_valid_response(
    doc: &Valid<ExecutableDocument>,
    schema: &Valid<Schema>,
) -> Result<Value, ResponseError> {
    let mut buf = [0u8; 2048];
    rand::rng().fill(&mut buf);
    let mut u = Unstructured::new(&buf);

    ResponseBuilder::new(&mut u, doc, schema).build()
}

Using RandProvider

Use RandProvider to wrap any rand::Rng:

use apollo_smith::{RandProvider, ResponseBuilder};

let mut rng = RandProvider(rand::rng());
let response = ResponseBuilder::new(&mut rng, &doc, &schema)
    .with_min_list_size(1)
    .with_max_list_size(5)
    .with_null_ratio(1, 4)
    .build()?;

Configuring generation per type

Use with_generator to register a [Generator] for any named GraphQL type — scalars, objects, interfaces, or unions. The same trait powers both leaf and composite generation; the fields argument is empty for scalar types and contains the requested selection (flattened across fragments and grouped by response key) for composite types.

To swap one of the built-in scalar defaults:

use apollo_smith::{ResponseBuilder, StringGenerator};
use apollo_compiler::Name;

let response = ResponseBuilder::new(&mut rng, &doc, &schema)
    .with_generator(
        Name::new_unchecked("ID".into()),
        StringGenerator { min_len: 8, max_len: 8 },
    )
    .build()?;

Or with completely custom generation logic:

use apollo_smith::{Generator, Generators, RandomProvider, ResponseBuilder, ResponseError};
use apollo_compiler::executable::Field;
use apollo_compiler::{Name, Node};
use indexmap::IndexMap;
use serde_json_bytes::Value;

struct IncrementingGenerator {
    id: i32,
}

impl<R: RandomProvider> Generator<R> for IncrementingGenerator {
    fn generate(
        &mut self,
        _rng: &mut R,
        _generators: &mut Generators<R>,
        _fields: &IndexMap<String, Vec<Node<Field>>>,
    ) -> Result<Value, ResponseError> {
        self.id += 1;
        Ok(Value::Number(self.id.into()))
    }
}

let response = ResponseBuilder::new(&mut rng, &doc, &schema)
    .with_generator(
        Name::new_unchecked("ID".into()),
        IncrementingGenerator { id: 0 },
    )
    .build();

For composite types, the generator's return value is used as-is — the builder does not recurse into it. The generator receives the requested fields already flattened across fragments and grouped by response key, so it can return only what the caller asked for:

use apollo_smith::{Generator, Generators, RandomProvider, ResponseBuilder, ResponseError};
use apollo_compiler::executable::Field;
use apollo_compiler::{Name, Node};
use indexmap::IndexMap;
use serde_json_bytes::{Map, Value};

/// Generator for the federation `_Service` type that returns the real schema SDL.
struct ServiceGenerator {
    sdl: String,
}

impl<R: RandomProvider> Generator<R> for ServiceGenerator {
    fn generate(
        &mut self,
        _rng: &mut R,
        _generators: &mut Generators<R>,
        fields: &IndexMap<String, Vec<Node<Field>>>,
    ) -> Result<Value, ResponseError> {
        let mut obj = Map::new();
        for (response_key, group) in fields {
            // The first field in the group is representative — multiple entries only
            // appear when the same response key shows up in several fragments.
            if group[0].name == "sdl" {
                obj.insert(response_key.clone(), Value::String(self.sdl.clone().into()));
            }
        }
        Ok(Value::Object(obj))
    }
}

let response = ResponseBuilder::new(&mut rng, &doc, &schema)
    .with_generator(
        Name::new_unchecked("_Service"),
        ServiceGenerator { sdl },
    )
    .build()?;

Accessing the default generators

The generators argument passed to every Generator::generate call is the same registry the builder is using. It starts from Generators::default(), which pre-registers a generator for each of the five standard GraphQL scalars:

Type	Generator	Default range
Boolean	BooleanGenerator	true or false
Int	IntGenerator	0..=100
Float	FloatGenerator	-1.0..=1.0
String	StringGenerator	1–10 alphanumeric characters
ID	IdGenerator	0..=100, serialized as a string

Each per-type struct is public, so a custom composite generator can:

• delegate a single field back to the registry via generators.generate_scalar(type_name, rng) — uses whatever is registered for that scalar (default or user-supplied), falling back to a StringGenerator if nothing is registered;
• dispatch to any registered type by name with generators.try_generate(type_name, rng, fields) — returns None if no generator is registered for that type;
• construct one of the built-in per-type generators directly (e.g. IntGenerator { min: -10, max: 10 }.generate(rng, generators, &fields)) when you want a tuned instance without registering it.

The example below uses generate_scalar so each field is filled by whichever generator is registered for its scalar type — including any overrides the caller has installed via with_generator:

impl<R: RandomProvider> Generator<R> for PartialUserGenerator {
    fn generate(
        &mut self,
        rng: &mut R,
        generators: &mut Generators<R>,
        fields: &IndexMap<String, Vec<Node<Field>>>,
    ) -> Result<Value, ResponseError> {
        let mut obj = Map::new();
        for (response_key, group) in fields {
            let field = &group[0];
            let value = if field.name == "id" {
                Value::String(self.next_id().into())
            } else {
                generators.generate_scalar(field.ty().inner_named_type(), rng)?
            };
            obj.insert(response_key.clone(), value);
        }
        Ok(Value::Object(obj))
    }
}

Limitations

• Recursive object type not yet supported (example : myType { inner: myType })

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.