by Ben Kraft

In my last post, I talked about genqlient, our new Go GraphQL client which generates code based on your schema and queries to provide more type safety with less boilerplate than other clients. In this post, I want to talk more about the technical challenges we encountered in writing genqlient, and how genqlient solves them.

To get started, let’s revisit the example from the previous post. We write:

_ = `# @genqlient
  query GetVideoTitle($id: ID!) { video(id: $id) { title } }
`
resp, err := generated.GetVideoTitle(ctx, client, "123")
fmt.Println(resp.Video.Title, err)
// Output: El Niño and La Niña 

Then genqlient generates:

type GetVideoTitleResponse struct {
	Video GetVideoTitleVideo
}
type GetVideoTitleVideo struct {
	Title string

type __GetVideoTitleInput struct {
	Id string
}
func GetVideoTitle(
	ctx context.Context,
	client graphql.Client,
	id string,
) (*GetVideoTitleResponse, error) {
	__input := __GetVideoTitleInput{Id: id}
	var retval GetVideoTitleResponse
	err := client.MakeRequest(
		ctx, "GetVideoTitle",
		`query GetVideoTitle ($id: ID!) { video(id: $id) { title } }`,
		&retval, &__input)
	return &retval, err
}

While in this simple case, the bulk of the generated code is in the function GetVideoTitle, it’s quite formulaic: it looks basically the same for all queries, just with different types. Here, the types are simple: GraphQL object types become structs, and scalars become the corresponding Go builtin types.

However, as queries get more complex, the entrypoint functions stay about the same, while the types get much more complex. Constructing these more complex types proved to be one of the most difficult parts of writing genqlient. In the rest of this post, we’ll discuss some of those challenges, and how we solved them.

VeryLongTypeNamesAndWhyWeNeedThem

Even here, there’s already trouble with names: a single type in the GraphQL schema may need to map to many different Go types on the client! For example, consider the query

query GetVideoAndPrereqs($id: ID!) {
  video(id: $id) {    # type: Video
    title             # type: String
    prerequisites {   # type: Video
      title           # type: String
    }
  }
}

We need to generate two different types corresponding to the GraphQL type Video: one for video (with fields title and prerequisites), and another for prerequisites (with just title). And we don’t want to just call them Video1 and Video2; in addition to being less descriptive, such names would be quite unstable. (For example, adding a new field of type Video at the top of the query would mean Video1 would now be Video2, and existing code that doesn’t need the new field would no longer compile. The problem gets even worse if we have multiple queries in the same Go package, where the change might be to another query.)

Luckily, this particular path has been trod before: other code generation tools like Apollo generate type names consisting of the operation name followed by the the full field/type path to the node, and genqlient follows their lead. (The full algorithm is described in the source.) This means the two Video types in question would be GetVideoAndPrereqsVideo and GetVideoAndPrereqsVideoPrerequisitesVideo. What a mouthful! In our codebase we’ve often ended up using type aliases to give the types friendlier names (here, perhaps Video and Prereq); more recently we’ve also added an option by which the query author can specify what name to use. Luckily, we’ve found that this isn’t as much of a problem as it might seem: the most frequently-referenced types are input and enum types, which have simpler names (because there’s no need to choose which fields you want). For the rest of the names, most queries need just a few user-defined type aliases.

Interfaces and the JSON problem

To handle fields which may return one of several types, GraphQL has interfaces and unions. We saw an example in the last post, where we ask for a piece of content and request different fields depending on whether it’s an Exercise, a Video, or something else:

query GetContent($id: ID!) {
  content(id: $id) {
    title
    ... on Exercise {
      problems {
        type: problemType
        correctAnswer
      }
    }
    ... on Video {
      duration
    }
  }
}

Go also has interfaces, and so we can convert the GraphQL interface Content into a Go interface type GetContentContent (including getter methods for shared fields), and convert each GraphQL type which implements the interface into a Go struct type, so we have GetContentExercise (with fields title and problems), GetContentVideo (with title and duration), GetContentArticle (with just title), and so on, and ensure each implements the interface. It’s a bit verbose at times, but so far so good.

Readers with experience using the Go JSON library, encoding/json, may spot the problem: json.Unmarshal can’t handle interfaces. And rightly so: it sees a field Content GetContentContent, and it can’t possibly know what struct needs to go there! GraphQL has a mechanism for this: we can request a special field __typename, to tell us which. But to tell encoding/json how to use it, we need to add an UnmarshalJSON method somewhere — and since interfaces can’t have methods of their own, that method needs to go on the containing type, here GetContentResponse.

So genqlient writes out such a method, using some tricks to keep from reimplementing the whole JSON library. This gets especially complicated if, instead of a field of type Content containing a single such item, we have a field [Content] containing a potentially heterogeneous list, or even [[[Content]]]; in Go we’d generate a field of type [][][]GetContentContent, but must fill in each item of that triply-nested slice with the right implementation type for JSON unmarshaling to proceed. The generated code ends up looking a bit strange, but it’s all automatic. This provides stronger type safety than other GraphQL clients: the types make it clear that duration and problems will never both be set, and ensure that the unmarshaling will succeed regardless of the concrete type returned by the server.

Code sharing

In many cases, we want to get some data from one of several different queries, and analyze it using the same code. For example, we might have two queries:

query GetVideoByID($id: ID!) {
  video(id: $id) {
    id
    title
    url
  }
}
query GetVideoByURL($url: String!) {
  videoByUrl(videoUrl: $url) {
    id
    title
    url
  }
}

and we might want to have shared code which renders the ID, title, and URL into a link to the video. Naively, genqlient generates two totally separate types for those, GetVideoByIDVideo and GetVideoByURLVideoByUrlVideo, so even though the two have the same fields we can’t pass them to the same function without casts.

This turns out to be a surprisingly thorny problem to solve with a code generator: while it’s easy for a human to see that it might be useful to share a type here, a code generator doesn’t know how you plan to use these queries; maybe they’re totally unrelated! (Even if we know to share them, which name do we use?) We’ve ended up adding a few different mechanisms to genqlient so you can tell it that you want to be able to share code.

The first takes its cue from how GraphQL allows you to share a list of fields: fragments. In this method, you write

fragment VideoFields {
  id
  title
  url
}
query GetVideoByID($id: ID!) {
  video(id: $id) {
    ...VideoFields
  }
}
query GetVideoByURL($url: String!) {
  videoByUrl(videoUrl: $url) {
    ...VideoFields
  }
}

This query guarantees systematically that the two types must match, and provides a natural name: genqlient generates a type VideoFields and embeds it into both queries. This is nice in that it allows you to take advantage of all the features of GraphQL: you can mix and match fragments with non-shared fields, nest them, use them in conjunction with interfaces, and more.

Sadly, we found in some cases this was still a bit too much indirection for our needs: we still have types like GetVideoByURLVideoByUrlVideo, even if they embed some shared type. So genqlient also supports an option to explicitly say what name to use for a field’s type. If that type name is used in several places, genqlient validates that they all request the same fields, and it uses the given name. So we could do:

query GetVideoByID($id: ID!) {
  # @genqlient(typename: "Video")
  video(id: $id) {
    id
    title
    url
  }
}
query GetVideoByURL($url: String!) {
  # @genqlient(typename: "Video")
  videoByUrl(videoUrl: $url) {
    id
    title
    url
  }
}

This approach is less flexible: if GetVideoByURL wants to request an extra field we can no longer share the type. But in simple cases, it does exactly what you want with minimal boilerplate.

Supporting code sharing within and between queries is still an active area of work on genqlient, as we find new places where genqlient’s current options don’t allow us to share code in the way we’d like. It’s always a tradeoff: each new customization option can add significant complexity to the code generation process, so we want to make sure we add them judiciously, while still supporting the bulk of queries that developers want to make in as ergonomic a fashion as possible.

Putting it all together

As with so many software projects, these challenges all interact: we need to be able to support code sharing for interface types with both named and inline fragments. It took several iterations to figure out how to structure genqlient’s code to keep the complexity manageable. But after working our way through all these problems, and converting most of our production queries, we’re proud to say that genqlient supports all standard GraphQL query syntax, as well as several additional options to customize the generated types according to developers’ preferences.

As a reminder, genqlient is open source and ready for you to use! Check out our getting started guide to use genqlient in your project, and if you come across more challenges in your use of genqlient, file an issue or send a pull request on GitHub. And if these challenges sound like fun, check out our careers page for more on how we solve them while supporting millions of learners.

Thanks to Benjamin Tidor, Craig Silverstein, Gaurav Singh, Kevin Dangoor, and Slim Lim for comments on drafts of this post, and to Craig Silverstein, Mark Sandstrom, and many more of our colleagues for design feedback and in-depth code reviews on genqlient.