Developer Products

Roy Derks

For many companies, performance is the main reason to go with GraphQL. But is that a valid argument? In this post we'll compare performance of GraphQL and REST with k6 using StepZen.


Compare REST with GraphQL For Performance Testing Using StepZen and k6

Developer

Lucia Cerchie


## Introduction

Recently, at the [ASC 2021 API Specifications Conference](/developers/videos/openapi-conference), Anant Jhingran gave a talk about bridging the gap between the OpenAPI spec and GraphQL. Here we'll deep dive into the code behind the talk to explore some practical ways of briding the gap.

## A Critical Problem that Arises in API Design

The conventions of backend and frontend developers create a central problem in their communication process. Backend engineers, with a "producer's" perspective, are often concerned with completeness - are they surfacing all the data possible? Meanwhile, frontend engineers look for simplicity. Is it clear which methods are available to surface the data they need to power the experiences they build? They have the "consumer's" perspective.

Neither the frontend nor the backend view is perfect. The former can eliminate a lot of useful information and require manual mapping. The latter can flood the frontend developer with a lot of interfaces, oftentimes with complicated, auto-generated names. API design requires an approach that bridges teh gap.

## An Example of this Conflict of Perspective

#### The Frontend

Say the consumer is looking for the weather temperature and description from a Weather API to surface it to a widget on their UI - the only thing they'll be concerned with is a single graphql query:

```
type Query {
   weather(city:String!):WeatherReport
}

type WeatherReport {
   temp: Float
   description: String
}

```

#### The Backend

But starting from the backend, from the producer's perspective, the developer is often using the [OpenAPI specification](https://swagger.io/specification/), which defines a common interface for REST APIs that makes it easy to introspect them. Thinking in terms of creating a complete schema, the backend engineer can autogenerate the schema for a REST API with an OpenAPI spec with dozens and dozens of queries.

## Our Example

Let's take a case where a fullstack team is making a connection to two different APIs for a project.

The goal is to use a [Cloudmersive API](https://api.cloudmersive.com/) to detect the language of a given DEV.to comment.

Here's a snippet of code from the schema that Anant built for this example. An extension technically accomplishes the goal:

```graphql
extend type Devto_getArticlesResponse {
    Language(
        cloudmersive_apiKey: Secret!
    ): Cloudmersive_LanguageDetectionResponse
        @materializer(
            query: "cloudmersive_LanguageDetection_getLanguage"
            arguments: [
                { name: "cloudmersive_apiKey", argument: "cloudmersive_apiKey" }
                { name: "textToDetect", field: "description" }
            ]
        )
}
```

> Note: here the [StepZen custom directive @materializer](/docs/features/linking-types) surfaces the result of the cloudmersive language detection query to the `Devto_getArticlesResponse` type, no resolvers necessary.

The backend developers have a plan: because both of these APIs participate in the OpenAPI spec, they can easily introspect both of them for complete coverage.

The Open API spec represents years of investment. The designers of these APIs were thinking about the data that the union of _all users_ wanted, not about each individual user separately from the whole. They also carefully considered other needs, including access control, performance, and error handling.

This means that our backend engineers' introspection of both APIs results in a multitude of queries! This is dizzying for the frontend developer who just wants one result from these connections. Ironically, this gap in API design is created by too much coverage.

![screenshot of dozens of queries in the graphiql editor](/images/blog/too-much-coverage.png)

It also means that our team needs to adopt a curation process to make the GraphQL API workable.

With StepZen, curating the API implemention to a manageable number of queries is straightforward.

'Curating' here means pruning the API, transforming it from completeness for the backend to accuracy for the frontend. What queries are needed, and what aren't?

## Closing the Gap in API Design

Once the developer knows which queries they want to emphasize, it's a quick process to comment out unneeded queries in GraphQL files, and create new 'curated' schemas. And with StepZen, preventing them from surfacing to the endpoint is the same process.

To make schemas accessible to an endpoint deployed on StepZen, a developer specifies them in a file called `index.graphql`.

```grapqhl
schema
  @sdl(
    files: [
      "curateddevto.graphql"
      "curatedcloudmersive.graphql"
      # "devto.graphql"
      # "cloudmersive.graphql"
      # "extend.graphql"
    ]
  ) {
  query: Query
}
```

As you can see, you can comment out the non-curated versions of the APIs, leaving us with a more manageable query set.

![screenshot of two queries in the graphiql](/images/blog/two-queries.png)

With StepZen's `index.graphql` file, backend engineers can generate a complete schema based on the OpenAPI spec, and enable the frontend engineers with "views" of the data they need to accomplish their goals.

## Conclusion

Producers (typically, backend engineers) and consumers (typically, frontend engineers) have different "views" of the data they're exposing or consuming - the language, perspective, and implementations of the sides are different. Therefore they can run into a "communication gap" when it comes to implementing GraphQL schemas with the OpenAPI spec. But bridging the gap is possible. Furthermore, bridging the gap enables developers to leverage investment in REST and OpenAPI and take advantage of the flexibility and developer experience of GraphQL.

For more on this topic see:

-   Anant Jhingran's Open API spec presentation [video replay](/developers/videos/openapi-conference) and
-   [OpenAPI and GraphQL: The Chunnel Problem ](/blog/openapi-and-graphql-chunnel-problem) blog post


A common problem in API design is how to make the needs of backend and frontend developers meet. We explore how GraphQL and StepZen make this easier.


OpenAPI and GraphQL: Bridging the Gap

Cofounder & CEO

Anant Jhingran


The Channel Tunnel, or [Chunnel](https://en.wikipedia.org/wiki/Channel_Tunnel), is an engineering marvel that connects England with France. Trains, and cars that ride on the trains, run/drive on the left-hand side in England and on the right-hand side in France. Somewhere in the middle of the Strait of Dover, the handedness changes, seamlessly. This bridging of gaps between two different perspectives needs to occur often in IT systems and is sometimes done well, like the Chunnel, and sometimes not so well. In this post, we’ll explore the bridging of the gaps between frontend and backend systems.

Frontends — your mobile app, your website, etc. — want to deal with data and concepts that make sense to their users. For example, they may want to provide the delivery status of a customer’s order or a customer’s updated shopping cart on an eCommerce app. Backend systems, on the other hand, typically expose APIs that reflect the shape of the data on the backend. For example, they store and process data that makes sense in the context of the business and the technology — normalized tables, REST interfaces, and so on.

The two are like the two sides of the Chunnel. And increasingly, the frontends want to deal with data in the form of GraphQL. So queries that the apps make using GraphQL have to get connected with data and actions that require SQL and REST interfaces. So the more specific Chunnel problem is translating GraphQL into its subparts from SQL and REST. For the purpose of this blog, we will focus on REST.

## Which Direction Do You Travel in?

For a minute, assume that the English end of the Chunnel represents the frontend, and the backend systems are the French end.

If you start with the English side, you might have:

```graphql
type Query {
  weather(city: String!): WeatherReport
}

type WeatherReport {
  temperature: Float
  description: String
}
```

But a weather API at the backend might take `lat`, `lon` as opposed to `city`, might return a weather forecast for the next 10 days, might produce fields that say `temp` as opposed to `temperature`, might encode `description` in a two-letter code and so on. So the call to the backend and the response from the backend must be made conformant to the frontend (English) view.

However, if you start on the French side, you might take the same JSON response and create a GraphQL API that looks like:

```graphql
type Query {
  OneAPI25ByLatLon(lat: Float!, lon: Float!): OneAPI25ByLatLonResponse
}
type OneAPI25ByLatLonResponse {
  coord: CoordResponse
  weather: WeatherResponse
  base: String
  main: MainResponse
  ...
}
...
type Main {
  temperature: Float
  ...
}
```

In this way, you get very high fidelity to the backend, but you also get all the details the backend produces. Any autogeneration is forced to generate names of types and queries.

As you can see, neither the frontend-first (English) view nor the backend-first (French) view is perfect. The former eliminates a lot of useful information and requires manual or semi-manual mapping. The latter floods the frontend developer with a lot of interfaces, oftentimes with complicated, auto-generated names. What you want is a meet-in-the-middle approach.

## Other Core Problems in the Chunnel
While we will not go into details, there are other issues to be addressed:

- There may be no way to meet in the middle. Perhaps the two sides are fundamentally irreconcilable. For example, a frontend API that returns all orders for a product to facilitate recalls might require a backend API that is the reverse of “all products of an order,” and such an API might not exist.
- The tracks might be really slow. While this could happen for a number of reasons, a common reason is that GraphQL allows recursive paths through enterprise data. So, often, the backends that support it end up being inundated with requests. (A GraphQL request might generate 500 calls for the different backends!)
- The gauges on the two sides might be different. (Am I carrying the analogy of Chunnel too far?) This is a fundamental gap. GraphQL is strongly typed, but often the underlying REST is weakly typed. I like the analogy of Golang/Java as strongly typed, but JavaScript is weakly typed. Neither is good nor bad; they are just different.
- Road signs in and out could be missing. In the REST world, a machine-readable description of what exists is available via OpenAPI specs. However, in our analysis, only one in 100 APIs had completely correct OpenAPI specs. This leads to guessing on the part of someone building the connectivity between frontend and backends. GraphQL, on the other hand, is completely introspectable, unless it has been deliberately turned off.
- The pagination model is common in GraphQL, but is left more as an exercise to the developer in REST.
- The filter and selection model is ad hoc in GraphQL; it is equally ad hoc in REST, but the two are almost always different.
- Versioning is central to REST constructs, while GraphQL takes a strong opinion on avoiding versioning.
- AuthN/AuthZ is sufficient in REST, but it is insufficient in GraphQL.

## Summary
Fundamentally, there are no frontends without backends. However, the language, perspective, and implementations of the two sides are completely different. The Chunnel can and must be built. But it requires a good understanding of the two sides and intelligent “meet-in-the-middle” approaches. We at StepZen are building a [set of capabilities](/docs/what-is-stepzen) to help build that Chunnel.

---
This article was originally published in The New Stack: [![The New Stack](/media/blog/TNS-chunnel-repost.png)](https://thenewstack.io/openapi-and-graphql-the-chunnel-problem/)







GraphQL APIs are excellent for frontend developers, providing acceleration for application development and an improved developer experience. However, new challenges for the security of these APIs arise. Here we explore finding the right balance for GraphQL API technology on a continuum between safety and progress.


OpenAPI and GraphQL: The Chunnel Problem

Samuel Hill


Hard; building an app that fits a user's wants and desires. 

Harder; building an app and logging activity to understand all wants and desires of a user.

Lucky for developers, [React Hooks](https://reactjs.org/docs/hooks-reference.html) is an effective way of handling state management and event listeners on a frontend. The event tracking lifecycle with React Hooks and GraphQL is comprised of two steps.

1. Properly capturing DOM events to ensure a single, accurate event is logged. 
2. Asynchronously fetching an API without disrupting the DOM document event tracking.

You can find the code for this example in the [StepZen](https://github.com/stepzen-dev/demo-event-tracking) GitHub repo.

### Create React App

On your machine, run [create-react-app](https://reactjs.org/docs/create-a-new-react-app.html) and create a project called `demo-event-tracking`.

```bash
npx create-react-app demo-event-tracking
cd demo-event-tracking
npm start
```

Once running you should see the app running at `localhost:3000`.

### Setting up Google Analytics

Let's create a Google Analytics account to track page activity for this react application. To properly track with the [`react-ga`](https://www.npmjs.com/package/react-ga) npm dependency, create a [Universal Analytics Property](https://support.google.com/analytics/answer/10269537) for the react application. The `react-ga` dependency generates a site visitor `clientId`, which we will capture and pass into the GraphQL Mutation later on.

![universal-analytics-property](/images/blog/google-analytics-universal.png)

Install the `react-ga` dependency in the root of your React application with the following command:

```bash
npm install react-ga
```

### Create an Amplitude Account

The next step is prepping a platform that you can send the click events to. You could use [Google Analytics Event Measurements](https://developers.google.com/analytics/devguides/collection/analyticsjs/events) to track:
- `[eventCategory]`
- `[eventAction]`
- `[eventLabel]`
- `[eventValue]`

However, creating custom events in google analytics is complex and limited to these specific labels. To create custom event key-value with ease, I went with amplitude's API.

Sign up for a free account at [Amplitude.com](https://amplitude.com/) and go to account settings. Copy the api and secret key to reference later.

![amplitude-settings](/images/blog/amplitude-settings.png)

### Create the StepZen Endpoint

> Note: If you do not have a stepzen account, [sign up](https://stepzen.com/signup) for one now for free and follow the [stepzen cli installation](https://stepzen.com/docs/quick-start#install-the-stepzen-cli) instructions to run the command below.

In the root of our project, run the following command. This will create a stepzen folder in a single command.

For Mac
```bash
mkdir stepzen; cd stepzen; touch index.graphql; touch amplitude.graphql; touch config.yaml
```

For Windows
```bash
mkdir stepzen; cd stepzen; type nul >> "index.graphql"; type nul >> "amplitude.graphql"; type nul >> "config.yaml"
```

Now in our code editor, open `index.graphql` and add the `amplitude.graphql` schema file to our @sdl schema index.

```graphql
# index.graphql

schema
  @sdl(
    files: [
      "amplitude.graphql"
    ]
  ) {
  query: Query
}
```

In the `amplitude.graphql` file, copy the code below to properly write a event mutation to our amplitude account.

```graphql
# amplitude.graphql

type Amplitude_Event {
  code: Int
  events_ingested: Int
  payload_size_bytes: Int
  server_upload_time: Int
}

type UserSearch {
  id: ID!
  lastUsed: String
  email: String
}

type Query {
  userSearch(email: String!): UserSearch
    @rest(
      setters: [
        { field: "email", path: "user_id" }
        { field: "id", path: "amplitude_id" }
        { field: "lastUsed", path: "last_seen" }
      ]
      resultroot: "matches[]"
      endpoint: "https://$apikey;:$secret;@amplitude.com/api/2/usersearch"
      configuration: "amplitude_config"
    )
}

type Mutation {
  amplitude_event(
    userId: String!
    eventType: String!
    eventTime: String!
    path: String!
  ): Amplitude_Event
    @rest(
      endpoint: "https://api2.amplitude.com/batch"
      postbody: "{\"api_key\":\"{{ .Get \"apikey\" }}\",\"events\":[{\"user_id\":\"{{ .Get \"userId\" }}\",\"event_type\":\"{{ .Get \"eventType\" }}\",\"event_properties\":{\"path\":\"{{ .Get \"path\" }}\"},\"event_time\":\"{{ .Get \"eventTime\" }}\"}]}"
      method: POST
      contenttype: "application/json"
      configuration: "amplitude_config"
    )
}
```

In this file, there is a `configuration: "amplitude_config"` that is sourcing from the `config.yaml`. Add the api and secret keys from the amplitude account settings into this yaml file.

```yaml
# config.yaml

configurationset:
  - configuration:
      name: amplitude_config
      apikey:
      secret: 
```

Now with our endpoint schema properly configured, run the following command to generate the GraphQL endpoint.

```bash
stepzen start --endpoint demo/events
```

A successful deploy will return a StepZen endpoint on your account that you can also explore from the StepZen dashboard. From the dashboard, test the mutation below to ensure the endpoint is sending the proper data to amplitude.

```graphql
# amplitude.graphql

mutation MyMutation {
  amplitude_event(
    eventTime: "09/08/2014, 2:35:56 AM"
    eventType: "click"
    path: "app_header"
    userId: "1232342345"
  ) {
    code
    events_ingested
    payload_size_bytes
    server_upload_time
  }
}
```

If you can see an event has been sent to your amplitude account, the mutation ran successfully! Now we can set up the frontend to send this event <em>on click</em> in our application.

# React App Event Tracking

In the root of your project, paste the newly generated endpoint into a `.env` file.

```env
REACT_APP_ANALYTICS_STEPZEN_API_URL=https://youraccount.stepzen.net/demo/events/__graphql
REACT_APP_ANALYTICS_STEPZEN_API_KEY=
REACT_APP_REACTGA_ID=UA-XXXXX-X
```

This is also where you will provide your [StepZen API key](https://dashboard.stepzen.com/account) and google analytics universal code. With our environment configurations set up, the first step is to create the context of our analytics tracker.

### Tracking Pageviews and Sessions with ClientId in Google Analytics

From the root of your project, run the following command. 

For Mac
```bash
touch .env; mkdir src/contexts; cd src; cd contexts; touch analytics.js
```

For Windows
```bash
type nul >> ".env"; mkdir src/contexts; cd src; cd contexts; type nul >> "analytics.js"
```

Open the newly created `analytics.js` file and add the following code.

```javascript
// src/contexts/analytics.js

import { createContext, useEffect, useState } from "react";
import ReactGA from "react-ga";

const analyticsContext = createContext();

const {
  REACT_APP_ANALYTICS_STEPZEN_API_URL,
  REACT_APP_ANALYTICS_STEPZEN_API_KEY,
  REACT_APP_REACTGA_ID,
} = process.env;

const url = window.location.pathname + window.location.search

const AnalyticsProvider = (props) => {
  const [userId, setUserID] = useState("unknown");
  useEffect(() => {
    ReactGA.initialize(`${REACT_APP_REACTGA_ID}`, { debug: true });
    ReactGA.ga((tracker) => {
      ReactGA.ga("set", "userId", "" + tracker.get("clientId"));
      setUserID(tracker.get("clientId"));
    });
    ReactGA.pageview(url);
  }, []);

  const state = {
    userId: userId
  }

  return (
    <analyticsContext.Provider value={state}>
      {props.children}
    </analyticsContext.Provider>
  );
};

export { analyticsContext, AnalyticsProvider };
```

Google Analytics is initialized in the `useEffect` React Hook. Once initialized, the `userId` is assigned to the `clientId`, and the pageview is captured from `window.location.pathname + window.location.search`.

```javascript
useEffect(() => {
    ReactGA.initialize(`${REACT_APP_REACTGA_ID}`, { debug: true });
    ReactGA.ga((tracker) => {
      ReactGA.ga("set", "userId", "" + tracker.get("clientId"));
      setUserID(tracker.get("clientId"));
    });
    ReactGA.pageview(url);
  }, []);
 ```
 
The dependencies are react hooks, `createContext`, `useEffect` that you can read about more [here](https://reactjs.org/docs/hooks-reference.html), and `react-ga` for analytics in google.

```javascript
import { createContext, useEffect, useState } from "react";
import ReactGA from "react-ga";
```

The `analyticsContext` is a wrapper around the application to initialize `reactGA` as a global element. This creates global variables for state management, similar to Redux but built into the react core.

```javascript
const analyticsContext = createContext();
```

For context, you can pass any state object into child components. In this case, you can pass `userId` if you want to pass the clientId to any other components.

```javascript
const state = {
    userId: userId
}
```

### Pass the Context to the Index.js Root File

To create the wrapper, go to the `index.js` file and pass `AnalyticsProvider` into the render of our ReactDOM.

```javascript
// src/index.js

import React from "react";
import ReactDOM from "react-dom";
import App from "./App";

import { AnalyticsProvider } from "./contexts/analytics";

ReactDOM.render(
  <React.StrictMode>
    <AnalyticsProvider>
      <App />
    </AnalyticsProvider>
  </React.StrictMode>,
  document.getElementById("root")
);
```

With `AnalyticsProvider` in the ReactDOM, we can track google analytics pageviews on each application render. 

#### Spin Up Google Analytics

Run `npm run start` in the root of the application and open the `localhost:3000` app. The pageviews should now be tracking in the analytics account you created!

![realtime-firing](/images/blog/google-analytics-realtime.png)

To further enhance our tracking, setting up the [User-ID](https://support.google.com/analytics/answer/3123666#zippy=%2Cin-this-article) feature in Google Analytics to associate sessions with the `clientId`.

### Creating the Event Listener

The next step is creating a function that captures the click events on the app. 

```javascript
  // This is the click handler on document.body that tracks analytics
  useEffect(() => {
    const handler = async (evt) => {
      const el = evt?.target?.closest("*");
      // See what is being tracked in the closest capture.
      console.log(el)
    };

    document.body.addEventListener("click", handler);

    return () => {
      document.body.removeEventListener("click", handler);
    };
  });
```

The newly created function, `handler` is going to be called by the [addEventListener](https://developer.mozilla.org/en-US/docs/Web/API/EventTarget/addEventListener), wrapped by a separate useEffect hook in `analytics.js`.

When executed, the handler should return the closest HTMLElement `evt?.target?.closest("*");` to the click event. The click event is being captured by the DOM listener, and removed when the event is completed.


### Write a GraphQL Mutation

To enhance the `handler` function, a graphql mutation can be written and run asynchronously with the click event.

```javascript
const handler = async (evt) => {
      // The current date / time
      const now = new Date().toLocaleString();

      // Get the nearest element.
      const el = evt?.target?.closest("*");

      // Get the full DOM path by traversing up the tree
      let path = [];
      let ref = el;

      do {
        const tag = ref.tagName.toLowerCase();
        const id = ref.id ? `#${ref.id}` : "";
        let classes = [];
        ref.classList.forEach((c) => classes.push(`.${c}`));

        path = [`${tag}${classes.join("")}${id}`, ...path];
        ref = ref.parentNode;
      } while (ref.parentNode);

      path = path.join(" > ");

      // Create an action object. It's either our specifically marked one,
      // or derived from the event
      const action = {
        path: el.dataset.analyticsData || path,
        type: el.dataset.analyticsAction || evt.type,
      };

      // This is the query we are sending
      const query = `
        mutation MyMutation {
          amplitude_event(
            eventTime: "${now}"
            eventType: "${action.type}"
            path: "${action.path}"
            userId: "${userId}"
          ) {
            code
            events_ingested
            payload_size_bytes
            server_upload_time
          }
        }
      `;

      console.log(query)

      if (REACT_APP_ANALYTICS_STEPZEN_API_URL) {
        await fetch(REACT_APP_ANALYTICS_STEPZEN_API_URL, {
          body: JSON.stringify({ query }),
          headers: {
            Authorization: `Apikey ${REACT_APP_ANALYTICS_STEPZEN_API_KEY}`,
            "Content-Type": "application/json",
          },
          method: "POST",
        }).then((response) => {
          console.log(response);
          response.json().then((data) => {
            console.log("the response", data);
          });
        });
      }
    };
```

In the fetch request, the GraphQL Mutation takes four arguments.

```graphql
eventTime: "${now}"
eventType: "${action.type}"
path: "${action.path}"
userId: "${userId}"
```

`const now = new Date().toLocaleString('en-US', { timeZone: 'PST' });` will create a variable associated with time of the click event.

`const el = evt?.target?.closest("*");` creates a variable that is looped over in a `do{}while{}` loop and captures parent HTMLElements of the event action. This loop is reassigned to the variable `path`.

The path and type are assigned to an object `action`.

```javascript
const action = {
    path: el.dataset.analyticsData || path,
    type: el.dataset.analyticsAction || evt.type,
  };
```

The state variable `userId` is assigned to `clientId`. This `clientId` is being generated by the `react-ga` dependency.

### Capture the Click Event

The userId from google analytics and event listeners of click events are ready to be timestamped and sent in a GraphQL fetch request. Stringify the query, provide the proper headers, and send the data to the generated StepZen endpoint.

Re-run the app if it it is not already running:

```bash
npm run start
```

Now at `localhost:3000`, the react app should be running. Clicking anywhere in the application will capture the click event. After the click event occurs, a response from the fetch request should appear in our console!

![click-event-mutation](/images/blog/click-event-mutation.jpg)

The GraphQL Response

```json
{
  "data": {
    "amplitude_event": {
      "code": 200,
      "events_ingested": 1,
      "payload_size_bytes": 234,
      "server_upload_time": -601656214
    }
  }
}
```

Amplitude should be tracking one click event by time, path, type, and userId. The `clientId` being passed from the `react-ga` tracker is the `userId` in Amplitude, and should be the same `userId` being tracked in google analytics.

![amplitude-event](/images/blog/amplitude-event.jpg)

Congrats, you have now successfully captured an event on a single-page react application that is passing to two separate analytics platforms!

### Next Steps

To capture DOM events with Event Listeners in alignment with react hooks is valuable for modern application analysis. Once these DOM events are properly sending data logs to third-party analytics platforms, the need to send and modify data logs to multiple locations beyond google analytics and amplitude can be the next challenge for analytics in an application. 

As the backend source for data logging, GraphQL and StepZen make POST'ing these API events easily configurable, and more importantly, scalable for any application.

I do plan on showing how StepZen can scale these tracking events in a part 2 series of Event Tracking. Reach out [on our Discord](https://discord.gg/T2st3mCp) if you want me to expedite the series to get your analytics firing on React.

You can sign up for StepZen [here](https://stepzen.com/signup) to give it a try.

Handling event listeners and DOM tracking in a react application. React Hooks take control of state management, and when implemented properly, can empower a product team to analyze features and functionality of a web platform with ease. 


Logging Event Listeners with GraphQL and React Hooks


Introspection is a heavy word, but in systems architecture it essentially means: “I will be able to tell you what I can do programmatically.”

So if I am a database, I can tell you the tables I have, the columns and views, etc. If I am a REST backend, I can tell you the curl you need to make, the optional parameters and what I return as a response, including error codes. And if I am a GraphQL endpoint, I can tell you the queries, mutations and types that I support.

As a developer, you can always read docs; make SQL, curl, or GraphQL calls; and learn the backend capabilities by trying. Then why bother about introspection? Because introspection is the only way for a program to build something extraordinary on top of it. It could be a tool like [Swagger Docs](https://swagger.io/tools/swaggerhub/hosted-api-documentation/) that creates docs. It could be a tool like [GraphiQL](https://github.com/graphql/graphiql) that lets developers submit queries and see responses. It could be a database optimization program that tries to create indexes automatically. It could be an IDE that needs a pulldown of the queries supported by an application. It could be a change management tool that runs periodically and determines the impacts of the backend changes on the application. It could be an autonomous agent that bootstraps itself by starting an understanding of what the landscape of systems is. It could be many things, but the important thing is that it is not a human being.

## The State of Introspection for REST APIs
The current gold standard for describing REST APIs is [OpenAPI](https://swagger.io/specification/). OpenAPI 3.0 was the first official release of the specification after being donated to the OpenAPI Initiative and renamed from the Swagger Specification to OpenAPI specification in 2015. OpenAPI is a description, in YAML or JSON, of what the backend capabilities are. So, for example, here is a snippet from devto.io:

```bash
---
paths:
  /articles:
    get:
      description: |
        ...
      parameters:
        - $ref: "#/components/parameters/pageParam"
        - $ref: "#/components/parameters/perPageParam30to1000"

```

As you can see, it specifies that the API supports `GET /articles?pageParam`.

I won’t get into the exact syntax here — there are plenty of references. The great thing about the OpenAPI spec is that it solves the documentation problem for a machine; with tooling, it also solves the documentation problem for human beings. Two birds with one stone!

However, creating an OpenAPI spec for an endpoint is entirely voluntary. Twitter does not publish its OpenAPI spec, and the OpenAPI spec for HolidayAPI is out of date! I estimate that less than 5 percent of the more than 100,000 APIs that are publicly available have good, consistent and up-to-date OpenAPI specs. So that leaves 95,000 APIs with no OpenAPI specs and no way for any programmatic access to them.


## The State of Introspection for GraphQL APIs
GraphQL has emerged as the preferred paradigm for frontend developers to fetch and modify backend data in the past few years. GraphQL is intuitive and performant, and the amount of tooling available around it is impressive. And GraphQL, by default, is introspective. What that means is that GraphQL endpoints support a GraphQL query (wow!) like this:

```graphql
{
  __schema {
    queryType {
      name
    }
  }
}
```

Fundamentally, we ask a GraphQL endpoint to tell us what queries it supports using a GraphQL query (this reminds me of SQL introspection in most databases!). This is convenient, actually awesome, and allows a whole set of third-party tools to work against any GraphQL endpoint.

Furthermore, if the GraphQL endpoint changes, the introspection query returns a different — current or updated — result. Consequently, as long as the program issues the query periodically, the production of an API endpoint and its use stays up to date. How awesome is that? In contrast, in OpenAPI, the spec is a separate doc. So if that is not updated, the endpoint will do something, and the spec will say something else. Bad!

## The ‘We Don’t Have OpenAPI Specs for 95% of REST APIs’ Problem

[Postman](https://www.postman.com/) has some tooling to take a Postman collection and generate an OpenAPI spec out of it. However, it is experimental, and there is no sure shot that it works. Some people have created some OpenAPI specs against the APIs they like, but again, they are typically incomplete or wrong as a third-party effort. For example, one can get the OpenAPI spec for a small subset of AccuWeather API, but it is woefully outdated. So, in essence, if the API provider does not want to provide, or does not provide for whatever reason, an OpenAPI spec, the developer is basically out of luck.

However, if you consume a GraphQL API, you get introspection for free. And you get all the benefits of GraphQL. So no surprise that developers are increasingly demanding that the APIs that they deal with are GraphQL-ized.

If an API provider decides to GraphQL-ize their REST endpoints, some tricks help make the investment worthwhile:

1. Write the OpenAPI spec. Then it is simple, with tools like IBM’s openapi-to-graphql and StepZen’s OpenAPI introspection, to create a GraphQL endpoint. Thus, you kill two birds with one stone: You get an OpenAPI spec for your REST API consumers, and you also get a fantastic GraphQL API for your GraphQL consumers. We recommend this path.
2. Take your existing curl and Postman collections and generate a pretty good GraphQL out of it (using tools like StepZen). The advantage of this is that you get GraphQL through simple clicks and responses.

## Summary
There are two decades of investment in REST, yet the future of APIs that deal with modern frontend experiences is primarily being written around GraphQL. Unfortunately, the investment in introspection in REST has not kept up with its popularity. Meanwhile, GraphQL comes with auto-introspection.

We believe that some targeted investments by the API providers, and using an increasing number of REST-to-GraphQL open source and vendor tools, will make the transition much more straightforward. Come on, let’s get a move on!


---
Originally published: [![The New Stack](/media/blog/TNS-introspection-repost.png)](https://thenewstack.io/the-state-of-introspection-for-rest-and-graphql-apis/)







You can always read docs; make SQL, curl, or GraphQL calls; and learn the backend capabilities by trying them. Why bother about introspection? It is the way for a program to build something extraordinary on top of it. It could be Swagger docs, GraphiQL, a database optimization program, a change management tool, or another programmatic approach. 


The State of Introspection for REST and GraphQL APIs

Developer Advocate

Anthony Campolo


CMSs are a crucial component in any developer’s stack. But a CMS may not include everything your project needs. Sometimes we need to mix and match data from CMSs and other data sources. With StepZen, you can easily mix data from an external CMS with your own project by incorporating StepZen’s custom directives.

[Storyblok](https://www.storyblok.com/home) is a headless CMS that includes a visual editor and [various APIs](https://www.storyblok.com/docs/api) to access your content. It gives your team the ability to manage content for multiple use-cases including corporate websites, eCommerce sites, help desks, mobile apps, and screen displays. In this example we use StepZen's [`@graphql` directive](https://stepzen.com/docs/connecting-backends/how-to-connect-a-graphql-api) to connect to Storyblok's [GraphQL API](https://www.storyblok.com/docs/graphql-api).

In addition to connecting to the Storyblok GraphQL API, we can also use StepZen's other custom directives to bring in data from any kind of REST endpoint we want. This example will include a Shopify backend to bring in products along with the blog posts from Storyblok.

You can find the code for this example on the [StepZen Samples](https://github.com/stepzen-dev/stepzen-storyblok) GitHub.

## 1. Create a Blog Post on Storyblok

### Sign up for Storyblok

You can sign up for Storyblok at the [following link](https://app.storyblok.com/#!/signup).

### Create a New Space

After signing up, you are asked whether you want to use a demo or create a new space. A space is a content repository for keeping all the content related to one project. Each space has its own components, datasources, assets, environments, domains, collaborators, and permissions.

![01-choose-your-own-adventure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wnbcfftvuwtkfpvj62fs.png)

Choose to create a new space instead of playing with a demo so we can build up our content from scratch.

### Storyblok Dashboard

After creating a new space you are taken to your Storyblok dashboard. You can manage your content here. Create a new folder and give your folder a name and slug, then create a new entry and use the Post content blueprint to create a blogpost.

![02-create-your-first-entry](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0qsb83sjey4ijpog0quk.png)

The post editor gives you the ability to create your blog post and include images, an intro, and an author.

![03-post-editor-with-post-data](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cnr0mbx5g7295lglzcma.png)

The JSON response looks like the following:

```json
{
  "_uid": "ec395043-549c-43d5-a389-6e3334e1f35a",
  "component": "Post",
  "title": "My Super Awesome Post",
  "image": "//a.storyblok.com/f/122199/1920x1080/823e44a555/gorillas-shark-explosion-high-five-wallpaper.jpeg",
  "intro": "This is the intro to my super awesome post. It isn't a very long intro.",
  "long_text": {
    "type": "doc",
    "content": [
      {
        "type": "paragraph",
        "content": [
          {
            "type": "text",
            "text": "This is the long text of my super awesome post. It is slightly longer than the intro."
          }
        ]
      }
    ]
  },
  "author": ""
}
```

## 2. Connect to the Storyblok GraphQL API

You can find your API keys listed under the Settings tab. Use this token to access Storyblok's GraphiQL editor.

![04-settings-api-keys](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5apxi6yqm01rcjktl10z.png)

### Storyblok GraphiQL Editor

Storyblok includes a [GraphiQL editor](https://gapi-browser.storyblok.com/) similar to StepZen to explore your data. Include `?token=YOUR_TOKEN` after the URL to access your specific content. Open the Explorer tab to see the different queries that are available. The queries include the content itself as well as information about your content and space.

![05-graphiql-explorer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2p7py4pjlflb2q1jfkn7.png)

Send the following test query returning the `title`, `intro`, and `long_text` of the blog posts to make sure you are connected.

```graphql
{
  PostItems {
    items {
      content {
        title
        intro
        long_text
      }
    }
  }
}
```

![06-graphiql-query](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3q56dms0a4nty4fk5qqz.png)

## 3. Create a StepZen Project

Now that we have data in our CMS and our API key, we can create our StepZen project. Lets create schema files called `storyblok.graphql` and `products.graphql` along with the other files we need to setup our project including `stepzen.config.json`, `config.yaml`, and `index.graphql`.

```bash
mkdir -p stepzen-storyblok/schema
cd stepzen-storyblok
touch index.graphql schema/storyblok.graphql schema/products.graphql stepzen.config.json config.yaml
```

In this example there are only two files in the `schema` to show the base case of connecting to the CMS and an eCommerce backend. As your schema grows it is important to separate out the schema into separate files.

### `stepzen.config.json`

`stepzen.config.json` gives you the ability to configure different aspects of your StepZen project such as the endpoint name or root directory. Include the following JSON object to specify that our endpoint is called `api/stepzen-storyblok`.

```json
{
  "endpoint": "api/stepzen-storyblok"
}
```

If you do not have a `stepzen.config.json` file then the CLI asks for a name for your endpoint the first time you deploy your project with `stepzen start`.

### `index.graphql`

`index.graphql` is the glue that pulls together all the other `.graphql` files in your schema. Our `index.graphql` file only includes two schema files called `storyblok.graphql` and `products.graphql`.

```graphql
schema
  @sdl(
    files: [
      "schema/storyblok.graphql"
      "schema/products.graphql"
    ]
  ) {
  query: Query
}
```

### `config.yaml`

`config.yaml` is a file written in YAML to relay information regarding your configuration and keys to StepZen. This is where we set our API token we received from Storyblok. Including `storyblok_config` in the `@graphql` query lets us authenticate our queries to the Storyblok GraphQL API.

```yaml
configurationset:
  - configuration:
      name: storyblok_config
      token: <YOUR_TOKEN_HERE>
  - configuration:
      name: shopify_config
      api_key: <YOUR_API_KEY>
      api_password: <YOUR_API_PASSWORD>
      store_name: <YOUR_STORE_NAME>
```

Include your own token and information for your Shopify store.

### `storyblok.graphql`

In our schema directory is a file called `storyblok.graphql` for our types and queries. Our types include a `PostItem` with a `name` for the post and a `Content` type for the `title` and `intro` of the post. The `PostItems` type returns an array of those `PostItem` objects.

```graphql
type PostItems {
  items: [PostItem]
}

type PostItem {
  name: String
  content: Content
}

type Content {
  title: String
  intro: String
}
```

This example shows how to query for just a small subset of all the information available from the Storyblok GraphQL API. If you want to generate a complete schema without needing to code it by hand, you can introspect your endpoint with [`stepzen import graphql`](https://stepzen.com/docs/connecting-backends/graphql-introspection).

```graphql
type Query {
  PostItems: PostItems
    @graphql(
      endpoint: "https://gapi.storyblok.com/v1/api"
      headers: [
        { name:"Token" value:"$token" }
      ]
      configuration: "storyblok_config"
    )
}
```

Our query includes the `endpoint` and sets the `headers` for our token. `configuration` includes our token from `config.yaml`.

### Deploy your GraphQL API to StepZen

`stepzen start` deploys the code in your current directory to the specified endpoint on StepZen. It also watches the directory for changes and automatically deploys them to the endpoint specified.

```bash
stepzen start
```

You can now use the StepZen dashboard to test your API by exploring the queries and types available and querying the API running on StepZen.

### Test the Endpoint

Run the following test query for the `intro` and `title` of your post.

```graphql
{
  PostItems {
    items {
      content {
        intro
        title
      }
    }
  }
}
```

This returns the information we created early in the Storyblok editor.

```json
{
  "data": {
    "PostItems": {
      "items": [
        {
          "content": {
            "intro": "This is the intro to my super awesome post. It isn't a very long intro.",
            "title": "My Super Awesome Post"
          }
        }
      ]
    }
  }
}
```

## 4. Connect an eCommerce REST API

Now that we have connected our blog to our StepZen project, we can pull in products from an eCommerce backend by using the `@rest` directive. This example connects to Shopify and queries for a set of products.

### `products.graphql`

`products.graphql` has a `Product` type and a `products` query that returns an array of `Product` objects. Our product type only has a few properties.

```graphql
type Product {
  id: ID!
  handle: String
  title: String
}
```

Our `products` query connects to the Shopify REST API using StepZen's `@rest` directive. The `@rest` directive accepts the URL of the REST endpoint that we want to connect to.

```graphql
type Query {
  products: [Product]
    @rest(
      resultroot: "products[]"
      endpoint: "https://$api_key:$api_password@$store_name.myshopify.com/admin/api/2020-01/products.json"
      configuration: "shopify_config"
    )
}
```

### Query for Products and Posts

With our blog and shop connected into our main schema, we can query for both the blog posts and the products with a single query.

```graphql

{
  PostItems {
    items {
      content {
        title
        intro
      }
    }
  }
  
  products {
    title
    id
    handle
  }
}
```

This returns the following output:

```json
{
  "data": {
    "PostItems": {
      "items": [
        {
          "content": {
            "intro": "This is the intro to my super awesome post. It isn't a very long intro.",
            "title": "My Super Awesome Post"
          }
        }
      ]
    }
    "products": [
      {
        "handle": "this-is-a-product",
        "id": "661261903800",
        "title": "BUY ME"
      }
    ]
  }
}
```

## Next Steps

With the two directives pulling in data from two completely separate backends, StepZen does the heavy lifting of merging the schemas together and providing the resolvers under the hood. This allows the developer to get all the data they need with just a single query.

Curious to learn more about Storyblok? You can see Facundo Giuliani and Anthony Campolo walk through creating content in Storyblok and connecting to StepZen in [How to mix data from Storyblok CMS with your own project using StepZen](https://www.youtube.com/watch?v=gDxYEUIzRMQ).

You can sign up for StepZen [here](https://stepzen.com/signup) to give it a try. If you have more questions or just want to hang out you can also join us [on our Discord](https://discord.gg/T2st3mCp).


Sometimes we need to mix and match data from CMSs and other data sources. With StepZen, you can easily mix data from an external CMS with your own project by incorporating StepZen’s custom directives.
