Backstage is an excellent framework for building an internal developer portal. It provides all of the fundamental building blocks to improve developer experience in an organization.

Core to Backstage is its Catalog of entities. The Catalog provides a database of software components, resources, libraries, and other kinds of software items. It provides client code and an API backend to retrieve items in the catalog, along with the software interfaces required to populate the entity catalog. Its model is flexible, customizable, and powerful.

However, with great power comes great responsibility. Without experience, it’s easy to develop anti-patterns in Backstage catalog usage. These anti-patterns can then turn into major performance issues at scale. This in turn leads affects trust and usage of the product as a whole.

At Roadie, we provide an out-of-the-box version of Backstage for our customers. We have come across many of the ways in which non-optimal Catalog client usage can affect performance of the application as a whole. We have seen these performance issues result in lagging page loads and (in extreme cases) causing page loads to fail in Backstage.

By applying the patterns explained in this post, you could see a huge improvement in Catalog response time. In some cases, you may even see Catalog queries perform 48x faster!

Architecture of the Entity catalog

The entity catalog in Backstage is made up of three components. A Catalog client, the Catalog backend, and the Catalog database. When Backstage starts up for the first time, it will have a Catalog database and a catalog backend. When you visit the Backstage application in your browser, it will make use of the Catalog client to retrieve data from the Catalog backend. The Catalog backend in turn retrieves the requested catalog items from the Catalog database.

Using the Backstage Catalog Client

Soon after deploying Backstage in an organization, users will want to customize it.

Frequent customizations we come across include loading entities into the Catalog from an in-house platform or visualizing data from an internal system in the Backstage UI. Customization is normal and is a sign that Backstage is adding value for teams.

When developers write extensions to Backstage, it’s likely they will come across the need to interact with the Catalog. There are two ways they can do this:

1. via a Frontend Backstage extension
2. via a Backend Backstage extension

To make use of the Catalog client in a frontend Backstage extension, you are likely to be using the useApi hook, along with a useAsync function.

import { catalogApiRef } from '@backstage/plugin-catalog-react';
import { useApi } from '@backstage/core-plugin-api';
import { stringifyEntityRef } from '@backstage/catalog-model';
import useAsync from 'react-use/lib/useAsync';

export const CustomReactComponent = () => {
  const catalogApi = useApi(catalogApiRef);
  const {
    value: entities,
  } = useAsync(async () => {
    const response = await catalogApi.getEntities();
    return response.items || [];
  }, [catalogApi]);

  return (<>{entities.map(stringifyEntityRef).join('\n')}</>)
}

In a backend Backstage extension, you are likely to be constructing the Catalog client using the discovery client. The discovery client is a helper that allows plugins to discover the API location of other clients.

Generally, if you are writing a backend plugin, like a new REST API or a Catalog processor, you will have access to the discovery client. Depending on your particular situation, you may have access to the discovery client in a different way.

import { CatalogClient } from '@backstage/catalog-client';
import { DiscoveryApi } from '@backstage/core-plugin-api';

export const getAllEntities = async (discovery: DiscoveryApi) => { 
  const catalogApi = new CatalogClient({
    discoveryApi: discovery,
  });

  const response = await catalogApi.getEntities();
  return response.items;
}

You will notice that once you have an instance of the CatalogApi, it is used in the same way in either the frontend or a backend extension.

(await catalogClient.getEntities()).items;

Check the Backstage docs for a more comprehensive explanation of the full Catalog interface.

How big can a Backstage Catalog get?

When thinking about Catalog size, it’s useful to think about two things:

• How big is each individual entity?
• How many entities do you have?

A typical Entity

A typical Backstage Entity looks like this:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: artist-web
  description: The place to be, for great artists
  labels:
    example.com/custom: custom_label_value
  annotations:
    example.com/service-discovery: artistweb
    circleci.com/project-slug: github/example-org/artist-website
  tags:
    - java
  links:
    - url: https://admin.example-org.com
      title: Admin Dashboard
      icon: dashboard
      type: admin-dashboard
spec:
  type: website
  lifecycle: production
  owner: artist-relations-team
  system: public-websites

It describes a website called the artist-web. It has a few basic Backstage entity properties, and some annotations, tags, and links. It’s encoded in YAML here, but in Backstage’s database, it is stored as plain text in JSON format.

Uncompressed, this entity definition is about half a kilobyte. Therefore, a Catalog containing about 20,000 similarly sized entities would add up to about 10 megabytes of data uncompressed. That’s a pretty big chunk of data to be sending over the wire.

However, we haven’t seen anything yet…

The Backstage Catalog model defines an API Kind. These are used to document the endpoints that services make available in Backstage. API entities often contain an embedded OpenAPI doc.

For example:

apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  name: artist-api
  description: Retrieve artist details
spec:
  type: openapi
  lifecycle: production
  owner: artist-relations-team
  system: artist-engagement-portal
  # The embedded OpenAPI spec is in the defiition
  definition: |
    openapi: "3.0.0"
    info:
      version: 1.0.0
      title: Artist API
      license:
        name: MIT
    servers:
      - url: http://artist.spotify.net/v1
    paths:
      /artists:
        get:
          summary: List all artists
    ...

At Roadie, we have seen multiple customers with API kind entities in their Catalog with embedded OpenAPI docs as large as 1 megabyte in size. It’s easy for even a small-sized engineering organization to have 50 such APIs documented in Backstage. Unoptimized, that’s 50MB+ of data that’s being transferred every time we query the full Catalog.

This Catalog size is important because poor use of the Catalog APIs can cause huge database queries and API response sizes to result, which will cause both unwarranted traffic across the network and unwanted wasted time transferring, encoding, and decoding that data.

How to Make Good Use of the Entity Catalog

With all this said, we wanted to run through some good practices that are going to help with improving the Backstage experience. The examples we use in the tables below are measured in the browser on a production Catalog that has 14k entities.

Unoptimized, we’re looking at 2.16 seconds and 59.5 MB of data. That’s our starting point. Now each experiment we do below is going to improve that data.

Only query the entities fields that you need

By default, when retrieving entities from the Backstage Catalog, Backstage will return the whole entity for each item listed. As mentioned above, an entity might be as large as 1 megabyte. As such, limiting fields requested to the ones that are strictly required can help a lot. For example, you might have code like the following that is requesting every entity in the Catalog and then converting the result into an array of entity references.

(await catalogClient.getEntities()).items.map(stringifyEntityRef);

If you look under the hood, you’ll find that the stringifyEntityRef function only makes use of the kind, name, and namespace. As such, we can cut down on the amount of data transferred across the network by limiting the fields that are requested.

(await catalogClient.getEntities({
  fields: ['kind', 'metadata.name', 'metadata.namespace']
})).items.map(stringifyEntityRef);

Make use of the filter option to retrieve only entities that you need

We have seen a pattern develop whereby the Catalog client is used to retrieve all of the entities in the Catalog, and then the list is filtered client side.

(await catalogClient.getEntities()).items.filter((entity) => entity.kind === 'Group');

It is more efficient to send a filter to the catalog client so that filtering is done either in the Backstage backend or the Backstage database.

(await catalogClient.getEntities({ filter: { kind: ['Group'] } })).items

Avoid retrieving all entities in order to count entities

A common pattern we see in Backstage is for developers to download the whole contents of the Catalog in order to count the entities in that Catalog. The following code will cause Backstage to query the database for every entity, then the client will need to decode the JSON it retrieves in order to count the number of entities.

(await catalogApi.getEntities({})).items.length

There is a far more performant way to do this, using the query API. The following requests a limit of 1 entity to be returned, and also requests that the uid field from the entity is the only item that is returned for that entity. The query API always returns the total count of entities for that query. As such it gives us what we need with out downloading the whole Catalog to the client.

(await catalogClient.queryEntities({
  fields: ['metadata.uid'],
  limit: 1,
}).totalItems;

The change suggested here is going to save work for the Backstage database, the Backstage backend, and the Backstage frontend.

Enable Gzip Compression

When not using the Catalog Client, we recommend using gzip encoding to reduce the amount of data transferred. This is crucial because requests for large amounts of data directly from the Backstage APIs can be massive. Enabling compression significantly decreases the data volume sent to the client. You can achieve this by including the Accept-Encoding header with your client requests.

Keep Backstage up to date

The first, and perhaps the most important thing to consider is to keep Backstage up to date. If you are using Roadie, you are already using a very recent version of Backstage. However, if you are managing Backstage yourself, you may have fallen behind. Backstage releases new versions at least once a month, and these versions often contain very valuable performance improvements to the Catalog.

For example, in version 1.6.7 of the Catalog client library, there was an optimization. Previously, the Catalog client would sort all entities before returning them to the caller. This is a nice, helpful utility until there are thousands of entities to sort. Often, it is not necessary or optimal to receive a sorted list of entities.

Collaborate on the OSS Backstage core project

As part of research for this document, we spoke with the core maintainers of Backstage, and there are some great ideas about how to continue to improve the performance. For example, it has been discussed that by default, the getEntities function should be replaced by an iterator object. That iterator would be used to page over the list of entities rather than retrieving the whole list.

As such, keeping up to date with Backstage releases will allow you to benefit from these performance improvements.

Conclusion

This article is illustrative of some of the performance gains that can be achieved, and your mileage may vary. We have not delved into the performance implications of these changes on backend memory and database query performance. However, we can say that these changes can greatly improve these items too. It’s difficult to quantify; however, at Roadie, we were seeing huge memory spikes and large garbage collections occurring in Backstage when the whole Catalog is queried. This is possibly due to the physical sizes of the entity Catalogs and the serialization and deserialization that occurs between the client, backend, and database.

We have shown that making use of some good patterns can result in a much improved load times for users. We have shown some examples where timings are reduced from multiple seconds to sub-second. We have also shown that the sizes sent across the wire can be greatly reduced from multiple megabytes to tens of kilobytes.

A well-managed and optimized internal developer portal can make your software engineers more efficient and empower them with the information they need. When load times are reduced from multiple seconds to sub 1 second, developers enjoy a fast, responsive experience that means they’re more likely to use Backstage and find what they need.