APIs and applications

Here’s a conundrum that I’ve been running into:

APIs

On one hand, a company like Val Town really needs well-maintained REST APIs and SDKs. Users want to extend the product, and APIs are the way to do it.

The tools for building APIs are things like Fastify, with an OpenAPI specification that it generates from typed routes, with SDKs generated by things like Stainless or so.

APIs are versioned - that’s basically what makes them APIs, that they provide guarantees and they don’t change overnight without warning.

Applications

On the other hand, we have frameworks like Remix and Next that let us iterate fast, and are increasingly using traditional forms and FormData to build interfaces. They have server actions and are increasingly abstracting-away things like the URLs that you hit to submit or receive data.

Though we have the skew problem (new frontends talking to old versions of backends), there is still a greater degree of freedom that we feel when building these “application endpoints” - we’re more comfortable updating them without keeping backwards-compatibility for other API consumers.

Other contraints

• dogfooding is essential: if your company doesn’t use your own APIs, it probably won’t catch bugs or notice areas for improvement in time.
• Performance is essential: we don’t want to make a lot of network hops when dealing with vital functionality.

Solve for X

So how do you solve for this? How do you end up with a REST API with nice versions and predictable updates, and a nimble way to build application features that touch the database? Is it always going to be a REST API for REST API consumers, and an application API for the application?

• At Val Town, we currently have a Remix server that powers the application, and a Fastify server that powers an API interface to the content. There are many places of duplication: you have specifically-optimized routes in Remix to show filtered views of vals or to create new resources, that are designed to fit into the application and its error handling. And you have routes in Fastify that expose the same sorts of resources, but are designed with more general patterns in mind. Sometimes they share database queries under the hood, but they are still separate.
• Companies like Stripe and (ha, to a lesser level) Mapbox build applications on the same APIs that they expose to the public, I think. For Mapbox, this meant that those API endpoints were pretty slow to update and held some applications back. Stripe apparently discovered the secret to fast API evolution while maintaining backwards-compatibility, but that kind of magic is not commonly available.
• We could hit the Fastify server from the Remix server, the “backend for frontend” pattern, but this would introduce some network hops to each request, and some adaptation from one to the other’s request shapes - forms in Val Town use FormData, mostly, and the Fastify server uses JSON. This seems a little complex, and if there was a new API feature in the Fastify server, we’d have to deploy it, generate a new TypeScript SDK with Stainless, update the SDK in Remix, and then implement the new request: I do not think this is a nimble enough pattern for an early-stage startup to move fast enough.
• We could support Remix’s API as the REST API. But this seems deeply weird: does anyone generate OpenAPI specs from Remix routes? And the URL structure of our website is meant to be user-friendly, whereas the URL structure of the REST API is meant to be explicit, computer-friendly, and to use nesting structure in a pedantic way.

In short, I’m not sure! We’re taking a some-of-each approach to this problem for now.