Imagine its's the year 3000 (😂) and API version 1.0 is out. How would you like it to be?
As I recently mentioned in a comment on DEV, designing APIs is hard, don't believe anyone that will tell you it's easy :-D
An API has many stakeholders, here some of them and some of the questions I started asking myself when I started thinking about API v1:
those who host and serve content of the platform: How resource intensive is the API? Does its performance holds by scaling traffic? How costly it is? Will all Forems support the same version of the API at the same time? What happens to performance if a small Forem suddenly gets a very popular integration that uses its API? How do we keep track of usage of all API endpoints on all Forems?
the machines (not just the end HTTP client a developer writes, but the internet is full proxies, caching servers and others): Is it discoverable? Is a public schema available? What's the deprecation policy? Will the software automatically know if something is deprecated? Can a piece of software navigate it without human input?
the developers: Is it documented? Is there a developer community around this API? Is the content available to all or needs authentication? What type of authentication? Why just one type? Is the client code autogenerated by the specification? Do I have to provide personal identifiable information to get access to this API? Is it security audited? Is a SLA available? How does versioning work? How forward thinking the API is?
the system integrators: *How do I make "my" users interact with that API without them being users of that Forem? Is access centralized or decentralized? How do I get support?
The API world is full of opinions and full of many, many, keywords: resources, API evolution, HTTP, RESTish, REST, GraphQL, JSON schema, OpenAPI, SOAP, RPC, authentication, authorization, webhooks, web sockets, Cap’n Proto, Protocol Buffers, binary formats, pub/sub, streaming and many, many more.
We'd like to understand how you'd envision a future API to work, not just in the sense of formats and protocols.
What does this API should enable you to build? What does the API should enable people to build, for non technical users in other Forems?
Bonus
I have hints of ideas (totally not fully formed) for the year-4000-maybe-API-v2 as well: should the API allow cross Forem requests? Let's say you're authenticated on Forems A, B and D. Should, with some discoverability and auth wizardry, an API call on Forem A be able to retrieve your content on B and D? How?
Top comments (4)
Consistency! 🤣 You knew I would come across this post at some point and bring up the various API inconsistencies. (Yes they are all basically the same inconsistency but still! 😂)
At its core, I'd want an API to feel consistent throughout. This goes beyond my previous comments about "tags" vs "tags_list", this is stuff like how "/articles/me" vs "/articles/me/published" (where two differently named endpoints return the same data), to differences in default page size for pagination (articles with video are 24, articles without are 30).
I've recently been integrating with Stripe for a project and I have been loving their API - it feels slick and quite natural to work with. Every entity has the same endpoints available to cover all CRUD operations.
With Forem's API, take for example Videos - from my point of view, videos are posts with a video yet there is an entirely separate API endpoint for retrieving posts with a video. I feel like it would be better to support query string parameters to drill down what I want from the API. You have Listings API "/listings" supports category filtering by query string BUT I also see you have "/listings/category/{category}" which does by path. I get mixed messages about what one is actually best to do it by.
I feel like that is enough of a rant about this topic, now to move onto some of the other bits you raised:
I think integrations are a big ticket item here. Forem can't be everything for everyone but an API strategically built around supporting certain integrations would be a big step forward.
Those integrations could be connecting Forem's chat to a third-party solution (a la Slack). It could be web hooks for emailing certain people based on the tags of a post. It could be unified authentication for more enterprise-ish deployments.
An API that can do everything a user can do (to a point) is a HUGE but ideal end goal in my opinion.
I probably see more integrations with existing websites and social media. Its getting posts here to auto-share on Facebook or Twitter. It is getting relevant content to the right people.
I haven't messed around with the OAuth flow but if it is anything how GitHub does it when authorising an application, you could build very slick tools for non technical users.
YES. The first natural approach to me would be having a unified authentication system like how Stack Exchange does it for all of their sites. That said, it puts Forem in a weird spot like the arbiter of authentication which you may not exactly want. (Unless maybe you allow anyone to stand up their own authentication system so they can having their own communities of Forem sites).
Alternatively you could essentially have Forem itself authenticate with other Forems in the same way we authenticate currently with GitHub or Twitter. Without a directory of some sort for the code to look it up by, you might need the user to supply the domain to it. Could have a dropdown list or something available if the site admins have configured "likely" ones you might want to link.
Once done (and permission is given for apps etc to use it), you could actually have the browser extension for switching between Forem instances exist in-application instead. Because the application would be authenticated with other Forem instances, an API endpoint that allows retrieving basic metadata information (name, icon, URL, number of notifications/chat messages), you'd have everything nicely together. Probably could sync themes and other settings between instances (except maybe Bio - might be worth following Stack Exchanges lead here and have Bio be optionally configurable per site or the same across all sites).
Yes, that's definitely going to be a pillar in API v1 :-) That's why I'm thinking of employing a design first approach.
We vibe on that, Stripe is one of my favorite examples of well designed REST APIs!
Thanks for this feedback!
Interesting. Yes, designing an API isn't just about serving different resources with a new format. It's also thinking about how authentication can work from machine to machine as you say. Especially in the context of Forem's architecture which is going to be decentralized by definition as it's not a multi tenant architecture.
So, mainly the ability to register webhooks in a pub/sub manner if I undestand correctly. Something like (using pseudo language):
Correct? I have a question here: wouldn't that be already be accomplishable by RSS and IFFFT or Zapier?
I'm mainly asking this to think out loud what's the right balance between having a giant behemot that can do everything (potentially million of webhooks to be activated daily) vs being a good citizen to other platforms that can integrate with Forem.
I see OAuth 2 by itself a bit limiting as it requires the app's registration and if we'll end up with decentralized identities (James Turner being on Forem.dev and James Turner on DEV are already 2 different users, even if behind them there's the same human, the two installations don't know that) we need to decentralize auth as well. This way, don't even know if that's technically feasible yet, these potential apps can follow the user across Forems if authorized to do so.
I guess we're talking more about auth here than the actual API but I'm glad, as I see them complementary in a sense, though the scope of API v1 will inevitably become bigger.
I have been thinking about the same a bit lately. If it's still the same, StackExchange's architecture, has a centralized authorisation system and they don't allow self installs so this greatly simplify their authentication. I would imagine they have a central identity store where "rhymes" has a universal ID and each StackExchange website links their identity to that user when authorized to do so. This way "rhymes's identity" can move freely from one site to the other.
Our system is going to be different. As you can infer from A big day for Forem Systems written by @jdoss , each Forem is a unit by itself. At the present moment they don't know about the others. So, by speculating a future where they do, I would imagine none of them to hold "the entire history of me" (quoting Black Mirror here is not casual :D) but to hold a piece of it.
There are systems like IndieAuth designed to solve this issue of not having websites and apps have to register to each other in advance but I'm still studying all the details to understand it better. Do you have any experience with that?
That said, a possible API would need this mechanism to be already in place I guess to be able to trigger a query of "get all posts by rhymes from forem.dev, letsbuild.gg but not DEV published since January 2020" for example.
I feel like we're crossing between API and product design but I like your ideas! If it's truly decentralized, what's the authorative version of rhymes? Is it my profile on DEV? My profile on letsbuild.gg ? My profile on a future Forem? Do I get to choose? Does the machine choose for me? Would there even by a page where I can see all my profiles?
What if we want to build an ecosystem where I can be the same human, with different usernames and profiles and don't want them to communicate because I don't want my employer to know that I have a profile on Forem A, B and C ? Thus I don't want the API to follow me there as well?
Keep in mind that Forem needs to be a safe place, so people need to be able to delete their data, to stop sharing data with an app, to not let the owner of one Forem know they are a user on another and in theory they should be able to have completely different identities (email and username) on different Forems.
Alas, I don't have all the answers yet, but thank you so much for this feedback! I hope we can continue this conversation and that others will join!
Yeah, I'm thinking more that Forem brings the tools that others can build these experiences rather than Forem do it itself. So with webhooks, it would be allowing relatively fine-grained webhooks to be built and linked to. You don't want to be on the receiving end of hundreds of posts a second but you might want to for say, a specific tag.
Yeah, my thought for this part was less technology per-se but more experience. The GitHub auth-flow experience feels really nice when using it. Having a non-technical user go through something that felt like that experience when connecting apps/integrations would be ideal in my opinion.
Yep, exactly - I don't think you'd want to for security/privacy reasons let alone just raw data reasons. It would be a pain to manage.
I haven't used it but it seems relatively straight forward conceptually. Take a URL (any URL) and identify the authorization server behind it (which it must have, it can't be any old link), from there you follow an OAuth-like process with a sprinkle of additional URL-checking at the end. That's my take from it anyway.
Besides the discovery portion (which could be useful for adding additional authentication providers like how Forem has Twitter & GitHub), it doesn't seem to bring much more to the table.
Yeah, that's my "ideal" view if this functionality was here tomorrow. Just from that written example though, my brain says "looks more GraphQL-esque than REST" - that isn't a problem per-se but I didn't see it that way when I wrote my previous comment - brains are weird. 🤷♂️
Glad you like them! Yeah I know it isn't API-related directly but is an interesting use case that I felt was a natural benefit of such functionality.
Here is a crazy thought - why do we need an authorative version of a profile? I don't think a single authorative version is needed but I do think somewhere that you can view these linked accounts across Forem instances would be fundamental.
Given every Forem instances has the same core-code, every instance could technically have this page and do a "network" query across your linked accounts across different Forem instances.
With that in mind, I guess it is a case of whether you link profiles or not. I don't see a way to link profiles together and not share information to the various linked parties. You wouldn't really gain anything out of it if the goal was privacy between the different Forems.
Without knowing what the code behind a specific instance of Forem is doing (knowing that people could, in theory, modify it to do custom stuff), the only way for the privacy to exist is to never link between them.
I completely agree about different emails and usernames on different Forems. They shouldn't be a technical (or otherwise) reason to not allow that while still supporting linking of profiles.
1000% agree there would have to be procedures in place to delete their data and stop sharing it with an app. It would be kinda hard to not let an owner of one Forem to know your identity on another Forem if you linked profiles etc together but otherwise, yeah - privacy and user controls need to be easily accessible and usable for anyone on any Forem instance.
Look, I'm happy to talk about this stuff all day and bounce ideas etc off of. Feel free to reach out at any point. 🙂
Yeah, we agree on that. No centralized data store ;-)
Yeah, still have to go through the entire spec to see if it's enough or not and/or if it's flexible enough to be extendable. The jury is still out on IndieAuth. I also wonder what we can borrow from ActivityPub and if would make sense for us to become a server for it.
Interesting. I also think of API Gateways that are able to stitch different microservices together to build a response and they predate GraphQL in a way. The difference here is that each "microservice" will be a Forem installation with the same exact code. The technology behind all of this is an avenue to be explored. I suspect how we build "discovery" is going tell us what the right architecture here is.
Agreed. Linking one another is an opt-in. Fundamentally a person could spend their online life using two Forems that never talk to each other and be fine with it.
Back to the discovery part ;-)
True that! As Forem can be modified and self-installed, the trust level by default should be share nothing, but then if you share nothing, as you say, what's to be gained to link them? Forem's code is AGPL which requires modifications to be published in theory. This might help. Another possibility is to trust only servers managed by Forem.com but that could limit the network effect.
Thank you!