Investigate how to handle deprecated API endpoints

[jonbulz]

Motivation

We currently don't use versioning for the API. Thus, changes to endpoints might make the API incompatible with older app versions without warning. This mostly affects mobile app users, since the web app can be updated and deployed in sync with the API.

Proposed Solution/Scope for Research

Timebox: 4 hours
Outcome: Recommendation on:

• need for implementation (if no, they next two outcomes can be ignored, but reasoning for decision needs to be documented)
• which of the two solutions outlined below should be implemented
• new issue for said implementation:

Two possible solutions have been outlined so far:

• introduce versioning to the API and alert/force users to use compatible versions
• keep endpoints backwards compatible and only change/remove endpoints after a long time period

Process: Form opinion, coordinate meeting with app-team to determine need for implementation, record results of meeting

Alternatives

Leave it as it is and hope that users regularly update their apps or find a better solution.

User Story

We as developers want to find a straight forward and concise way to handle changes in the API.

Additional Context

Design Requirements

Related Issues

[JoeyStk]

keep endpoints backwards compatible and only change/remove endpoints after a long time period

this solution is probably not feasible for us and we would need a different solution

[JoeyStk]

Discussion about this on-going, @jonbulz / @PeterNerlich

[PeterNerlich]

I think I'm beginning to wrap my head around it and how to best do this in integreat-cms to satisfy the following goals:

1. We want to provide an interface for the app to fetch content and (in our case in much smaller capacity) submit data back to us (e.g. feedback)
2. We want that interface to be stable and backwards compatible, since there is no guarantee that users update their app in any reasonable time frame
3. At the same time we want to retain the ability to add new functionality to the API and change the behaviour of existing endpoints
4. With potentially many different versions, we still want to minimize code duplication but also keep the code easy to understand where which endpoint is implemented
5. We want to facilitate easy and clear communication with the app team about development of the API, meaning new versions and what they bring in additions, changes or when versions are being deprecated

(I listed them here to have them out for discussion at least somewhere)
I have a hunch for points 1 (that's trivial, it's our primary objective and what we already satisfy) through 4, and part of 5. I think our indicated attempt at a versioning scheme with /api/v3/ is sufficient and a good base to build on. But speaking only to the last point as it is actually what this issue is about:

To deprecate an endpoint or entire API version, we should implement the HTTP Deprecation header (RFC 9745) (best through a custom decorator) and ask the app team to check for it and after detection, warn users (e.g. once per app startup and/or day) that they should update to a newer version soon or the app might not be able to fulfil some functionality.
Note that we did recently is decide that a single value in a JSON response for one endpoint was now deprecated, while all the other ones was still valid. That case is not covered by the Deprecation header – it would be a potentially breaking change to the existing API and thus necessarily spawn a new version into existence.

After deprecation, we should take care not to sunset the API version too soon. Instead, we should monitor the successful API calls made for that version and only decide to pull the plug when they have essentially vanished. For the very minimum time, 6 months between deprecation and sunset seems reasonable. If we work really quickly and decide to deprecate something we just created (a situation we will hopefully avoid), we will need to pay extra attention to where the app team stands with using it. If it gets used in an upcoming app release, or just to be sure, the deprecation period of minimum 6 months should start with the release of the next app version using a successor to the deprecated API and no day earlier.

[jonbulz]

@steffenkleinle Do you have an opinion? If necessary, we can also schedule a meeting

[steffenkleinle]

Good question, are we talking about some endpoint/fields in specific or more about how to handle (potentially) breaking changes?
I think there are basically three different options on the CMS side and which one should be applied depends a lot on the concrete case in mind imo:

1. Just break stuff (both whole endpoints and properties in endpoints, e.g. changing last_update in place from some custom date format to ISO8601)
2. Maintain both old and new versions/properties/behavior in a single endpoint/version (e.g. keeping the old last_update and adding a new display_date)
   a) Add some kind of deprecation to a property and/or endpoint
   b) No deprecation notice
3. Use a new API version and maintain both versions for a bit

We currently use all three approaches depending on the specific case and I think that actually makes sense:

1. Potentially breaks stuff but is easier to maintain since there is only one version to support: Should be used e.g. for non-breaking changes, features that are not yet implemented/live in the apps, if there was enough time for users to update (e.g. one year), or minor things that can be handled in the app with enough preparation time
2. Harder to maintain but still relatively easy to implement, duplication might be confusing. Difficult to decide and remember when to remove deprecated fields.
3. Major changes to the whole structure of the API (we currently use API version v3), difficult to maintain two whole API version but clean separation

So in summary, I think all three methods are valid and which one should be used depends a lot on the concrete case in mind imo. I think its not possible to have one general approach for all breaking API changes 🤔 I think we basically have to decide together for every API change which one to apply.

The only thing that we basically don't do at the moment is deprecation notices, both to properties nor to endpoints. However, I think by documenting that clearly in the API specs together with clear communication when you change behavior in the endpoints would be enough for us to react. I think adding deprecation headers or similar is overkill and more necessary if it would be an API intended for public use.

In the long term, we might look into Open API specs to clearly document the API.
What do you think? I'd definitely be open for a meeting as well :)