feat(resolver): Add extra methods for token set operations

[AngelRionCervi]

This PR adds utility functions to the resolver to manipulate and filter resolved tokens.

Motivations

My team and I have trouble converting our codebase to use the resolver, one of the reasons is that we have to do some amount of filtering of the resolved tokens ourselves, which we could avoid when using legacy modes.

For example, for optimization reasons, we want to be able to filter out all the tokens common to every modifiers (meaning their value don't change), in order to create a common.css file, and then have tokens which values are exclusive to a modifier context in their own {modifier}-{context}.css files. This way, tokens with the same values across all modifiers are stored in one file, and we avoid having duplicated tokens in other files.

Changes

This PR adds 2 methods in resolver.extra:

intersection(options): returns all tokens that don't change value across modifiers

symmetricDifference(options): returns all tokens on which the value changes across modifiers

How to Review

Check the added tests in packages/parser/test/resolver.test.ts and verify that they pass

---

This is very much an idea open for suggestions 👍

[changeset-bot]

🦋 Changeset detected

Latest commit: 2a6da5b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@terrazzo/parser	Major
@terrazzo/plugin-css-in-js	Major
@terrazzo/plugin-css	Major
@terrazzo/plugin-js	Major
@terrazzo/plugin-sass	Major
@terrazzo/plugin-swift	Major
@terrazzo/plugin-tailwind	Major
@terrazzo/plugin-token-listing	Major
@terrazzo/plugin-vanilla-extract	Major
@terrazzo/cli	Major
@terrazzo/token-tools	Major
@terrazzo/token-types	Major
@terrazzo/use-color	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[drwpow]

This is awesome! 💯 I totally see the value in this addition, and am in favor of adding.

Before I approve though, would love your thoughts on the API change, and otherwise would like to see the new tests split out into their own fixture to make sure all the changes are 100% additive and backwards-compatible. But overall love the direction and thought here.

[drwpow]

Suggestion: I love this ability to filter tokens based on whether they are the source of a value or not! But could we adjust the API a little bit, and reach the same end result, but allow for more usecases?

/**
 * Filter tokens based on specific criteria
 */
filter?: (token: TokenNormalized) => boolean;

So for example, for 'primitive', you could do:

filter: (token) => !token.aliasOf?.length

And for 'alias':

filter: (token) => token.aliasOf?.length > 0

These should fit y’alls usecases. But I’m also thinking of other design systems I’ve come across where they really use namespacing in a significant way, and want to do something like:

- filter: (token) => !token.aliasOf?.length
+ filter: (token) => !token.aliasOf?.length && !token.id.startsWith('deprecated.')

Open to thoughts! My only hesitance with this API as-written is not the practicality of it, it’s more that the idea of “primitive” tokens isn’t a concept or verbiage that exists elsewhere in Terrazzo. But the idea of a configurable function is a pretty common pattern, that we can provide some “recipes” like this that equate to the same things.

[AngelRionCervi]

Ah yes ! I agree it's better, I will make that change 👍

[drwpow]

2 things:

1. we could just add to the object here, rather than creating a duplicate object in memory inside addResolverExtras
2. I don’t mind everything on .extras just being part of the core API here 🙂. resolver.intersection and resolver.semanticDifference are a lot cleaner

[drwpow]

Change: rather than change this test fixture, can we just copy this and fork it into a new one? I can see that a lot of the changes aren’t a huge departure from the original, but there are lots of little changes here I want to make sure the original tests don’t have any hidden mini-regressions here.

Even if there’s some bit of test duplication, that’s OK 🙂 I don’t mind cleaning up later

[AngelRionCervi]

Thanks for the review and for taking the time to explain your points !

I pushed modifications based on your comments but I also converted the PR to a draft to add the methods and a couple of examples to the doc if that's ok

[AngelRionCervi]

Added the doc
PR now open for review 👍

edit:
cc @drwpow ^^