Query typegen for non-root gatsby-* files

[Js-Brecht]

New issue to track conversation started in #44, regarding query type generation when gatsby-* configuration files are located in a directory other than the project root.

Following is the conversation thread up to now.

[Js-Brecht]

I just realized that I have to deal with outside of src to support gatsby-node. Thank you so much @Js-Brecht !

Out of curiosity, how do you plan to handle that? I ask because gatsby-plugin-ts-config could potentially conflict with that process, because of Js-Brecht/gatsby-plugin-ts-config#2. If the path to gatsby-node is hard coded to the root of the project, then whoever uses gatsby-plugin-ts-config may not be able to use gatsby-plugin-typegen to generate types from queries in gatsby-node.

This won't be an issue if babel is used to do ahead-of-time transpiling, because it will be running the compiled gatsby-node out of a subdirectory in the cache (or somewhere similar), but if something is used that does JIT transpiling (like @babel/register), then gatsby-node will not be able to be stored in the project root without causing node ownership conflicts.

Perhaps this would be better as a separate issue? Do you have anything tracking this work?

Originally posted by @Js-Brecht in #44 (comment)

[Js-Brecht]

@Js-Brecht It is... hard to make it compatible with your project for now. (or please let me know if you have an idea)

The gatsby-* files in the project root, and the files in the src path and gatsby dependency, are what gatsby assumes for end-user projects and plugins/themes so it's easiest to follow that assumption.

I think you should talk more closely with the core team about that. It is likely to be included in the next breaking change.

Originally posted by @cometkim in #44 (comment)

[Js-Brecht]

It is... hard to make it compatible with your project for now. (or please let me know if you have an idea)

Would it be feasible to include an option allowing the user to define various additional files to pull types from? Perhaps that would be too labor intensive (and probably far too much maintenance), since I think it would require implementing Gatsby's query extraction apart from Gatsby itself. Correct me if I'm wrong.

The issues with node ownership will actually be resolved when Gatsby is able to read Typescript gatsby-* files natively, because then it is no problem keeping those files in the project root.

I think you should talk more closely with the core team about that. It is likely to be included in the next breaking change.

Will do, thank you

Originally posted by @Js-Brecht in #44 (comment)

[Js-Brecht]

Would it be feasible to include an option allowing the user to define various additional files to pull types from? Perhaps that would be too labor intensive (and probably far too much maintenance), since I think it would require implementing Gatsby's query extraction apart from Gatsby itself.

I wonder if that could be as simple as using Gatsby's FileParser, like they do in the parseQueries function, to gather queries from custom file paths 🤔. Since executing the queries isn't necessarily an issue, running the FileParser should, at most, cause your GraphQL parser to run, because it fires the redux action.

The queries won't be watched for changes, but perhaps that could be arranged another way.

In the recent version of gatsby-plugin-ts-config, if the default export of the user's gatsby-config or gatsby-node is a function, then one of the parameters will be the collection of resolved user gatsby-* files. That would make it easy for the user to feed those particular files into your plugin for query typing. It could be left as just that basic collection, but it would be trivial to extend it to include the files that are require()'d by those files, too (in case they also include queries that aren't already parsed by Gatsby).

Originally posted by @Js-Brecht in #44 (comment)

[Js-Brecht]

I just took a moment to read your code, instead of just skimming it, and I see you are already using the FileParser. So, I was just getting ahead of myself there ☝️

Originally posted by @Js-Brecht in #44 (comment)

[Js-Brecht]

Would it be feasible to include an option allowing the user to define various additional files to pull types from? Perhaps that would be too labor intensive (and probably far too much maintenance), since I think it would require implementing Gatsby's query extraction apart from Gatsby itself. Correct me if I'm wrong.

It can be easily done using graphql-tag-pluck. (Even I have a simple RegExp which can do almost the same thing as the Gatsby's file parser)

I just took a moment to read your code, instead of just skimming it, and I see you are already using the FileParser. So, I was just getting ahead of myself there

Anyway, as you saw, I switched to using Gatsby's action and the file parser instead of watching the user directory to extract the queries directly. Removing the custom watcher process was a big goal for v1, so it's hard to revert it at this moment.

It seems off-topic. @Js-Brecht Should we have another issue for this?

Originally posted by @cometkim in #44 (comment)

[Js-Brecht]

Removing the custom watcher process was a big goal for v1, so it's hard to revert it at this moment.

No, no, I wouldn't ask you to revert changes/improvements you've made. I was just referring to Gatsby's query watcher. These queries wouldn't be watched by Gatsby, but since they're located in configuration files that only really get processed during the build, then the behavior won't be any different than what people are already used to. Perhaps eventually there will be support for watching changes in config files.

It can be easily done using graphql-tag-pluck. (Even I have a simple RegExp which can do almost the same thing as the Gatsby's file parser)

If you're willing to accept an array of file paths that you can scan for queries, it shouldn't be too hard to accomplish this, yeah? Like I mentioned before, gatsby-plugin-ts-config already passes the resolved endpoints to the user, so they could take them and pass them right into your plugin as an extra include option, or something like that.

I can document the process in my plugin, because it would be a good example for showing how to use that particular bit of data.

The general (simplified) process would be like this:

1. /gatsby-config.js

module.exports = {
    plugins: [
        {
            resolve: `gatsby-plugin-ts-config`,
            options: {
                configDir: `foo`
            }
        }
    ]
}

2. gatsby-plugin-ts-config/gatsby-config.js

require(`${projectRoot}/foo/gatsby-config`).default(resolvedEndpoints);

3. /foo/gatsby-config.ts:

export default ({ endpoints }) => ({
    plugins: [
        {
            resolve: 'gatsby-plugin-typegen',
            options: {
                include: Object.values(endpoints)
            }
        }
    ]
})

That endpoints object looks like this, for now:

type IConfigTypes = 'config' | 'node' | 'browser' | 'ssr';
type IGatsbyEndpoints = {
    [k in IConfigTypes]?: string;
}

But it can be changed if it needs to be to make it easier for the user to consume it. Especially considering it might be worth including any files that are imported into those endpoints as well.

So, say gatsby-node does:

import ExecuteQuery from './utils/query-handler';

that query-handler file could also be checked for queries rather easily, because it could be included in the resolved endpoints collection. The endpoints collection that gets passed to the user could very well just become an array of files, instead of an object. The user doesn't really have a lot of use for knowing what is config, and what is node. They just might want to tell other plugins what files are being used by their gatsby-* files, like the case here.

[cometkim]

In theory, this plugin will start track of new queries when they are notified by the FileParser.

I'll be doing some PoC soon.

[Js-Brecht]

In theory, this plugin will start track of new queries when they are notified by the FileParser.

Should, but I was hoping to not have to use FileParser in ts-config plugin, because that exceeds its scope. What would be great is if we could tell Gatsby where to pick up additional files and look for queries in them 🤔.