feat: add custom walker support by tobim · Pull Request #698 · numtide/treefmt

tobim · 2026-05-15T06:45:01Z

This adds configurable custom walkers. A custom walker runs a configured command, reads newline- or NUL-delimited paths from stdout, normalizes them relative to treefmt's tree root, and feeds existing regular files into the formatter pipeline.

This PR is stacked on #705, which adds the shared path stream reader used by custom walkers.

Example: Format a git repo + a specific submodule:

walk = "repoPlusSubmodule"

[walker.repoPlusSubmodule]
command = "bash"
options = [
  "-c",
  '''
    submodule="path/to/submodule"
    {
      git ls-files --cached --others --exclude-standard --full-name
      git -C "$submodule" ls-files --cached --others --exclude-standard --full-name \
        | sed "s#^#$submodule/#"
    }
  ''',
]

Example: Format a pijul repo:

walk = "pijul"

[walker.pijul]
command = "pijul"
options = ["ls"]

jfly

@tobim, thanks for the contribution! I am supportive of this change, but haven't had a chance to read the code yet.

I wanted to warn you that #694 touches some of the same code you've touched here. It may land before this PR, apologies if the conflicts are annoying to deal with.

jfly

Thanks! This looks like a solid approach.

Not a blocker, but I'm not in love with a 3rd copy of the walk code. I didn't read it particularly closely. I think 2 copies (jj and git) is OK. 3 is when it probably makes sense to to think of an abstraction. Did you think about that at all?

Accept documented camel-case walker names even when Viper normalizes table keys

What is camel-case? That's kebab-case. This is camelCase (or CamelCase).

Fix custom walker pipe handling

I did not have time to grok this commit during my review. I'll need some more handholding before I can approve it. Does whatever bug you're fixing here apply to the git and jj walkers as well? Is it related to this bug @Mic92 found in #694 (commit labeled "walk: unblock producers when Close() is called before EOF").

jfly · 2026-05-27T04:46:19Z

+When you pass directory paths to `treefmt`, the walker command still runs for the tree root.
+`treefmt` filters the command output to the requested directories.
+The walker command doesn't need to implement path argument handling.


While I appreciate the simplicity of this decision, I'm not sure it's the right decision. I imagine people are most inclined to use the "format a directory" feature in very large repos, where the time spent for some VCS to determine which files are in the entire repo might be a lot slower than discovering just the files in a specific directory. #694 is an example of someone reporting just how slow it is for git to report all files in a repo.

How crazy would it be for the walk command to have to implement path argument handling? IIUC, it would be straightforward for git.

I admit that I didn't pay too much attention to optimizing for subtrees. It's no concern for my own use cases.

For a regular git walker this might not be a problem, but as soon as you want a more custom file list - like in the first example from the PR description - implementing the filtering yourself becomes error-prone rather quickly.

Maybe we'll add a toggle so the user can switch between both methods?

jfly · 2026-05-27T04:52:11Z

+# You can also set this to the name of a configured custom walker
 # Env $TREEFMT_WALK
 # walk = "filesystem"
+# walk = "myWalker"


I wonder if we should instead do walker.myWalker (or perhaps custom-walker.myWalker) here. Then we wouldn't have to worry about (or detect) conflicts with the builtin walkers.

jfly · 2026-05-27T05:14:37Z

-		reader, err = NewReader(Git, root, path, db, statz)
-		if err != nil {
-			reader, err = NewReader(Jujutsu, root, path, db, statz)
+	if selector.Custom != nil {


This diff would be a lot simpler, and the if selector.Custom == nil && selector.Type == Stdin change below wouldn't have had to happen if we introduced a new selector.Type == Custom. Did you consider that?

tobim · 2026-05-27T09:13:45Z

Thank you for the review. I hope to be able to address the individual comments in the next few days.

Not a blocker, but I'm not in love with a 3rd copy of the walk code. I didn't read it particularly closely. I think 2 copies (jj and git) is OK. 3 is when it probably makes sense to to think of an abstraction. Did you think about that at all?

I didn't really consider that because I wanted to keep the new path somewhat isolated.
We could also replace git and jj with built in custom walkers now that the machinery is there.

Does whatever bug you're fixing here apply to the git and jj walkers as well? Is it related to this bug @Mic92 found in #694 (commit labeled "walk: unblock producers when Close() is called before EOF").

I'll have to look into that.

jfly · 2026-05-27T12:55:18Z

I didn't really consider that because I wanted to keep the new path somewhat isolated.
We could also replace git and jj with built in custom walkers now that the machinery is there.

This sounds reasonable to me

tobim · 2026-06-01T08:06:35Z

So I did somewhat of a refactor of the walker component now. Now there is a generic PathStreamReader abstraction that consumes a stream of paths that are separated by either a newline or a NULL character. jj and git are now constructed in terms of that PathStreamReader.
After that are a couple of clean-up commits that simplify the state representation and related control flow.

jfly · 2026-06-01T20:12:15Z

So I did somewhat of a refactor of the walker component now. Now there is a generic PathStreamReader abstraction that consumes a stream of paths that are separated by either a newline or a NULL character. jj and git are now constructed in terms of that PathStreamReader.

I am excited to read this. I'll be travelling for the next ~1 week, though, and don't expect to be able to take a look until I'm back. Could you rework this change as separate PR we could land before we add the custom walker support?

brianmcgee · 2026-06-02T11:16:50Z

Been on holiday myself this past week or so. Just getting up to speed with this. I'm liking the direction, need to give it some proper brain time though, as it's a core change.

Document that treefmt does not support paths containing newlines before introducing command-backed path stream walkers.

Introduce a shared reader for walker commands that emit paths on stdout. Parse newline- and NUL-delimited records, normalize emitted paths inside the tree root, post-filter requested path filters, and convert existing regular files into walk.File values. Own the stdout and stderr pipes instead of using Cmd.StdoutPipe so the command can be waited for while Read drains stdout. Close cancels the command to unblock producers when callers stop before EOF, and treats post-cancel signal exits as expected.

Build the uncached reader in a helper and apply the cache wrapper once in NewReader. Keep auto fallback inside uncached construction so trying built-in walkers does not recurse through cache setup.

Store filesystem path filters as a slice and walk them sequentially in the filesystem reader. Use one filesystem reader for multiple requested paths instead of building one reader per path. Add coverage for multi-path filesystem walking.

Run git ls-files from the tree root and parse its NUL-delimited output with the shared path stream reader. Pass requested paths to git as path filters after -- so subtree formatting can prune the git output inside git.

Run jj file list from the tree root and parse its output with the shared path stream reader. Pass requested paths to jj after -- so paths that start with dashes are handled as path filters. Remove the generic CompositeReader now that all built-in walkers consume the same validated path-filter list.

tobim · 2026-06-07T14:55:35Z

I opened #705 to split of the refactoring. This PR should now be stacked on top of that, but GitHub is crappy and you can't do that.

When no explicit path filters are passed, run `jj file list` with `.` as the positional filter. In non-colocated jj workspaces, running from a subdirectory can otherwise emit paths outside the treefmt tree root. Add a regression test for the jj-only workspace shape from numtide#704.

Document selecting a custom walker with the global walk option and defining its command and options under [walker.<name>]. Document that custom walker commands run from the tree root, receive requested path filters as positional arguments, and emit newline- or NUL-delimited path records.

Parse [walker.<name>] tables into the configuration model and validate walker names, commands, and walk values. Treat custom walker names as exact configuration keys and keep the default walk value in NewViper.

Represent walker selection as either a built-in walk type or a custom walker configuration instead of passing only the built-in enum through the format command. Keep selector fields private and expose IsBuiltin for callers that need to test for stdin.

Implement custom walkers as a small adapter around the shared path stream reader. Run the configured command from the tree root and pass validated path filters as positional arguments after configured options.

Cover selecting a custom walker from treefmt.toml, passing configured walker options, and forwarding requested path filters to the walker command. Use sandbox-compatible bash commands, exact lowercase walker names, and the long-form --clear-cache option in CLI coverage.

This was referenced May 15, 2026

feat: Allow recursing into submodules when using git walker #507

Closed

Darcs walker #656

Open

jfly reviewed May 18, 2026

View reviewed changes

jfly reviewed May 27, 2026

View reviewed changes

tobim force-pushed the push-spzzmzrpkzzq branch from 7e5f3af to 69c5a22 Compare June 1, 2026 07:53

jfly mentioned this pull request Jun 5, 2026

treefmt-tree-root config option isn't respected in Jujutsu workspaces #704

Open

tobim added 5 commits June 7, 2026 15:17

Document unsupported newline paths

02fdb68

Document that treefmt does not support paths containing newlines before introducing command-backed path stream walkers.

Split walker reader construction

2f52fe6

Build the uncached reader in a helper and apply the cache wrapper once in NewReader. Keep auto fallback inside uncached construction so trying built-in walkers does not recurse through cache setup.

Let filesystem walker handle multiple paths

6c8fb0e

Store filesystem path filters as a slice and walk them sequentially in the filesystem reader. Use one filesystem reader for multiple requested paths instead of building one reader per path. Add coverage for multi-path filesystem walking.

Use path stream reader for git walker

f4b79f9

Run git ls-files from the tree root and parse its NUL-delimited output with the shared path stream reader. Pass requested paths to git as path filters after -- so subtree formatting can prune the git output inside git.

tobim force-pushed the push-spzzmzrpkzzq branch from 63f7764 to 63d350b Compare June 7, 2026 14:35

tobim force-pushed the push-spzzmzrpkzzq branch from 63d350b to 871c19b Compare June 7, 2026 14:47

tobim added 6 commits June 7, 2026 17:44

Add custom walker configuration

0db4f01

Parse [walker.<name>] tables into the configuration model and validate walker names, commands, and walk values. Treat custom walker names as exact configuration keys and keep the default walk value in NewViper.

Use explicit walker selectors

5ebf693

Represent walker selection as either a built-in walk type or a custom walker configuration instead of passing only the built-in enum through the format command. Keep selector fields private and expose IsBuiltin for callers that need to test for stdin.

Add custom walker reader

0757075

Implement custom walkers as a small adapter around the shared path stream reader. Run the configured command from the tree root and pass validated path filters as positional arguments after configured options.

tobim force-pushed the push-spzzmzrpkzzq branch from 871c19b to 4e1f4b8 Compare June 7, 2026 15:46

Conversation

tobim commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfly left a comment

Choose a reason for hiding this comment

Uh oh!

jfly left a comment

Choose a reason for hiding this comment

Uh oh!

jfly May 27, 2026

Choose a reason for hiding this comment

Uh oh!

tobim May 27, 2026

Choose a reason for hiding this comment

Uh oh!

jfly May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jfly May 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tobim commented May 27, 2026

Uh oh!

jfly commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tobim commented Jun 1, 2026

Uh oh!

jfly commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brianmcgee commented Jun 2, 2026

Uh oh!

tobim commented Jun 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tobim commented May 15, 2026 •

edited

Loading

jfly commented May 27, 2026 •

edited

Loading

jfly commented Jun 1, 2026 •

edited

Loading