Skip to content

tooling for generating upstream PRs (including exportify) - #20

Merged
missinglink merged 2 commits into
masterfrom
exportify
Jul 8, 2020
Merged

tooling for generating upstream PRs (including exportify)#20
missinglink merged 2 commits into
masterfrom
exportify

Conversation

@missinglink

@missinglink missinglink commented Jul 8, 2020

Copy link
Copy Markdown
Member

This PR contains a bunch of commands required to author this PR, these make generating patches to WOF 💯 times easier 🎉

Summary:

  • new command wof fs import <dir> takes a stream of WOF features and writes them to a filesystem, it creates the nested directory structure if required and optionally runs the WOF 'exportify' tool via docker, this makes them ready to merge upstream 😱
  • new command wof feature exportify, same as above except it's streams-in & streams-out (basically just a convenient way of running 'exportify' which supports streams)
  • some changes to the wof sqlite fix <db> command:
    • it occurred to me that we may add more fixes in the future and need a way of running them independently, so --hierarchies is now false by default and developers will need to explicitly specify which fixes they would like to apply
    • added --cat flag which prints any documents which were fixed to stdout
    • added --dryrun flag which prevents the DB from being updated, this operation takes time so it's convenient to skip it when using the --cat flag in some workflows

So the cool thing about all this is that it's now possible to write a through2 stream to update WOF records and then write them back to the repo ready for a PR!

It's even possible to import them to a SQLite db and use the rest of the collection to fix integrity issues and again write it back to the filesystem ready for a PR 🧙

Eg:

# clone repo (fa984290faaef47d83c795159558cf7ee5560ca4)
git clone https://github.com/whosonfirst-data/whosonfirst-data-admin-ca.git

# convert to sqlite db
wof git export whosonfirst-data-admin-ca \
  | wof sqlite import ca-fix.db

# fix hierarchies, print updated records to stdout, exportify them and write them back to the filesystem
wof sqlite fix --hierarchies --dryrun --cat ca-fix.db \
  | wof fs import -v --exportify whosonfirst-data-admin-ca/data

@missinglink

Copy link
Copy Markdown
Member Author

cc/ @Joxit I think you're going to like this 😄

…specifying which fixes to apply manually eg "--hierarchies"
…mport" command to write files a stream of features to the filesystem
@missinglink
missinglink merged commit ec51cc2 into master Jul 8, 2020
@missinglink
missinglink deleted the exportify branch July 8, 2020 10:37
@missinglink

Copy link
Copy Markdown
Member Author
npm publish
+ @whosonfirst/wof@0.27.0

@Joxit

Joxit commented Jul 10, 2020

Copy link
Copy Markdown
Member

Yeah ! That's pretty cool !
In fact I already wrote something for exportify 😅 I can run the exportify on all files for a specific commit or all stagged files.
I'm trying to rewrite the full exportify in rust but it's a pain... 😓

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants