Integration tests between bitcoin impls and krux

[qlrd]

Last months i was working on idea of run embit through an integration test with a bitcoin-core on regtest. Built some experiments with a pet-project[^1] through studying the floresta's test framework that @joaozinhom made[^2] (a impressive refactor on previous test framework).

It have a nice proposition (correct me if is incorrect): use pytest fixtures to run daemons and pytest will deal with threading. So i glued it on that on https://github.com/qlrd/bornal as flags on pytest.

While i do not want to replace any custom framework on any project, neither claim what was already did (is FOSS anyway) (EDIT: but) i faced with a "make plumbing problem" -- a minimal compilation, correct logs/data folders, thread safety between many nodes[^3] (mostly because multiple IO on data/logs), and other challenges (before available local ports) -- before real integration tests. So i think it's worth to share what i learned in floresta to krux ecosystem.

Saw some events currently on our repo and don't wanna bloat the project with a tests/integration folder on repo, so IMO this could be added as a selfcustody/krux-integration-tests using that bornal as module. WDYT @odudex @jdlcdl @kkdao @bitcoisas (and joao)

--

[^1] https://github.com/qlrd/embln.
[^2] https://github.com/getfloresta/Floresta/tree/master/tests
[^3] I'm against do it with pre-built binaries but understand who choose that, it's boring and have a CI pumbling on cache too (i expect to solve with some GH action)

[odudex]

The scope is not clear for me, will it test Krux or Embit? Can you define a clear outcome/goal of the project?
How the proposed extra tooling and repo can achieve that goal, and current tests can't? Will it be more powerful, efficient or easy to maintain?

[qlrd]

Thxs for the concerns @odudex. I will try to answer them one by one in a summarized way:

The scope is not clear for me, will it test Krux or Embit?

For now, just krux.

Can you define a clear outcome/goal of the project?

I will try not be verbose, but have some

• to code some of the daily user routines with bitcoin reference implementation
• to code some of routines made by @jdlcdl in some gists and examples
• to explore possibilities with lianad,florestad, with simulator (hope someday with a headless sparrow)
• pedagogical for newcomers (i have some personal plans with that)

How the proposed extra tooling and repo can achieve that goal, and current tests can't?

I don't think current tests couldn't do what those proposition propose. I mean, starting from a "local lab model" POV, we code our tests given our material conditions during the procedures of what
we think/know/verified as correct, on firmware, at test code.

Although we could use the reference implementation outside of the code, the reference implementation isn't in the code itself (direct/indirect references). I don't know if i make myself clear and hope you guys can correct me on what is wrong).

Will it be more powerful, efficient or easy to maintain?

I don't think it will be more powerful or efficient; my aim is somewhat like "this firmware is compliant to reference(s) implementation(s)".

I would risk a "efficient" in sense to separation of concerns (this is why i proposed a separated repository -- a selfcustody/integration-tests-like, not the bornal one). Like krux-installer, i would like to give it to the organization property.

About the "easy" part, IMO yes in the POV of separation of concerns, but no in the sense that could burden the maintenance of selfcustody organization (this duality is the reason to open this discussion).