Optimization - CI/CD

[nathanjhood]

Chore

Describe the chore

The CI/CD is all driven from the GitHub workflows files. Currently, I am maintaining one per target platform (current total 4: x64-windows, x64-linux, x64-osx, arm64-osx), due primarily to the subtle differences between platforms that make a generic workflow difficult to define. The primary issues include path handling, especially with regards to caching, restoring, and distributing build artefacts.

I am also driving builds and tests for two build configurations - Debug and Release. The Debug build is really handy to have, and quite conventional to distribute as part of a pre-built package from a package manager (vcpkg...). These are again currently specified uniquely within each workflow file, despite only one single var being different between both builds.

Clearly, we can benefit a lot from using matrixes for build types. I have only hesitated since I have been experimenting with using mutli-config generators (Ninja Multi-Config, etc) to speed things up.

Unfortunately, JUCE doesn't seem to play nice with the typical CMAKE_CONFIGURATION_TYPES var usually used to tell the multi-config generator which build types to produce. When using this setup, find_package(juce CONFIG REQUIRED) fails on the CI/CD runs... perhaps we could use a less restrictive call to find the juce CMake package, but this will probably open up a whole new level of issues which are currently locked down due to the strictness of the design, an integrity which I'd much prefer to maintain.

Matrixes can also work for the multi-platform handling, as per pamplejuce... however, I find that nice clear workflow files which resemble an actual human interaction with a project's required build steps to be very useful and informative, personally, when perusing other users' projects. Given the 'tutorial-ish' nature of the project, I quite prefer to have nice clear workflows I can point visitors to, so that they may understand the entire CMake configure/build/test/install/pack routine.

Spaghetti-like workflows are not only unclear and un-informative for project newcomers; they are also much more prone to failing for subtle reasons, and are much harder to debug.

For now, I have been very deliberate and repetitive in the workflow runs because the clarity is of much greater benefit than boiling it all down into some sort of yaml-flavoured delicacy which even I won't understand, when I revisit this project after the next long break...

I don't see that repetition as a bad thing, in the case of the workflows - DRY is more important within the codebase itself, whereas the workflows should work, and if well-made, will also be very useful to others.

However, I am currently generating multiple build caches per push. This doesn't feel responsible of me. There is also more work to be done to address version management, and logic to control whether or not any artefacts even should be cached (think validation).

Generally the CI/CD is working great now that I have done all of the dirty repetitive work, so I am not in a big hurry to go and break it all. But, every time I watch the thing run, I feel that this could be done much more efficiently and programmatically.

I have been able to regularize/generalize the less meaty parts of the workflows, particularly the CMake config/build/test/install part of the routine. The dependency resolution(s) and artefact caching/uploading are a whole other matter, and would likely still require a distinct job each, meaning that we'd have these various un-related pre- and post-build steps sandwiching a matrix... Very tricky because the workspace gets cleared between jobs... so, caching a buildpath-per-matrix-run scenario starts to arise... this kind of complexity makes me wonder if it would actually be preferable over the current implementation, or not...

This issue will be a dumping ground for further thoughts and ideas on the subject, before deciding on how to improve and optimize the current CI/CD. I probably won't make any further changes for the time being, until my thoughts on what are more fully established here.

[nathanjhood]

Started some CMake presets that deign to offer a valid 'default' that should run on all platforms, plus some different generators, options such as build type and verbosity.

I had quite overlooked how the provision of presets actually integrates into downstream user processes, such as CI/CD...

In theory, we can move all of the GitHub workflows into CMakePresets, and then easily just call the preset from the command line (perhaps using a matrix to cycle through target platforms and archs).

I hadn't realized how the "workflowPresets" in particular are literally what we're doing with CMake in the actions files.

One huge benefit of this approach is that we can be more assured that our CI/CD actually matches the user experiences - since we'd all be running literally the same file.

Even more benefits in the shape of IDE integration...

Weirdly, I've got Ninja Multi-Config working just fine via this method.

I am not sure how deep I will go with the presets as these multiply with each new option, and tend to require steady planning of inheritance. For example, providing an x86 set of configs along with the x64 config doubles the workload instantly.

As a rule of thumb: I've realized there really is very little need for any -D define args at all when using the presets, and ensuring that reasonable defaults are set for all CMake Project logic (e.g., always build tests, but only when project is top-level).

In a beautiful play by Kitware, we can boil CMake down into a much more universal command-line UX along these sorts of lines:

$ cmake --fresh --preset=linux-ninja-debug-verbose --toolchain=${VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake

$ cmake --build

# etc...

[nathanjhood]

Due to my absolute lack of attention span, I've created some CMake presets while trying to work on the parameters.

(#84)

They only apply to linux and don't offer any different compiler toolchain options (just generator and built type, plus a verbose build option). I was sure to establish a --preset=default for each level, which only makes the assumption of using Ninja, and specifies a build output directory. Everything else just takes (I assume) your CMake environment's defaults and uses that.

For most of the CI/CD, I think the default presets will probably be sufficient... Platform-specific edge cases will be really easily handled by wrapping the fix into a (non-default) preset.

I will re-focus on the parameter work, as I don't really have the means to be machine-hopping to create the full set of presets right now, but those initial Linux presets should be taken in when starting work on this issue.

[nathanjhood]

Considering attempting to wrap some custom commands-as-targets, for handling stuff like the vst validator and AU code signing. Should in theory be easy to add, for example, an AU codesigning step as a custom command that gets built (last - by 'depending' on the plugin target) as a target, for AU builds on macOS.

Raises a really interesting part of the project that I'd like to develop further for CI/CD - adding options to the CMake project for specifying which targets to configure and build. Perhaps a user (or CI run) doesn't need to build the standalone for ARM machines... I can offer options for this.

They should use CMake's "dependent" options, providing a level of restriction, such that for example the AU option will only be optionable IF ${CMAKE_HOST_SYSTEM} STREQUAL DARWIN, and so forth.

Reasonable default behaviour would be to fall back to whatever the project is currently doing as of writing, which is the JUCE API default behaviour and all passing CI/CD nicely. Maybe this can be enhanced to include more things (or differing things) in time, but I of course assume team JUCE have many dilligent reasons for the provided default behaviour implementation. We don't need to always learn every lesson the hard way.

Also I seem to have no issue at all using Ninja Multi-Config to generate both the Debug and Release target types concurrently. I really am not sure why making this change in the CI/D was causing fails, but I have a suspiscion that JUCE somehow breaks when you set CMAKE_CONFIGURATION_TYPES, for some reason I have yet to discern. Could be something else entirely... no idea, right now.

Will be really good to switch to multi-config generation in the CI/CD, then use a matrix to configure, build, test, and install the matrix of build types. Packing should be done global-per-platform because packages usually ship the Release and Debug builds together in one single archive (this is our current implementation anyway, but if swapping to a matrix strategy then the matrix must end after installing, before packing).