Feature/hardware ci#127
Open
JOOpdenhoevel wants to merge 22 commits into
Open
Conversation
Signed-off-by: Jan-Oliver Opdenhövel <joo@mail.uni-paderborn.de>
…line" This reverts commit 347394e.
This reverts commit 716c93f.
Signed-off-by: Jan-Oliver Opdenhövel <joo@mail.uni-paderborn.de>
Signed-off-by: Jan-Oliver Opdenhövel <Jan-Oliver.Opdenhovel@amd.com>
Signed-off-by: Jan-Oliver Opdenhövel <jopdenho@amd.com>
This reverts commit 7d6f1f5.
This reverts commit 11c9f07.
Signed-off-by: Jan-Oliver Opdenhövel <joo@mail.uni-paderborn.de>
Signed-off-by: Jan-Oliver Opdenhövel <joo@mail.uni-paderborn.de>
…ing it Signed-off-by: Jan-Oliver Opdenhövel <joo@mail.uni-paderborn.de>
Signed-off-by: Jan-Oliver Opdenhövel <joo@mail.uni-paderborn.de>
Signed-off-by: Jan-Oliver Opdenhövel <jopdenho@amd.com>
Signed-off-by: Jan-Oliver Opdenhövel <jopdenho@amd.com>
Signed-off-by: Jan-Oliver Opdenhövel <jopdenho@amd.com>
This reverts commit f018dc0.
Signed-off-by: Jan-Oliver Opdenhövel <jopdenho@amd.com>
Open
Collaborator
Author
Related issuesThis PR covers almost all points in #18 and thus might be sufficient to close #18. The only missing point is testing on real hardware, which has not been been implemented yet as previously mentioned. However, once the V80s in Otus are regularly available, extending this pipeline to also do the testing should be trivial. |
Signed-off-by: Jan-Oliver Opdenhövel <jopdenho@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The main issue with GitHub Actions is that it is based on Docker, and Vivado can't be practically shipped in a Docker container. Thus, this PR adds the necessary configuration files to run Vivado-based tasks and everything that depends on it on the Otus supercomputer using a Gitlab CI/CD setup.
Overview of the CI pipeline solution
The first part of this setup is the mirror repository in Paderborn's Gitlab instance, which regularly pulls new changes from the upstream repo on GitHub and evaluates the .gitlab-ci.yml file for every branch that has received new commits.
In the configuration that this PR provides, there is one job to (optionally) build the base design and package the linker wheel, one job to build and run all working examples in emulation and simulation, and one job for each working example that builds the hardware VBIN. Jobs to run these hardware VBINs on a V80 are not included since the FPGA partition within Otus is not production-ready yet. This pipeline runs for all pushes to the branches
mainanddev, as well as all branches matching the globpr/*.The linker packaging step uses a trick to always pull the artifacts of the last successful run of this job and, depending of the user's input, either packages the linker with the previous base design or builds it newly from scratch. This way, we can both semi-automatically build the base design and reuse it as often as desired, without changing any configuration files.
The last piece of the puzzle is a Gitlab CI runner that's running on Otus. It receives the jobs that the CI pipeline needs to execute and submits them as Slurm jobs to the cluster.
External access and external pull requests
Since this CI pipeline both handles tasks with very high runtimes and produces artifacts that must not be redistributed, it is only really available to the primary SLASH developers and maintainers. Outside users can see the status and some of the logs of a CI pipeline, but they can not download any artifacts.
Also, the Gitlab mirror does not automatically pull in branches from external pull requests; Only the primary SLASH developers and maintainers can push changes to the upstream repository and thus the mirror and its CI pipeline. However, external pull-requests can be pulled in as separate branches with the import-pr.yml workflow. Once this PR has been merged, it will be available for the primary SLASH developers and maintainers as a manual workflow.
Other changes needed to build the software stack on Otus
I had to make two changes to the build system in order to build the software stack on Otus: First of all, the CLI11 library is not available as a library on Otus. However, since it is a header-only library, it is sufficient to pull it in from GitHub during the CMake configuration phase.
The other, rather minute change is that I changed the
target_link_librariescommand for the VRT library since the nameJsonCpp::JsonCppgot resolved to the static JsonCpp library, which wasn't compiled with-fPICon Otus and thus can't be linked into another shared library, such as VRT.jsoncpp_libshould now always resolve to the shared object.