mirrors/qemu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2a339fee45
qemu/.gitlab-ci.d/crossbuild-template.yml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

134 lines
4.3 KiB
YAML
Raw Normal View History

gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
.cross_system_build_job:
gitlab: convert build/container jobs to .base_job_template This converts the main build and container jobs to use the base job rules, defining the following new variables - QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired. - QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness. - QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream As build-tools-and-docs runs on master we declare the requirement of building amd64-debian-container optional as it should already exits once we merge. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20220526110705.59952-5-berrange@redhat.com> [AJB: fix upstream typo, mention optional container req] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org>
2022-05-27 18:36:01 +03:00
extends: .base_job_template
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
stage: build
gitlab: centralize the container tag name We use a fixed container tag of 'latest' so that contributors' forks don't end up with an ever growing number of containers as they work on throwaway feature branches. This fixed tag causes problems running CI upstream in stable staging branches, however, because the stable staging branch will publish old container content that clashes with that needed by primary staging branch. This makes it impossible to reliably run CI pipelines in parallel in upstream for different staging branches. This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to change which tag container publishing uses. Initially it can be set by contributors as a git push option if they want to override the default use of 'latest' eg git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish this is useful if contributors need to run pipelines for different branches concurrently in their forks. Reviewed-by: Michael Tokarev <mjt@tls.msk.ru> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20230608164018.2520330-2-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-06-08 19:40:14 +03:00
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
cache:
paths:
- ccache
key: "$CI_JOB_NAME"
when: always
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
timeout: 80m
gitlab: display /packages.txt in build jobs The lcitool created containers save the full distro package list details into /packages.txt. The idea is that build jobs will 'cat' this file, so that the build log has a record of what packages were used. This is important info, because when it comes to debug failures, the original container is often lost. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20240724095505.33544-3-berrange@redhat.com> Acked-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20240729144414.830369-3-alex.bennee@linaro.org>
2024-07-29 17:44:02 +03:00
before_script:
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- source scripts/ci/gitlab-ci-section
- section_start setup "Pre-script setup"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- JOBS=$(expr $(nproc) + 1)
gitlab: display /packages.txt in build jobs The lcitool created containers save the full distro package list details into /packages.txt. The idea is that build jobs will 'cat' this file, so that the build log has a record of what packages were used. This is important info, because when it comes to debug failures, the original container is often lost. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20240724095505.33544-3-berrange@redhat.com> Acked-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20240729144414.830369-3-alex.bennee@linaro.org>
2024-07-29 17:44:02 +03:00
- cat /packages.txt
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end setup
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
script:
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
- export CCACHE_BASEDIR="$(pwd)"
- export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
- export CCACHE_MAXSIZE="500M"
- export PATH="$CCACHE_WRAPPERSDIR:$PATH"
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
- mkdir build
- cd build
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
- ccache --zero-stats
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_start configure "Running configure"
gitlab-ci.d: Build with --enable-fdt=system by default By using --enable-fdt=system we can make sure that the configure script does not try to check out the "dtc" submodule. This should help to safe some precious CI minutes in the long run. While we're at it, also drop some now-redundant --enable-slirp and --enable-capstone statements. These used to have the "=system" suffix in the past, too, which has been dropped when the their corresponding submodules had been removed. Since these features are auto-enabled anyway now (since the containers have the right libraries installed), we do not need the explicit --enable-... statements anymore. Message-Id: <20230207201447.566661-6-thuth@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-02-07 23:14:47 +03:00
- ../configure --enable-werror --disable-docs --enable-fdt=system
--disable-user $QEMU_CONFIGURE_OPTS $EXTRA_CONFIGURE_OPTS
system: Remove support for CRIS target We are about to remove the CRIS target, so remove the sysemu part. This remove the CRIS 'none' machine. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Acked-by: Edgar E. Iglesias <edgar.iglesias@amd.com> Message-ID: <20240904143603.52934-13-philmd@linaro.org>
2024-07-24 14:47:57 +03:00
--target-list-exclude="arm-softmmu
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds We need to cut down compile time by excluding more targets. Both these targets still have their 64-bit variant enabled, so the loss of coverage is mitigated to some degree. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210810140653.3969823-2-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-08-10 17:06:52 +03:00
mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
gitlab: skip many more targets in windows cross builds The windows cross builds still take way too long in gitlab CI, so need more targets to be skipped. We don't want to hurt coverage of other cross builds more though, so we let jobs fine tune with a new env variale $CROSS_SKIP_TARGETS. We take the set of targets that are considered relatively niche or very old architectures, and skip approx half of them in win32 builds and the other half of them in win64. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210810140653.3969823-3-berrange@redhat.com> [thuth: Swapped the "CROSS_SKIP_TARGETS:" lines as suggested by philmd] Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-08-10 17:06:53 +03:00
sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS"
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end configure
- section_start build "Building QEMU"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- make -j"$JOBS" all check-build
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end build
- section_start test "Running tests"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- if test -n "$MAKE_CHECK_ARGS";
then
$MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
fi
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end test
- section_start installer "Building the installer"
ci: build & store windows installer Build windows installer for qemu in gitlab CI, store the result as artifact. Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20210623091137.1156959-2-kraxel@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-23 12:11:35 +03:00
- if grep -q "EXESUF=.exe" config-host.mak;
then make installer;
gitlab: fall back to commit hash in qemu-setup filename Personal repos may not have release tags (v6.0.0, v6.1.0, etc) and this causes cross_system_build_job to fail when pretty-printing a unique qemu-setup-*.exe name: version="$(git describe --match v[0-9]*)"; ^^^^^^^^^^ fails ^^^^^^^^^^^ mv -v qemu-setup*.exe qemu-setup-${version}.exe; Fall back to the short commit hash if necessary. This fixes CI failures that Greg Kurz and I experienced in our personal repos. Cc: Greg Kurz <groug@kaod.org> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Greg Kurz <groug@kaod.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20220125173454.10381-1-stefanha@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20220204204335.1689602-14-alex.bennee@linaro.org>
2022-02-04 23:43:22 +03:00
version="$(git describe --match v[0-9]* 2>/dev/null || git rev-parse --short HEAD)";
ci: build & store windows installer Build windows installer for qemu in gitlab CI, store the result as artifact. Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20210623091137.1156959-2-kraxel@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-23 12:11:35 +03:00
mv -v qemu-setup*.exe qemu-setup-${version}.exe;
fi
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end installer
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
- ccache --show-stats
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
# Job to cross-build specific accelerators.
#
# Set the $ACCEL variable to select the specific accelerator (default to
# KVM), and set extra options (such disabling other accelerators) via the
# $EXTRA_CONFIGURE_OPTS variable.
.cross_accel_build_job:
gitlab: convert build/container jobs to .base_job_template This converts the main build and container jobs to use the base job rules, defining the following new variables - QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired. - QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness. - QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream As build-tools-and-docs runs on master we declare the requirement of building amd64-debian-container optional as it should already exits once we merge. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20220526110705.59952-5-berrange@redhat.com> [AJB: fix upstream typo, mention optional container req] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org>
2022-05-27 18:36:01 +03:00
extends: .base_job_template
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
stage: build
gitlab: centralize the container tag name We use a fixed container tag of 'latest' so that contributors' forks don't end up with an ever growing number of containers as they work on throwaway feature branches. This fixed tag causes problems running CI upstream in stable staging branches, however, because the stable staging branch will publish old container content that clashes with that needed by primary staging branch. This makes it impossible to reliably run CI pipelines in parallel in upstream for different staging branches. This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to change which tag container publishing uses. Initially it can be set by contributors as a git push option if they want to override the default use of 'latest' eg git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish this is useful if contributors need to run pipelines for different branches concurrently in their forks. Reviewed-by: Michael Tokarev <mjt@tls.msk.ru> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20230608164018.2520330-2-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-06-08 19:40:14 +03:00
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
timeout: 30m
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
cache:
paths:
- ccache/
key: "$CI_JOB_NAME"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
before_script:
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- source scripts/ci/gitlab-ci-section
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- JOBS=$(expr $(nproc) + 1)
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
script:
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
- export CCACHE_BASEDIR="$(pwd)"
- export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
- export CCACHE_MAXSIZE="500M"
- export PATH="$CCACHE_WRAPPERSDIR:$PATH"
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
- mkdir build
- cd build
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_start configure "Running configure"
gitlab: remove redundant setting of PKG_CONFIG_PATH The PKG_CONFIG_PATH variable is not defined in GitLab CI envs and even if it was, we don't need to set it to its existing value. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Message-Id: <20221103173044.3969425-2-berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-11-03 20:30:43 +03:00
- ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
--disable-tools --enable-${ACCEL:-kvm} $EXTRA_CONFIGURE_OPTS
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end configure
- section_start build "Building QEMU"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- make -j"$JOBS" all check-build
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end build
- section_start test "Running tests"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- if test -n "$MAKE_CHECK_ARGS";
then
$MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
fi
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end test
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
.cross_user_build_job:
gitlab: convert build/container jobs to .base_job_template This converts the main build and container jobs to use the base job rules, defining the following new variables - QEMU_JOB_SKIPPED - jobs that are known to be currently broken and should not be run. Can still be manually launched if desired. - QEMU_JOB_AVOCADO - jobs that run the Avocado integration test harness. - QEMU_JOB_PUBLISH - jobs that publish content after the branch is merged upstream As build-tools-and-docs runs on master we declare the requirement of building amd64-debian-container optional as it should already exits once we merge. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20220526110705.59952-5-berrange@redhat.com> [AJB: fix upstream typo, mention optional container req] Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org>
2022-05-27 18:36:01 +03:00
extends: .base_job_template
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
stage: build
gitlab: centralize the container tag name We use a fixed container tag of 'latest' so that contributors' forks don't end up with an ever growing number of containers as they work on throwaway feature branches. This fixed tag causes problems running CI upstream in stable staging branches, however, because the stable staging branch will publish old container content that clashes with that needed by primary staging branch. This makes it impossible to reliably run CI pipelines in parallel in upstream for different staging branches. This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to change which tag container publishing uses. Initially it can be set by contributors as a git push option if they want to override the default use of 'latest' eg git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish this is useful if contributors need to run pipelines for different branches concurrently in their forks. Reviewed-by: Michael Tokarev <mjt@tls.msk.ru> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Message-Id: <20230608164018.2520330-2-berrange@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
2023-06-08 19:40:14 +03:00
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
cache:
paths:
- ccache/
key: "$CI_JOB_NAME"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
before_script:
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- source scripts/ci/gitlab-ci-section
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- JOBS=$(expr $(nproc) + 1)
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
script:
gitlab: enable ccache for many build jobs The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
2023-08-29 19:15:17 +03:00
- export CCACHE_BASEDIR="$(pwd)"
- export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
- export CCACHE_MAXSIZE="500M"
gitlab: Extract crossbuild job templates to crossbuild-template.yml Extract the crossbuild job templates to a new file (crossbuild-template.yml) to be able to reuse them without having to run all the jobs included, which are mainly useful for mainstream CI. Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Message-Id: <20210519185504.2198573-5-f4bug@amsat.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-19 21:54:58 +03:00
- mkdir build
- cd build
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_start configure "Running configure"
gitlab: remove redundant setting of PKG_CONFIG_PATH The PKG_CONFIG_PATH variable is not defined in GitLab CI envs and even if it was, we don't need to set it to its existing value. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Message-Id: <20221103173044.3969425-2-berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-11-03 20:30:43 +03:00
- ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
gitlab: reduce targets in cross_user_build_job We already limit the scope of the cross system build to reduce the cross build times. With the recent addition of more targets we are also running into timeout issues for some of the cross user builds. I've selected a few of those linux-user targets which are less likely to be in common use as distros don't have pre-built rootfs for them. I've also added the same CROSS_SKIP_TARGETS variable as is occasionally used to further limit cross system builds. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220914155950.804707-2-alex.bennee@linaro.org>
2022-09-14 18:59:21 +03:00
--disable-system --target-list-exclude="aarch64_be-linux-user
linux-user: Remove support for CRIS target As per the deprecation notice in commit c7bbef4023: The CRIS architecture was pulled from Linux in 4.17 and the compiler is no longer packaged in any distro making it harder to run the `check-tcg` tests. Unless we can improve the testing situation there is a chance the code will bitrot without anyone noticing. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Acked-by: Edgar E. Iglesias <edgar.iglesias@amd.com> Message-ID: <20240904143603.52934-5-philmd@linaro.org>
2024-07-24 14:47:57 +03:00
alpha-linux-user m68k-linux-user microblazeel-linux-user
target/nios2: Remove the deprecated Nios II target The Nios II target is deprecated since v8.2 in commit 9997771bc1 ("target/nios2: Deprecate the Nios II architecture"). Remove: - Buildsys / CI infra - User emulation - System emulation (10m50-ghrd & nios2-generic-nommu machines) - Tests Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Acked-by: Marek Vasut <marex@denx.de> Message-Id: <20240327144806.11319-3-philmd@linaro.org>
2024-03-27 14:10:58 +03:00
or1k-linux-user ppc-linux-user sparc-linux-user
gitlab: reduce targets in cross_user_build_job We already limit the scope of the cross system build to reduce the cross build times. With the recent addition of more targets we are also running into timeout issues for some of the cross user builds. I've selected a few of those linux-user targets which are less likely to be in common use as distros don't have pre-built rootfs for them. I've also added the same CROSS_SKIP_TARGETS variable as is occasionally used to further limit cross system builds. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20220914155950.804707-2-alex.bennee@linaro.org>
2022-09-14 18:59:21 +03:00
xtensa-linux-user $CROSS_SKIP_TARGETS"
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end configure
- section_start build "Building QEMU"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- make -j"$JOBS" all check-build
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end build
- section_start test "Running tests"
.gitlab-ci.d: Split build and test in cross build job templates In the native_build_job_template we have separate steps in the script for the build and the test steps. This is helpful because then gitlab will give separate timestamps in the log view for each, and you can see how long it took to compile vs how long to test. In the templates in crossbuild-template.yml, however, we do both the build and test in a single 'make' invocation, and so we don't get the separate timing information. Split the build and test, in the same way we do in the native build template. This will also give us a place to separate out how parallel we want to do the build by default from how parallel we want to do the tests by default, which might be helpful in future. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-2-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:48 +03:00
- if test -n "$MAKE_CHECK_ARGS";
then
$MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
fi
.gitlab-ci.d: Make separate collapsible log sections for build and test GitLab lets a CI job create its own collapsible log sections by emitting special escape codes, as documented here: https://docs.gitlab.com/ee/ci/yaml/script.html#expand-and-collapse-job-log-sections Use these to make "configure", "build" and "test" separate collapsible stages. As recommended by the GitLab docs, we use some shell which is sourced in the CI job to define functions to emit the magic lines that start and end sections, to hide the ugliness of the printf lines from the log. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-ID: <20240918125449.3125571-3-peter.maydell@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-18 15:54:49 +03:00
- section_end test
tests: ensure we export job results for some cross builds We do run tests on some cross builds. Provide a template to ensure we export the testlog to the build artefacts and report the test results via the junit. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230228190653.1602033-13-alex.bennee@linaro.org>
2023-02-28 22:06:41 +03:00
# We can still run some tests on some of our cross build jobs. They can add this
# template to their extends to save the build logs and test results
.cross_test_artifacts:
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
gitlab: explicit set artifacts publishing criteria If not set explicitly, gitlab assumes 'when: on_success" as the publishing criteria for artifacts. This is reasonable if the artifact is an output deliverable of the job. This is useless if the artifact is a log file to be used for debugging job failures. This change makes the desired criteria explicit for every job that publishes artifacts. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230503145535.91325-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230630180423.558337-2-alex.bennee@linaro.org>
2023-06-30 21:03:46 +03:00
when: always
tests: ensure we export job results for some cross builds We do run tests on some cross builds. Provide a template to ensure we export the testlog to the build artefacts and report the test results via the junit. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reported-by: Peter Maydell <peter.maydell@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Message-Id: <20230228190653.1602033-13-alex.bennee@linaro.org>
2023-02-28 22:06:41 +03:00
expire_in: 7 days
paths:
- build/meson-logs/testlog.txt
reports:
junit: build/meson-logs/testlog.junit.xml
Reference in New Issue Copy Permalink