This avoids fetching blobs and tree references for branches we are not
going to worry about. Also skip tag references which are similarly not
useful and keep the default --prune. This keeps the .git data to
around 100M rather than the ~400M even a shallow clone takes.
So we can check the savings we also run a quick du while setting up
the build.
We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the
Windows build, the migration legacy test and the custom runners. In
the case of the custom runners we also move the free floating variable
to the runner template.
Reviewed-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240312170011.1688444-1-alex.bennee@linaro.org>
Now we don't build msys2-32bit we don't need the abstraction out of the
common msys2 handling from the 32-vs-64-bit specifics. Collapse it
down into the msys2-64bit job definition.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240222130920.362517-4-peter.maydell@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
MSYS2 is dropping support for 32-bit Windows. This shows up for us
as various packages we were using in our CI job no longer being
available to install, which causes the job to fail. In commit
8e31b744fd we dropped the dependency on libusb and spice, but the
dtc package has also now been removed.
For us as QEMU upstream, "32 bit x86 hosts for system emulation" have
already been deprecated as of QEMU 8.0, so we are ready to drop them
anyway.
Drop the msys2-32bit CI job, as the first step in doing this.
This is cc'd to stable, because this job will also be broken for CI
on the stable branches. We can't drop 32-bit support entirely there,
but we will still be covering at least compilation for 32-bit Windows
via the cross-win32-system job.
Cc: qemu-stable@nongnu.org
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20240220165602.135695-1-peter.maydell@linaro.org
When msys2 updated their libusb packages to libusb 1.0.27, they
dropped support for building them for mingw32, leaving only mingw64
packages. This broke our CI job, as the 'pacman' package install now
fails with:
error: target not found: mingw-w64-i686-libusb
error: target not found: mingw-w64-i686-usbredir
(both these binary packages are from the libusb source package).
Similarly, spice is now 64-bit only:
error: target not found: mingw-w64-i686-spice
Fix this by dropping these packages from the list we install for our
msys2-32bit build. We do this with a simple mechanism for the
msys2-64bit and msys2-32bit jobs to specify a list of extra packages
to install on top of the common ones we install for both jobs.
Cc: qemu-stable@nongnu.org
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2160
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
Message-id: 20240215155009.2422335-1-peter.maydell@linaro.org
We also --disable-plugins for the two mingw based cross builds as
although they have dlltool they seem to be unhappy linking.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
The `ccache` tool can be very effective at reducing compilation times
when re-running pipelines with only minor changes each time. For example
a fresh 'build-system-fedora' job will typically take 20 minutes on the
gitlab.com shared runners. With ccache this is reduced to as little as
6 minutes.
Normally meson would auto-detect existance of ccache in $PATH and use
it automatically, but the way we wrap meson from configure breaks this,
as we're passing in an config file with explicitly set compiler paths.
Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For
unknown reasons if doing this in msys though, gcc becomes unable to
invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc'
before invoking 'configure' instead.
A second problem with msys is that cache misses are incredibly
expensive, so enabling ccache massively slows down the build when
the cache isn't well populated. This is suspected to be a result of
the cost of spawning processes under the msys architecture. To deal
with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only'
strategy. This avoids extra spawning of the pre-processor during
cache misses, with the downside that is it less likely ccache will
find a cache hit after semantically benign compiler flag changes.
This is the lesser of two evils, as otherwise we can't use ccache
at all under msys and remain inside the job time limit.
If people are finding ccache to hurt their pipelines, it can be
disabled by setting the 'CCACHE_DISABLE=1' env variable against
their gitlab fork CI settings.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230804111054.281802-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
The FF_SCRIPT_SECTIONS=1 variable should ordinarily cause output from
each line of the job script to be presented in a collapsible section
with execution time listed.
While it works on Linux shared runners, when used with Windows runners
with PowerShell, this option does not create any sections, and actually
causes echo'ing of commands to be disabled, making it even worse to
debug the jobs.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230801130403.164060-9-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Building at -O2, adds 33% to the build time, over -O2. IOW a build that
takes 45 minutes at -O0, takes 60 minutes at -O2. Turning off debug
symbols drops it further, down to 38 minutes.
IOW, a "-O2 -g" build is 58% slower than a "-O0" build on msys in the
gitlab CI windows shared runners.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230801130403.164060-8-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The cache is used to hold the msys installer. Even if the build phase
fails, we should still populate the cache as the installer will be
valid for next time.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230801130403.164060-6-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The gitlab cache is limited to only handle content within the
$CI_PROJECT_DIR hierarchy, and as such relative paths are always
implicitly relative to $CI_PROJECT_DIR.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230801130403.164060-5-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
We current reference an msys installer binary from mid-2022, which means
after installation, it immediately has to re-download a bunch of newer
content. This wastes precious CI time.
The msys project publishes an installer binary with a fixed URL that
always references the latest content. We cache the downloads in gitlab
though and so once downloaded we would never re-fetch the installer
leading back to the same problem.
To deal with this we also fetch the pgp signature for the installer
on every run, and compare that to the previously cached signature. If
the signature changes, we re-download the full installer.
This ensures we always have the latest installer for msys, while also
maximising use of the gitlab cache.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230801130403.164060-4-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
It is hard to get visibility into where time is consumed in our Windows
msys jobs. Adding a few log console messages with the timestamp will
aid in our debugging.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230801130403.164060-3-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Although they share a common parent, the two msys jobs still have
massive duplication in their script definitions that can easily be
collapsed.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230801130403.164060-2-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Rename build directory to "build", like most other CI builds.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20230620153720.514882-2-marcandre.lureau@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Include the mingw-w64-x86_64-spice package so SPICE is covered:
C compiler for the host machine: cc -m64 -mcx16 (gcc 12.2.0 "cc (Rev10, Built by MSYS2 project) 12.2.0")
...
Run-time dependency spice-protocol found: YES 0.14.4
Run-time dependency spice-server found: YES 0.15.1
In particular this would have helped catching the build issue
reported as https://gitlab.com/qemu-project/qemu/-/issues/1553:
[1851/5253] Compiling C object libcommon.fa.p/ui_spice-core.c.obj
FAILED: libcommon.fa.p/ui_spice-core.c.obj
../ui/spice-core.c: In function 'watch_remove':
../ui/spice-core.c:152:5: error: implicit declaration of function 'qemu_close_to_socket' [-Werror=implicit-function-declaration]
152 | qemu_close_to_socket(watch->fd);
| ^~~~~~~~~~~~~~~~~~~~
../ui/spice-core.c:152:5: error: nested extern declaration of 'qemu_close_to_socket' [-Werror=nested-externs]
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230322135721.61138-2-philmd@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
By using --enable-fdt=system we can make sure that the configure
script does not try to check out the "dtc" submodule. This should
help to safe some precious CI minutes in the long run.
While we're at it, also drop some now-redundant --enable-slirp
and --enable-capstone statements. These used to have the "=system"
suffix in the past, too, which has been dropped when the their
corresponding submodules had been removed. Since these features
are auto-enabled anyway now (since the containers have the right
libraries installed), we do not need the explicit --enable-...
statements anymore.
Message-Id: <20230207201447.566661-6-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The qtests are not stable in the msys2-32bit job yet - especially
the test-hmp and the qom-test are failing randomly. Until this is
fixed, let's better disable the qtests here again to avoid failing
CI tests.
Message-Id: <20230105204819.26992-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The windows jobs (especially the 32-bit job) recently started to
hit the timeout limit. Bump it a little bit to ease the situation
(80 minutes is quite long already - OTOH, these jobs do not have to
wait for a job from the container stage to finish, so this should
still be OK).
Additionally, some update on the container side recently enabled
OpenGL in these jobs - but the corresponding code fails to compile.
Thus disable OpenGL here for the time being until someone figured
out the proper fix in the shader code for this.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20230104123559.277586-1-thuth@redhat.com
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
qTests don't run successfully with "--without-default-devices",
so let's exclude the qtests from CI for now.
Suggested-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Bin Meng <bin.meng@windriver.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20221125114100.3184790-3-bmeng.cn@gmail.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
At present the build scripts of 32-bit and 64-bit are inconsistent.
Let's keep them consistent for easier maintenance.
While we are here, add some comments to explain that for the 64-bit
job, "--without-default-devices" is a must have, at least for now.
Signed-off-by: Bin Meng <bin.meng@windriver.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20221125114100.3184790-2-bmeng.cn@gmail.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
At present the prerequisite packages for 64-bit and 32-bit builds
are slightly different. Let's use the same packages for both for
easier maintenance in the future.
Signed-off-by: Bin Meng <bin.meng@windriver.com>
Message-Id: <20221125114100.3184790-1-bmeng.cn@gmail.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Adds our build-time dependencies to containers which build qemu-hexagon,
but aren't covered by libvirt-ci.
Signed-off-by: Anton Johansson <anjo@rev.ng>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220804115548.13024-11-anjo@rev.ng>
Message-Id: <20220929114231.583801-9-alex.bennee@linaro.org>
When CI fails we don't know what causes the failure. Displaying the
meson test logs can be helpful.
Signed-off-by: Bin Meng <bin.meng@windriver.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220925113032.1949844-53-bmeng.cn@gmail.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The sed processing of build/config-host.mak seems to be no longer
needed, and there is no such in the 32-bit build too. Drop it.
Signed-off-by: Bin Meng <bin.meng@windriver.com>
Message-Id: <20220908132817.1831008-5-bmeng.cn@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The Windows jobs are currently aborting at weird places - and
there's the suspicion that it's due to memory constraints in
the Windows containers. Let's switch to single-threaded compilation
to decrease the pressure on the memory load, and to make the
job more deterministic for further investigations.
Message-Id: <20220825193323.104768-1-thuth@redhat.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Message-Id: <20220728200422.1502-1-luoyonggang@gmail.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
The following error message was seen during the configure:
"ln: failed to create symbolic link
'x86_64-softmmu/qemu-system-x86_64.exe': No such file or directory"
By default the MSYS environment variable is not defined, so the runtime
behavior of winsymlinks is: if <target> does not exist, 'ln -s' fails.
At the configure phase, the qemu-system-x86_64.exe has not been built
so creation of the symbolic link fails hence the error message.
Set winsymlinks to 'native' whose behavior is most similar to the
behavior of 'ln -s' on *nix, that is:
a) if native symlinks are enabled, and whether <target> exists
or not, creates <destination> as a native Windows symlink;
b) else if native symlinks are not enabled, and whether <target>
exists or not, 'ln -s' creates as a Windows shortcut file.
Signed-off-by: Bin Meng <bin.meng@windriver.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220725123000.807608-1-bmeng.cn@gmail.com>
Message-Id: <20220725140520.515340-6-alex.bennee@linaro.org>
This converts the main build and container jobs to use the
base job rules, defining the following new variables
- QEMU_JOB_SKIPPED - jobs that are known to be currently
broken and should not be run. Can still be manually
launched if desired.
- QEMU_JOB_AVOCADO - jobs that run the Avocado integration
test harness.
- QEMU_JOB_PUBLISH - jobs that publish content after the
branch is merged upstream
As build-tools-and-docs runs on master we declare the requirement of
building amd64-debian-container optional as it should already exits
once we merge.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-5-berrange@redhat.com>
[AJB: fix upstream typo, mention optional container req]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org>
Now that we allow compiling with Capstone v3.0.5 again, all our supported
build hosts should provide at least this version of the disassembler
library, so we do not need to ship this as a submodule anymore.
Message-Id: <20220516145823.148450-4-thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
lcitool doesn't support MSYS2 targets, so manually remove
this now unnecessary library.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220121154134.315047-7-f4bug@amsat.org>
Message-Id: <20220204204335.1689602-11-alex.bennee@linaro.org>
The new msys2-64bit job is often running for more than 50 minutes - and
if the CI is currently loaded, it times out after 60 minutes. The job
has been declared with a bigger timeout, but seems like this is getting
ignored on the shared Gitlab-CI Windows runners, so we're currently
seeing a lot of failures with this job. Thus we have to reduce the time
it takes to finish this job. Since we want to test compiling the WHPX
and HAX accelerator code with this job, switching to another target CPU
is not really a good option, so let's reduce the amount of code that we
have to compile with the --without-default-devices switch instead.
Message-Id: <20211216082253.43899-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Gitlab also provides runners with Windows, we can use them to
test compilation with MSYS2, in both, 64-bit and 32-bit.
However, it takes quite a long time to set up the VM, so to stay
in a reasonable time frame, we can only compile and check one
target here.
Message-Id: <20211115140623.104116-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
