Commit Graph

199 Commits

Author SHA1 Message Date
Daniel P. Berrangé
f13abca0a3 gitlab: fix passing of TEST_TARGETS env to cirrus
A typo meant the substitution would not work, and the placeholder in the
target file didn't even exist.

The result was that tests were never run on the FreeBSD and macOS jobs,
only a basic build.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210915125452.1704899-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210917162332.3511179-10-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Daniel P. Berrangé
dcbad7a6ed gitlab: skip the check-patch job on the upstream repo
The check-patch job is intended to be used by contributors or
subsystem maintainers to see if there are style mistakes. The
false positive rate is too high to be used in a gating scenario
so should not run it on the upstream repo ever.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210915125452.1704899-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210917162332.3511179-9-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Richard Henderson
9f62025141 gitlab: Add cross-riscv64-system, cross-riscv64-user
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210914185830.1378771-3-richard.henderson@linaro.org>
[AJB: add allow_failure]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210917162332.3511179-6-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Alex Bennée
5c24acf320 tests/docker: promote debian-riscv64-cross to a full image
To be able to cross build QEMU itself we need to include a few more
libraries. These are only available in Debian's unstable ports repo
for now so we need to base the riscv64 image on sid with the the
minimal libs needed to build QEMU (glib/pixman).

The result works but is not as clean as using build-dep to bring in
more dependencies. However sid is by definition a shifting pile of
sand and by keeping the list of libs minimal we reduce the chance of
having an image we can't build. It's good enough for a basic cross
build testing of TCG.

Cc: "Daniel P. Berrangé" <berrange@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210914185830.1378771-2-richard.henderson@linaro.org>
[AJB: tweak allow_failure]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210917162332.3511179-5-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Peter Maydell
7b7ab2d6c9 gitlab-ci: Mark manual-only jobs as allow_failure
If a gitlab CI job is marked as manual-only but is not marked
as allow_failure, then gitlab considers that the pipeline is
"blocked" until the job has been manually triggered. We need
to mark these manual-only jobs as also allow_failure: true
so that gitlab doesn't insist that they have run before it
will consider the pipeline to be complete.

Fixes: 4c9af1ea14
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20210915123412.8232-1-peter.maydell@linaro.org
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
2021-09-15 16:43:16 +01:00
Peter Maydell
4c9af1ea14 gitlab-ci: Make more custom runner jobs manual, and don't allow failure
Currently we define a lot of jobs for our custom runners:
for both aarch64 and s390x we have
 - all-linux-static
 - all
 - alldbg
 - clang (manual)
 - tci
 - notcg (manual)

This is overkill.  The main reason to run on these hosts is to get
coverage for the host architecture; we can leave the handling of
differences like debug vs non-debug to the x86 CI jobs.

The jobs are also generally running OK; they occasionally fail due to
timeouts, which is likely because we're overloading the machine by
asking it to run 4 CI jobs at once plus the ad-hoc CI.

Remove the 'allow_failure' tag from all these jobs, and switch the
s390x-alldbg, aarch64-all, s390x-tci and aarch64-tci jobs to manual.
(We keep -all on s390x and -alldbg on aarch64 just for diversity
of coverage.)

This will let us make the switch for s390x and aarch64 hosts from
the ad-hoc CI to gitlab.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-id: 20210913101948.12600-1-peter.maydell@linaro.org
2021-09-14 17:03:03 +01:00
Thomas Huth
2ffd4d815e gitlab-ci: Don't try to use the system libfdt in the debian job
libfdt in Debian is too old to be usable for QEMU. So far we were
silently falling back to the internal dtc submodule, but since
this is wrong, let's remove the --enable-fdt=system switch here now.

Message-Id: <20210827151718.178988-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-09-06 09:07:00 +02:00
Peter Maydell
9c03aa87e5 Testing and plugin updates:
- fix typo in execlog plugin
   - clean-up and document gitlab FOO_RUNNER_AVAILABLE vars
   - fix plugin build issue on OSX and modules
   - add multi-core support to cache modelling plugin
   - clean-ups for plugin arg=FOO handling
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmEwqF4ACgkQ+9DbCVqe
 KkTCcAf/fCHK1hhXJIaxpFyMGvCkhYzUgKlpL4b05ofXFyQJ4JWp+0OadQzBcVIN
 PwsVVsXik/5ibKvQiud8wt3/kbr9gj5RZdwVITZMEpod6hy9Nt5oaUhxEZpiVs3f
 XZG17zaK8huwTwU6EtHWOnvIWMCuvBXpw8cM/6jF6pEiq162VJbiRo9L/aQ+n0Io
 TZTy9YqEbrZUTj+XrvgPV0TewjM6T/zTujXFJtyCzYJ3P3kx9z5a3/KzfyJI9qkk
 TYw0MX9MY8J5/sXcP6OjOqXqppsRe+G5Uaz0BoW9SvOmR/0nNGy/N8mVCZTD6mqs
 zMsWw5RyIZsZCyWEYj4TYWEORv5Kiw==
 =CBs1
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-for-6.2-020921-1' into staging

Testing and plugin updates:

  - fix typo in execlog plugin
  - clean-up and document gitlab FOO_RUNNER_AVAILABLE vars
  - fix plugin build issue on OSX and modules
  - add multi-core support to cache modelling plugin
  - clean-ups for plugin arg=FOO handling

# gpg: Signature made Thu 02 Sep 2021 11:33:02 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-for-6.2-020921-1: (22 commits)
  docs/devel: be consistent about example plugin names
  docs/deprecated: deprecate passing plugin args through `arg=`
  tests/plugins/syscalls: adhere to new arg-passing scheme
  tests/plugins/mem: introduce "track" arg and make args not positional
  tests/plugins/insn: made arg inline not positional and parse it as bool
  tests/plugins/bb: adapt to the new arg passing scheme
  docs/tcg-plugins: new passing parameters scheme for cache docs
  plugins/howvec: adapting to the new argument passing scheme
  plugins/hwprofile: adapt to the new plugin arguments scheme
  plugins/lockstep: make socket path not positional & parse bool arg
  plugins/hotblocks: Added correct boolean argument parsing
  plugins/hotpages: introduce sortby arg and parsed bool args correctly
  plugins/api: added a boolean parsing plugin api
  plugins: allow plugin arguments to be passed directly
  docs/devel/tcg-plugins: added cores arg to cache plugin
  plugins: sort exported symbol list
  plugins/cache: supported multicore cache modelling
  plugins: do not limit exported symbols if modules are active
  gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them
  gitlab-ci: Remove superfluous "dnf install" statement
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-09-03 14:23:36 +01:00
Thomas Huth
40e916b4bc gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them
The patch that recently introduced the S390X_RUNNER_AVAILABLE variable
in custom-runners.yml missed that the bottom half of the file is rather
about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to
AARCH64_RUNNER_AVAILABLE in those jobs.

Finally mention both variables in our CI documentation, too.

Fixes: c5dd0f0342 ("Improve rules for the staging branch")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210730143809.717079-4-thuth@redhat.com>
[AJB: moved due to docu changes]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210806141015.2487502-5-alex.bennee@linaro.org>
2021-09-02 11:29:34 +01:00
Thomas Huth
d3ac658fbe gitlab-ci: Remove superfluous "dnf install" statement
The container already features meson and ninja, so there is no need
to try to install it with dnf again.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210730143809.717079-3-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210806141015.2487502-4-alex.bennee@linaro.org>
2021-09-02 11:29:34 +01:00
Thomas Huth
8d316275ad gitlab-ci: Merge "build-disabled" with "build-without-default-features"
Both jobs are testing more or less the same thing (building QEMU with
features disabled), so we are wasting precious CI cycles here by doing
this twice. Merge the jobs by using --without-default-features by default
and just adding some additional --disable-... switches which are not
covered by the generic switch (yet). And while we're at it, also test
compilation with "--disable-fdt" (which forces us to change the list
of targets in this job, though, since some targets do not work without
fdt).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210730143809.717079-2-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210806141015.2487502-3-alex.bennee@linaro.org>
2021-09-02 11:29:34 +01:00
Vladimir Sementsov-Ogievskiy
c88f078948 iotests: move 222 to tests/image-fleecing
Give a good name to test file.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210824083856.17408-29-vsementsov@virtuozzo.com>
[hreitz: Adjust .gitlab-ci.d/buildtest.yml]
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 14:37:14 +02:00
Daniel P. Berrangé
a1f0f36838 gitlab: skip many more targets in windows cross builds
The windows cross builds still take way too long in gitlab CI, so need
more targets to be skipped. We don't want to hurt coverage of other
cross builds more though, so we let jobs fine tune with a new env
variale $CROSS_SKIP_TARGETS.

We take the set of targets that are considered relatively niche or
very old architectures, and skip approx half of them in win32 builds
and the other half of them in win64.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210810140653.3969823-3-berrange@redhat.com>
[thuth: Swapped the "CROSS_SKIP_TARGETS:" lines as suggested by philmd]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-08-11 08:52:31 +02:00
Daniel P. Berrangé
a62354915b gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
We need to cut down compile time by excluding more targets. Both these
targets still have their 64-bit variant enabled, so the loss of coverage
is mitigated to some degree.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210810140653.3969823-2-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-08-11 07:52:43 +02:00
Thomas Huth
c5dd0f0342 gitlab-ci.d/custom-runners: Improve rules for the staging branch
If maintainers are currently pushing to a branch called "staging"
in their repository, they are ending up with some stuck jobs - unless
they have a s390x CI runner machine available. That's ugly, we should
make sure that the related jobs are really only started if such a
runner is available. So let's only run these jobs if it's the
"staging" branch of the main repository of the QEMU project (where
we can be sure that the s390x runner is available), or if the user
explicitly set a S390X_RUNNER_AVAILABLE variable in their CI configs
to declare that they have such a runner available, too.

Fixes: 4799c21023 ("Jobs based on custom runners: add job definitions ...")
Message-Id: <20210728173857.497523-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 08:04:25 +02:00
Thomas Huth
e37264ebe3 gitlab-ci.d/buildtest: Mark the aarch64 and ppc64-s390x CFI jobs as manual
These two jobs are currently failing very often - the linker seems to
get killed due to out-of-memory problems. Since apparently nobody has
currently an idea how to fix that nicely, let's mark the jobs as manual
for the time being until someone comes up with a proper fix.

Message-Id: <20210728075141.400816-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 08:04:25 +02:00
Thomas Huth
db1e119238 gitlab-ci.d/buildtest: Disable iotests 197 and 215
The iotests 197 and 215 are occasionally failing in the gitlab-CI now.
According to the log, the failure is "./common.rc: Killed" which might
be an indication that the process has been killed due to out-of-memory
reasons. Both tests are doing a big read with 2G that likely causes
this issue. It used to work fine in the gitlab-CI in the past, but
either the program is now requiring more free memory, or the the CI
containers have changed, so that the OOM condition now sometimes occurs.

Anyway, these two tests are not really suitable for CI containers if
they are doing things like huge reads (which is likely also the reason
why they haven't been added to the "auto" group in the past), so let's
simply disable them in the gitlab-CI now, too.

Message-Id: <20210727162542.318882-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 08:04:25 +02:00
Philippe Mathieu-Daudé
c217fd8e36 gitlab-ci: Fix 'when:' condition in OpenSBI jobs
Jobs depending on another should not use the 'when: always'
condition, because if a dependency failed we should not keep
running jobs depending on it. The correct condition is
'when: on_success'.

Fixes: c6fc0fc1a7 ("gitlab-ci.yml: Add jobs to build OpenSBI firmware binaries")
Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210727142431.1672530-5-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 07:56:01 +02:00
Philippe Mathieu-Daudé
59e8b62b22 gitlab-ci: Fix 'when:' condition in EDK2 jobs
Jobs depending on another should not use the 'when: always'
condition, because if a dependency failed we should not keep
running jobs depending on it. The correct condition is
'when: on_success'.

Fixes: 71920809ce ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries")
Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210727142431.1672530-4-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 07:56:01 +02:00
Philippe Mathieu-Daudé
d3a4e41da2 gitlab-ci: Fix 'when:' condition in acceptance_test_job_template
Jobs depending on another should not use the 'when: always'
condition, because if a dependency failed we should not keep
running jobs depending on it. The correct condition is
'when: on_success'.

Fixes: f56bf4caf7 ("gitlab: Run Avocado tests manually (except mainstream CI)")
Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210727142431.1672530-3-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 07:56:01 +02:00
Peter Maydell
34fd92ab41 Doc, metadata, plugin and testing updates for 6.1-rc1:
- git ignore some file editor detritus
   - add overview on device emulation terminology
   - remove needless if leg in configure custom devices logic
   - numerous gitdm/mailmap updates
   - fix plugin_exit race for linux-user
   - fix a few bugs in cache modelling plugin
   - fix plugin calculation of physical address
   - handle pure assembler/linker tcg tests outside of docker
   - add tricore build to gitlab
   - remove superfluous MacOSX task
   - generalise the OpenBSI gitlab rules
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmD67ioACgkQ+9DbCVqe
 KkTKcwf/UlEl2wlYrvymoJy7caasD/UrlNwpaK90zRTxuofJjG2bXX/58EGBlgxe
 cjbX3841702HXAFDxzswmWX3yeXQVgNiXrEy5FapB7L3h/qC/BDVS4pz987TITNW
 b4lw/d05bQVGavcdJXPnX0CdV8IK4hpieDkdh/LGj5+CSAC39jvyVGTYBpfLKl8D
 cps5qcJcJuQjz4hatB1haZOV1WHLyw8KI0srfcEqkh3IImZFZ09frs3qqAxrYRqE
 Y62d7SIN5TX1XgWuTaf1mmaMVmbNi9DTdCxWpGYr4ZBLhhzKeljQvKbF45ZGQ0z9
 eBVBdxGJTX/WAKrimcbHxLo43pCo2Q==
 =7vzj
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-for-6.1-rc1-230721-1' into staging

Doc, metadata, plugin and testing updates for 6.1-rc1:

  - git ignore some file editor detritus
  - add overview on device emulation terminology
  - remove needless if leg in configure custom devices logic
  - numerous gitdm/mailmap updates
  - fix plugin_exit race for linux-user
  - fix a few bugs in cache modelling plugin
  - fix plugin calculation of physical address
  - handle pure assembler/linker tcg tests outside of docker
  - add tricore build to gitlab
  - remove superfluous MacOSX task
  - generalise the OpenBSI gitlab rules

# gpg: Signature made Fri 23 Jul 2021 17:28:26 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-for-6.1-rc1-230721-1: (28 commits)
  gitlab-ci: Extract OpenSBI job rules to reusable section
  gitlab-ci: Remove the second superfluous macos task
  gitlab: enable a very minimal build with the tricore container
  tests/tcg/configure.sh: add handling for assembler only builds
  plugins: Fix physical address calculation for IO regions
  plugins/cache: Fixed "function decl. is not a prototype" warnings
  plugins/cache: limited the scope of a mutex lock
  plugins/cache: Fixed a bug with destroying FIFO metadata
  tcg/plugins: implement a qemu_plugin_user_exit helper
  contrib/gitdm: add more individual contributor entries.
  contrib/gitdm: add a new interns group-map for GSoC/Outreachy work
  contrib/gitdm: add an explicit academic entry for BU
  contrib/gitdm: add group-map for Netflix
  contrib/gitdm: add domain-map for NVIDIA
  contrib/gitdm: add domain-map for Crudebyte
  contrib/gitdm: un-ironically add a mapping for LWN
  contrib/gitdm: add domain-map/group-map for Wind River
  contrib/gitdm: add domain-map for Eldorado
  contrib/gitdm: add domain-map/group-map mappings for Samsung
  gitdm.config: sort the corporate GroupMap entries
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-07-26 11:00:15 +01:00
Philippe Mathieu-Daudé
0a9487d80a gitlab-ci: Extract OpenSBI job rules to reusable section
All jobs depending on 'docker-opensbi' job must use at most all
the rules that triggers it. The simplest way to ensure that
is to always use the same rules. Extract all the rules to a
reusable section, and include this section (with the 'extends'
keyword) in both 'docker-opensbi' and 'build-opensbi' jobs.

The problem was introduced in commit c6fc0fc1a7 ("gitlab-ci.yml:
Add jobs to build OpenSBI firmware binaries"), but was revealed in
commit 91e9c47e50 ("docker: OpenSBI build job depends on OpenSBI
container").

This fix is similar to the one used with the EDK2 firmware job in
commit ac0595cf6b ("gitlab-ci: Extract EDK2 job rules to reusable
section").

Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210720164829.3949558-1-philmd@redhat.com>
Message-Id: <20210720232703.10650-30-alex.bennee@linaro.org>
2021-07-23 17:22:16 +01:00
Thomas Huth
e90c3c3c87 gitlab-ci: Remove the second superfluous macos task
While there might have been bigger differnces between the -base and
the -xcode images in the beginning, they almost vanished in the
current builds, e.g. when comparing the output of the "configure"
step after cleaning up the differences due to temporary path names,
I only get:

  $ diff -u /tmp/base.txt /tmp/xcode.txt
  --- /tmp/base.txt	2021-07-16 09:16:24.211427940 +0200
  +++ /tmp/xcode.txt	2021-07-16 09:16:43.029684274 +0200
  @@ -19,14 +19,14 @@
   Build type: native build
   Project name: qemu
   Project version: 6.0.50
  -C compiler for the host machine: cc (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.29)")
  +C compiler for the host machine: cc (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.28)")
   C linker for the host machine: cc ld64 609.8
   Host machine cpu family: x86_64
   Host machine cpu: x86_64
   Program sh found: YES (/bin/sh)
   Program python3 found: YES (/usr/local/opt/python@3.9/bin/python3.9)
   Program bzip2 found: YES (/usr/bin/bzip2)
  -C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.29)")
  +C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.28)")
   C++ linker for the host machine: c++ ld64 609.8
   Objective-C compiler for the host machine: clang (clang 12.0.0)
   Objective-C linker for the host machine: clang ld64 609.8

Since we're not using Xcode itself at all, it seems like it does not
make much sense anymore to waste compute cycles with two images here.
Thus let's delete the -xcode job now.

[AJB: fix up commit formatting which trips up b4]

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210719073051.1559348-1-thuth@redhat.com>
Message-Id: <20210720232703.10650-29-alex.bennee@linaro.org>
2021-07-23 17:22:16 +01:00
Alex Bennée
39ce923732 gitlab: enable a very minimal build with the tricore container
Rather than base of the shared Debian 10 container which would require
us to bring in even more dependencies just bring in what is needed for
building tricore-softmmu in GitLab. We don't even remove the container
from the DOCKER_PARTIAL_IMAGES lest we cause more confusion.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210720232703.10650-28-alex.bennee@linaro.org>
2021-07-23 17:22:16 +01:00
Daniel P. Berrangé
eafadbbbac gitlab: only let pages be published from default branch
GitLab will happily publish pages generated by the latest CI pipeline
from any branch:

https://docs.gitlab.com/ee/user/project/pages/introduction.html

  "Remember that GitLab Pages are by default branch/tag agnostic
   and their deployment relies solely on what you specify in
   .gitlab-ci.yml. You can limit the pages job with the only
   parameter, whenever a new commit is pushed to a branch used
   specifically for your pages."

The current "pages" job is not limited, so it is happily publishing
docs content from any branch/tag in qemu.git that gets pushed to.
This means we're potentially publishing from the "staging" branch
or worse from outdated "stable-NNN" branches

This change restricts it to only publish from the default branch
in the main repository. For contributor forks, however, we allow
it to publish from any branch, since users will have arbitrarily
named topic branches in flight at any time.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210723113051.2792799-1-berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-07-23 16:02:56 +02:00
Gerd Hoffmann
8619b5ddb5 ci: build & store windows installer
Build windows installer for qemu in gitlab CI,
store the result as artifact.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210623091137.1156959-2-kraxel@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-19 09:33:39 +02:00
Peter Maydell
438951e883 Testing and plugin updates:
- custom runner playbooks for configuring GitLab runners
   - integrate Cirrus jobs into GitLab via cirrus-run
   - clean-up docker package lists
   - bump NetBSD to 9.2
   - bump OpenBSD to 6.9
   - make test-mmap more hexagon friendly
   - fixup handling of hostaddr for plugins
   - disallow some incompatible plugin configurations
   - fix handling of -ldl for BSDs
   - remove some old unused symbols from the plugin symbol map
   - enable plugins by default for most TCG builds
   - honour main build -Wall settings for plugins
   - new execlog plugin
   - new cache modelling plugin
   - fix io_uring build regression
   - disable modular TCG on Darwin
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmDu+xsACgkQ+9DbCVqe
 KkQ2XAf+MRBeSMX9J091iloiqSjpuobAcVuoIFciG7gEqWd2iTQVPDqJNgBJUANQ
 vTA7f89ljxj/FZjSZwLgR6cqY6X0ufXI1BLEIt7s78LJtVp14sHVo89GwNBzmRwo
 615T49KG8b5EBBU5YlVcAW/m8DlfgI4b1ufS/qHldOukKegu+haoCDjGG6RNpYNx
 mmXgLOBJiB/p2u2S73KTIa+3AEIIlsTJZpdLPus8hby+/Q4qB9t8YbdHaweyM9qs
 NjiojczvlbHLFd/IQSl6fqBS9QI+KWf4+oFd6ZB4THhr9/GUYTiMeLjRKucBgDQG
 TwGtd1RN3gAPI5aEJ0xlgCL/crz1qg==
 =dQl6
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-testing-and-plugins-140721-5' into staging

Testing and plugin updates:

  - custom runner playbooks for configuring GitLab runners
  - integrate Cirrus jobs into GitLab via cirrus-run
  - clean-up docker package lists
  - bump NetBSD to 9.2
  - bump OpenBSD to 6.9
  - make test-mmap more hexagon friendly
  - fixup handling of hostaddr for plugins
  - disallow some incompatible plugin configurations
  - fix handling of -ldl for BSDs
  - remove some old unused symbols from the plugin symbol map
  - enable plugins by default for most TCG builds
  - honour main build -Wall settings for plugins
  - new execlog plugin
  - new cache modelling plugin
  - fix io_uring build regression
  - disable modular TCG on Darwin

# gpg: Signature made Wed 14 Jul 2021 15:56:27 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-testing-and-plugins-140721-5: (44 commits)
  MAINTAINERS: Added myself as a reviewer for TCG Plugins
  docs/devel: Added cache plugin to the plugins docs
  plugins/cache: Added FIFO and LRU eviction policies
  plugins/cache: Enable cache parameterization
  plugins: Added a new cache modelling plugin
  docs/devel: tcg-plugins: add execlog plugin description
  contrib/plugins: add execlog to log instruction execution and memory access
  contrib/plugins: enable -Wall for building plugins
  tcg/plugins: enable by default for most TCG builds
  configure: stop user enabling plugins on Windows for now
  configure: add an explicit static and plugins check
  configure: don't allow plugins to be enabled for a non-TCG build
  tcg/plugins: remove some stale entries from the symbol list
  meson.build: relax the libdl test to one for the function dlopen
  meson.build: move TCG plugin summary output
  plugins: fix-up handling of internal hostaddr for 32 bit
  tests/tcg: make test-mmap a little less aggressive
  tests/vm: update openbsd to release 6.9
  tests/vm: update NetBSD to 9.2
  tests/docker: expand opensuse-leap package list
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-07-15 16:06:12 +01:00
Philippe Mathieu-Daudé
ac0595cf6b gitlab-ci: Extract EDK2 job rules to reusable section
All jobs depending on 'docker-edk2' job must use at most all
the rules that triggers it. The simplest way to ensure that
is to always use the same rules. Extract all the rules to a
reusable section, and include this section (with the 'extends'
keyword) in both 'docker-edk2' and 'build-edk2' jobs.

The problem was introduced in commit 71920809ce ("gitlab-ci.yml:
Add jobs to build EDK2 firmware binaries"), but was revealed in
commit 1925468ddb ("docker: EDK2 build job depends on EDK2
container") and eventually failed on CI:
https://gitlab.com/qemu-project/qemu/-/pipelines/335995843

Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210714101003.3113726-1-philmd@redhat.com>
2021-07-14 22:28:17 +02:00
Alex Bennée
ba4dd2aabc tcg/plugins: enable by default for most TCG builds
Aside from a minor bloat to file size the ability to have TCG plugins
has no real impact on performance unless a plugin is actively loaded.
Even then the libempty.so plugin shows only a minor degradation in
performance caused by the extra book keeping the TCG has to do to keep
track of instructions. As it's a useful feature lets just enable it by
default and reduce our testing matrix a little.

We need to move our linker testing earlier so we can be sure we can
enable the loader module required. As we have ruled out static &
plugins in an earlier patch we can also reduce the indent a little.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210709143005.1554-33-alex.bennee@linaro.org>
2021-07-14 15:54:13 +01:00
Daniel P. Berrangé
0e103a65ba gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
This adds support for running 4 jobs via Cirrus CI runners:

 * FreeBSD 12
 * FreeBSD 13
 * macOS 11 with default XCode
 * macOS 11 with latest XCode

The gitlab job uses a container published by the libvirt-ci
project (https://gitlab.com/libvirt/libvirt-ci) that contains
the 'cirrus-run' command. This accepts a short yaml file that
describes a single Cirrus CI job, runs it using the Cirrus CI
REST API, and reports any output to the console.

In this way Cirrus CI is effectively working as an indirect
custom runner for GitLab CI pipelines. The key benefit is that
Cirrus CI job results affect the GitLab CI pipeline result and
so the user only has look at one CI dashboard.

[AJB: remove $TEMPORARILY_DISABLED condition, s/py37/py38/]

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210625172211.451010-3-berrange@redhat.com>
Message-Id: <20210709143005.1554-8-alex.bennee@linaro.org>
2021-07-14 14:33:36 +01:00
Cleber Rosa
4799c21023 Jobs based on custom runners: add job definitions for QEMU's machines
The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests.  This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-5-crosa@redhat.com>
Message-Id: <20210709143005.1554-5-alex.bennee@linaro.org>
2021-07-14 14:31:48 +01:00
Cleber Rosa
b38a04f71f Jobs based on custom runners: documentation and configuration placeholder
As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place.  One of their primary
goals of catching and preventing regressions on a wider number of host
systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own
machine configuration documentation/scripts, and accompanying job
definitions.  As a general rule, those newly added contributed jobs
should run as "non-gating", until their reliability is verified (AKA
"allow_failure: true").

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-2-crosa@redhat.com>
Message-Id: <20210709143005.1554-2-alex.bennee@linaro.org>
2021-07-14 14:31:48 +01:00
Wainer dos Santos Moschetta
6f651a6d84 python: Configure tox to skip missing interpreters
Currently tox tests against the installed interpreters, however if any
supported interpreter is absent then it will return fail. It seems not
reasonable to expect developers to have all supported interpreters
installed on their systems. Luckily tox can be configured to skip
missing interpreters.

This changed the tox setup so that missing interpreters are skipped by
default. On the CI, however, we still want to enforce it tests
against all supported. This way on CI the
--skip-missing-interpreters=false option is passed to tox.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630184546.456582-1-wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-07-13 15:44:16 -04:00
John Snow
6f84d726f3 python: rename 'venv-check' target to 'check-pipenv'
Well, Cleber was right, this is a better name.

In preparation for adding a different kind of virtual environment check
(One that simply uses whichever version of Python you happen to have),
rename this test 'check-pipenv' so that it matches the CI job
'check-python-pipenv'.

Remove the "If you don't know which test to run" hint, because it's not
actually likely you have Python 3.6 installed to be able to run the
test. It's still the test I'd most prefer you to run, but it's not the
test you are most likely to be able to run.

Rename the 'venv' target to 'pipenv' as well, and move the more
pertinent help text under the 'check-pipenv' target.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-30 21:54:04 -04:00
Alex Bennée
66cf70149a scripts/checkpatch: roll diff tweaking into checkpatch itself
Rather than relying on external tweaks lets just do it inside
checkpatch's direct commitish handling which is QEMU specific code
anyway.

Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210623102749.25686-4-alex.bennee@linaro.org>
2021-06-25 10:08:33 +01:00
Philippe Mathieu-Daudé
e2d3017809 gitlab-ci: Split gprof-gcov job
This job is hitting the 70min limit, so split it in 2 tasks.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210525082556.4011380-7-f4bug@amsat.org>
2021-06-07 14:42:47 +01:00
Alex Bennée
b114a0b943 gitlab: work harder to avoid false positives in checkpatch
This copies the behaviour of patchew's configuration to make the diff
algorithm generate a minimal diff.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210602153247.27651-1-alex.bennee@linaro.org>
2021-06-07 14:42:47 +01:00
Peter Maydell
8c345b3e6a * Update the references to some doc files (use *.rst instead of *.txt)
* Bump minimum versions of some requirements after removing CentOS 7 support
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAmC3L1IRHHRodXRoQHJl
 ZGhhdC5jb20ACgkQLtnXdP5wLbU8wA//aRAIn0Qa3xoZdsT434P99GHauCZ5ePq3
 meY4co69c+TfkQZ/b0xlyvT3+7bd9Ni92CQL7n/LtX5bs3pIhRjHNIsm50e4x6Da
 0jb422cKWfRIltXlCfdL/dnFRtJluH83M0seRGMvmhveuWPZ19oSIasEiQjeg//e
 KMNK5z4HEnJ1czb3Bf8p38bmmY/O/QEAA5wAqk7iJkJHz6T/GlqImLYYwpFPlHj6
 JQttm0aWsHrsqEXxnuV0/DT1yHyXDB6S4iuAvABZWhv/M/nCaXo0ib0gW5NPtRPo
 Yf3HO163F9/fewJCc1AUsBe1C/2UwmSWRhEtxpr9uuW2Mv9qEl3hkJwd4k6sEOvh
 U4i+GONC4eElPcmECKUfHA9EP+7faDs6xnM6Ev/PIEp+cPJ2QRfklZv4qpMUWYtb
 3KkADchOyVZAsdB8cGmnznDEVmno1Dt0adVUq8CF6uW6MwD3pb4838arrfwIfwOp
 g4yTI1AQQykkxxOaR7GSNoxWRti7TH4fzLhx/xXDd9TKIOWuOepiyuhB7+Q48WAJ
 6EW/JUIzOe7k3GsI0iBsk+y67ED2tpATiRKWWw4QS1BwIhNEMUqgZkucFfKE6Ze1
 B1Xw+Di+1CzrLHV6lCjtdLVmqWqcGcO43HqhSK4sodE87BDGGwy9UxE6oBVt2C1x
 HPQ6/omDlFk=
 =RAkI
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/thuth-gitlab/tags/pull-request-2021-06-02' into staging

* Update the references to some doc files (use *.rst instead of *.txt)
* Bump minimum versions of some requirements after removing CentOS 7 support

# gpg: Signature made Wed 02 Jun 2021 08:12:18 BST
# gpg:                using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg:                issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg:                 aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg:                 aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg:                 aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3  EAB9 2ED9 D774 FE70 2DB5

* remotes/thuth-gitlab/tags/pull-request-2021-06-02:
  configure: bump min required CLang to 6.0 / XCode 10.0
  configure: bump min required GCC to 7.5.0
  configure: bump min required glib version to 2.56
  tests/docker: drop CentOS 7 container
  tests/vm: convert centos VM recipe to CentOS 8
  crypto: drop used conditional check
  crypto: bump min gnutls to 3.5.18, dropping RHEL-7 support
  crypto: bump min gcrypt to 1.8.0, dropping RHEL-7 support
  crypto: drop back compatibility typedefs for nettle
  crypto: bump min nettle to 3.4, dropping RHEL-7 support
  patchew: move quick build job from CentOS 7 to CentOS 8 container
  block/ssh: Bump minimum libssh version to 0.8.7
  docs: fix references to docs/devel/s390-dasd-ipl.rst
  docs: fix references to docs/specs/tpm.rst
  docs: fix references to docs/devel/build-system.rst
  docs: fix references to docs/devel/atomics.rst
  docs: fix references to docs/devel/tracing.rst

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-06-02 17:08:11 +01:00
Daniel P. Berrangé
6f0748a048 tests/docker: drop CentOS 7 container
It has been over two years since RHEL-8 was released, and thus per the
platform build policy, we no longer need to support RHEL-7 as a build
target.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210514120415.1368922-10-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02 09:11:32 +02:00
Daniel P. Berrangé
d4c7ee330c crypto: bump min gnutls to 3.5.18, dropping RHEL-7 support
It has been over two years since RHEL-8 was released, and thus per the
platform build policy, we no longer need to support RHEL-7 as a build
target. This lets us increment the minimum required gnutls version

Per repology, current shipping versions are:

             RHEL-8: 3.6.14
      Debian Buster: 3.6.7
 openSUSE Leap 15.2: 3.6.7
   Ubuntu LTS 18.04: 3.5.18
   Ubuntu LTS 20.04: 3.6.13
            FreeBSD: 3.6.15
          Fedora 33: 3.6.16
          Fedora 34: 3.7.1
            OpenBSD: 3.6.15
     macOS HomeBrew: 3.6.15

Ubuntu LTS 18.04 has the oldest version and so 3.5.18 is the new minimum.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210514120415.1368922-7-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
[thuth: rebased to use .gitlab-ci.d/buildtest.yml]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02 07:10:35 +02:00
Daniel P. Berrangé
b33a84632a crypto: bump min gcrypt to 1.8.0, dropping RHEL-7 support
It has been over two years since RHEL-8 was released, and thus per the
platform build policy, we no longer need to support RHEL-7 as a build
target. This lets us increment the minimum required gcrypt version and
assume that HMAC is always supported

Per repology, current shipping versions are:

             RHEL-8: 1.8.5
      Debian Buster: 1.8.4
 openSUSE Leap 15.2: 1.8.2
   Ubuntu LTS 18.04: 1.8.1
   Ubuntu LTS 20.04: 1.8.5
            FreeBSD: 1.9.2
          Fedora 33: 1.8.6
          Fedora 34: 1.9.3
            OpenBSD: 1.9.3
     macOS HomeBrew: 1.9.3

Ubuntu LTS 18.04 has the oldest version and so 1.8.0 is the new minimum.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210514120415.1368922-6-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
[thuth: rebased to use .gitlab-ci.d/buildtest.yml]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02 07:10:18 +02:00
Daniel P. Berrangé
20ba7a4a34 crypto: bump min nettle to 3.4, dropping RHEL-7 support
It has been over two years since RHEL-8 was released, and thus per the
platform build policy, we no longer need to support RHEL-7 as a build
target. This lets us increment the minimum required nettle version and
drop a lot of backwards compatibility code for 2.x series of nettle.

Per repology, current shipping versions are:

             RHEL-8: 3.4.1
      Debian Buster: 3.4.1
 openSUSE Leap 15.2: 3.4.1
   Ubuntu LTS 18.04: 3.4
   Ubuntu LTS 20.04: 3.5.1
            FreeBSD: 3.7.2
          Fedora 33: 3.5.1
          Fedora 34: 3.7.2
            OpenBSD: 3.7.2
     macOS HomeBrew: 3.7.2

Ubuntu LTS 18.04 has the oldest version and so 3.4 is the new minimum.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210514120415.1368922-4-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
[thuth: rebased to use .gitlab-ci.d/buildtest.yml]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02 07:03:25 +02:00
John Snow
6b9c277797 gitlab: add python linters to CI
Add a Python container that has just enough juice for us to run the
Python code quality analysis tools. Base this container on Fedora,
because Fedora has very convenient packaging for testing multiple Python
versions.

We need python3, pip (for pulling packages), pipenv and virtualenv for
creating virtual environments, and tox for running tests. make is needed
for running 'make check-tox' and 'make venv-check' targets. Python3.10
is needed explicitly because the tox package only pulls in 3.6-3.9, but
we wish to test the forthcoming release of Python as well to help
predict any problems. Lastly, we need gcc to compile PyPI packages that
may not have a binary distribution available.

Add two tests:

check-python-pipenv uses pipenv to test a frozen, very explicit set of
packages against our minimum supported python version, Python 3.6. This
test is not allowed to fail. The dependencies this test uses do not
change unless python/Pipfile.lock is changed.

check-python-tox uses tox to install the latest versions of required
python dependencies against a wide array of Python versions from 3.6 to
3.9, even including the yet-to-be-released Python 3.10. This test is
allowed to fail with a warning.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-32-jsnow@redhat.com
[Fix rebase conflict over .gitlab-ci.yml --js]
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-01 16:21:21 -04:00
Philippe Mathieu-Daudé
2d9e45b9cb gitlab: Convert check-dco/check-patch jobs to the 'rules' syntax
Per GitLab documentation [*]:

  "rules replaces only/except and they can’t be used together
   in the same job."

Since the 'rules' syntax is more powerful and we are already using
it, convert the check-dco/check-patch jobs so no job use the 'only/
except' syntax.

[*] https://docs.gitlab.com/ee/ci/yaml/#rules

Inspired-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210525132418.4133235-3-f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
1f88fad50e gitlab: Use $CI_DEFAULT_BRANCH instead of hardcoded 'master'
We want to skip the checkpatch and DCO signoff jobs when
pushing to the default branch. Currently this branch is
called 'master', but we don't need to hardcode this in
the CI configuration, because the $CI_DEFAULT_BRANCH
env variable exposes it.

References:

- https://sfconservancy.org/news/2020/jun/23/gitbranchname/
- https://about.gitlab.com/blog/2021/03/10/new-git-default-branch-name/

Suggested-by: Savitoj Singh <savsingh@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210525153826.4174157-1-philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
f56bf4caf7 gitlab: Run Avocado tests manually (except mainstream CI)
Due to a design problem and misunderstanding between the Avocado
framework and QEMU, Avocado is fetching many asset artifacts it
shouldn't be fetching, exhausting the jobs CI timeout.

Since Avocado artifacts are cached, this is not an issue with old
forks, which already have populated the cache and do not need to
download new artifacts to run the tests.

However this is very confusing to new contributors who start to
fork the project and keep having failing CI pipelines.

As a temporary kludge, add the QEMU_CI_AVOCADO_TESTING variable
to allow old forks to keep running the Avocado tests, while still
allowing new forks to use the mainstream set of CI tests.

Keep the tests enabled by default on the mainstream namespace
which is old enough to have a populated cache, hoping we will
keep this cache long enough until the Avocado/QEMU design issue
is fixed.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210525082556.4011380-9-f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
f230dba714 gitlab: Keep Avocado reports during 1 week
Sometimes pull requests are merged during the week-end, triggering
a CI pipeline. Currently if such pipeline fails, the Avocado reports
are available for 2 days. For the reviewers working on the project
during office hours, the reports are already discarded when they
want to look at them. Increase this time to 1 week, which should
give reviewers enough time.

Only keep the reports on failure, which is the only case we'll
look at them.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210525082556.4011380-8-f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
961889f00c gitlab: Extract cross-container jobs to container-cross.yml
Extract the jobs preparing the cross containers into a new file
(container-cross.yml).

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210525082556.4011380-5-f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
a151926c80 gitlab: Move current job set to qemu-project.yml
To allow forks to easily decide which jobs they want to run,
but without disrupting the current default, move the current
set of jobs to a new file corresponding to the jobs run by
the mainstream project CI:
https://gitlab.com/qemu-project/qemu/-/pipelines

Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210511072952.2813358-11-f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
f2f5c3e629 gitlab: Extract all default build/test jobs to buildtest.yml
Extract the build/test jobs run by default on the mainstream
CI into a new file (buildtest.yml).

Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210511072952.2813358-9-f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
6b505a4e4f gitlab: Extract core container jobs to container-core.yml
It is not possible to use the previously extracted templates
without this set of core containers. Extract them into a new
file (container-core.yml) to be able to build them without
having to build all the other containers by default.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210519185504.2198573-10-f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
6cd5251b40 gitlab: Extract default build/test jobs templates
To be able to reuse the mainstream build/test jobs templates,
extract them into a new file (buildtest-template.yml).

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210519185504.2198573-8-f4bug@amsat.org>
[thuth: Keep the "acceptance_test_job_template" name for now]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
07df3dfafa gitlab: Extract build stages to stages.yml
Extract the build stages used by our job templates to a new file
(stages.yml) to be able to include it with the other templates,
without having to run all the jobs included in the default
.gitlab-ci.yml, which are mainly useful for mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210519185504.2198573-7-f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
f659e6584a gitlab: Extract DCO/style check jobs to static_checks.yml
Extract the DCO / checkpatch jobs to a new file (static_checks.yml)
to be able to run them without having to run all the jobs included
in the default .gitlab-ci.yml, which are mainly useful for the
mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210519185504.2198573-6-f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
ecac85fe49 gitlab: Extract crossbuild job templates to crossbuild-template.yml
Extract the crossbuild job templates to a new file
(crossbuild-template.yml) to be able to reuse them
without having to run all the jobs included, which
are mainly useful for mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210519185504.2198573-5-f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-27 16:39:23 +02:00
Philippe Mathieu-Daudé
fb3c0334de gitlab: Extract container job template to container-template.yml
Extract the container job template to a new file
(container-template.yml) to be able to reuse it
without having to run all the jobs included, which
are mainly useful for mainstream CI.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210519185504.2198573-4-f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-27 16:39:23 +02:00
Richard Henderson
840cafced1 gitlab: Enable cross-i386 builds of TCI
We're currently only testing TCI with a 64-bit host -- also test
with a 32-bit host.  Enable a selection of softmmu and user-only
targets, 32-bit LE, 64-bit LE, 32-bit BE, as there are ifdefs for each.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210502235727.1979457-27-richard.henderson@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Richard Henderson
e4af2160e1 gitlab: Rename ACCEL_CONFIGURE_OPTS to EXTRA_CONFIGURE_OPTS
Suggested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210502235727.1979457-26-richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2021-05-27 16:39:23 +02:00
Alex Bennée
1aab5f0536 gitlab: add special rule for the hexagon container
The hexagon container is always manually built but of course not
everyone will be building it themselves and pushing to their
registries. We still need to create a "local" registry copy for the
actual gitlab tests to run. We don't build it in this case, just pull
it across from the upstream registry. We disable this rule from
running on the qemu-project itself so it doesn't accidentally wipe out
our master copy.

Fixes: 910c40ee94 ("gitlab: add build-user-hexagon test")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Cornelia Huck <cohuck@redhat.com>
Message-Id: <20210520174303.12310-4-alex.bennee@linaro.org>
2021-05-25 16:52:44 +01:00
Alex Bennée
0b84609bbd gitlab: explicitly reference the upstream registry
Since c8e6793903 ("containers.yml: build with docker.py tooling") we
don't need to manually pull stuff from the upstream repository. Just
set the -r field to explicitly use that rather than the current
registry.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20210520174303.12310-3-alex.bennee@linaro.org>
2021-05-25 09:22:47 +01:00
Matheus Ferst
cf9b6d215e tests/docker: gcc-10 based images for ppc64{,le} tests
A newer compiler is needed to build tests for Power10 instructions. As
done for arm64 on c729a99d27, a new '-test-cross' image is created for
ppc64 and ppc64le. As done on 936fda4d77, a test for compiler support
is added to verify that the toolchain in use has '-mpower10'. Finally,
Unused images (docker-power-cross and docker-ppc64-cross) are removed.

Signed-off-by: Matheus Ferst <matheus.ferst@eldorado.org.br>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210423205757.1752480-2-matheus.ferst@eldorado.org.br>
Message-Id: <20210512102051.12134-27-alex.bennee@linaro.org>
2021-05-18 09:36:21 +01:00
Philippe Mathieu-Daudé
1d8b96126e gitlab-ci: Replace YAML anchors by extends (container_job)
'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210418233448.1267991-2-f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-05-14 12:28:01 +02:00
Philippe Mathieu-Daudé
1c13514449 gitlab-ci: Add KVM mips64el cross-build jobs
Add a new job to cross-build the mips64el target without
the TCG accelerator (IOW: only KVM accelerator enabled).

Only build the mips64el target which is known to work
and has users.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20210428170410.479308-31-f4bug@amsat.org>
2021-05-02 16:49:35 +02:00
Alex Bennée
a9eb2df27f gitlab: default to not building the documentation
In d0f26e68a0 ("gitlab: force enable docs build in Fedora, Ubuntu,
Debian") we made sure we can build the documents on more than one
system. However we don't want to build documents all the time as it's
a waste of cycles (and energy). So lets reduce the total amount of
documentation we build while still keeping coverage of at least one
build on each supported target.

Fixes: a8a3abe0b3 ("gitlab: move docs and tools build across from Travis")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210323165308.15244-23-alex.bennee@linaro.org>
2021-03-24 14:25:48 +00:00
Philippe Mathieu-Daudé
91e9c47e50 docker: OpenSBI build job depends on OpenSBI container
Add missing dependency build-opensbi -> docker-opensbi.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210303130646.1494015-4-philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-03-09 06:03:53 +01:00
Philippe Mathieu-Daudé
1925468ddb docker: EDK2 build job depends on EDK2 container
Add missing dependency build-edk2 -> docker-edk2.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210303130646.1494015-3-philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-03-09 06:03:53 +01:00
Daniel P. Berrangé
764a0747a7 gitlab: add fine grained job deps for all build jobs
This allows the build jobs to start running as soon as their respective
container image is ready, instead of waiting for all container builds
to finish.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210216132954.295906-3-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-02-19 06:29:05 +01:00
Daniel P. Berrangé
c31fa24e96 gitlab: always build container images
Currently we attempt to skip building container images if the commits do
not involve changes to the dockerfiles or gitlab CI definitions.

Conceptually this makes sense, but there is a challenge in the real
world implementation of this in gitlab.

In the case of a CI pipeline triggered from a merge request, GitLab
knows the common ancestor of the merge request and the main git repo,
so it can trivially determine if any of the commits associated with
the MR change the dockerfiles.

In the case of a CI pipeline triggered from a push to a branch, it is
much more difficult. There is no concept of a common ancestor in this
case. Instead GitLab looks at the set of commits in the git push event.

On the surface this may sound reasonable, but it doesn't take into
account that a push event does not always contain the full set of
patches from a branch.

For example, consider pushing 5 commits, one of which contains a
dockerfile change. This will trigger a CI pipeline for the
containers. Now consider you do some more work on the branch and push 3
further commits, so you now have a branch of 8 commits. For the second
push GitLab will only look at the 3 most recent commits, the other 5
were already present. Thus GitLab will not realize that the branch has
dockerfile changes that need to trigger the container build.

This can cause real world problems:

 - Push 5 commits to branch "foo", including a dockerfile change

    => rebuilds the container images with content from "foo"
    => build jobs runs against containers from "foo"

 - Refresh your master branch with latest upstream master

    => rebuilds the container images with content from "master"
    => build jobs runs against containers from "master"

 - Push 3 more commits to branch "foo", with no dockerfile change

    => no container rebuild triggers
    => build jobs runs against containers from "master"

The "changes" conditional in gitlab is OK, *provided* your build
jobs are not relying on any external state from previous builds.

This is NOT the case in QEMU, because we are building container
images and these are cached. This is a scenario in which the
"changes" conditional is not usuable.

The only other way to avoid this problem would be to use the git
branch name as the container image tag, instead of always using
"latest". The downside of this approach is that the user's gitlab
registry will grow significantly until it starts to trigger
GitLab's automatic deletion policy.  Every time the user starts
a new branch they will have to trigger a rebuild of the container
images. Given this, we might as well just drop the conditional
and always build the container images. Most of the time docker
will be able to use the layer cache to avoid the most expensive
part of the rebuild process (installing all the RPMs/debs/etc)

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210216132954.295906-2-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-02-19 06:29:04 +01:00
Jiaxun Yang
b22786eab0 gitlab-ci: Add alpine to pipeline
[thuth: Enable "make check" tests, too, after tests/check-block.sh got fixed]

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
Message-Id: <20210118063808.12471-10-jiaxun.yang@flygoat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-20 10:46:54 +01:00
Cho, Yu-Chen
b115ea3a0d gitlab-ci.yml: Add openSUSE Leap 15.2 for gitlab CI/CD
Add build-system-opensuse jobs and opensuse-leap.docker dockerfile.
Use openSUSE Leap 15.2 container image in the gitlab-CI.

Signed-off-by: Cho, Yu-Chen <acho@suse.com>
Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20201229085046.8536-1-acho@suse.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:08 +01:00
Thomas Huth
ee381b7fe1 gitlab-CI: Test 32-bit builds with the fedora-i386-cross container
After adding some missing packages, it's possible to check 32-bit
builds and tests with the fedora-i386-cross container in the gitlab-CI,
too. Unfortunately, the code in subprojects/ ignores the --extra-cflags
(on purpose), so the vhost-user part has to be disabled for this.

While we're at it, update the container to Fedora 31. Unfortunately the
gcc from the later versions emits some very dubious format-truncation
warnings, so Fedora 32 and 33 are currently unsuitable for this job.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20201215083451.92322-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2021-01-02 21:03:09 +01:00
Alex Bennée
f3890e71ac gitlab: include aarch64-softmmu and ppc64-softmmu cross-system-build
Otherwise we miss coverage of KVM support in the cross build. To
balance it out add arm-softmmu (no kvm, subset of aarch64),
cris-softmmu and ppc-softmmu to the exclude list which do get coverage
elsewhere.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20201210190417.31673-3-alex.bennee@linaro.org>
2021-01-02 21:03:09 +01:00
Philippe Mathieu-Daudé
84eda11079 gitlab-ci: Add Xen cross-build jobs
Cross-build ARM and X86 targets with only Xen accelerator enabled.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201207131503.3858889-6-philmd@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-12-09 08:04:34 +01:00
Philippe Mathieu-Daudé
b16999a32c gitlab-ci: Add KVM s390x cross-build jobs
Cross-build s390x target with only KVM accelerator enabled.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201207131503.3858889-5-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-12-09 08:04:34 +01:00
Philippe Mathieu-Daudé
0cdd3b0aa4 gitlab-ci: Introduce 'cross_accel_build_job' template
Introduce a job template to cross-build accelerator specific
jobs (enable a specific accelerator, disabling the others).

The specific accelerator is selected by the $ACCEL environment
variable (default to KVM).

Extra options such disabling other accelerators are passed
via the $ACCEL_CONFIGURE_OPTS environment variable.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201207131503.3858889-4-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-12-09 08:04:34 +01:00
Philippe Mathieu-Daudé
b62ea74648 gitlab-ci: Replace YAML anchors by extends (cross_system_build_job)
'extends' is an alternative to using YAML anchors
and is a little more flexible and readable. See:
https://docs.gitlab.com/ee/ci/yaml/#extends

More importantly it allows exploding YAML jobs.

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201207131503.3858889-3-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-12-09 08:04:34 +01:00
Rebecca Cran
ca905bec44 gitlab-ci: Use $CI_REGISTRY instead of hard-coding registry.gitlab.com
Update containers.yml to use the $CI_REGISTRY variable as other files
such as edk2.yml do.

Signed-off-by: Rebecca Cran <rebecca@nuviainc.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201113172519.31056-1-rebecca@nuviainc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-11-17 09:45:24 +01:00
Alex Bennée
7025111a19 .gitlab-ci.d/check-patch: tweak output for CI logs
We don't need running commentary for the CI logs and by keeping it
short we might just see the problem on the first page. While we are at
it flush the previous line so order is maintained between script and
sub process.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20201113174404.19608-1-alex.bennee@linaro.org>
2020-11-16 11:08:40 +00:00
Daniel P. Berrangé
faf9828e2f gitlab: skip checkpatch.pl checks if no commit delta on branch
If the current branch is synced to the current upstream git master,
there are no commits that need checking. This causes checkpatch.pl
to print an error that it found no commits. We need to avoid calling
checkpatch.pl in this case.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201019143537.283094-2-berrange@redhat.com>
Message-Id: <20201021163136.27324-5-alex.bennee@linaro.org>
2020-10-27 09:53:48 +00:00
Daniel P. Berrangé
34ed46a284 gitlab: add a CI job to validate the DCO sign off
While checkpatch.pl can validate DCO sign off that job must always be
advisory only since it is expected that certain patches will fail some
code style rules.

We require the DCO sign off to be mandatory for all commits though, so
it benefits from being validated in a standalone job.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200918132903.1848939-3-berrange@redhat.com>
[thuth: Use "stage: build" to let it run earlier]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-10-13 12:48:17 +02:00
Daniel P. Berrangé
1f47547256 gitlab: add a CI job for running checkpatch.pl
This job is advisory since it is expected that certain patches will fail
the style checks and checkpatch.pl provides no way to mark exceptions to
the rules.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200918132903.1848939-2-berrange@redhat.com>
[thuth: Use "stage: build" to let it run earlier]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-10-13 12:48:17 +02:00
Thomas Huth
254ef17e3f gitlab-ci: Increase the timeout for the cross-compiler builds
Some of the cross-compiler builds (the mips build and the win64 build
for example) are quite slow and sometimes hit the 1h time limit.
Increase the limit a little bit to make sure that we do not get failures
in the CI runs just because of some few minutes.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200921174320.46062-7-thuth@redhat.com>
Message-Id: <20200925154027.12672-15-alex.bennee@linaro.org>
2020-10-02 12:29:21 +01:00
Thomas Huth
e3755276d1 tests/docker: Remove old Debian 9 containers
We do not support Debian 9 in QEMU anymore, and the Debian 9 containers
are now no longer used in the gitlab-CI. Time to remove them.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200921174320.46062-6-thuth@redhat.com>
Message-Id: <20200925154027.12672-14-alex.bennee@linaro.org>
2020-10-02 12:29:15 +01:00
Thomas Huth
cf63177e59 tests/docker: Update the tricore container to debian 10
We do not support Debian 9 anymore, thus update the Tricore container
to Debian 10 now.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200921174320.46062-4-thuth@redhat.com>
Message-Id: <20200925154027.12672-12-alex.bennee@linaro.org>
2020-10-02 12:29:08 +01:00
Thomas Huth
5fdf6e22d9 gitlab-ci: Remove the Debian9-based containers and containers-layer3
According to our support policy, Debian 9 is not supported by the
QEMU project anymore. Since we now switched the MinGW cross-compiler
builds to Fedora, we do not need these Debian9-based containers
in the gitlab-CI anymore, and can now also get rid of the "layer3"
container build stage this way.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200921174320.46062-3-thuth@redhat.com>
Message-Id: <20200925154027.12672-11-alex.bennee@linaro.org>
2020-10-02 12:29:04 +01:00
Thomas Huth
93cc0506f6 tests/docker: Use Fedora containers for MinGW cross-builds in the gitlab-CI
According to our support policy, we do not support Debian 9 in QEMU
anymore, and we only support building the Windows binaries with a
very recent version of the MinGW toolchain. So we should not test
the MinGW cross-compilation with Debian 9 anymore, but switch to
something newer like Fedora. To do this, we need a separate Fedora
container for each build that provides the QEMU_CONFIGURE_OPTS
environment variable.
Unfortunately, the MinGW 64-bit compiler seems to be a little bit
slow, so we also have to disable some features like "capstone" in the
build here to make sure that the CI pipelines still finish within a
reasonable amount of time.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200921174320.46062-2-thuth@redhat.com>
Message-Id: <20200925154027.12672-10-alex.bennee@linaro.org>
2020-10-02 12:29:02 +01:00
Thomas Huth
6bcb5fc0f7 gitlab-ci: Add cross-compiling build tests
Now that we can use all our QEMU test containers in the gitlab-CI, we can
easily add some jobs that test cross-compilation for various architectures.
There is just only small ugliness: Since the shared runners on gitlab.com
are single-threaded, we have to split each compilation job into two parts
(--disable-user and --disable-system), and exclude some additional targets,
to avoid that the jobs are running too long and hitting the timeout of 1 h.

Message-Id: <20200823111757.72002-8-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-09-07 12:34:17 +02:00
Bin Meng
a52ea3e7d2 gitlab-ci/opensbi: Update GitLab CI to build generic platform
This updates the GitLab CI opensbi job to build opensbi bios images
for the generic platform.

Signed-off-by: Bin Meng <bin.meng@windriver.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Message-Id: <1596439832-29238-7-git-send-email-bmeng.cn@gmail.com>
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
2020-08-21 22:37:55 -07:00
Thomas Huth
6f60a240f6 gitlab-ci/containers: Add missing wildcard where we should look for changes
The tests/docker/* wildcard seems to only match the files that are directly
in the tests/docker folder - but changes to the files in the directory
tests/docker/dockerfiles are currently ignored. Seems like we need a
separate entry to match the files in that folder. With this wildcard added,
the stages now get re-run successfully when something in the dockerfiles
has been changed.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200713182235.30379-1-thuth@redhat.com>
2020-07-15 11:52:43 +01:00
Alex Bennée
c8e6793903 containers.yml: build with docker.py tooling
Instead of building the docker files directly use the same docker.py
scripting as we do for building locally. This should help ensure we
use the exact same steps and allow us to cache properly when building
locally.

To get this working you have to have a fairly recent docker binary
otherwise you will see the error message:

 => ERROR importing cache manifest from registry.gitlab....

So far docker 19.03.12 works (from the docker apt repos) but 18.09.1,
build 4c52b90 which is packaged in Debian Buster fails.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200701135652.1366-39-alex.bennee@linaro.org>
2020-07-11 15:53:00 +01:00
Alex Bennée
f2c5466c78 gitlab: limit re-builds of the containers
Most of the time we are just rebuilding the same things. We can skip
this although currently there is no mechanism for picking up new
distro releases.

Rather than try to be too fine grained allow any change to trigger all
the images being rebuilt.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Message-Id: <20200701135652.1366-38-alex.bennee@linaro.org>
2020-07-11 15:53:00 +01:00
Alex Bennée
27ebeda0c0 tests/docker: add a linux-user testing focused image
We happily use all the cross images for both cross-building QEMU as
well as building the linux-user tests. However calling docker from
within docker seems not to work. As we can build in Debian anyway why
not include an image that has all the compilers available for
non-docker invocation.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200701135652.1366-33-alex.bennee@linaro.org>
2020-07-11 15:53:00 +01:00
Alex Bennée
8a8a50a957 gitlab: build containers with buildkit and metadata
According to the documentation to be able to use --cache-from for
remote registries you need to enable both buildkit and inline the
metadata. We want to do this to support pulling from gitlab when users
build their local docker images.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200701135652.1366-24-alex.bennee@linaro.org>
2020-07-11 15:53:00 +01:00
Daniel P. Berrangé
fa821f23e1 gitlab: build all container images during CI
We have a number of container images in tests/docker/dockerfiles
that are intended to provide well defined environments for doing
test builds. We want our CI system to use these containers too.

This introduces builds of all of them as the first stage in the
CI, so that the built containers are available for later build
jobs. The containers are setup to use the GitLab container
registry as the cache, so we only pay the penalty of the full
build when the dockerfiles change. The main qemu-project/qemu
repo is used as a second cache, so that users forking QEMU will
see a fast turnaround time on their CI jobs.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200622153318.751107-3-berrange@redhat.com>
[AJB: tweak the tag format]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200701135652.1366-22-alex.bennee@linaro.org>
2020-07-11 15:53:00 +01:00
Daniel P. Berrangé
d0caa0a881 gitlab: introduce explicit "container" and "build" stages
If no stage is listed, jobs get put in an implicit "test" stage.
Some jobs which create container images to be used by later stages
are currently listed as in a "build" stages.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200622153318.751107-2-berrange@redhat.com>
Message-Id: <20200701135652.1366-21-alex.bennee@linaro.org>
2020-07-11 15:53:00 +01:00
Thomas Huth
5117ba25d2 gitlab-ci: Fix the change rules after moving the YML files
The edk2.yml and opensbi.yml files have recently been moved/renamed,
but the change has not been reflected in the rules in the YML files
yet.

Fixes: 922febe2af ("Move edk2 and opensbi YAML files to .gitlab-ci.d folder")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20200625151627.24986-1-thuth@redhat.com>
Message-Id: <20200701135652.1366-20-alex.bennee@linaro.org>
2020-07-11 15:53:00 +01:00
Thomas Huth
922febe2af gitlab-ci: Move edk2 and opensbi YAML files to .gitlab-ci.d folder
We have a dedicated folder for the gitlab-ci - so there is no need
to clutter the top directory with these .yml files.

Message-Id: <20200525131823.715-5-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2020-05-28 11:00:39 +02:00
Bin Meng
c6fc0fc1a7
gitlab-ci.yml: Add jobs to build OpenSBI firmware binaries
Add two GitLab jobs to build the OpenSBI firmware binaries.

The first job builds a Docker image with the packages requisite
to build OpenSBI, and stores this image in the GitLab registry.
The second job pulls the image from the registry and builds the
OpenSBI firmware binaries.

The docker image is only rebuilt if the GitLab YAML or the
Dockerfile is updated. The second job is only built when the
roms/opensbi/ submodule is updated, when a git-ref starts with
'opensbi' or when the last commit contains 'OpenSBI'. The files
generated are archived in the artifacts.zip file.

With OpenSBI v0.6, it took 2 minutes 56 seconds to build
the docker image, and 1 minute 24 seconds to generate the
artifacts.zip with the firmware binaries (filesize: 111KiB).

See: https://gitlab.com/lbmeng/qemu/pipelines/120520138

Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2020-03-16 17:03:50 -07:00
Philippe Mathieu-Daudé
71920809ce gitlab-ci.yml: Add jobs to build EDK2 firmware binaries
Add two GitLab job to build the EDK2 firmware binaries.

The first job build a Docker image with the packages requisite
to build EDK2, and store this image in the GitLab registry.
The second job pull the image from the registry and build the
EDK2 firmware binaries.

The docker image is only rebuilt if the GitLab YAML or the
Dockerfile is updated.
The second job is only built when the roms/edk2/ submodule is
updated, when a git-ref starts with 'edk2' or when the last
commit contains 'EDK2'. The files generated are archived in
the artifacts.zip file.

With edk2-stable201905, it took 2 minutes 52 seconds to build
the docker image, and 36 minutes 28 seconds to generate the
artifacts.zip with the firmware binaries (filesize: 10MiB).

See: https://gitlab.com/philmd/qemu/pipelines/107553178

Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2020-01-21 14:15:48 +01:00