mirrors/qemu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
70f2011015
qemu/.gitlab-ci.d/containers.yml
Daniel P. Berrangé c31fa24e96 gitlab: always build container images 
Currently we attempt to skip building container images if the commits do
not involve changes to the dockerfiles or gitlab CI definitions.

Conceptually this makes sense, but there is a challenge in the real
world implementation of this in gitlab.

In the case of a CI pipeline triggered from a merge request, GitLab
knows the common ancestor of the merge request and the main git repo,
so it can trivially determine if any of the commits associated with
the MR change the dockerfiles.

In the case of a CI pipeline triggered from a push to a branch, it is
much more difficult. There is no concept of a common ancestor in this
case. Instead GitLab looks at the set of commits in the git push event.

On the surface this may sound reasonable, but it doesn't take into
account that a push event does not always contain the full set of
patches from a branch.

For example, consider pushing 5 commits, one of which contains a
dockerfile change. This will trigger a CI pipeline for the
containers. Now consider you do some more work on the branch and push 3
further commits, so you now have a branch of 8 commits. For the second
push GitLab will only look at the 3 most recent commits, the other 5
were already present. Thus GitLab will not realize that the branch has
dockerfile changes that need to trigger the container build.

This can cause real world problems:

 - Push 5 commits to branch "foo", including a dockerfile change

    => rebuilds the container images with content from "foo"
    => build jobs runs against containers from "foo"

 - Refresh your master branch with latest upstream master

    => rebuilds the container images with content from "master"
    => build jobs runs against containers from "master"

 - Push 3 more commits to branch "foo", with no dockerfile change

    => no container rebuild triggers
    => build jobs runs against containers from "master"

The "changes" conditional in gitlab is OK, *provided* your build
jobs are not relying on any external state from previous builds.

This is NOT the case in QEMU, because we are building container
images and these are cached. This is a scenario in which the
"changes" conditional is not usuable.

The only other way to avoid this problem would be to use the git
branch name as the container image tag, instead of always using
"latest". The downside of this approach is that the user's gitlab
registry will grow significantly until it starts to trigger
GitLab's automatic deletion policy.  Every time the user starts
a new branch they will have to trigger a rebuild of the container
images. Given this, we might as well just drop the conditional
and always build the container images. Most of the time docker
will be able to use the layer cache to avoid the most expensive
part of the rebuild process (installing all the RPMs/debs/etc)

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210216132954.295906-2-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-02-19 06:29:04 +01:00

252 lines
5.8 KiB
YAML
Raw Blame History

.container_job_template: &container_job_definition
image: docker:stable
stage: containers
services:
- docker:dind
before_script:
- export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
- export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest"
- apk add python3
- docker info
- docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
script:
- echo "TAG:$TAG"
- echo "COMMON_TAG:$COMMON_TAG"
- docker pull "$TAG" || docker pull "$COMMON_TAG" || true
- ./tests/docker/docker.py --engine docker build
-t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
-r $CI_REGISTRY_IMAGE
- docker tag "qemu/$NAME" "$TAG"
- docker push "$TAG"
after_script:
- docker logout
amd64-alpine-container:
<<: *container_job_definition
variables:
NAME: alpine
amd64-centos7-container:
<<: *container_job_definition
variables:
NAME: centos7
amd64-centos8-container:
<<: *container_job_definition
variables:
NAME: centos8
amd64-debian10-container:
<<: *container_job_definition
variables:
NAME: debian10
amd64-debian11-container:
<<: *container_job_definition
variables:
NAME: debian11
alpha-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-alpha-cross
amd64-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-amd64-cross
amd64-debian-user-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-all-test-cross
amd64-debian-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-amd64
arm64-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-arm64-cross
arm64-test-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian11-container']
variables:
NAME: debian-arm64-test-cross
armel-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-armel-cross
armhf-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-armhf-cross
hppa-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-hppa-cross
m68k-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-m68k-cross
mips64-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-mips64-cross
mips64el-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-mips64el-cross
mips-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-mips-cross
mipsel-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-mipsel-cross
powerpc-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-powerpc-cross
ppc64-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-ppc64-cross
ppc64el-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-ppc64el-cross
riscv64-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-riscv64-cross
s390x-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-s390x-cross
sh4-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-sh4-cross
sparc64-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-sparc64-cross
tricore-debian-cross-container:
<<: *container_job_definition
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
NAME: debian-tricore-cross
xtensa-debian-cross-container:
<<: *container_job_definition
variables:
NAME: debian-xtensa-cross
cris-fedora-cross-container:
<<: *container_job_definition
variables:
NAME: fedora-cris-cross
amd64-fedora-container:
<<: *container_job_definition
variables:
NAME: fedora
i386-fedora-cross-container:
<<: *container_job_definition
variables:
NAME: fedora-i386-cross
win32-fedora-cross-container:
<<: *container_job_definition
variables:
NAME: fedora-win32-cross
win64-fedora-cross-container:
<<: *container_job_definition
variables:
NAME: fedora-win64-cross
amd64-ubuntu1804-container:
<<: *container_job_definition
variables:
NAME: ubuntu1804
amd64-ubuntu2004-container:
<<: *container_job_definition
variables:
NAME: ubuntu2004
amd64-ubuntu-container:
<<: *container_job_definition
variables:
NAME: ubuntu
amd64-opensuse-leap-container:
<<: *container_job_definition
variables:
NAME: opensuse-leap
Reference in New Issue View Git Blame Copy Permalink