Commit Graph

68 Commits

Author SHA1 Message Date
Stefan Hajnoczi
c2d3189667 docker.py: Python 2.6 argparse compatibility
Add the scripts/ directory to sys.path so Python 2.6 will be able to
import argparse.

Cc: Fam Zheng <famz@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: John Snow <jsnow@redhat.com>
Acked-by: Fam Zheng <famz@redhat.com>
Message-id: 20170825155732.15665-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2017-08-30 12:02:11 +01:00
Fam Zheng
0b95ff72cb docker.py: Improve subprocess exit code handling
A few error handlings are missing because we ignore the subprocess exit
code, for example "docker build" errors are currently ignored.

Introduce _do_check() aside the existing _do() method and use it in a
few places.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170712075528.22770-3-famz@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-07-17 11:34:20 +08:00
Fam Zheng
58bf7b6d8c docker.py: Drop infile parameter
The **kwargs can do this just well.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <20170712075528.22770-2-famz@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-07-17 11:34:20 +08:00
Philippe Mathieu-Daudé
438d116872 docker: rebuild image if 'extra files' checksum does not match
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé
4c84f662c2 docker: add --include-files argument to 'build' command
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé
2499ee9fad docker: let _copy_with_mkdir() sub_path argument be optional
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2017-06-21 15:01:48 +01:00
Philippe Mathieu-Daudé
06cc355171 tests/docker: support proxy / corporate firewall
if ftp_proxy/http_proxy/https_proxy standard environment variables available,
pass them to the docker daemon to build images.
this is required when building behind corporate proxy/firewall, but also help
when using local cache server (ie: apt/yum).

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170306205520.32311-1-f4bug@amsat.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-03-07 18:20:40 +08:00
Alex Bennée
414a8ce57e tests/docker: add basic user mapping support
Currently all docker builds are done by exporting a tarball to the
docker container and running the build as the containers root user.
Other use cases are possible however and it is possible to map a part
of users file-system to the container. This is useful for example for
doing cross-builds of arbitrary source trees. For this to work
smoothly the container needs to have a user created that maps cleanly
to the host system.

This adds a -u option to the docker script so that:

  DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \
    -u --include-executable=arm-linux-user/qemu-arm \
    debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker

Will build a container that can then be run like:

  docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \
    --user=alex:alex -w /src/ debian:armhf \
    sh -c "make clean && ./configure -s && make"

All docker containers built will add the current user unless
explicitly disabled by specifying NOUSER when invoking the Makefile:

  make docker-image-debian-armhf-cross NOUSER=1

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20170220105139.21581-2-alex.bennee@linaro.org>
Signed-off-by: Fam Zheng <famz@redhat.com>
2017-02-24 14:18:11 +08:00
Fam Zheng
97cba1a1d1 docker: Terminate instances at SIGTERM and SIGHUP
Signed-off-by: Fam Zheng <famz@redhat.com>
Message-Id: <1474429768-25027-9-git-send-email-famz@redhat.com>
Reviewed-by: Daniel P. Berrange <berrange@redhat.com>
2016-09-23 11:48:32 +08:00
Eduardo Habkost
0679f98b40 docker: Handle exceptions when looking for docker command
When trying to run docker tests on a host without the docker
command,  we get the following Python backtrace:

  $ make docker-test-quick@centos6 V=1
  .../qemu/tests/docker/docker.py build qemu:centos6 .../qemu/tests/docker/dockerfiles/centos6.docker
  Traceback (most recent call last):
    File ".../qemu/tests/docker/docker.py", line 339, in <module>
      sys.exit(main())
    File ".../qemu/tests/docker/docker.py", line 336, in main
      return args.cmdobj.run(args, argv)
    File ".../qemu/tests/docker/docker.py", line 231, in run
      dkr = Docker()
    File ".../qemu/tests/docker/docker.py", line 98, in __init__
      self._command = _guess_docker_command()
    File ".../qemu/tests/docker/docker.py", line 41, in _guess_docker_command
      stdout=DEVNULL, stderr=DEVNULL) == 0:
    File "/usr/lib64/python2.7/subprocess.py", line 523, in call
      return Popen(*popenargs, **kwargs).wait()
    File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
      errread, errwrite)
    File "/usr/lib64/python2.7/subprocess.py", line 1343, in _execute_child
      raise child_exception
  OSError: [Errno 2] No such file or directory
  .../qemu/tests/docker/Makefile.include:47: recipe for target 'docker-image-centos6' failed
  make: *** [docker-image-centos6] Error 1

Change _guess_docker_command() to handle OSError exceptions
raised by subprocess.call(), so we will keep looking for other
commands and print a better error message.

New output will be:

  $ make docker-test-quick@centos6 V=1
  .../qemu/tests/docker/docker.py build qemu:centos6 .../qemu/tests/docker/dockerfiles/centos6.docker
  Traceback (most recent call last):
    File ".../qemu/tests/docker/docker.py", line 343, in <module>
      sys.exit(main())
    File ".../qemu/tests/docker/docker.py", line 340, in main
      return args.cmdobj.run(args, argv)
    File ".../qemu/tests/docker/docker.py", line 235, in run
      dkr = Docker()
    File ".../qemu/tests/docker/docker.py", line 102, in __init__
      self._command = _guess_docker_command()
    File ".../qemu/tests/docker/docker.py", line 49, in _guess_docker_command
      commands_txt)
  Exception: Cannot find working docker command. Tried:
    docker
    sudo -n docker
  .../qemu/tests/docker/Makefile.include:47: recipe for target 'docker-image-centos6' failed
  make: *** [docker-image-centos6] Error 1

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <1474369559-16903-1-git-send-email-ehabkost@redhat.com>
[exceptions.OSError -> OSError and drop the import. - Fam]
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-23 11:38:57 +08:00
Sascha Silbe
f8042deafa docker: silence debootstrap when --quiet is given
If we silence docker when --quiet is given, we should also silence the
.pre script (i.e. debootstrap).

Only discards stdout, so some diagnostics (e.g. from git clone) are
still printed. Most of the verbose output is gone however and this way
we still have a chance to see error messages.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Message-Id: <1473192351-601-9-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Sascha Silbe
c977257045 docker.py: don't hang on large docker output
Unlike Popen.communicate(), subprocess.call() doesn't read from the
stdout file descriptor. If the child process produces more output than
fits into the pipe buffer, it will block indefinitely.

If we don't intend to consume the output, just send it straight to
/dev/null to avoid this issue.

Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com>
Reviewed-by: Janosch Frank <frankja@linux.vnet.ibm.com>
Message-Id: <1473192351-601-2-git-send-email-silbe@linux.vnet.ibm.com>
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-09-08 19:56:34 +08:00
Fam Zheng
4b08af6019 docker: Add "images" subcommand to docker.py
This is a wrapper for the 'docker images' command.

Signed-off-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-9-git-send-email-famz@redhat.com
2016-07-20 19:19:43 +08:00
Alex Bennée
6e733da676 tests/docker/docker.py: add update operation
This adds a new operation to the docker script to allow updating of
binaries in an existing container. This is because it would be
inefficient to re-build the whole container just for an update to the
QEMU binary.

To update the executable run:

    ./tests/docker/docker.py update \
        debian:armhf ./arm-linux-user/qemu-arm

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-6-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Alex Bennée
920776ea5e tests/docker/docker.py: check and run .pre script
The docker script will now search for an associated $dockerfile.pre
script which gets run in the same build context as the dockerfile will
be. This is to support pre-seeding the build context before running the
docker build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-4-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Alex Bennée
504ca3c208 tests/docker/docker.py: support --include-executable
When passed the path to a binary we copy it and any linked libraries (if
it is dynamically linked) into the docker build context. These can then
be included by a dockerfile with the line:

  # Copy all of context into container
  ADD . /

This is mainly intended for setting up foreign architecture docker
images which use qemu-$arch to do cross-architecture linux-user
execution. It also relies on the host and guest file-system following
reasonable multi-arch layouts so the copied libraries don't clash with
the guest ones.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 1468934445-32183-3-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Alex Bennée
a9f8d03891 tests/docker/docker.py: docker_dir outside build
Instead of letting the build_image create the temporary working dir we
move the creation to the build command. This is preparation for the
later patches where additional files can be added to the build context
before the build step is run.

We also ensure we remove the build context after we are done (mkdtemp
doesn't do this automatically for you).

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Fam Zheng <famz@redhat.com>
Message-id: 1468934445-32183-2-git-send-email-famz@redhat.com
Signed-off-by: Fam Zheng <famz@redhat.com>
2016-07-20 19:19:43 +08:00
Fam Zheng
4485b04be9 tests: Add utilities for docker testing
docker.py is added with a number of useful subcommands to manager docker
images and instances for QEMU docker testing. Subcommands are:

run: A wrapper of "docker run" (or "sudo -n docker run" if necessary),
which takes care of killing and removing the running container at
SIGINT.

clean: Tear down all the containers including inactive ones that are
started by docker_run.

build: Compare an image from given dockerfile and rebuild it if they're
different.

Signed-off-by: Fam Zheng <famz@redhat.com>
Message-id: 1464755128-32490-2-git-send-email-famz@redhat.com
2016-06-01 17:25:50 +08:00