tests: thread-based client harness
This replaces the old test harness with a new one.
The old harness relied on fork()'ing each test which makes tests independent,
but makes debugging them harder. The new harness runs client code in a thread
instead of a new process. A side-effect of not fork()'ing anymore is that any
failure will stop running a test series short. Fortunately we do not have any
tests that are expected to crash or fail.
The old harness executed 'weston' from Meson, with lots of setup as both
command line options and environment variables. The new harness executes
wet_main() instead: the test program itself calls the compositor main function
to execute the compositor in-process. Command line arguments are configured in
the test program itself, not in meson.build. Environment variables aside, you
are able to run a test by simply executing the test program, even if it is a
plugin test.
The new harness adds a new type of iteration: fixtures. For now, fixtures are
used to set up the compositor for tests that need a compositor. If necessary, a
fixture setup may include a data array of arbitrary type for executing the test
series for each element in the array. This will be most useful for running
screenshooting tests with both Pixman- and GL-renderers.
The new harness outputs TAP formatted results into stdout. Meson is not
switched to consume TAP yet though, because it would require a Meson version
requirement bump and would not have any benefits at this time. OTOH outputting
TAP is trivial and sets up a clear precedent of random test chatter belonging
to stderr.
This commit migrates only few tests to actually make use of the new features:
roles is a basic client test, subsurface-shot is a client test that
demonstrates the fixture array, and plugin-registry is a plugin test. The rest
of the tests will be migrated later.
Once all tests are migrated, we can remove the test-specific setup from
meson.build, leaving only the actual build instructions in there.
The not migrated tests and stand-alone tests suffer only a minor change: they
no longer fork() for each TEST(), otherwise they keep running as before.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
2019-11-01 15:02:15 +03:00
|
|
|
/*
|
|
|
|
* Copyright 2019 Collabora, Ltd.
|
|
|
|
*
|
|
|
|
* Permission is hereby granted, free of charge, to any person obtaining
|
|
|
|
* a copy of this software and associated documentation files (the
|
|
|
|
* "Software"), to deal in the Software without restriction, including
|
|
|
|
* without limitation the rights to use, copy, modify, merge, publish,
|
|
|
|
* distribute, sublicense, and/or sell copies of the Software, and to
|
|
|
|
* permit persons to whom the Software is furnished to do so, subject to
|
|
|
|
* the following conditions:
|
|
|
|
*
|
|
|
|
* The above copyright notice and this permission notice (including the
|
|
|
|
* next paragraph) shall be included in all copies or substantial
|
|
|
|
* portions of the Software.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
|
|
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
|
|
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
|
|
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
|
|
|
|
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
|
|
|
|
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
|
|
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
|
|
* SOFTWARE.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef WESTON_TESTSUITE_DATA_H
|
|
|
|
#define WESTON_TESTSUITE_DATA_H
|
|
|
|
|
2023-06-28 13:45:07 +03:00
|
|
|
#include <assert.h>
|
|
|
|
#include <errno.h>
|
|
|
|
#include <semaphore.h>
|
|
|
|
|
tests: thread-based client harness
This replaces the old test harness with a new one.
The old harness relied on fork()'ing each test which makes tests independent,
but makes debugging them harder. The new harness runs client code in a thread
instead of a new process. A side-effect of not fork()'ing anymore is that any
failure will stop running a test series short. Fortunately we do not have any
tests that are expected to crash or fail.
The old harness executed 'weston' from Meson, with lots of setup as both
command line options and environment variables. The new harness executes
wet_main() instead: the test program itself calls the compositor main function
to execute the compositor in-process. Command line arguments are configured in
the test program itself, not in meson.build. Environment variables aside, you
are able to run a test by simply executing the test program, even if it is a
plugin test.
The new harness adds a new type of iteration: fixtures. For now, fixtures are
used to set up the compositor for tests that need a compositor. If necessary, a
fixture setup may include a data array of arbitrary type for executing the test
series for each element in the array. This will be most useful for running
screenshooting tests with both Pixman- and GL-renderers.
The new harness outputs TAP formatted results into stdout. Meson is not
switched to consume TAP yet though, because it would require a Meson version
requirement bump and would not have any benefits at this time. OTOH outputting
TAP is trivial and sets up a clear precedent of random test chatter belonging
to stderr.
This commit migrates only few tests to actually make use of the new features:
roles is a basic client test, subsurface-shot is a client test that
demonstrates the fixture array, and plugin-registry is a plugin test. The rest
of the tests will be migrated later.
Once all tests are migrated, we can remove the test-specific setup from
meson.build, leaving only the actual build instructions in there.
The not migrated tests and stand-alone tests suffer only a minor change: they
no longer fork() for each TEST(), otherwise they keep running as before.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
2019-11-01 15:02:15 +03:00
|
|
|
/** Standard return codes
|
|
|
|
*
|
|
|
|
* Both Autotools and Meson use these codes as test program exit codes
|
|
|
|
* to denote the test result for the whole process.
|
|
|
|
*
|
|
|
|
* \ingroup testharness
|
|
|
|
*/
|
|
|
|
enum test_result_code {
|
|
|
|
RESULT_OK = 0,
|
|
|
|
RESULT_SKIP = 77,
|
|
|
|
RESULT_FAIL = 1,
|
|
|
|
RESULT_HARD_ERROR = 99,
|
|
|
|
};
|
|
|
|
|
|
|
|
struct weston_test;
|
|
|
|
struct weston_compositor;
|
|
|
|
|
|
|
|
/** Weston test types
|
|
|
|
*
|
|
|
|
* \sa weston_test_harness_execute_standalone
|
|
|
|
* weston_test_harness_execute_as_plugin
|
|
|
|
* weston_test_harness_execute_as_client
|
|
|
|
*
|
|
|
|
* \ingroup testharness_private
|
|
|
|
*/
|
|
|
|
enum test_type {
|
|
|
|
TEST_TYPE_STANDALONE,
|
|
|
|
TEST_TYPE_PLUGIN,
|
|
|
|
TEST_TYPE_CLIENT,
|
|
|
|
};
|
|
|
|
|
2023-06-28 13:45:07 +03:00
|
|
|
/** Safely handle posting a semaphore to wake a waiter
|
|
|
|
*
|
|
|
|
* \ingroup testharness_private
|
|
|
|
*/
|
|
|
|
static inline void wet_test_post_sem(sem_t *sem)
|
|
|
|
{
|
|
|
|
int ret = sem_post(sem);
|
|
|
|
assert(ret == 0); /* only fails on programming errors */
|
|
|
|
}
|
|
|
|
|
|
|
|
/** Safely handle waiting on a semaphore
|
|
|
|
*
|
|
|
|
* \ingroup testharness_private
|
|
|
|
*/
|
|
|
|
static inline void wet_test_wait_sem(sem_t *sem)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
do {
|
|
|
|
ret = sem_wait(sem);
|
|
|
|
} while (ret != 0 && errno == EINTR);
|
|
|
|
assert(ret == 0); /* all other failures are programming errors */
|
|
|
|
}
|
|
|
|
|
|
|
|
/** An individual breakpoint set for the server
|
|
|
|
*
|
|
|
|
* This breakpoint data is created and placed in a list by either the server
|
|
|
|
* (when handling protocol messages) or the client (when directly manipulating
|
|
|
|
* the list during a breakpoint).
|
|
|
|
*
|
|
|
|
* It must be freed by the client.
|
|
|
|
*
|
|
|
|
* \ingroup testharness_private
|
|
|
|
*/
|
|
|
|
struct wet_test_pending_breakpoint {
|
|
|
|
/** breakpoint type - enum weston_test_breakpoint from protocol */
|
|
|
|
uint32_t breakpoint;
|
|
|
|
/** type-specific resource to filter on (optional) */
|
|
|
|
void *resource;
|
|
|
|
struct wl_list link; /**< wet_testsuite_breakpoints.list */
|
|
|
|
};
|
|
|
|
|
|
|
|
/** Information about the server's active breakpoint
|
|
|
|
*
|
|
|
|
* This breakpoint data is created by the server and passed to the client when
|
|
|
|
* the server enters a breakpoint.
|
|
|
|
*
|
|
|
|
* It must be freed by the client.
|
|
|
|
*
|
|
|
|
* \ingroup testharness_private
|
|
|
|
*/
|
|
|
|
struct wet_test_active_breakpoint {
|
|
|
|
/** libweston compositor instance in use */
|
|
|
|
struct weston_compositor *compositor;
|
|
|
|
/** type-specific pointer to resource which triggered this breakpoint */
|
|
|
|
void *resource;
|
|
|
|
/** on release, reinsert the template to trigger next time */
|
|
|
|
bool rearm_on_release;
|
|
|
|
/** client's original breakpoint request */
|
|
|
|
struct wet_test_pending_breakpoint *template_;
|
|
|
|
};
|
|
|
|
|
|
|
|
/** Client/compositor synchronisation for breakpoint state
|
|
|
|
*
|
|
|
|
* Manages the set of active breakpoints placed for the server, as well as
|
|
|
|
* signalling the pausing/continuing of server actions.
|
|
|
|
*
|
|
|
|
* \ingroup testharness_private
|
|
|
|
*/
|
|
|
|
struct wet_testsuite_breakpoints {
|
|
|
|
/** signalled by the server when it reaches a breakpoint */
|
|
|
|
sem_t client_break;
|
|
|
|
/** signalled by the client to resume server execution */
|
|
|
|
sem_t server_release;
|
|
|
|
|
|
|
|
/** Pushed by the server when a breakpoint is triggered, immediately
|
|
|
|
* before it signals the client_break semaphore. Client consumes this
|
|
|
|
* and takes ownership after the wait succeeds. */
|
|
|
|
struct wet_test_active_breakpoint *active_bp;
|
|
|
|
|
|
|
|
/** client-internal state; set by consuming active_bp, cleared by
|
|
|
|
* signalling server_release */
|
|
|
|
bool in_client_break;
|
|
|
|
|
|
|
|
/** list of pending breakpoints: owned by the server during normal
|
|
|
|
* execution (ordinarily added to by a protocol request, and
|
|
|
|
* traversed to find a possible breakpoint to trigger), and owned by
|
|
|
|
* the client wtihin a breakpoint (pending breakpoints may be added
|
|
|
|
* or removed). Members are wet_test_pending_breakpoint.link */
|
|
|
|
struct wl_list list;
|
|
|
|
};
|
|
|
|
|
tests: thread-based client harness
This replaces the old test harness with a new one.
The old harness relied on fork()'ing each test which makes tests independent,
but makes debugging them harder. The new harness runs client code in a thread
instead of a new process. A side-effect of not fork()'ing anymore is that any
failure will stop running a test series short. Fortunately we do not have any
tests that are expected to crash or fail.
The old harness executed 'weston' from Meson, with lots of setup as both
command line options and environment variables. The new harness executes
wet_main() instead: the test program itself calls the compositor main function
to execute the compositor in-process. Command line arguments are configured in
the test program itself, not in meson.build. Environment variables aside, you
are able to run a test by simply executing the test program, even if it is a
plugin test.
The new harness adds a new type of iteration: fixtures. For now, fixtures are
used to set up the compositor for tests that need a compositor. If necessary, a
fixture setup may include a data array of arbitrary type for executing the test
series for each element in the array. This will be most useful for running
screenshooting tests with both Pixman- and GL-renderers.
The new harness outputs TAP formatted results into stdout. Meson is not
switched to consume TAP yet though, because it would require a Meson version
requirement bump and would not have any benefits at this time. OTOH outputting
TAP is trivial and sets up a clear precedent of random test chatter belonging
to stderr.
This commit migrates only few tests to actually make use of the new features:
roles is a basic client test, subsurface-shot is a client test that
demonstrates the fixture array, and plugin-registry is a plugin test. The rest
of the tests will be migrated later.
Once all tests are migrated, we can remove the test-specific setup from
meson.build, leaving only the actual build instructions in there.
The not migrated tests and stand-alone tests suffer only a minor change: they
no longer fork() for each TEST(), otherwise they keep running as before.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
2019-11-01 15:02:15 +03:00
|
|
|
/** Test harness specific data for running tests
|
|
|
|
*
|
|
|
|
* \ingroup testharness_private
|
|
|
|
*/
|
|
|
|
struct wet_testsuite_data {
|
|
|
|
void (*run)(struct wet_testsuite_data *);
|
|
|
|
|
2023-06-28 13:45:07 +03:00
|
|
|
void *wl_client;
|
|
|
|
|
tests: thread-based client harness
This replaces the old test harness with a new one.
The old harness relied on fork()'ing each test which makes tests independent,
but makes debugging them harder. The new harness runs client code in a thread
instead of a new process. A side-effect of not fork()'ing anymore is that any
failure will stop running a test series short. Fortunately we do not have any
tests that are expected to crash or fail.
The old harness executed 'weston' from Meson, with lots of setup as both
command line options and environment variables. The new harness executes
wet_main() instead: the test program itself calls the compositor main function
to execute the compositor in-process. Command line arguments are configured in
the test program itself, not in meson.build. Environment variables aside, you
are able to run a test by simply executing the test program, even if it is a
plugin test.
The new harness adds a new type of iteration: fixtures. For now, fixtures are
used to set up the compositor for tests that need a compositor. If necessary, a
fixture setup may include a data array of arbitrary type for executing the test
series for each element in the array. This will be most useful for running
screenshooting tests with both Pixman- and GL-renderers.
The new harness outputs TAP formatted results into stdout. Meson is not
switched to consume TAP yet though, because it would require a Meson version
requirement bump and would not have any benefits at this time. OTOH outputting
TAP is trivial and sets up a clear precedent of random test chatter belonging
to stderr.
This commit migrates only few tests to actually make use of the new features:
roles is a basic client test, subsurface-shot is a client test that
demonstrates the fixture array, and plugin-registry is a plugin test. The rest
of the tests will be migrated later.
Once all tests are migrated, we can remove the test-specific setup from
meson.build, leaving only the actual build instructions in there.
The not migrated tests and stand-alone tests suffer only a minor change: they
no longer fork() for each TEST(), otherwise they keep running as before.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
2019-11-01 15:02:15 +03:00
|
|
|
/* test definitions */
|
|
|
|
const struct weston_test_entry *tests;
|
|
|
|
unsigned tests_count;
|
|
|
|
int case_index;
|
|
|
|
enum test_type type;
|
|
|
|
struct weston_compositor *compositor;
|
|
|
|
|
|
|
|
/* client thread control */
|
|
|
|
int thread_event_pipe;
|
2023-06-28 13:45:07 +03:00
|
|
|
struct wet_testsuite_breakpoints breakpoints;
|
tests: thread-based client harness
This replaces the old test harness with a new one.
The old harness relied on fork()'ing each test which makes tests independent,
but makes debugging them harder. The new harness runs client code in a thread
instead of a new process. A side-effect of not fork()'ing anymore is that any
failure will stop running a test series short. Fortunately we do not have any
tests that are expected to crash or fail.
The old harness executed 'weston' from Meson, with lots of setup as both
command line options and environment variables. The new harness executes
wet_main() instead: the test program itself calls the compositor main function
to execute the compositor in-process. Command line arguments are configured in
the test program itself, not in meson.build. Environment variables aside, you
are able to run a test by simply executing the test program, even if it is a
plugin test.
The new harness adds a new type of iteration: fixtures. For now, fixtures are
used to set up the compositor for tests that need a compositor. If necessary, a
fixture setup may include a data array of arbitrary type for executing the test
series for each element in the array. This will be most useful for running
screenshooting tests with both Pixman- and GL-renderers.
The new harness outputs TAP formatted results into stdout. Meson is not
switched to consume TAP yet though, because it would require a Meson version
requirement bump and would not have any benefits at this time. OTOH outputting
TAP is trivial and sets up a clear precedent of random test chatter belonging
to stderr.
This commit migrates only few tests to actually make use of the new features:
roles is a basic client test, subsurface-shot is a client test that
demonstrates the fixture array, and plugin-registry is a plugin test. The rest
of the tests will be migrated later.
Once all tests are migrated, we can remove the test-specific setup from
meson.build, leaving only the actual build instructions in there.
The not migrated tests and stand-alone tests suffer only a minor change: they
no longer fork() for each TEST(), otherwise they keep running as before.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
2019-11-01 15:02:15 +03:00
|
|
|
|
|
|
|
/* informational run state */
|
|
|
|
int fixture_iteration;
|
2021-02-15 15:43:14 +03:00
|
|
|
const char *fixture_name;
|
tests: thread-based client harness
This replaces the old test harness with a new one.
The old harness relied on fork()'ing each test which makes tests independent,
but makes debugging them harder. The new harness runs client code in a thread
instead of a new process. A side-effect of not fork()'ing anymore is that any
failure will stop running a test series short. Fortunately we do not have any
tests that are expected to crash or fail.
The old harness executed 'weston' from Meson, with lots of setup as both
command line options and environment variables. The new harness executes
wet_main() instead: the test program itself calls the compositor main function
to execute the compositor in-process. Command line arguments are configured in
the test program itself, not in meson.build. Environment variables aside, you
are able to run a test by simply executing the test program, even if it is a
plugin test.
The new harness adds a new type of iteration: fixtures. For now, fixtures are
used to set up the compositor for tests that need a compositor. If necessary, a
fixture setup may include a data array of arbitrary type for executing the test
series for each element in the array. This will be most useful for running
screenshooting tests with both Pixman- and GL-renderers.
The new harness outputs TAP formatted results into stdout. Meson is not
switched to consume TAP yet though, because it would require a Meson version
requirement bump and would not have any benefits at this time. OTOH outputting
TAP is trivial and sets up a clear precedent of random test chatter belonging
to stderr.
This commit migrates only few tests to actually make use of the new features:
roles is a basic client test, subsurface-shot is a client test that
demonstrates the fixture array, and plugin-registry is a plugin test. The rest
of the tests will be migrated later.
Once all tests are migrated, we can remove the test-specific setup from
meson.build, leaving only the actual build instructions in there.
The not migrated tests and stand-alone tests suffer only a minor change: they
no longer fork() for each TEST(), otherwise they keep running as before.
Signed-off-by: Pekka Paalanen <pekka.paalanen@collabora.com>
2019-11-01 15:02:15 +03:00
|
|
|
|
|
|
|
/* test counts */
|
|
|
|
unsigned counter;
|
|
|
|
unsigned passed;
|
|
|
|
unsigned skipped;
|
|
|
|
unsigned failed;
|
|
|
|
unsigned total;
|
|
|
|
};
|
|
|
|
|
|
|
|
#endif /* WESTON_TESTSUITE_DATA_H */
|