replayer: add replayer executable

v2:
  - Update the README.md with further documentation.

v3:
  - Exit with "1" when the inner result is negative, marking a CRASH,
    and with "2" when the inner result is positive, marking a FAIL.

v4:
  - Use newly added compare result enumeration.

Signed-off-by: Andres Gomez <agomez@igalia.com>
Reviewed-by: Dylan Baker <dylan@pnwbakers.com>
Part-of: <https://gitlab.freedesktop.org/mesa/piglit/-/merge_requests/353>

3 files changed, 392 insertions, 0 deletions

diff --git a/replayer/CMakeLists.txt b/replayer/CMakeLists.txt
new file mode 100644
index 000000000..779549860
--- /dev/null
+++ b/replayer/CMakeLists.txt
@@ -0,0 +1,50 @@
+# coding=utf-8
+#
+# Copyright 2012 Intel Corporation
+# Copyright © 2020 Valve Corporation.
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation
+# files (the "Software"), to deal in the Software without
+# restriction, including without limitation the rights to use,
+# copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following
+# conditions:
+#
+# This permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+# PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE AUTHOR(S) BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
+# OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+#
+# SPDX-License-Identifier: MIT
+
+set(piglit_replayer_script ${piglit_SOURCE_DIR}/replayer/replayer.py)
+set(piglit_replayer_output_dir ${piglit_BINARY_DIR}/bin)
+
+set(piglit_replayer_outputs
+	${piglit_replayer_output_dir}/replayer.py
+	)
+
+set(piglit_replayer_depends
+	${CMAKE_SOURCE_DIR}/replayer/replayer.py
+	)
+
+add_custom_command(
+	OUTPUT ${piglit_replayer_outputs}
+	DEPENDS ${piglit_replayer_depends}
+	COMMAND ${CMAKE_COMMAND} -E copy ${piglit_replayer_script} ${piglit_replayer_output_dir}
+	)
+
+add_custom_target(piglit_replayer ALL
+	DEPENDS ${piglit_replayer_outputs}
+	)
+
+install(PROGRAMS ${piglit_replayer_outputs} DESTINATION ${PIGLIT_INSTALL_LIBDIR}/bin)
diff --git a/replayer/README.md b/replayer/README.md
new file mode 100644
index 000000000..d03a21b7c
--- /dev/null
+++ b/replayer/README.md
@@ -0,0 +1,196 @@
+Replayer
+========
+
+### Traces definition file
+
+The trace definition file contains information about the traces to run along
+with their expected image checksums on each device, and optionally from where to
+download them. An example:
+
+```yaml
+traces-db:
+  download-url: https://minio-packet.freedesktop.org/mesa-tracie-public/
+
+traces:
+  - path: glmark2/jellyfish.rdc
+    expectations:
+      - device: gl-intel-0x3185
+        checksum: 58359ea4caf6ad44c6b65526881bbd17
+      - device: gl-vmware-llvmpipe
+        checksum: d82267c25a0decdad7b563c56bb81106
+  - path: supertuxkart/supertuxkart-antediluvian-abyss.rdc
+    expectations:
+      - device: gl-intel-0x3185
+        checksum: ff827f7eb069afd87cc305a422cba939
+```
+
+The `traces-db` entry can be absent, in which case it is assumed that
+the traces can be found in the `CWD/replayer-db` directory.
+
+Traces that don't have an expectation for the current device are skipped
+during trace replay.
+
+Adding a new trace to the list involves uploading the trace to the
+remote `traces-db` and adding an entry to the `traces` list. The
+reference checksums can be calculated with the `checksum` command.
+Alternatively, an arbitrary checksum can be used, and during replay
+(see below) the scripts will report the mismatch and expected
+checksum.
+
+### Trace-db download urls
+
+The trace-db:download-url property contains an HTTPS url from which traces can
+be downloaded, by appending traces:path properties to it.
+
+### Trace files
+
+replayer supports renderdoc (.rdc), apitrace (.trace, .trace-dxgi) and
+gfxreconstruct (.gfxr) files. Trace files need to have the correct
+extension so that replayer can detect them properly.
+
+The trace files that are contained in public `traces-db` stores must
+be legally redistributable. This is typically true for FOSS games and
+applications. Traces for proprietary games and application are
+typically not redistributable, unless specific redistribution rights
+have been granted by the publisher.
+
+In order to have reliable comparisons, trace files in a given store
+are expected to be immutable. Any change to a trace file means that it
+needs to be renamed and updated in the traces definition file (eg. by
+appending a _v2 suffix to the file).
+
+### Replaying traces
+
+If running from the source tree, replayer needs the
+`PIGLIT_SOURCE_DIR` env variable:
+
+   ```sh
+   $ PIGLIT_SOURCE_DIR="../" ./replayer.py ...
+   ```
+
+replayer features a series of commands to deal with traces:
+ * `checksum`: will calculate the checksum for a given image file.
+ * `compare`: will download a trace or all the traces in a traces
+   definition file for a given device, replay them and compare their
+   checksum against the expected ones.
+ * `download`: will download a file, given a relative path, from a
+   remote url.
+ * `dump`: will dump as images the last call or specified calls from a
+   trace file, given a specific device.
+ * `query`: will return the queried information from a given traces
+   definition file.
+
+You can get a more complete help running:
+
+   ```sh
+   $ replayer.py <command> --help
+   ```
+
+Examples:
+
+   ```sh
+   $ replayer.py checksum ./vkcube.gfxr-9.png
+   ```
+
+   ```sh
+   $ replayer.py compare trace \
+                 --output ./results \
+		 --device-name gl-vmware-llvmpipe \
+		 --download-url https://minio-packet.freedesktop.org/mesa-tracie-public/ \
+		 glmark2/desktop-blur-radius=5:effect=blur:passes=1:separable=true:windows=4.rdc \
+		 8867f3a41f180626d0d4b7661ff5c0f4
+   ```
+
+   ```sh
+   $ replayer.py compare yaml \
+		 --yaml-file ./traces.yml \
+		 --device-name gl-vmware-llvmpipe \
+		 --keep-image
+   ```
+
+   ```sh
+   $ replayer.py download \
+		 --download-url https://minio-packet.freedesktop.org/mesa-tracie-public/ \
+		 --db-path ./traces-db \
+		 --force-download \
+		 glmark2/desktop-blur-radius=5:effect=blur:passes=1:separable=true:windows=4.rdc
+   ```
+
+   ```sh
+   $ replayer.py dump \
+                 --config ./piglit.conf \
+                 --output ./results \
+		 --device-name gl-vmware-llvmpipe \
+		 --calls "3,8,9" \
+		 glmark2/desktop-blur-radius=5:effect=blur:passes=1:separable=true:windows=4.rdc
+   ```
+
+   ```sh
+   $ replayer.py query \
+		 --yaml-file ./traces.yml \
+                 checksum \
+                 --device-name gl-vmware-llvmpipe \
+		 glmark2/desktop-blur-radius=5:effect=blur:passes=1:separable=true:windows=4.rdc
+   ```
+
+   ```sh
+   $ replayer.py query \
+		 --yaml-file ./traces.yml \
+                 traces \
+                 --device-name gl-vmware-llvmpipe \
+                 --trace-extensions ".trace,.rdc"
+                 --checksum
+   ```
+
+   ```sh
+   $ replayer.py query \
+		 --yaml-file ./traces.yml \
+                 traces_db_download_url
+   ```
+
+Unless specified when comparing or dumping, replayer places the
+produced artifacts at the `CWD/results` directory. By default, created
+images from traces are only stored in case of a checksum
+mismatch. This can be overriden with the `--keep-image` parameter to
+force storing images, e.g., to get a complete set of reference images.
+
+By default when dumping, only the image corresponding to the last frame
+of the trace is created.  This can be changed with the `--calls`
+parameter.
+
+Unless specified with the `--output` parameter, the dumped images are
+stored in the subdirectory `./test/<device-name>/<trace_file_path/` ,
+with names of the form `trace_file_name-call_num.png`.  The full log
+of any commands used while dumping the images is also saved in a file
+in the 'test/<device-name>' subdirectory, named after the trace name
+with '.log' appended.
+
+### Specific dependencies for dumping depending of the trace type
+
+Depending on the target 3D API, replayer requires a recent version
+of apitrace (and eglretrace) being in the path, and also the renderdoc
+python module being available, for GL traces.
+
+To ensure python3 can find the renderdoc python module you need to set
+`PYTHONPATH` to point to the location of `renderdoc.so` (binary python modules)
+and `LD_LIBRARY_PATH` to point to the location of `librenderdoc.so`. In the
+renderdoc build tree, both of these are in `renderdoc/<builddir>/lib`. Note
+that renderdoc doesn't install the `renderdoc.so` python module.
+
+In the case of Vulkan traces, replayer needs a recent version of
+gfxrecon-info and gfxrecon-replay being in the path, and also the
+`VK_LAYER_LUNARG_screenshot` Vulkan layer from LunarG's VulkanTools.
+
+To ensure that this layer can be found when running the trace you need
+to set `VK_LAYER_PATH` to point to the location of
+`VkLayer_screenshot.json` and `LD_LIBRARY_PATH` to point to the
+location of `libVkLayer_screenshot.so`.
+
+In the case of DXGI traces, replayer requires Wine, a recent version
+of DXVK installed in the default `WINEPREFIX`, and a recent binary
+version of apitrace (and d3dretrace) for Windows which should be
+reachable through Windows' `PATH` environment variable.
+
+Alternatively, all of the paths for specific binaries can be set via
+piglit's configuration file or env variables. Check the documenation
+at [piglit.conf.example](piglit.conf.example) for further details.
diff --git a/replayer/replayer.py b/replayer/replayer.py
new file mode 100755
index 000000000..914bab68c
--- /dev/null
+++ b/replayer/replayer.py
@@ -0,0 +1,146 @@
+#!/usr/bin/env python3
+# coding=utf-8
+#
+# Copyright (c) 2014, 2019 Intel Corporation
+# Copyright © 2020 Valve Corporation.
+#
+# Permission is hereby granted, free of charge, to any person
+# obtaining a copy of this software and associated documentation
+# files (the "Software"), to deal in the Software without
+# restriction, including without limitation the rights to use,
+# copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following
+# conditions:
+#
+# This permission notice shall be included in all copies or
+# substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+# KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+# WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+# PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE AUTHOR(S) BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
+# OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+#
+# SPDX-License-Identifier: MIT
+
+
+""" Wrapper for replayer executable
+
+This imports functions from the replay framework and calls them with the
+argument parts that the parser defined here doesn't know how to parse.
+
+It is very important that the final parser not generate a help message
+(add_help=False in the constructor arguments), otherwise this parser will
+capture -h/--help and the results will not be useful.
+
+"""
+
+import argparse
+import os
+import os.path as path
+import sys
+
+
+def setup_module_search_path():
+    """Add Piglit's data directory to Python's module search path.
+
+    This enables Python to import Piglit's framework module.
+
+    CAUTION: This script must import the framework that *belongs to this
+    script*. Mayhem occurs if this script accidentally imports a framework
+    module that belongs to a different Piglit source tree or belongs to
+    a different Piglit installation.
+
+    CAUTION: This script file must be located in the Piglit source tree or in
+    an installed location.  Otherwise this function may fail to find the
+    framework module or, worse, it may succeed in finding a different Piglit's
+    framework module.
+    """
+
+    piglit_data_dir = os.environ.get('PIGLIT_SOURCE_DIR', None)
+    if piglit_data_dir is None:
+        print('error: the PIGLIT_SOURCE_DIR env variable is not set. '
+              'exiting ...', file=sys.stderr)
+        sys.exit(1)
+
+    if path.exists(path.join(piglit_data_dir, 'framework')):
+        sys.path.append(piglit_data_dir)
+        return
+
+    print('error: failed to find piglit data directory. '
+          'exiting...',file=sys.stderr)
+    sys.exit(1)
+
+
+setup_module_search_path()
+import framework.replay.programs.compare as compare
+import framework.replay.programs.query as query
+import framework.replay.programs.download as download
+import framework.replay.programs.checksum as checksum
+import framework.replay.programs.dump as dump
+from framework.replay.compare_replay import Result
+
+
+def main():
+    """ Parse argument and call other executables """
+    input_ = sys.argv[1:]
+
+    parser = argparse.ArgumentParser()
+
+    # Add a destination due to
+    # https://github.com/python/cpython/pull/3027#issuecomment-330910633
+    subparsers = parser.add_subparsers(dest='command', required=True)
+
+    parser_checksum = subparsers.add_parser(
+        'checksum',
+        add_help=False,
+        help=('Calculates the checksums of a local image file.'))
+    parser_checksum.set_defaults(func=checksum.checksum)
+
+    parser_compare = subparsers.add_parser(
+        'compare',
+        add_help=False,
+        help=('Trace replay compare methods.'))
+    parser_compare.set_defaults(func=compare.compare)
+
+    parser_download = subparsers.add_parser(
+        'download',
+        add_help=False,
+        help=('Downloads a remote file into the db path.'))
+    parser_download.set_defaults(func=download.download)
+
+    parser_dump = subparsers.add_parser(
+        'dump',
+        add_help=False,
+        help=('Dumps image(s) from a trace file.'))
+    parser_dump.set_defaults(func=dump.dump)
+
+    parser_query = subparsers.add_parser(
+        'query',
+        add_help=False,
+        help=('Queries for specific information '
+              'from a traces description file listing traces '
+              'and their checksums for each device.'))
+    parser_query.set_defaults(func=query.query)
+
+    # Parse the known arguments (replayer compare or replayer query for
+    # example), and then pass the arguments that this parser doesn't know about
+    # to that executable
+    parsed, args = parser.parse_known_args(input_)
+    try:
+        runner = parsed.func
+    except AttributeError:
+        parser.print_help()
+        sys.exit(1)
+
+    result = runner(args)
+    if result is not Result.MATCH:
+        sys.exit(1 if result is Result.FAILURE else 2)
+
+
+if __name__ == '__main__':
+    main()