XorgTesting.mdwn


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

# Using the X Test Suite

The X Test Suite is the test suite for core rendering in the X Server.  For other X testing tools, see [[http://www.freedesktop.org/wiki/Software/XTesting/]]

## Build xts5

This is the tree with all the actual test code.  It has a framework for running tests and comparing results, but I don't recommend using them.  You can learn more about some of those tools in the README if you want.

    # git clone git://anongit.freedesktop.org/git/xorg/test/xts.git
    # cd xts
    # ./autogen.sh
    # make
    # cd ..

## Build xtsttopng

This tool takes the output logs of tests that generate image comparisons, remaps the pixel values to visibly different colors, and generates pngs for the two (-0 being the rendered image and -1 being the reference image).

Note that these images don't necessarily represent what was *actually* rendered, since the reference image is generated automatically, and for tests that don't care about some area of the drawable, it'll smash the areas it doesn't care about to a standard value in both the rendered and reference image.

    # git clone git://anongit.freedesktop.org/git/xorg/app/xtsttopng.git
    # cd xtsttopng
    # ./autogen.sh
    # make
    # sudo make install

## Get piglit-xts

This is a fork of piglit (the GL test suite project)

    # git clone git://anongit.freedesktop.org/git/piglit.git
    # cd piglit
    # git remote add anholt git://people.freedesktop.org/\~anholt/piglit
    # git fetch anholt
    # git checkout -b xts anholt/xts
    # mkdir bin
    # ln -s ~/my/path/to/xts bin/xtest

Now, here's an example of using this to do some tests comparing two rendering modes in Xephyr.  Xlib9 is the set of core rendering tests (not counting the window movement code, which also does rendering and is covered in another chapter).

    # Xephyr :5 -noreset -ac -screen 1024x768 &
    # DISPLAY=:5 ./piglit-run.py -t Xlib9 tests/xts.py results/xts
    # killall Xephyr
    # Xephyr :5 -noreset -ac -screen 1024x768 -glamor &
    # DISPLAY=:5 ./piglit-run.py -t Xlib9 tests/xts.py results/xts-glamor
    # ./piglit-summary-html.py summary/mysum results/xts results/xts-glamor
    # firefox summary/mysum/index.html

Take a look at changes.html for a common way of looking at piglit results as a developer (I don't care if things are still broken, I want to know if I broke or fixed new things).  Also, note that in current piglit-xts, the images present in the failure reports on tests get overwritten in each test run (because they're saved in the top level instead of in the results/ directory you specify.  It's just a bug).