1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
|
# The X11R6.9/X11R7 Release Testing Plan
[[!toc ]]
## Detailed test instructions
This section outlines the test procedure to follow when testing a release candidate. We need for everyone to test both the X11``R6.9 (monolithic source tree) and the X11``R7.0 (modular source tree). When you have completed a test run, please submit your test report to our [[results database|http://results.x.org/index.php]] so that we can track the progress of the release. We need these reports to determine if the code is ready for release or needs more work.
There are four main areas that we are interested in testing:
* Build tests: Does the release build properly?
* Install tests: Does the release install properly?
* Conformance tests: Does the release conform to the specifications (i.e., does it pass the X Test Suite?)
* Run tests: Does the release run the standard desktop and applications properly?
Each of these are described below in detail.
Note that for any test(s) that fail, please file a bugzilla report in [[bugzilla|https://bugs.freedesktop.org/enter_bug.cgi?product=xorg]] and mark it as a blocker of the [[release bug|https://bugs.freedesktop.org/bugzilla/show_bug.cgi?id=1690]].
### Build tests
#### Monolithic build tests
For the monolithic source tree, each of the following build tests can be performed by copying the _sample_ host.def file (or the _alternate_) to the xc/config/cf directory and the running `make World >& World.LOG` (or other such command as appropriate for your platform), and then checking the World.LOG file for any failures.
1. Build with empty host.def file ([[sample|http://www.freedesktop.org/~kem/build-tests/1/host.def]] [[alternate|http://www.freedesktop.org/~kem/build-tests/1a/host.def]])
1. Build with Build``Servers``Only defined as YES ([[sample|http://www.freedesktop.org/~kem/build-tests/2/host.def]] [[alternate|http://www.freedesktop.org/~kem/build-tests/2a/host.def]])
1. Build with Do``Loadable``Server defined as NO ([[sample|http://www.freedesktop.org/~kem/build-tests/3/host.def]] [[alternate|http://www.freedesktop.org/~kem/build-tests/3a/host.def]])
Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each build requirement above, defining `HasFreetype2` as NO is permitted. Each _alternate_ host.def file above have this define included.
#### Modular build tests
For the modular source tree, we ask that you build from CVS using the methods described in the [[Modular developer's guide|Building_the_X_Window_System]]. The `build.sh` script in CVS will allow you to build and install all of the module components in the appropriate order. By default, the build.sh script will exit if it fails to build or install any component.
If you would like to have the `build.sh` script warn you about build or install failures, then you can use the `-n` command-line flag to tell it to print error messages instead of exiting. For example, you could run `build.sh -n /tmp/modular >& build.log` and then once the script completes, you could `grep '*****' build.log` to see if there are any errors and which packages failed.
One additional test that further helps find problems is to have the `build.sh` script also run `make distcheck` on each package, which can be enabled with the `-d` command-line flag.
### Install tests
#### Monlithic install tests
For the monolithic source tree, each of the following install tests can be performed by building the release (as described above using the _sample_ or _alternate_ host.def file provided), running `make Install >& Install.LOG` (or other such command as appropriate for your platform), and checking the Install.LOG output for any failures.
1. Build and install with no host.def file ([[sample|http://www.freedesktop.org/~kem/install-tests/1/host.def]] [[alternate|http://www.freedesktop.org/~kem/install-tests/1a/host.def]])
1. Build and install with: Project``Root defined to be something other than the default, and Nothing``Outside``Project``Root defined as YES ([[sample|http://www.freedesktop.org/~kem/install-tests/2/host.def]] [[alternate|http://www.freedesktop.org/~kem/install-tests/2a/host.def]])
Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each install requirement above, defining Has``Freetype2 as NO is permitted. Each _alternate_ host.def file above have this define included.
#### Modular install tests
Since most of the modular source tree components have dependencies on other module components being built and installed, the `build.sh` script will both build and install all of the components in the appropriate dependency order. If the script completed without errors in the build step above, then the modular source tree both built and installed correctly.
### Conformance tests
After installing the full release of either the monolithic, modular or both source trees, the conformance tests can be run using the X Test Suite (XTS). Note that XTS has been updated recently and the build instructions have changed from what they were for past releases.
#### Setting up the X Test Suite
We have prepared an XTS package for those that are running Linux on an x86 machine. You can download the prebuilt package [[here|http://xorg.freedesktop.org/tests/xts5-Linux-i686.tgz]]. Untar this package and cd into the `xts5-Linux-i686` subdirectory.
For everyone else or if you have problems with the pre-built package, you will need to build XTS on your local system. To build it, download the `mktestpkg.sh` script from CVS and run it, which you can download [[here|http://cvs.freedesktop.org/*checkout*/xtest/utils/mktestpkg.sh?rev=1.4]]. This script will get the appropriate files, check out the latest version of XTS from CVS (Note when it asks you for a password, just hit enter). Then, change to the subdir that is created by the build script.
Now you should be ready to begin testing.
Note that an updated helper script (called `xreg`), which is used to run XTS, is already included in the test suite subdirectory.
#### Examples of how to use the xreg script
Here are some examples of how to use xreg to run the X test suite:
1. `xreg -xtest -xvfb`
* This runs xtest at all default depths using the Xvfb server.
* The default depths are 8, 15, 16, and 24+32.
* The "24+32" depth is one that uses a depth of 24 with a frame buffer bits per pixel of 32 (i.e., -depth 24 -fbbpp 32).
1. `xreg -xtest -xorg -d 16`
* This runs xtest at depth 16 using the Xorg server.
1. `xreg -xtest -xvfb -d 15 -test XCopyArea`
* This runs xtest at depth 15 using the Xvfb server, but it only runs the XCopy``Area test.
* Selecting individual tests is very useful to track down test failures.
1. `xreg -xtest -xvfb -d 16 -xvfbwidth 1280 -xvfbheight 1024 -test XFillRectangles`
* This runs xtest at depth 16 using the Xvfb server running at 1280x1024, but it only runs the XFill``Rectangles test.
Notes on using xreg:
* The output from these test runs are stored in `results` subdirectory by default. You can change the default output dir using the `-O` xreg command-line option.
* The material below assumes that you have done a full install of the system to /usr/X11``R6. However, if you are using a different `ProjectRoot` (for the monolithic tree) or a different `--prefix` (for the modular tree), you can use the following command-line option to the `xreg` script to run from that alternate location: `-projroot` _path-to-your-project-root-or-prefix_
* Running XTS at all bit depths can take a long time (6 hours on a P3 1GHz), so we suggest running them overnight.
* The files that are generated from an xreg run of xtest are:
1. `X-setup.DEPTH.DATE.TIME.output` -- this file contains the output of the X server during the setup phase
1. `xts5.DEPTH.DATE.TIME.errors` -- this file contains the list of errors found during the test run at depth _DEPTH_ made on date _DATE_ at time _TIME_.
1. `xts5.DEPTH.DATE.TIME.report` -- this file contains the report of all tests run at depth _DEPTH_ made on date _DATE_ at time _TIME_.
1. `xts5.DEPTH.DATE.TIME.summary` -- this file contains a summary of the errors found during the test run at depth _DEPTH_ made on date _DATE_ at time _TIME_. The summary file is only useful during full test runs (e.g., not when running individual tests).
1. `xts5.DEPTH.DATE.TIME.results` -- this directory contains the journal from the tests run at depth _DEPTH_ made on date _DATE_ at time _TIME_ as well as any error images generated.
* After running xtest, you can check to see if everything passed by looking at the summary/errors/report file(s) to see if there are any failures.
* As we get more test results in, we will be collecting a list of known failures. We can then update the `xreg` script to take those known errors into account in the summary file.
* The xreg script has only been tested on Linux systems. If there are problems with these scripts, please post patches to the [[xorg@lists.freedesktop.org|mailto:xorg@lists.freedesktop.org]] mailing list.
* There are many other options to xreg (and it can be used to run other tests such as x11perf). Run `xreg -help` to see the usage message.
After the XTS has been run with `xreg`, the report file lists the failures and a summary of the test run. Here is an example of the test report generated:
[[!format txt """
VSW5 SUMMARY RESULTS REPORT
Test suite version: 5.1.5
Specification version: Open Group Window Management (X11R5) document set
Test run by: guest
System: Linux t5 2.6.13-1.1576_FC5 #1 Sat Sep 24 15:23:34 EDT 2005 i686
Test run started: Monday November 07, 2005 11:52:31 PM
Test run ended: Monday November 07, 2005 01:23:18 AM
Journal file: /tmp/xtest/xts5-Linux-i686/results/xts5.16bpp.20051107.204759.results/journal
TCC command line: tcc -e -s /tmp/xreg.tet_scen.26369 -x /tmp/xreg.tetexec.cfg.26369 -i /tmp/xtest/xts5-Linux-i686/results/xts5.16bpp.20051107.204759.results xts5 all
Report type: -d 1 -s 1
CASES TESTS PASS UNSUP UNTST NOTIU WARN FIP FAIL UNRES UNIN ABORT
Xproto 122 389 267 2 120 0 0 0 0 0 0 0
Xlib3 109 161 129 3 28 1 0 0 0 0 0 0
Xlib4 29 324 275 17 27 5 0 0 0 0 0 0
Xlib5 15 84 77 2 5 0 0 0 0 0 0 0
Xlib6 8 50 20 0 30 0 0 0 0 0 0 0
Xlib7 58 172 150 9 13 0 0 0 0 0 0 0
Xlib8 29 165 133 10 22 0 0 0 0 0 0 0
Xlib9 46 1472 1176 46 36 201 8 0 5 0 0 0
Xlib10 23 95 58 2 35 0 0 0 0 0 0 0
Xlib11 33 195 87 22 43 43 0 0 0 0 0 0
Xlib12 27 138 108 2 15 12 0 0 1 0 0 0
Xlib13 32 269 161 3 102 3 0 0 0 0 0 0
Xlib14 45 58 49 0 5 0 0 0 4 0 0 0
Xlib15 45 159 126 0 33 0 0 0 0 0 0 0
Xlib16 30 105 82 1 22 0 0 0 0 0 0 0
Xlib17 55 131 110 0 21 0 0 0 0 0 0 0
Xopen 8 127 125 2 0 0 0 0 0 0 0 0
Xt3 21 73 73 0 0 0 0 0 0 0 0 0
Xt4 33 192 94 0 98 0 0 0 0 0 0 0
Xt5 10 69 28 0 41 0 0 0 0 0 0 0
Xt6 7 71 71 0 0 0 0 0 0 0 0 0
Xt7 11 106 97 0 6 3 0 0 0 0 0 0
Xt8 7 43 35 0 4 0 0 0 4 0 0 0
Xt9 33 189 132 2 55 0 0 0 0 0 0 0
Xt10 8 17 16 0 1 0 0 0 0 0 0 0
Xt11 58 285 249 0 34 0 0 0 1 1 0 0
Xt12 22 67 56 0 11 0 0 0 0 0 0 0
Xt13 39 178 129 0 47 0 0 0 1 1 0 0
Xt14 2 18 18 0 0 0 0 0 0 0 0 0
Xt15 1 2 0 2 0 0 0 0 0 0 0 0
XtC 29 147 90 1 56 0 0 0 0 0 0 0
XtE 1 1 1 0 0 0 0 0 0 0 0 0
TOTAL 996 5552 4222 126 910 268 8 0 16 2 0 0
"""]]
If the `TOTAL` line at the end of your report is similar to the one shown above, then we consider that your system has passed XTS. We are working to resolve the remaining failures and unresolved tests. From above, there are 16 failed and 2 unresolved tests. We will update the list above as we resolve more of these tests. If you are successful in fixing them, please let us know on the [[xorg@lists.freedesktop.org|mailto:xorg@lists.freedesktop.org]] mailing list.
#### Actually running the conformance tests
For the purposes of the testing the monolithic and modular source trees for this release, we ask that you run one of the following scenarios:
* For platforms based on a XFree86-style DDX, run XTS on Xorg either using the card driver for your video card or the `dummy` driver.
* For example: `xreg -xtest -xorg`
* This will run xtest at all the default depths using the Xorg server.
* Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass.
* For all other platforms, ``Xvfb`` should be used. This step is optional for those who ran with the Xorg server above.
* For example: `xreg -xtest -xvfb`
* This will run xtest at all of the default depths using the Xvfb server.
* Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass.
Additional notes:
* The ``Xvfb`` server is special X server that uses a virtual framebuffer. It is normally built and installed with the full release. See the `Xvfb(1)` for more information about this server.
* The ``dummy`` driver is a special driver available with the XFree86 DDX. To use the dummy driver, simply substitue it for your normal card driver in the `Device` section of your `xorg.conf` configuration file and comment out any card-specific options from that section. For example, if you normally uses an ATI driver, then you will have a `Device` section with `Driver "ati"` to let the X server know that you want it to load and use the ATI driver. It is not necessary to run both the card driver and the dummy driver for the conformance tests, but if you run into issues with the card driver not passing XTS, we recommend that you try running the dummy driver to see if the failures are resolved. This will help us narrow down any failures.
### Run tests
After installing the full release and running the conformance tests, run the subset of tests listed below that applies to the platform being tested. We would like to get as much coverage as possible, so please run these tests on as many cards as possible.
Tests to run for each card:
1. X Test Suite (in case you haven't already run it)
1. x11perf (Note that you can use `xreg` to automatically run x11perf -- see the script for details)
1. rendertest (found in the xapps CVS repository on freedesktop.org)
1. Standard graphical environment (e.g., Gnome or KDE desktop)
1. GL tests: glxgears, gloss, [[quake3|http://www.freedesktop.org/~jg/quake3.tar.gz]]
1. Switch to/from VTs (on Linux)
After running these tests, please enter them into our [[results database|http://results.x.org/index.php]]. This will allow us to track progress of the release. To see the current set of test results, go to the [[results summary page|http://results.x.org/show.php]].
|
