summaryrefslogtreecommitdiff
path: root/XorgTesting.mdwn
blob: 05897dbcd4ddd770ac8cdd072975a629c09faeb8 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118


# The X.Org Foundation Testing Plan


## Detailed test instructions

This section outlines the test procedure to follow when testing a release candidate.  When a test run has been completed, please e-mail the [[xorg@lists.freedesktop.org|mailto:xorg@lists.freedesktop.org]] mailing list with the following information so that we can track what has and has not yet been tested. Note that we are interested in progress: please do not wait to complete all phases of testing to send in reports. 

                     1. Your Name 
                     1. The date tested 
                     1. The platform you tested: 
                                             * The operating system tested (e.g., AIX, Cygwin, FreeBSD, HP-UX, Linux, etc.) 
                                             * The architecture tested (e.g., Alpha, AMD64, EM64T, IA-32, IA-64, Sparc, etc.) 
                                             * The distribution and release tested (e.g., Red Hat FC2, SUSE 9.1, Debian unstable, Solaris 9, etc.) 
                     1. The snapshot or release candidate tag tested (e.g., XORG-6_7_99_1, etc.) 
                     1. Build test status: passed or failed or untested 
                     1. Install test status: passed or failed or untested 
                     1. Conformance test status: passed or failed or untested 
                     1. Run test status: passed or failed or untested 
                                             * List the tests run 
For any test(s) that failed, please include in your report the test(s) that failed, and file a bugzilla report if no one has already filed one against the failure(s) you found. 


### Build tests

Each of the following build tests can be performed by copying the _sample_ host.def file (or the _alternate_) to the xc/config/cf directory and the running `make World >& World.LOG` (or other such command as appropriate for your platform), and then checking the World.LOG file for any failures. 

                     1. Build with empty host.def file ([[sample|http://www.freedesktop.org/~kem/build-tests/1/host.def]] [[alternate|http://www.freedesktop.org/~kem/build-tests/1a/host.def]]) 
                     1. Build with [[BuildServersOnly|BuildServersOnly]] defined as YES ([[sample|http://www.freedesktop.org/~kem/build-tests/2/host.def]] [[alternate|http://www.freedesktop.org/~kem/build-tests/2a/host.def]]) 
                     1. Build with [[DoLoadableServer|DoLoadableServer]] defined as NO ([[sample|http://www.freedesktop.org/~kem/build-tests/3/host.def]] [[alternate|http://www.freedesktop.org/~kem/build-tests/3a/host.def]]) 
Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each build requirement above, defining [[HasFreetype2|HasFreetype2]] as NO is permitted.  Each _alternate_ host.def file above have this define included. 


### Install tests

Each of the following install tests can be performed by building the release (as described above using the _sample_ or _alternate_ host.def file provided), running `make Install >& Install.LOG` (or other such command as appropriate for your platform), and checking the Install.LOG output for any failures. 

                     1. Build and install with no host.def file ([[sample|http://www.freedesktop.org/~kem/install-tests/1/host.def]] [[alternate|http://www.freedesktop.org/~kem/install-tests/1a/host.def]]) 
                     1. Build and install with: Project``Root defined to be something other than the default, and Nothing``Outside``Project``Root defined as YES ([[sample|http://www.freedesktop.org/~kem/install-tests/2/host.def]] [[alternate|http://www.freedesktop.org/~kem/install-tests/2a/host.def]]) 
Note that some systems do not have a compatible version of Freetype2 installed on their system, so in addition to each install requirement above, defining Has``Freetype2 as NO is permitted.  Each _alternate_ host.def file above have this define included. 


### Conformance tests

After installing the full release, the conformance tests can be run using the X test suite, which can be downloaded [[here|http://www.freedesktop.org/~kem/testing/xtest.tar.gz]].  A helper script (called ``xreg``) is used to run the X test suite, which can be downloaded [[here|http://www.freedesktop.org/~kem/testing/xreg]].  See the next two sections below for more information on how to setup and use these tools. 


#### Setting up the X test suite

Here are some brief instructions on how to download and set up everything that you will need to run the X test suite: 

                     1. Follow the directions at [[BuildingXtest|BuildingXtest]] 
                     1. `wget http://www.freedesktop.org/~kem/testing/xreg` 
Now you should be ready to begin testing. 


#### Examples of how to use the xreg script

Here are some examples of how to use xreg to run the X test suite: 

                     1. `xreg -xtest -xvfb` 
                                             * This runs xtest at all default depths using the Xvfb server. 
                                             * The default depths are 8, 15, 16, and 24+32. 
                                             * The "24+32" depth is one that uses a depth of 24 with a frame buffer bits per pixel of 32 (i.e., -depth 24 -fbbpp 32). 
                     1. `xreg -xtest -xorg -d 16` 
                                             * This runs xtest at depth 16 using the Xorg server. 
                     1. `xreg -xtest -xvfb -d 15 -test XCopyArea` 
                                             * This runs xtest at depth 15 using the Xvfb server, but it only runs the XCopy``Area test. 
                                             * Selecting individual tests is very useful to track down test failures. 
                     1. `xreg -xtest -xvfb -d 16 -xvfbwidth 1280 -xvfbheight 1024 -test XFillRectangles -n 3-5` 
                                             * This runs xtest at depth 16 using the Xvfb server running at 1280x1024, but only runs the third through the fifth tests of the XFill``Rectangles test. 
Notes on using xreg: 

                     * The output from these test runs are stored in `pwd`/results by default.  You can change the default output dir using the -O command line option. 
                     * The material below assumes that you have done a full install of the system to /usr/X11``R6.  However, if you are using a different Project``Root, you can use the following command line option to the xreg script to run from that alternate location: `-projroot` _path-to-your-project-root_ 
                     * The files that are generated from an xreg run of xtest are: 
                                             1. `X-setup..output` -- this file contains the output of the X server during the setup phase 
                                             1. `xtest.DEPTH.DATE.TIME.errors` -- this file contains the list of errors found during the test run at depth _DEPTH_ made on date _DATE_ at time _TIME_. 
                                             1. `xtest.DEPTH.DATE.TIME.report` -- this file contains the report of all tests run at depth _DEPTH_ made on date _DATE_ at time _TIME_. 
                                             1. `xtest.DEPTH.DATE.TIME.summary` -- this file contains a summary of the errors found during the test run at depth _DEPTH_ made on date _DATE_ at time _TIME_.  The summary file is only useful during full test runs (e.g., not when running individual tests). 
                                             1. `xtest.DEPTH.DATE.TIME.results` -- this directory contains the journal from the tests run at depth _DEPTH_ made on date _DATE_ at time _TIME_ as well as any error images generated. 
                     * After running xtest, you can check to see if everything passed by looking at the summary/errors/report file(s) to see if there are any failures. 
                     * There are some known failures that the summary file attempts to take into account.  The first part of the summary file is the list of failures, and at the end of the summary file is a diff between the known failures (e.g., XDraw``Arcs) and what the failures were for this run. 
                     * The xreg script has only been tested on Linux systems.  If there are problems with these scripts, please post patches to the [[xorg@lists.freedesktop.org|mailto:xorg@lists.freedesktop.org]] mailing list. 
                     * There are many other options to xreg (and it can be used to run other tests such as x11perf).  Run `xreg -help` to see the usage message. 

#### Actually running the conformance tests

For this section, one of the following should be used for testing: 

                     * For platforms based on a XFree86-style DDX, the ``dummy`` driver should be used. 
                                             * For example: `xreg -xtest -xorg` 
                                             * This will run xtest at all the default depths using the Xorg server. 
                                             * Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass. 
                     * For all other platforms, ``Xvfb`` should be used. 
                                             * For example: `xreg -xtest -xvfb -d "15 16 24+32"` 
                                             * This will run xtest at depths 15, 16 and 24+32 using the Xvfb server. 
                                             * Check the output of each report or summary file to make sure that all tests that are expected to pass do actually pass. 
                                             * Note that Xvfb does not currently run at depth 8, so the example above limits the testing to depths 15, 16 and 24+32.  Update, this problem has been fixed in CVS now and Xvfb at depth 8 works again. 
Additional notes: 

                     * The ``Xvfb`` server is special X server that uses a virtual framebuffer.  It is normally built and installed with the full release.  See the `Xvfb(1)` for more information about this server. 
                     * The ``dummy`` driver is a special driver available with the XFree86 DDX.  To use the dummy driver, simply substitue it for your normal card driver in the `Device` section of your `xorg.conf` configuration file.  For example, if you normally uses an ati driver, then you will have a `Device` section with `Driver "ati"` to let the X server know that you want it to load and use the ati driver; however, for these conformance tests, you would change that line to `Driver "dummy"` and remove any other ati specific options from the `Device` section. 

### Run tests

After installing the full release, you can run the subset of tests listed below that applies to the platform being tested.  Please run these tests on at least two different driver families (where applicable).  For example, on an IA-32 system running Linux, you could run the tests using one card from the ATI driver family and another card from the NVIDIA driver family. 

Tests for each driver family: 

                     1. X test suite (listed above) 
                     1. x11perf 
                     1. rendertest (found in the xapps CVS repository on freedesktop.org) 
                     1. Standard graphical environment 
                     1. GL tests: glxgears, gloss, [[quake3|http://www.freedesktop.org/~jg/quake3.tar.gz]] 
                     1. Switch to/from VTs (on Linux) 
-- [[KevinMartin|KevinMartin]] - 18 Jan 2005