1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
|
IGT Test documentation
======================
Legacy way
----------
IGT has been providing a way to document tests in runtime by using tree macros:
- IGT_TEST_DESCRIPTION(test_file_description)
- igt_describe(subtest_description) and igt_describe_f(format, args...)
This is limited, as:
- Each test/subtest has just one “field”: description. It doesn't
allow specifying what features are tested and/or special requirements;
- It is meant to produce a very concise documentation;
- Integration with external platforms to group tests per category
is not possible, as there's no way to tell what category each test
belongs;
- Format is not easily expansible;
- The build system doesn’t verify if all tests are documented.
As time passes, documentation tends to be outdated or forgotten,
as new patches modify the test set.
Documentation via structured comments (testplan)
------------------------------------------------
With the addition of Xe driver, a new way to document tests was added.
It is based on special comment-like annotation inside the C code.
It was written to be flexible, so the valid fields to be used by are
described on a JSON configuration file. That makes easy to add/modify
the fields as needed. It is also easy to use the documentation to
integrate with external reporting tools.
As an additional benefit, the documentation tags will be generating a
Restructured Text file. It is possible to add enriched test descriptions
if needed.
The build system can also optionally enforce a check at build time to
verify if the documentation is in place.
testplan configuration file
---------------------------
The configuration file contains the fields to be used for documenting
tests and test names. It may also mark a property as mandatory.
A typical example is:
```
{
"description": "JSON example file",
"name": "Tests for XYZ Driver",
"files": [ "test.c" ],
"fields": {
"Feature": {
"_properties_": {
"description": "Feature to be tested"
}
},
"Description" : {
"_properties_": {
"mandatory": true,
"description": "Provides a description for the test/subtest."
}
}
}
}
```
Documenting tests via testplan
------------------------------
A typical documentation markup at the test source code looks like:
```
/**
* TEST: Check if new IGT test documentation logic functionality is working
* Mega-feature: IGT test documentation
* Category: Software build block
* Sub-category: documentation
* Functionality: test documentation
* Issue: none
* Description: Complete description of this test
*
* SUBTEST: foo
* Description: do foo things.
* Foo description continuing on another line
*
* SUBTEST: bar
* Description:
* Do bar things.
* Bar description continuing on another line
*/
```
It is also possible to add variables to the documentation with:
```
/**
* SUBTEST: test-%s-binds-with-%ld-size-%s
* Description: Test arg[3] arg[1] binds with arg[2] size
*
* SUBTEST: test-%s-%ld-size
* Description: Test arg[1] with %arg[2] size
*
* arg[1]:
*
* @large: large
* something
* @small: small
* something
*
* arg[2]: buffer size
*
* arg[3]:
*
* @misaligned-binds: misaligned
* @userptr-binds: user pointer
*/
```
The first "%s" will be replaced by the values of arg[1] for the subtest
name. At the description, arg[1] will be replaced by the string after
the field description. The same applies to the subsequent wildcards.
The above will produce the following output (using the example
configuration file described at the past session:
```
``igt@test@test-large-<buffer size>-size``
:Description: Test large something with <buffer size> size
``igt@test@test-large-binds-with-<buffer size>-size-misaligned-binds``
:Description: Test misaligned large something binds with <buffer size> size
``igt@test@test-small-binds-with-<buffer size>-size-misaligned-binds``
:Description: Test misaligned small something binds with <buffer size> size
``igt@test@test-small-<buffer size>-size``
:Description: Test small something with <buffer size> size
``igt@test@test-large-binds-with-<buffer size>-size-userptr-binds``
:Description: Test user pointer large something binds with <buffer size> size
``igt@test@test-small-binds-with-<buffer size>-size-userptr-binds``
:Description: Test user pointer small something binds with <buffer size> size
```
Wildcard replacement can also be used to maintain an argument with the
same name at the replaced description. That makes easy to document
numeric arguments with a fixed testset, like:
```
/**
* SUBTEST: unbind-all-%d-vmas
* Description: unbind all with %arg[1] VMAs
*
* arg[1].values: 2, 8
* arg[1].values: 16, 32
*/
/**
* SUBTEST: unbind-all-%d-vmas
* Description: unbind all with %arg[1] VMAs
*
* arg[1].values: 64, 128
*/
```
|
