1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
|
This repository contains a collection of traces that represent
applications in the wild. These aim to give an overall feel as to
whether cairo is faster for everyday use. In particular, they represent
behaviour that someone thought worthy of further study and analysis!
Building
--------
Traces are stored in the archive as compressed files. The make command
uncompresses them and optimizes the traces for the system cairo:
git clone git://anongit.freedesktop.org/cairo-traces
cd cairo-traces && make && cd ..
Alternately, cairo-perf-trace also looks for cairo traces at
/usr/src/cairo-traces and /usr/share/cairo-traces. Or, you can point
the environment variable CAIRO_TRACE_DIR to any location you prefer.
To capture and replay traces you will need the cairo-trace and
cairo-perf-trace tools from cairo. If you are testing against the
system installed cairo library (the default behavior), you'll need the
system cairo perf tools as well; this may be packaged separately from
cairo itself, depending on your distro (e.g. look for cairo-tools on
RedHat-alikes or cairo-perf-utils on Debian-alikes).
On the other hand, if you want to test against a local cairo build
rather than the system cairo, just unpack the traces you need and run
them with cairo-perf-trace directly. For example:
lzma -cd cairo-traces/benchmark/ocitysmap.lzma > ./ocitysmap.cs
cairo/perf/cairo-perf-trace ./ocitysmap.cs
Capturing Traces
----------------
Using cairo-trace you can record the exact sequence of graphic
operations made by an application and replay them later. These traces
can then be used by cairo-perf-trace to benchmark the various backends
and patches.
To record a trace:
cairo-trace --profile <application> [args...]
This will produce a application.$pid.lzma file. If you think it is
interesting add it to this repository, give it a short but descriptive
name, and append a note to this file explaining what the trace captures.
Or place it into cairo/perf/cairo-traces (or set CAIRO_TRACE_DIR to
point at your trace directory), and the trace will be included in your
cairo performance testing when running make perf.
Another way to collect traces is:
cairo-trace --no-mark-dirty --no-callers <application> [args...]
--no-mark-dirty is useful for applications that are paranoid about
surfaces being modified by external plugins outside of their control,
the prime example here is firefox.
--no-callers disables the symbolic caller lookup and so speeds tracing
(dramatically for large c++ programs) and similarly speeds up the replay
as the files are much smaller.
The output file will be called $APPLICATION.$PID.trace, the actual path
written to will be displayed on the terminal.
Replaying Traces
----------------
The macro-benchmarks are run by a single program called
cairo-perf-trace:
cairo-perf-trace <trace-name>
cairo-perf-trace produces the same output and takes the
same arguments as cairo-perf-micro.
By default cairo-perf-trace will loop over all traces within
cairo-traces for all test targets. To run against a subset specify a
series of traces on the command line. Some examples of running it:
# Report on all tests with default number of iterations:
cairo-perf-trace
# Report on 100 iterations of all firefox tests:
cairo-perf-trace -i 100 firefox
# Generate raw results for 10 iterations into cairo.perf
cairo-perf-trace -r -i 10 > cairo.perf
# Append 10 more iterations of the poppler tests
cairo-perf-trace -r -i 10 poppler >> cairo.perf
# Run all the firefox tests, but only against the xlib backend
CAIRO_TEST_TARGET=xlib cairo-perf-trace firefox
Analyzing Traces
----------------
Several tools are available for post-processing the performance test
output, such as cairo-perf-diff. See cairo/perf/README for more
details.
Guidelines
----------
Initial thoughts...
1. Nothing obscene, or otherwise unsuitable for viewing by children or in the
workplace - please be considerate to others. The trace should also respect
copyright where appropriate. Even though the content is not usually visible
during a replay, it is not acceptable to distribute such material via
freedesktop.org.
2. Each trace must be "representative of a significant use". The goal is a
set of traces that capture the behaviour of real applications during active
sessions. (We must guard against the tests becoming too artificial as that
is better served by purpose-written micro-benchmarks.)
3. For practical reasons (i.e. continuous performance testing), each trace
should neither be profligate in the resources consumed nor the duration of
the run. However, I'm loathe to exclude tests simply because they take too
long or can not be run on tiny machines. (Instead, I plan to create subset
of tests that have been truncated to run within 60s on a 2GHz Intel Core2.)
4. If the application significantly improves such that the current trace
is no longer representative of that application, it must be replaced. This
is to ensure that we do not try to optimise for "historically broken
behaviour".
Traces (found in the full/ directory)
-------------------------------------
firefox-20090601 - browsing various news site, in particular the
international BBC News website.
firefox-36-20090609 - browsing news sites and blogs at font size 36
Warning: this needs lots of RAM, working set > 512 MiB.
firefox-36-20090611 - browsing http://www.bbc.co.uk/worldservice/languages/,
single page at a time with font size 36
Warning: this needs lots of RAM, working set > 512 MiB.
firefox-woodtv - A page with a gratiutiously large and obscured background,
can cause misbehaving drivers to freeze!
http://www.woodtv.com/
firefox-world-map - Rotating a vector map of the world:
http://people.mozilla.com/~jmuizelaar/world-map.html
firefox-periodic-table - Resizing a German periodic table of the elements
http://upload.wikimedia.org/wikipedia/commons/6/63/Periodic_table_(german)_4.svg
firefox-talos-gfx-20090702 - Firefox Talos performance suite, gfx tests (SYNTHETIC)
https://wiki.mozilla.org/StandaloneTalos tgfx
firefox-talos-svg-20090702 - Firefox Talos performance suite, svg tests (SYNTHETIC)
https://wiki.mozilla.org/StandaloneTalos tsvg
gnome-terminal-20090601 - exercising the various performance tests within vte
gnome-terminal-20090728 - ls -lR /usr
A real-world example of a many glyphs micro-benchmark!
evolution-20090605 - perusing some old cairo@ mail
swfdec-giant-steps-full - http://michalevy.com/wp-content/uploads/Giant%20Steps%202007.swf
swfdec-youtube-full - http://www.youtube.com/v/l6IAoPAjzpw&hl=en&fs=1&
swfdec-fill-rate - an example slideshow created from www.slide.com
swfdec-fill-rate-2xaa - ditto, but with 2x FSAA
swfdec-fill-rate-4xaa - ditto, but with 4x FSAA
poppler-bug-12266 - A notorious document that is slow to render in poppler,
and suffered a severe regression with the clipping
rework.
poppler-record - Experimental traces to exercise the recording surface
poppler-record-ordered
poppler-record-drunkard
epiphany-20090810 - Downloading a 2 GB file, mainly updating a progress
bar and associated text.
epiphany-webkit-20090810 - Running epiphany using the webkit backend over a
couple of planets and the internal BBC news.
It appears the cairo backend is not very mature...
gnome-system-monitor-20090821 - A clean cairo version of gnome-system-monitor,
graphing CPU/memory/network activity whilst
reading the news with epiphany.
Benchmarks
----------
The benchmarks are a set of trimmed traces that should run in a reasonable
timeframe (I'm aiming for <10 seconds on a 2GHz Core2 with cairo-image, using
./csi-trace --trim=10). The idea here is that this allows us to quickly
determine a representative metric without running the whole suite -- and is
useful for the casual user. The full traces are kept as they will exercise
corner cases much more thoroughly and are not compromised by the conversion.
To run the benchmark subset, use cairo-perf-trace benchmark.
firefox-talos-gfx - The synthetic firefox benchmark (identical to
firefox-talos-gfx-20090702) that exercise the core renderer.
firefox-talos-svg - The synthetic firefox benchmark (identical to
firefox-talos-svg-20090702) that exercise the SVG renderer.
firefox-planet-gnome - Scrolling the full height of http://planet.gnome.org/
firefox-fishtank - A HTML5 canvas demo, courtesy of
http://ie.microsoft.com/testdrive/Performance/FishIETank/Default.html
This is a sprite based animation, and mostly tests the
performance of unaligned, scaled alpha blits.
firefox-fishbowl - A HTML5 canvas demo, courtesy of
http://ie.microsoft.com/testdrive/Performance/fishbowl/Default.html
A slightly more complex fishtank.
firefox-paintball - A HTML5 canvas demo, courtesy of
http://ie.microsoft.com/testdrive/Performance/Paintball/Default.html
firefox-particles - A HTML5 canvas demo, courtesy of
http://ie.microsoft.com/testdrive/Performance/ParticleAccelration
firefox-chalkboard - Another HT|ML5 canvas demo written by Microsoft to
demonstrate their own virtues by highlighting poor
performance elsewhere.
http://ie.microsoft.com/testdrive/Performance/Chalkboard
firefox-canvas - A pair of synthetic canvas benchmarks, found in the wild
firefox-canvas-alpha at http://flashcanvas.net/examples/dl.dropbox.com/u/1865210/mindcat/canvas_perf.html
firefox-canvas-scroll- An slideshow of cats and dogs on a horizontally scrolling canvas.
http://manichord.com/opensign/ffdebug/ffscroll.html
firefox-asteroids - A HTML5 game benchmark,
http://www.kevs3d.co.uk/dev/asteroidsbench/
firefox-scrolling - A bug report concerning an ancient performance regression
http://home.arcor.de/bazonbloch/scrolltest/whyisscrollingonlinuxslowerthanonwindows.htm
firefox-tron - An animated digital book by Disney advertising Tron Legacy
http://disneydigitalbook.go.com/tron
gvim - Scrolling within a window showing a UTF-8 document
gnome-terminal-vim - The same test but using vim inside a gnome-terminal,
but repeated more often to give a comparable time.
evolution - based on full/evolution-20090607
gnome-system-monitor - based on full/gnome-system-monitor-20090821
midori-zoomed - scroll http://www.jakob-persson.com/styles_demo/?s=5
whilst zoomed in.
ocitysmap - ocitysmap-render -f png -c Sanguinet
popper - based on full/poppler-20090811
poppler-reseau - A vector based map of a French city, which is
abnormally slow in cairo:
https://bugs.freedesktop.org/show_bug.cgi?id=38927
swfdec-giant-steps - based on full/swfdec-giant-steps
swfdec-youtube - based on full/swfdec-youtube
xfce4-terminal-a1 - scrolling 'man gcc' using the terminus font
grads-heat-map - provided by Jennifer Adams, a typical output from
GraDS, an open-source program for the analsyis and
display of meteorlogical data. The trace is heavy in
unantialiased fills and strokes.
chromium-tabs - provided by Jindrich Makovicka to exercise an issue
he found will switching between tabs in Chromium, where
the xlib backend on his machine was over 200x slower
than the image backend.
gtkperf - a refined capture from the GtkPerf benchmark using the
AdWaita theme. Only interesting from the pathological
GTK+ perspective.
thunar - a capture of using a selection box in Thunar. This is
noteworthy for their new rendering technique of using a
MULTIPLY composite operation for the selection rectangle.
|