[[!meta title="The X New Developer’s Guide: Xlib and XCB"]]
# Xlib and XCB
*Alan Coopersmith*
[[!toc levels=3 startlevel=2]]
The two most popular questions about Xlib and XCB may be
"What are they?" and "What's the difference?"
Most programming languages make it awkward for X
applications to spit raw X protocol down the network and
take apart the protocol coming back. Thus, X toolkits and
applications are a lot easier to write if some library
handles these jobs for them, providing an API that fits with
the programming language and environment for connecting to
the X server.
At the bottom level of the X client library stack are Xlib
and XCB, two helper libraries (really sets of libraries)
that provide API for talking to the X server. Xlib and XCB
have different design goals, and were developed in different
periods in the evolution of the X Window System.
Most application developers should call Xlib and XCB
sparingly. Higher level toolkits provide more efficient
programming models, and support features expected in modern
applications, including support for complex
internationalized input and output, accessibility, and
integration with desktop environments. However, sometimes
applications will find themselves needing to make calls to
the raw underlying X11 libraries for operations not
supported by toolkits. An application might need to make
calls to X extension API's not covered in the current
version of the toolkit's programming model. It is also
common for drawing not to be wrapped by toolkit API's. [Yes?
--po8]
The original C-language X11 API is libX11, often referred to
as "Xlib". It was designed to look like a traditional
library API, hiding the fact that calls result in protocol
requests to a server. Calls that don't require a response
from the X server are queued in a buffer to be sent as a
batch of requests to the server. Those that require a
response flush all the buffered requests and then block
until the response is received.
Xlib's mix of synchronous and asynchronous behaviors causes
some problems. Xlib's behaviour is often confusing to new
programmers. Calls appear to work sometimes and not others,
because it is not obvious which calls implicitly flush the
buffer. The asynchronous nature of many calls makes it
difficult to debug problems. When an error is reported, the
stack trace shows the call that was being made when the
error was received and processed, often many calls after the
one that caused the error. Finally, Xlib's synchronous calls
incur avoidable round-trip latency. This latency has a
notable effect on application performance; in particular,
startup times are often greatly increased.
After many years of experience with Xlib, and learning from
it and other protocol interface libraries, a second attempt
was made at defining a C language binding for X11: the "X11
C Binding" layer XCB. XCB makes the client-server nature of
the protocol explicit in its design. The client is in charge
of deciding when to flush the request buffer, when to read
results and when to wait for the server to respond.
For instance, to lookup a window property, the Xlib code is
a single function call:
XGetWindowProperty(dpy, win, atom, 0, 0, False, AnyPropertyType, &type_ret, &format_ret, &num_ret, &bytes_after, &prop_ret);
Xlib generates the request to the X server to retrieve the
property and appends it to its buffer of requests. Since
this is a request that requires a response, Xlib then
flushes the buffer, sending the contents to the X
server. Next, Xlib waits until the X server processes all
the requests preceding the property retrieve request, and
sends the property retrieve reply. Xlib then returns the
reply to the client.
Xlib also provides convenience functions that wrap a
property request. These convenience functions retrieve
specific properties, knowing the details of each property
and how to request and decode it. Examples include
XGetWMName and XGetWMHints. Some of these functions could be
written outside Xlib, but many use Xlib internals in
non-trivial ways and are thus inseparable. [Yes? --po8]
XCB on the other hand, provides functions generated directly
from the protocol descriptions in an "obvious" mechanistic
way. XCB functions map directly onto the protocol, with
separate functions to put requests into the outgoing buffer
and to read results back from the X server asynchronously
later. The XCB version of the above code is:
prop_cookie = xcb_get_property (dpy, False, win, atom, XCB_GET_PROPERTY_TYPE_ANY, 0, 0);
prop_reply = xcb_get_property_reply (dpy, prop_cookie, NULL);
The power of XCB is in allowing those two steps to have as
much code as you want between them. The programmer decides
when to wait for data, instead of being forced to wait for
the data returned by a request at the time the request is
issued.
## Example: Converting xwininfo from Xlib to XCB
The program xwininfo is a command-line utility to print
information about windows on an X server. It knows in
advance, from the command line options, most of the data it
needs to request information for each window from the
server. Thus, xwininfo can make its requests all at once,
and then wait for the results to start coming in. When using
the -tree option to walk the window tree, xwininfo can
request the data for all the children of the current window
at once, batching even further. On a local connection on a
single CPU server, this means less context switches between
X client and server. On a multi-core/CPU server, the X
server can process requests on one core while the client is
handling the responses on another core as they become
available, improving performance. On remote connections, the
requests can be grouped into packets closer to the MTU size
of the connection, instead of just sending whatever requests
are in the buffer when a request is made that needs a
response.
Version 1.1 of xwininfo was converted from Xlib to XCB by
Alan Coopersmith. It was tested with a GNOME desktop session
with a few clients. xwininfo was run as "xwininfo -root
-all": this started xwininfo at the root of the window
hierarchy and asked it to traverse the tree, requesting all
the information available for each window along the way. In
this sample session it found 114 windows. (In X, a window is
simply a container for drawing output and receiving
events. X windows are often regions of, or borders around,
the "user windows"). When running locally on a four-core
Intel Nehalem CPU, both versions ran so fast (0.05 seconds
or less) that the difference in time was too small to
accurately measure. To measure remote performance, "ssh -X"
was used to tunnel an X11 connection from California to a
computer in China, and from there back to the workstation in
California, introducing a huge amount of latency. With this
setup, the difference was dramatic between the two:
Xlib: 0.03u 0.02s 8:19.12 0.0%
xcb: 0.00u 0.00s 0:45.26 0.0%
Of course, xwininfo is an unusual X application in a few ways:
* xwininfo runs through the requests as fast as it can and
then exits, not waiting for user input (unless you use
the mode where you click on a window to choose it, after
which it runs through as normal). Once they are up and
running, most X applications spend most of their time
waiting for user input, so the overall runtime won't
decrease as much by reducing the time spent
communicating with the X server. However, application
startups are typically dominated by round-trip times,
and so proper use of XCB reduces the huge startup times
of X applications running over high-latency (and even
medium-latency) connections.
* xwininfo uses only the core protocol and shape
extension. It does not use the more complex extensions,
such as Render or Xinput, that most modern applications
use. Xinput, XKB, and GLX are especially problematic, as
those have not yet been fully supported in an XCB
release, though support has been worked on through some
Google Summer of Code projects.
* xwininfo is small enough that a complete reworking to
use XCB in one shot was feasible. Most applications are
much larger than this. XCB is aimed primarily at new
code and at toolkits: it is designed specifically to
interoperate with existing Xlib applications. Calls to
Xlib and XCB can be mixed, so Xlib applications can be
converted partially or incrementally if desired.
* xwininfo used only raw Xlib, without any toolkit. Thus,
it did not have to worry about which X library the
toolkit used.
* xwininfo uses only a few of the Xlib helper
functions. This makes it more directly mappable to
XCB. Applications that rely on Xlib's input method
framework, compose key handling or character set
conversion, for example, would be harder to
port. Fortunately, modern toolkits handle most of this
functionality in the toolkit layer anyway.
xwininfo did rely on Xlib helper functions for converting
the window name property from other character sets---the XCB
version currently only works for UTF-8 and Latin-1 window
names. Since most modern toolkits use UTF-8, no one is
likely to notice. Older applications with localized window
names will fail, but there are few of these in use.
## Mixing Xlib & XCB calls
As mentioned above, XCB provides a method for incremental
conversion from Xlib to XCB. One can use libX11 to open the
display and pass the Display pointer it returns to existing
code, toolkits, and libraries. To call an XCB function, one
can convert the Display pointer to an xcb_connection_t
pointer for the same connection. This enables calling into
Xlib and XCB from the same application.
Xlib and XCB compatibility was achieved by rebuilding libX11
as a layer on top of libxcb. Xlib and XCB share the same X
server connection and pass control of it back and
forth. That option was introduced in libX11 1.2, and is now
always present (no longer optional) since the 2010 release
of libX11 1.4.
## Example: Converting xdpyinfo extension queries to XCB
xdpyinfo is another command-line tool in the standard X
Window System toolset. Like xwininfo, xdpyinfo prints a lot
of information about the X server. xdpyinfo calls many
extensions, and few of its calls block waiting for a
response from the server. If you add the "-queryExt" option,
though, for every extension xdpyinfo calls XQueryExtension
to print which request, event, and error ids are assigned to
that extension in the currently running server. These ids
are dynamically assigned, and vary depending on the set of
extensions enabled in a given server
build/configuration. Thus, the list of extension ids is
critical information to have when debugging X error reports
that reference them. Using "xdpyinfo -queryExt" is
especially needed when reporting an X error message that
comes from a custom error handler like the one in the gtk
toolkit: such error handlers typically omit the extension
information found in the default Xlib error handler, so the
person reading the bug report will be unable to identify the
extension in which the error was encountered.
The Xlib call XQueryExtension takes one extension name at a
time, sends a request to the X server for the id codes for
that extension, and waits for a response so it can return
those ids to the caller. On the Xorg 1.7 server used as the
test system for this conversion, there were 30 active X
extensions, so that's 30 tiny packets sent to the X server,
30 times that the xdpyinfo client blocks in poll() waiting
for a response, and 30 times that the X server goes through
the client handling and request scheduling code before going
back to block again on its own select() loop.
Note: XListExtensions can be used to get a list of available
extensions that can be called with XQueryExtension.
A simple patch to xdpyinfo replaced just that loop of calls
to XQueryExtension with two loops. The first loop called
xcb_query_extension for each extension. When the entire
batch of queries had been issued, a second loop called
xcb_query_extension_reply to start collecting the batched
replies. Gathering system call counts with "truss -c" showed
the expected reduction in a number of system calls made by
the xdpyinfo client:
[[!table header="yes" class="mointable" data="""
System call|Xlib|xcb
writev| 40| 11
poll| 80| 22
recv|117| 29
total|237| 62
"""]]
Over a TCP connection, the switch to XCB for this
transaction reduced both the number of packets and (due to
tcp packet header overhead) the overall amount of data:
[[!table header="yes" class="mointable" data="""
|Xlib|xcb
TCP packets| 93| 35
TCP bytes|11554|7726
"""]]
This sort of change is far more feasible than wholesale
conversion to XCB for most applications. Find the hotspots
where the application is waiting for data from the server
and convert those. There are almost always opportunities in
application startup code, when the application is gathering
the information about the X server and session. Converting
just those calls to more efficient sets of XCB calls can
have major performance benefits. Earlier work by X
developers reduced the latency of many applications by
converting repeated calls to XInternAtom with a single call
to fetch multiple atoms at once via XInternAtoms. XCB
permits a generalization of this principle.
## Extension libraries
Each new extension to the X11 protocol adds requests that
clients can make to the X server. To allow client software
to utilize these requests, most extensions offer API's built
on top of Xlib or XCB. These API's use the library's
connection marshalling to include their requests in the
stream sent to the X server.
In the early X11 releases, many of the smaller and more
common extensions were grouped into a common library,
libXext. You will find several there today which are still
in use, such as the MIT-SHM Shared Memory extension, the
SHAPE extension for non-rectangular windows, and the SYNC
extension for event synchronization. However, libXext also
includes some API's for extensions no longer found in
current Xorg server releases, such as App-Group and
Low-Bandwidth X (LBX), as well as extensions many apps never
use, such as DPMS for display power management. Since these
extension API's cannot be removed from libXext without
breaking any existing application which may be using them,
the code is stuck in there.
Accordingly, new Xlib extension API's are no longer added to
libXext. Instead a new library utilizing libX11 is created
for each extension. Having a library per extension makes it
easier to evolve the API for that extension, to deprecate an
obsolete extension and to only link it into the clients that
actually need it. Almost all modern extensions have their
own Xlib API library---libXrender for the RENDER extension,
libXcomposite for the COMPOSITE extension, and so on. A
handful of extensions are so core to the protocol
interaction that they are supported directly in libX11
itself, such as BigRequests, XC-MISC, and XKB.
When XCB added its API style to the mix, it followed the
newer style and created a "libxcb"-prefixed library for each
extension---libxcb-composite, libxcb-render, etc. Since XCB
can generates the API code for an extension automatically
from an XML description of the extension protocol, new
extension API's are created by simply adding the extension
description to the xcb-proto package and rebuilding.
Unfortunately, some of the older extensions have complex
protocols that are not easily described in the XML
syntax. Work is ongoing to extend the syntax and code
generator to handle these. The XKB & GLX protocols are
current challenges.
## API documentation
To write code using Xlib or XCB, you'll need to know the
details of the library API. Xlib includes man pages for
most functions, providing a good API reference. libXext
includes man pages for some of the extension API's it
includes, but not all of them. Man page coverage is even
more spotty in the individual Xlib-based extension
libraries.
There's also a more complete guide to the Xlib API, and
specifications for many extension API's, in the X.Org online
doc set at .
For extensions without Xlib-style API documentation, the
calls are usually simple mappings to the protocol
specifications provided in the above-linked doc set.
For XCB, the documentation relies even more heavily on the
protocol specifications. The generated API is an exact
mapping to the X protocol; it translates the C call data
into X protocol encoded packets as straightforwardly as
possible. The connection management and other functions in
the XCB API are documented at
. Work is in progress on
adding support to XCB to generate Unix style reference man
pages from the XML protocol descriptions as well, for
developer convenience.
There is also a XCB tutorial, "Basic Graphics Programming
With The XCB Library" at
.
Helping us improve our API documentation for either library
stack is always appreciated. See the [Documentation chapter](../documentation)
later in this guide for more information.
[[!meta link="../" rel="contents"]]
[[!meta link="../client-ecosystem" rel="prev"]]
[[!meta link="../using-extensions" rel="next"]]