path: root/Building_the_X_Window_System.mdwn

# Building the X Window System from Source 

[[!toc startlevel=2 levels=2]]


## Introduction

This guide is for developers who wish to build the X Window System from source.  If your area of interest is limited to a single package, like a driver or an application, check with your O/S first for development facilities.

<a name="RequiredTools"></a>
## Required Tools and Software Dependencies

The most common cause of build failures for first time builders is the lack of required tools. There are over 30 tools required to build all of the X Window System 200+ packages and half of them are not installed by default on most O/S or Linux distributions. The minimum version needed is shown in brackets where applicable.

It is strongly advised you install such packages from your O/S official packaging system. Check the [[RequiredPackages|RequiredPackages]] for your O/S. Some of the tools may not be required to build some areas of X. Some GNU utility, like make, have native O/S equivalent on non GNU systems.

### Source Code Version Control

* [[git|http://git-scm.com/]] &ndash; The fast version control system

<a name="GNUBuildSystem"></a>
### GNU Build System

* [[autoconf (2.62)|http://www.gnu.org/software/autoconf/]] &ndash; Produce shell scripts that automatically configure software source code packages
* [[automake (1.11)|http://www.gnu.org/software/automake/]] &ndash; Create GNU standards-compliant Makefiles from template files
* [[autopoint|https://www.gnu.org/software/gettext/]] &ndash; Copies standard gettext (i18n) infrastructure files into a source package
* [[gmake|http://www.gnu.org/software/make/]] &ndash; The GNU version of the "make" utility  to maintain groups of programs
* [[libtool (2.2)|http://www.gnu.org/software/libtool/]] &ndash; Provide generic shared library support
* [[pkg-config (0.22)|http://pkg-config.freedesktop.org/wiki/]] &ndash; Return metainformation about installed libraries

### Development Tools

* [[yacc|http://en.wikipedia.org/wiki/Yacc]] &ndash; The original AT&T parser generator or an upward compatible version such as [[bison|http://www.gnu.org/software/bison/]] or [[byacc|http://invisible-island.net/byacc/]] without using its specific features
* [[lex|http://en.wikipedia.org/wiki/Lex_%28software%29]] &ndash; The original AT&T lexical analyser generator or an upward compatible version such as [[flex|http://flex.sourceforge.net/]] whitout using its specific features
* [[gcc (2.95)|http://www.gnu.org/software/gcc/]] &ndash; The GNU C compiler
* [[gettext|http://www.gnu.org/software/gettext/]] &ndash; GNU Internationalization utilities
* [[gperf (3.0.1)|http://www.gnu.org/software/gperf/]] &ndash; Perfect hash function generator
* [[m4 (1.4)|http://www.gnu.org/software/m4/]] &ndash; A macro processing language
* [[ncurses (5)|http://www.gnu.org/software/ncurses/]] &ndash; Basic terminal type definitions

### Development Tools/Libraries

* [[perl (5)|http://www.perl.org/]] &ndash; Larry Wall's Practical Extraction and Report Language
* [[intltool (0.30)|http://freedesktop.org/wiki/Software/intltool]] &ndash; Utility scripts for internationalizing XML
* [[libpng (1.2.44)|http://www.libpng.org/pub/png/]] &ndash; Reading and writing PNG (Portable Network Graphics) format files
* [[llvm|http://www.llvm.org/]] &ndash; The Low-Level Virtual Machine (LLVM)
* [[libtalloc|http://talloc.samba.org/talloc/doc/html/index.html]] &ndash; A hierarchical pool based memory allocator with destructors
* [[zlib (1.2.5)|http://zlib.net/]] &ndash; Software library used for data compression
* [[libgbm|http://mesa3d.sourceforge.net/]] &ndash; GBM buffer management library for Glamor
* [[libmtdev|http://bitmath.org/code/mtdev/]] &ndash;  Multitouch Protocol Translation Library
* [[libudev|http://www.freedesktop.org/software/systemd/libudev/]] &ndash;  Dynamic library to access udev device information

### Cryptography

The X server requires one of the following cryptographic libraries.

* [[libgcrypt|http://directory.fsf.org/project/libgcrypt/]] &ndash; General purpose cryptographic library
* [[libcrypto|http://www.openssl.org/docs/crypto/crypto.html]] &ndash; OpenSSL cryptographic library
* [[libsha1|https://github.com/dottedmag/libsha1]] &ndash; Tiny SHA1 implementation for embedded devices
* [[libmd|http://martin.hinner.info/libmd/]] &ndash; Cryptographic message digest library
* [[CommonCrypto|http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man3/Common%20Crypto.3cc.html]] &ndash; libSystem digest library on MAC OS X

### Fonts

* [[fontconfig (2.2.3)|http://www.freedesktop.org/wiki/Software/fontconfig]] &ndash; A library for configuring and customizing font access
* [[freetype (2.2)|http://freetype.sourceforge.net/index2.html]] &ndash; A Free, High-Quality, and Portable Font Engine

### Documentation/XML Tools

These tools are optionals. If they are missing, the documentation will not be generated.

* [[Asciidoc|http://www.methods.co.nz/asciidoc/index.html]] &ndash; Highly configurable text format for writing documentation
* [[Doxygen (1.6.1)|http://www.stack.nl/~dimitri/doxygen/]] &ndash; Documentation system for C, C++, Java, Python and other languages
* [[Xmlto (0.0.22)|https://fedorahosted.org/xmlto/]] &ndash; XML-to-any converter
* [[Xsltproc (1.1.26)|http://xmlsoft.org/XSLT/xsltproc2.html]] &ndash; XSLT command line processor

### Freedesktop.org Dependencies

These freedesktop.org projects are required software dependencies to build X. If you use a build process based on the build.sh script or on JHBuild, the source will be obtained for you.

* [[Mesa|http://www.mesa3d.org/]] &ndash; The Mesa 3D Graphics Library
* [[Pixman|http://cgit.freedesktop.org/pixman/]] &ndash; Low Level Pixel Manipulation Library
* [[XKeyboardConfig|http://www.freedesktop.org/wiki/Software/XKeyboardConfig]] &ndash; X Keyboard Configuration Database
* [[libevdev|http://www.freedesktop.org/wiki/Software/libevdev/]] &ndash; 	Wrapper library for evdev devices


## Selecting a Build Process

We are assuming you have installed the [[required tools|ModularDevelopersGuide]] but nothing else. There are two maintained build processes you can use. One is based on a simple Bourne script (build.sh) and one is a customization of the Gnome JHBuild process.


### Build Process Based on build.sh Script

This build process simply runs a `build.sh` script that downloads the source code from the git repositories and builds each package in dependency order.

The _source tree_ is where the source code is extracted from git. It would typically be something like $HOME/src. The _installation tree_ is where the build generated objects (binaries, scripts, etc...) are created, typically $HOME/build.

The script is itself part of a package which needs to be obtained from a git repository. The following command will extract the source code from git:
[[!format txt """
cd $HOME/src
git clone git://anongit.freedesktop.org/git/xorg/util/modular util/modular
"""]]
You are now ready to launch a full build by invoking the build script. <a name="FullBuild"></a>
[[!format txt """
mkdir $HOME/build
cd $HOME/src
./util/modular/build.sh --clone $HOME/build
"""]]
There are a good number of environment variables to be set and directories to be created. The script does a good job of setting default values. The `--help` option will list those variables and what they are used for. A common one is PREFIX which you may set to the installation directory. There is no variable for the source tree, you invoke commands from the top of the tree.

Take a look at the features available in that script. You will certainly appreciate the `--autoresume` when the build stops on error. If you do not need to build all the packages, you can customize a list using the `-L` and use it with the `--modfile` options.


#### Building from Archive Files

Alternatively, the script can build the source from archive files. X.Org regularly publishes versions of the source code in the form of [[tarballs|http://www.x.org/releases/X11R7.6/src/]]. The example shown involves tarballs from the X11R7.6 release. The script can also build a mixture of tarballs and source from git.

You must first download the tarballs before running the build script. An input file containing URLs to read can be provided. You do not need to extract the tarballs. A sample command:
[[!format txt """
wget http://www.x.org/releases/X11R7.6/src/util/util-macros-1.11.0.tar.gz
"""]]
You are now ready to launch a full build by invoking the build script. You can leave the `--clone` option in, any missing source will be obtained from git, at the current development level.
[[!format txt """
mkdir $HOME/build
cd $HOME/src
./util/modular/build.sh $HOME/build
"""]]

### Build Process Based on JHBuild

The second build process uses [[jhbuild|http://www.freedesktop.org/wiki/Software/jhbuild]], a program written by James Henstridge that can be used to automatically download module components from Git and then build them in the correct dependency order. Refrer to the [[JhBuildInstructions|JhBuildInstructions]].


## Features of the Build Script

The `build script` first sets up environment variables and creates missing directories. It then cycles through a list of packages in dependency order to download them and run the Autotools from the  [[GNU Build System|ModularDevelopersGuide]]. Some of the features require a conceptual understanding of Autoconf and Automake. Use the `--help` for a complete list of options.


### Environment Variables

You normally do not need to have any environment variable set to run a full build. The script assigns reasonable defaults. The `--help` option will list those variables and what they are used for.


### Downloading and Updating Source Code

As seen when launching a [[full build|ModularDevelopersGuide]], the `--clone` option will download the source code from git if it is missing on disk. The `-p` option will issue a `git -pull --rebase` to obtain the latest code currently in git.


### Stop/Continue/Resume On Error

All builds eventually stop due to errors, sooner rather than later. The `-n` option allows the build script to continue the build with the next module. When the script terminates, it reports which ones failed. The `--autoresume` option writes a list of modules and can restart where it left off after you fixed the build error.


### Specifying Modules to Build

The build script contains a full list of all X.Org modules and its dependent projects. That's a lot, over 200. You will most likely not need all of them. The `-L` will write this list to a file which you can edit and feed back to script using `--modfile` option. Make sure you understand the dependencies so as not to remove critical modules.


### Making Tarballs

The X.Org modules are published as tarballs on the web. To create such tarballs, one can simply specify the `-D` option, but preferably `-d` as it performs a number of tests while creating the tarballs.


### Debug Mode

The Automake default CFLAGS are -g and -O2. Using the -g option will add -g3 -O0 for additional debugging.

The Automake silent rules are enabled by default at version 1.11 and higher. You may need additional context to debug the error. The `--retry-v1` will cause a rebuild of the module with the silent rules disabled.


### Advanced Commands and Configuration

The build script provides additional flexibility in letting you specify arbitrary commands. You need more internal knowledge of the script and good experience with configuring and building modules. There is no checking to ensure that what you are trying to do _makes sense_.

The `-a` option will skip the Autoconf configuration. This speeds up the build when you know that all modules have already been configured and do not need their source to be updated. You can rely, up to a point, on the make build rules to reconfigure the module if it is out of date.

The `--cmd` option allows you to invoke a custom make or git command. You may want to run `git status` on all modules to ensure there are no local code changes for example.

The `--confflags` option allows you to supply any Autoconf `configure` option. One example that comes to mind is `--disable-docs` which will suppress the building of user documention. These may conflict will default values assigned to environment variables set by the script.


### Building Individual Modules Yourself

The build script does not mind if you cd to an individual module and build it yourself. It will have no adverse affect on the build when you resume the use of the script. You just need some basic understanding of the Autotools. Beware that the build script sets environment variables for you which are not available when you type the commands yourself in the terminal.

Let's set those variables the way the script does it for you:


[[!format txt """
export PREFIX=$HOME/build
export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig:$PREFIX/share/pkgconfig
export ACLOCAL="aclocal -I $PREFIX/share/aclocal"
export LD_LIBRARY_PATH=$PREFIX/lib
export PATH=$PREFIX/bin:$PATH
"""]]
Now that have exported those variables, the build script will use them in lieu of its internal defaults. You can now switch back and forth between the build script and the terminal and keep the same environment.

The script also creates directories for you:
[[!format txt """
mkdir -p $PREFIX/share/aclocal
mkdir -p $PREFIX/var/log
"""]]
Download the font/util module and its dependency util/macros from git.
[[!format txt """
cd $HOME/src
git clone git://anongit.freedesktop.org/git/xorg/util/macros util/macros
git clone git://anongit.freedesktop.org/git/xorg/font/util font/util
"""]]
Run the `autogen.sh` script, which will create and run the Autoconf configuration, and invoke the make command.
[[!format txt """
cd $HOME/src/util/macros
./autogen.sh --prefix $PREFIX && make install
cd $HOME/src/font/util
./autogen.sh --prefix $PREFIX && make install
"""]]
This is exactly what the script does for you, note the order and the importance of running the _install_ target. Without it, the font util module would not be able to use xorg-macros.m4 in $PREFIX/share/aclocal. For complete details on the packaging structure of a module read the INSTALL file in the root directory of the module.

When building from tarballs, the configuration has already been created and only needs to be run. The commands would be:
[[!format txt """
cd $HOME/src/util/macros
./configure --prefix $PREFIX && make install
cd $HOME/src/font/util
./configure --prefix $PREFIX && make install
"""]]
The autogen.sh script is not part of the GNU Build System architecture and is kept for historical reasons. Other projects have different name and content for it. It should not be included in the tarball. If you need to recreate the configuration, `autoreconf -vfi` will do and is only what autogen.sh calls.


## Using the Build

This section describes how to use the newly compiled X libraries, applications and X server.


### X libraries

The X libraries which are built will typically be in $PREFIX/lib.

To use them you have to configure the linker to find them.  On GNU/Linux systems you do:


[[!format txt """
export LD_LIBRARY_PATH=$PREFIX/lib
"""]]
Alternately, you might edit the `/etc/ld.so.conf` file and run `ldconfig`. There may be other ways to configure the linker on other systems.

Once this is done X applications should use the newly compiled libraries.


### X applications

The X client programs which are built will be found in $PREFIX/bin.  Generally, these programs can be run with the default user permissions.


### Running the X server

Verify that the Xorg binary that will get executed is really the one you just build. If not, adjust $PATH.
[[!format txt """
ls -l `which Xorg`
"""]]
The Xorg binary needs to be _suid root_ which requires root privilege.
[[!format txt """
chown root Xorg
chmod 4711 Xorg
"""]]
Run the new X server using the most basic command. Your computer screen will blank out and your desktop will be replaced with a small X terminal window with no borders. To return to your desktop, you can type `exit` or you can press `Ctrl Alt F7` (on most platforms).

**Please read the above paragraph carefully before issuing the following command.**
[[!format txt """
Xorg :1 -terminate & sleep 2 ; DISPLAY=:1 xterm
"""]]

## Working with git submodules

Submodules allow foreign repositories to be embedded within a dedicated subdirectory of the source tree, always pointed at a particular commit. This section assumes you are comfortable with the similar tasks when performed without git submodules. You might be interested in the [[Kernel Git Submodule Tutorial|https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial]].


### Cloning a module with submodules

After cloning the module, the git submodule command will clone and initialize the submodules specified in the .gitmodules file. In this example, the submodule path is _m4_. Git keeps track of which submodule commit to checkout. Later versions of the _git clone_ command have a _--recursive_ option.
[[!format txt """
git clone git://anongit.freedesktop.org/xcb/util
git submodule update --init
"""]]

### Updating a module

It may be that only the module has changed since the last update, or that both the module and the submodule have changed.
[[!format txt """
git pull --rebase
git submodule update
"""]]
It may be that only the submodule has changed and that these changes should be picked-up by the module.
[[!format txt """
cd m4
git pull origin master
git status
cd ..
git commit -a -s
"""]]

### Rolling back a module

It may happen you need to revert the state of a module and a submodule to a previous point in time. The module should then point to the matching commit of the submodule. The commit numbers are valid, you can run the commands.
[[!format txt """
git reset --hard 02289ca
git submodule update
"""]]
You should see this output:
[[!format txt """
HEAD is now at 02289ca Bump version to 0.3.8
Submodule path 'm4': checked out '55e8069773efd794a91d5fb37bfceeebae2e378a'
"""]]
The log [[http://cgit.freedesktop.org/xcb/util/log/m4|http://cgit.freedesktop.org/xcb/util/log/m4]] will show the commit history for the submodule.


### Hooking up a submodule to a module

A submodule has a git repository like any other module. It is only different in the way it gets used. Several modules can _include_ the submodule containing reusable code.
[[!format txt """
git submodule add git://anongit.freedesktop.org/xcb/util-common-m4 m4
git submodule update --init
"""]]
You should see this output from _git status_:
[[!format txt """
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       new file:   .gitmodules
#       new file:   m4
#
"""]]

### Detaching a submodule from a module

There is no git submodule remove command. Given our example where the submodule path is _m4_,
[[!format txt """
git config --remove-section submodule.m4
git config --file .gitmodules --remove-section submodule.m4
git rm --cached m4
rm -fr m4
"""]]
Create a commit for the changes incurred by the module and you are done. Note that the submodule still exists at fdo.org, you have only detached and deleted your module copy.


### Applying changes to a submodule


#### Working from within a submodule

To directly modify sources referenced by a submodule, just change to the directory of the submodule and work with it like with a normal repository.  Note however, that the state of a submodule is controlled by the supermodule and that  under certain conditions it might even be possible to lose unpublished work. For more, see the Gotchas section of the [[Kernel Git Submodule Tutorial|https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial#Gotchas]].


#### Using a local clone of a submodule

It is also possible to separately clone the repository and redirect the upstream URL of the submodule to the local version. For example, the configuration of a submodule m4 in ` m4/.git/config ` could look like:


[[!format txt """
[remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*
        url = git://anongit.freedesktop.org/xcb/util-common-m4.git
"""]]
Simply set the value of `url` to the path of the local repository.


[[!format txt """
[remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*
        url = /path/to/util-common-m4.git
"""]]
Note, that we are changing the configuration of the submodule at ` m4/.git/config ` and not the configuration of the supermodule at `.git/config`.

 To use a different location from the start, it is also possible to change the submodule configuration in the supermodule before updating the submodule for the first time.  The file `.gitmodules` specifies the official location of the submodule and changes to it are being tracked.  But that file does not need to be changed for our purpose.  The following command copies the contents of `.gitmodules` to the configuration of the supermodule at `.git/config`:


[[!format txt """
git submodule init
"""]]
The submodule has not been checked out yet, and so it is possible to select a different location.  For example, the configuration in `.git/config` could look like:


[[!format txt """
[submodule "m4"]
        url = git://anongit.freedesktop.org/xcb/util-common-m4.git
"""]]
Insert the preferred location like above and then update the submodule with:


[[!format txt """
git submodule update
"""]]

### Creating a submodule

A git submodule is created just like any other git module. The reasons for using a module as a submodule is beyond the scope of this section. In the XCB example, the goal was code reuse.


## Crosscompiling

There is a separate page dedicated to issues around [[CrossCompilingXorg|CrossCompilingXorg]].


## New Modules

When adding modules to the tree, you may want to read the [[NewModuleGuidelines|NewModuleGuidelines]].