diff options
| author | luporl <luporl@FreeBSD.org> | 2020-12-09 11:11:15 -0300 |
|---|---|---|
| committer | Povilas Kanapickas <povilas@radix.lt> | 2021-04-16 14:40:49 +0000 |
| commit | 7e142cb2a848acb6af986fa91d254d4c23963b24 (patch) | |
| tree | 813e91dc5fb8a9f39e2d1732e3f1fc1843e04602 /render | |
| parent | 5be3b80b8d084ca5721be8791910d5827d1b6014 (diff) | |
xserver: fix RGB mask handling
On FreeBSD 13.0-CURRENT for PowerPC64 big-endian (BE), X was
crashing in some cases. For instance, when twm was started
and the background was clicked to open its menu, X crashed
with a segmentation fault, trying to dereference a null pointer
at CreatePicture().
There were 2 issues with xorg-server handling of RGB masks that
caused the pointer above to be null and thus the crash:
- wrong use of ffs() to get the RGB offsets from the masks
- overflow when shifting a 16-bit integer
This change fixes both issues. They happen when the system is BE
but has a video adapter using a little-endian (LE) ARGB32
framebuffer. In order to display the correct colors, this setup
requires a BE RGBA32 color format to be used by X, by setting
the RGB masks appropriately, that didn't work properly because of
the issues above.
Diffstat (limited to 'render')
| -rw-r--r-- | render/picture.c | 12 |
1 files changed, 6 insertions, 6 deletions
diff --git a/render/picture.c b/render/picture.c index 876316dc1..afa0d258f 100644 --- a/render/picture.c +++ b/render/picture.c @@ -523,12 +523,12 @@ PictureMatchVisual(ScreenPtr pScreen, int depth, VisualPtr pVisual) return format; } else { - if (format->direct.redMask << format->direct.red == - pVisual->redMask && - format->direct.greenMask << format->direct.green == - pVisual->greenMask && - format->direct.blueMask << format->direct.blue == - pVisual->blueMask) { + if ((unsigned long)format->direct.redMask << + format->direct.red == pVisual->redMask && + (unsigned long)format->direct.greenMask << + format->direct.green == pVisual->greenMask && + (unsigned long)format->direct.blueMask << + format->direct.blue == pVisual->blueMask) { return format; } } |