[howto] grsecurity + NOUVEAU + Compiz + Seg Fault

Assuming that you have a grsec + PaX enabled kernel, you would realise that the nvidia-drivers are a bad choice. Quite a few applications will fail(the ones that use libGLcore.so). Use the nouveau driver for your card, as it’s pretty much stable and works with good 3D acceleration.

compiz under NOUVEAU + PaX

To get compiz working NOUVEAU under hardened linux, first enable the kernel DRM module for nouveau. Follow this link: The X Server Configuration HOWTO

Build the kernel, and install it. Edit the VIDEO_CARDS variable in your make.conf to say only nouveau, nothing more, nothing less.

Unmask the following packages: media-libs/mesa, x11-libs/pixman, x11-drivers/xf86-video-nouveau, x11-base/xorg-drivers, x11-base/xorg-server, x11-libs/libdrm, x11-drivers/xf86-input-evdev, x11-drivers/xf86-input-keyboard, x11-drivers/xf86-input-mouse

Install the above packages, make sure you’ve done a emerge -C nvidia-drivers nvidia-settings prior to the merge.

Reboot the system, it should all work out of the box, compiz will fail with a segmentation fault, look into your logs. You’ll see something like the following:

2011-05-11T17:22:24.760922+05:08 halcyon-82 kernel: [ 2026.893377] grsec: denied
 resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/b
in/compiz[compiz:20146] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[
bash:11847] uid/euid:1000/1000 gid/egid:1000/1000

2011-05-11T17:26:07.848848+05:08 halcyon-82 kernel: [ 2249.981362] compiz[20378]
: segfault at ffffffffffffffff ip 00000284c5f39fa1 sp 0000039c50e0ee00 error 6 i
n nouveau_dri.so[284c5cc3000+38b000]

Simply disable pax for compiz and emerald, do the following as root:

# paxctl -zm /usr/bin/compiz
# paxctl -zm /usr/bin/emerald

Now, start compiz as usual and your all set.
On a side note, flash player will show a similar issue too, so disable PaX for that too.

WARNING: Disabling PaX for compiz, emerald and flash is a security risk.