Troubleshooting SELinux – Part 1

Harderning with SELinux

I finally got around to hardening my web server over the holiday break. A very important step in hardening linux is ensuring selinux is both enabled and enforcing. Of course the risk is that Security Enhanced Linux starts blocking things that should not be blocked…

Undeserved Reputation?

I have been reluctant to have SELinux enforcing on all my instances of linux for a long time. It turns out that nowadays, with the maturity of the feature and its associated tools, this is pretty much an indefensible position.

Sure, back in the mid-noughties SELinux had its issues and caused much grief for sysadmins, but things have changed… a lot. This post documents me catching up, and will hopefully be useful to anyone trying to do the same!

Why SEliNux?

SELinux is the most essential security tool available on linux. Period. No other single tool succeeds succeeds so well at securing a system from the inside out.

It originated from a 3-letter agency with a special interest in security. You can read more about it at wikipedia.


I used Centos 7 for this post. Most of my machines use Centos without a GUI installed. The tools should be identical on Red Hat (RHEL) and very similar on any modern linux distro. Of course the system commands may differ on your platform.

For the rest of the post I’ll assume you are using Centos 7.

[sysadmin@imago ~]$ cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)

How SELinux Works

For those familiar with networking, SELinux works a little like a firewall, in that it controls activity between sources and targets based on a configured policy. That policy may include definitions of the source and target objects and rules about the types of activity allowed between them.

SELinux differs from a firewall in that it is interested in the activity within your system, not between systems on a network. And it does a very good job of this.

Further, source and target objects are not limited to hosts. They include a whole range of linux “things”, such as process, files, directories, devices and sockets. Basically, any element of a linux system.

SELinux has total visibiility and control of the activity between any of these objects. For example, when a process tries to read a file, or open a network socket.


While SELinux can is very flexible and get very complicated, the basics are actually quite simple.

Every object on the system is labelled with a type. For example if you were to look at the files in a directory with the “-Z” option, you will see the SELinux attributes associated with each object (file or directory), including the type.


Here is how my home directory looks:

[sysadmin@server ~]$ ls -aZ
drwx------. sysadmin sysadmin system_u:object_r:user_home_dir_t:s0 .
drwxr-xr-x. root root system_u:object_r:home_root_t:s0 ..
-rw-------. sysadmin sysadmin system_u:object_r:user_home_t:s0 .bash_history
-rw-r--r--. sysadmin sysadmin system_u:object_r:user_home_t:s0 .bash_logout
-rw-r--r--. sysadmin sysadmin system_u:object_r:user_home_t:s0 .bash_profile
-rw-r--r--. sysadmin sysadmin system_u:object_r:user_home_t:s0 .bashrc
-rw-------. sysadmin sysadmin system_u:object_r:user_home_t:s0 .lesshst
drwx------. sysadmin sysadmin system_u:object_r:ssh_home_t:s0 .ssh
-rw-rw-r--. sysadmin sysadmin system_u:object_r:user_home_t:s0 tmp1
-rw-rw-r--. sysadmin sysadmin system_u:object_r:user_home_t:s0 tmp2
-rw-------. sysadmin sysadmin unconfined_u:object_r:user_home_t:s0 .viminfo
-rw-r--r--. sysadmin sysadmin system_u:object_r:user_home_t:s0 .vimrc

The type here is denoted by “_t”. Most files here are type “user_home_t”.

The “_u” denotes user, and “_r” denote roles, but these related to SELinux’s role-based access system and can safely be ignored by most sysadmins.

Process Object Types

Processes also have a type, and simillarly, these can be shown with the “-Z” switch. Most of the standard utilities will use this switch:

[sysadmin@server ~]$ ps -aZ
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 2744 pts/0 00:00:00 sudo
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 2748 pts/0 00:00:00 su
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 2749 pts/0 00:00:00 bash
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 3648 pts/0 00:00:00 tail
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 3649 pts/0 00:00:00 grep
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 5524 pts/1 00:00:00 ps

Here, all the processes my user has initiated are of type “unconfined_t”.


Rules basically boil down to which type of sources can access which types of targets. That’s pretty much all you need to know to get started.

For example can a process of type “unconfined_t” write to a directory with the “user_home_dir_t” type? Can it read from the same directory? Or modify a particular file in that directory?

Remember, these rules are separate from the file permissions system on the linux filesystem, and selinux runs in series. It won’t alter or override those permissions.


The default policy is also sufficient to get started.

Instead of changing the policy, we will attempt to resolve any issues with the system just by changing the type of any objects we encounter in our logs.

Checking the Status of SELinux

SELinux has two states enabled or disabled.

The enabled state has two modes: permissive or enforcing.

In permissive mode SELinux is doing everything except denying any activity. Log messages will appear just as though SELinux was enforcing policy on your system.

It is highly recommended to start in permissive mode and check your logs before going to enforcing, which we will do below. Permissive mode seems to be the default mode on most linux distros.

The most useful commands for checkiong the status of SELinux are:


On Centos 7, these utilities can be found in the following packages which are part of the base installation:

[sysadmin@server ~]$ sudo yum -q whatprovides /sbin/sestatus
policycoreutils-2.5-29.el7.x86_64 : SELinux policy core utilities
Repo : @base

[sysadmin@server ~]$ sudo yum -q whatprovides /sbin/getenforce
libselinux-utils-2.5-14.1.el7.x86_64 : SELinux libselinux utilies
Repo : @base
SELinux Disabled

In disabled mode, SELinux is not loaded into the kernel, and no functionality is available.

[sysadmin@server ~]# sestatus
SELinux status:                 disabled

If your machine looks like above, then check the following file:

cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.

You will need to enable SELinux by changing “disabled” to “permissive” and then rebooting your machine.

sudo vim /etc/selinux/config

This should be a safe action. I performed this without issue on a cloud machine on which I have no console access. The machine took a little longer to boot the first time, but came back as expected. Of course, there’s no guarantees so take care!

SELinux ENabled

getenforce provides a simple one-line output, useful for confirming which enabled mode your system is in:

[sysadmin@server ~]$ getenforce

sestatus provides a more detailed output:

[sysadmin@server ~]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

The most useful information here is the mode (permissive) and the policy name (targeted).

From here we’ll assume you are in permissive mode.

How to find SELinux logs

On Centos, SELinux uses the linux audit daemon (service), known as auditd.

Auditd logs everything to /var/log/audit/audit.log which requires elevated privileges to read:

[sysadmin@server ~]$ cat /var/log/audit/audit.log
cat: /var/log/audit/audit.log: Permission denied

[sysadmin@server ~]$ sudo more /var/log/audit/audit.log
type=USER_AUTH msg=audit(1546124135.971:451246): pid=30238 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/sbin/sshd" hostname= addr= terminal=ssh res=failed'
type=USER_AUTH msg=audit(1546124138.373:451247): pid=30238 uid=0 auid=4294967295 ses=4294967295 msg='op=password acct="root" exe="/usr/sbin/sshd" hostname=? addr= terminal=ssh res=failed'

On the plus side, everything you need is in one place. On the negative side, this file can get quite large, contains a log of messages unrelated to SELinux, and the output is a little cryptic.

As a first step we can just filter down to the relevant logs, which will begin with “type=AVC”:

[sysadmin@server ~]$ sudo cat /var/log/audit/audit.log | fgrep "type=AVC" | more

On some of my systems there were no “AVC” entries in the audit file. The rest of this post will focus on a cloud-located server that had multiple entries – in fact, the server hosting this blog.

Later on, we’ll use other tools to interpret these entries and even make recommendations.

In Part 2…

In the next part we will start looking at some example logs, where SELinux would have denied some activity on my system, had it been enforcing.