Working with rsyslog in Solaris 11

In this exercise you will:

• Install rsyslog and setup rsyslog to log messages to individual log files. 
• You’ll setup a central rsyslog server and configure an rsyslog client to send messages to the central server.
• You’ll also learn to customize the message format using templates.

Setup:
Perform this exercise on any system that has access to a pkg repository so you can install the rsyslog package.
For this example, I’ve setup two virtual solaris systems in VirtualBox.
A central log server: hostname logserver IP 10.100.0.189
A remote client: hostname solaris IP 10.100.0.191

I. Perform these steps on the host named “logserver”

 

1.  First set the hostname to logserver and type:

# hostname logserver

If using DHCP,  make an entry in /etc/nodename to make the hostname persistent as follows:

logserver

2. Check for the rsyslog pkg:

# pkg list rsyslog
pkg list: No packages matching 'rsyslog' installed

3. Make sure you have access to a package repository

# pkg publisher
PUBLISHER  TYPE     STATUS   P  LOCATION
solaris    origin   online   F  file:///IPS/11.4/

4. If the package is not installed, install it as follows:

# pkg install rsyslog
Packages to install: 4
Services to change: 1
Create boot environment: No
Create backup boot environment: No

Planning linked: 0/1 done; 1 working: zone:zonea
Linked image 'zone:zonea' output:
`
Planning linked: 1/1 done
DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 4/4 309/309 2.9/2.9 0B/s

Downloading linked: 0/1 done; 1 working: zone:zonea
Downloading linked: 1/1 done
PHASE ITEMS
Installing new actions 426/426
Updating package state database Done
Updating package cache 0/0
Updating image state Done
Creating fast lookup database Done
Executing linked: 0/1 done; 1 working: zone:zonea
Executing linked: 1/1 done
Updating package cache 1/1

4. Disable the syslog daemon:

# svcadm disable svc:/system/system-log:default

5. Make a copy of the /etc/rsyslog.conf file

# cp /etc/rsyslog.conf /etc/rsyslog.conf.ORG

6. Use the vi editor to edit the /etc/rsyslog.conf file.   Locate this section of the file (approx. line 24):

# Log all the auth, daemon & mail messages in one place.
auth.* /var/log/auth.log
daemon.* /var/log/daemon.log
mail.* /var/log/mail.log

7. Make these modifications (in red) to the file:

# Log all the auth, daemon & mail messages in one place.
auth.* /var/log/auth.log
daemon.* /var/log/daemon.log
mail.* /var/log/mail.log
mail.info /var/log/mail.info
mail.warn /var/log/mail.warn
mail.err /var/log/mail.err
daemon.* /var/log/daemon.log
kern.* /var/log/kern.log
user.* /var/log/user.log

8. Locate this section of the /etc/rsyslog.conf file (at the end of the file) and uncomment the lines highlighted in red:

# ######### Receiving Messages from Remote Hosts ##########
# TCP Syslog Server:
# provides TCP syslog reception and GSS-API
$ModLoad imtcp.so       # load module
$InputTCPServerRun 514  # start up TCP listener at port 514
 
# UDP Syslog Server:
$ModLoad imudp.so       # provides UDP syslog reception
$UDPServerAddress *     # listen to all IP addresses
$UDPServerRun 514       # start a UDP syslog server at standard port 514

9. Enable the rsyslog service as follows:

# svcadm enable svc:/system/system-log:rsyslog

10. Reboot the system and notice the new system messages on the console during the boot process:

# init 6 

11. Login and change to the /var/log directory and view the new log files:

auth.log 
kern.log 
misc.log 
daemon.log 
mail.info 
mail.log 
user.log 

12. From a remote system, ssh into the server named logserver and check the /var/log/auth.log file for an entry like this:

Aug 28 07:21:10 solaris sshd[1208]: [ID 800047 auth.info] Accepted keyboard-interactive/pam for root from 10.10.0.5 port 62192 ssh2

II. Setup Remote Logging to a Central Host

Perform these steps on a remote client, for this example, my remote client is named “solaris.”  You’ll configure rsyslog on this client to send messages to a remote server named “logserver”

13. Make sure rsyslog is installed on this client, the rsyslog service is enabled and syslog is disabled as described in steps 1-5

14. Use the vi editor to edit the /etc/rsyslog.conf file.
Locate this section of the file (approx. lines 37-47) and make the changes I’ve highlighted in red:

# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName uniqName # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @@logserver:514

Note: The above section instructs the rsyslog daemon to send all log messages, regardless of the facility or severity, to the host named logserver (IP address can also be used) via 514/TCP port.
The logserver is prefixed with either @@ or @. @@=TCP, @=UDP
If you only wanted to send specific log messages, for instance, assuming you want to send only a specific facility message to a remote log server, such as all related mail messages regardless of the priority level, add the below line to rsyslog configuration file:
mail.* @logserver:514

15. Make an entry in the /etc/hosts file for logserver

     logserver

16. Restart the rsyslog service as follows:

# svcadm restart svc:/system/system-log:rsyslog

Watch the rsyslog activity on the logserver

Use either snoop or wireshark (tshark) to watch the network activity for port 514 (rsyslog messages). For this example we’ll use snoop

17. Start the snoop utility on the logserver as follows:

# snoop port 514 
Using device net0 (promiscuous mode)

As messages get sent to the logserver from the client, you’ll see activity similar to this:

10.100.0.189 -> 10.100.0.191 RSHELL C port=33277 <39>Aug 28 07:59:18
10.100.0.191 -> 10.100.0.189 RSHELL R port=33277

You can also use tshark (terminal version of wireshark) if it’s installed as follows:

# tshark -f "tcp port 514"
Capturing on 'net0'
1 0.000000 10.100.0.189 ? 10.100.0.191 RSH 316 Client -> Server data
2 0.068643 10.100.0.191 ? 10.100.0.189 TCP 66 514 ? 61615 [ACK] Seq=1 Ack=251 Win=64074 Len=0 TSval=334322 TSecr=260944
3 6.841524 10.100.0.189 ? 10.100.0.191 RSH 205 Client -> Server data
4 6.898634 10.100.0.191 ? 10.100.0.189 TCP 66 514 ? 61615 [ACK] Seq=1 Ack=390 Win=64074 Len=0 TSval=335005 TSecr=261628

18. Reboot the client and watch the /var/log/daemon.log file on the logserver and notice that all of the boot messages are getting logged in that file. Each entry has the client’s hostname as follows:

Aug 28 07:54:30 solaris devchassisd[863]: [ID 752255 daemon.info] devchassis daemon started.

19. When the client reboots, you’ll see several messages in the snoop session as boot messages get transferred to the logserver.

Using Custom Templates with rsyslog

The default message template in Solaris is specified by the following entry located near the top of the /etc/rsyslog.conf file:

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

The above string will generate a message that is similar to the syslogd style message which looks like this:

Aug 28 08:45:43 solaris sshd[1159]: [ID 800047 auth.info] Accepted keyboard-interactive/pam for root from 10.100.0.191 port 45171 ssh2

 

Specify a custom message format by using the $template statement in the rsyslog.conf file as follows:

$template precise,"%syslogseverity-text%,%syslogfacility-text%,%timegenerated%,%HOSTNAME%,%syslogtag%,%msg%\n"

Apply this template to a message as follows:

auth.* /var/log/auth.log;precise

This will generate a message that looks like this:

info,auth,Aug 28 10:14:37,solaris,sshd[1408]:, [ID 800047 auth.info] Accepted keyboard-interactive/pam for root from 10.10.0.5 port 64069 ssh2

All of the properties available for customizing your template can be found here:
https://www.rsyslog.com/doc/v7-stable/configuration/properties.html