Azure Sentinel and Rsyslog – When things went bad with Syslog-ng
You probably see the title above and might think to yourself, what can go wrong with the Syslog-ng mechanism. When you forward logs to Azure Sentinel, the data connector configuration with the Azure Sentinel agent and the syslog-ng settings are straightforward. Still, can be many issues, mostly from the syslog-ng side.
Things can go wrong when a dedicated security system cannot work seamlessly with syslog-ng and cannot forward the logs as you wish to Azure Sentinel. In this scenario, the rsyslog can be a better solution and do the job like a star.
I recently came across a specific case with a complex scenario with a demand to connect an old WAF system to Azure Sentinel. The old WAF system sends encoded logs with no facility, the structure is unique, no option to work with CEF, and the data from the WAF is sent to the syslog server with a specific file. Of course, the requirement was to work with TLS between the old WAF system and the syslog server.
This blog post focused on how to connect Rsyslog to Azure Sentinel and includes many tips.
The Good, the Bad, and the Ugly
The good, the bad, and the ugly – when Rsyslog is your last hope.
Someone once said there is no bad technology. You only need to adjust the technology to the situation. When it comes to a Syslog server on Linux, the preferred engine is the syslog-ng, but sometimes there are specific scenarios when you need other engines to do the job, so what we’ve got with the syslog engine? Basically, they are all the same in how they all permit data logging from different types of systems in a central repository.
There are three projects, each project trying to improve the previous one with more reliability and functionalities.
Syslog – The Syslog project was the very first project. It started in 1980. It is the root project of the Syslog protocol. At this time, Syslog is a straightforward protocol. In the beginning, it only supports UDP for transport, so that it does not guarantee the delivery of the messages.
The daemon, also known as sysklogd and is the default LM in standard Linux distributions. The Syslog isn’t flexible, and you can redirect log flux sorted by facility and severity to files and over the network-based (TCP, UDP).
Syslog-ng was the next one that came in 1998. It extends basic Syslog protocol with new features like:
- Content-based filtering
- Logging directly into a database
- TCP for transport
- TLS encryption
The Syslog-ng is the “Next-Gen.” engine, and it’s the best way to manage logs. For example, everything is an object like source, destination, filter, and the very forwarding rule, and the syntax is straightforward.
Rsyslog came in 2004, and it extends the Syslog protocol with new features. The Rsyslog is an “advanced” version of sysklogd where the config file remains the same, but you have a lot of new cool stuff coming, such as:
- Listen to TCP/UDP connections, with restrictions (ports, Source IPs)
- Load a lot of modules.
- Discriminate the log filtering by program, source, message, PID, etc. (for instance, each message tagged with the message “connexion closed” to the file closed.log)
- Discard messages after one or more rules.
- RELP Protocol support
- Buffered operation support
TIP: you can copy a Syslog. conf file directly into rsyslog.conf, and it works
Let’s say that today they are three concurrent projects that have grown separately upon versions but also developed in parallel regarding what the neighbors were doing. From my perspective, Syslog-ng is the reference in most cases. It’s the most mature project offering the main features you may need and an easy and comprehensive setup and configuration in many scenarios.
Install and Configure Rsyslog
For the Syslog server, I used Ubuntu 18.04 or 20.04. Hence make sure to install Ubuntu and perform some updates. The Rsyslog does not come by default in Ubuntu, and if you check for installed services, it will be inactive mode.
In case it is not installed, you need to install Rsyslog and make sure it run correctly. Use the command below:
apt install rsyslog // install Rsyslog engine
service start rsyslog // Check Rsyslog status
Configure RSyslog to receive remote messages
First, we need to enable the socket on which Rsyslog is listening to receive remote messages. By default Rsyslog only logs from the local system.
To configure Rsylsog to listen and to receive remote messages, you need to edit the following file: /etc/rsyslog.conf
In the MODULES section, you will see two sockets, an UDP and TCP socket. You need to enable the default UDP and Port 514
module(load=”imuxsock”) # provides support for local system logging
module(load=”imklog” permitnonkernelfacility=”on”)
After editing the rsyslog.conf file, we first need to restart the service to check if the server is listening on UDP Port 514 for remote messages.
# check if the server listening on UDP and TCP Port 514
# ss displays socket statistics and replaces the deprecated netstat command in Linux.
ss -a -4 or ss -anu or netstat -anu netstat -anu | grep 514
To restrict access to the log server, we can define senders that will send messages. Therefore we can create AllowedSender directives in the GLOBAL DIRECTIVES section from rsyslog.conf as follows
$AllowedSender <type>, ip[/bits], ip[/bits]
$AllowedSender UDP, 127.0.0.1, 192.0.2.0/24, [::1]/128, *.example.net, somehost.example.com
$AllowedSender TCP, 127.0.0.1, 192.0.2.0/24, [::1]/128, *.example.net, somehost.example.com
Rsyslog also detects some malicious reverse DNS entries. In any case, using DNS names adds an extra layer of vulnerability. We recommend sticking with hard-coded IP addresses wherever possible.
Finally, we need to add a rsyslog template to store messages from a remote server to a specific file and path.
To create a new template, I will add a further section named Templates to the rsyslog.conf file with my custom template RemoteLogs.
This will create a folder named with the IP from the remote host sending the messages, and in this folder, a file named sbc.log will be created with the actual remote messages from the remote host.
#Custom template
$template RemoteLogs, “/var/log/remote/%FROMHOST-IP%/sbc.log”
. ?RemoteLogs
$ModLoad imfile
$InputFileName /var/log/log.log ## the log file location
$InputFileTag Elli
$InputFileSeverity error
$InputFileSeverity info
$InputFileSeverity warning
$InputFileSeverity error
$InputFileFacility local4
$InputFileFacility syslog
$InputRunFileMonitor
# Set the default permissions for all log files #
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
Azure-Sentinel-4-SecOps/rsyslog.conf at master · eshlomo1/Azure-Sentinel-4-SecOps (github.com)
The Azure Sentinel Side
You can stream events from Linux-based, Syslog-supporting machines or appliances into Azure Sentinel, using the Log Analytics agent for Linux (formerly known as the OMS agent). You can do this for any machine that allows you to install the Log Analytics agent directly on the machine. The machine’s native Syslog daemon will collect local events of the specified types, and forward them locally to the agent, which will stream them to your Log Analytics workspace.
Syslog is an event logging protocol that is common to Linux. When the Log Analytics agent for Linux is installed on your VM or appliance, the installation routine configures the local Syslog daemon to forward messages to the agent on TCP port 25224.
When it comes to the Azure Sentinel, the process is simple, and we need to install the agent whether it is on Azure VM or in other places. From Azure, Sentinel data connector go to Syslog and from there install the relevant agent
TIP: On a Syslog server, it is unnecessary to add the oms facility to the conf files because the rsyslog file pulls the specific file and sends it to Azure.
Once the Azure Sentinel syslog agent installs on the rsyslog server, we configure the facility. In this example, I used the Syslog facility with all settings.
TIP: you don’t need to configure an additional configuration file like cef, because syslog works differently from cef. But, you can custom the deamon and facilities from the rsyslog default configuration.
Connect Syslog data to Azure Sentinel | Microsoft Docs
Syslog and Severities
What are Syslog facilities?
The facility value indicates which machine process created the message. The Syslog protocol was originally written on BSD Unix, so Facilities reflect the names of UNIX processes and daemons.
| Number | Source | Number | Source |
|---|---|---|---|
| 0 | kernel messages | 12 | NTP subsystem |
| 1 | user-level messages | 13 | log audit |
| 2 | mail system | 14 | log alert |
| 3 | system daemons | 15 | clock daemon |
| 4 | security/authorization messages | 16 | local use 0 (local0) |
| 5 | messages generated internally by Syslog | 17 | local use 1 (local1) |
| 6 | line printer subsystem | 18 | local use 2 (local2) |
| 7 | network news subsystem | 19 | local use 2 (local3) |
| 8 | UUCP subsystem | 20 | local use 2 (local4) |
| 9 | clock daemon | 21 | local use 2 (local5) |
| 10 | security/authorization messages | 22 | local use 2 (local6) |
| 11 | FTP daemon | 23 | local use 2 (local7) |
Syslog severities
The following table provides a list of Syslog severity levels with descriptions and suggested actions for each.
| Number | Severity | Suggested Actions |
|---|---|---|
| 0 | Emergency | A “panic” condition affecting multiple applications, servers, or sites. The system is unusable. Notify all technical staff on call. |
| 1 | Alert | A condition requiring immediate correction, for example, the loss of a backup ISP connection. Notify staff who can fix the problem. |
| 2 | Critical | A condition requiring immediate correction or indicating a failure in a primary system, for example, a loss of a primary ISP connection. Fix CRITICAL issues before ALERT-level problems. |
| 3 | Error | Non-urgent failures. Notify developers or administrators as errors must be resolved within a given time. |
| 4 | Warning | Warning messages are not errors, but they indicate that an error will occur if the required action is not taken. An example is a file system that is 85% full. Each item must be resolved within a given time. |
| 5 | Notice | Events that are unusual but are not error conditions. These items might be summarized in an email to developers or administrators to spot potential problems. No immediate action is required. |
| 6 | Informational | Normal operational messages. These may be harvested for network maintenance functions like reporting and throughput measurement. No action is required. |
| 7 | Debug | Information is useful to developers for debugging an application. This information is not useful during operations. |
Outcome
The Azure Sentinel Syslog deployment is effortless and straightforward, and most other setting is on the syslog server side. In specific cases, you need to configure the rsyslog.conf file to allow data forwarding to Azure Sentinel. I can say that most Syslog cases are simple, but as you know, sometimes the floor is crooked, and we have to adjust to it.
