logging — Kernun firewall logging facility
All native applications of the Kernun firewall use a special interface
to contact the syslog daemon, and a common format of messages. Several
attributes of the logging interface can be set in each application
configuration by the same means.
Information about packet filter (PF) events are read from the pflog(4) and pfsync(4) pseudodevices by the pf-control
daemon and logged in a similar way. The PF logging is configured also
by the same means; moreover, you can precisely configure which events
of PF will be logged by special log option of PF rules.
For detailed description of the logging configuration syntax, see
the Section 5 application manual page or log(5).
There are two logs created by Kernun: the stats log and the debug log.
The stats
log logs each event (request, session, email, etc.
— depending on the traffic type) as a single
record. The statistics is generated out of this log. The log is (by
default) stored in /var/log/kernun-stats.
Since there can be a limitation to the maximal length of a single row in the logging facility, a long record can be split into more rows. In that case, there is '\' at the end of the record to be continued, and '~' at the beginning of record that is the continuation of some record.
The statistics is generated from the stats log.
The format of the stats log records is described in their particular man pages: AKHP-888(6), DNSP-888(6), FTPP-888(6), HTTP-888(6), ICAP-888(6), IMAP-888(6), MMCG-888(6), MMCP-888(6), PFLG-888(6), POP3-888(6), SIPP-888(6), SMTP-888(6), SQLP-888(6), TCPP-888(6), UDPP-888(6)
The debug log
contains verbose information, and is intended for thorough
investigation the traffic or other troubleshooting. The debug log is
(by default) stored in
/var/log/kernun-debug.
Kernun log messages look like the following example:
Sep 8 7:40:22 fw ftp-in[2018]: FTPP-110-E Message text
Parts of the message:
It is a part of the message and is prepared by the application.
It can be either the name of the application, or any string set by the firewall administrator in application configuration.
If the application needs to distinguish among several “problems” solved by one process (e.g. several requests served by a UDP-based proxy), it uses also a numerical suffix to PID. The “.0” suffix means the “main program”, other suffices denote particular request tracks.
This identification can facilitate post-processing of the log file, as every message has its own identification and all of them have the same form: Kernun component code, Message number, Severity code.
See below.
Each library module or application has its own code. This code has either four letters, or three letters as a prefix and fourth one for detailed distinguishing of submodules. A list of the component codes follows:
ADFI - Adaptive Firewall - core
AFHP - Adaptive Firewall - honeypot module
AFLD - Adaptive Firewall - reload tool
AFWD - Adaptive Firewall - watchdog module
ARGS - argument handling
ASN1 - ASN.1 parser utilities
ATRM - ATR monitoring daemon
AUTx - authentication
H - general authentication library
R - RADIUS
CASE - configuration integrity check
CFGx - configuration
L - lists handling
P - parsing
R - reader utilities
CHSC - Character set conversion library
CIBR - Configuration definition reader
CKGB - CML Kernun Generation Base
CMLx - CML/KAT tools
I - command line interface
K - Kernun Admin Tool
M - CML main program
R - reading configuration
S - showing configuration
T - tree management
CWBP - Clear Web ByPass
CWCD - Clear Web automatic categorization daemon (cwcatd)
CWDx - Clear Web DataBase
B - database engine
X - configuration checks
DHCP - DHCP server configuration
DHDR - document header handling
DNSx - DNS-proxy
C - nameserver cache management
E - DNS proxy and resolver engine
I - configuration and post-config initialisation
P - proxy itself
R - resource records utilties
X - configuration checks
FTPx - FTP-proxy
P - FTP-proxy, main program
S - FTP-proxy, control connection operations
T - FTP-proxy, data connection operations
H - FTP-proxy, HTTP <-> FTP gateway
H225 - H.323-proxy, H.225 Parser
H245 - H.323-proxy, H.245 Parser
HTCT - HTTP cookie table
HTCW - Clear Web DataBase
HTTx - HTTP-proxy
A - HTTP authentication proxy
F - module mod-ftp-dir
H - HTTP header processing
P - HTTP-proxy main program
X - configuration checks
ICAx - ICAP server
P - ICAP server
B - ICAP server, ICAP BNF parser/printer
R - ICAP server, server request control module
S - ICAP server, main program
IFSC - pikemon, interface status checking
IMAP - IMAP4-proxy
IPCx - IPC facilities
L - locks
M - shared memory
IPSE - IPsec
KERN - Kernun general messages
KEYV - keyword-value handling
LDAP - LDAP authentication
LIBx - general library functions
A - ACL
I - IP utilities
P - process utilities
T - time utilities
U - general utilities
LICC - license checking
LIST - doubly linked lists
LSQL - SQLite library
LSTN - listening sockets management
MAVC - module mod-antivirus
MAVP - module mod-antivirus
MCHU - module mod-chunked
MEMM - memory managemenent
MENC - MIME encoding/decoding utilities
MGZI - module mod-gzip
MHTF - module mod-html-filter
MIME - MIME type utilities
MIMF - module mod-image-filter
MIMX - MIME features configuration checks
MLSN - module mod-listen
MMAT - module mod-match
MMCx - H.323 Multimedia Communication proxy
C - control protocol (H.225, H.245)
D - multimedia data flow
G - gatekeeper proxy
P - proxy itself
R - Registration and Admission Service
Y - RAS Yellow Pages
MMIM - module mod-mime-magic
MNIO - module mod-netio
MNUL - module mod-null
MODM - module management
MONI - runtime monitoring
MPWF - ICAP interface to Proventia Web Filter
MRDF - module mod-read-file
MRWD - module mod-rw-data
MSPA - module mod-antispam
MWRF - module mod-write-file
NATT - NAT utilities
NETx - network library functions
L - network I/O library
S - select handling library
NTIF - network interface and routing utilities
NTLM - NTLM authentication module
NTPC - NTP configuration resolver
OOBA - out-of-band authentication
OSSL - OpenSSL support
OVPN - OpenVPN
PFCD - Packet-filter - configuration daemon
PFLG - Packet-filter - logger
PIKE - PIKE monitoring daemon
PING - PING group monitoring library
POP3 - POP3-proxy
PRXY - proxy configuration support
RCSL - Revision Control System
RDST - get real destination of transparent connection
RGAI - resolver, getting address info
RGHD - resolver gethostbydns() function
RGHT - resolver gethostbyht() function
RSLx - resolver
C - resolver, name compressing
I - resolver, initialisation
M - resolver, DNS making query
N - resolver, low-level API
Q - resolver, DNS query formulation
V - resolver, Kernun top-level routines
RSND - resolver, DNS query sending
SDPB - SIP-proxy, SDP BNF parser
SDPC - SIP-proxy, data channels management
SIPx - SIP-proxy
B - SIP-proxy, SIP BNF parser
C - SIP-proxy, control channels management
M - SIP-proxy, SIP messages management
P - SIP-proxy, main program
R - SIP-proxy, SIP requests management
S - SIP-proxy, SIP sessions management
Y - SIP-proxy, SIP YP map management
SLOG - system logging itself
SMTx - SMTP-proxy and mail processing proxies
B - mailing proxies, BNF parser
C - mailing proxies, configuration
D - mailing proxies, mail document module
I - SMTP-proxy, initialization
N - SMTP-proxy, DSN creation module
P - SMTP-proxy, main program
R - SMTP-proxy, client-side (reader)
S - SMTP-proxy, server-side (sender)
T - SMTP-proxy, tools
V - SMTP-proxy, client verification
X - mailing proxies, configuration checks
SQLx - SQL*Net proxy
P - proxy itself
S - TNS session layer
T - SQL RPC transport layer
TCPC - TCP client
TCPP - TCP-proxy
TCPS - TCP server
TCPX - TCP-proxy, configuration checks
TEST - configuration tester
UDPP - UDP-proxy
UDPS - UDP server
USBA - script for auto-configuration from an USB device (usb-setup.pl)
URIP - URI parsing and printing
Each Kernun message has assigned a severity code, expressed as the last part of identification - a single letter. Every Kernun severity code corresponds to one syslog severity level and has assigned a numeric value:
X - 0,LOG_EMERG - system is unusable
A - 1,LOG_ALERT - potential security problem detected
C - 2,LOG_CRIT - critical error, application fails
E - 3,LOG_ERR - error, current connection fails
W - 4,LOG_WARNING - potential error
N - 5,LOG_NOTICE - normal but noticeable condition
K - 5,LOG_KERNUN - Kernun message
(non-maskable LOG_NOTICE messages)
I - 6,LOG_INFO - statistical message
D - 7,LOG_DEBUG - debugging message
T - 8,LOG_DEBUG - tracing message
F - 9,LOG_DEBUG - full log message
The firewall administrator can decide what level of debugging they
desire; the lowest available is 'E' (error
messages).
The full debug level is very exhaustive for the system resources
and disc capacity. It is recommended to set this level only when hunting
bugs, to direct logging to a file (see log(5) for deatils) instead of syslogd,
and to do so for a single process only and for as short
time as possible. The logging level can be increased and decreased
by sending the SIGUSR1/SIGUSR2 signals.
Every (non-debugging) log message has its own Section 6 manual page, the name of which is equal to the first and second message identification parts. For example, the above example would correspond to the FTPP-110 manual page. This manual page shows the message text (if the message contains various values, they are substituted by adequate C-printf style directives — %s, %d etc.) and describes the meaning of the message.
There are five types of message texts:
Panic messages
**PANIC** [ftp-proxy.c:97] ftpadr(): Bad IP type (0)
These messages are logged in the case of unexpected internal errors. The program immediately fails in this case. The information in the square brackets (source module name and line) and before the parentheses (function name) locates the error in the source code and is important when reporting such an error to a support technician. The section 6 manual pages contain only the last part of the messages.
Errno messages
[log.c:97] open(): Permission denied (EACCES=13)
These messages are logged when a syscall returns an error state that cannot be reached in a natural way. All messages of this type have KERN-100 log ID and are usually followed by some “high-level” message describing the situation, in which the error occurs and its consequences.
For instance, when a write syscall fails,
the application (or library) will log the appropriate errno message
and then another message describing what kind of connection has
failed.
However, if the reason of the failure is “peer has
closed connection”, no errno message is generated and only
the “high-level” message appears. This kinds of error
need not necessarily stop the application.
The information in square brackets
(source module name and line) locates the error in the source code.
The name before the parentheses is the name of the syscall
that has caused the error.
The text after colon is the standard
“strerrno” text, the name
in the last parentheses is the proper errno
constant, the number is the errno value.
Configuration error messages
Line 21, char 1: Exactly one of DENY and ACCEPT must be specified
FTP-PROXY.ACL-1: Exactly one of DENY and ACCEPT must be specified
These messages are printed when the configuration reading utility or the CML tool finds an error in the syntax or semantics of the configuration. If the configuration is read by a proxy, the printed line and char numbers approximately locate where in the configuration file the error occurs. In the case of verification by the CML tool, the “configuration path” printed points to the place in the configuration where the error occurs. This path can be used as a parameter of the /SHOW command.
Ordinary log messages
closecfg(): Configuration failed, exiting
Such a log message is produced in the case of an error or of a normal, but significant condition. The function name (before parentheses) can sometimes be replaced by the '%s' symbol when the same message is produced by several functions.
Statistical messages
ACL PHASE=2 CLIENT=[127.0.0.1]:2471 SERVER=localhost:21 USER=des
PARENT=normal NAME=all ACCEPTED
The last type of message is statistical information. It is
supposed to be automatically post-processed, which is why its
format is less human-readable, but stricter.
It begins with a keyword
describing the message type, followed by couples
keyword=valueACCEPTED
or REJECTED at the end of the
message.
By default, Kernun applications log messages up to level 5 (LOG_NOTICE). The firewall administrator can change the logging level limit to a value between 3 (error messages) and 9 (full debug) by several means. K-level messages are logged in any case and cannot be switched off.
The configuration-time log level can be set using the
-d option on the command line. The value of the
option can range from 3 to 9, corresponding to the numeric values of
severity levels.
The run-time log level can be set using
the level item
of the log section of the particular application
configuration file. The logging levels are expressed
mnemonically there: error, warning,
normal, debug,
trace, and full.
The last way of setting the logging level is to send a running process the SIGUSR1 (to increase level by one) or SIGUSR2 (to decrease it) signal. The logging level cannot get outside the range 3-9.
By default, Kernun applications log via syslog using the LOCAL4 facility. Alternatively, the log output can be directed to a file. It is possible to specify how to handle the situation when the writing of a log message fails.
Another variant of the logging output is logging to memory. It is independent to and can be used simultaneously with syslog/file logging. Even the log level can be set differently for memory logging. The principle of logging to memory is that each process has a fixed circular memory buffer. Log messages are written to the buffer and if the buffer becomes full, the oldest messages are overwritten. The buffer is mapped to a file, hence it can be viewed by any program that displays file contents (e.g., less). When a proxy process terminates successfully, its memory log file is deleted. If the process fails, the file is retained. The memory log is not physically written to the disk until the termination of the process and it takes only a fixed amount of space for each process. It is therefore faster and takes smaller disk space than the normal log. The memory log level can be set to D or T, in order to get detailed records of the last moments of failed processes.