Introduction

One of the things which makes GNU/Linux a great operating system is that virtually anything and everything happening on and to the system may be logged in some manner. This information is invaluable for using the system in an informed manner, and should be one of the first resources you use to trouble-shoot the system and application issues. The logs can tell you almost anything you need to know, as long as you have an idea where to look first.

Your Ubuntu system provides vital information using various system log files. These log files are typically plain ASCII text in a standard log file format, and most of them sit in the traditional system log subdirectory /var/log . Many are generated by the system log daemon, syslogd on behalf of the system and certain applications, while some applications generate their own logs by writing directly to files in /var/log .

This guide talks about how to read and use several of these system log files, how to use and configure the system logging daemon, syslogd , and how log rotation works. See the Resources section for additional information.

Target Audience

This guide will be simple enough to use if you have any experience using the console and editing text files using a text editor. See the end of this document for some essential commands that may help you find your way around these files if you"re relatively new to the command line.

System Logs

System logs deal primarily with the functioning of the Ubuntu system, not necessarily with additional applications added by users. Examples include authorization mechanisms, system daemons, system messages, and the all-encompassing system log itself, syslog.

Authorization Log

The Authorization Log tracks usage of authorization systems, the mechanisms for authorizing users which prompt for user passwords, such as the Pluggable Authentication Module (PAM) system, the sudo command, remote logins to sshd and so on. The Authorization Log file may be accessed at /var/log/auth.log . This log is useful for learning about user logins and usage of the sudo command.

Use grep to cut down on the volume. For example, to see only information in the Authorization Log pertaining to sshd logins, use this:

grep sshd /var/log/auth.log | less

Daemon Log

A daemon is a program that runs in the background, generally without human intervention, performing some operation important to the proper running of your system. The daemon log at /var/log/daemon.log and contains information about running system and application daemons such as the Gnome Display Manager daemon gdm , the Bluetooth HCI daemon hcid , or the MySQL database daemon mysqld . This can help you trouble-shoot problems with a particular daemon.

Again, use grep to find specific information, plugging in the name of the daemon you"re interested in.

Debug Log

The debug log at /var/log/debug and provides detailed debug messages from the Ubuntu system and applications which log to syslogd at the DEBUG level.

Kernel Log

The kernel log at /var/log/kern.log provides a detailed log of messages from the Ubuntu Linux kernel. These messages may prove useful for trouble-shooting a new or custom-built kernel, for example.

Kernel Ring Buffer

The kernel ring buffer is not really a log file per se, but rather an area in the running kernel you can query for kernel bootup messages via the dmesg utility. To see the messages, use this:

dmesg | less

Or to search for lines that mention the Plug & Play system, for example, use grep like this:

dmesg | grep pnp | less

By default, the system initialization script /etc/init.d/bootmisc.sh sends all bootup messages to the file /var/log/dmesg as well. You can view and search this file the usual way.

System Log

The system log typically contains the greatest deal of information by default about your Ubuntu system. It is located at /var/log/syslog , and may contain information other logs do not. Consult the System Log when you can"t locate the desired log information in another log. It also contains everything that used to be in /var/log/messages .

Application Logs

Many applications also create logs in /var/log . If you list the contents of your /var/log subdirectory, you will see familiar names, such as /var/log/apache2 representing the logs for the Apache 2 web server, or /var/log/samba , which contains the logs for the Samba server. This section of the guide introduces some specific examples of application logs, and information contained within them.

Apache HTTP Server Logs

The default installation for Apache2 on Ubuntu creates a log subdirectory: /var/log/apache2 . Within this subdirectory are two log files with two distinct purposes:

/var/log/apache2/access.log - records of every page served and every file loaded by the web server.

/var/log/apache2/error.log - records of all error conditions reported by the HTTP server

By default, every time Apache accesses a file or page, the access logs record the IP address, time and date, browser identification string, HTTP result code and the text of the actual query, which will generally be a GET for a page view. Look at the Apache documentation for a complete rundown; quite a lot can be gleaned from this file, and indeed many statistical packages exist that perform analyzes of these logs.

Also, every time any error occurs, Apache adds a line to the error log. If you run PHP with error and warning messages disabled, this can be your only way to identify bugs.

CUPS Print System Logs

The Common Unix Printing System (CUPS) uses the default log file /var/log/cups/error_log to store informational and error messages. If you need to solve a printing issue in Ubuntu, this log may be a good place to start.

Rootkit Hunter Log

The Rootkit Hunter utility (rkhunter) checks your Ubuntu system for backdoors, sniffers and rootkits, which are all signs of compromise of your system. The log rkhunter uses is located at /var/log/rkhunter.log .

Samba SMB Server Logs

The Server Message Block Protocol (SMB) server, Samba is popularly used for sharing files between your Ubuntu computer and other computers which support the SMB protocol. Samba keeps three distinct types of logs in the subdirectory /var/log/samba:

log.nmbd - messages related to Samba"s NETBIOS over IP functionality (the network stuff)

log.smbd - messages related to Samba"s SMB/CIFS functionality (the file and print sharing stuff)

log. - messages related to requests for services from the IP address contained in the log file name, for example, log.192.168.1.1 .

X11 Server Log

The default X11 Windowing Server in use with Ubuntu is the Xorg X11 server, and assuming your computer has only one display defined, it stores log messages in the file /var/log/Xorg.0.log . This log is helpful for diagnosing issues with your X11 environment.

Non-Human-Readable Logs

Some log files found in the /var/log subdirectory are designed to be readable by applications, not necessarily by humans. Some examples of such log files which appear in /var/log follow.

Login Failures Log

The login failures log located at /var/log/faillog is actually designed to be parsed and displayed by the faillog command. For example, to print recent login failures, use this:

faillog

Last Logins Log

The last logins log at /var/log/lastlog should not typically be parsed and examined by humans, but rather should be used in conjunction with the lastlog command. For example to see a listing of logins with the lastlog command, displayed one page per screen with the less command, use the following command:

lastlog | less

Login Records Log

The file /var/log/wtmp contains login records, but unlike /var/log/lastlog above, /var/log/wtmp is not used to show a list of recent logins, but is instead used by other utilities such as the who command to present a listed of currently logged in users. This command will show the users currently logged in to your machine:

who

System Logging Daemon (syslogd)

The system logging daemon syslogd , also known as sysklogd , awaits logging messages from numerous sources and routes the messages to the appropriate file or network destination. Messages logged to syslogd usually contain common elements like system hostnames and time-stamps in addition to the specific log information.

Configuration of syslogd

The syslogd daemon"s configuration file is /etc/syslog.conf . Each entry in this file consists of two fields, the selector and the action. The selector field specifies a facility to be logged, such as for example the auth facility which deals with authorization, and a priority level to log such information at, such as info, or warning. The action field consists of a target for the log information, such as a standard log file (i.e. /var/log/syslog), or the hostname of a remote computer to send the log information to.

Echoing Messages to syslogd With Logger

A neat utility exists in the logger tool, which allows one to place messages into the System Log (i.e. /var/log/syslog) arbitrarily. For example, assume your user name is buddha , and you would like to enter a message into the syslog about a particularly delicious pizza you"re eating, you could use a command such as the following at a terminal prompt:

logger This Pizza from Vinnys Gourmet Rocks

and you would end up with a line in the /var/log/syslog file like this:

Jan 12 23:34:45 localhost buddha: This Pizza from Vinnys Gourmet Rocks

You can even specify a tag the messages come from, and redirect the output standard error too.

# # sample logger error jive # logmsg="/usr/bin/logger -s -t MyScript " # announce what this script is, even to the log $logmsg "Directory Checker FooScript Jive 1.0" # test for the existence of Fred" s home dir on this machine if [ -d /home/fred ]; then $logmsg "I. Fred"s Home Directory Found" else $logmsg "E. Fred"s Home Directory was NOT Found. "Boo Hoo." exit 1 fi

Executing this script as chkdir.sh on the machine butters where Fred does not have a home directory, /home/fred , gives the following results:

bumpy@butters:~$./chkdir.sh MyScript: Directory Checker FooScript Jive 1.0 MyScript: E. Fred's Home Directory was NOT Found. Boo Hoo. bumpy@butters:~$tail -n 2 /var/log/syslog Jan 12 23:23:11 localhost MyScript: Directory Checker FooScript Jive 1.0 Jan 12 23:23:11 localhost MyScript: E. Fred's Home Directory was NOT Found. Boo Hoo.

So, as you can see, we received the messages both via standard error, at the terminal prompt, and they also appear in our syslog.

Log Rotation

When viewing directory listings in /var/log or any of its subdirectories, you may encounter log files with names such as daemon.log.0 , daemon.log.1.gz , and so on. What are these log files? They are "rotated" log files. That is, they have automatically been renamed after a predefined time-frame, and a new original log started. After even more time the log files are compressed with the gzip utility as in the case of the example daemon.log.1.gz . The purpose of log rotation is to archive and compress old logs so that they consume less disk space, but are still available for inspection as needed. What handles this functionality? Why, the logrotate command of course! Typically, logrotate is called from the system-wide cron script /etc/cron.daily/logrotate , and further defined by the configuration file /etc/logrotate.conf . Individual configuration files can be added into /etc/logrotate.d (where the apache2 and mysql configurations are stored for example).

This guide will not cover the myriad of ways logrotate may be configured to handle the automatic rotation of any log file on your Ubuntu system. For more detail, check the Resources section of this guide.

NOTE: You may also rotate system log files via the cron.daily script /etc/cron.daily/sysklogd instead of using logrotate. Actually, the utility savelog may produce unexpected results on log rotation which configuring logrotate seems to have no effect on. In those cases, you should check the cron.daily sysklogd script in /etc/cron.daily/sysklogd and read the savelog manual page to see if savelog is not in fact doing the rotation in a way that is not what you are specifying with logrotate.

Essential Commands

If you"re new to the console and the Linux command line, these commands will get you up and running to the point where you can work with log files at a basic level.

Getting Started

To change to the log directory, where most of these files sit, use the cd command. This saves having to type out a full path name for every subsequent command:

cd /var/log

Editing Files

You can view and edit files in GEdit or Kate, the simple text editors that come with Ubuntu and Kubuntu respectively, but these can be overkill when all you want to do is look at a file or make simple changes. The easiest editor to use from the console is nano, which is less powerful but also less complicated than vim or emacs. The command to edit a particular logfile /var/log/example.log using nano is:

nano example.log

Press Ctrl+X to exit. It will ask if you want to save your changes when you exit, but unless you run it with the sudo command the files won"t be writable. In general, you won"t want to save your changes to log files, of course.

Viewing Files

To simply look at a file, an editor is overkill. Use the less command, which pages through a file one screen at a time:

less example.log

You don"t need sudo to look at a file. Press h for help, or q to quit. The cursor keys and page up/down keys will work as expected, and the slash key ("/") will do a case- sensitive search; the n key repeats the last search.

Viewing the Beginning of Files

To see the first ten lines of a file, use the head command:

head example.log

To see some other number of lines from the beginning of the file, add the -n switch, thus:

head -n 20 example.log

Viewing the End of Files

To see the final ten lines of a file, the analogous command is tail:

tail example.log

Again, the -n switch gives you control over how many lines it displays:

tail -n 20 example.log

Watching a Changing File

Also, the -f ("follow") switch puts tail into a loop, constantly waiting for new additions to the file it"s displaying. This is useful for monitoring files that are being updated in real time:

tail -f example.log

Press Ctrl+C to quit the loop.

Searching Files

Because log files can be large and unwieldy, it helps to be able to focus. The grep command helps you strip out only the content you care about. To find all the lines in a file containing the word "system", for example, use this:

grep "system" example.log

To find all the lines containing "system" at the beginning of the line, use this:

grep "^system" example.log

Note the caret symbol, a regular expression that matches only the start of a line. This is less useful for standard log files, which always start with a date and time, but it can be handy otherwise. Not all files have a standard format.

Any time the result of a grep is still too long, you can pipe it through less:

grep "system" example.log | less

Resources

Additional information on system and application logs and syslogd is available via the following resources:

Local System Resources

	System manual page for the dmesg kernel ring buffer utility
	System manual page for the faillog command (and also the faillog configuration file via man 5 faillog)
	System manual page for the grep pattern searching utility
	System manual page for the head utility
	System manual page for the kernel log daemon (klogd)
	System manual for the last command which shows last logged in users
	System manual page for the less paging utility
	System manual page for the logger command-line interface to syslog utility
	System manual page for the logrotate utility
	System manual page for the savelog log file saving utility
	System manual page for the system log daemon (syslogd)
	System manual page for the syslogd configuration file
	System manual page for the tail utility

When viewing log files (log files) in Linux, sometimes you need to monitor new log entries in real time. That is, you specify which log file (or files) you want to view and monitor in real time for new entries in that file.

Monitoring log files with the tail command

If you run the command tailFileName without any additional arguments, the last 10 lines of the file will be printed and the command will complete its work.

In order for the tail command to continuously output the last notes in the file, that is, if new entries appeared in the file, then the information on the screen was updated, the -f option is used:
tail -f LogFileName

Run the tail -f command to output the log file /var/log/syslog

tail -f /var/log/syslog

Since the option is used -f , the tail command does not complete its work, but waits for new entries to appear in the log file. As soon as new entries are added to the log file, they will immediately be displayed in the terminal.

To interrupt the execution of a command, press the keyboard shortcut Ctrl+C

tailf command

The tail -f command is equivalent to the tailf command.

Usage:

Tailf /var/log/mylogfile.log

An important difference between the tailf command and tail -f is that tailf does not access the file when it is not modified. As a consequence, the file access time is not updated and the system does not continuously flush the file to disk when the file is not updated.

The description of the tailf command indicates that it is useful for monitoring log files on laptops. Since the disk is not accessed unnecessarily and battery life is preserved.

tail -F. If the file has been renamed or deleted

Typically, in Linux, log files are not written indefinitely, otherwise such a file would be very inconvenient to use in the future. Instead, so-called file rotation is used. When the log file becomes large, it is either deleted or renamed (a backup copy of the file is created), and further messages begin to be written to a new, empty file.

The tail command has two options: -f and -F

• If used option -f and the tracked file is renamed, the tail command continues to track the already renamed file. Team tail in this case it is tied to the identifier (inode) of the file.
• If used option -F and the tracked file is renamed, the tail command will detect this, and as soon as a new log file is created (with the name that we specified to the command tail), team tail will start tracking this new file.

Let's look at an example.

We will monitor the log file /var/log/apache2/error.log. Execute the tail command with the -F option

Tail -F /var/log/apache2/error.log

If the system moves (renames) the error.log file to error.log.1 and creates a new error.log file, then our tail command will continue to monitor the new error.log file

If we were to use the option in this example -f , then the tail command would continue to monitor the error.log.1 file, which is no longer relevant for us when viewing logs in real time.

Monitor multiple log files simultaneously

The tail command supports tracking multiple files at once. To do this, you must specify file names separated by spaces.

Tail -f /var/log/apache2/error.log /var/log/apache2/access.log

As soon as any of the files changes, the name of this file and the new entries in it are displayed on the screen.

For simultaneous monitoring of several log files, there is a very convenient multitail utility

It does not just display data, but creates its own window (area) for each file and displays the data in this window. With its help, it is very convenient to track many log files at once and see them in one terminal window.

The multitail utility can be installed from the standard repositories of your distribution. To install, run the command (select the appropriate command for your distribution):

Sudo apt install multitail sudo yum install multitail sudo dnf install multitail

Usage:

Multitail /var/log/apache2/access.log /var/log/apache2/error.log

To exit the utility, press the q key

All log files are located in /var/log directory. In that directory, there are specific files for each type of logs. For example, system logs, such as kernel activities are logged in syslog file.

Some of the most common log files in that directory is:

In directory apt there is a file history.log which saves all the package installation and removal information even the initial system build as Live CD. You can open this file to see this very interesting file.

In directory dist-upgrade there is a file apt.log which logs the information during distribution upgrades

In directory installer the log files which are created during installation can be found.

There is an apport.log file which saves information about crashes in your system and reporting them.

The file auth.log includes information about the authentication activities such as when you authenticate as root user via sudo.

The file dpkg.log saves the low level details of package installation and removal related with dpkg . You might be aware that the apt system depends on dpkg for package installation and removal.

boot.log includes information of each boot.

kern.log saves kernel information such as warnings, errors etc.

alternatives.log includes the history of all the alternatives set by various packages and their removal via update-alternatives command.

Another important log file is Xorg.log which includes information about the graphics driver, its failures, warnings etc.

Some other types of Log files may be there depending on your installed packages. For example, My system also includes a log files epoptes.log which will only be there if you install epoptes package.

Changes after systemd

With the advent of systemd , logging is mostly handled by journalctl utility and store the logs in binary format in /var/lib/systemd/catalog/database file. This file enumerates all logs including kernel, boot and application logs and provides required logs via journalctl utility.

Here is a good article on journalctl on how you can use it to fetch required log info.

During its operation, the system monitors and saves in special files some events that it considers important or simply necessary for use in order to correct and debug errors, faulty configurations, etc. The files in which these events are stored are called log files or registration files . Often, logging files take up too much disk space, which can indicate a system malfunction, configuration errors, or simply incorrect configuration of the event logging daemons that monitor and collect everything. Thus, working with an event registration system is an important component in the work of any system administrator, on which the quality of system service and, as a result, their reliability and durability entirely depend.

How does the event registration system work?

Experienced system administrators know that it is necessary to review and analyze registration logs (files) regularly and with special care. The information contained in the logs often helps to quickly resolve problems or identify hidden problems in the system configuration. To track events by the system, check logs, record, store, archive and delete information from these logs, special regulations must be developed and approved for the organization operating and/or maintaining systems, servers and networks.

The main event logging tool in UNIX and Linu still remains syslogd daemon Syslog systems. But you should also keep in mind that for a long time, due to the variety of various branches of UNIX and Linux versions Many software packages, service scripts, and network daemons use their own logs, sometimes with exotic formats.

In general, the Syslog system (and other specialized programs) intercepts the monitored event and registers it in a log file. The registered event itself is a line of text containing data about the date/time, type, and severity of the event. Depending on the situation, this set may also include other data. The line of the registered event itself is broken up by separating characters: spaces, tabs, and punctuation marks to highlight the specified components.

Logs are easy to view because they are regular text files. For efficient work with magazines the most standard tools from the basic distribution of any distribution - commands and . If you need to comb through very large and complex logs, then you can (and even need) instead of the grep utility use another tool that is much more productive and flexible in such tasks - the . The text processing language Perl is also very well suited for this.

A typical Syslog system log entry typically looks like this:

Dec 18 15:12:42 backup.main.superhosting.ru sbatchd: sbatchd/main: ls_info() failed: LIM is down; try later; trying ... Dec 18 15:14:28 system.main.superhosting.ru pop-proxy: Connection from 186.115.198.84 Dec 18 15:14:30 control.main.superhosting.ru pingem : office.main.superhosting.ru has not answered 42 times Dec 18 15:15:05 service.main.superhosting.ru vmunix: Multiple softerrors: Seen 100Corrected Softerrors from SIMM J0201 Dec 18 15:15:16 backup.main.superhosting.ru PAM_unix: (sshd) session closed "for user trent

In this case, you can see that one of the Syslog logs contains events from several sources: sbathd, pingem, pop-proxy programs. You can also see that events are logged for several hosts interacting with this system: backup, system, office and service.

log files and their location in Linux

As already noted, in UNIX systems and Linux there are no clear conventions about where and how logs should be stored. They can be scattered throughout the entire file system, so it is important for each administrator to immediately understand where and for which packages and daemons the corresponding log files are located. But despite the lack of clear formal regulations regarding where logs are stored, there is still a traditional rule that these files should be located in the /var/log, /var/log/syslog, and also in /var/adm directories.

As a rule, only the superuser is granted access to read files in specified directories, but there is nothing wrong with setting up a more “democratic” access mode for frequently viewed logs that also do not contain important system information. Typically, this option is also used for convenience and to save time when you need to frequently and regularly examine some logs, for example for the Apache web server, which are usually located in /var/log/apache2 or /var/log/httpd.

It is also worth remembering that there are cases when (especially on faulty configurations) the total volume of log files increases sharply, and there is a high risk of crashing the system. For ease of monitoring free space on storage devices, as well as for reliability, the /var directory is often placed in a separate file system on a separate section.

Some special log files

The following table provides information about some log files, the information from which is very useful for system administration:

File	Program	Place	Frequency	Systems	Purpose
acpid	acpid	F	64k	RZ	Power System Events
auth.log	sudo and others	S	M	U	Authorization information
apache2/*	httpd or apache2	F	D	ZU	Apache Web Server Logs
apt*	APT	F	M	U	Package installers
boot.log	Launch scripts	F	M	R	Launch script logs
boot.msg	Core	IN	-	Z	Kernel message buffer image
cron	cron	S	N	RAH	Logs and information about the cron daemon
cups/*	CUPS	F	N	ZRU	Messages related to the printing system
daemon.log	Miscellaneous	S	N	U	Demon Tool Messages
debug	Miscellaneous	S	D	U	Debugging messages
dmesg	Core	IN	-	RU	Kernel message buffer image
dpkg.log	dpkg	F	M	U	Package installers
faillog	login	N	N	RZU	Information about failed authorization attempts
apache2/*	Httpd or apache2	F	D	R	Apache web server logs for /etc directory
kern.log	login	IN	-	RZ	All kernel tool messages
lastlog	login	IN	-	RZ	Last login time for each user (this file is binary)
mail*	Email programs	S	N	All	Electronic communications
messages	Miscellaneous	S	N	RZUS
rpmpkgs	cron.daily	IN	D	R	List of installed RPM packages
samba/*	smbd and others	F	N	-	Information about the Samba server
secure	sshd and others	S	M	R	Confidential authorization messages
sulog	su	F	-	SAH	Information about successful and unsuccessful attempts to use the su command
syslog*	Miscellaneous	S	H	SUH	Main system log file
warn	wpar	S	H	Z	System warning/error level events
wpars/*	wpar	F	-	A	Boot partition event information
wtmp	login	IN	M	All	System registration messages (binary file)
xen/*	Xen	F	1m	RZU	Information from the Xen Virtual Machine Monitor
Xorg.n.log	Xorg	F	N	R.S.	X Windows Server Error Messages
yum.log	yum	F	M	R	Package management log

The following designations apply for this table: S - Syslog, B - built-in name, F - configuration file, D - daily, N - weekly, M - monthly, NN - size in kilobytes or megabytes, Z - SUSE, R - Red Hat and CentOS, S - Solaris, H - HP-UX, A - AIX. The Frequency column indicates the frequency with which outdated information related to time or file size is deleted. The Program column indicates the program that created the file.

It should also be noted that most of the messages for the files presented in the table are sent to the Syslog system. The severity level and the program that creates the file are specified in the /etc/initlog.conf configuration file. - this is how the Syslog system works. The faillog file is binary and can therefore be read by the failog utility.

If you find an error, please highlight a piece of text and click Ctrl+Enter.