RHCSA and RHCE Chapter 9 - System Logging, Monitoring, and Automation
Syslog
From Red Hat Enterprise Linux 6 Deployment Guide:
Chapter 20. Viewing and Managing Log Files Log files are files that contain messages about the system, including the kernel, services, and applications running on it. There are different log files for different information. For example, there is a default system log file, a log file just for security messages, and a log file for cron tasks.
Log files can be very useful when trying to troubleshoot a problem with the system such as trying to load a kernel driver or when looking for unauthorized login attempts to the system. This chapter discusses where to find log files, how to view log files, and what to look for in log files.
Some log files are controlled by a daemon called rsyslogd. A list of log files maintained by rsyslogd can be found in the /etc/rsyslog.conf configuration file.
rsyslog is an enhanced, multi-threaded syslog daemon which replaced the sysklogd daemon. rsyslog supports the same functionality as sysklogd and extends it with enhanced filtering, encryption protected relaying of messages, various configuration options, or support for transportation via the TCP or UDP protocols. Note that rsyslog is compatible with sysklogd.
Here is how the configuration file is broken down for rsyslog:
20.1. Configuring rsyslog The main configuration file for rsyslog is /etc/rsyslog.conf. It consists of global directives, rules or comments (any empty lines or any text following a hash sign (#)). Both, global directives and rules are extensively described in the sections below.
20.1.1. Global Directives Global directives specify configuration options that apply to the rsyslogd daemon. They usually specify a value for a specific pre-defined variable that affects the behavior of the rsyslogd daemon or a rule that follows. All of the global directives must start with a dollar sign ($). Only one directive can be specified per line. The following is an example of a global directive that specifies the maximum size of the syslog message queue:
$MainMsgQueueSize 50000
The default size defined for this directive (10,000 messages) can be overridden by specifying a different value (as shown in the example above).
You may define multiple directives in your /etc/rsyslog.conf configuration file. A directive affects the behavior of all configuration options until another occurrence of that same directive is detected.
A comprehensive list of all available configuration directives and their detailed description can be found in /usr/share/doc/rsyslog-version-number/rsyslog_conf_global.html.
20.1.2. Modules Due to its modular design, rsyslog offers a variety of modules which provide dynamic functionality. Note that modules can be written by third parties. Most modules provide additional inputs (see Input Modules below) or outputs (see Output Modules below). Other modules provide special functionality specific to each module. The modules may provide additional configuration directives that become available after a module is loaded. To load a module, use the following syntax:
$ModLoad module
where $ModLoad is the global directive that loads the specified module and MODULE represents your desired module. For example, if you want to load the Text File Input Module (imfile — enables rsyslog to convert any standard text files into syslog messages), specify the following line in your /etc/rsyslog.conf configuration file:
$ModLoad imfile
20.1.3. Rules A rule is specified by a filter part, which selects a subset of syslog messages, and an action part, which specifies what to do with the selected messages. To define a rule in your /etc/rsyslog.conf configuration file, define both, a filter and an action, on one line and separate them with one or more spaces or tabs.
20.1.3.1. Filter Conditions rsyslog offers various ways how to filter syslog messages according to various properties. This sections sums up the most used filter conditions.
Facility/Priority-based filters The most used and well-known way to filter syslog messages is to use the facility/priority-based filters which filter syslog messages based on two conditions: facility and priority. To create a selector, use the following syntax:
FACILITY.PRIORITY
where:
- FACILITY specifies the subsystem that produces a specific syslog message. For example, the mail subsystem handles all mail related syslog messages. FACILITY can be represented by one of these keywords: auth, authpriv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp, and local0 through local7.
- PRIORITY specifies a priority of a syslog message. PRIORITY can be represented by one of these keywords (listed in an ascending order): debug, info, notice, warning, err, crit, alert, and emerg. By preceding any priority with an equal sign (=), you specify that only syslog messages with that priority will be selected. All other priorities will be ignored. Conversely, preceding a priority with an exclamation mark (!) selects all syslog messages but those with the defined priority. By not using either of these two extensions, you specify a selection of syslog messages with the defined or higher priority.
In addition to the keywords specified above, you may also use an asterisk (*) to define all facilities or priorities (depending on where you place the asterisk, before or after the dot). Specifying the keyword none serves for facilities with no given priorities.
To define multiple facilities and priorities, simply separate them with a comma (,). To define multiple filters on one line, separate them with a semi-colon (;).
The following are a few examples of simple facility/priority-based filters:
kern.* # Selects all kernel syslog messages with any priority mail.crit # Selects all mail syslog messages with priority crit and higher. cron.!info,!debug # Selects all cron syslog messages except those with the info or debug priority.
20.1.3.2. Actions Actions specify what is to be done with the messages filtered out by an already-defined selector. The following are some of the actions you can define in your rule:
Saving syslog messages to log files The majority of actions specify to which log file a syslog message is saved. This is done by specifying a file path after your already-defined selector. The following is a rule comprised of a selector that selects all cron syslog messages and an action that saves them into the /var/log/cron.log log file:
cron.* /var/log/cron.log
Use a dash mark (-) as a prefix of the file path you specified if you want to omit syncing the desired log file after every syslog message is generated.
Your specified file path can be either static or dynamic. Static files are represented by a simple file path as was shown in the example above. Dynamic files are represented by a template and a question mark (?) prefix.
If the file you specified is an existing tty or /dev/console device, syslog messages are sent to standard output (using special tty-handling) or your console (using special /dev/console-handling) when using the X Window System, respectively.
Sending syslog messages over the network rsyslog allows you to send and receive syslog messages over the network. This feature allows to administer syslog messages of multiple hosts on one machine. To forward syslog messages to a remote machine, use the following syntax:
@[(option)]host:[port]
where:
- The at sign (@) indicates that the syslog messages are forwarded to a host using the UDP protocol. To use the TCP protocol, use two at signs with no space between them (@@).
- The OPTION attribute can be replaced with an option such as zNUMBER. This option enables zlib compression for syslog messages; the NUMBER attribute specifies the level of compression. To define multiple options, simply separate each one of them with a comma (,).
- The HOST attribute specifies the host which receives the selected syslog messages.
- The PORT attribute specifies the host machine’s port.
When specifying an IPv6 address as the host, enclose the address in square brackets ([, ]).
The following are some examples of actions that forward syslog messages over the network (note that all actions are preceded with a selector that selects all messages with any priority):
*.* @192.168.0.1 # Forwards messages to 192.168.0.1 via the UDP protocol *.* @@example.com:18 # Forwards messages to "example.com" using port 18 and the TCP protocol *.* @(z9)[2001::1] # Compresses messages with zlib (level 9 compression) # and forwards them to 2001::1 using the UDP protocol
Here is how the default configuration for rsyslog looks like:
[root@rhel1 ~]# cat /etc/rsyslog.conf
#rsyslog v3 config file
# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance
#### MODULES ####
$ModLoad imuxsock.so # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by rklogd)
#$ModLoad immark.so # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spppl/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
Here is the Command Line Configuration:
20.1.4. rsyslog Command Line Configuration Some of rsyslog’s functionality can be configured through the command line options, as sysklogd’s can. Note that as of version 3 of rsyslog, this method was deprecated. To enable some of these option, you must specify the compatibility mode rsyslog should run in. However, configuring rsyslog through the command line options should be avoided.
To specify the compatibility mode rsyslog should run in, use the -c option. When no parameter is specified, rsyslog tries to be compatible with sysklogd. This is partially achieved by activating configuration directives that modify your configuration accordingly. Therefore, it is advisable to supply this option with a number that matches the major version of rsyslog that is in use and update your /etc/rsyslog.conf configuration file accordingly. If you want to, for example, use sysklogd options (which were deprecated in version 3 of rsyslog), you can specify so by executing the following command:
~]# rsyslogd -c 2
Options that are passed to the rsyslogd daemon, including the backward compatibility mode, can be specified in the /etc/sysconfig/rsyslog configuration file.
Here are the parameters used for my rsyslog daemon:
[root@rhel1 ~]# ps -ef | grep rsyslog | grep -v grep
root 1005 1 0 14:27 ? 00:00:00 /sbin/rsyslogd -c 4
Logrotate
Next we move onto logrotate:
20.2. Locating Log Files Most log files are located in the /var/log/ directory. Some applications such as httpd and samba have a directory within /var/log/ for their log files.
You may notice multiple files in the /var/log/ directory with numbers after them (for example, cron-20100906). These numbers represent a timestamp that has been added to a rotated log file. Log files are rotated so their file sizes do not become too large. The logrotate package contains a cron task that automatically rotates log files according to the /etc/logrotate.conf configuration file and the configuration files in the /etc/logrotate.d/ directory.
20.2.1. Configuring logrotate The following is a sample /etc/logrotate.conf configuration file:
# rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # uncomment this if you want your log files compressed compress
All of the lines in the sample configuration file define global options that apply to every log file. In our example, log files are rotated weekly, rotated log files are kept for the duration of 4 weeks, and all rotated log files are compressed by gzip into the .gz format. Any lines that begin with a hash sign (#) are comments and are not processed
You may define configuration options for a specific log file and place it under the global options. However, it is advisable to create a separate configuration file for any specific log file in the /etc/logrotate.d/ directory and define any configuration options there.
The following is an example of a configuration file placed in the /etc/logrotate.d/ directory:
/var/log/messages { rotate 5 weekly postrotate /usr/bin/killall -HUP syslogd endscript }
The configuration options in this file are specific for the /var/log/messages log file only. The settings specified here override the global settings where possible. Thus the rotated /var/log/messages log file will be kept for five weeks instead of four weeks as was defined in the global options.
The following is a list of some of the directives you can specify in your logrotate configuration file:
weekly — Specifies the rotation of log files on a weekly basis. Similar directives include:
- daily
- monthly
- yearly
compress — Enables compression of rotated log files. Similar directives include:
- nocompress
- compresscmd — Specifies the command to be used for compressing.
- uncompresscmd
- compressext — Specifies what extension is to be used for compressing.
- compressoptions — Lets you specify any options that may be passed to the used compression program.
- delaycompress — Postpones the compression of log files to the next rotation of log files.
rotate INTEGER — Specifies the number of rotations a log file undergoes before it is removed or mailed to a specific address. If the value 0 is specified, old log files are removed instead of rotated.
mail ADDRESS — This option enables mailing of log files that have been rotated as many times as is defined by the rotate directive to the specified address. Similar directives include:
- nomail
- mailfirst — Specifies that the just-rotated log files are to be mailed, instead of the about-to-expire log files.
- maillast — Specifies that the about-to-expire log files are to be mailed, instead of the just-rotated log files. This is the default option when mail is enabled.
So cron runs logrotate periodically. Here is the script that cron runs daily:
[root@rhel1 ~]# cat /etc/cron.daily/logrotate
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf >/dev/null 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
That basically runs /usr/sbin/logrotate /etc/logrotate.conf
. Checking out /etc/logrotate.conf file:
[root@rhel1 ~]# cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
#compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0600 root utmp
rotate 1
}
That has very generic settings and has settings for /var/log/wtmp and /var/log/btmp log files. But we also include everything that is the under /etc/logrotate.d folder and checking under that directory:
[root@rhel1 ~]# ls /etc/logrotate.d/
dracut setroubleshoot subscription-manager syslog up2date vsftpd yum
We can see package specific settings.
Remote Syslog Server
Let’s setup our RHEL6 machine to be a syslog Server to accept logs on UDP 514 and let’s configure the RHEL 5 machine to send logs remotely to the RHEL6 machine on UDP 514.
Syslog Server
On the RHEL 6 machine edit /etc/rsyslog.conf and un-comment the following lines:
# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514
Restart the rsyslogd daemon:
[root@rhel1 ~]# service rsyslog restart
Shutting down system logger: rsyslogd
Starting system logger: rsyslogd
Make sure it’s listening on UDP 514:
[root@rhel1 ~]# netstat -anup --inet | grep 514
udp 0 0 0.0.0.0:514 0.0.0.0:* 1524/rsyslogd
Open up the firewall for that port:
[root@rhel1 ~]# iptables -I INPUT 5 -p udp -m udp --dport 514 -j ACCEPT
[root@rhel1 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:
Syslog Client
On the RHEL 5 machine, edit /etc/syslog.conf and add the following lines:
# Send logs to remote syslog server
*.* @192.168.56.102
then restart the syslog services:
[root@rhel2 ~]# service syslog restart
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Then send a test message from the client:
[root@rhel2 ~]# logger test
Check tcpdump on the syslog server, to make sure the logs are coming in:
[root@rhel1 ~]# sudo tcpdump -i eth1 udp port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
18:17:25.347833 IP 192.168.56.103.syslog > 192.168.56.102.syslog: SYSLOG user.notice, length: 15
lastly check the logs in the server to make sure the logs are there:
[root@rhel1 log]# grep 192.168.56.103 messages
Jun 11 18:15:59 192.168.56.103 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 11 18:17:25 192.168.56.103 root: test
There is our test message.
Send Remote logs to a specific file
If you don’t want to seep through /var/log/message for remote logs, you can add a rule to put remote logs to another file. On the server add the following to the /etc/rsyslog.conf file on top of all the Rules:
#### RULES ####
if $fromhost-ip == '192.168.56.103' then /var/log/remote/192-168-56-103.log
& ~
The bottom line ensures that the logs don’t get logged to the regular /var/log/messages file as well. Restart rsyslog:
[root@rhel1 ~]# service rsyslog restart
Shutting down system logger: rsyslogd
Starting system logger: rsyslogd
Do another test and make sure it shows up under the appropriate file:
[root@rhel1 ~]# tail /var/log/remote/192-168-56-103.log
Jun 11 18:31:48 192.168.56.103 root: test
That looks good.
User Login Information
From this RHEL Forum:
The lastb command is able to show a list of failed users logins since the creation date of the /var/log/btmp file. The /var/log/btmp file is a binary file which can not be viewed with a normal text editor.
and from this RHEL forum:
The last command will display all users logged into a system along with their current tty’s. However, it will only display login information starting from the time /var/log/wtmp was created.
There is also another command called lastlog. Here is how each look like:
[root@rhel1 ~]# lastb
root ssh:notty 192.168.56.1 Wed Jun 12 16:26 - 16:26 (00:00)
btmp begins Wed Jun 12 16:26:27 2013
So the latest failed login was from root.
[root@rhel1 ~]# last
root pts/0 192.168.56.1 Wed Jun 12 16:26 still logged in
root pts/0 192.168.56.1 Wed Jun 12 16:14 - 16:26 (00:11)
reboot system boot 2.6.32-131.0.15. Wed Jun 12 16:13 - 16:42 (00:28)
root pts/0 192.168.56.1 Tue Jun 11 18:34 - down (00:00)
root pts/0 192.168.56.1 Tue Jun 11 14:27 - 18:34 (04:06)
reboot system boot 2.6.32-131.0.15. Tue Jun 11 14:27 - 18:34 (04:06)
root tty1 Mon Jun 10 18:22 - down (00:00)
We can see a bunch of logins and their corresponding times.
[root@rhel1 ~]# lastlog
Username Port From Latest
root pts/0 192.168.56.1 Wed Jun 12 16:26:28 -0600 2013
bin **Never logged in**
daemon **Never logged in**
adm **Never logged in**
lp **Never logged in**
We can see the last login for each user that exists on the system. We can see some system users were never logged in, which is expected. We can also check out currently logged in users, from this RHEL Forum:
To quickly check who is currently log in on the server and the programs they are currently running, run the ‘w’ command. This command would also show how they are logged in, if they logged in remotely and the current time, how long the their login session has lasted. Alternatively, you can use the ‘who’ command. Here is how each looks like:
[root@rhel1 ~]# w
16:47:40 up 34 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 192.168.56.1 16:26 0.00s 0.05s 0.00s w
and here is the other one:
[root@rhel1 ~]# who
root pts/0 2013-06-12 16:26 (192.168.56.1)
Monitoring The System
From Red Hat Enterprise Linux 6 Performance Tuning Guide:
3.3. Built-in Command-line Monitoring Tools In addition to graphical monitoring tools, Red Hat Enterprise Linux provides several tools that can be used to monitor a system from the command line. The advantage of these tools is that they can be used outside run level 5. This section discusses each tool briefly, and suggests the purposes to which each tool is best suited.
top
The top tool provides a dynamic, real-time view of the processes in a running system. It can display a variety of information, including a system summary and the tasks currently being managed by the Linux kernel. It also has a limited ability to manipulate processes. Both its operation and the information it displays are highly configurable, and any configuration details can be made to persist across restarts.
By default, the processes shown are ordered by the percentage of CPU usage, giving an easy view into the processes that are consuming the most resources. For detailed information about using top, refer to its man page: man top.
ps
The ps tool takes a snapshot of a select group of active processes. By default this group is limited to processes owned by the current user and associated with the same terminal.
It can provide more detailed information about processes than top, but is not dynamic. For detailed information about using ps, refer to its man page: man ps.
vmstat
vmstat (Virtual Memory Statistics) outputs instantaneous reports about your system’s processes, memory, paging, block I/O, interrupts and CPU activity.
Although it is not dynamic like top, you can specify a sampling interval, which lets you observe system activity in near-real time.
For detailed information about using vmstat, refer to its man page: man vmstat.
sar
sar (System Activity Reporter) collects and reports information about today’s system activity so far. The default output covers today’s CPU utilization at ten minute intervals from the beginning of the day:
12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 0.10 0.00 0.15 2.96 0.00 96.79 12:20:01 AM all 0.09 0.00 0.13 3.16 0.00 96.61 12:30:01 AM all 0.09 0.00 0.14 2.11 0.00 97.66 ...
This tool is a useful alternative to attempting to create periodic reports on system activity through top or similar tools.
For detailed information about using sar, refer to its man page: man sar.
Here is how top looks like:
top - 17:03:28 up 50 min, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 73 total, 1 running, 72 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 511508k total, 161764k used, 349744k free, 9136k buffers
Swap: 1048568k total, 0k used, 1048568k free, 101512k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1334 root 20 0 11400 3224 2592 S 0.3 0.6 0:00.20 sshd
1434 root 20 0 2540 1096 880 R 0.3 0.2 0:00.09 top
1 root 20 0 2852 1384 1192 S 0.0 0.3 0:00.80 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.02 events/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 sync_supers
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 bdi-default
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0
16 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kblockd/0
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpid
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_hotplug
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/0
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_aux
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksuspend_usbd
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/0
26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/0
We can see a lot of information from top. CPU usage, Memory Usage and Priority of each process running on the system. From this output we can figure what process is taking the most resources on the system. From the above output, we can see that the system is pretty idle:
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Our top CPU process is sshd:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1334 root 20 0 11400 3224 2592 S 0.3 0.6 0:00.20 sshd
1434 root 20 0 2540 1096 880 R 0.3 0.2 0:00.09 top
1 root 20 0 2852 1384 1192 S 0.0 0.3 0:00.80 init
And that is a whopping 0.3% :) Here is how ps looks like:
[root@rhel1 ~]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.2 2852 1384 ? Ss 16:13 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S 16:13 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 16:13 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S 16:13 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S 16:13 0:00 [migration/0]
root 6 0.0 0.0 0 0 ? S 16:13 0:00 [watchdog/0]
root 7 0.0 0.0 0 0 ? S 16:13 0:00 [events/0]
root 8 0.0 0.0 0 0 ? S 16:13 0:00 [cpuset]
root 9 0.0 0.0 0 0 ? S 16:13 0:00 [khelper]
root 10 0.0 0.0 0 0 ? S 16:13 0:00 [netns]
root 11 0.0 0.0 0 0 ? S 16:13 0:00 [async/mgr]
root 12 0.0 0.0 0 0 ? S 16:13 0:00 [pm]
root 13 0.0 0.0 0 0 ? S 16:13 0:00 [sync_supers]
root 14 0.0 0.0 0 0 ? S 16:13 0:00 [bdi-default]
root 15 0.0 0.0 0 0 ? S 16:13 0:00 [kintegrityd/0]
root 16 0.0 0.0 0 0 ? S 16:13 0:00 [kblockd/0]
root 17 0.0 0.0 0 0 ? S 16:13 0:00 [kacpid]
root 18 0.0 0.0 0 0 ? S 16:13 0:00 [kacpi_notify]
root 19 0.0 0.0 0 0 ? S 16:13 0:00 [kacpi_hotplug]
root 20 0.0 0.0 0 0 ? S 16:13 0:00 [ata/0]
root 21 0.0 0.0 0 0 ? S 16:13 0:00 [ata_aux]
root 22 0.0 0.0 0 0 ? S 16:13 0:00 [ksuspend_usbd]
root 23 0.0 0.0 0 0 ? S 16:13 0:00 [khubd]
root 24 0.0 0.0 0 0 ? S 16:13 0:00 [kseriod]
root 25 0.0 0.0 0 0 ? S 16:13 0:00 [md/0]
root 26 0.0 0.0 0 0 ? S 16:13 0:00 [md_misc/0]
root 27 0.0 0.0 0 0 ? S 16:13 0:00 [khungtaskd]
root 28 0.0 0.0 0 0 ? S 16:13 0:00 [kswapd0]
root 29 0.0 0.0 0 0 ? SN 16:13 0:00 [ksmd]
root 30 0.0 0.0 0 0 ? S 16:13 0:00 [aio/0]
root 31 0.0 0.0 0 0 ? S 16:13 0:00 [crypto/0]
root 36 0.0 0.0 0 0 ? S 16:13 0:00 [kthrotld/0]
root 38 0.0 0.0 0 0 ? S 16:13 0:00 [kpsmoused]
root 39 0.0 0.0 0 0 ? S 16:13 0:00 [usbhid_resumer]
root 69 0.0 0.0 0 0 ? S 16:13 0:00 [kstriped]
root 208 0.0 0.0 0 0 ? S 16:13 0:00 [scsi_eh_0]
root 209 0.0 0.0 0 0 ? S 16:13 0:00 [scsi_eh_1]
root 217 0.0 0.0 0 0 ? S 16:13 0:00 [scsi_eh_2]
root 272 0.0 0.0 0 0 ? S 16:13 0:00 [kdmflush]
root 276 0.0 0.0 0 0 ? S 16:13 0:00 [kdmflush]
root 291 0.0 0.0 0 0 ? S 16:13 0:00 [jbd2/dm-0-8]
root 292 0.0 0.0 0 0 ? S 16:13 0:00 [ext4-dio-unwrit]
root 363 0.0 0.1 2680 956 ? S 16:13 0:00 /sbin/udevd -d
root 653 0.0 0.0 0 0 ? S 16:13 0:00 [jbd2/sda1-8]
root 654 0.0 0.0 0 0 ? S 16:13 0:00 [ext4-dio-unwrit]
root 691 0.0 0.0 0 0 ? S 16:13 0:00 [kauditd]
root 858 0.0 0.0 0 0 ? S 16:13 0:00 [flush-253:0]
root 981 0.0 0.1 12896 796 ? S 16:13 0:00 auditd
root 983 0.0 0.1 13432 768 ? S 16:13 0:00 /sbin/audispd
root 987 0.0 0.4 15108 2140 ? S 16:13 0:00 /usr/sbin/sedispatch
root 999 0.0 0.3 45792 1548 ? Sl 16:13 0:00 /sbin/rsyslogd -c 4
rpc 1018 0.0 0.1 2536 612 ? Ss 16:13 0:00 rpcbind
dbus 1029 0.0 0.2 15112 1124 ? Ssl 16:13 0:00 dbus-daemon --system
root 1061 0.0 0.1 9256 896 ? Ss 16:13 0:00 /usr/sbin/sshd
root 1069 0.0 0.1 3152 876 ? Ss 16:13 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 1080 0.0 0.1 7556 576 ? Ss 16:13 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
root 1165 0.0 0.4 12312 2428 ? Ss 16:13 0:00 /usr/libexec/postfix/master
postfix 1172 0.0 0.4 12388 2388 ? S 16:13 0:00 pickup -l -t fifo -u
postfix 1173 0.0 0.4 12456 2428 ? S 16:13 0:00 qmgr -l -t fifo -u
root 1175 0.0 0.2 5876 1260 ? Ss 16:13 0:00 crond
root 1192 0.0 0.0 1972 388 ? Ss 16:13 0:00 /usr/bin/rhsmcertd 240
root 1204 0.0 0.0 1980 484 tty1 Ss+ 16:13 0:00 /sbin/mingetty /dev/tty1
root 1206 0.0 0.0 1980 484 tty2 Ss+ 16:13 0:00 /sbin/mingetty /dev/tty2
root 1208 0.0 0.0 1980 484 tty3 Ss+ 16:13 0:00 /sbin/mingetty /dev/tty3
root 1210 0.0 0.0 1980 480 tty4 Ss+ 16:13 0:00 /sbin/mingetty /dev/tty4
root 1212 0.0 0.0 1980 484 tty5 Ss+ 16:13 0:00 /sbin/mingetty /dev/tty5
root 1214 0.0 0.0 1980 488 tty6 Ss+ 16:13 0:00 /sbin/mingetty /dev/tty6
root 1222 0.0 0.3 3336 1760 ? S< 16:13 0:00 /sbin/udevd -d
root 1223 0.0 0.3 3336 1760 ? S< 16:13 0:00 /sbin/udevd -d
root 1334 0.0 0.6 11400 3224 ? Ss 16:26 0:00 sshd: root@pts/0
root 1339 0.0 0.3 5228 1756 pts/0 Ss 16:26 0:00 -bash
root 1431 0.0 0.1 2796 592 ? Ss 17:01 0:00 /usr/sbin/anacron -s
root 1436 0.0 0.1 4776 1012 pts/0 R+ 17:07 0:00 ps aux
If you want to figure out specific information regarding a process then ps is your friend. Top shows you an overall overview of the system, where ps can dig deeper into each process ID. We can even see the full path of the command that was used to start the process, this can be helpful as well. Let’s see we wanted to find out any processes that have to do with postfix:
[root@rhel1 ~]# ps -eaf | grep postfix
root 1165 1 0 16:13 ? 00:00:00 /usr/libexec/postfix/master
postfix 1172 1165 0 16:13 ? 00:00:00 pickup -l -t fifo -u
postfix 1173 1165 0 16:13 ? 00:00:00 qmgr -l -t fifo -u
We can see a couple of processes that are started with the postfix user and the master process started by the root user. Similar tools exist like pidof and pgrep to find out PID of process names. Those tools are usually used in conjuction with kill,pkill,nice, and renice. Information regarding each of those can be seen in “Controlling the system”. Here are brief descriptions of each:
pgrep This command is useful for finding the process id of a particular process when you know part of its name.
kill To kill processes on your system, you will need their pid’s or id’s . Use ps or pstree to find out the process id’s (pid’s), or use jobs to find out id’s.
killall Kill a process by it’s name, uses names instead of process id’s (pid’s). Use -v to have killall report whether the kill was successful or not and -i for interactive mode (will prompt you before attempting to kill).
pkill pkill is used to kill processes according to an extended regular expression. Use the -u option to kill using a user name(s) and process name (for example to only kill a process of a certain user). pkill can also send specific signals to processes.
nice Sets the priority for a process. nice -20 is the maximum priority (only administrative users can assign negative priorities), nice 20 is the minimum priority. You must be root to give a process a higher priority, but you can always lower the priority of your own processes…
renice Changes the priority of an existing command. You may use the options -u to change the priorities of all processes for a particular user name and -g to change priorities for all processes of a particular group. The default is to change via the process id number.
So let’s renice our rhsmcertd process and then kill it. First let’s take a look at the current process:
[root@rhel1 ~]# ps -FcC rhsmcertd
UID PID PPID CLS PRI SZ RSS PSR STIME TTY TIME CMD
root 1192 1 TS 19 493 388 0 16:13 ? 00:00:00 /usr/bin/rhsmcertd 240
We can see that the Priority is currently 19 and the PID is 1192. Using the other tools, we can determine the PID like so:
[root@rhel1 ~]# pgrep -l rhsmcertd
1192 rhsmcertd
we can also use the renice command together with pgrep, like so:
[root@rhel1 ~]# renice +4 `pgrep rhsmcertd`
1192: old priority 0, new priority 4
Here is how the Priority looked like after the change:
[root@rhel1 init.d]# ps -FcC rhsmcertd
UID PID PPID CLS PRI SZ RSS PSR STIME TTY TIME CMD
root 1192 1 TS 15 493 388 0 17:49 ? 00:00:00 /usr/bin/rhsmcertd 240
So two ways to kill our process would be:
[root@rhel1 ~]# pkill rhsmcertd
[root@rhel1 ~]# kill `pgrep rhsmcertd`
After I restarted it, here was the new PID:
[root@rhel1 init.d]# pidof rhsmcertd
1807
Now that was a very little process managament guide, let’s get back to system monitoring. I ran a quick dd command on one terminal:
[root@rhel1 ~]# dd if=/dev/zero of=test.dd bs=1M count=100
on the other terminal I ran the vmstat command and here is what I saw:
[root@rhel1 ~]# vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 324448 26072 104216 0 0 22 2 21 13 0 0 99 0 0
0 0 0 324440 26072 104216 0 0 0 0 18 7 0 0 100 0 0
0 1 0 251268 26128 173076 0 0 56 32776 146 73 0 9 63 28 0
0 1 0 179596 26128 240448 0 0 0 95448 296 166 0 10 0 90 0
0 1 0 129128 26136 290608 0 0 8 44064 259 127 0 10 0 90 0
0 1 0 78040 26136 341160 0 0 0 45856 284 179 0 10 0 90 0
1 1 0 31788 26136 386600 0 0 0 45760 294 209 1 10 0 89 0
0 2 0 6180 23636 414972 0 0 4 45744 352 364 0 11 0 89 0
0 1 0 9172 6848 438548 0 0 22 178 21 14 0 0 99 1 0
0 0 0 9172 6852 438548 0 0 0 4 22 19 0 0 98 2 0
0 0 0 9172 6852 438548 0 0 0 0 20 10 0 0 100 0 0
0 0 0 9196 6852 438548 0 0 0 0 17 9 1 0 99 0 0
We definitely see a decrease in free memory (the filesystem caches a lot of the IO), an increase in IO (bo = Blocks sent to a block device), and of course an increase in interrupts (system in) since we were writing to a disk. For more information regarding IRQs, check out Interrupts and IRQ Tuning from the Performance Tuning guide.
Automation
From Red Hat Enterprise Linux 6 Deployment Guide:
Chapter 21. Automating System Tasks Tasks, also known as jobs, can be configured to run automatically within a specified period of time, on a specified date, or when the system load average decreases below 0.8.
Red Hat Enterprise Linux is pre-configured to run important system tasks to keep the system updated. For example, the slocate database used by the locate command is updated daily. A system administrator can use automated tasks to perform periodic backups, monitor the system, run custom scripts, and so on.
Red Hat Enterprise Linux comes with the following automated task utilities: cron, anacron, at, and batch.
Every utility is intended for scheduling a different job type: while Cron and Anacron schedule recurring jobs, At and Batch schedule one-time jobs.
21.1. Cron and Anacron Both, Cron and Anacron, are daemons that can schedule execution of recurring tasks to a certain point in time defined by the exact time, day of the month, month, day of the week, and week.
Cron jobs can run as often as every minute. However, the utility assumes that the system is running continuously and if the system is not on at the time when a job is scheduled, the job is not executed.
On the other hand, Anacron remembers the scheduled jobs if the system is not running at the time when the job is scheduled. The job is then executed as soon as the system is up. However, Anacron can only run a job once a day.
21.1.1. Installing Cron and Anacron To install Cron and Anacron, you need to install the cronie package with Cron and the cronie-anacron package with Anacron (cronie-anacron is a sub-package of cronie).
To determine if the packages are already installed on your system, issue the rpm -q cronie cronie-anacron command. The command returns full names of the cronie and cronie-anacron packages if already installed or notifies you that the packages are not available.
To install the packages, use the yum command in the following form:
yum install package
For example, to install both Cron and Anacron, type the following at a shell prompt:
~]# yum install cronie cronie-anacron
Note that you must have superuser privileges (that is, you must be logged in as root) to run this command.
Let’s make sure we have it installed:
[root@rhel1 ~]# rpm -qa cronie*
cronie-1.4.4-7.el6.i686
cronie-anacron-1.4.4-7.el6.i686
That looks good, moving on with the guide:
21.1.2.1. Starting and Stopping the Cron Service To determine if the service is running, use the command service crond status. To run the crond service in the current session, type the following at a shell prompt as root:
service crond start
To configure the service to be automatically started at boot time, use the following command:
chkconfig crond on
This command enables the service in runlevel 2, 3, 4, and 5.
21.1.2.2. Stopping the Cron Service To stop the crond service, type the following at a shell prompt as root
service crond stop
To disable starting the service at boot time, use the following command:
chkconfig crond off
This command disables the service in all runlevels.
Let’s make sure the service is started and is setup to be automatically started:
[root@rhel1 ~]# service crond status
crond (pid 1175) is running...
[root@rhel1 ~]# chkconfig --list crond
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
That looks fabuluos. Moving onto anacron:
21.1.3. Configuring Anacron Jobs The main configuration file to schedule jobs is the /etc/anacrontab file, which can be only accessed by the root user. The file contains the following:
SHELL=/bin/sh PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # the maximal random delay added to the base delay of the jobs RANDOM_DELAY=45 # the jobs will be started during the following hours only START_HOURS_RANGE=3-22 #period in days delay in minutes job-identifier command 1 5 cron.daily nice run-parts /etc/cron.daily 7 25 cron.weekly nice run-parts /etc/cron.weekly @monthly 45 cron.monthly nice run-parts /etc/cron.monthly
The first three lines define the variables that configure the environment in which the anacron tasks run:
- SHELL — shell environment used for running jobs (in the example, the Bash shell)
- PATH — paths to executable programs
- MAILTO — username of the user who receives the output of the anacron jobs by email If the MAILTO variable is not defined (MAILTO=), the email is not sent.
The next two variables modify the scheduled time for the defined jobs:
RANDOM_DELAY — maximum number of minutes that will be added to the delay in minutes variable which is specified for each job The minimum delay value is set, by default, to 6 minutes.
If RANDOM_DELAY is, for example, set to 12, then between 6 and 12 minutes are added to the delay in minutes for each job in that particular anacrontab. RANDOM_DELAY can also be set to a value below 6, including *** . When set to ** *, no random delay is added. This proves to be useful when, for example, more computers that share one network connection need to download the same data every day. START_HOURS_RANGE — interval, when scheduled jobs can be run, in hours
In case the time interval is missed, for example due to a power failure, the scheduled jobs are not executed that day.
The remaining lines in the /etc/anacrontab file represent scheduled jobs and follow this format:
period in days delay in minutes job-identifier command
- period in days — frequency of job execution in days The property value can be defined as an integer or a macro (@daily, @weekly, @monthly), where @daily denotes the same value as integer 1, @weekly the same as 7, and @monthly specifies that the job is run once a month regarless of the length of the month.
- delay in minutes — number of minutes anacron waits before executing the job The property value is defined as an integer. If the value is set to 0, no delay applies.
- job-identifier — unique name referring to a particular job used in the log files
- command — command to be executed The command can be either a command such as ls /proc » /tmp/proc or a command which executes a custom script.
Looking over the file on the RHEL6 machine, I saw the following:
[root@rhel1 ~]# cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly
It was the same as the example, anacron just takes care of any scheduling on a daily, weekly, and monthly basis. Here are some things that are run daily on the system:
[root@rhel1 ~]# ls /etc/cron.daily/
logrotate makewhatis.cron mlocate.cron rhsm-complianced
Looks like logrotate (which we discussed above), the MAN database, the locate database, and the RHEL compliance check. If we wanted we could add our own script to be daily if we needed to. Moving onto Cron:
21.1.4. Configuring Cron Jobs The configuration file for cron jobs is the /etc/crontab, which can be only modified by the root user. The file contains the following:
SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root HOME=/ # For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * username command to be executed
The first three lines contain the same variable definitions as an anacrontab file: SHELL, PATH, and MAILTO.
In addition, the file can define the HOME variable. The HOME variable defines the directory, which will be used as the home directory when executing commands or scripts run by the job.
The remaining lines in the /etc/crontab file represent scheduled jobs and have the following format:
minute hour day month day of week username command
The following define the time when the job is to be run:
- minute — any integer from 0 to 59
- hour — any integer from 0 to 23
- day — any integer from 1 to 31 (must be a valid day if a month is specified)
- month — any integer from 1 to 12 (or the short name of the month such as jan or feb)
- day of week — any integer from 0 to 7, where 0 or 7 represents Sunday (or the short name of the week such as sun or mon)
The following define other job properties:
- username — specifies the user under which the jobs are run
- command — the command to be executed
For any of the above values, an asterisk (*) can be used to specify all valid values. If you, for example, define the month value as an asterisk, the job will be executed every month within the constraints of the other values.
A hyphen (-) between integers specifies a range of integers. For example, 1-4 means the integers 1, 2, 3, and 4.
A list of values separated by commas (,) specifies a list. For example, 3, 4, 6, 8 indicates exactly these four integers.
The forward slash (/) can be used to specify step values. The value of an integer will be skipped within a range following the range with /integer. For example, minute value defined as 0-59/2 denotes every other minute in the minute field. Step values can also be used with an asterisk. For instance, if the month value is defined as */3, the task will run every third month.
Any lines that begin with a hash sign (#) are comments and are not processed.
Users other than root can configure cron tasks with the crontab utility. The user-defined crontabs are stored in the /var/spool/cron/ directory and executed as if run by the users that created them.
To create a crontab as a user, login as that user and type the command crontab -e to edit the user’s crontab with the editor specified in the VISUAL or EDITOR environment variable. The file uses the same format as /etc/crontab. When the changes to the crontab are saved, the crontab is stored according to username and written to the file /var/spool/cron/username. To list the contents of your crontab file, use the crontab -l command.
Let’s create a cron job to run a script which appends the date into a file every 5 minutes. First let’s edit the file:
[root@rhel1 ~]# vi /tmp/script.bash
then add the following into the file:
#/bin/bash
# Append date into a file
/bin/date +"%m-%d-%Y_%H.%M.%S" >> /tmp/date
Now let’s make the file executable:
[root@rhel1 ~]# chmod +x /tmp/script.bash
Now let’s run it to make sure it works as expected:
[root@rhel1 ~]# /tmp/script.bash
[root@rhel1 ~]# cat /tmp/date
06-16-2013_17.17.05
That looks good. Now let’s check the current cron table for the root user:
[root@rhel1 ~]# crontab -l
no crontab for root
We can see that it’s currently empty. Now let’s edit our cron table:
[root@rhel1 ~]# crontab -e
That will fire an editor of your choice with an empty file. Then add the following entry into the file:
*/5 * * * * /tmp/script.bash
Save the file and exit your editor. After that is done you should see the following in your cron table:
[root@rhel1 ~]# crontab -l
*/5 * * * * /tmp/script.bash
You can also see the contents of the cron table by checking out the following file:
[root@rhel1 ~]# cat /var/spool/cron/root
*/5 * * * * /tmp/script.bash
Wait 10 minutes and check out the output file and you should see the following:
[root@rhel1 ~]# cat /tmp/date
06-16-2013_17.17.05
06-16-2013_17.25.01
06-16-2013_17.30.01
We can also control access to cron:
21.1.5. Controlling Access to Cron To restrict the access to Cron, you can use the /etc/cron.allow and /etc/cron.deny files. These access control files use the same format with one username on each line. Mind that no whitespace characters are permitted in either file.
If the cron.allow file exists, only users listed in the file are allowed to use cron, and the cron.deny file is ignored.
If the cron.allow file does not exist, users listed in the cron.deny file are not allowed to use Cron.
The Cron daemon (crond) does not have to be restarted if the access control files are modified. The access control files are checked each time a user tries to add or delete a cron job.
The root user can always use cron, regardless of the usernames listed in the access control files.
At and Batch
From the deployment guide:
21.2. At and Batch While Cron is used to schedule recurring tasks, the At utility is used to schedule a one-time task at a specific time and the Batch utility is used to schedule a one-time task to be executed when the system load average drops below 0.8.
21.2.1. Installing At and Batch To determine if the at package is already installed on your system, issue the rpm -q at command. The command returns the full name of the at package if already installed or notifies you that the package is not available.
To install At and Batch, type the following at a shell prompt:
~]# yum install at
21.2.2. Running the At Service>br> The At and Batch jobs are both picked by the atd service. This section provides information on how to start, stop, and restart the atd service, and shows how to enable it in a particular runlevel.
21.2.2.1. Starting and Stopping the At Service To determine if the service is running, use the command service atd status.
To run the atd service in the current session, type the following at a shell prompt as root:
service atd start
To configure the service to start automatically at boot, use the following command:
chkconfig atd on
This command enables the service in runlevel 2, 3, 4, and 5
So let’s see if atd is installed:
[root@rhel1 ~]# rpm -q at
package at is not installed
It looks like I don’t have it installed. So let’s go ahead and install it:
[root@rhel1 ~]# yum install at
..
..
Installed:
at.i686 0:3.1.10-43.el6
Complete!
[root@rhel1 ~]#
Not let’s make sure it enabled and running:
[root@rhel1 ~]# service atd status
atd is stopped
[root@rhel1 ~]# chkconfig --list atd
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
It’s enabled but it’s not running. So let’s start it:
[root@rhel1 ~]# service atd start
Starting atd: atd
[root@rhel1 ~]# service atd status
atd (pid 1532) is running...
That looks good. Now let’s keep down the deployment guide:
21.2.3. Configuring an At Job To schedule a one-time job for a specific time with the At utility, do the following:
On the command line, type the command at TIME, where TIME is the time when the command is to be executed. The TIME argument can be defined in any of the following formats:
- HH:MM specifies the exact hour and minute; For example, 04:00 specifies 4:00 a.m.
- midnight specifies 12:00 a.m.
- noon specifies 12:00 p.m.
- teatime specifies 4:00 p.m.
- MONTHDAYYEAR format; For example, January 15 2012 specifies the 15th day of January in the year 2012. The year value is optional.
- MMDDYY, MM/DD/YY, or MM.DD.YY formats; For example, 011512 for the 15th day of January in the year 2012.
- now + TIME where TIME is defined as an integer and the value type: minutes, hours, days, or weeks. For example, now + 5 days specifies that the command will be executed at the same time five days from now. The time must be specified first, followed by the optional date. For more information about the time format, refer to the /usr/share/doc/at-VERSION/timespec text file.
If the specified time has past, the job is executed at the time the next day.
In the displayed at> prompt, define the job commands:
- Type the command the job should execute and press Enter. Optionally, repeat the step to provide multiple commands.
- Enter a shell script at the prompt and press Enter after each line in the script. The job will use the shell set in the user’s SHELL environment, the user’s login shell, or /bin/sh (whichever is found first).
- Once finished, press Ctrl+D on an empty line to exit the prompt.
If the set of commands or the script tries to display information to standard output, the output is emailed to the user.
To view the list of pending jobs, use the atq command.
First let’s remove our cron table:
[root@rhel1 ~]# crontab -r
[root@rhel1 ~]# crontab -l
no crontab for root
The table is now empty. Also let’s remove the output file:
[root@rhel1 ~]# rm /tmp/date
Now let’s run the same command, that we run from cron, 2 minutes from now:
[root@rhel1 ~]# date; at now + 2 minutes
Sun Jun 16 18:15:52 MDT 2013
at> /tmp/script.bash
at> EOT
job 1 at 2013-06-16 18:17
We can see that I started at 18:15 and the job will be run at 18:17. Let’s also check the queue:
[root@rhel1 ~]# atq
1 2013-06-16 18:17 a root
When the time has passed, we can check out the file:
[root@rhel1 ~]# cat /tmp/date
06-16-2013_18.17.00
and the date is correct. The batch utility works in the same way except you don’t specify the time but the process is the same other than that. You can also block access to at just like with cron, from the deployment guide:
21.2.7. Controlling Access to At and Batch You can restrict the access to the at and batch commands using the /etc/at.allow and /etc/at.deny files. These access control files use the same format defining one username on each line. Mind that no whitespace are permitted in either file.
If the file at.allow exists, only users listed in the file are allowed to use at or batch, and the at.deny file is ignored.
If at.allow does not exist, users listed in at.deny are not allowed to use at or batch.
The at daemon (atd) does not have to be restarted if the access control files are modified. The access control files are read each time a user tries to execute the at or batch commands.
The root user can always execute at and batch commands, regardless of the content of the access control files.