Tools: BASIC
# uptime
23:27:29 up 4 min, 2 users, load average: 0.07, 0.26, 0.13
The load average counts runnable threads on CPU, or runnable and waiting. Includes tasks blocked in disk
I/O.
These are exponentially damped moving averages, with time constraints of 1, 5 and 15 minutes.
If the load is greater than CPU count, (CPUs are 100% utilized) and threads are suffering scheduler latency.
That can be disk I/O factor too.
# top
# iostat
%util depends on the target – virtual devices backed up by multiple disks may accept work at 100% utilization.
Calculates I/O controller stats by summing their devices.
Would like to see disk errors add “-e”.
# vmstat
#swapon
Shows which device is used for swap.
$ swapon -s
# lsof
# free
[root@rhel7 ~] # free
total used free shared buff/cache available
Mem: 1867292 592332 752844 10880 522116 1036852
Swap: 2097148 0 2097148
# ping
Simple ICMP test.
# nicstat
# dstat
The output above indicates:
CPU stats: cpu usage by a user (usr) processes, system (sys) processes, as well as the number of idle (idl) and
waiting (wai) processes, hard interrupt (hiq) and soft interrupt (siq).
Disk stats: total number of read (read) and write (writ) operations on disks.
Network stats: total amount of bytes received (recv) and sent (send) on network interfaces.
Paging stats: number of times information is copied into (in) and moved out (out) of memory.
System stats: number of interrupts (int) and context switches (csw).
Tools: INTERMEDIATE
tps – Transactions per second (this includes both read and write)
rtps – Read transactions per second
wtps – Write transactions per second
bread/s – Bytes read per second
bwrtn/s – Bytes written per second
$ sar -d (for individual block device statistics) (can also use sar -p -d)
$ sar -w (for the total number of processes created per second, and total number of context switches per second)
$ sar -q (for run queue size and load average)
$ sar -n <KEYWORD> (for network statistics)
DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
ALL – This displays all the above information. The output will be very long.
# netstat
Gets network protocol statistics
$ netstat -na (to list all the connections and ports)
$ netstat -s (for getting individual statistics report)
and so on….
# pidstat
For breaking down the processes. The pidstat command is used for monitoring individual tasks currently being
managed by the Linux kernel.
$ pidstat 1
$ pidstat -drl
-d: reports I/O stats
-r: page faults and memory utilization
-l: command name and all its args.
-p <PID>: allows to get the data for a specific process.
-u: CPU utilization
$ pidstat -r -p xxxx 2 5
$ pidstat -C “fox|bird” -r -p ALL
$ pidstat -T CHILD -r 2 5
minflt/s: Total number of minor faults the task has made per second, those which have not required loading a memory
page from disk.
majflt/s: Total number of major faults the task has made per second, those which have required loading a memory
page from disk.
VSZ: virtual size
RSS: resident Set Size: non swappable physical memory
StkSize: memory reserverd for task as stack
StkRef: memory used as stack
# strace
to debug a process
strace /path/executable-name <arguments-if-any>
to print a summary
strace -c dd if=/dev/zero of=/dev/null bs=512 count=1024k
# gdb
# tcpdump
$ tcpdump -n (To get the hostnames and ports)
-vv (very verbose)
-i <int_name> (to specify an interface)
-i any (will listen to all interfaces)
-w /path/to/file (to save the output to a file)
-r /path/to/file (to read the file)
-s <number> (specify capture size of each packets)
-c <number>: to specify number of packets to be captured
# tcpdump -w /var/tmp/tcpdata.pcap -i any -c 10 -vvv
# blktrace
Allows block device event tracing and investigating I/O latency. Need to mount debugfs on /sys/kernel/debug
$ mount -t debugfs none /sys/kernel/debug
$ btrace /dev/sdb -w N_seconds -n N_buffer -b buff_size
# iotop
Shows disk I/O by process.
The IO> is the time the thread was waiting on I/O
CONFIG_TASK_IO_ACCOUNTING needs to be enabled for this to work.
$ iotop -bod 5
# slabtop
Shows where kernel memory is being utilized (check for CACHE SIZE)
$ slaptop -sc
# sysctl
The /sbin/sysctl command is used to view, set, and automate kernel settings in the /proc/sys/ directory.
/proc
Debug:
Obtain stack trace, dynamic object dependencies, address map, open file descriptors of the processes & using the gcore utility get a core
file of the process