Resources: Disk, Memory, CPU

Disk full triage
free: available vs free vs buff/cache
Load average vs nproc (cores)

Must-know cold

df -h · df -i · lsblk -f · du -xh --max-depth=1 /var | sort -rh
free -h → watch available, not only free · swapon --show
uptime / load · nproc · lscpu · vmstat -y 1 5
Full disk breaks “unrelated” services (logs, temp, deploys)

Toolchain map

Definition: Resource RCA is six jobs. df is mounted capacity; lsblk is block devices; du is a tree. Do not treat them as one command.

Job	Tools
1 Space & mounts	`df`, `du`, `lsblk`, `findmnt`
2 Memory & swap	`free`, `swapon --show`
3 CPU inventory	`lscpu`, `nproc`, `uptime`
4 Live samples	`top` / `htop`, `vmstat`, `mpstat`, `iostat`
5 Per-process	`ps`, `pidstat`, `pmap`
6 Holders / pressure	`lsof`, `fuser`, `/proc/pressure/*`

Quick interpret

Definition: How to read full disk, inode exhaustion, memory pressure, and high load.

Signal	Meaning
`df` Use% → 100%	Unprivileged writes fail (ext reserved blocks can bite before 100%; root may still write)
`df -i` 100%	No inodes (many tiny files) even if space remains
`free` low available	Memory pressure risk (new work may swap/fail)
Load ≫ `nproc`	Oversubscribed (runnable or uninterruptible I/O) — not “CPU %”
High `wa` in top/vmstat	I/O wait

Commands

`df`

Definition: Report free disk space and capacity per mounted filesystem.

Option	Argument	Meaning	Example
(none)	—	Disk space per filesystem	`df`
`-h`	—	Human sizes (G/M)	`df -h`
`-T`	—	Print filesystem type	`df -hT`
`-i`	—	Inode usage instead of block space	`df -i`
`-h`	`PATH`	FS containing PATH	`df -h /var`
`--output`	`FIELDS`	Custom columns (GNU)	`df -h --output=source,size,used,avail,pcent,target`
`-l`	—	Local only (skip NFS)	`df -hl`
`-t`	`TYPE`	Only type	`df -t ext4`
`-x`	`TYPE`	Exclude type	`df -x tmpfs`

Flag combos

Combo	Meaning	Example
`df -h`	First look at block space	Always
`df -i`	Inode table (run separately; `-i` replaces size columns)	Full mystery
`df -h /var/lib/jenkins`	Filesystem that contains that path	Agents

`du`

Definition: Estimate disk space used by files and directories.

Option	Argument	Meaning	Example
(none)	`PATH`	Disk usage of tree	`du dir`
`-h`	`PATH`	Human sizes	`du -h dir`
`-s`	`PATH`	Summary total only	`du -sh dir`
`-a`	`PATH`	All files not just dirs	`du -ah dir`
`-x`	`PATH`	One filesystem only (no other mounts)	`du -xh /`
`-d` / `--max-depth`	`N PATH`	Limit depth	`du -h --max-depth=1 /var`
`-c`	`PATH`	Grand total	`du -ch *`
`--inodes`	`PATH`	Count inodes (GNU)	`du --inodes -s *`

Flag combos

Combo	Meaning	Example
`du -xh --max-depth=1 /var \| sort -rh`	Biggest dirs on same FS	Disk full
`du -sh * \| sort -rh`	Here, largest first
`du -sh /var/lib/docker`	Known fat path	Agents

`lsblk`

Definition: List block devices from sysfs/udev (lsblk(8)). Tree of disks → partitions → LVM/mapper. Default columns change; scripts should pass -o. RAM disks omitted unless -a.

Option	Meaning	Example
(none)	Tree of block devices	`lsblk`
`-f`	FS columns (FSTYPE, LABEL, UUID, FSUSE%, MOUNTPOINTS)	`lsblk -f`
`-p`	Full `/dev` paths	`lsblk -p`
`-l`	List (no tree); each device once (util-linux ≥ 2.34)	`lsblk -l`
`-o LIST`	Columns (`NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS`)	`lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT`
`-d`	Device only, no holders/slaves	`lsblk -d /dev/sda`
`-t`	Topology (rotational, scheduler, sector size)	`lsblk -t`

Authoritative FS probe is blkid. MOUNTPOINT is one path; MOUNTPOINTS lists all (bind mounts).

`findmnt`

Definition: Find a filesystem / list mounts (findmnt(8)). Default source is kernel /proc/self/mountinfo. --target PATH walks up to the mount of that path (like df PATH).

Option	Meaning	Example
(none)	All mounts, tree	`findmnt`
`-T` / `--target`	FS containing PATH	`findmnt -T /var/lib/jenkins`
`-t TYPE`	Filter type	`findmnt -t ext4,xfs`
`-D`	df-like columns (excludes pseudo FS)	`findmnt -D`
`-s` / `--fstab`	Search `/etc/fstab`	`findmnt --fstab -t nfs`
`-n -o TARGET,SOURCE`	Stable columns for scripts

Default output is not stable — use -o. Exit 1 if nothing matches.

`free`

Definition: Display RAM and swap usage (watch available, not only free).

Option	Argument	Meaning	Example
(none)	—	Memory + swap	`free`
`-h`	—	Human	`free -h`
`-m`	—	Mebibytes	`free -m`
`-g`	—	Gibibytes	`free -g`
`-s`	`N`	Refresh every N seconds	`free -h -s 2`
`-c`	`N`	With `-s`: N times then exit	`free -h -s 1 -c 5`
`-w`	—	Wide (buffers/cache separate)	`free -wh`
`-t`	—	Total line	`free -ht`

Columns: total · used · free · shared · buff/cache · available (what matters for new work)

Flag combos

Combo	Meaning	Example
`free -h`	Default interview command
`free -h -s 2`	Watch pressure live

`swapon`

Definition: Show or enable swap (swapon(8)). --show is the table; -s / cat /proc/swaps is deprecated in favour of --show. swapoff can fail with OOM (exit 2 since util-linux 2.36).

Option	Meaning	Example
`--show`	Swap areas (size, used, prio)	`swapon --show`
`-a`	Enable all fstab swap (skip `noauto`)	Boot scripts
`-s`	Old summary ≡ `/proc/swaps`	Prefer `--show`

Priority (pri=) 0–32767, higher first. Swap files with holes / CoW (plain Btrfs) are rejected; Btrfs needs nocow (kernel ≥ 5.0).

`uptime`

Definition: Show how long the system has been up and load averages.

Option	Argument	Meaning	Example
(none)	—	Up time, users, load average 1/5/15 min	`uptime`

Load ≈ tasks runnable (R) + uninterruptible (D), 1/5/15 min averages. Compare to nproc (available units, often SMT threads). Load 8 on 8 units ≈ busy; load 8 on 2 units ≈ overloaded. High load with low %us is often I/O (wa / D), not CPU.

Flag combos

Combo	Meaning	Example
`watch -n 1 uptime`	Live load	While stressing

`nproc`

Definition: Print processing units available to this process (affinity / cgroup / OMP_*), not “physical cores.” Counts SMT threads as units.

Option	Argument	Meaning	Example
(none)	—	Units this process may use	`nproc`
`--all`	—	Installed processors (ignores cgroup quota / OpenMP)	`nproc --all`

Flag combos

Combo	Meaning	Example
`echo "load $(uptime); cores $(nproc)"`	Context for load	Spoken RCA

`lscpu`

Definition: CPU architecture from sysfs + /proc/cpuinfo (lscpu(1)). Reports guest topology in a VM (often not the host). Cache sizes are a summary across CPUs (not per-core since util-linux 2.34). Default output changes — do not parse it.

Option	Meaning	Example
(none)	Human summary (sockets, cores, threads, NUMA, model)	`lscpu`
`-e`	Per-CPU table	`lscpu -e`
`-p`	Parseable	Scripts
`-C`	Cache details	`lscpu -C`
`-y`	Physical IDs from the kernel

Compare CPU(s) here with nproc (available to this process) and load from uptime.

`vmstat`

Definition: Report virtual memory, processes, and CPU/I/O samples over time. From procps; install if missing.

Option	Argument	Meaning	Example
(none)	—	Since-boot averages for I/O/CPU (procs/memory are current)	Ignore as “now”
(none)	`N`	Then a sample every N seconds	`vmstat 1`
(none)	`N C`	Every N sec, C reports (first is still since-boot)	`vmstat 1 5`
`-y`	—	Omit the since-boot first line (procps)	`vmstat -y 1 5`
`-s`	—	Event counters summary	`vmstat -s`
`-d`	—	Disk stats	`vmstat -d`
`-w`	—	Wide columns	`vmstat -w 1`

CPU columns: us user · sy system · id idle · wa I/O wait · st steal (VM)

Flag combos

Combo	Meaning	Example
`vmstat -y 1 10`	10 live samples (skip since-boot line)	High load RCA

`iostat`

Definition: Show CPU and per-device disk I/O statistics. Optional: sysstat package.

Option	Argument	Meaning	Example
(none)	—	CPU + device I/O (since boot if no interval/`-y`)	`iostat`
`-x`	—	Extended device stats (`await`, `%util`, …)	`iostat -x`
`-y`	—	Omit the since-boot first report	`iostat -yxz 1 5`
`-h` / `-m`	—	Human / MiB units (version-dependent)	`iostat -xm 1`
(none)	`N C`	Interval + count (first report is since-boot unless `-y`)	`iostat -x 1 5`
`-p`	`DEV`	Device and its partitions	`iostat -xp sda 1`

Flag combos

Combo	Meaning	Example
`iostat -yxz 1 5`	Extended, skip idle devices, skip since-boot	Disk latency (`%util`≈100% saturates serial disks; not SSDs)

`mpstat`

Definition: Per-processor utilisation (mpstat(1), sysstat). No interval → since boot. Default report is CPU (-u). %iowait = idle with outstanding disk I/O; %idle = idle without. %sys does not include IRQ/softirq (%irq / %soft).

Option	Meaning	Example
`interval [count]`	Sample (first line is still since-boot unless you ignore it)	`mpstat 1 5`
`-P ALL`	Every CPU + `all` average	`mpstat -P ALL 1 5`
`-u`	Utilisation (`%usr` `%nice` `%sys` `%iowait` `%irq` `%soft` `%steal` `%idle`)	Default
`-I SUM`	Interrupts/s per CPU
`-A`	`-n -u -I ALL` (implies `-P ALL -N ALL` unless set)

Offline CPUs are omitted. Needs /proc.

`pidstat`

Definition: Per-task stats (pidstat(1), sysstat). No -p → active tasks only (non-zero stats), not every PID. No interval → since boot. Default metric is CPU (-u). Units labelled kB are KiB.

Option	Meaning	Example
`1 5`	Five CPU reports, 1 s apart, active tasks	`pidstat 1 5`
`-p PID\|ALL`	One PID, or every task (including idle)	`pidstat -p 1643 1 5`
`-u`	`%usr` `%system` `%guest` `%wait` `%CPU`	Default
`-r`	minflt/majflt, VSZ, RSS, `%MEM`	`pidstat -r 1 5`
`-d`	I/O kB_rd/s kB_wr/s (kernel ≥ 2.6.20)	Disk hogs
`-t`	Threads (TGID/TID)
`-l`	Command and args
`-I`	Divide `%CPU` by CPU count (SMP)

pidstat is live rates over the interval — closer to top than ps %cpu.

`lsof`

Definition: List open files/sockets by process; find deleted-but-open space hogs.

Option	Argument	Meaning	Example
(none)	—	List open files (huge)	Avoid bare
`-p`	`PID`	Files for PID	`lsof -p 1234`
`+L1`	—	Open files with link count 0 (deleted but open)	`sudo lsof +L1`
`-i`	—	Network files	See networking sheet
`-n`	—	No DNS resolve	`lsof -n`
`-u`	`USER`	By user	`lsof -u jenkins`

Flag combos

Combo	Meaning	Example
`sudo lsof +L1 \| head`	Space held after delete	`df` full, `du` small
`sudo lsof /var \| head`	Who uses /var

`fuser`

Definition: fuser names processes using a file or mount. Linux PSI (/proc/pressure/{cpu,memory,io}) is stall time, not utilisation.

Command	Meaning	Example
`sudo fuser -vm /var`	PIDs on that mount (`-v` verbose, `-m` name is a mount)	Unmount busy
`cat /proc/pressure/memory`	`some`/`full` `avg10=` `avg60=` `avg300=`	Memory stall
`cat /proc/pressure/io`	I/O stall	High `wa` correlation
`cat /proc/pressure/cpu`	Runnable but no CPU	Load vs `nproc`

avg10 is a 10 s average of the fraction of time some (or all) tasks were stalled. Missing files → kernel built without PSI.

Common recipes

Goal	Command
Space overview	`df -h`
Inodes	`df -i`
Largest under /var	`sudo du -xh --max-depth=1 /var \| sort -rh \| head`
Memory	`free -h`
Load vs cores	`uptime; nproc`
CPU/I/O samples	`vmstat -y 1 5`
Deleted-open space	`sudo lsof +L1`
Block devices	`lsblk -f`
Mount of a path	`findmnt -T /var/lib/jenkins`
Swap	`swapon --show`
CPU topology	`lscpu`
Per-CPU live	`mpstat -P ALL 1 5`
One PID I/O	`pidstat -d -p PID 1 5`
PSI	`cat /proc/pressure/memory`
OOM evidence	`journalctl -k --since "today" \| grep -iE 'oom\|killed process'`

Pitfalls

available is the estimate for new work (includes reclaimable cache, excludes some unevictable). Low free alone ≠ crisis. Modern used ≈ total − available.
df -i replaces the space columns; it is not “space plus inodes.” Run df -h and df -i.
ext-style reserved blocks (often ~5% for root) make unprivileged writes fail before Use% looks “full.”
du does not see space held by deleted-but-open files — use lsof +L1 and restart/truncate the holder.
du without -x walks other mounts (NFS, extra disks). Same-device bind mounts can still double-count even with -x.
du -sh * skips hidden (.*) names.
First line of vmstat / iostat is since boot unless you pass -y. Do not RCA from that line.
ps %CPU is lifetime CPU/elapsed, not the live burn top shows.
nproc is units this process may use (cgroup/affinity/OpenMP), not socket×core. --all ignores quota.
rm -rf large trees under load can make I/O worse — prefer targeted cleanup.
Load high + low CPU user% → check wa, NFS, D state, mpstat %iowait, /proc/pressure/io.
lsblk default columns change; so does lscpu / findmnt. Pin -o / do not parse the pretty summary.
mpstat / pidstat with no interval are since boot, same class of trap as vmstat / iostat.
pidstat without -p hides idle tasks. pidstat -p ALL is everyone.
nproc ≠ lscpu “CPU(s)” in a cgroup. lscpu in a VM is the guest.
swapon -s is deprecated; swapon --show. swapoff can OOM (exit 2).

For more details, try man <command> in your terminal.

Resources: Disk, Memory, CPU

Disk full triage
free: available vs free vs buff/cache
Load average vs nproc (cores)

Must-know cold

df -h · df -i · lsblk -f · du -xh --max-depth=1 /var | sort -rh
free -h → watch available, not only free · swapon --show
uptime / load · nproc · lscpu · vmstat -y 1 5
Full disk breaks “unrelated” services (logs, temp, deploys)

Toolchain map

Definition: Resource RCA is six jobs. df is mounted capacity; lsblk is block devices; du is a tree. Do not treat them as one command.

Job	Tools
1 Space & mounts	`df`, `du`, `lsblk`, `findmnt`
2 Memory & swap	`free`, `swapon --show`
3 CPU inventory	`lscpu`, `nproc`, `uptime`
4 Live samples	`top` / `htop`, `vmstat`, `mpstat`, `iostat`
5 Per-process	`ps`, `pidstat`, `pmap`
6 Holders / pressure	`lsof`, `fuser`, `/proc/pressure/*`

Quick interpret

Definition: How to read full disk, inode exhaustion, memory pressure, and high load.

Signal	Meaning
`df` Use% → 100%	Unprivileged writes fail (ext reserved blocks can bite before 100%; root may still write)
`df -i` 100%	No inodes (many tiny files) even if space remains
`free` low available	Memory pressure risk (new work may swap/fail)
Load ≫ `nproc`	Oversubscribed (runnable or uninterruptible I/O) — not “CPU %”
High `wa` in top/vmstat	I/O wait

Commands

`df`

Definition: Report free disk space and capacity per mounted filesystem.

Option	Argument	Meaning	Example
(none)	—	Disk space per filesystem	`df`
`-h`	—	Human sizes (G/M)	`df -h`
`-T`	—	Print filesystem type	`df -hT`
`-i`	—	Inode usage instead of block space	`df -i`
`-h`	`PATH`	FS containing PATH	`df -h /var`
`--output`	`FIELDS`	Custom columns (GNU)	`df -h --output=source,size,used,avail,pcent,target`
`-l`	—	Local only (skip NFS)	`df -hl`
`-t`	`TYPE`	Only type	`df -t ext4`
`-x`	`TYPE`	Exclude type	`df -x tmpfs`

Flag combos

Combo	Meaning	Example
`df -h`	First look at block space	Always
`df -i`	Inode table (run separately; `-i` replaces size columns)	Full mystery
`df -h /var/lib/jenkins`	Filesystem that contains that path	Agents

`du`

Definition: Estimate disk space used by files and directories.

Option	Argument	Meaning	Example
(none)	`PATH`	Disk usage of tree	`du dir`
`-h`	`PATH`	Human sizes	`du -h dir`
`-s`	`PATH`	Summary total only	`du -sh dir`
`-a`	`PATH`	All files not just dirs	`du -ah dir`
`-x`	`PATH`	One filesystem only (no other mounts)	`du -xh /`
`-d` / `--max-depth`	`N PATH`	Limit depth	`du -h --max-depth=1 /var`
`-c`	`PATH`	Grand total	`du -ch *`
`--inodes`	`PATH`	Count inodes (GNU)	`du --inodes -s *`

Flag combos

Combo	Meaning	Example
`du -xh --max-depth=1 /var \| sort -rh`	Biggest dirs on same FS	Disk full
`du -sh * \| sort -rh`	Here, largest first
`du -sh /var/lib/docker`	Known fat path	Agents

`lsblk`

Definition: List block devices from sysfs/udev (lsblk(8)). Tree of disks → partitions → LVM/mapper. Default columns change; scripts should pass -o. RAM disks omitted unless -a.

Option	Meaning	Example
(none)	Tree of block devices	`lsblk`
`-f`	FS columns (FSTYPE, LABEL, UUID, FSUSE%, MOUNTPOINTS)	`lsblk -f`
`-p`	Full `/dev` paths	`lsblk -p`
`-l`	List (no tree); each device once (util-linux ≥ 2.34)	`lsblk -l`
`-o LIST`	Columns (`NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS`)	`lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT`
`-d`	Device only, no holders/slaves	`lsblk -d /dev/sda`
`-t`	Topology (rotational, scheduler, sector size)	`lsblk -t`

Authoritative FS probe is blkid. MOUNTPOINT is one path; MOUNTPOINTS lists all (bind mounts).

`findmnt`

Definition: Find a filesystem / list mounts (findmnt(8)). Default source is kernel /proc/self/mountinfo. --target PATH walks up to the mount of that path (like df PATH).

Option	Meaning	Example
(none)	All mounts, tree	`findmnt`
`-T` / `--target`	FS containing PATH	`findmnt -T /var/lib/jenkins`
`-t TYPE`	Filter type	`findmnt -t ext4,xfs`
`-D`	df-like columns (excludes pseudo FS)	`findmnt -D`
`-s` / `--fstab`	Search `/etc/fstab`	`findmnt --fstab -t nfs`
`-n -o TARGET,SOURCE`	Stable columns for scripts

Default output is not stable — use -o. Exit 1 if nothing matches.

`free`

Definition: Display RAM and swap usage (watch available, not only free).

Option	Argument	Meaning	Example
(none)	—	Memory + swap	`free`
`-h`	—	Human	`free -h`
`-m`	—	Mebibytes	`free -m`
`-g`	—	Gibibytes	`free -g`
`-s`	`N`	Refresh every N seconds	`free -h -s 2`
`-c`	`N`	With `-s`: N times then exit	`free -h -s 1 -c 5`
`-w`	—	Wide (buffers/cache separate)	`free -wh`
`-t`	—	Total line	`free -ht`

Columns: total · used · free · shared · buff/cache · available (what matters for new work)

Flag combos

Combo	Meaning	Example
`free -h`	Default interview command
`free -h -s 2`	Watch pressure live

`swapon`

Definition: Show or enable swap (swapon(8)). --show is the table; -s / cat /proc/swaps is deprecated in favour of --show. swapoff can fail with OOM (exit 2 since util-linux 2.36).

Option	Meaning	Example
`--show`	Swap areas (size, used, prio)	`swapon --show`
`-a`	Enable all fstab swap (skip `noauto`)	Boot scripts
`-s`	Old summary ≡ `/proc/swaps`	Prefer `--show`

Priority (pri=) 0–32767, higher first. Swap files with holes / CoW (plain Btrfs) are rejected; Btrfs needs nocow (kernel ≥ 5.0).

`uptime`

Definition: Show how long the system has been up and load averages.

Option	Argument	Meaning	Example
(none)	—	Up time, users, load average 1/5/15 min	`uptime`

Flag combos

Combo	Meaning	Example
`watch -n 1 uptime`	Live load	While stressing

`nproc`

Definition: Print processing units available to this process (affinity / cgroup / OMP_*), not “physical cores.” Counts SMT threads as units.

Option	Argument	Meaning	Example
(none)	—	Units this process may use	`nproc`
`--all`	—	Installed processors (ignores cgroup quota / OpenMP)	`nproc --all`

Flag combos

Combo	Meaning	Example
`echo "load $(uptime); cores $(nproc)"`	Context for load	Spoken RCA

`lscpu`

Definition: CPU architecture from sysfs + /proc/cpuinfo (lscpu(1)). Reports guest topology in a VM (often not the host). Cache sizes are a summary across CPUs (not per-core since util-linux 2.34). Default output changes — do not parse it.

Option	Meaning	Example
(none)	Human summary (sockets, cores, threads, NUMA, model)	`lscpu`
`-e`	Per-CPU table	`lscpu -e`
`-p`	Parseable	Scripts
`-C`	Cache details	`lscpu -C`
`-y`	Physical IDs from the kernel

Compare CPU(s) here with nproc (available to this process) and load from uptime.

`vmstat`

Definition: Report virtual memory, processes, and CPU/I/O samples over time. From procps; install if missing.

Option	Argument	Meaning	Example
(none)	—	Since-boot averages for I/O/CPU (procs/memory are current)	Ignore as “now”
(none)	`N`	Then a sample every N seconds	`vmstat 1`
(none)	`N C`	Every N sec, C reports (first is still since-boot)	`vmstat 1 5`
`-y`	—	Omit the since-boot first line (procps)	`vmstat -y 1 5`
`-s`	—	Event counters summary	`vmstat -s`
`-d`	—	Disk stats	`vmstat -d`
`-w`	—	Wide columns	`vmstat -w 1`

CPU columns: us user · sy system · id idle · wa I/O wait · st steal (VM)

Flag combos

Combo	Meaning	Example
`vmstat -y 1 10`	10 live samples (skip since-boot line)	High load RCA

`iostat`

Definition: Show CPU and per-device disk I/O statistics. Optional: sysstat package.

Option	Argument	Meaning	Example
(none)	—	CPU + device I/O (since boot if no interval/`-y`)	`iostat`
`-x`	—	Extended device stats (`await`, `%util`, …)	`iostat -x`
`-y`	—	Omit the since-boot first report	`iostat -yxz 1 5`
`-h` / `-m`	—	Human / MiB units (version-dependent)	`iostat -xm 1`
(none)	`N C`	Interval + count (first report is since-boot unless `-y`)	`iostat -x 1 5`
`-p`	`DEV`	Device and its partitions	`iostat -xp sda 1`

Flag combos

Combo	Meaning	Example
`iostat -yxz 1 5`	Extended, skip idle devices, skip since-boot	Disk latency (`%util`≈100% saturates serial disks; not SSDs)

`mpstat`

Definition: Per-processor utilisation (mpstat(1), sysstat). No interval → since boot. Default report is CPU (-u). %iowait = idle with outstanding disk I/O; %idle = idle without. %sys does not include IRQ/softirq (%irq / %soft).

Option	Meaning	Example
`interval [count]`	Sample (first line is still since-boot unless you ignore it)	`mpstat 1 5`
`-P ALL`	Every CPU + `all` average	`mpstat -P ALL 1 5`
`-u`	Utilisation (`%usr` `%nice` `%sys` `%iowait` `%irq` `%soft` `%steal` `%idle`)	Default
`-I SUM`	Interrupts/s per CPU
`-A`	`-n -u -I ALL` (implies `-P ALL -N ALL` unless set)

Offline CPUs are omitted. Needs /proc.

`pidstat`

Definition: Per-task stats (pidstat(1), sysstat). No -p → active tasks only (non-zero stats), not every PID. No interval → since boot. Default metric is CPU (-u). Units labelled kB are KiB.

Option	Meaning	Example
`1 5`	Five CPU reports, 1 s apart, active tasks	`pidstat 1 5`
`-p PID\|ALL`	One PID, or every task (including idle)	`pidstat -p 1643 1 5`
`-u`	`%usr` `%system` `%guest` `%wait` `%CPU`	Default
`-r`	minflt/majflt, VSZ, RSS, `%MEM`	`pidstat -r 1 5`
`-d`	I/O kB_rd/s kB_wr/s (kernel ≥ 2.6.20)	Disk hogs
`-t`	Threads (TGID/TID)
`-l`	Command and args
`-I`	Divide `%CPU` by CPU count (SMP)

pidstat is live rates over the interval — closer to top than ps %cpu.

`lsof`

Definition: List open files/sockets by process; find deleted-but-open space hogs.

Option	Argument	Meaning	Example
(none)	—	List open files (huge)	Avoid bare
`-p`	`PID`	Files for PID	`lsof -p 1234`
`+L1`	—	Open files with link count 0 (deleted but open)	`sudo lsof +L1`
`-i`	—	Network files	See networking sheet
`-n`	—	No DNS resolve	`lsof -n`
`-u`	`USER`	By user	`lsof -u jenkins`

Flag combos

Combo	Meaning	Example
`sudo lsof +L1 \| head`	Space held after delete	`df` full, `du` small
`sudo lsof /var \| head`	Who uses /var

`fuser`

Definition: fuser names processes using a file or mount. Linux PSI (/proc/pressure/{cpu,memory,io}) is stall time, not utilisation.

Command	Meaning	Example
`sudo fuser -vm /var`	PIDs on that mount (`-v` verbose, `-m` name is a mount)	Unmount busy
`cat /proc/pressure/memory`	`some`/`full` `avg10=` `avg60=` `avg300=`	Memory stall
`cat /proc/pressure/io`	I/O stall	High `wa` correlation
`cat /proc/pressure/cpu`	Runnable but no CPU	Load vs `nproc`

avg10 is a 10 s average of the fraction of time some (or all) tasks were stalled. Missing files → kernel built without PSI.

Common recipes

Goal	Command
Space overview	`df -h`
Inodes	`df -i`
Largest under /var	`sudo du -xh --max-depth=1 /var \| sort -rh \| head`
Memory	`free -h`
Load vs cores	`uptime; nproc`
CPU/I/O samples	`vmstat -y 1 5`
Deleted-open space	`sudo lsof +L1`
Block devices	`lsblk -f`
Mount of a path	`findmnt -T /var/lib/jenkins`
Swap	`swapon --show`
CPU topology	`lscpu`
Per-CPU live	`mpstat -P ALL 1 5`
One PID I/O	`pidstat -d -p PID 1 5`
PSI	`cat /proc/pressure/memory`
OOM evidence	`journalctl -k --since "today" \| grep -iE 'oom\|killed process'`

Pitfalls

available is the estimate for new work (includes reclaimable cache, excludes some unevictable). Low free alone ≠ crisis. Modern used ≈ total − available.
df -i replaces the space columns; it is not “space plus inodes.” Run df -h and df -i.
ext-style reserved blocks (often ~5% for root) make unprivileged writes fail before Use% looks “full.”
du does not see space held by deleted-but-open files — use lsof +L1 and restart/truncate the holder.
du without -x walks other mounts (NFS, extra disks). Same-device bind mounts can still double-count even with -x.
du -sh * skips hidden (.*) names.
First line of vmstat / iostat is since boot unless you pass -y. Do not RCA from that line.
ps %CPU is lifetime CPU/elapsed, not the live burn top shows.
nproc is units this process may use (cgroup/affinity/OpenMP), not socket×core. --all ignores quota.
rm -rf large trees under load can make I/O worse — prefer targeted cleanup.
Load high + low CPU user% → check wa, NFS, D state, mpstat %iowait, /proc/pressure/io.
lsblk default columns change; so does lscpu / findmnt. Pin -o / do not parse the pretty summary.
mpstat / pidstat with no interval are since boot, same class of trap as vmstat / iostat.
pidstat without -p hides idle tasks. pidstat -p ALL is everyone.
nproc ≠ lscpu “CPU(s)” in a cgroup. lscpu in a VM is the guest.
swapon -s is deprecated; swapon --show. swapoff can OOM (exit 2).

For more details, try man <command> in your terminal.