Text Processing

Must-know cold

• grep -i PATTERN FILE · grep -v · grep -c · grep -n · grep -E · grep -F · grep -r
• Prefer grep -E (ERE: + ? | () {n,m}). Bare grep is BRE, not ERE.
• grep -Eo 'host=[^ ]+' · grep -En 'ERROR|WARN|FATAL' · grep -Ev 'level=INFO'
• sed -i.bak 's/old/new/g' FILE · sed -n '10,20p'
• awk '{print $1}' · awk -F',' '{print $2}'
• sort | uniq -c | sort -nr | head
• Shape: extract → transform → aggregate

Commands

grep

Definition: Print lines matching a pattern (basic regex by default).

Flag combos

grep -E

Definition: grep with extended regex (+, ?, |, (), {n,m}) for flexible log/code search.

Why default to -E: Basic regex makes + ? | ( ) { } awkward or inert unless backslash-escaped. Extended regex is the syntax ops uses most.
vs -F: use -F for literal strings (paths, IPs, fixed messages).
vs -P: Perl regex is not portable; stick to -E for scripts.

Invocation & high-value flag combos with -E

BRE vs ERE (what changes with -E)

Anchors & position

Dot, classes, and escapes

Quantifiers

Greedy tip for logs: prefer [^ ]+ / [^:]+ over .* so tokens stop at spaces.

Groups, alternation, backrefs

Ops-ready pattern cookbook (grep -E)

Extract → aggregate recipes (grep -E first)

Quoting rules for -E patterns

Pitfalls specific to grep -E

• grep -E 'file.txt' matches fileXtxt — escape dots: file\.txt or use -F.
• .* is greedy and crosses fields — prefer [^ ]+ for log tokens.
• Alternation has low precedence — use groups: grep -E '^(ERROR|WARN):' not ambiguous forms.
• Inside double quotes, the shell may eat \ — prefer single quotes for patterns.
• grep -E is line-oriented; multi-line stack traces need flatten-first.
• -o + alternation prints each alternative match separately (useful for counting tokens).
• Character classes are not Unicode “letters” unless locale/PCRE — for ASCII logs stay with [A-Za-z0-9].
• \s / \w are GNU extensions ([[:space:]] / [_[:alnum:]]). \t is not a tab in GNU grep ERE.
• egrep is deprecated in some docs; write grep -E in scripts.

sed

Definition: Stream editor: filter or transform text line-by-line (substitute, delete, print ranges).

Common scripts (first column = program text):

Flag combos

awk

Definition: Pattern-scanning language for field-based reports, filters, and aggregation.

Program pieces:

Flag combos

cut

Definition: Extract selected columns/bytes from delimited lines.

Flag combos

sort

Definition: Sort lines of text (optionally numeric, by key, unique).

Flag combos

uniq

Definition: Report or omit adjacent duplicate lines (usually after sort).

Flag combos

tr

Definition: Translate, squeeze, or delete characters from a stream.

Flag combos

pipe

Definition: Connect stdout of the left command to stdin of the right.

Common recipes

Pitfalls

• Bare grep is BRE. grep -E 'a|b' is alternation; GNU BRE writes that as backslash-pipe. POSIX BRE has no |.
• GNU grep ERE does not treat \t as tab (\t usually matches t). \s is a GNU synonym for [[:space:]], not POSIX ERE.
• grep -r and grep -R differ on GNU: only -R follows every symlink.
• grep -Eq PAT f && exit 1 as the last line of a script exits 1 even when PAT is absent (grep’s own status). Use if grep -Eq …; then exit 1; fi.
• uniq without sort only collapses adjacent duplicates. uniq -u keeps lines that appear once; it is not sort -u.
• sort -k2 is field 2 through end of line; field 2 alone is -k2,2. sort -u with -k uniques by the key, not the whole line.
• awk '{$1=$2=""; print}' leaves leading spaces; it does not delete the fields.
• Unescaped . in regex matches any character — use grep -F or \..
• sed -i without backup is irreversible. GNU form is -i.bak; BSD/macOS wants -i '' / -i '.bak'.
• cut (and simple awk -F,) break on quoted CSV commas — use a real CSV tool or Python.
• Prefer grep -E for regex; grep -F for literals; grep -P is GNU-only and not for portable scripts.