auditd / collectl – Linux system auditing and monitoring

Auditd
Digital Ocean has a really great tutorial for Auditd

IntroductionThe Linux Auditing System helps system administrators create an audit trail, a log for every action on the server. We can track security-relevant events, record the events in a log file, and detect misuse or unauthorized activities by inspecting the audit log files. We can choose which actions on the server to monitor and to what extent. Audit does not provide additional security to your system, rather, it helps track any violations of system policies and enables you to take additional security measures to prevent them.This tutorial explains the audit system, how to configure it, how to generate reports, and how to read these reports. We will also see how to search the audit logs for specific events.

with auditd running, you can see when/who removes files, logs in, logs out, switches user access files .. basically anything that requires a system call gets logged to your session. This is really helpful

I like to combine auditd with collectl for a full view of the system at a given time. While auditd is nice to report back events, collectl can show me processes that were running at a particular point in time.

The scenario of a server crashing – unless you have collectl or some other service that collects process information, it is very difficult to go back in time to check what processes were running or what the state of the machine was at the time of a fault. Collectl allows you to do just this:

Collectl:
https://www.tecmint.com/linux-performance-monitoring-with-collectl-tool/

collectl is a very nice feature rich command-line utility that can be used to collect performance data that describes the current system status. Unlike most of the other monitoring tools, collectl does not focus in a limited number of system metrics, instead it can gather information on many different types of system resources such as cpu, disk, memory, network, sockets, tcp, inodes, infiniband, lustre, memory, nfs, processes, quadrics, slabs and buddyinfo.

my favorite way to run this is:

collectl -p lnyctomtest-20170714-131249.raw.gz --top

### RECORD 1 >>> lnyctomtest <<< (1500052440.001) (Fri Jul 14 13:14:00 2017) ### # TOP PROCESSES sorted by time (counters are /sec) 13:14:00 # PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command 2250 root 20 967 2 S 521M 41M 3 0.03 0.78 1 54:44.28 0 0 0 3 /usr/bin/python 25706 root 20 1 0 R 172M 20M 0 0.04 0.22 0 00:00.45 0 1 0 1 /usr/bin/perl 508 root 20 1 0 S 96M 40M 0 0.09 0.04 0 08:54.92 0 0 0 30 /usr/lib/systemd/systemd-journald 967 root 20 1 2 S 503M 27M 0 0.02 0.09 0 05:34.87 0 0 0 0 /usr/bin/python 970 root 20 1 2 S 565M 24M 2 0.04 0.03 0 05:19.42 0 3 0 1 /usr/sbin/rsyslogd 10553 root 20 1 0 S 223M 5M 1 0.03 0.04 0 00:08.52 0 0 0 0 /usr/bin/vmtoolsd 9 root 20 2 0 S 0 0 3 0.04 0.00 0 01:22.88 0 0 0 0 rcu_sched 2907 root 20 2 0 S 0 0 3 0.04 0.00 0 00:04.03 0 0 0 0 kworker/3:1 645 root 20 2 0 S 0 0 3 0.02 0.00 0 01:34.28 0 0 0 0 xfsaild/dm-4 661 root 20 2 0 S 0 0 1 0.02 0.00 0 01:32.75 0 0 0 0 xfsaild/dm-3 ### RECORD 2 >>> lnyctomtest <<< (1500052500.001) (Fri Jul 14 13:15:00 2017) ### # TOP PROCESSES sorted by time (counters are /sec) 13:15:00 # PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command 2250 root 20 967 2 S 521M 41M 0 0.02 0.78 1 54:45.08 0 0 0 3 /usr/bin/python 25706 root 20 1 0 R 172M 20M 2 0.04 0.21 0 00:00.70 0 1 0 0 /usr/bin/perl 508 root 20 1 0 S 96M 40M 2 0.08 0.04 0 08:55.04 0 0 0 30 /usr/lib/systemd/systemd-journald 967 root 20 1 2 S 503M 27M 2 0.01 0.08 0 05:34.96 0 0 0 0 /usr/bin/python 970 root 20 1 2 S 565M 24M 2 0.04 0.04 0 05:19.50 0 3 0 1 /usr/sbin/rsyslogd 10553 root 20 1 0 S 223M 5M 0 0.03 0.04 0 00:08.59 0 0 0 0 /usr/bin/vmtoolsd 9 root 20 2 0 S 0 0 3 0.06 0.00 0 01:22.94 0 0 0 0 rcu_sched 2907 root 20 2 0 S 0 0 3 0.04 0.00 0 00:04.07 0 0 0 0 kworker/3:1 645 root 20 2 0 S 0 0 1 0.03 0.00 0 01:34.32 0 0 0 0 xfsaild/dm-4 661 root 20 2 0 S 0 0 1 0.02 0.00 0 01:32.77 0 0 0 0 xfsaild/dm-3 ### RECORD 3 >>> lnyctomtest <<< (1500052560.001) (Fri Jul 14 13:16:00 2017) ### # TOP PROCESSES sorted by time (counters are /sec) 13:16:00 # PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command 2250 root 20 967 2 S 521M 41M 0 0.03 0.76 1 54:45.87 0 0 0 3 /usr/bin/python 25706 root 20 1 0 R 172M 20M 1 0.03 0.23 0 00:00.96 0 1 0 0 /usr/bin/perl 508 root 20 1 0 S 96M 41M 3 0.07 0.05 0 08:55.16 0 0 0 30 /usr/lib/systemd/systemd-journald 10553 root 20 1 0 S 223M 5M 1 0.03 0.05 0 00:08.67 0 0 0 0 /usr/bin/vmtoolsd 967 root 20 1 2 S 503M 27M 3 0.00 0.07 0 05:35.03 0 0 0 0 /usr/bin/python 970 root 20 1 2 S 565M 24M 2 0.04 0.03 0 05:19.56 0 2 0 1 /usr/sbin/rsyslogd 9 root 20 2 0 S 0 0 3 0.04 0.00 0 01:22.98 0 0 0 0 rcu_sched 2907 root 20 2 0 S 0 0 3 0.04 0.00 0 00:04.11 0 0 0 0 kworker/3:1 661 root 20 2 0 S 0 0 1 0.03 0.00 0 01:32.80 0 0 0 0 xfsaild/dm-3 645 root 20 2 0 S 0 0 1 0.02 0.00 0 01:34.34 0 0 0 0 xfsaild/dm-4