My initial guess was a failing SSD. SMART values did not indicate as such. The drive was only a few years old and was from some obscure brand (PNY).
IO profiling indicated that process [jdb2/sdd1-8] had high IO write wait times while the system was "stalled". I reproduced the behavior by writing a large random file to the disk.
Fast forward a few hours of research and unsuccessful attempts at modifying filesystem settings. The solution was to disable journaling for this "previous generation" SSD drive. It is work noting that the problem does not occur on my "newer generation" Samsung 840 root drive.
Lesson learned? It would have been nice to "call IT" and have them fix it. Not much else. I guess that learned about IO system profiling and file system journalling. It's time to get back to real work.
# monitor journalling process sudo iotop -obtqqq | grep "jbd2" # write large random file head -c 10G < /dev/urandom > tmp # disable journaling, need to unmount first sudo tune2fs -O ^has_journal /dev/sdd1 # drive information === START OF INFORMATION SECTION === Device Model: SSD2SC120G3LA726B087-327 Firmware Version: 560ABBF0
https://www.harddrivebenchmark.net/hdd.php?hdd=SSD2SC120G3LA726B087-327