Saturday, July 7, 2018

Fix journalling issue older generation SSD performance in Linux

I found a solution to an issue that has been hampering my development efficiency for months.  The symptom of the problem was any large file I/O operations on my secondary SSD stalled my Linux system.  The drive only contains applications and it is not a working volume.

My initial guess was a failing SSD.   SMART values did not indicate as such.  The drive was only a few years old and was from some obscure brand (PNY).

IO profiling indicated that process [jdb2/sdd1-8] had high IO write wait times while the system was "stalled".  I reproduced the behavior by writing a large random file to the disk.

Fast forward a few hours of research and unsuccessful attempts at modifying filesystem settings.  The solution was to disable journaling for this "previous generation" SSD drive.  It is work noting that the problem does not occur on my "newer generation" Samsung 840 root drive.

Lesson learned?  It would have been nice to "call IT" and have them fix it.  Not much else.  I guess that learned about IO system profiling and file system journalling.  It's time to get back to real work.

# monitor journalling process
sudo iotop -obtqqq | grep "jbd2"

# write large random file
head -c 10G < /dev/urandom > tmp

# disable journaling, need to unmount first
sudo tune2fs -O ^has_journal /dev/sdd1

# drive information
Device Model: SSD2SC120G3LA726B087-327
Firmware Version: 560ABBF0