Corrupted Blocks on ext4 FS
Recently my Intel 1TB boot SSD started exhibiting some odd behavior. Frequently Ubuntu would crash unexpectedly whenever I would navigate into a folder which had large image files in it and Nautilus would attempt to display the thumbnails.
Whenever I looked in the dmesg logs I would see errors such as these
[202118.117543] print_req_error: I/O error, dev sda, sector 836148808
[202118.117555] Buffer I/O error on dev sda1, logical block 104518345, async page read
[202118.117585] ata4: EH complete
[202118.820841] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[202118.820848] ata4.00: irq_stat 0x40000001
[202118.820855] ata4.00: failed command: READ DMA
[202118.820868] ata4.00: cmd c8/00:08:e8:72:3d/00:00:00:00:00/e7 tag 14 dma 4096 in
res 51/40:08:e8:72:3d/00:00:00:00:00/e7 Emask 0x9 (media error)
[202118.820874] ata4.00: status: { DRDY ERR }
[202118.820878] ata4.00: error: { UNC }
[202118.877330] ata4.00: configured for UDMA/133
[202118.877355] sd 3:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[202118.877361] sd 3:0:0:0: [sda] tag#14 Sense Key : Medium Error [current]
[202118.877367] sd 3:0:0:0: [sda] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed
[202118.877373] sd 3:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 07 3d 72 e8 00 00 0
Backing up the Failing Drive
I booted my machine from a USB drive and immediately created an image of the failing drive via:
dd if=/dev/sda of=/media/backup/failing_drive.backup.img status=progress conv=noerror,sync
The noerror,sync is very important since when the drive is failing, a read from the block device returns failing return codes which dd will quit upon seeing unless given the noerror flag where it ignores such errors and carriers on.
The sync flag enables dd to pad the block to the correct size, in the case that it could not read from block I/O due to an error.
Using SMART Tool
Since this is a relatively new Intel SSD, I decided to install the SmartTools package to interrogate the drive as to whether it was close to end of life. The command I used is:
sudo apt-get install smartmontool
I got the SMART report from the harddrive via
sudo smartctl –all /dev/sda
The output was as follows:
=== START OF INFORMATION SECTION ===
Device Model: INTEL SSDSCKKW010X6
Serial Number: CVLY6116008R1P0G
LU WWN Device Id: 5 5cd2e4 14cb5b0d3
Firmware Version: LSBG100
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: M.2
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 (minor revision not indicated)
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Feb 3 20:03:14 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 60) minutes.
SCT capabilities: (0x0039) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 1024
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 436
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 90
170 Unknown_Attribute 0x0033 098 098 010 Pre-fail Always - 0
171 Unknown_Attribute 0x0032 100 100 010 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 010 Old_age Always - 0
174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 59
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 299
190 Airflow_Temperature_Cel 0x0032 052 066 000 Old_age Always - 52 (Min/Max 28/66)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 59
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 3
225 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 206243
226 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 0
227 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 0
228 Power-off_Retract_Count 0x0032 100 100 000 Old_age Always - 0
232 Available_Reservd_Space 0x0033 027 027 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 098 098 000 Old_age Always - 0
241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 206243
242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 179722
249 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 3172
252 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 5
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 0
Warning: ATA Specification requires self-test log structure revision number = 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 70403103932424 70403103932424 Not_testing
2 70403103932424 70403103932424 Not_testing
3 70403103932424 70403103932424 Not_testing
4 70403103932424 70403103932424 Not_testing
5 70403103932424 70403103932424 Not_testing
Selective self-test flags (0x4008):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
It seemed that the drive reported its status as fine, but I wanted to make sure. We can trigger a SMART scan of the drive via:
sudo smartctl -t long /dev/sda
The output of which is as follows:
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 60 minutes for test to complete.
Test will complete after Sun Feb 3 21:05:21 2019
Use smartctl -X to abort test.
After the test is done we can see the results via
sudo smartctl –all /dev/sda
The output of which was:
=== START OF INFORMATION SECTION ===
Device Model: INTEL SSDSCKKW010X6
Serial Number: CVLY6116008R1P0G
LU WWN Device Id: 5 5cd2e4 14cb5b0d3
Firmware Version: LSBG100
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: M.2
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 (minor revision not indicated)
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Feb 3 21:09:32 2019 UTC
SMART support is: Available – device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 60) minutes.
SCT capabilities: (0x0039) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 1024
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 436
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 90
170 Unknown_Attribute 0x0033 098 098 010 Pre-fail Always - 0
171 Unknown_Attribute 0x0032 100 100 010 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 010 Old_age Always - 0
174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 59
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 299
190 Airflow_Temperature_Cel 0x0032 043 066 000 Old_age Always - 43 (Min/Max 28/66)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 59
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 3
225 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 206243
226 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 0
227 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 0
228 Power-off_Retract_Count 0x0032 100 100 000 Old_age Always - 0
232 Available_Reservd_Space 0x0033 023 023 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 098 098 000 Old_age Always - 0
241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 206243
242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 179722
249 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 3180
252 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 5
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
1 Extended offline Completed without error 00% 436 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 70403103932424 70403103932424 Not_testing
2 70403103932424 70403103932424 Not_testing
3 70403103932424 70403103932424 Not_testing
4 70403103932424 70403103932424 Not_testing
5 70403103932424 70403103932424 Not_testing
Selective self-test flags (0x4008):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
So that did not help a lot in the least, I tried then running badblocks in non-destructive mode to see if there are any read/write errors on the drive via:
Finding Bad Blocks
sudo badblocks -svvn -c 262144 /dev/sda1
The output of which was:
Checking for bad blocks in non-destructive read-write mode
From block 0 to 960032767
Checking for bad blocks (non-destructive read-write test)
Testing with random pattern: 60732788done, 19:18 elapsed. (0/0/0 errors)
60732789done, 19:18 elapsed. (1/0/0 errors)
60732790done, 19:18 elapsed. (2/0/0 errors)
60732791done, 19:18 elapsed. (3/0/0 errors)
[..snip..]
418073378one, 2:12:01 elapsed. (114/0/0 errors)
418073379one, 2:12:02 elapsed. (115/0/0 errors)
418073380one, 2:12:02 elapsed. (116/0/0 errors)
418073381one, 2:12:02 elapsed. (117/0/0 errors)
418073382one, 2:12:02 elapsed. (118/0/0 errors)
418073383one, 2:12:02 elapsed. (119/0/0 errors)
done
Pass completed, 120 bad blocks found. (120/0/0 errors
Fixing Bad Blocks
Aha! Now we are getting somewhere, let’s try to fix via:
sudo e2fsck -cfpv /dev/sda1
The output of which was:
/dev/sda1: File /home/cookie/Desktop/photos.zip (inode #18881802, mod time Sun Mar 26 00:47:18 2017)
has 4 multiply-claimed block(s), shared with 1 file(s):
/dev/sda1: (inode #1, mod time Mon Feb 4 03:49:04 2019)
/dev/sda1: Updating bad block inode.
/dev/sda1: Inode 18885483 extent tree (at level 1) could be shorter. IGNORED.
/dev/sda1: Inode 18972432 extent tree (at level 1) could be shorter. IGNORED.
/dev/sda1: Inode 19660885 extent tree (at level 1) could be shorter. IGNORED.
/dev/sda1: Duplicate or bad block in use!
/dev/sda1: Multiply-claimed block(s) in inode 18881802: 80541305--80541308
/dev/sda1: Multiply-claimed block(s) in inode 19664781: 48251809--48251812
/dev/sda1: Multiply-claimed block(s) in inode 20061946: 66410385--66410388 66426325
/dev/sda1: Multiply-claimed block(s) in inode 41824574: 81660364--81660367
/dev/sda1: Multiply-claimed block(s) in inode 41824769: 84694106--84694109
/dev/sda1: Multiply-claimed block(s) in inode 41825243: 95786870--95786873
/dev/sda1: Multiply-claimed block(s) in inode 41825867: 104518342--104518345
/dev/sda1: Multiply-claimed block(s) in inode 42118786: 15183197
/dev/sda1: (There are 8 inodes containing multiply-claimed blocks.)
/dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
Hrm… Ok, let’s try:
sudo e2fsck -cfv /dev/sda1
Output of which:
e2fsck 1.44.1 (24-Mar-2018)
Checking for bad blocks (read-only test): done
/dev/sda1: Updating bad block inode.
Pass 1: Checking inodes, blocks, and sizes
Inode 18885483 extent tree (at level 1) could be shorter. Fix? yes
Inode 18972432 extent tree (at level 1) could be shorter. Fix? yes
Inode 19660885 extent tree (at level 1) could be shorter. Fix? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Running additional passes to resolve blocks claimed by more than one inode…
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 18881802: 80541305--80541308
Multiply-claimed block(s) in inode 19664781: 48251809--48251812
Multiply-claimed block(s) in inode 20061946: 66410385--66410388 66426325
Multiply-claimed block(s) in inode 41824574: 81660364--81660367
Multiply-claimed block(s) in inode 41824769: 84694106--84694109
Multiply-claimed block(s) in inode 41825243: 95786870--95786873
Multiply-claimed block(s) in inode 41825867: 104518342--104518345
Multiply-claimed block(s) in inode 42118786: 15183197
Pass 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
(There are 8 inodes containing multiply-claimed blocks.)
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #1 (6783, counted=6753).
Fix? yes
Free blocks count wrong for group #463 (65535, counted=0).
Fix? yes
Free blocks count wrong for group #1472 (8346, counted=8350).
Fix? yes
Free blocks count wrong for group #2026 (65532, counted=0).
Fix? yes
Free blocks count wrong for group #2027 (65535, counted=0).
Fix? yes
Free blocks count wrong for group #2457 (15156, counted=15160).
Fix? yes
Free blocks count wrong for group #2492 (4027, counted=4031).
Fix? yes
Free blocks count wrong for group #2584 (859, counted=863).
Fix? yes
Free blocks count wrong for group #2923 (65532, counted=0).
Fix? yes
Free blocks count wrong for group #3189 (1113, counted=1117).
Fix? yes/dev/sda1:
FILE SYSTEM WAS MODIFIED
5584074 inodes used (9.31%, out of 60006400)
8811 non-contiguous files (0.2%)
2257 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 5501966/1199/6
212208359 blocks used (88.42%, out of 240008192)
30 bad blocks
8 large files
5184043 regular files
284048 directories
222 character device files
100 block device files
1 fifo
77 links
115551 symbolic links (80472 fast symbolic links)
100 so
Results
After this I was able to mount the partition and retrieve my files. Some of my files were corrupted but the vast majority seemed fine. Thankfully almost everything was backed up, so this was mostly just an interesting experiment.
I still don’t quite know what corrupted the drive, but since I see these errors in the kernel log, it feels like a hardware issue. I even changed the slot of the drive from one M.2 slot to another with no positive results.
Either way this drive is not to be trusted anymore!
Good luck!