У меня есть два сервера, которые настроены точно так же (одинаковые SSD, RAM и т. Д.). На них обоих работает ESXI. В одном из ящиков был квестовый хозяин, у которого было много жалоб на запись на диск. Оказывается, на хосте не хватало оперативной памяти, и он сильно загружался в своп. Это объяснило бы, почему мы увидели ошибки. SSD использовался в качестве оперативной памяти и не успевал. Я запустил несколько экземпляров
dd bs=2GB if=/dev/zero of=/dev/null
На коробке, чтобы коробка перешла в режим подкачки, чтобы SSD использовался в качестве оперативной памяти. На box1 (где у нас возникла проблема) потребовалось много усилий, но мы смогли вызвать те же ошибки. На box2 мы запускали одни и те же тесты снова и снова и только один раз успешно получили ошибку. Это заставило меня поверить, что проблема была в диске на box1. Я использовал fdisk, а также посмотрел на smartd, и отчеты для обеих систем, похоже, показывают, что диски в порядке. я что-то упустил?
box1 (плохой ящик):
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 131
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 24
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 100 100 000 Old_age Always - 12
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 20
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 48
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 064 049 000 Old_age Always - 36 (Min/Max 0/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Bogus_Current_Pend_Sect 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 665
202 Percent_Lifetime_Remain 0x0030 100 100 001 Old_age Offline - 0
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 3801713422
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 62354066
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 39135457
root@ubuntu:~# smartctl --all /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron MX500 SSDs
Device Model: CT500MX500SSD1
Serial Number: 2002E2330303
LU WWN Device Id: 5 00a075 1e2330303
Firmware Version: M3CR023
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jul 20 09:34:19 2020 UTC
==> WARNING: This firmware returns bogus raw values in attribute 197
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 30) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 131
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 24
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 100 100 000 Old_age Always - 12
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 20
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 48
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 064 049 000 Old_age Always - 36 (Min/Max 0/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Bogus_Current_Pend_Sect 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 665
202 Percent_Lifetime_Remain 0x0030 100 100 001 Old_age Offline - 0
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 3801713422
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 62354066
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 39135459
SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 1
ATA Error Count: 0
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 0 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
00 ec 00 00 00 00 00
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
c8 00 00 00 00 00 00 00 00:00:00.000 READ DMA
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 121 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute dela```
box2 (good box)
```root@ubuntu:/usr/src/gsmartcontrol-1.1.3# smartctl -A /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 295
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 20
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 099 099 000 Old_age Always - 28
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 13
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 36
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 059 049 000 Old_age Always - 41 (Min/Max 0/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Bogus_Current_Pend_Sect 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 4
202 Percent_Lifetime_Remain 0x0030 099 099 001 Old_age Offline - 1
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 8068464723
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 136319482
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 81387530
root@ubuntu:/usr/src/gsmartcontrol-1.1.3# smartctl --all /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron MX500 SSDs
Device Model: CT500MX500SSD1
Serial Number: 2002E2330312
LU WWN Device Id: 5 00a075 1e2330312
Firmware Version: M3CR023
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jul 20 19:57:11 2020 UTC
==> WARNING: This firmware returns bogus raw values in attribute 197
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x03) Offline data collection activity
is in progress.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 1303) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 30) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0031) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 295
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 20
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 099 099 000 Old_age Always - 28
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 13
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 36
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 059 049 000 Old_age Always - 41 (Min/Max 0/51)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Bogus_Current_Pend_Sect 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 4
202 Percent_Lifetime_Remain 0x0030 099 099 001 Old_age Offline - 1
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 8068464723
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 136319482
248 FTL_Program_Page_Count 0x0032 100 100 000 Old_age Always - 81387535
SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 1
ATA Error Count: 0
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 0 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
00 ec 00 00 00 00 00
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
ec 00 00 00 00 00 00 00 00:00:00.000 IDENTIFY DEVICE
c8 00 00 00 00 00 00 00 00:00:00.000 READ DMA
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Self_test_in_progress [90% left] (0-65535)
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x10):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.```
Я не вижу ничего, что указывает на то, что "плохой" диск в данный момент выходит из строя, кроме UDMA_CRC_Error_Count что немного завышено. Попробуйте заменить кабель SSD.
sudo cat /dev/sda > /dev/null
echo $?
Должен возвращать 0 и не должно быть sda связанные ошибки или предупреждения в dmesg.
sudo smartctl -t long /dev/sda
# Some 20 minutes or later (you will be told how much it takes)
sudo smartctl -a /dev/sda
Вы должны увидеть что-то вроде
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 1234 -
Если это что-то, кроме «Завершено без ошибок», вероятно, вам следует заменить диск как можно скорее.
Имейте в виду, что производители SSD часто не полностью поддерживают стандарт SMART, поэтому smartctl -t long /dev/sda может вообще не работать.
badblocks -b 4096 -s -n /dev/sda