У меня в cron активен rsync, и я начал получать электронные письма после каждого rsync
This is an automatically generated mail message from mdadm
running on titan707
A DegradedArray event had been detected on md device /dev/md/2.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdb3[1] sda3[0]
7995840 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdb2[1](F) sda2[0]
499712 blocks super 1.2 [2/1] [U_]
md2 : active raid1 sdb4[1](F) sda4[0]
968130304 blocks super 1.2 [2/1] [U_]
unused devices:
Но позже smartctl и mdadmin не покажут никаких проблем, см. Ниже журналы от mdadm, smartctl.
$ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] [linear] [multipath]
md0 : active raid1 sda1[0] sdb1[1]
33553336 blocks super 1.2 [2/2] [UU]
md1 : active raid1 sdb2[1] sda2[0]
524276 blocks super 1.2 [2/2] [UU]
md3 : active raid1 sdb4[1] sda4[0]
1822442815 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdb3[1] sda3[0]
1073740664 blocks super 1.2 [2/2] [UU]
unused devices:
$ smartctl -a /dev/sda
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-24-generic] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda XT
Device Model: ST33000651AS
Serial Number: Z291E1TG
LU WWN Device Id: 5 000c50 03f2f8fbc
Firmware Version: CC45
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Wed Mar 19 09:20:26 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 600) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x103f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 152015022
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 6
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 40795438
9 Power_On_Hours 0x0032 077 077 000 Old_age Always - 20281
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 6
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 053 046 045 Old_age Always - 47 (Min/Max 43/54)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 6
194 Temperature_Celsius 0x0022 047 054 000 Old_age Always - 47 (0 23 0 0)
195 Hardware_ECC_Recovered 0x001a 021 003 000 Old_age Always - 152015022
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 253145372446521
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2852285811
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 811308464
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 20193 -
# 2 Short offline Completed without error 00% 20185 -
# 3 Extended offline Completed without error 00% 5723 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
$ smartctl -a /dev/sdb
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-24-generic] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda XT
Device Model: ST33000651AS
Serial Number: Z2917JDM
LU WWN Device Id: 5 000c50 03f1b6146
Firmware Version: CC45
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Wed Mar 19 09:20:53 2014 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 609) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x103f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 144398334
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 6
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 41707682
9 Power_On_Hours 0x0032 077 077 000 Old_age Always - 20281
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 6
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 057 049 045 Old_age Always - 43 (Min/Max 39/51)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 6
194 Temperature_Celsius 0x0022 043 051 000 Old_age Always - 43 (0 23 0 0)
195 Hardware_ECC_Recovered 0x001a 021 003 000 Old_age Always - 144398334
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 38959648362297
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 162809159
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1526676264
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 20218 -
# 2 Short offline Completed without error 00% 20185 -
# 3 Extended offline Completed without error 00% 5723 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
$
$ mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Jul 27 13:40:57 2012
Raid Level : raid1
Array Size : 33553336 (32.00 GiB 34.36 GB)
Used Dev Size : 33553336 (32.00 GiB 34.36 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Mon Mar 17 12:24:57 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : rescue:0
UUID : 28ad38a2:f3df9bbc:2f1f4d98:2006ce16
Events : 22
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
$ mdadm -D /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Fri Jul 27 13:40:57 2012
Raid Level : raid1
Array Size : 524276 (512.07 MiB 536.86 MB)
Used Dev Size : 524276 (512.07 MiB 536.86 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Wed Mar 19 06:25:43 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : rescue:1
UUID : 659022e1:e93cfcb9:c7b533ae:5a81c83b
Events : 25
Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 18 1 active sync /dev/sdb2
$ mdadm -D /dev/md2
/dev/md2:
Version : 1.2
Creation Time : Fri Jul 27 13:40:58 2012
Raid Level : raid1
Array Size : 1073740664 (1024.00 GiB 1099.51 GB)
Used Dev Size : 1073740664 (1024.00 GiB 1099.51 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Wed Mar 19 09:21:40 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : rescue:2
UUID : b79d3e48:62b55d0b:8501355c:2f905ef2
Events : 34
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
$ mdadm -D /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Fri Jul 27 13:40:58 2012
Raid Level : raid1
Array Size : 1822442815 (1738.02 GiB 1866.18 GB)
Used Dev Size : 1822442815 (1738.02 GiB 1866.18 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Wed Mar 19 09:21:09 2014
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : rescue:3
UUID : fdb07043:8bd52646:9f267e1b:d0a43f0e
Events : 22
Number Major Minor RaidDevice State
0 8 4 0 active sync /dev/sda4
1 8 20 1 active sync /dev/sdb4
$
Я тоже ничего не могу найти в dmesg
$ dmesg | grep "md" [ 1.957908] md: raid0 personality registered for level 0 [ 1.959091] md: raid1 personality registered for level 1 [ 2.069112] md: bind [ 2.070684] md: bind [ 2.072032] md: bind [ 2.116159] md: bind [ 2.117310] md/raid1:md3: active with 2 out of 2 mirrors [ 2.117380] md3: detected capacity change from 0 to 1866181442560 [ 2.124174] md: bind [ 2.138621] md3: unknown partition table [ 2.140113] md: bind [ 2.141326] md/raid1:md2: active with 2 out of 2 mirrors [ 2.141398] md2: detected capacity change from 0 to 1099510439936 [ 2.162685] md2: unknown partition table [ 2.230596] md: bind [ 2.231715] md/raid1:md1: active with 2 out of 2 mirrors [ 2.231786] md1: detected capacity change from 0 to 536858624 [ 2.233100] md1: unknown partition table [ 2.436160] md: bind [ 2.437387] md/raid1:md0: active with 2 out of 2 mirrors [ 2.437456] md0: detected capacity change from 0 to 34358616064 [ 2.444765] md0: unknown partition table [ 2.456675] md: raid6 personality registered for level 6 [ 2.456738] md: raid5 personality registered for level 5 [ 2.456797] md: raid4 personality registered for level 4 [ 2.458570] md: raid10 personality registered for level 10 [ 2.462736] md: linear personality registered for level -1 [ 2.463538] md: multipath personality registered for level -4 [ 8.213448] EXT4-fs (md2): mounted filesystem with ordered data mode. Opts: (null) [ 11.334852] Adding 33553332k swap on /dev/md0. Priority:-1 extents:1 across:33553332k [ 11.337379] EXT4-fs (md2): warning: checktime reached, running e2fsck is recommended [ 11.359536] EXT4-fs (md2): re-mounted. Opts: (null) [ 11.700105] EXT3-fs (md1): warning: checktime reached, running e2fsck is recommended [ 11.778306] EXT3-fs (md1): using internal journal [ 11.778310] EXT3-fs (md1): mounted filesystem with ordered data mode [ 12.155704] EXT4-fs (md3): warning: checktime reached, running e2fsck is recommended [ 12.218303] EXT4-fs (md3): mounted filesystem with ordered data mode. Opts: (null) $ dmesg | grep "sd" [ 1.870244] sd 0:0:0:0: [sda] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB) [ 1.870251] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 1.870487] sd 0:0:0:0: [sda] Write Protect is off [ 1.870637] sd 1:0:0:0: [sdb] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB) [ 1.870638] sd 1:0:0:0: Attached scsi generic sg1 type 0 [ 1.870667] sd 1:0:0:0: [sdb] Write Protect is off [ 1.870668] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 1.870697] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.870989] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 1.870999] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 1.916610] sda: sda1 sda2 sda3 sda4 sda5 [ 1.917195] sd 0:0:0:0: [sda] Attached SCSI disk [ 1.928325] sdb: sdb1 sdb2 sdb3 sdb4 sdb5 [ 1.929042] sd 1:0:0:0: [sdb] Attached SCSI disk [ 2.069112] md: bind [ 2.070684] md: bind [ 2.072032] md: bind [ 2.116159] md: bind [ 2.124174] md: bind [ 2.140113] md: bind [ 2.230596] md: bind [ 2.436160] md: bind
Сценарий Cron, который я использую как резервный пользователь, для синхронизации содержимого между двумя серверами, которыми я управляю
#!/bin/bash #follow instructions to setup mybackup account and sh keys from https://blogs.oracle.com/jkini/entry/how_to_scp_scp_and rsync -a -r -u mybackup@tralev.com:/tralev/images /home/tralev/backup echo finished tralev images sleep 2s rsync -a -r -u mybackup@tralev.com:/backup/* /home/tralev/backup/db echo finished tralev db sleep 2s #backup numbeo files to tralev server rsync -a -r -u /numbeo/* mybackup@tralev.com:/numbeo/backup echo finished numbeo files like images sleep 2s rsync -a -r -u /root/backup/* mybackup@tralev.com:/numbeo/db_backup echo finished numbeo db backup sleep 2s
Я могу воспроизвести проблему только при запуске из cron, когда я запускаю скрипт на сервере, у меня не возникает такой же проблемы.
Есть идеи, что может пойти не так?
РЕДАКТИРОВАТЬ: Оказалось, что я проверял не тот сервер. Более того, оба диска на сервере titan707 вышли из строя, поэтому мне пришлось заменить сервер из резервной копии! Человеческая ошибка!
Вы проверяете не тот сервер. Второй вывод / proc / mdstat (с 4 массивами raid) не от titan707, у которого есть три массива raid.