Так что у меня есть Dell PowerEdge R610 сервер работает Debian Jessie который, согласно некоторым документам, имеет неисправный чипсет: https://www.novell.com/support/kb/doc.php?id=7014344 (Intel 5500 rev 13).
Предоставленный тест оболочки подтверждает наличие этого чипсета: lspci -nn | grep -qE '8086:(340[36].*rev 13|3405.*rev (12|13|22))' && echo "Interrupt remapping is broken"
Оно имеет VT-d отключен в BIOS и имеет GRUB_CMDLINE_LINUX_DEFAULT="quiet intremap=off" добавлен в /etc/grub/default.
У меня вопрос, влияет ли это на MSI-X особенность сетевых карт (которая должна распределять прерывания между ядрами)?
На этом конкретном сервере действительно только одно ядро обслуживает прерывания, но я не уверен, является ли это результатом отключенных настроек BIOS и ядра. Или может быть что-то еще?
(К сожалению, у меня нет доступа к BIOS для изменения настроек и тестирования с включенными функциями).
Спасибо.
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 43 0 0 0 0 0 0 0 IO-APIC-edge timer
1: 3 0 0 0 0 0 0 0 IO-APIC-edge i8042
8: 1 0 0 0 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
12: 4 0 0 0 0 0 0 0 IO-APIC-edge i8042
16: 15459 0 0 0 0 0 0 0 IO-APIC-fasteoi megasas
17: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
18: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
19: 28 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1
20: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb6
21: 398 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5
23: 50805 0 0 0 0 0 0 0 IO-APIC-fasteoi ata_piix
72: 0 0 0 0 0 0 0 0 PCI-MSI-edge PCIe PME
73: 0 0 0 0 0 0 0 0 PCI-MSI-edge PCIe PME
74: 0 0 0 0 0 0 0 0 PCI-MSI-edge PCIe PME
75: 0 0 0 0 0 0 0 0 PCI-MSI-edge PCIe PME
76: 0 0 0 0 0 0 0 0 PCI-MSI-edge PCIe PME
95: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
96: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
97: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
98: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
99: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
100: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
101: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
102: 2 0 0 0 0 0 0 0 PCI-MSI-edge ioat-msix
103: 126464280 0 0 0 0 0 0 0 PCI-MSI-edge eth2-0
104: 92145436 0 0 0 0 0 0 0 PCI-MSI-edge eth2-1
105: 89096948 0 0 0 0 0 0 0 PCI-MSI-edge eth2-2
106: 88057555 0 0 0 0 0 0 0 PCI-MSI-edge eth2-3
107: 81812555 0 0 0 0 0 0 0 PCI-MSI-edge eth2-4
108: 89599897 0 0 0 0 0 0 0 PCI-MSI-edge eth2-5
109: 88031686 0 0 0 0 0 0 0 PCI-MSI-edge eth2-6
110: 87201395 0 0 0 0 0 0 0 PCI-MSI-edge eth2-7
112: 377125739 0 0 0 0 0 0 0 PCI-MSI-edge eth0-0
113: 72913 0 0 0 0 0 0 0 PCI-MSI-edge eth0-1
114: 1890421 0 0 0 0 0 0 0 PCI-MSI-edge eth0-2
115: 162242342 0 0 0 0 0 0 0 PCI-MSI-edge eth0-3
116: 53494 0 0 0 0 0 0 0 PCI-MSI-edge eth0-4
117: 85112 0 0 0 0 0 0 0 PCI-MSI-edge eth0-5
118: 44281 0 0 0 0 0 0 0 PCI-MSI-edge eth0-6
119: 41480 0 0 0 0 0 0 0 PCI-MSI-edge eth0-7
NMI: 39120 194 382 197 337 184 276 186 Non-maskable interrupts
LOC: 217815785 1972192 4721954 2056033 2252897 1834404 1894099 1739779 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 39120 194 382 197 337 184 276 186 Performance monitoring interrupts
IWI: 8 0 0 0 0 0 0 0 IRQ work interrupts
RTR: 7 0 0 0 0 0 0 0 APIC ICR read retries
RES: 133678 35960 21042 9616 10918 8463 8607 8143 Rescheduling interrupts
CAL: 1815 3264 1626 3885 1970 3807 1972 4000 Function call interrupts
TLB: 896 110 258 139 200 110 187 81 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 171 171 171 171 171 171 171 171 Machine check polls
HYP: 0 0 0 0 0 0 0 0 Hypervisor callback interrupts
ERR: 0
MIS: 0
Настройки сетевой карты:
root@server:~# lspci -v
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
Flags: bus master, fast devsel, latency 0, IRQ 36
Memory at d6000000 (64-bit, non-prefetchable) [size=32M]
Capabilities: [48] Power Management version 3
Capabilities: [50] Vital Product Data
Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [a0] MSI-X: Enable+ Count=9 Masked-
Capabilities: [ac] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-bd
Capabilities: [110] Advanced Error Reporting
Capabilities: [150] Power Budgeting <?>
Capabilities: [160] Virtual Channel
Kernel driver in use: bnx2
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
Flags: bus master, fast devsel, latency 0, IRQ 48
Memory at d8000000 (64-bit, non-prefetchable) [size=32M]
Capabilities: [48] Power Management version 3
Capabilities: [50] Vital Product Data
Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
Capabilities: [ac] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-bf
Capabilities: [110] Advanced Error Reporting
Capabilities: [150] Power Budgeting <?>
Capabilities: [160] Virtual Channel
Kernel driver in use: bnx2
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
Flags: bus master, fast devsel, latency 0, IRQ 32
Memory at da000000 (64-bit, non-prefetchable) [size=32M]
Capabilities: [48] Power Management version 3
Capabilities: [50] Vital Product Data
Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [a0] MSI-X: Enable+ Count=9 Masked-
Capabilities: [ac] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-c1
Capabilities: [110] Advanced Error Reporting
Capabilities: [150] Power Budgeting <?>
Capabilities: [160] Virtual Channel
Kernel driver in use: bnx2
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Subsystem: Dell PowerEdge R610 BCM5709 Gigabit Ethernet
Flags: bus master, fast devsel, latency 0, IRQ 42
Memory at dc000000 (64-bit, non-prefetchable) [size=32M]
Capabilities: [48] Power Management version 3
Capabilities: [50] Vital Product Data
Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
Capabilities: [ac] Express Endpoint, MSI 00
Capabilities: [100] Device Serial Number 84-2b-2b-ff-fe-fe-b3-c3
Capabilities: [110] Advanced Error Reporting
Capabilities: [150] Power Budgeting <?>
Capabilities: [160] Virtual Channel
Kernel driver in use: bnx2
root@server:~# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
root@server:~# ethtool -i eth0
driver: bnx2
version: 2.2.5
firmware-version: 7.12.17 bc 7.10.0 NCSI 2.0.13
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
Да. Для полноценной работы MSI-X необходимо включить переопределение прерываний.
Из файла Intel PDF: Технология виртуализации Intel® для направленного ввода-вывода Спецификация архитектуры Октябрь 2014 г.
5.1 Переназначение прерывания
Архитектура переназначения прерываний позволяет системному программному обеспечению контролировать и подвергать цензуре внешние запросы прерывания, генерируемые всеми источниками, в том числе от контроллеров прерываний (I / OxAPIC), устройств с поддержкой MSI / MSI-X, включая конечные точки, корневые порты и интегрированный конечный модуль Root-Complex. точки.