Solved

NX-8235-G7 Switch Port Configuration

  • 14 November 2019
  • 5 replies
  • 2039 views
A

Good Morning:

We are in the process of deploying new NX-8235-G7 nodes that are connected to a pair of Cisco Nexus switch. Everything seems to be working well, with the exception of the switch occasionally receiving RX Pause frames from the servers. While this may not necessarily be a problem, I am doing my due dilligence.

The switch ports the nodes are connected to are configured as follows:

  • Connected as active/standby NICs
  • Connected via 1 meter 10GE DAC cables
  • 802.1Q trunk
  • Speed/duplex auto-negotiation
  • Flow control auto-negotiation

The nodes themselves are configured with the default network configuration that comes from the factory with the exception of anything that is necessary to support the use of ESXi and our network environment.

Any suggestions or validations of how the switch ports should be configured?

 

Thanks!

icon

Best answer by avelardo 15 November 2019, 17:59

View original
This topic has been closed for comments

5 replies

S
Rank
Userlevel 3
Badge +3

@avelardo Have you ran ncc health_checks on the cluster and if so what are they saying about networking in the cluster? This could probably help narrowing it to a specific node in the cluster. It “may be” that we have to increase send and receive buffer sizes in the esxi side. Below could be a good KB to read for this issue:

https://portal.nutanix.com/#/page/kbs/details?targetId=kA0600000008d25CAA

Let me know if this helps at all.

A

@sbarabThanks for the suggestions. I will review the link and I think one of our administrators recently ran ncc health_checks, but I will double check.

Alona
Rank
Userlevel 6
Badge +5

@avelardo On the note of Nexus ports configuration recommendations and examples here is another link for you KB-2455 Cisco Nexus Recommended Practices

A

Thanks @Alona. That was a useful document as well. I have not run the ncc health_checks yet, but I was able to gather statistics from the vmnics connected to the switch interfaces where I am seeing RX PAUSE counter changes.

It is clear that the rx_missed_errors, tx_flow_control_xon, tx_flow_control_xoff counters are going up on the vmnics. This would correlate with what I am seeing on the switch statistics. I noticed that different amounts of packets are counted in the different tx/rx queue counters, is this based on a QoS scheme?

 

Here is an output from one of the vmnics.

NIC statistics:
     rx_packets: 3344510250
     tx_packets: 2090940316
     rx_bytes: 4199449818807
     tx_bytes: 2040903105372
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 0
     multicast: 1242564
     collisions: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_fifo_errors: 0
     rx_missed_errors: 8537
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     rx_pkts_nic: 3344510250
     tx_pkts_nic: 2023678635
     rx_bytes_nic: 4226218803062
     tx_bytes_nic: 2053460219087
     lsc_int: 1
     tx_busy: 0
     non_eop_descs: 0
     broadcast: 1556398
     rx_no_buffer_count: 0
     tx_timeout_count: 0
     tx_restart_queue: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     tx_flow_control_xon: 99
     rx_flow_control_xon: 0
     tx_flow_control_xoff: 111
     rx_flow_control_xoff: 0
     rx_csum_offload_errors: 0
     rx_header_split: 0
     alloc_rx_page_failed: 0
     alloc_rx_buff_failed: 0
     rx_no_dma_resources: 0
     hw_rsc_aggregated: 0
     hw_rsc_flushed: 0
     fdir_match: 0
     fdir_miss: 0
     fdir_overflow: 0
     fcoe_bad_fccrc: 0
     fcoe_last_errors: 0
     rx_fcoe_dropped: 0
     rx_fcoe_packets: 0
     rx_fcoe_dwords: 0
     fcoe_noddp: 0
     fcoe_noddp_ext_buff: 0
     tx_fcoe_packets: 0
     tx_fcoe_dwords: 0
     os2bmc_rx_by_bmc: 0
     os2bmc_tx_by_bmc: 0
     os2bmc_tx_by_host: 0
     os2bmc_rx_by_host: 0
     tx_queue_0_packets: 137598803
     tx_queue_0_bytes: 107727994850
     tx_queue_1_packets: 471789106
     tx_queue_1_bytes: 412602748162
     tx_queue_2_packets: 1478148133
     tx_queue_2_bytes: 1516711930468
     tx_queue_3_packets: 3401309
     tx_queue_3_bytes: 3858609567
     tx_queue_4_packets: 2537
     tx_queue_4_bytes: 1266353
     tx_queue_5_packets: 0
     tx_queue_5_bytes: 0
     tx_queue_6_packets: 0
     tx_queue_6_bytes: 0
     tx_queue_7_packets: 0
     tx_queue_7_bytes: 0
     tx_queue_8_packets: 0
     tx_queue_8_bytes: 0
     tx_queue_9_packets: 0
     tx_queue_9_bytes: 0
     tx_queue_10_packets: 0
     tx_queue_10_bytes: 0
     tx_queue_11_packets: 0
     tx_queue_11_bytes: 0
     tx_queue_12_packets: 0
     tx_queue_12_bytes: 0
     tx_queue_13_packets: 0
     tx_queue_13_bytes: 0
     tx_queue_14_packets: 0
     tx_queue_14_bytes: 0
     tx_queue_15_packets: 428
     tx_queue_15_bytes: 555972
     rx_queue_0_packets: 136286341
     rx_queue_0_bytes: 111460884442
     rx_queue_1_packets: 0
     rx_queue_1_bytes: 0
     rx_queue_2_packets: 1418425670
     rx_queue_2_bytes: 1987709924521
     rx_queue_3_packets: 0
     rx_queue_3_bytes: 0
     rx_queue_4_packets: 1788335579
     rx_queue_4_bytes: 2099211490505
     rx_queue_5_packets: 0
     rx_queue_5_bytes: 0
     rx_queue_6_packets: 1460697
     rx_queue_6_bytes: 1066902002
     rx_queue_7_packets: 0
     rx_queue_7_bytes: 0
     rx_queue_8_packets: 1963
     rx_queue_8_bytes: 617337
     rx_queue_9_packets: 0
     rx_queue_9_bytes: 0
     rx_queue_10_packets: 0
     rx_queue_10_bytes: 0
     rx_queue_11_packets: 0
     rx_queue_11_bytes: 0
     rx_queue_12_packets: 0
     rx_queue_12_bytes: 0
     rx_queue_13_packets: 0
     rx_queue_13_bytes: 0
     rx_queue_14_packets: 0
     rx_queue_14_bytes: 0
     rx_queue_15_packets: 0
     rx_queue_15_bytes: 0
     tx_pb_0_pxon: 0
     tx_pb_0_pxoff: 0
     tx_pb_1_pxon: 0
     tx_pb_1_pxoff: 0
     tx_pb_2_pxon: 0
     tx_pb_2_pxoff: 0
     tx_pb_3_pxon: 0
     tx_pb_3_pxoff: 0
     tx_pb_4_pxon: 0
     tx_pb_4_pxoff: 0
     tx_pb_5_pxon: 0
     tx_pb_5_pxoff: 0
     tx_pb_6_pxon: 0
     tx_pb_6_pxoff: 0
     tx_pb_7_pxon: 0
     tx_pb_7_pxoff: 0
     rx_pb_0_pxon: 0
     rx_pb_0_pxoff: 0
     rx_pb_1_pxon: 0
     rx_pb_1_pxoff: 0
     rx_pb_2_pxon: 0
     rx_pb_2_pxoff: 0
     rx_pb_3_pxon: 0
     rx_pb_3_pxoff: 0
     rx_pb_4_pxon: 0
     rx_pb_4_pxoff: 0
     rx_pb_5_pxon: 0
     rx_pb_5_pxoff: 0
     rx_pb_6_pxon: 0
     rx_pb_6_pxoff: 0
     rx_pb_7_pxon: 0
     rx_pb_7_pxoff: 0

A

I found this Community thread and it seems like it might be relevant. Our 8X35 nodes have the Intel 82599 NICs mentioned and currently the configuration of ESX looks like:

[root@vdi-esx02:~] esxcfg-module -g ixgbe
ixgbe enabled = 1 options = ‘’

 

Based on my read of links in the thread, the configuration should look something like:

ixgbe enabled = 1 options = 'InterruptType=2,2 VMDQ=16,16'

Terms & Conditions