Network Slowness due LLDP process get stuck

Today i have observed a new issue in my environment, out of six floors from one of floor users are facing network slowness, their system got struck. Since the users are using VDI machines even a little traffic fluctuations will hamper performance.

I gone through various troubleshooting steps and found that LLDP processes for the interfaces were stuck and after gracefully restarting the processes things work well.

Following were the troubleshooting steps

  1. Ping all 364 tech floor switches and check latency or packet drops if any—Normal
  2. Ping from Core Switch and reverse path--- normal latency and no packet drops .
  3. Check logs at Core Switch and all aggregated interfaces were flapping intermittently which are connected to 364 GF L3 Switch

Sep  5 01:52:21  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 574, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36
Sep  5 01:52:21  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 578, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36.0
Sep  5 01:52:51  JUN-L3-FF-Cluster lldpd[1352]: LLDP_NEIGHBOR_UP: A neighbor has come up for interface ge-0/0/36.0. Now, this interface has 1 neighbor/s .
Sep  5 01:58:26  JUN-L3-FF-Cluster chassism[1307]: ifd_process_flaps IFD: ge-0/0/36, sent flap msg to RE, Downstate
Sep  5 01:58:26  JUN-L3-FF-Cluster lldpd[1352]: LLDP_NEIGHBOR_DOWN: A neighbor of interface ge-0/0/36.0 has gone down. Now, this interface has 0 neighbor/s.
Sep  5 01:58:26  JUN-L3-FF-Cluster chassism[1307]:       Link status change event: ifd ge-0/0/36 MAC ctrl reg0 :: 0x8BE5, MAC port status reg0 :: 0x6802, MAC auto-neg reg :: 0xB0F4
Sep  5 01:58:26  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36.0 index 2147404488 <Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:58:26  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36 index 223 <Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:58:26  JUN-L3-FF-Cluster chassism[1307]:      Link status change event: ifd ge-0/0/36 PHY Link Status: DOWN,LP-AN capable: NO
Sep  5 01:58:26  JUN-L3-FF-Cluster chassism[1307]:      Link status change event: ifd ge-0/0/36 AN Status: Pending, Speed: 1000 Mbps, Duplex: HALF DUPLEX,Remote Link Fault: NO
Sep  5 01:58:26  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_DOWN: ifIndex 574, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/36
Sep  5 01:58:29  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36.0 index 2147404488 <Up Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:58:29  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36 index 223 <Up Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:58:29  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 574, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36
Sep  5 01:58:29  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 578, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36.0
Sep  5 01:58:59  JUN-L3-FF-Cluster lldpd[1352]: LLDP_NEIGHBOR_UP: A neighbor has come up for interface ge-0/0/36.0. Now, this interface has 1 neighbor/s .
Sep  5 01:59:31  JUN-L3-FF-Cluster lldpd[1352]: LLDP_NEIGHBOR_DOWN: A neighbor of interface ge-0/0/36.0 has gone down. Now, this interface has 0 neighbor/s.
Sep  5 01:59:31  JUN-L3-FF-Cluster chassism[1307]: ifd_process_flaps IFD: ge-0/0/36, sent flap msg to RE, Downstate
Sep  5 01:59:31  JUN-L3-FF-Cluster chassism[1307]:       Link status change event: ifd ge-0/0/36 MAC ctrl reg0 :: 0x8BE5, MAC port status reg0 :: 0x6802, MAC auto-neg reg :: 0xB0F4
Sep  5 01:59:31  JUN-L3-FF-Cluster chassism[1307]:      Link status change event: ifd ge-0/0/36 PHY Link Status: DOWN,LP-AN capable: NO
Sep  5 01:59:31  JUN-L3-FF-Cluster chassism[1307]:      Link status change event: ifd ge-0/0/36 AN Status: Pending, Speed: 1000 Mbps, Duplex: HALF DUPLEX,Remote Link Fault: NO
Sep  5 01:59:31  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36.0 index 2147404488 <Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:59:31  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36 index 223 <Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:59:31  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_DOWN: ifIndex 574, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/36
Sep  5 01:59:34  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36.0 index 2147404488 <Up Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:59:34  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36 index 223 <Up Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 01:59:34  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 574, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36
Sep  5 01:59:34  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 578, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36.0
Sep  5 02:00:04  JUN-L3-FF-Cluster lldpd[1352]: LLDP_NEIGHBOR_UP: A neighbor has come up for interface ge-0/0/36.0. Now, this interface has 1 neighbor/s .
Sep  5 02:02:19  JUN-L3-FF-Cluster chassism[1307]: ifd_process_flaps IFD: ge-0/0/36, sent flap msg to RE, Downstate
Sep  5 02:02:19  JUN-L3-FF-Cluster lldpd[1352]: LLDP_NEIGHBOR_DOWN: A neighbor of interface ge-0/0/36.0 has gone down. Now, this interface has 0 neighbor/s.
Sep  5 02:02:19  JUN-L3-FF-Cluster chassism[1307]:       Link status change event: ifd ge-0/0/36 MAC ctrl reg0 :: 0x8BE5, MAC port status reg0 :: 0x6802, MAC auto-neg reg :: 0xB0F4
Sep  5 02:02:19  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_DOWN: ifIndex 574, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/36
Sep  5 02:02:19  JUN-L3-FF-Cluster chassism[1307]:      Link status change event: ifd ge-0/0/36 PHY Link Status: DOWN,LP-AN capable: NO
Sep  5 02:02:19  JUN-L3-FF-Cluster chassism[1307]:      Link status change event: ifd ge-0/0/36 AN Status: Pending, Speed: 1000 Mbps, Duplex: HALF DUPLEX,Remote Link Fault: NO
Sep  5 02:02:19  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36.0 index 2147404488 <Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 02:02:19  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36 index 223 <Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 02:02:22  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36.0 index 2147404488 <Up Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 02:02:22  JUN-L3-FF-Cluster rpd[1329]: EVENT <UpDown> ge-0/0/36 index 223 <Up Broadcast Multicast> address #0 50.c5.8d.aa.72.83
Sep  5 02:02:22  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 574, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36
Sep  5 02:02:22  JUN-L3-FF-Cluster mib2d[1328]: SNMP_TRAP_LINK_UP: ifIndex 578, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/0/36.0
Sep  5 02:02:52  JUN-L3-FF-Cluster lldpd[1352]: LLDP_NEIGHBOR_UP: A neighbor has come up for interface ge-0/0/36.0. Now, this interface has 1 neighbor
  1.       Check the same thing at 364 ground floor switch. 

Sep  5 00:18:36  JUN-4200-364GF-L3 mib2d[830]: SNMP_TRAP_LINK_DOWN: ifIndex 598, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/47
Sep  5 00:18:49  JUN-4200-364GF-L3 mib2d[830]: SNMP_TRAP_LINK_DOWN: ifIndex 598, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/47
Sep  5 00:18:59  JUN-4200-364GF-L3 mib2d[830]: SNMP_TRAP_LINK_DOWN: ifIndex 596, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/46
Sep  5 00:19:31  JUN-4200-364GF-L3 mib2d[830]: SNMP_TRAP_LINK_DOWN: ifIndex 598, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/47
Sep  5 00:20:19  JUN-4200-364GF-L3 mib2d[830]: SNMP_TRAP_LINK_DOWN: ifIndex 598, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/47
Sep  5 00:24:08  JUN-4200-364GF-L3 mib2d[830]: SNMP_TRAP_LINK_DOWN: ifIndex 598, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/47
5.       
6.       
  1.  Then check lldp statistics which is showing some abnormalities because frames are generating at downstream interfaces but at upstream 1.e at core switch frames generation is not proper only few frames were generated.


hikumar@JUN-4200-364GF-L3> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info     System Name
ge-0/0/8.0         -                   00:18:fe:57:9d:80   24           HP-L2-364-WFMBAY
ge-0/0/20.0        -                   00:1d:b3:4a:5c:80   50           HP-L2-364-GF-SW8
ge-0/0/47.0        ae0.0               50:c5:8d:aa:72:80   ge-0/0/36.0  JUN-L3-FF-Cluster
ge-0/0/45.0        ae0.0               50:c5:8d:aa:72:80   ge-0/0/37.0  JUN-L3-FF-Cluster
ge-0/0/44.0        ae0.0               50:c5:8d:aa:72:80   ge-1/0/36.0  JUN-L3-FF-Cluster
ge-0/0/46.0        ae0.0               50:c5:8d:aa:72:80   ge-1/0/37.0  JUN-L3-FF-Cluster

At core switch 

hikumar@JUN-L3-FF-Cluster> show lldp neighbors
Local Interface    Parent Interface    Chassis Id          Port info          System Name
ge-0/0/4.0         ae4.0               00:16:b9:0e:d3:80   22                 HP-L3-GF-365
xe-0/1/0.0         -                   00:2a:6a:fc:3f:a2   Ethernet1/27       FPGGNFI365A-A
ge-0/0/39.0        ae2.0               28:c0:da:31:58:00   ge-0/0/44.0        JUN-4200-NB-FF
ge-0/0/38.0        ae2.0               28:c0:da:31:58:00   ge-0/0/45.0        JUN-4200-NB-FF
ge-1/0/38.0        ae2.0               28:c0:da:31:58:00   ge-0/0/46.0        JUN-4200-NB-FF
ge-1/0/39.0        ae2.0               28:c0:da:31:58:00   ge-0/0/47.0        JUN-4200-NB-FF
ge-1/0/36.0        ae0.0               28:c0:da:36:5a:80   ge-0/0/44.0        JUN-4200-364GF-L3
ge-0/0/37.0        ae0.0               28:c0:da:36:5a:80   ge-0/0/45.0        JUN-4200-364GF-L3
ge-1/0/37.0        ae0.0               28:c0:da:36:5a:80   ge-0/0/46.0        JUN-4200-364GF-L3
ge-0/0/36.0        ae0.0               28:c0:da:36:5a:80   ge-0/0/47.0        JUN-4200-364GF-L3
  1.         Then check lldp statistics which is showing some abnormalities because frames are generating at downstream interfaces but at upstream 1.e at core switch frames generation is not proper only few frames were generated.
hikumar@JUN-4200-364GF-L3> show lldp statistics
Interface    Parent Interface  Received  Unknown TLVs  With Errors  Discarded TLVs  Transmitted  Untransmitted
ge-0/0/1.0   -                 1667644   0             0            0               1548161      0
ge-0/0/2.0   -                 1673510   0             0            0               1548158      0
ge-0/0/3.0   -                 1560456   0             0            0               1548101      0
ge-0/0/6.0   -                 1673653   0             0            0               1548156      0
ge-0/0/7.0   -                 1673578   0             0            0               1548074      0
ge-0/0/8.0   -                 1525684   0             0            0               1525569      0
ge-0/0/9.0   -                 1109772   0             0            0               1085581      0
ge-0/0/13.0  -                 1673560   0             0            0               1548144      0
ge-0/0/14.0  -                 1673712   0             0            0               1548156      0
ge-0/0/20.0  -                 1539599   0             0            0               1548167      0
ge-0/0/44.0  ae0.0             1673406   0             0            0               1548144      0
ge-0/0/45.0  ae0.0             1673650   0             0            0               1548167      0
ge-0/0/46.0  ae0.0             1673123   0             0            0               1547873      0
ge-0/0/47.0  ae0.0             35256     0             0            0               31542        0


hikumar@JUN-L3-FF-Cluster> show lldp statistics | match ae0
ge-1/0/36.0  ae0.0             323       0             0            0               351          0
ge-1/0/37.0  ae0.0             323       0             0            0               350          0
ge-0/0/36.0  ae0.0             323       0             0            0               350          0
ge-0/0/37.0  ae0.0             323       0             0            0               350          0

Then gracefully restart lldp services which makes the things works and counter start increasing at a proper rate.
Run following commands for gracefully restart the lldp protocols
sh system processes
restart lldp-services gracefully
for verification, run following commands:

sh lldp neighbors
sh lldp statistics


7.       Also, after that interface “ae0” flapping has also stopped.
Monitor things and get it confirmed from the floor user, no issue has duplicated

Thanks












Comments

Post a Comment