In Service Software Updrade (ISSU). Sups are in Active/Standby Mode all the time. Modules are online insertion and removal supported on all chassis (OIR). Sup that boots first become Active, Active SUP controls the control and data plane decisions. Active SUP synchronize config and protocol state info to Standby SUP. Interfaces on Standby SUP are oprtational during the Standby state of SUP. Route Processor Redundancy (RPR): In this Standby software is not fully initialized. Upon Switchover the standby sup become active but it has to complate the boot Process. along with that all line cards will be reloaded and reprogrammed accordingly. In RPR+ Standby sup is completely initiallized and line cards will not reload upon switchover. Running config is synchronized between both Active and Standby sups. No link layer or control-plane info is synchronized between sups. NFS: It introduce the capabilities of Graceful restarts. It separates the functionality of Control-plan and data-plan during the SUP Switchover. In event of Control-plan failure it use the information from CEF to forward the data, it will not tear down the neighbor relationship with the peer. NSF aware router understand the GRACEFUL restart mechanisms. SSO: SSO introduce more capabilities of synchronize the protocol states and any other service related parameters. With the help of SSO Spanning tree info on Standyby SUP will be ketp up-to-date. Routing Protocol will be restarted and all dynamic route entries will be flushed and needs to relearn. Static routes will not be affected in switchover. Upgrade IOS on SUPs: Load new IOS on Standby SUP. Once IOS updated on SUP, reset currently active module so newly loaded IOS SUP will become active. Now upload new IOS on Standby SUP. ES-20 Card: ES-20 card add the MPLS Application supports in Cisco 7600. ES-20 card can be inserted in slot 1 to 8 in 7613. ES-20 card can have two 10GB (Single Mode fiber) interfaces or twenty(20) 1GB interface. It does supports OIA (Online Insert/HOT Swappable capability). Es-20 Card is not supported on Cisco 7603. ES-20 card will only supported on chassis with Sup-720. BGP NSF: ES-20 During BGP session establishment BGP Peer negotiate neighbor capabilities using OPEN messages. During SUP switchover NSF aware BGP peer mark all routes as "STALE" routes and continue to forwarde the packate for perticular time(120 seconds). After switchover newly active SUP will negotiate BGP session with peer and form new adjacency. "bgp graceful-restart" command under router bgp will enable the NSF capability.
ATM & FR works same way. ATM & FR; NBMA technology doesn’t have native capability of broadcast/Multicast. You need to define broadcast keyword in layer-3 to layer-2 binding (in frame-relay map in FR as well as broadcast ip in ATM) In Ethernet router send only one multicast packet per interface. FR/ATM router sends multiple multicast packets on per PVC which has broadcast keyword per interface (ex: if we have total 5 pvc on interface and have 3 pvc enabled with broadcast keyword router will make three broadcast packet for 3 pvc with broadcast capability).
Multipoint FR/ATM interface: need to define layer-3 to layer-2 mapping mandatory or use inverse ARP in FR and ilmi in ATM. FR use Frame-relay map command | ATM use protocol ip under pvc n/n configuration command.
HDLC and PPP are point to point only.
PPP: Extends functionality of L2 protocol like:
i. Clear text username and password
i. Clear text username and MD5 hashed password
PPPoE: PPPoE using VPDN support only one PPPoE profile; PPPoE using bba-profile support the multiple PPPoE profile on Routers. ISO support two variation: PPPoE Dialin (PE): Accept Connections from Client PPPoE Client(CE): Initiate PPPoE negotiation to Server Peer Neighbor Route: Client receive the IP address from PE, and CE will installed the assigned IP address as well as Virtual-temp IP address on PE (client IP address) into routing table using peer neighbor route.So peer neighbor route will install the client ip address assigned by PE router into CE routing table to make that client reachable. Virtual-Temp interface: It always remain in down/down under show ip interface brief output. You can use it for defining layer-3 property as well as PPP layer-2 property like authentication parameters. Client Dialer interface: Dialer pool number on interface dialer and pppoe-client dialer-pool number on Ethernet/fast-Ethernet interface of client should match the number. Layer1/2/3 Connectivity issue: Main issues Performance: Latency, jitter, packet loss. & Connectivity: link, reach ability. In SLOW network complains look at the statics on interface to see the excessive amount of broadcast, there might be broadcast storm. Layer1: Connectivity: Check for link Traffic: Packets are passing or not? If yes then look at how many? Speed/Duplex: check for mismatch on both side. To resolve it: 1. there might be NIC or driver issue 2. There might be faulty cable (some tools are available in IOS cli to check it) a. Check for type of cable used: fiber/Cat5/Cat6 b. Do TDR test on cable; only available for copper ports. i. TDR find out cable fault ii. Cat5 has four cable pair iii. TDR detects fault in cable pair such as opens or short iv. TDR can determine position of cable v. TDR is test is intensive for CPU; avoid using on production vi. Supported on 6500/4500/3750. vii. Command “test cable-diagnostics tdr interface gig3/3” viii. To see TDR result “show cable-diagnostics tdr int gig 3/3”
3. To much traffic can be broadcast storm, oversubscribed interface. 4. Hardcode configuration of speed and duplex. 5. DOM: Digital Optical Monitoring do testing on optical cable a. Run on selected GBIC/SFP Physical interface: Do show interface command to see any input/output counters incrementing as well as check for any errors/crc/collisions. (This issues mostly because of duplex/speed mismatch) Port startup delay: port is having delay to come up because of DHCP (ip address resolution) or 802.1X.(Layer-2 authentication). You can configure “switchport host” on access port, its same as port-fast.
Check Duplex setting and verify topology: Do show cdp neighbor details; there you can see connected interface as well as duplex (at the end of output of command) of the connected port.
Layer-2 troubleshooting: 1. Trunk is forming or not 2. Ether channel is forming or not 3. Bridge table 4. Spanning tree is working as expected or not? Trunk issue: 1. ISL & 802.1Q mismatch check 2. Check for Dynamic Trunking Protocol configuration (non-negotiate/Desirable/ON/AUTO) a. DTP packets will be sent on VLAN 1 in ISL and On native VLAN in 802.1Q b. Port will not participate in STP during negotiation c. Use “show interface g 3/3 trunk” to check for which mode is running on port (Desirable or ON or AUTO or non-negotiate) 3. Native VLAN mismatch
Etherchannel: 1. Check for PAGP and LACP mismatch, use LACP in multi-vendor environment. 2. Port will be in err-disable state if you have misconfigured mode of PAGP/LACP on both end.
Layer-3 Troubleshooting: Show ip route/show ip arp/extended ping with various option like record route/traceroute VTP: Switch VLAN configuration changed with out any manual configuration: You might have added new switch with higher configuration revision number and VTP server mode. Avoid this condition by putting new switch with transparent mode and then add new switch in network. Configuration revision number increase by 1 if you will change the configuration of VLAN on switch. “show vtp status” will give you switch VTP mode as well as VTP configuration revision number. You can reset (0) the configuration revision number by changing the vtp domain name (assign bogus or any VTP domain to reset the switch configuration revision number and then again configure the required VTP domain name.) STP Troubleshooting: Document the STP topology to make troubleshooting easy. STP Feature (Also supported in MSTP and PVST+):
MSTP (802.1S): Here you can bundle the numbers of VLAN to the part of one instance and run STP on instance rather than running STP on individual switch. Much less CPU utilization, highly scalable, reduce the complexity of topologies, Use two instance to achieve the proper utilization of resources using load balancing of two different instance. RSTP (802.1W): To improve convergence time. No need to tune timers. Backwards compatibl ewith STP and PVST+ so you can have network running different protocol at same time. Troubleshoot STP: Use show spaning-tree vlan 1 brief, you can identify the which protocol switch is running:STP or PVST etc. Show spanning-tree summary to identify the Root bridge of STP. Show spanning-tree VLAN 1 details, check for when and number of topology changes received, check for number BPDU sent and received. Enabling port-fast on access port will suppress the TCN generation regarding to that port as user is connected to that. Check for process CPU utilization for spanning-tree process. UDLD: It will detect if one way fiber link fail as switch use two different fiber cable one for only receive and one only for transmitting. So if switch RX cable is faulty the it will not receive BPDU from the Rootbridge and It will announce itself as a ROOTBRIDGE which will cause STP instability, UDLD will help in such a conditions. “Udld aggressive” will put port in error-disable mode if there is condition of UDLD. “show udld gig 3/3” to verify the UDLD status. Enable UDLD globally by using “udld enable” on global configuration mode. IOS Troubleshooting: sh interface | in drops sh interface | in errors (run both the commands multiple time to see the drops or errors number increasing or not, as well as check for when you have clear the interface status and if it has not done since long then first of clear the counters.) Show interface summary (You will have status of 5 min input and output traffic status. You can configure "load-interval" on interface or interface range command depends on platform to smaller value than 5 mins depends on your requirement) Configure logging buffer before running any debug commands Keep large memory in router and have two image of IOS for redundancy. To isolate Hardware Failure try removing module at a time and reboot the system so that way you can identify the card is faulty or module is faulty. If your router is not detecting new hardware card then check for IOS compatibility for that hardware. Power Issue: Two power supplied use redundant mode (default), you can set it to combined mode where both POWER supply work same time. SIP/SPA: SIP is processor cards for particular type of port adapters. So make sure the proper combination of SIP/SPA. As some kind of SIP is not supported some kind of SPA so you can not insert any SPA cards into SIP slots. Data Flow Troubleshooting: Inbound Packet Loss: Runts: Packet too small Giants: Packet too large CRC: CRC check failed Frame: Packet framed incorrectly Ignored: Packet NOT accepted by interface processor Throttles: No buffer to refill receive ring, this happen because of interface interrupts too often Overrun: Interface processor is copying the packets faster than memory can accept it.
Receive and Process Packet Loss: Inputs or received packets are needed process queue for routing as well as control traffic.
CEF: CEF does per source and destination pair loadbalancing. Use "show ip cef exect-route" to fidn out which path packet flow between particular source and destination is following on. CEF Adjacency Type:
Process switch does per packet loadbalancing and that will help you to find out the faulty link. By default ether channel do the per source MAC address loadbalancing o, SUP-720 does the per SOURCE-DESTINATION IP loadbalancing.
show proc cpu gives an total CPU utilization by process and total CPU utilization percentage because of interrupt (look for "CPU Utilization for five second: n%/n%), if you see the interrupt percentage is high then it is mostly because of interface interrupts, check for traffic statics. MALLOC Failures: will tell you amount of memory the process attempted to allocate and fail to allocate it. “ip route-cache” enable the Fast switching, “no ip route-cache” enable the process switch.
IOS Device reboots reason: Cisco router reboot or router restart or switch reboot or switch restart reason from show version •power-on — System was reset with the initial power on or a power cycling of the device. •s/w peripheral — System was reset due to a software peripheral. •s/w nmi — System was reset by a non-maskable interrupt (NMI) originating in the system software. For example, on some systems, you can configure the device to reset automatically if two or more fans fail. •push-button—System was reset by manual activation of a RESET push-button (also called a hardware NMI). •watchdog—System was reset due to a watchdog process. •unexpected value—May indicate a bus error, such as for an attempt to access a nonexistent address (for example, "System restarted by bus error at PC 0xC4CA, address 0x210C0C0"). When GRE Tunnel goes down: · There is no route to the tunnel destination address. · The interface that anchors the tunnel source is down. · The route to the tunnel destination address is through the tunnel itself. If you have static default route then also Tunnel interface will remain up. Keepalive is the solution to get the correct status of tunnel. RIB: Store best routes, RIB is sync with FIB, and FIB has binding with actual interface for destination (CEF table). Cisco IOS remove the EIGRP neighbor before shutting down the Interface. Static ROUTE pointing to the Ethernet interface need to do ARP every time and number of ARP depends on the number of routers connected on that Segment. So this kind of static route coz the large numbers of ARP as well as layer-2 rewrite function (so don’t use Broadcast interfaces for this kind of static routes). Static Routes and routing protocol: OSPF and ISIS will not include static route pointing to the connected interface in the routing table. Whereas EIGRP and BGP will advertise that static routes pointing to the connected interface without redistribution if BGP or EIGRP has network command for the subnet of that interface (take care of this). Routing protocols with Null routes: EIGRP/OSPF/ISIS: if you configure summary address (area range in OSFP) for connected subnet, it will automatically introduce the NULL Route in RIB. It helps to break the loops in network if the one component of the summery is active and other is down. "no discard route" in OSPF will prevent the generation of Discard or Null route when you generate the summary. Redistribution will take place only on the routes which are available in RIB. Similarly filter applying to the Redistribution only take action on the routes which are in RIB. Applying TAG as well as redistribution by matching tag is the best way to simplify the redistribution and loop prevention in network. Prefix list:
ACLs: 1. Use ACL to stop unauthorized access 2. ACL to recognize DOS attack 3. ACL to stop DOS attack 4. IP Fragment handling 5. NetFLow with ACL. "LOG" keyword at the end of ACL will generate the LOG message at console. "LOG-INPUT" at the end of ACL will include input interface and source MAC or VC in logging message.
SMURF ATTACK (ICMP): Attacker sends a ping echo request to broadcast address on network with source address of the victim. So All host will send echo-reply to the victim host instead of attacker IP address. access-list 170 permit icmp any any echo access-list 170 permit icmp any any echo-reply access-list 170 permit ip any any
interface serial 0 ip access-group 170 in
FRAGGLE ATTACK (UDP ECHO): Its variation of Smurf Attack. Instead of using ICMP echo attacker uses UDP echoes.
access-list 180 permit udp any any eq echo access-list 180 permit udp any eq echo any access-list 180 permit ip any any
interface serial 0 ip access-group 180 in
SYN FLOOD ATTACK (TCP SYN): SYN floods are DoS attacks cause by sending large number of TCP SYN to server without completing handshake. Upon receiving SYN message server will keep sending SYN-ACK.
access-list 190 permit tcp any any established access-list 190 permit tcp any any access-list 190 permit ip any any
interface serial 0 ip access-group 190 in
CAR: CAR to RATE-LIMIT ICMP Floods interface serial 0 rate-limit input access-group 110 256000 8000 8000 conform-action transmit exceed-action drop
access-list 110 permit icmp any any echo access-list 110 permit icmp any any echo-reply
CAR to Rate-Limit SYN Floods: access-list 110 deny tcp any any established access-list 110 permit tcp any any
interface serial0 rate-limit input access-group 110 64000 8000 8000 conform-action transmit exceed- action drop
Dropping and Policing using MQC: 1. Define Class map to match traffic 2. Define Policy policy-map mypolicy class-map myclass drop 3. Apply policy-map to interface.
uRPF: Router examine the source of packet arrived on uRPF enabled interface. If router find the source address in routing table and matches the interface on which packet was received. (Same as RPF in multicast, here unicast source IP address should be reachable via the received interface)
Router(cofnig)# access-list 101 deny ip 10.1.1.0 0.0.0.255 any log-input Router(config)# access-list 101 permit ip 172.16.1.0 0.0.0.255 any log-input Router(config)# interface <interface-name> Router(config-if)# ip verify unicast reverse-path 101
As 10.1.1.0 is internal network so when packet received on uRPF enable interface then router will do RPF lookup for source IP address in routing table, as 10.1.1.0 is not reachable via the interface on which packet received it fails the uRPF check so it will be drop.
NetFlow: "ip route-cache flow" and "ip flow ingress" perform the same function. to look at flow data use "show ip cache flow" You can export the netflow data to external system using "ip flow-export" on global configuration mode.
|