UPDATE: I completed the CCIE R&S Written Exam in December 2011, so I will begin to study for the LAB. I already created an MS Project Plan (According to this, I will take the LAB in August).

I will use only INE materials, RFCs, books, Dynamips, IOU and rack rentals – hope it will be enough:)

** Problem description

Every 12 days, your router running 12.4(24)T5 is crashing, always due to ISDN L2 Process being unable to allocate a packet :

002261: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_FATAL: ISDN FATAL ERROR: file ../isdn/lif_common.c, function LIF_GetPkt, message: malloc of PKT failed
002262: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_EXIT: malloc of PKT failed

LIF_GetPkt calls LIF_Get_block which allocates a chunk in isdn_pkt_chunk only if the amount of allocated packets is lower than a limit (isdn_pkt_limit) :

In the coredump, we see that you reached the limit of 6014 packets :

(cisco-6.6.2-r4k-gdb) p/d Used_pkt
$97 = 6014
(cisco-6.6.2-r4k-gdb) p/d isdn_pkt_limit
$99 = 6014

Used_pkt reached isdn_pkt_limit so ISDN triggers a crash (which is a bit aggressive in my opinion) <- xcke:

At some point, you also provided 'show isdn memory' which confirms this :

VGW#Show isdn memory 
        MEMORY POOL STATISTICS
        BlockType           in use max used      limit    initial    increment
        PKT_BTYPE             5700     5702     6014       62         31

At that point in time, we were already at 5700.

You also provided q921 debugs. If we filter the INFO frames with sapi=16, we see this :

$ grep INFO\ isdn.log
000380: *May 19 2011 10:19:52.885 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=65 nr=0
000639: *May 19 2011 10:22:52.869 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=66 nr=0
000905: *May 19 2011 10:25:52.849 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=67 nr=0
001164: *May 19 2011 10:28:52.885 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=68 nr=0
001424: *May 19 2011 10:31:52.869 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=69 nr=0
001683: *May 19 2011 10:34:52.853 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=70 nr=0
001942: *May 19 2011 10:37:52.833 CET: ISDN BR2/3 Q921: Net RX <- INFO sapi=16 tei=3, ns=71 nr=0

INFO frames are received every 3 minutes. To fill up the ISDN packet chunk with one packet every 3 minutes, it takes 3*6014/60/24 = 12 days.

sapi=16 is X.25. However, you have no X25 configuration on your router.

** Action plan

Could you provide the configuration from the other side of BRI2/3 ? We probably need to disable X.25 there to stop the leak.

To confirm this, after you've changed the config on the other side, monitor 'sh isdn memory | i PKT_BTYPE' and check that it no longer increments (for more than 3 minutes).

WTF?:) This will be an EVIL FAX device I think

335414: May 19 19:07:49 CEST: %ISDN-6-CONNECT: Interface Serial0/0/0:26 is now connected to xxx N/A
335415: May 19 19:07:51 CEST: %ISDN-6-DISCONNECT: Interface Serial0/0/0:26 disconnected from xxx , call lasted 2 seconds
335417: May 19 19:08:18 CEST: %ISDN-6-DISCONNECT: Interface Serial0/0/0:27 disconnected from xxx , call lasted 871 seconds
335418: May 19 19:08:20 CEST: %ISDN-6-CONNECT: Interface Serial0/0/0:30 is now connected to xxx N/

You have an option however, to filter out these type of log messages with Message discriminator. For example we can define our discriminator (in one line):

logging discriminator NOISDNLO severity drops 
6 facility drops ISDN msg-body drops connected

Here, we will drop log messages, which has a Severity of 6, a facility of “ISDN” and the message body contains “connected” word. Be careful to not filter out important messages.

Once you defined your discriminator, you can tie it to a syslog session, or to the console / monitor / buffer objects.

logging host 1.1.1.1 discriminator NOISDNLO
logging console discriminator NOISDNLO
logging monitor discriminator NOISDNLO

You can check the result with the show logging command:

RouterA#sh logging
Syslog logging: enabled (0 messages dropped, 1612 messages rate-limited,
1 flushes, 0 overruns, xml disabled, filtering disabled)

Active Message Discriminator:
NOISDNLO severity group drops 6
facility drops ISDN
msg-body drops connected

No Inactive Message Discriminator.

Console logging: level debugging, 139092 messages logged, xml disabled,
filtering disabled, discriminator(NOISDNLO),
0 messages rate-limited, 198075 messages dropped-by-MD
Monitor logging: level debugging, 30127 messages logged, xml disabled,
filtering disabled, discriminator(NOISDNLO),
0 messages rate-limited, 198075 messages dropped-by-MD
Logging to: vty514(0)
Buffer logging: level debugging, 337323 messages logged, xml disabled,
filtering disabled
Logging Exception size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled

No active filter modules.

ESM: 0 messages dropped

Trap logging: level informational, 337330 message lines logged
Logging to 1.1.1.1 (udp port 514, audit disabled,
authentication disabled, encryption disabled, link up),
139255 message lines logged,
0 message lines rate-limited,
198075 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering enabled, discriminator (NOISDNLO)

You can read more about the feature here.

parser config cache interface

But what does this feature do? Cisco says:

Until the Configuration Generation Performance Enhancement feature was introduced, NVGEN always had to query the entire system and could generate only a total configuration. The time required to process the running configuration creates performance problems for configuration management, because completion of the NVGEN operation can take many minutes.

The Configuration Generation Performance Enhancement feature reduces the execution time for NVGEN processes and is especially useful for managing large configuration files that contain numerous interface configurations. This feature provides faster execution of commands that process the running system configuration by caching interface configuration information in system memory, and by retrieving only configuration information that has changed.

If you are using IOS 12.3(7)T or newer, and the router has enough memory, you can speed up the things. After I configured the feature, the sh run command was ready in less then a sec.

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t7/feature/guide/gtinvgen.html

Actually, the configuration is very basic:

On R2:

interface Ethernet0/0
 ip address 10.10.10.2 255.255.255.252
 ip nat outside
 ip virtual-reassembly

interface Ethernet0/1 
 ip address 10.10.11.1 255.255.255.252 
 ip nat inside 
 ip virtual-reassembly

ip nat inside source static tcp 1.1.1.1 23 1.1.1.1 666 extendable

Now, if I telnet from R1 to 1.1.1.1:666, it works, and I can see the following debug messages in R2

*Apr 24 20:48:37.543: NAT*: TCP s=52634, d=666->23
*Apr 24 20:48:37.547: NAT*: TCP s=23->666, d=52634
*Apr 24 20:48:37.559: NAT*: TCP s=52634, d=666->23
*Apr 24 20:48:37.559: NAT*: TCP s=52634, d=666->23
*Apr 24 20:48:37.611: NAT*: TCP s=23->666, d=52634
*Apr 24 20:48:37.611: NAT*: TCP s=23->666, d=52634
*Apr 24 20:48:37.615: NAT*: TCP s=52634, d=666->23

During the testing, I tried to make this work using the LAN address of the router (10.10.11.2).  So I changed the NAT rule:

ip nat inside source static tcp 10.10.11.2 23 10.10.11.2 666 extendable

After this, I could see the following log messages. I also issued a sh ip arp command on R2.

*Apr 24 23:04:31.367: %IP-4-DUPADDR: Duplicate address 10.10.11.2 on Ethernet0/1, sourced by aabb.cc00.0310
*Apr 24 23:05:01.371: %IP-4-DUPADDR: Duplicate address 10.10.11.2 on Ethernet0/1, sourced by aabb.cc00.0310
*Apr 24 23:05:33.283: %IP-4-DUPADDR: Duplicate address 10.10.11.2 on Ethernet0/1, sourced by aabb.cc00.0310
*Apr 24 23:06:04.279: %IP-4-DUPADDR: Duplicate address 10.10.11.2 on Ethernet0/1, sourced by aabb.cc00.0310

R2#sh ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.10.10.1 144 aabb.cc00.0100 ARPA Ethernet0/0
Internet 10.10.10.2 – aabb.cc00.0200 ARPA Ethernet0/0
Internet 10.10.11.1 – aabb.cc00.0210 ARPA Ethernet0/1
Internet 10.10.11.2 – aabb.cc00.0210 ARPA Ethernet0/1

As you can see, the 10.10.11.2 address seems to be now an address of Ethernet0/1 on R2. I tried this on real 2811 router, in GNS3 with 7200 router and also with IOU. I really don’t know the cause of this. And here is the ARP debug:

*Apr 24 23:11:49.279: IP ARP: rcvd rep src 10.10.11.2 aabb.cc00.0310, dst 10.10.11.2 Ethernet0/1
*Apr 24 23:11:49.279: IP ARP: Gratuitous ARP throttled.
*Apr 24 23:11:49.279: IP ARP: 10.10.11.2 added to arp_defense_Q
*Apr 24 23:11:49.375: IP ARP: 10.10.11.2 removed from arp_defense_Q
*Apr 24 23:11:49.375: IP ARP: sent rep src 10.10.11.2 aabb.cc00.0210, dst 10.10.11.2 aabb.cc00.0210 Ethernet0/1

If you have an idea, why IOS act like this, please comment:)

Right now, I’m preparing for the CCIE R&S written

Code test