28 May

Cisco Voice Gateway crash

Well, this will be a story about a voice gateway, which is restarting every 12th day:)
For mounts we didn’t notices the problem, because the router was quickly rebooted. After we implemented central logging (I will write about this a post), we received mail about the reboot. We begun writing mails to the Cisco TAC. Personally, I have always bad feeling, when I have to give a mysterious issue to the vendor’s support team.
This was however a nice job. I would like to include the final mail about the problem, because I think it is “funny”:) Of course you can imagine, that we had to escalate the problem many times, until we received the solution…but Cisco TAC, I like you.

** Problem description

Every 12 days, your router running 12.4(24)T5 is crashing, always due to ISDN L2 Process being unable to allocate a packet :

002261: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_FATAL: ISDN FATAL ERROR: file ../isdn/lif_common.c, function LIF_GetPkt, message: malloc of PKT failed
002262: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_EXIT: malloc of PKT failed

LIF_GetPkt calls LIF_Get_block which allocates a chunk in isdn_pkt_chunk only if the amount of allocated packets is lower than a limit (isdn_pkt_limit) :

In the coredump, we see that you reached the limit of 6014 packets :

(cisco-6.6.2-r4k-gdb) p/d Used_pkt
$97 = 6014
(cisco-6.6.2-r4k-gdb) p/d isdn_pkt_limit
$99 = 6014

Used_pkt reached isdn_pkt_limit so ISDN triggers a crash (which is a bit aggressive in my opinion) <- xcke: 🙂 At some point, you also provided 'show isdn memory' which confirms this :

VGW#Show isdn memory 
        BlockType           in use max used      limit    initial    increment
        PKT_BTYPE             5700     5702     6014       62         31

At that point in time, we were already at 5700.

You also provided q921 debugs. If we filter the INFO frames with sapi=16, we see this :

$ grep INFO\ isdn.log
000380: *May 19 2011 10:19:52.885 CET: ISDN BR2/3 Q921: Net RX <- INFO sapi=16 tei=3, ns=65 nr=0 000639: *May 19 2011 10:22:52.869 CET: ISDN BR2>

WTF?:) This will be an EVIL FAX device I think 🙂