Cisco Voice Gateway crash

xcke — Sat, 28 May 2011 09:38:32 +0000

Well, this will be a story about a voice gateway, which is restarting every 12th day:)
For mounts we didn’t notices the problem, because the router was quickly rebooted. After we implemented central logging (I will write about this a post), we received mail about the reboot. We begun writing mails to the Cisco TAC. Personally, I have always bad feeling, when I have to give a mysterious issue to the vendor’s support team.
This was however a nice job. I would like to include the final mail about the problem, because I think it is “funny”:) Of course you can imagine, that we had to escalate the problem many times, until we received the solution…but Cisco TAC, I like you.

** Problem description

Every 12 days, your router running 12.4(24)T5 is crashing, always due to ISDN L2 Process being unable to allocate a packet :

002261: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_FATAL: ISDN FATAL ERROR: file ../isdn/lif_common.c, function LIF_GetPkt, message: malloc of PKT failed
002262: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_EXIT: malloc of PKT failed

LIF_GetPkt calls LIF_Get_block which allocates a chunk in isdn_pkt_chunk only if the amount of allocated packets is lower than a limit (isdn_pkt_limit) :

In the coredump, we see that you reached the limit of 6014 packets :

(cisco-6.6.2-r4k-gdb) p/d Used_pkt
$97 = 6014
(cisco-6.6.2-r4k-gdb) p/d isdn_pkt_limit
$99 = 6014

Used_pkt reached isdn_pkt_limit so ISDN triggers a crash (which is a bit aggressive in my opinion) <- xcke:

At some point, you also provided 'show isdn memory' which confirms this :
VGW#Show isdn memory 
        MEMORY POOL STATISTICS
        BlockType           in use max used      limit    initial    increment
        PKT_BTYPE             5700     5702     6014       62         31
At that point in time, we were already at 5700.

You also provided q921 debugs. If we filter the INFO frames with sapi=16, we see this :

$ grep INFO\ isdn.log
000380: *May 19 2011 10:19:52.885 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=65 nr=0
000639: *May 19 2011 10:22:52.869 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=66 nr=0
000905: *May 19 2011 10:25:52.849 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=67 nr=0
001164: *May 19 2011 10:28:52.885 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=68 nr=0
001424: *May 19 2011 10:31:52.869 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=69 nr=0
001683: *May 19 2011 10:34:52.853 CET: ISDN BR2/3 Q921: Net RX <- info sapi=16 tei=3, ns=70 nr=0
001942: *May 19 2011 10:37:52.833 CET: ISDN BR2/3 Q921: Net RX <- INFO sapi=16 tei=3, ns=71 nr=0

INFO frames are received every 3 minutes. To fill up the ISDN packet chunk with one packet every 3 minutes, it takes 3*6014/60/24 = 12 days.

sapi=16 is X.25. However, you have no X25 configuration on your router.

** Action plan

Could you provide the configuration from the other side of BRI2/3 ? We probably need to disable X.25 there to stop the leak.

To confirm this, after you've changed the config on the other side, monitor 'sh isdn memory | i PKT_BTYPE' and check that it no longer increments (for more than 3 minutes).

WTF?:) This will be an EVIL FAX device I think

Internetworking.hu » ISDN

Cisco Voice Gateway crash