Well, this will be a story about a voice gateway, which is restarting every 12th day:)
For mounts we didn’t notices the problem, because the router was quickly rebooted. After we implemented central logging (I will write about this a post), we received mail about the reboot. We begun writing mails to the Cisco TAC. Personally, I have always bad feeling, when I have to give a mysterious issue to the vendor’s support team.
This was however a nice job. I would like to include the final mail about the problem, because I think it is “funny”:) Of course you can imagine, that we had to escalate the problem many times, until we received the solution…but Cisco TAC, I like you.
** Problem description
Every 12 days, your router running 12.4(24)T5 is crashing, always due to ISDN L2 Process being unable to allocate a packet :
002261: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_FATAL: ISDN FATAL ERROR: file ../isdn/lif_common.c, function LIF_GetPkt, message: malloc of PKT failed
002262: *May 20 2011 01:52:55.757 CET: %ISDN-2-ISDN_EXIT: malloc of PKT failed
LIF_GetPkt calls LIF_Get_block which allocates a chunk in isdn_pkt_chunk only if the amount of allocated packets is lower than a limit (isdn_pkt_limit) :
In the coredump, we see that you reached the limit of 6014 packets :
(cisco-6.6.2-r4k-gdb) p/d Used_pkt
$97 = 6014
(cisco-6.6.2-r4k-gdb) p/d isdn_pkt_limit
$99 = 6014
Used_pkt reached isdn_pkt_limit so ISDN triggers a crash (which is a bit aggressive in my opinion) <- xcke: 🙂 At some point, you also provided 'show isdn memory' which confirms this :VGW#Show isdn memory MEMORY POOL STATISTICS BlockType in use max used limit initial increment PKT_BTYPE 5700 5702 6014 62 31
At that point in time, we were already at 5700.
You also provided q921 debugs. If we filter the INFO frames with sapi=16, we see this :
$ grep INFO\ isdn.log
000380: *May 19 2011 10:19:52.885 CET: ISDN BR2/3 Q921: Net RX <- INFO sapi=16 tei=3, ns=65 nr=0 000639: *May 19 2011 10:22:52.869 CET: ISDN BR2>
WTF?:) This will be an EVIL FAX device I think 🙂