Saturday, February 03, 2007

Asterisk, WiFi, HomePlug, and an Avaya SIP Phone

The Digital Aggregates corporate data center, such as it is, is a mash-up of bits and pieces of hardware spread all over the palatial Overclock estate. The hardware was selected by the tried and true method of mostly whatever works and sometimes whatever was handy at the time.

The central network stack is a pile of several LinkSys boxes on top one of the A/V cabinets in the family room. This is simply because our broadband internet provider is also our cable television provider, Comcast. Proximity to the cable was necessary for the enablement of two critical activities in the Overclock household, watching the new Battlestar Galactica, and cruising the web. One of the LinkSys boxes is a cable modem, and another, predictably, is a router/WiFi access point.

There are a lot of devices on our network, including an H-P full-duplex color laser printer with integrated print server (running VxWorks, as it turns out), several Linux servers, Mrs. Overclock's Mac Powerbook, my IBM Thinkpad, my Palm PDA, our TiVo, an Avaya 4610SW SIP (VOIP) phone, and probably some other stuff that I've forgotten about.

Although the router has a four-port Ethernet switch, until now only one device was plugged into that switch, our TiVo digital video recorder, the machine without which life as we know it could not exist. As Mrs. Overclock has been known to say, "TiVo loves us and wants us to be happy!" The only reason the TiVo warranted such special treatment was its physical proximity to the router. Otherwise, it would have been a citizen of the WiFi network, just like all the other devices on the heavily-armed Overclock compound.

The SIP phone is my principle telephone in my home office. It is serviced by one of the Linux servers which runs Asterisk, an open source PBX. The Asterisk server has a four-port Digium card that connects it to an analog phone (used only for troubleshooting), and to the PSTN via our second phone line. I chose the Avaya SIP phone because it had both of what are for me absolutely killer features: a speakerphone with great sound quality, and a jack for a high quality two-ear Plantronics headset. The former is necessary for the way I tend to work during boring phone calls (present company excepted of course). The latter is for my less than perfect hearing, which is the result of a misspent youth (no, not rock music, but firearms, motorcycles, and mainframe computer rooms). The Avaya 4610SW is a professional quality phone. It is not cheap, but its sound quality and headphone capabilities make it worth every single penny. It looks kinda cool too.

Note that this is a completely unsupported configuration: an Avaya 4610SW connected to a WiFi network serviced by a SIP proxy and registrar running inside Asterisk. Once upon a time when I was doing product development for Avaya, someone from their CTO organization actually called me up and warned me not to ever admit that I was using Asterisk, nor that I was using an Avaya SIP phone with it, nor to ever help anyone ever set up such a configuration, nor to even admit that such a configuration was even possible. So here is my disclaimer: if you have an Avaya SIP phone and are thinking of running it with Asterisk, quit reading this article right now. You have been warned. And if you are an Avaya executive, please tell your CTO people to pull their heads out of their asses.

This configuration worked pretty well for a couple of years. In the past few months though I have been having a disconcerting problem of very occasionally having phone calls from the PSTN to my SIP desk phone drop. Picking up an analog phone connected to the same phone line revealed that the other party was still on the line, and in fact had no idea that the call had been dropped from my point of view. Several hours here and there spent testing, perusing Asterisk logs, running Ethereal traces, and just about anything I could think of, turned up nothing.

Then one day while cruising the web, I started to notice articles complaining about how when the density of WiFi networks in a neighborhood reaches a certain point, connections start to drop unexpectedly. When I first brought WiFi into the Overclock household, there was exactly one other WiFi network visible in our neighborhood. Now there are at least a half dozen, and maybe more which, like my network, do not publish their SSIDs. I began to realize that not only had I seen SIP calls drop occasionally, but once in a while I had problems just downloading web pages too, and had to tell my browser to stop and reload, finding that it worked fine the second time.

So began a quest to find another LAN solution, and after some research, HomePlug presented itself. HomePlug is an industry standard that allows you to run Ethernet over your household electrical wiring at rates somewhere around thirty megabits a second, depending on whose propaganda you believe. After reading a lot of reviews, I departed from my usual network vendor, LinkSys, and settled on NetGear.

Why the change? You may recall that in my article on the book Five Myths of Consumer Behavior, I mentioned how the authors preached simplicity and ease of use as the keys to rapid consumer adoption. The LinkSys manuals for their HomePlug gear had long, multi-step setup procedures. The reviews of the NetGear boxes had users saying "I just plugged it in and it worked!" Yeah, baby, that's what I'm talking about, give me some of that.

So I ordered three NetGear XE104NA "PowerLine" four-port HomePlug Ethernet bridges from what has become my favorite computer vendor, Amazon.com. One for the Asterisk server in the basement, one for the router in the family room, and one for the SIP phone in my office.

Installation was almost that simple. The Digital Aggregates network became a combination of 100BaseT, WiFi, and HomePlug. I could telnet from my WiFi laptop to the HomePlug server. I could ping from one of the WiFi servers to the SIP phone. And when I booted the SIP phone, it could download its firmware and configuration from the HomePlug server. But the damn phone would not register with the SIP registrar on the very same HomePlug server.

I'll spare you the drama of the Ethereal traces, the traceroutes, the pings, the firewall experiments that took up a couple of hours on a Saturday morning. None of them contributed in any way to the solution except to convince me that the problem was something a lot less obvious. And so it was. To me anyway. All solutions are obvious once you know the answer.

The 4610SW implements the 802.1Q standard, a mechanism that allows multiple bridged virtual LANs to share the same physical LAN while keeping their data packets segregated. This makes a lot of sense for corporate environments. The default value for 802.1Q on the 4610SW was "auto", which also makes a lot of sense. Unfortunately, it doesn't seem to work on my new network configuration. Even through the SIP phone and the Asterisk server are on the same HomePlug network, the Netgear bridges apparently make them appear to be on different VLANs. I disabled this feature on the phone, and five seconds later all was well.

That's all it took. Otherwise, it was just a matter of freeing up some wall outlets for the Netgear boxes, which are each about the size of a largish power brick, moving the Ethernet cables from the WiFi boxes to the Netgear boxes, and standing back. It was that simple. It is too soon to tell if this will solve my problem of dropped VOIP calls, but it has had an expected side benefit. The HomePlug network apparently has lowered the network latency to where the echo canceller in the 4610SW can handle it. Prior to this, on the WiFi network, I had some echo back of my own voice, probably from the hybrid analog interface to the PSTN, which, fortunately, only I could hear. That seems to be gone. Huzzah!

Time will tell.

Update

By way of an update, I've had several longish calls recently, in addition to my own test calls, using this new configuration and have yet to have a VOIP disconnect. This proves nothing, but it is at least encouraging. On the flip side, if WiFi does have a problem when densities get high, then unfortunately as a technology it has no where to go by down.

Update

So here's the latest on Asterisk, HomePlug, and my Avaya SIP phone. As of yesterday, I had a couple of calls drop in the middle. This is the first time this has happened since switching from WiFi to HomePlug to connect my Asterisk server to my Avaya 4620SW. Investigation continues.

Meanwhile, today I upgraded to Asterisk 1.2.16 (I'm embarressed to admit how old my prior version was, but I installed it almost exactly one year ago today), and also to the Zaptel 1.2.15 drivers. I did this primary to take advantage of Digium's proprietary host-based High-Performance Echo Canceller (HPEC) which they offered free to folks with a Digium card still under warranty (barely, in my case). So far initial testing is all good: with no echo cancellation, the hybrid echo was terrible. With the open source echo canceller, it was barely bearable. With HPEC, it's gone completely. (I try not to think how many cycles this is consuming, but it's ONE phone and a server dedicated to Asterisk and some internal Apache use.)

I initially tried this with Zaptel 1.2.14 (the latest rev at that time) and could not get the zaphpec_enable tool to recognize that I had HPEC enabled ("it appears that this driver was not built with HPEC enabled", which was patently untrue as any number of tools verified). But Zaptel 1.2.15 seems to work fine.

Update

I used a LinkSys 100Mb/s WiFi bridge, a little box that converts from wired Ethernet to WiFi. The phone has a power-over-ethernet brick that sits between it and the WiFi bridge (now between it and the HomePlug adaptor). I converted both the Avaya 4610SW SIP phone and my Asterisk server from WiFi to HomePlug.

I'm still having problems with very occasional disconnects while using HomePlug. I recently upgraded the Avaya phone to the latest firmware (from 46xxSIP_101005 to 46xxSIP_032207) and am waiting to see if that helps.

I've never been able to reproduce this disconnect problem at will, so testing this is a pain. I now wonder if it's something like the old problem where the message waiting update from Asterisk would crash the phone. This was a known problem with the SIP firmware caused, according to Avaya, by a malformed or otherwise invalid SIP message from Asterisk. You can disable the feature in Asterisk by commenting out the "mailbox=" line in your sip.conf file.

I recently installed the proprietary host-based echo cancellation software from Digium. I recommend it if you're using Asterisk and a Digium analog card to connect to the PSTN. It definitely made a big difference in sound quality.

Update

Just a couple of days ago I had another rash of call droppings. The original call was coming in from the PSTN analog to Asterisk, then SIP via HomePlug to my Avaya 4610. It dropped twice on the same caller. (Sorry about that.) So it's back to debugging.

Update

I'm admitting defeat. After a couple of years of trying to get an Avaya 4610SW SIP phone to work reliably with Asterisk, I've just ordered a LinkSys (Cisco) SPA942 to replace it. My two must-have applications for a desk phone are [1] a decent speaker phone, and [2] a Plantronics headphone jack. We'll see how the SPA942 fairs in this regard. Just when I think I've got the 4610 working reliably, it drops a call (something my two soft phones I've used have never done). I'm calling it quits.

Update

I installed the LinkSys (Cisco) SPA942 on my Asterisk server this AM. The hardest part was sorting through the admin web pages (the phone as a built-in web server, like all the LinkSys gear I've used) to find where the IP address of the SIP proxy (my Asterisk server) was administered. Other than that, everything else was just gravy. I did some machinations in my Asterisk dial plan to attach a second phone number to the phone, just because I could. The voice quality of the handset, speakerphone, and headset, all seem quite acceptable. I had to buy an inexpensive 2.5mm adaptor for my Plantronics headset, but other than that it all pretty much worked right out of the box.

Update

My ancient Digium line card finally gave up the ghost. I had to install the latest hardware from Digium which, happily, is available with a hardware echo canceler. I also had to upgrade to Asterisk 1.8.3.2, which was a moderate bit of effort. So far, so good.

18 comments:

Chip Overclock said...

I have since posted a follow-up article with photographs of some of the equipment I described here. Check out "The Secret Life of Chip Overclock Revealed At Last!".

Chip Overclock said...

By way of an update, I've had several longish calls recently, in addition to my own test calls, using this new configuration and have yet to have a VOIP disconnect. This proves nothing, but it is at least encouraging. On the flip side, if WiFi does have a problem when densities get high, then unfortunately as a technology it has no where to go by down.

Jim Hickstein said...

Any thoughts on the Avaya 9620/9630 for (my intended) use with Asterisk? A local reseller has agreed to let me come over and touch one (there's far too much plastic crap and I will never again buy a telephone sight-unseen), but the hear-no-Asterisk/see-no-Asterisk thing may extend even to this reseller.

Why shouldn't I pick up a used 4610 on eBay instead?

Chip Overclock said...

I've only seen the new Avaya 96xx phones "at a distance": I've handled one but never even made a call with it on an Avaya CM system much less Asterisk.

Depending on the price differentials, used 46xx models capable of SIP might be a pretty good deal, particularly since they're known to work with Asterisk. I am frankly impressed with the build quality of the 46xx phones, less so with the 96xx that I briefly handled.

If you get one of the new ones, and get it working with Asterisk, for sure it'd be great if you'd post a link to your experience here!

Thanks!

Chip Overclock said...

So here's the latest on Asterisk, HomePlug, and my Avaya SIP phone. As of yesterday, I had a couple of calls drop in the middle. This is the first time this has happened since switching from WiFi to HomePlug to connect my Asterisk server to my Avaya 4620SW. Investigation continues.

Meanwhile, today I upgraded to Asterisk 1.2.16 (I'm embarressed to admit how old my prior version was, but I installed it almost exactly one year ago today), and also to the Zaptel 1.2.15 drivers. I did this primary to take advantage of Digium's proprietary host-based High-Performance Echo Canceller (HPEC) which they offered free to folks with a Digium card still under warranty (barely, in my case). So far initial testing is all good: with no echo cancellation, the hybrid echo was terrible. With the open source echo canceller, it was barely bearable. With HPEC, it's gone completely. (I try not to think how many cycles this is consuming, but it's ONE phone and a server dedicated to Asterisk and some internal Apache use.)

I initially tried this with Zaptel 1.2.14 (the latest rev at that time) and could not get the zaphpec_enable tool to recognize that I had HPEC enabled ("it appears that this driver was not built with HPEC enabled", which was patently untrue as any number of tools verified). But Zaptel 1.2.15 seems to work fine.

Josh said...

Does the phone have built in wifi or were you using something else. If so what? I'm trying to use IP Phones in my house with out rewiring. I was thinking of using the HomePlug adapters but wanted to see how you did it with wireless.

Chip Overclock said...

I used a LinkSys 100Mb/s WiFi bridge, a little box that converts from wired Ethernet to WiFi. The phone has a power-over-ethernet brick that sits between it and the WiFi bridge (now between it and the HomePlug adaptor). I converted both the Avaya 4610SW SIP phone and my Asterisk server from WiFi to HomePlug.

I'm still having problems with very occasional disconnects while using HomePlug. I recently upgraded the Avaya phone to the latest firmware (from 46xxSIP_101005 to 46xxSIP_032207) and am waiting to see if that helps.

I've never been able to reproduce this disconnect problem at will, so testing this is a pain. I now wonder if it's something like the old problem where the message waiting update from Asterisk would crash the phone. This was a known problem with the SIP firmware caused, according to Avaya, by a malformed or otherwise invalid SIP message from Asterisk. You can disable the feature in Asterisk by commenting out the "mailbox=" line in your sip.conf file.

I recently installed the proprietary host-based echo cancellation software from Digium. I recommend it if you're using Asterisk and a Digium analog card to connect to the PSTN. It definitely made a big difference in sound quality.

Jim Hickstein said...

I finally bought a 4610SW IP, partly on your recommendation, but frankly mostly because I was able to touch one, and it does suck less.

Now I'm in firmware hell. You used to work Avaya, I see, which might explain why this was apparently effortless for you. Where's the reset-NVRAM switch inside? I've got it looking for VLAN ID=0 now, and I'll have to turn the lab upside-down to make it happy.

Anonymous said...

Just hold down the # key and the phone will reset.

Chip Overclock said...

I apologize for not responding to some of these comments earlier, but I just managed to stumble across them and discover that my spam filter has started removing all of the comment email from Blogger. Oops. Hopefully this is now rectified.

Yes, I spent a decade in the Lucent/Avaya camp. Although I never worked on Avaya phone firmware, nor even used a VOIP phone in my day-to-day work, I did have access to developers that did. But everything "internal" I know about the 4610 came from the Avaya "4610SW IP Telephone Release 2.2 User Guide" which I keep near my 4610 on my desk. This guide tells me everthing I've needed to know so far about magic keypresses to administer and reset the phone.

Good luck and sorry again about the delay.

Chip Overclock said...

Just a couple of days ago I had another rash of call droppings. The original call was coming in from the PSTN analog to Asterisk, then SIP via HomePlug to my Avaya 4610. It dropped twice on the same caller. (Sorry about that.) So it's back to debugging.

Star said...

Thot i'd post my experiences with the AVAYA SIP phones.

- The current version of the firmware for the 46xx phones still doesn't seem to support the message waiting indicator (MWI) light. And as a side-effect of pending voice mail, these phones seem to think they are not registered after a seeming random period of time, refusing to allow outgoing calls. I had to disable voice mail on all extensions with AVAYA phones. Wasn't too bad, I could live without the MWI.

- The 1.xx version of SIP firmware for the 9620/9630/9640 series of phones would register fine on Asterisk, allow outgoing calls but refuse to accept incoming calls.

- AVAYA have recently release version 2 of the SIP firmware for the 96xx phones. Thought I'd give it a try the other day. I was pleasantly surprised to find that it worked perfectly, including the MWI!! Excellent speaker-phone, and great voice quality.

I've been testing it for about 4 days now and am totally happy with the results.

- StarProwler

Chip Overclock said...

I'm pleased to hear Avaya is taking Asterisk seriously enough that their phones aren't crippled somehow when used on it.

Back when I worked at Avaya I raised a small stink about the fact that my Avaya 4610 SIP phone (for which I paid retail from my own pocket) had firmware that crashed when it received certain SIP messages from my Asterisk system. Messages that another developer assured me were invalid or otherwise malformed. No problem in my book with rejecting such messages but the phone shouldn't crash.

A few days later I got a call from an Avaya marketing executive in New Jersey (where the company is headquartered) warning me not to discuss Asterisk or the fact that Avaya phones could be used it. I appreciated his call because he really was doing me a favor (in a number of respects).

I'm sure there are folks at certain other companies that warn their employees not to write software that works under Linux, or to discuss the fact that it might be portable to Linux. Yeah, that'll really put a dent in the popularity of open source solutions.

(I left Avaya a short while after that experience.)

I'll check out the 96xx -- thanks so much for your report.

Star said...

Now that you mention your experience, I recall the local AVAYA people insisting that their phones will not work at all on Asterisk, when I first approached them for the MWI issue on the 46xx phones.

BTW, an AVAYA insider let me know a few weeks ago that they actually discuss Asterisk during their induction training sessions.

Chip Overclock said...

Maybe I'm just in a more charitable mood than normal, but as a former Avayan developer, I totally get why Avaya might have denied any interoperability between Avaya SIP phones and Asterisk. Supporting known end stations and known PBXen on the many different networks found in the field is hard enough. Supporting known endstations on a customer's roll-their-own PBX could be downright painful. But to deny the market force that is Asterisk seems nuts to me. Like a friend of mine who is still at Avaya said, call processing is now a commodity.

Chip Overclock said...

I'm admitting defeat. After a couple of years of trying to get an Avaya 4610SW SIP phone to work reliably with Asterisk, I've just ordered a LinkSys (Cisco) SPA942 to replace it. My two must-have applications for a desk phone are [1] a decent speaker phone, and [2] a Plantronics headphone jack. We'll see how the SPA942 fairs in this regard. Just when I think I've got the 4610 working reliably, it drops a call (something my two soft phones I've used have never done). I'm calling it quits.

Chip Overclock said...

I installed the LinkSys (Cisco) SPA942 on my Asterisk server this AM. The hardest part was sorting through the admin web pages (the phone as a built-in web server, like all the LinkSys gear I've used) to find where the IP address of the SIP proxy (my Asterisk server) was administered. Other than that, everything else was just gravy. I did some machinations in my Asterisk dial plan to attach a second phone number to the phone, just because I could. The voice quality of the handset, speakerphone, and headset, all seem quite acceptable. I had to buy an inexpensive 2.5mm adaptor for my Plantronics headset, but other than that it all pretty much worked right out of the box.

Anonymous said...

This was an interesting read. I am currently playing around with PBX In A Flash (PIAF), and have it working with a Cisco 7912G, an Avaya 4610SW, and a softphone (X-Lite). I continue to have the problem with the Avaya phone unregistering after awhile. I tried deleting the mailbox= command from its' configuration in the sip_additional_conf file, but the problem continues. Well, I have access to more Cisco phones, so the Avaya will go back to the pile.

Regarding the Avaya 96xx versus 46xx phones...if you use DHCP to send the phone its' call manager IP and TFTP server addresses, you should know that the 46xx phones use DHCP Option 176 for this purpose, but the new 96xx phones use DHCP Option 242. Apparently DHCP options were redefined to expand the range, so Option 242 will be the future for Avaya. Fortunately, it's very easy to configure Option 242, as it will contain exactly the same information that is contained in Option 176. In fact, for a mixed environment containing both 46xx and 96xx phones, just put both Option 176 and Option 242 into the DHCP server and the phone will find the Option it needs.

-rb