Ten years of VoIP & SIP

I woke up today and realised that as of summer 2016, I’ll have been doing VoIP & SIP for ten years. It was about ten years ago that I first connected an Asterisk 1.2 server to my home landline via an FXO card, registered a Snom 190 to it and got down to business with my copy of O’Reilly’s Asterisk: The Future of Telephony.

snom190

I know that 2006 isn’t exactly old-school by the standards of Asterisk early adopters, and especially not by those of butt set-wearing colleagues who think E&M Wink is too newfangled  and won’t drive cars made after 1975 due to all those new anti-pollution requirements. If you did Dialogic IVR programming on NT boxes in the 1990s, or rolled out AT&T WATS lines in the 1970s, you won’t be impressed. However, this field encompasses a third of my life span and nearly my entire adult life and career. So, from my perspective, it’s been a minute.

I was never an early adopter. At my small ISP job of the time, several people were tinkering with Asterisk from 2003 onward with varying degrees of commercial implementation success. I was introduced to the notion in 2005, and at first responded with relative indifference. In fact, if you had told me I was going to be doing anything with voice or telephones before 2006, I would have looked at you like you were crazy.

As the modem boom rapidly wound down and our ISP floundered desperately in search of a new business model, the connectivity side became a strictly Layer 3 proposition, a kind of sales channel for the ILEC. They owned the network and transport we sold. Do you remember the brief window in which independent ADSL providers were allowed to exist in the US, in the early-mid 2000s? That was us.

Nevertheless, I had a burgeoning interest in the technical and business side of networking that grew out of my deepening responsibilities at work. As I aggressively pursued that curiosity, I plumbed down the protocol stack and became intensely interested in how all these circuits worked at the physical level (which was out of our reach as we did not own the network). T1: A Survival Guide was a very enlightening tome that I literally could not put down. Anyway, as one slices through the OSI burrito, the subject matter rapidly converges with the history of voice and the Bell system, through the pathway of digital trunking facilities and digital loop carriers, multiplexing, etc. Now I was interested in voice. The whole Asterisk and open-source IP PBX phenomenon was not central to my interests, but critical to pursuing them; at the right time and place in this developmental process, it pushed down telephony from the mystical realm of the proprietary into commodity hardware and closer to the application level, which meant I could actually afford to get my hands dirty with it.

Speaking of commodity hardware, my foray into voice was, as my forays into many things as a twenty year-old, chaotic and irresponsible. I took a PBX PC with a Digium quad FXS/FXO card home from work and impertinently formatted it into a fresh Debian install. Why not? It seemed to be sitting downstairs doing nothing for a few months, give or take. Spare hardware, right? As it turns out, its disk contained–well, had contained–the only copy of a home-spun Asterisk PBX CD distribution project into which my immediate supervisor had invested many months of work. There were no backups. Oops. I still feel terrible about it. Granted, there were no change control processes, asset tagging, version control, or project management systems, but common sense might have invited one to ask first.

Anyway, I returned the hardware, built my own Asterisk server, got my own FXO card, and plugged into my land line. Those who frequented the 24/7 coffee shop Hot Corner back in Athens, GA in those days (summer 2006) might remember me sitting there with a telephone plugged into my laptop, which I was using to bridge it onto the WiFi network. Making calls out of my home landline from the coffee shop! How cool was that!

Alex_BR_DC1

In a data centre in 2007. The smile is definitely posed; I do not typically enjoy my time in data centres in this much.

Here, my career took an interesting turn and perhaps suffered from a bit of confusion. I’ve never been a so-called visionary for innovation, but my twenty year-old self was especially impaired as a barometer of industry trends. I didn’t really realise that the classical TDM stuff was on its way out. I saw Asterisk as a pedestrian and entry-level window into the Really Serious Stuff, a stepping stone to the exalted heights of multiplexers, DACSs, and big-iron switches. Instead of stopping for a moment to wonder why the O’Reilly book was called “Asterisk: The Future of Telephony”, I was suddenly enamoured with things like ISDN, SS7, and SONET. I wanted to get into hardcore CLEC operations–interconnection, switch translations, the works.

So, instead of getting deeper into VoIP per se or taking an overly active interest in the new >= Layer 5 capabilities Asterisk opened up, I spent a few years investing in that career direction, rather the opposite of where things were going. I was a bit of an enigma; a guy who got into VoIP and discovered his love for the CSU/DSU.

I kept up my Asterisk skills and used it frequently, and I also started to get into OpenSER around this time. However, it wasn’t really what I cared about the most. Real excitement came at moments like when a colleague of mine set me loose on his Cisco AS5300–the first time I encountered the voice side of IOS–to get a few inbound PRIs working and routed to a SIP server. That was living!

Besides the fact that my core areas of interest were not so much hot topics in 2006-07 as they were moribund, there were a few other things I didn’t understand in my very early twenties:

  • TDM & physical-layer stuff didn’t pay, for the most part.

    Back at my ISP job, I thought the BellSouth CWAs who came in to reprogram our building mux in TL1 via the craft port were highly-paid, highly-skilled, in-demand demigods, true engineering luminaries, practitioners of the dark and recondite arts. Anyone can do good old IT folk traditions; few can program a fibre mux!

    I didn’t realise they’re actually considered kind of blue-collar, not especially well-paid, and that their work consists largely of fastidiously adhering to procedures that are labouriously articulated in three-ring binders.

  • Engineering vs. operations.

    Diving into the voice service provider and CLEC world from the angle that I did, I ended up tracked for an “operations” career for a while without realising it. It took me some time to learn that this was a thing, and moreover, that I, as a developer historically, more belong on the engineering side. It dawned on me when I ended up a senior NOC tech at a mid-size company (a few hundred staff) and I noticed that only “engineers” actually get to implement things or change code; “operations” people just do things like server administration and monitoring.

    Coming from a small-company background (our ISP was ~5 people), I certainly didn’t know that Corporate America makes a distinction between “engineering” and “operations”. As you might imagine, in an environment of half a dozen or so, everyone did everything, and owned their responsibilities end-to-end. In that sense, it was a fairly seamless transition from how I grew up doing things at a hobbyist level to a professional environment. I was already accustomed to having to know all aspects of what I was doing.

    That’s why I was so confused when the word “DevOps” first popped up. What the hell is “DevOps”? It sounds like just “how we normally work”, right? Are there really developers who think their work ends when the IDE closes or whatever? All of us had to both write the code and cultivate the skill set to deploy it in the real world, secure it, etc. Are there really sysadmin that throw anything involving for loops and if statements or SQL over to the Programming Department, because that’s not really “operational”? Our sysadmin back at the ISP may not have been professional software engineers, but if they couldn’t write their own utilities and script glue, or execute the odd inner join, they couldn’t do their job for more than five minutes.

    So, nobody was more surprised to learn than I that I was apparently in something called “operations”, where we had to get three “engineers” on a conference call to fix that one Bash script called from the one cron job. Shouldn’t I just do that for them? I thought I was in the get-things-done department? Bumping up against that division of labour caused me some problems both as an employee and as an employer.

  • The misery of proprietary platforms.

    I grew up breathing open-source like air, so I kind of took the culture, values and skill set that it brings for granted. I got a rude awakening and a newfound appreciation for it when I ended up in a role where my job was to be the Broadsoft specialist, deal with Metaswitch and Acme Packet, etc.Suddenly, no O’Reilly books, no conferences, no mailing lists, no forums, and, worst of all, no Google and no source code. Operating these platforms required an entirely different skill set; strong-arming the vendor and playing political games.

    What little documentation existed was not conceptual in nature; it read like a reference manual–fine if you already know what you want to find out and how to express that in the vendor’s proprietary vernacular, but completely useless if you don’t know what you should want or what it’s called. Worse yet, it’s very clear that a lot of these platforms aren’t seriously designed for their users to run them, but for large operators who rely on the vendor’s consulting to architect, build and maintain their networks. That gets really fun if you don’t have official vendor support, but that’s another story.

    It turns out that bashing my head against the CLI of various black boxes to see what the autocomplete turns up and fruitlessly pleading with vendor support people for answers is not my cup of tea. I was really bad at that job and didn’t get a lot done. I know for a fact that there’s a certain kind of personality out there that is very good at amassing this kind of knowledge by more oblique means, and those people are indispensable in the enterprise world. However, I’m not one of them. If you give me a box under whose hood I cannot easily look and which is not augmented by a user community or freely available literature, I’m pretty useless.

    More importantly, I realised that it’s a dead end; should you choose to become (by way of rather expensive training and certification) a specialist in one of these platforms, your employability and fate will rise and fall strictly with the commercial fate of that platform. Knowing how to Combobulate the ANI Presentation Screen List on the Translations Profile of a Business Group is valuable, but it’s not Knowledge with a capital K, nor even a fundamental skill set per se–at least, not as I was accustomed to thinking of it. It’s just the privilege of carrying some part of the reference manual in your head. I wasn’t going to bracket myself into that. I’d rather be the guy who doesn’t know the answer offhand, but can Google it in 15 seconds, and has the strong fundamentals to be able to (a) find the answer in the results, (b) intuit that what he found really is the answer and (c) apply the information.

Anyway, the rest is largely history. In early 2008, I started a certain consulting practice and rapidly specialised in SIP service delivery platform development that heavily emphasises OpenSER / Kamailio. It’s been a slow, painful (ahem, “organic”) growth path that has taken far too long, but we’re well on our way to the product life, which is a generally positive development.

In hindsight, I don’t feel set back by the circuitous path I took to get here. Experience with development as well as infrastructure & operations goes far. Getting invested in the hard PSTN side of voice instead of viewing the world through the prism of open-source VoIP systems solely has empowered me to provide added value to my customers by being able to understand the migration path from “legacy” infrastructure intimately and to consult on intricate matters of PSTN-side economics. A lot of VoIP folks don’t really know what’s going on the other side of the fence; they just send their calls to the ITSP and provision DIDs without much of an inkling for how it works. One could argue that this sort of esoteric knowledge is obviously more useful so long as the PSTN is around and not as important in a world where voice is just another IP application, but, so far, even in 2016, the death of the PSTN has been greatly exaggerated.

Moreover, while there’s certainly a need to revolutionise paradigms and whatnot, being more deeply conversant with the folk traditions of telephony both allows one to play a more constructive role in bridging the gap and, with a little luck, might allow one to avoid some of the errors that those who don’t know history are doomed to repeat in any scenario, circuit-switched or packet-switched. There’s a reason it took decades to engineer voice to the standard of reliability POTS users have come to expect. I strongly believe there are some lessons in that even for the WebRTC-in-The-Cloud enthusiasts out there.

It’s an interesting industry and, without a doubt, a very small circle of key actors. From a business perspective, that has its upsides and its downsides (small market). I’ve been fortunate to make many good friends and colleagues along the way, and the support has been both indispensable and deeply appreciated.

That’s the best thing about being in a small industry; like Cheers, it’s nice to go where everybody [well, some people] knows your name. You are too numerous to list, but I owe a debt of gratitude to all of my friends, colleagues, customers, hiring managers and bosses for helping me to learn where I fit best.


From our friends in North Carolina

Ringfree Communications, a leading VoIP operator and provider of hosted PBX and SIP trunking services in the Asheville, NC area, sent me this inspiring homage.

likewisementalhealth


Kamailio as an SBC (Session Border Controller)

One of the most common enquiries we get is about using Kamailio as an SBC.  I don’t think it’s an exaggeration to say it’s our top FAQ, as a consulting organisation.

“SBC” is a term that has a lot of different meanings to different people in different contexts.  To answer this question, you need to:

  • Develop a proper understanding of “SBC”.
  • Determine whether you actually need an SBC, enlightened by a correct understanding of the concept.
  • Evaluate the extent to which Kamailio can play this role.

I think it’s safe to say, from my observations, that there are two primary understandings of the concept of an SBC: a pedestrian understanding, and a professional one, as used in an intra-industrial/domain-specific context.

  • Pedestrian definition: some kind of network edge proxy, administrative border and/or call transit element at the edge of a voice network, designed to face carriers and/or peer networks.  It is expected to provide routing functionality and some security and rate-limiting type capabilities, to shield the internal network elements from DDoS attacks and so on.

When most people come along and ask me, “Can you deploy Kamailio for us as an SBC?”, they are proceeding from the pedestrian definition, and haven’t had to look at the terminology critically.  It often turns out that they are just looking for an aggregation point to condense their variable IP endpoints to a single signaling endpoint, in order to interface with a carrier.  Many other customers are looking for LCR (least cost routing) gateway functionality, or some other type of internetwork routing intelligence.  They, too, some reason choose to put the “SBC” label on it, because that’s what industry marketing has taught them to do for anything that sits at the network edge.

That’s not what an SBC is, at least, not to professional VoIP topology architects and network implementors.

  • Professional definition: an appliance with a SIP B2BUA (back-to-back user agent) core, usually able to do some combination of policy-based access control, transcoding, topology concealment, call accounting (usually via RADIUS), QoS and call quality statistics, etc.

The most exhaustive list of SBC capabilities I’ve seen lately was compiled by Mark R. Lindsey in this post on the VoiceOps list, which I will quote here (with his prior permission):

  • Ability to gracefully single-source and distributed handle attacks from the Internet
  • Traffic from numerous VRFs (so that customer A’s 10.0.0.1 is not the same as customer B’s 10.0.0.1)
  • Hosted NAT Traversal
  • Transcoding media
  • Lawful Intercept hooks
  • Demultiplexed SIP trunks (so that each SIP “path” can come from a different IP address or port)
  • Constraints like Calls-per-second or Concurrent-calls (so legitimate customers don’t cause a failure)
  • Failover from one SBC to another
  • Distributed SBC models where one box handles signaling and many more handle media
  • 802.1q VLAN tagging support
  • Support for failover to BroadWorks-style registrar redundancy (where the mated registrar servers have distinct IP addresses)
  • SNMP management
  • Interworking SIP/UDP and SIP/TCP
  • Jumbo SIP (SIP datagrams over 1300 bytes, out of compliance with RFC 3261)
  • SIP/TLS and SRTP, and decryption thereof
  • Configurable media management (to release media when possible, steer it through the SBC when not)

Higher-end SBCs usually also incorporate at least for ASIC (application-specific integrated circuit)-assisted RTP relay and transcoding, in order to achieve higher port densities.

This is the sort of thing I, and others in my profession, hear when customers say “SBC”.  In most cases, that’s probably not what they mean, but it’s what we hear.  When I tell people all this, they usually say, “no, no, I don’t need all that”.  If you don’t need the bulk of the aforementioned capabilities, you don’t need an SBC, and you shouldn’t call it that.  Perhaps this meta draft can help with the classification of the network element itself.

Kamailio fails out of the gate on the B2BUA aspect, which seems to be a requirement of the SBC concept.  A B2BUA accepts one logical call leg A, and creates another, logically unrelated call leg B, and bridges signaling events between them in as transparent or opaque a manner as it desires.  A proxy receives logical call leg A, and pushes call leg A right back out, and by and large, that’s what proxies do–they proxy; garbage in, garbage out.

There are relatively few things that a proxy can modify in a SIP request or reply, formally speaking.  It can modify the Request URI and a few other message fields, and it can attach custom headers.  However, critically, it cannot rewrite or fundamentally regenerate the request or reply, as a B2BUA might.  This makes it a rather poor mediator for interoperability issues and security purposes–a key selling point of SBCs.

Kamailio has some hacks that allow it to go a bit beyond these limitations, in defiance of its standards-prescribed role as a proxy element, such as the UAC module’s ability to statefully rewrite the From and To header, and the topology hiding module‘s stateful encryption of topology-revealing details in Record-Route header parameters. However, these should be understood for what they are: hacks with limitations–limitations that a B2BUA does not have.

Kamailio can use rtpproxy to relay RTP on commodity hardware, which is mostly done for far-end NAT traversal (which Kamailio supports through the nathelper module) and topology concealment reasons (combined with topoh). This horizontally scales rather well, but is not exactly like the port-dense ASIC-driven RTP forwarding setup of a commercial SBC.  Also, there is no transcoding capability in rtpproxy.

With some effort, Kamailio can be set up to bridge signaling between multiple network interfaces, and rtpproxy can operate in a “bridging mode” which allows it do to the same.  However, implementing this is challenging, and does not provide the multi-realm flexibility of an SBC; it is, by its nature, a hard-coded setup, and if you’re expecting to maintain “realms” in half a dozen subnets, as with an SBC, you’re setting yourself up for a world of pain.  It is technically possible, but just because some things can be done does not mean they should, methodologically speaking, be done.

Kamailio can relay REGISTER messages upstream, and supports the Path extension.  Whether the upstream registrar supports it is another question; many do not.  It is possible to do some clever hacking involving encoding of the original Contact URI received from the registrant into the Contact header, but this is a dirty hack, not a true registrar front-end.

In sum, it is possible to deploy Kamailio in select subsets of an SBC role, but the probability is quite high that “SBC” is not really what is meant when the term is used.  Things I’ve seen classified as SBCs in the context of Kamailio project requisitions include:

  • Far-end NAT traversal gateways.
  • Load balancers.
  • Registrars.
  • Proxy-based security elements.
  • Traffic aggregation proxies.
  • Intelligent routing gateways for voice application silos.
  • Least Cost Routing gateways.
  • Class 4 carrier trunking interfaces.

None of these things, in and of themselves, an SBC make. In fact, most of these things are functions that an SBC would not directly perform, though it is likely to front-end network elements that do perform them.

On the other hand, trying to front-end a classic Class 5 type SIP subscriber softswitch, such as a Metaswitch or a Broadsoft assembly, ought to be classified as an anti-pattern. While it is possible, in principle, cajole a reluctant proxy into a mostly-functional role doing that, it is the wrong tool for the job, it will create as many management problems as the technical objectives it crudely achieves.  There are lots of threads on sr-users from hapless souls who got stuck in the quagmire of multiple Record-Route headers corresponding to different forwarding network interfaces, and conditional rewriting of SDP based on the originating and terminating subnet, and keeping it all straight by encoding it in some sort of dialog-persistent parameters.  From a best-practical point of view, you really don’t want to go there.

Bottom line: Kamailio is not an SBC.  Not at all.  It’s a proxy with some SIP server features inside. However, depending on what you’re actually looking for and whether you’re using the term correctly, that may not be a problem.