I woke up today and realised that as of summer 2016, I’ll have been doing VoIP & SIP for ten years. It was about ten years ago that I first connected an Asterisk 1.2 server to my home landline via an FXO card, registered a Snom 190 to it and got down to business with my copy of O’Reilly’s Asterisk: The Future of Telephony.
I know that 2006 isn’t exactly old-school by the standards of Asterisk early adopters, and especially not by those of butt set-wearing colleagues who think E&M Wink is too newfangled and won’t drive cars made after 1975 due to all those new anti-pollution requirements. If you did Dialogic IVR programming on NT boxes in the 1990s, or rolled out AT&T WATS lines in the 1970s, you won’t be impressed. However, this field encompasses a third of my life span and nearly my entire adult life and career. So, from my perspective, it’s been a minute.
I was never an early adopter. At my small ISP job of the time, several people were tinkering with Asterisk from 2003 onward with varying degrees of commercial implementation success. I was introduced to the notion in 2005, and at first responded with relative indifference. In fact, if you had told me I was going to be doing anything with voice or telephones before 2006, I would have looked at you like you were crazy.
As the modem boom rapidly wound down and our ISP floundered desperately in search of a new business model, the connectivity side became a strictly Layer 3 proposition, a kind of sales channel for the ILEC. They owned the network and transport we sold. Do you remember the brief window in which independent ADSL providers were allowed to exist in the US, in the early-mid 2000s? That was us.
Nevertheless, I had a burgeoning interest in the technical and business side of networking that grew out of my deepening responsibilities at work. As I aggressively pursued that curiosity, I plumbed down the protocol stack and became intensely interested in how all these circuits worked at the physical level (which was out of our reach as we did not own the network). T1: A Survival Guide was a very enlightening tome that I literally could not put down. Anyway, as one slices through the OSI burrito, the subject matter rapidly converges with the history of voice and the Bell system, through the pathway of digital trunking facilities and digital loop carriers, multiplexing, etc. Now I was interested in voice. The whole Asterisk and open-source IP PBX phenomenon was not central to my interests, but critical to pursuing them; at the right time and place in this developmental process, it pushed down telephony from the mystical realm of the proprietary into commodity hardware and closer to the application level, which meant I could actually afford to get my hands dirty with it.
Speaking of commodity hardware, my foray into voice was, as my forays into many things as a twenty year-old, chaotic and irresponsible. I took a PBX PC with a Digium quad FXS/FXO card home from work and impertinently formatted it into a fresh Debian install. Why not? It seemed to be sitting downstairs doing nothing for a few months, give or take. Spare hardware, right? As it turns out, its disk contained–well, had contained–the only copy of a home-spun Asterisk PBX CD distribution project into which my immediate supervisor had invested many months of work. There were no backups. Oops. I still feel terrible about it. Granted, there were no change control processes, asset tagging, version control, or project management systems, but common sense might have invited one to ask first.
Anyway, I returned the hardware, built my own Asterisk server, got my own FXO card, and plugged into my land line. Those who frequented the 24/7 coffee shop Hot Corner back in Athens, GA in those days (summer 2006) might remember me sitting there with a telephone plugged into my laptop, which I was using to bridge it onto the WiFi network. Making calls out of my home landline from the coffee shop! How cool was that!
Here, my career took an interesting turn and perhaps suffered from a bit of confusion. I’ve never been a so-called visionary for innovation, but my twenty year-old self was especially impaired as a barometer of industry trends. I didn’t really realise that the classical TDM stuff was on its way out. I saw Asterisk as a pedestrian and entry-level window into the Really Serious Stuff, a stepping stone to the exalted heights of multiplexers, DACSs, and big-iron switches. Instead of stopping for a moment to wonder why the O’Reilly book was called “Asterisk: The Future of Telephony”, I was suddenly enamoured with things like ISDN, SS7, and SONET. I wanted to get into hardcore CLEC operations–interconnection, switch translations, the works.
So, instead of getting deeper into VoIP per se or taking an overly active interest in the new >= Layer 5 capabilities Asterisk opened up, I spent a few years investing in that career direction, rather the opposite of where things were going. I was a bit of an enigma; a guy who got into VoIP and discovered his love for the CSU/DSU.
I kept up my Asterisk skills and used it frequently, and I also started to get into OpenSER around this time. However, it wasn’t really what I cared about the most. Real excitement came at moments like when a colleague of mine set me loose on his Cisco AS5300–the first time I encountered the voice side of IOS–to get a few inbound PRIs working and routed to a SIP server. That was living!
Besides the fact that my core areas of interest were not so much hot topics in 2006-07 as they were moribund, there were a few other things I didn’t understand in my very early twenties:
- TDM & physical-layer stuff didn’t pay, for the most part.
Back at my ISP job, I thought the BellSouth CWAs who came in to reprogram our building mux in TL1 via the craft port were highly-paid, highly-skilled, in-demand demigods, true engineering luminaries, practitioners of the dark and recondite arts. Anyone can do good old IT folk traditions; few can program a fibre mux!
I didn’t realise they’re actually considered kind of blue-collar, not especially well-paid, and that their work consists largely of fastidiously adhering to procedures that are labouriously articulated in three-ring binders.
- Engineering vs. operations.
Diving into the voice service provider and CLEC world from the angle that I did, I ended up tracked for an “operations” career for a while without realising it. It took me some time to learn that this was a thing, and moreover, that I, as a developer historically, more belong on the engineering side. It dawned on me when I ended up a senior NOC tech at a mid-size company (a few hundred staff) and I noticed that only “engineers” actually get to implement things or change code; “operations” people just do things like server administration and monitoring.
Coming from a small-company background (our ISP was ~5 people), I certainly didn’t know that Corporate America makes a distinction between “engineering” and “operations”. As you might imagine, in an environment of half a dozen or so, everyone did everything, and owned their responsibilities end-to-end. In that sense, it was a fairly seamless transition from how I grew up doing things at a hobbyist level to a professional environment. I was already accustomed to having to know all aspects of what I was doing.
That’s why I was so confused when the word “DevOps” first popped up. What the hell is “DevOps”? It sounds like just “how we normally work”, right? Are there really developers who think their work ends when the IDE closes or whatever? All of us had to both write the code and cultivate the skill set to deploy it in the real world, secure it, etc. Are there really sysadmin that throw anything involving for loops and if statements or SQL over to the Programming Department, because that’s not really “operational”? Our sysadmin back at the ISP may not have been professional software engineers, but if they couldn’t write their own utilities and script glue, or execute the odd inner join, they couldn’t do their job for more than five minutes.
So, nobody was more surprised to learn than I that I was apparently in something called “operations”, where we had to get three “engineers” on a conference call to fix that one Bash script called from the one cron job. Shouldn’t I just do that for them? I thought I was in the get-things-done department? Bumping up against that division of labour caused me some problems both as an employee and as an employer.
- The misery of proprietary platforms.
I grew up breathing open-source like air, so I kind of took the culture, values and skill set that it brings for granted. I got a rude awakening and a newfound appreciation for it when I ended up in a role where my job was to be the Broadsoft specialist, deal with Metaswitch and Acme Packet, etc.Suddenly, no O’Reilly books, no conferences, no mailing lists, no forums, and, worst of all, no Google and no source code. Operating these platforms required an entirely different skill set; strong-arming the vendor and playing political games.
What little documentation existed was not conceptual in nature; it read like a reference manual–fine if you already know what you want to find out and how to express that in the vendor’s proprietary vernacular, but completely useless if you don’t know what you should want or what it’s called. Worse yet, it’s very clear that a lot of these platforms aren’t seriously designed for their users to run them, but for large operators who rely on the vendor’s consulting to architect, build and maintain their networks. That gets really fun if you don’t have official vendor support, but that’s another story.
It turns out that bashing my head against the CLI of various black boxes to see what the autocomplete turns up and fruitlessly pleading with vendor support people for answers is not my cup of tea. I was really bad at that job and didn’t get a lot done. I know for a fact that there’s a certain kind of personality out there that is very good at amassing this kind of knowledge by more oblique means, and those people are indispensable in the enterprise world. However, I’m not one of them. If you give me a box under whose hood I cannot easily look and which is not augmented by a user community or freely available literature, I’m pretty useless.
More importantly, I realised that it’s a dead end; should you choose to become (by way of rather expensive training and certification) a specialist in one of these platforms, your employability and fate will rise and fall strictly with the commercial fate of that platform. Knowing how to Combobulate the ANI Presentation Screen List on the Translations Profile of a Business Group is valuable, but it’s not Knowledge with a capital K, nor even a fundamental skill set per se–at least, not as I was accustomed to thinking of it. It’s just the privilege of carrying some part of the reference manual in your head. I wasn’t going to bracket myself into that. I’d rather be the guy who doesn’t know the answer offhand, but can Google it in 15 seconds, and has the strong fundamentals to be able to (a) find the answer in the results, (b) intuit that what he found really is the answer and (c) apply the information.
Anyway, the rest is largely history. In early 2008, I started a certain consulting practice and rapidly specialised in SIP service delivery platform development that heavily emphasises OpenSER / Kamailio. It’s been a slow, painful (ahem, “organic”) growth path that has taken far too long, but we’re well on our way to the product life, which is a generally positive development.
In hindsight, I don’t feel set back by the circuitous path I took to get here. Experience with development as well as infrastructure & operations goes far. Getting invested in the hard PSTN side of voice instead of viewing the world through the prism of open-source VoIP systems solely has empowered me to provide added value to my customers by being able to understand the migration path from “legacy” infrastructure intimately and to consult on intricate matters of PSTN-side economics. A lot of VoIP folks don’t really know what’s going on the other side of the fence; they just send their calls to the ITSP and provision DIDs without much of an inkling for how it works. One could argue that this sort of esoteric knowledge is obviously more useful so long as the PSTN is around and not as important in a world where voice is just another IP application, but, so far, even in 2016, the death of the PSTN has been greatly exaggerated.
Moreover, while there’s certainly a need to revolutionise paradigms and whatnot, being more deeply conversant with the folk traditions of telephony both allows one to play a more constructive role in bridging the gap and, with a little luck, might allow one to avoid some of the errors that those who don’t know history are doomed to repeat in any scenario, circuit-switched or packet-switched. There’s a reason it took decades to engineer voice to the standard of reliability POTS users have come to expect. I strongly believe there are some lessons in that even for the WebRTC-in-The-Cloud enthusiasts out there.
It’s an interesting industry and, without a doubt, a very small circle of key actors. From a business perspective, that has its upsides and its downsides (small market). I’ve been fortunate to make many good friends and colleagues along the way, and the support has been both indispensable and deeply appreciated.
That’s the best thing about being in a small industry; like Cheers, it’s nice to go where everybody [well, some people] knows your name. You are too numerous to list, but I owe a debt of gratitude to all of my friends, colleagues, customers, hiring managers and bosses for helping me to learn where I fit best.