From the Trenches:
True Tales Pulled from OSI’s Customer Files
Customer Problem: Unexpected Late-Night Network Outage Due to Dead Cisco 6500 Ethernet Module
Solution: 10:30pm Red Eye from LAX to Philly, a 2-Hour Rental Car Drive, and a Datacenter Parking Lot Hand-Off of Two WS-X6716-10GE-3Cs
Who do you call when a server blade goes down at 8:00pm Eastern time Monday night and half the network running your customer’s VMWare applications is dark?
Monday evening, February 12th, 2018 | 5:38pm Pacific time
Darren Nevolo, OSI Hardware Senior Account Executive, and his wife Shani are preparing dinner for their three children. It’s been a relaxing weekend following months of turmoil. Just scant weeks ago their hometown not only endured California’s largest wildfire in recorded history but suffered a devastating body blow from a deadly flood triggered by monsoon-like rains. Unseasonably warm and sunny late winter weather have provided a welcomed respite for the young family.
Across the kitchen, Darren’s Samsung Galaxy Note 8 lights up: “What are the odds of you getting us a WS-X6716-10GE line card in a handful of hours? It’s for our partner Ironton. They are in a pinch and have no spare.”
The text is from Frank Clements, Director of Engineering at PenTelData, an ILEC serving Pennsylvania and New Jersey with over 9,000 miles of fiber optic cabling. The dark network is located in the data center of Ironton Global. A wholly owned subsidiary of Ironton Telephone Company, the firm operates a massive state-of-the-art fiber optic infrastructure and 60 Points of Presence throughout both PA and NJ. There’s a lot at stake; the 107-year old, family-owned CLEC has an unblemished track record of over a century of uninterrupted service to the residents of Lehigh Valley in eastern Pennsylvania.
Asking his wife to keep an eye on the marinara sauce simmering on the cooktop, Darren immediately dials PenTelData to delve into the problem more deeply. The issue is a blade server that went down 20 minutes ago. Intensive troubleshooting identified the culprit: an Ethernet module purchased only a few months ago from an OSI Hardware competitor had suddenly gone south.
With all eastern seaboard parts warehouses shuttered until dawn and no replacement WS-X6716-10GE-3C modules in sight, it promised to be a long night and a very rough morning ops meeting at PenTelData.
“Whaddya say, Darren? Can you guys help us out?”
Having travelled over 250,000 miles annually during his 12-year career at OSI, Nevolo knew PenTelData’s infrastructure well. He’d been working with the firm for several years, providing pre-owned WS-X6704-10GE network modules for their Cisco 7606s, saving the frugal but by-the-book firm hundreds of thousands.