True Tales From the Trenches: Darren’s Story

From the Trenches: Darren’s Story

True Tales Pulled from OSI’s Customer Files

Customer Problem: Unexpected Late-Night Network Outage Due to Dead Cisco 6500 Ethernet Module

Solution: 10:30pm Red Eye from LAX to Philly, a 2-Hour Rental Car Drive, and a Datacenter Parking Lot Hand-Off of Two WS-X6716-10GE-3Cs

Who do you call when a server blade goes down at 8:00pm Eastern time Monday night and half the network running your customer’s VMWare applications is dark?

Monday evening, February 12th, 2018 | 5:38pm Pacific time

Darren Nevolo, OSI Hardware Senior Account Executive, and his wife Shani are preparing dinner for their three children. It’s been a relaxing weekend following months of turmoil. Just scant weeks ago their hometown not only endured California’s largest wildfire in recorded history but suffered a devastating body blow from a deadly flood triggered by monsoon-like rains. Unseasonably warm and sunny late winter weather have provided a welcomed respite for the young family.

Across the kitchen, Darren’s Samsung Galaxy Note 8 lights up: “What are the odds of you getting us a WS-X6716-10GE line card in a handful of hours? It’s for our partner Ironton. They are in a pinch and have no spare.”

The text is from Frank Clements, Director of Engineering at PenTelData, an ILEC serving Pennsylvania and New Jersey with over 9,000 miles of fiber optic cabling. The dark network is located in the data center of Ironton Global. A wholly owned subsidiary of Ironton Telephone

Company, the firm operates a massive state-of-the-art fiber optic infrastructure and 60 Points of Presence throughout both PA and NJ. There’s a lot at stake; the 107-year old, family-owned CLEC has an unblemished track record of over a century of uninterrupted service to the residents of Lehigh Valley in eastern Pennsylvania.

Asking his wife to keep an eye on the marinara sauce simmering on the cooktop, Darren immediately dials PenTelData to delve into the problem more deeply. The issue is a blade server that went down 20 minutes ago. Intensive troubleshooting identified the culprit: an Ethernet module purchased only a few months ago from an OSI Hardware competitor had suddenly gone south.

With all eastern seaboard parts warehouses shuttered until dawn and no replacement WS-X6716-10GE-3C modules in sight, it promised to be a long night and a very rough morning ops meeting at PenTelData.

“Whaddya say, Darren? Can you guys help us out?”

Having travelled over 250,000 miles annually during his 12-year career at OSI, Nevolo knew PenTelData’s infrastructure well. He’d been working with the firm for several years, providing pre-owned WS-X6704-10GE network modules for their Cisco 7606s, saving the frugal but by-the-book firm hundreds of thousands.

Have WS-X6716-10GE-3C Modules, Will Travel

“You just never know when a component will go bad. But taking care of the customer makes everyone feel good.”
– Darren Nevolo

Airline ticket from LAX to PHL purchased 90 minutes prior to departure

Cost: Definitely over Darren’s travel voucher allowance.

Value to customer having network restored just in time: Priceless

A Bay-area transplant before he relocated to the “Silicon Beach” community of Santa Barbara, Darren felt at home with the regional telecom’s approachable but technically-savvy network engineers in Ironton and Lehigh. One of the fastest growing metropolitan regions in the nation, the area is a unique blend of rural values and big city astuteness. Recently Lehigh Valley was ranked as a top pick to locate finance and IT operations by Fortune Magazine.

“These boys know their stuff,” he’d commented to Christian Saunders, OSI Global IT’s CEO and co-founder, when he returned from his first visit. “They’re all over the latest cloud scaling stuff and they run a local exchange with an eye for detail that puts the national players to shame.”

A tight-knit family business, Ironton Telephone is a lot like the Lehigh Valley itself – just about everyone knows your business and you can count on your true friends and business associates to always have your back. Now they needed a miracle and they were counting on OSI to come through.

Logging on to OSI Hardware’s NetSuite inventory database from his home office computer, it looks like everyone is in luck.

“We’ve got six WS-X6716-10GE line cards in stock”, he found himself telling Christian a few minutes later, having pulled his old friend and boss away from family dinner. “We gotta help these guys. Can you meet me at the shop in half an hour? I’m going to hop a red-eye out of LAX the second they give me the green light.”

“Let’s do it,” was the immediate response as the call abruptly ended.

“Damn, he’ll probably be in the stockroom before I hit the 101”, Darren thought as he searched his mind for the precise explanation he’d unpack for Shani in just a few moments. Ten minutes later at the usually bustling 606 Olive ops center, Christian is handing him the sealed units wrapped in static-resistant poly.

“Here, take two and go,” Christian says. “We’ll catch up with the paperwork later.”

“Good thinking,” Darren agreed. “They still won’t have a spare after installing the replacement. Wait!”, he says freezing. “I don’t yet have definitive approval for the quote from PenTel.”

Just then his Note 8 lit up with a text. It was a “go”.

“Good night and safe travels”, Christian said, slapping him on the shoulder. “You rule.”

“Working at a place where the CEO drops his fork at 6:30 on Monday night and rushes down to open up the shop, now that rules”, Darren says to Shani as he jams a few shirts, a light jacket, and his laptop into a backpack.

ID, What ID? I Have to Fly Now

Ninety- minutes later he’s parked in an LAX parking structure and sprinting over to the American Airlines check-in counter.

LAX TSA Security Screening,
9:52pm “I bet that pasta sauce was fantastic,” Darren thinks ruefully as he waits in line.


“You’re all booked in and departing in 18 minutes, Mr. Nevolo,” says the friendly AA counterperson. “Now if I can just confirm your identify with a photo ID, you can board immediately.”

“Photo ID, photo ID”, he mutters opening his wallet. A mental image of handing his California driver’s license to Christian back at the shop two hours ago flashes before him. “He needed it to book my ticket online. And I forgot to get it back!”

“Wait, I may still have my old one with the outdated address. Yes!”, he shouts, pulling it from a little used corner of his wallet. “Another improbable hurdle dodged,” he thinks with relief.

Four hours and thirty-seven minutes later the Airbus A321 is touching down in Philly. As he maneuvers the rented Altima onto the nearby 248, he thanks the early morning hour for its virtually traffic-free condition. Arriving in Palmerton, 82 miles north-northwest of Philly spot on at 9:00am, he pulls up into the darkened PenTelData parking lot. He’s greeted in the lobby by Frank, the worried Director of Engineering who’d sent the last-ditch text message last night, but seemingly days ago.

Headquarters of Ironton Global

Located in the heart of bucolic Lehigh Valley where they do cloud communication the old-fashioned way: reliable, simple & affordable

“Man, it’s good to see you!” he exclaims. “We need to drive over to the Ironton Global HQ on Mauch Chunk Road in Coplay,” he says visibly relieved.

“Mauch Chunk Road,” Darren contemplates. “You can’t make this stuff up!”

Twenty minutes later they’re in front of Ironton’s offices. Gary Yencho, the top ops guy meets them just outside the front door. He’s definitely expecting them and the WS-X6716-10GE.

Yencho reaches out to shake his hand, “You are a lifesaver,” he says guiding them toward the datacenter.

“We bought these switches from, uh, one of your competitors. Well, you know who I mean,” he says tersely. “When this one went down, we reached out to them, and – nothing. Totally M.I.A. on us and that was not cool.”

After some small talk and sharing a few outage war stories, Darren is back in the Altima, blasting the heater and contemplating the journey back to Santa Barbara. He glances at his watch: 7:47am east coast time. “It’s almost 8am back home,” he thinks. The sights and smells of Brian Charrette’s freshly baked artisanal bread and a pan of crispy bacon frying in the OSI breakroom kitchen flood his imagination.

“They’ll be on the phones in two hours,” he muses. “I’m going to text Christian and let him know the customer is back up and that I’ll probably need some jet lag sleep before I get in the office Wednesday morning.”

The next day an email from Gary pops up on his Note 8:

Good Morning Darren,
We did bring the network up yesterday early evening. The good news is that you brought 2 blades, bad news is one blade failed – Was in service several hours. Currently, the status light illuminates red, flashes to green and then goes dark. This repeats and does not return to active. I was waiting for California time zone before calling you. Give me a call at your convenience regarding this.

Also, your response was the best advertisement for OSI and your commitment to customer service. Ironton is very appreciative and is happy to add OSI to our preferred vendor list.
Talk to you soon.

Regards,
Gary Yencho Supervisor,
Network Operations
Ironton Telephone Company

Waiting for the flight home A refreshment at Rock & Brews at LAX. “The customer support road has many twists and turns, and occasionally, a boilermaker,” Darren says to himself, dreaming of a long nap and some delicious homemade marinara sauce.

Waiting for the flight home

A refreshment at Rock & Brews at LAX. “The customer support road has many twists and turns, and occasionally, a boilermaker,” Darren says to himself, dreaming of a long nap and some delicious homemade marinara sauce.

OSIHardware_logofullTransparent

About OSI Hardware

OSI Hardware is a leading supplier of pre-owned, surplus and new in-box IT infrastructure hardware for organizations seeking to break free from the constraints imposed by outmoded and self-serving OEM business practices. Our expertise and strong partnerships within the secondary market empowers our customers to avoid unnecessary expenditures, minimize complexity, and maximize the buying power of their IT budgets. OSI Hardware’s massive inventory of name brand routers, switches, servers, optics and cabling manufactured by Cisco, Juniper, Brocade and dozens of other vendors are consistently priced at 10% to 40% less than any available OEM price discounts. Our global reach and dedication to ensuring technology serves the business requirements of our customers has made us the “go-to” technology resource for clients here in the U.S. and Canada, and across the globe. Let’s get to work.™

For more information, call 1-866-602-4674 or visit www.osihardware.com.