Migration crapness

I'm generally an easy going sucker. I don't think I get angry very easily, though I admit that I can get from being happy to being angry in an instant. Becoming angry happens rather rarely though.

Anyway, about 2 days ago I was involved in a network migration. From a single network, the traffic will be split into 2 networks, each handling a different role. Obviously there'll be network topology changes, with configuration being done on the firewalls, switches and routers. Arrived onsite to do pre-migration checking at around 3 pm. Final design documents arrived at 5.17 pm and migration begins at 9 pm. Fun! Sure, it'll be a bit challenging but nothing me and my colleague can't handle. Hell, if shit really happens, we could rebuild the entire network if we need to. We're THAT good! Well...ok, maybe not me. I'm barely average but my colleague's really REALLY good. He's the Jedi Master while I'm just the Padawan. Heh!

Anyway, we completed our configuration about 2 am or so. After that was the network verification part. Initially we had some problems with voice calls, video calls and sms but it was all sorted out in the end. Remember my colleague? Well he's the one who sorted it out. Me, I can handle the packet-switched networks pretty decent, I guess. Him, he's way more than decent at handling packet-switched network, circuit-switched network and radio access network. I have a looooooong way to go...

Once we verified the core network services, we proceed with verifying the value added services. Sure, there are some hiccups here and there but it was all resolved in the end. All in all, the migration was a success!

Of course, no good deed goes unpunished. Apparently connection to the billing domain wasn't up. They had some issues with connecting and running services to some of the network nodes. We've been working more or less non-stop for the past 24 hours so understandably we're all rather tired and worn out. For this issue, which is probably a minor routing problem, I asked another of my colleague to take a look at it. Despite my best efforts, I didn't really managed to get much rest. Sigh...too many phone calls from work.

Next day, I woke up late. Didn't go to work in the morning since I've got another night of activity later on anyway. Logged on to work from home and checked my email. All the feel-good factor of a successful migration disappeared in an instant. Imagine getting an email with title like "IMPORTANT: ALL NODES DOWN!! (CRITICAL)", and that email is cc'ed to practically the whole world. Bloody fuck!! It's only a minor routing issue which was resolved in about 2 hours anyway. Issue was raised around 12.30 pm yesterday, connectivity was restored around 2 pm. For God's sake, it's not even a LIVE network! Ok, fine. Just someone making a mountain out of a molehill.

Normally, I'd laugh this off but the part that REALLY pisses me off is the person who sent out that email. This is the same sucker whom when he needs help in resolving some networking issues, he comes straight to me. No emails, just a "Hi! We're having some connection problems. Can you assist us?". Now in this particular situation, he shoots of an email informing everyone that "all nodes down". Bastard!! When it's his problem, he wants to keep it quiet and approaches engineers directly. When it's other people's problem (in this case, mine), he shoots out email informing EVERYONE in the project.

Fucking hell!! Bloody two-faced bastard!! Grr... It's not like this is gonna reflect badly on me anyway. My scope was to migrate the network and make sure that the core and value added services are working as expected. Billing is not part of that scope. Dammit, I'm still pissed. From now on, no more charity work. If my routing, switching and firewall policies are fine, billing people can sort out their network themselves. I'm not going to do any traces or do any analysis on THEIR network. Let them do it themselves, assuming they even know how!