On Call It’s another tale from the world of telephony where everything goes wrong in this week’s On Call.

Today’s story comes from a Register reader Regomised as “Greg.” Greg was the applications manager for a now-defunct telco, and was the throat to choke for all the applications in the customer-facing side of the business. The network side, where telephone calls were actually connected, was a whole other ball game for which he was most definitely not responsible.

“On my side of the world,” he told us, not at all highlighting a worryingly siloed side to the business, “we had a customer service application, which is where the sales clerks entered the details of new customers.

“The clerk would type up the new customer’s details, the application would save them and tell the network side to set up the customer on the telephone exchanges. When done, the network side would tell the customer front end that this was all complete.”

The system hadn’t been so much designed as “evolved” over time, the dependencies likely lost in the mists of poorly commented code and unhelpful documentation that had accumulated over the years like crud in a cutlery drawer.

“It worked, most of the time,” he said, “but that was all you could say.”

And as much as one might hope for.

However, there came a time when a total power-down was needed. Everything in the building had to be turned off – computers, servers, everything. There was to be no UPS, no battery backup. All kit was to be shut down.

This shouldn’t have presented a problem in itself. The remote telephone switches would deal with network calls, but customer service was to be totally offline.

“It took weeks of planning,” said Greg. “We had never tested a total power down before, because we had redundant systems, so a…