T-Mobile Deutschland GmbH suffered a massive network outage Tuesday that left all of its 40 million customers unable to make calls or send texts for four hours.

The service collapse was caused by a software problem in the network that caused the Home Location Register (HLR) servers supplied by Nokia Siemens Networks to crash, which, with refreshing accountability, has issued an apology.

"We apologize to all T-Mobile customers for the service disruption," says a Nokia Siemens Networks (NSN) spokesman. "We're working closely with T-Mobile to identify the software problem and why the redundancy built into the system didn't kick in."

The HLR database holds all of a mobile operator's subscriber information -- such as location, preferences, account status -- which is critical for routing calls. If this database goes down, calls cannot be completed.

In T-Mobile's calamitous outage yesterday, the subscriber locations were not being updated on the central HLR database, according to an NSN spokesman. All the HLR servers had to be taken offline and brought back up one at a time, which explains why the outage lasted as long as four hours.

The HLR servers that went wrong yesterday were installed by T-Mobile as part of a major upgrade project to streamline its subscriber databases. The servers in question are part of the product line Nokia Siemens acquired when it bought Apertio Ltd. for $206 million in January 2008.

Counting the costs

A T-Mobile spokeswoman would not comment on how much yesterday's network outage would cost the operator in lost revenues.

But Germany-based telecom executive Chris Larmour, chief marketing officer at Actix Ltd. , which provides automated network status management systems to operators such as Vodafone Group plc (NYSE: VOD), estimates it could be about $100 million.

Larmour believes the severity of T-Mobile's network outage yesterday could have been limited.

"Some of this could have been avoided," says Larmour (who, incidentally, is not a T-Mobile customer). "It went wrong and no one was able to manage it. It took them four hours to figure it out. It will take them months to get back to normal."

The outage could be costly, not just for T-Mobile in terms of lost revenues, damaged market perception, and service reputation, but also for NSN if T-Mobile decides to seek compensation.

After a similar network outage back in 2004 at Bouygues Telecom in France, the operator sued its HLR supplier, Tekelec Inc. (Nasdaq: TKLC), for $81 million in damages.

Source:Unstrung