Over the past week, we had a small system failure that knocked a few thousand Ting devices temporarily offline. Some of you reading this might have experienced that failure directly. Many others will have experienced unusual hold times trying to get to our support team as we were dealing with that inflated, failure-related call volume. I wanted to take the opportunity here to explain and apologize to all of you.
Our GSM network provider was performing a planned migration to a new backend system that Ting and other “wholesale partners” like us use to provision and manage our customers on the network.
(As a complete aside, referring to our GSM network provider reminds me how much I loved how everyone would cringe every time Harry said Voldemort. And he was all, “Yeah, I said Voldemort. Please, I’ve defeated him like five books in a row now, I’ll say his name if I want to.” Sorry, anyway…)
This was a change that was not even intended to affect us at all. It was certainly not intended to affect any of our customers. As it was explained to us, devices would seamlessly find their way from the old system to the new system without any outage or inconvenience.
We tested the migration on dummy accounts and experienced no problem. Then we started to cautiously migrate small batches of real customers on Wednesday, July 25 and everything was going smoothly. When we scaled it out further on Thursday and Friday, we started encountering the failures.
Most phones were able to reconnect to the network when restarted. But customers understandably had no way of knowing that and reached out for support. Meanwhile, some IoT (Internet of Things) devices that could not easily be accessed and restarted were more difficult to address.
Between guiding customers through those restarts and managing some fixes with our network provider remotely, it seems that we have gotten through the outages now. Devices seem to be working properly and our support team is answering calls at the speeds you have come to expect.
But we are truly sorry that we did not do a better job identifying these issues and addressing them before they ever affected any of you. You trust us to protect you from exactly these sorts of technical failures and to be accessible to you any time you need us. We take that trust seriously and we will be smarter, more diligent and more cautious in the future.
Thank you for your patience.