An Update on Our Outage

As most of the Roblox neighborhood is conscious, we not too long ago experienced an extended outage throughout our system. We are sorry for the duration of time it took us to restore support. A crucial value at Roblox is “Respect the Community,” and in this case, we apologize for the inconvenience to our community.  

On Thursday afternoon, October 28th, customers began acquiring difficulties connecting with our system. This straight away grew to become our greatest priority. Groups started performing around the clock to identify the source of the trouble and get issues back to typical.  

This was an particularly complicated outage in that it involved a blend of various aspects. A core method in our infrastructure became overcome, prompted by a subtle bug in our backend service communications even though beneath significant load. This was not thanks to any peak in external targeted visitors or any individual practical experience. Rather the failure was brought about by the expansion in the selection of servers in our datacenters. The outcome was that most providers at Roblox were not able to successfully talk and deploy.  

Owing to the issue in diagnosing the actual bug, restoration took longer than any of us would have appreciated. Upon properly figuring out this root lead to, we were being equipped to take care of the challenge by overall performance tuning, re-configuration, and scaling back again of some load. We were capable to thoroughly restore assistance as of this afternoon.

We will publish a publish-mortem with more information after we have finished our analysis, together with the steps we’ll be having to keep away from these kinds of problems in the potential. In addition, we will carry out a plan to make our creator community economically total as a final result of this outage. There are extra details on this to come. As aspect of our “Respect the Community” price, we will keep on to be clear in our submit-mortem.

To the most effective of our know-how, there has been no decline of player persistence information, and your Roblox expertise need to now be thoroughly back again to ordinary. You can often get hold of our aid workforce if you expertise any hiccups using Roblox now or in the potential.

We are grateful for the persistence and help of our players, builders, and companions for the duration of this time.