Blackout in the Zuckerberg Group: Whatsapp, Facebook and Instagram down worldwide

Anyone who tried to log into Facebook, Whatsapp or Instagram only received an error message for about six hours. The reasons are still officially unclear.

Whatsapp messages can neither be sent nor received, the same is true on Facebook. Publishers like t3n cannot even publish new news there at the moment. The server API returns an error. Instagram users have problems loading feeds and stories – in short: the Zuckerberg empire is completely down.

Update: Facebook, Instagram, Whatsapp report back after a worldwide failure.

The German portal allestören.de is also literally flooded with error messages. For the Whatsapp messenger service alone, the portal received more than 20,000 fault reports in just over half an hour. The Facebook Messenger has completely stopped the service. He reports “no server connection”. In the meantime, even the status page is down, which is supposed to show which errors there are on Facebook.

Whatsapp has now reported via Twitter, but without offering any significant information or a perspective.

“Some people” would have problems accessing WhatsApp, it says, while actually no access to the service – be it via the app or the web – is possible. The most telling error message is currently provided by Instagram. Whoever calls up the website receives a 5xx error after a considerable waiting time. In general, it describes errors that are caused by the operator’s server infrastructure. So this is not just a misdirected DNS entry , as we noticed at Slack a few days ago.

In the meantime, Facebook has also dared to make a similar statement on Twitter. The best way to get information from the Zuckerberg empire seems to be on Twitter on Monday evening.

5xx errors can affect the server itself, but also upstream gateways or proxies. Facebook has all of it in action. So we only know: The problem lies with Facebook and with some certainty not with a service provider such as a CDN. Unforgotten in this context is the recent failure of large parts of the network because of problems with Fastly .

On Hacker News is speculating about DNS problems. But then they would have to affect the domain name services in the Facebook infrastructure. It goes without saying that they would be down if the servers themselves were also down. Shortly before 7 p.m., more and more details become known, even if there is still no meaningful statement from the Zuckerberg empire itself. It is now clear that all Facebook name servers are not only down, but no longer entered in the name services worldwide. Queries for facebook.com and whatsapp.com do not bring any result, i.e. they do not return the actual IP addresses, just as if the names were not assigned an address at all.

The Zuckerberg services could use this Can’t find each other either – this suggests that Oculus can no longer be reached.

Journalist Dylan Byers claims to have leaked an internal Facebook memo. It reads like this:

In essence, the memo confirms that the problem affects both internal and external resolution of IP addresses. This is strange in that internal name resolution would not go through external DNS servers. Speculations on a much larger problem seem justified.

A Reddit user wants a source on the technical team investigating the issues. This source is said to have said the following:

As many of you know, the DNS for FB services is affected. This is probably just a symptom of the real problem which is BGP peering with Facebook’s peering routers. This is most likely due to a configuration change that went into effect just before the outages (around 1540 UTC).

There are people now trying to get access to the peering routers procure to make corrections. But the people who have physical access are separate from the people who know how to authenticate to the systems and the people who know what to do. So it is now a logistical challenge to combine all this knowledge.

This is partly due to the lower staffing levels in the data centers due to the pandemic measures.

To shortly after 20 On Monday evening there are increasing indications that the problem could be a serious error in the DNS configuration of the Facebook domains. The security expert Brian Krebs even wants confirmation for this, but does not say where from.

In any case, Krebs writes that all Facebook DNS entries were removed from the global routing tables on Monday morning in California. Since Facebook has sole control over its DNS records, it is still unclear how this could happen. An internal, system-wide update is conceivable, which could have failed.

The network specialist Doug Madory specifies that the deleted routes are so-called BGP routes, which can be configured via remote connection. If you now delete these entries, you take remote access. So it could be that Facebook is actually handing over to them Fenced hardware has to be used, which can be difficult because hardly anyone is on site.

This explanation is quite close to the one mentioned above, which is said to have been posted on Reddit by the alleged member of the investigation team. From the point of view of the CDN service provider Cloudflare, a connection with a BGP error is obvious:

In the meantime, faults are also reported with other services, such as signal or telegram. In the opinion of an expert in network infrastructure, however, this has only a very indirect connection. The head of technology at the cloud service provider Cloudflare, John Graham-Cumming, pointed out that users and software were still trying to access Facebook services. This would fundamentally increase the load on DNS services worldwide, which could then lead to disruptions which, however, have nothing to do directly with the services concerned. In fact, no extensive interference with other services has been verified so far.

The The New York Times asked two cybersecurity experts whether a hacker attack was likely. They said no, because the technology behind the individual apps was so different that they couldn’t be brought offline with a single cyberattack.

NYT-Reportering Sheera Frenkel reports on Twitter from a phone call with a Facebook employee. He described a real chaos to her. If the workforce tried to enter the building on Monday morning, their digital keys would not work. Apparently, Facebook really has broken down into all its individual parts. There are apparently no areas that are not affected.

And the Internet so …

The Facebook share price was almost even faster than the servers: