Facebook, WhatsApp and Instagram crash in major outage

Nearly $50bn (£36.7bn) was wiped off Facebook’s value after its social media platforms went down for several hours due to a “faulty configuration change”. 

Facebook and Instagram were partially reconnected to users around the world around 10.45pm on Monday, nearly six hours into a global outage.

Facebook issued an apology for the outage which impacted the social media giant, along with Instagram and WhatsApp, which Facebook also owns.

Facebook Engineering said in a statement on Twitter late on Monday: “To the huge community of people and businesses around the world who depend on us: we’re sorry.

“We’ve been working hard to restore access to our apps and services and are happy to report they are coming back online now. Thank you for bearing with us.”

Mike Schroepfer, Facebook’s chief technology officer, said on Twitter: “*Sincereapologies to everyone impacted by outages of Facebook powered services right now.

“We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible.”

Nearly three hours later he said: “Facebook services coming back online now – may take some time to get to 100pc. To every small and large business, family, and individual who depends on us, I’m sorry.”

Just after 3am, Mr Schroepfer pushed out a link – also on Twitter – to a Facebook statement detailing the cause of the outage: “Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centres caused issues that interrupted this communication. 

“This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.

“Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.” Read the statement in full below

Shares in the technology giant fell 5pc as the worldwide outage persisted. 

The drop meant Mark Zuckerberg, Facebook’s founder and chief executive, suffered a $7bn fall in his paper wealth. An estimated 2.8bn people globally use at least one of Facebook, WhatsApp and Instagram daily. 

The failure also meant the company’s 60,000 employees were unable to access Facebook’s own internal communication systems, rendering many unable to do their work as the outage coincided with the start of the day at Facebook’s California headquarters. The Oculus virtual reality system that Facebook owns was also sent offline.

Network experts attributed the problem to Facebook’s own servers being unable to connect to a global system that directs web browsers and apps to the social network. The problem may have been caused by a bug or a mistaken code update.

The outage also left Facebook staff unable to access the company’s buildings after their electronic entry keys stopped working, The New York Times reported.

It added that a team of employees was dispatched to a California data centre to attempt to manually reset the company’s servers.

“We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologise for any inconvenience,” a spokesperson said.

Facebook bought Instagram in 2011 and WhatsApp in 2014, and its ownership of three of the world’s biggest internet services has seen it draw attention from competition watchdogs.

The company made around $320 million (£235m) per day in revenue in the three months to the end of June, or $13.3 million per hour, meaning Monday’s outage may have cost the company tens of millions of dollars.

Users first started noticing a problem shortly around 4.30pm, with the outage continuing into the night, and affecting every user. It is believed to be the longest total outage Facebook has experienced in several years.

The error is believed to have been down to the way the tech giant’s servers interact with the “Domain Name System”. The system, known as the “internet’s address book”, is a way of routing internet traffic that ensures web browsers go to Facebook when they type “facebook.com” into a browser. 

Jack Dorsey, the chief executive of Twitter, poked fun at the outage, tweeting “how much?” when the error made it appear as if the “facebook.com” address was for sale.

The company will be under further pressure on Tuesday when a former employee-turned-whistleblower testifies to a US Senate committee about Facebook’s impact on teenagers. Frances Haugen, who revealed her identity over the weekend, has accused the company of putting “profit over safety”.

It came as tech shares including Twitter, Apple and Amazon also saw significant declines in a sell-off driven by rising bond yields and a US stand-off over the debt ceiling. The Nasdaq declined by 2.3pc, putting it on course for one of its biggest falls of the year.

Has the issue been resolved?

When users started reporting problems, Whatsapp said in a statement: “We’re aware that some people are experiencing issues with WhatsApp at the moment.

“We’re working to get things back to normal and will send an update here as soon as possible. Thanks for your patience.”

At 12.17am on Tuesday, WhatsApp hopped on Twitter to update users: “Apologies to everyone who hasn’t been able to use WhatsApp today. We’re starting to slowly and carefully get WhatsApp working again.

“Thank you so much for your patience. We will continue to keep you updated when we have more information to share.”

Just before midnight, Instagram announced it was “slowly but surely coming back now”:

The issue also affected other platforms such as Twitter amid an increase in traffic on its website and app.

Twitter Support tweeted: “Sometimes more people than usual use Twitter. We prepare for these moments, but today things didn’t go exactly as planned.

“Some of you may have had an issue seeing replies and DMs as a result. This has been fixed. Sorry about that!”

It had earlier joked: “Hello literally everyone.”

How did this happen?

Several Facebook employees who declined to be named said they believed the outage was caused by an internal routing mistake to an internet domain that was compounded by the failures of internal communication tools and other resources that depend on that same domain in order to work, Reuters reported.

Facebook, which is the second-largest digital advertising platform in the world, was losing about $545,000 in US advertising revenue per hour during the outage, according to estimates from ad measurement firm Standard Media Index.

Security experts speculated the problem came following network changes, with the cause still unconfirmed.

Adam Leon Smith, of BCS, The Chartered Institute for IT and a software testing expert, said: “The outage is caused by changes made to the Facebook network infrastructure.

“Many of the recent high-profile outages have been caused by similar network level events.

“It is reported by unidentified Facebook sources on Reddit that the network changes have also prevented engineers from remotely connecting to resolve the issues, delaying resolution.

“Notably, many organisations now define their physical infrastructure as code, but most do not apply the same level of testing rigour when they change that code, as they would when changing their core business logic.”

However, cyber security specialist Jake Moore said there is a “chance” the issue could be related to a cyber attack.

He told the PA news agency: “There have been many reports and I’m struggling to find out exactly what has happened – I’m reading it could be DNS related, which means there is an issue with the connection not knowing where to go to your device.

“It could well be a human error or a software bug lurking in the shadows but whatever it is Facebook needs to do its best to mitigate the problem of causing more panic about this.

“The biggest problem is fears over a cyber attack but as we saw from Fastly in the summer I would hedge my bets on that not being the case as we’re talking about one of the biggest companies in the world, but there’s always a chance.”

The New York Times reported the issue likely stemmed from a misconfiguration of Facebook’s server computers, which were not letting people connect to its sites like Instagram and WhatsApp.

It said the problems appeared to be more complex and required some manual updating.

Some security experts said the disruption could be the result of an internal mistake, though sabotage by an insider would be theoretically possible.

“Facebook basically locked its keys in its car,” tweeted Jonathan Zittrain, director of Harvard’s Berkman Klein Centre for Internet and Society.

Facebook’s explanation to the world

To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms. 

We’ve been working as hard as we can to restore access, and our systems are now back up and running. 

The underlying cause of this outage also impacted many of the internal tools and systems we use in our day-to-day operations, complicating our attempts to quickly diagnose and resolve the problem.

Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centres caused issues that interrupted this communication. 

This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt.

Our services are now back online and we’re actively working to fully return them to regular operations. 

We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.

People and businesses around the world rely on us everyday to stay connected. 

We understand the impact outages like these have on people’s lives, and our responsibility to keep people informed about disruptions to our services. 

We apologise to all those affected, and we’re working to understand more about what happened today so we can continue to make our infrastructure more resilient.