Xen Project colocation facility shutdown leaves open source OSSTest facing major disruption

To developers, OSSTest is an essential automated testing and quality checking system for anyone submitting code to the Xen Project’s open-source hypervisor.

But all software runs somewhere. In the Xen Project’s case it’s virtual server infrastructure inside in a featureless colocation datacenter just outside Boston. 

Unfortunately — extraordinarily even — it seems that this facility can no longer be used because of a “sudden shutdown” as an announcement on the Xen Project website this week explained it.

“We received a notification from our colo facility provider Synoptek indicating the shutdown of their services in Marlborough, where our current facility is located,” wrote Xen Project community manager, Kelly Choi.

Then the bad news. “As a result, the OSSTest instance hosted at this facility will likely be disrupted with an anticipated shutdown date of October 31st, 2024.”

Last resort

Technically, OSSTest is a gating Xen Project continuous integration (CI) loop, a complicated way of describing a shared system that replicates standalone testing tools of old.

But in an era of rapid development DevOps, it’s a stage that accelerates open-source development by spotting issues at an early stage.

Not having access to it won’t stop Xen Project development but will reduce the number of bugs being caught, said Choi. “Contributors may experience quality issues with code, or face slower response times as our technical community spends more time resolving issues. We are aware of the critical nature of the services we provide and the significant impact this downtime may have.”

The project was assessing its options, all of which promise disruption in different ways. Those include “abandoning our infrastructure and creating a new OSSTest instance.”

Alternatively, OSSTest could move to a bare metal service provider, which would require rejigging OSSTest itself. There’s also a nuclear option: “As a last resort, abandoning OSSTest has been discussed,” said Choi, ominously.

Xen today, gone tomorrow

As with VMWare’s ESXi — a platform wracked with uncertainties of its own after it was acquired by Broadcom — Xen is a “bare metal” hypervisor. Its distinctive feature is its open-source status, hence its use in third-party virtualization products, including famously the Citrix Hypervisor (formerly XenServer).

Xen lacks the features of mega platforms such as ESXi, but that’s OK as third parties such as Citrix and others can use it as the basis for adding their own.

However, losing the OSSTest CI loop, even for a while, isn’t a good advert for the platform’s stability.

What’s astonishing in an age that barely stops to wonder about the colocation datacenters that make all this possible, is how abruptly the loss of a single facility can throw a spanner in the works.

It’s not clear what is happening to the Synoptek datacenter at the heart of this story (we asked for comment but had not received any at the time of publication), but it highlights the fragility of much software development.

Large private developers can cope with this kind of disruption, either because they own their own datacenters or have layers of redundancy in place. Smaller OSS projects working on a shoestring aren’t as insulated.

What happens next?

According to Choi, a possible outcome is that someone steps into the gap, for example a university or an organization sympathetic to Xen or to open source in general.

Or perhaps it could migrate, with some difficulty, to an alternative testing platform such as Gitlab CI.