Announcing Forto’s open-source project helm-rollback-web

Wouldn’t it be great if product managers/owners had the ability to hit the rollback button themselves if they realise the release their team did had unintended consequences?

How about the developers themselves? Not the on-call dev, but any developer who saw the problem and could confirm it was because of their change?

Well, here’s where there tends to arise a problem — not all product folk have the technical skill nor the access required to use helm rollback at the command line. Neither do all engineers (the engineer on-call has production access but not every engineer is on call at the time a release goes out).

Some companies solve this by having a “held” rollback step in their CI pipeline (build->test->(sometimes hold)->deploy->(hold)->rollback) but that is messy too because you’re going to rollback far less than every release. Worse still, that might mean your pipeline never completes (meaning your deployment never “goes green”, and sits there looking sad and gray, even though it went to production).

Enter: helm-rollback-web, a tool we developed at Forto to enable a subset of users (namely people such as engineers but also product managers) to rollback a release without cluster access. It also picked up a few neat features along the way.

Here’s a screenshot of how it looks when it’s running:

Note the blue color of the interface — that means we’re operating on the sandbox environment here, if you access it on our production environment, it’s a nice red color!

On the left, you can see a list of namespaces that are in the cluster and the most recent release in each of those namespaces. On the right, the top 10 most recently deployed releases.

If you click the namespace in the left hand side you get a full list of deployments within that namespace:

And here’s what happens if you click on a release itself:

Yes, helm-rollback-web can rollback releases of helm-rollback-web!

Now, some of you might be wondering what database the tool is using to track releases so it can display in a nice list like this (which one of our engineers made mobile-friendly because if someone calls you and you want to rollback, you then can). Here’s the clever bit, there is no database. The system uses the standard helm3 secrets in the cluster. In fact, to allow for maximum compatibility, it actually calls the helm binary.

So, in an emergency, you can rollback a bad release at the click of a button, authorized by the groups you belong to (in Forto’s case that’s product managers and engineers).

Here’s the rollback screen in full glory:

Note the little info panel on the right with a direct link to datadog where the application logs are — that’s handy! It’s also possible to link to the release list with a url like /rollback/{namespace}/{release-name} which allows for things to directly link to the rollback tool such as, for example, CI slack notifications. Did I mention helm-rollback-web also pings slack so people have visibility?

And so, as this tool is really useful for us, and we think it would be really useful for any company using helm, we’re open-sourcing it, giving back to the community that has given us so many great tools, by building on the shoulders of giants.

Let us know if you use the tool. We’d love to hear about how you like it (in the comments here, or on the github page), or perhaps you have some cool ideas about how to make it even better — PRs are most welcome.

Finally, if working on such tools seems like an exciting thing to you, why not check out our careers page. We are always looking for talented engineers who are themselves looking for a new and exciting place to work.