Yesterday, a developer accidentally reset the content of 186 Git repositories of the famous Jenkins project on GitHub. The content of those repositories got reset to an older version with no possibility to restore from remote without knowing the latest valid content.
In contrast to many other version control systems, Git has a force push option which allows you to change the history of repository in a non linear way, i.e. with no possibility to just go back to the previous commit to restore history. In a similar fashion, it is possible to remove entire branches without a possibility to restore them remotely (unless you still have a copy of them locally). The incident from yesterday is not the first one when this happened, the Eclipse foundation suffered from a similar case where almost all their Git branches have been accidentally deleted. Both Eclipse and the Jenkins community are higly professional organizations whose contributors belong to the top of the top. If incidents like this happen there, it is a fair assumption that they happen way more often in enterprises, in many cases even completely unnoticed (and definitely not published) – a nightmare from an enterprise compliance perspective (Basel II, SOX, etc).
Unfortunately, turning off force pushes completely is not the answer as there are many legitimate use cases for this feature. My previous post on GitHub’s private key exposure over source code search shows a very valid one: Being able to prune problematic IP (copyright violations, confidential information) for your repository. Other legitimate use cases are removing no longer used development branches from remote servers and cleaning branches from large binary files, making CI operations way faster.
CollabNet TeamForge’s Git backend comes with a feature called History Protection. Whenever we detect a history rewrite attempt at our servers, we won’t block it (given necessary permissions are granted in TeamForge). We know that there is a number of legitimate use cases out there (see above) why blocking is not an answer. Instead, we will log exactly what happened (who, when, what) in our tamper proof audit log, notify Git administrators and provide an ability for users of the repository to restore the previous content in a self-service manner (your administrators do not have to get involved). If needed, Git administrators can still permanently remove selected content (like accidentally pushed credentials or unlawful content) at a push of a button. With TeamForge History Protection, our customers can use Git’s history rewrite features with an audit compliant safety net. The feature is available both for hosted customers as well as on-premise.
If you like to find out more about CollabNet TeamForge’s History Protection feature or our Git integration in general, please visit our web site or watch an on demand webinar. If you have any specific questions or feedback on this blog post, feel free to drop a comment.