Let’s talk about the “git push” operation. It is commonly used as you need to push your local commits to the blessed server to share them with others. However it’s important to be aware that there are two use cases when it can be abused:
- removing remote branch
- force push aka history rewrite
Let’s take a minute to focus on these two use cases.
The most simple usage of a push operation is:
which is equivalent to:
git push origin master:master
So what does that mean? It means: “hey Git, push my local branch called ‘master’ (the one on the left hand side of colon in “master:master” parameter) to the remote branch ‘master’ on the server called ‘origin’”.For instance, let’s say that you want to push the content of branch ‘production-quick-fix’ to ‘stable’ branch on the origin server. For this purpose you’ll type:
git push origin production-quick-fix:stable
Easy as that. You have done your work, it’s Friday afternoon so you can happily go home and start your weekend … But, what if you made a little typo, and typed something like this:
git push origin production-quick-fix :stable
(Note that there is a space between ‘production-quick-fix’ and colon). ln this case, Git will create a remote branch called ‘production-quick-fix’ on the remote host and REMOVE the remote branch ‘stable’. Of course it’s not as scary as it sounds, you will have copy of ‘stable’ branch in ‘production-quick-fix’, but the only person who knows that is … you. In this case Git will inform you about the action performed, so if you look carefully at the Git output you’ll notice this immediately. But there is no warning message or question confirming this request, only information that the operation was successfully performed. Wouldn’t it be great if you could audit who removes remote branches and could be notified when it happens? Maybe even have the ability to resurrect deleted branches after some time?
OK, let’s talk about ‘force push’ now. By default, Git will not allow you to push changes that are not ‘fast forward’ (i.e. changes that modify past history). You are safe here because during a push operation Git automatically checks if your changes are ‘fast forward’ or not and in case of ‘non fast forward’ changes it will simply stop … However, one could force Git to ignore this rule by adding ‘-f’ parameter to the push command. This replaces server side history with the local one … and will not leave any traces of this operation unless you were careful enough to enable reflog logging on server side.
Maybe now you are scared about this small ‘-f’ parameter … well, you shouldn’t be. It is really useful in some cases, but you need to be aware of what you are doing. For example when you discover a file without a corporate header statement, you can easily add this header to past versions. Or perhaps you find some GPL code snippet in the code base, you can remove it by rewriting history. Or what if somebody accidentally checked in tons of binary files choking Git? This can also be fixed by rewriting history and force push. Git gives you tools like ‘filter-branch’ or ‘rebase interactive’, to do such rewrites effectively.
OK, so you know the pros and cons of a ‘force push’ operation. You are aware that in some cases it’s necessary, but would prefer to have this under control or at least be able to restore old history. So lets enable ‘reflog’ on the server. It is really easy, just log in into the server’s shell (yes you’ll need to have an SSH account there), go to the desired repository (and yes again, your user must have read permission there) and then type:
git config core.logAllRefUpdates true
This will require write permission. Now you are safe! … not really, because by default reflog entries older than 90 days will be pruned by git gc (‘garbage collect’). You need to tweak Git a bit more, so let’s turn off gc and just for to be sure set the ‘reflog’ expiration date to some huge number:
git config gc.auto 0 git config gc.reflogExpire 10.years git config gc.reflogExpireUnreachable 10.years
So now you are safe… or at least safer. So for a quick summary:
- your administrators need to be Git experts in order to properly configure and understand reflog,
- nobody will be notified about a ‘force push’ action, unless you add a special hook for this purpose,
- if you want to prune something completely from your history, you also need to find it manually in reflog and prune it (otherwise somebody can figure it out and restore the old repository state from ‘reflog’).
Is there an easier way to achieve the same result? Yes, with TeamForge’s ‘History Protect’ feature! So what does ‘History Protect’ do? It detects remove branch and force push operations and creates a special reference pointing to the current state. This reference contains all necessary information such as:
- who made the change,
- when was it done,
- what was the branch/ref name
- what are the old and new commit ids
Also it enables a notification email to be sent automatically to members of the Gerrit ‘Administrators’ group, and automatically creates a new entry in the audit log (not ‘reflog’). ‘History Protect’ can be enabled from TeamForge for a particular repository or forced to be active on a whole Gerrit instance in the global configuration file. When it is enforced globally, not even site admins can deactivate it.
You can also examine force push and branch deletion from within
or using the Git command line tool:
When you want to resurrect old entries, you only need to click on the “Resurrect” link in Web UI, give it a new branch name, and that’s it. You are done. If you prefer to use command line tools, you can just fetch your special ref locally, and re-push it as new branch. Also all refs under ‘refs/rewrite’ and ‘refs/delete’ are read only, nobody can push or remove from there. If for any reasons you want to delete your special ref, you need to use the Gerrit Web UI and be a member of Gerrit ‘Administrators’ group.
You can run Git ‘gc’ as often as you want to, as it will not break anything. You still will be able to access all historical repository states. Whenever you want to prune something from the repository permanently, just use ‘Delete permanently’ for the entry connected with this version using Gerrit Web UI and run ‘gc’ on the server repository. That way, all traces will be erased.
‘History Protect’ is transparent for end users, easy to configure and secure. With this feature activated, you will be protected from malicious operations and immediately informed about any such attempted operations. Also you are easily able to restore old repository states whenever it is required. In comparison to ‘reflog’ all administrative operations in ‘History Protect’ can be done using web browsers or Git client, you don’t need to have access to server shell or ask server administrators for assistance. No additional Git configuration and knowledge is required – just flip the bit and you are all set.
Have you ever had a problem with accidental force push or branch deletion in your organization? What happened and how was it resolved?
If you have any questions or comments to this post that I can address – just let me know.