What is this and why is it good?
Git-hg Mirror is a simple to use (at least this is the intention) web application to automatically keep a Git and a Mercurial repository or two Mercurial or two Git repositories in sync. This allows you to create mirrors of repositories in the other version control system. Now why is this useful? Because you might prefer to use one of these systems but you find a library, tool or anything that resides in a repo that runs on the other system. It's a bit of a hassle to integrate a repository in your own one that runs on the other VCS; Git-hg Mirror makes it possible to maintain an automatically updated mirror so you (and your team!) don't have to go through the hassle of keeping a diverse version control setup. Diversity is a good thing but not in your repository!
Please keep in mind that Git-hg Mirror is a free service run in our free time - we strive to keep it in a good shape but hickups can happen. We at Lombiq also use it to sync our own repositories (including the 100+ open source ones) so if it's broken we also feel the pain and try to fix it as soon as possible.
Who created Git-hg Mirror?
Lombiq Technologies is behind Git-hg Mirror. Lombiq is a company driven by technology enthusiasts dedicated to open source software development (mostly with .NET and web technologies). If you're somebody looking for a creative solution for your software problem (especially if it's web, and even more so if it's Orchard Core/Orchard CMS) then you should definitely get in touch with us, probably right now if it's not already too late!
What are the configurations that are tested and 100% working?
Only HTTPS pull/push is supported, SSH won't work!
We tested the following configs and are using them ourselves so they're working for sure:
- Git repo on GitHub to/from/two-way with a Mercurial repo on Bitbucket
- Mercurial and Git or another Mercurial repo on Bitbucket (either way)
- Two Mercurial repos on Bitbucket, two-way sync possible
- Want to hear a secret? Git-git mirroring works too (tested between GitHub and Bitbucket repos)! Just make sure to use either the "git to hg" or "two-way" direction and a git URL with the format "git+https://username:[email protected]/Lombiq/repo.git" (note the protocol being git+https and the URL ending in ".git"). This feature is kind of hidden for now but eventually we'll have a better, completely new UI.
If you use another type of configuration and you're happy with it please let us know and we'll add it here.
Whatever you do however, never rewrite already pushed history! E.g. never remove commits if you already pushed them (regardless of which source control system you do this in) because this can totally mess up syncing to the point that it should be restarted with a completely new destination repository.
I've rewritten history and I'm getting punished for it. What now?
As said, you should really never rewrite history in such synced repositories! The errors you'll see when you do this and Git-hg Mirror can't push are along the lines of "Cannot push non-fastforwardable reference" or "Cannot push because a reference that you are trying to update on the remote contains commits that are not present locally." You'll also get the latter if you have one-way sync between two git repos and in the destination repo you committed to a branch where the source repo also added commits.
Do the following to fix the problem:
- Since the local clone of Git-hg Mirror is now also messed up, first delete the mirroring configuration to get rid of that clone. If you want copy the config's data somewhere, because you'll need to re-create it.
- Fix the repositories by hand, which commonly can be done with force pushes, or as a last resort by re-creating the repos and pushing a known good state.
- Now create the mirroring configuration again. This will do a fresh clone so everything should get back to order.
What should I be careful about when creating a sync configuration?
Be sure to keep these in mind:
- Git-hg Mirror should be able to push to the destination repository (or to both repositories in case of two-way sync), so the credentials you've given in the repository URL should be eligible for write access. Similarly if the source repo is not public Git-hg Mirror should get read access to it. We recommend creating a new, limited user account that has access to the repositories just for the purpose to be used with Git-hg Mirror.
- Only related repositories can be synced, that means two repos that have the same root history. You don't have to think about this in most cases since this only requires that for one-way sync the destination repo be initially empty (don't push anything new to it and don't let e.g. GitHub create the repo with a Readme!). For two-way sync the best is too if in the beginning one of the repos is empty (and then Git-hg Mirror will sync the history from the other repo over).
How are branches handled?
Keep in mind that the equivalent of Git branches in Mercurial is not branches but bookmarks. Thus Git branches will be converted to Mercurial bookmarks and vice-versa. When syncing from Mercurial to Git bookmark names (that will appear as branches in Git) will be suffixed with "-git": this is because there can't be a bookmark with the same name as a branch (the default branch will have a bookmark called "master-git"). You can also do git commits that will be part of hard Mercurial branches if you add the following to the commit message in a new line:
branch : branch-name
But be aware that this should be added to every commit that should go into a branch, i.e. if you open a branch with such a commit, then add a subsequent commit then that should also contain the above text, otherwise hg-git will get mixed up and syncing will completely break!
So in case of two-way sync better have separate branches for Mercurial and Git users and merge between the two! E.g. have a "dev" branch in Mercurial (which will also appear as such in Git) and a "GitHub-dev" in Git (which will appear as a bookmark in Mercurial); then in Mercurial merge from the "GitHub-dev" bookmark into the "dev" branch and in Git merge from the "dev" branch into the "GitHub-dev" branch. Exception is the default branch with the master bookmark: as long as the bookmark is correctly carried in Mercurial too that will work.
In case of one-way mirroring in the target repo be careful not to commit into the common branches/bookmarks either because that's where commits from the source repo will be pushed.
Line endings are kind of mixed up. Why? Halp!
You won't need to care about this when syncing two Mercurial repos.
The thing is that (if you're a Windows user) Mercurial will store the line endings in text files with CRLF line endings (unless you do something fancy, because this is the Windows default). Git, however (if you use the default settings) will store files with the Linux-style LF line ending. This will cause that a Mercurial repo synced from git will contain text files with the LF line ending, which will cause mismatches when you start to edit them in your Mercurial working directory and add CRLF line endings. You can use the Mercurial eol extension to force consistent line endings. Note that when using the eol extension all users of the Mercurial repo should have it enabled prior to cloning the repo! Furthermore after you add the .eol file you should re-clone the repository!
After enabling the extension (it comes with Mercurial) add a .hgeol file to your repository and configure it according to your scenario:
When syncing one-way from Git to Mercurial
** = native
This will store all the files with the LF line endings in the repo (the Git default) by just using the default settings. The patterns section specifies that in the working directory files will use the OS-native line endings, which is CLRF on Windows. All in all this will supply the same experience you'd have when working on a standard Mercurial repo with an all-Windows team.
Note that when merging from a branch of the git repo (i.e. a git bookmark) you'll probably get a lot of "abort: inconsistent newline style in [filename]" errors. If there are only a small number of files you can normalize line endings of such files with an editor like Notepad++ in the local working directory. However if the number of mismatches is huge it's better to do it programmatically; you can use a utility like Swiss File Knife to convert the whole working directory to CRLF: put the exe into the repository root and run the
sfk195.exe addcr . *.* -yes command. Do this for the working directory but also for the branch to be merged from: update to the git bookmark, do the conversion, commit the files that now have changed (normalized) line endings in a temporary branch, then do the merge with the local branch.
Alternatively to resolving such issues yourself you can instruct the eol extension to also convert line endings to the canoncial version even if the line endings in a file aren't consistent. You can unfortunately only configure this for yourself (i.e. it can't be shared in the repo) by adding the following to your mercurial.ini file (or other Mercurial config files):
only-consistent = False
When syncing one-way from Mercurial to Git
No need to care about this scenario if you're an all-Windows team: users of both repos will only see CRLF. Otherwise or if you intend to also commit to the git repo use the .gitattributes config as outlined below.
When syncing two-way between Mercurial and Git
This is kind of complicated. It depends which source control system and which line ending style you started.
But if your team is all-Windows or all-Linux (or at least they can agree on a line ending style) and you started with the LF line ending style (which is the Git default, so most possibly this is the case if the repo was initially Git) you can use this .hgeol:
** = native
This will store files in the repo in LF but use the OS's standard line endings in the working directory. No need to configure anything on the Git side.
If the repo was originally Mercurial and you're a Windows team, i.e. if the repo contains CLRF line endings, then you can either migrate to LF (as above) or continue to use CRLF. For the latter you don't need to configure anything in Mercurial but you need to instruct Git to store files with CRLF. To do this add a .gitattributes file to the repository root with the following content:
This will instruct Git not to do any line ending conversion, i.e. files with CLRF will be stored as CLRF. Be aware that this requires that everyone on the team uses the CRLF line ending in their editors.
I'd like to use Git subrepos in a Mercurial repository or Mercurial submodules in a Git repository. How to do it?
If you want to get the worst of both worlds, it is possible:
- To be able to clone and pull (but not push) Git subrepos in a Mercurial repository you need to enable the hg-git Mercurial extension.
- To be able to clone and pull Mercurial subrepos in a Git repository, with experimental push support, then use git-hg (though the project seems abandoned and lacking support for the latest Mercurial).
Warning: if you start with a Mercurial repo containing subrepos and then sync it to Git then in Git the subrepos (so the .hgsub and .hgsubstate files) will be gone. This also means that syncing that back to Mercurial would create a different history - so if you set up two-way sync starting with a Mercurial repo then you'll get a second, unrelated history in your repository! This can also be preferred (to move to a synced repo setup while keeping the old subrepo-including history), but you may not expect this.
I get some connection-related errors under my configs. What do to?
Your mirroring configurations may fail with errors similar to the one below:
hg pull --rev 296 "url/to/your/repo" pulling from "url/to/your/repo" searching for changes adding changesets adding manifests adding file changes " and error " transaction abort! rollback completed abort: stream ended unexpectedly (got 3 bytes, expected 80)"
This is a connection-related error. This can happen for a variety of reasons, including the Mercurial repo being slow (either due to server load or network issues), Git-hg Mirror being under heavy load (and thus its network being slow). Either way, most possibly this is a temporary issue and you can resolve it by simply clicking "save" to retry syncing your repos. If the issue still perists get in touch with us.
Man, this is awesome, but it could be even more awesome. How can I contribute?
Glad you asked! Since multiple people asked for it Git-hg Mirror is now open source! Check out these repos on GitHub:
- Git-Hg Mirror Daemon: this Windows service does the actual syncing in the background.
- Git-Hg Mirror Common: the frontend of the service, so this is what you click through here.
This is not working for me and I'm unhappy, you ruined my day :-(. Where can I ask for help?
Sorry, sorry, we're really sorry! Drop us a line explaining what two repos are failing for you and we'll take a look. But since we know if something fails probably we're already working on fixing it!