Using git for XWiki development

Git is a free distributed version control system. Among its most important features for us:

• ultra fast
• distributed (P2P) development
• branch-friendly
• full local repository, with the entire history available
• very efficient storage
• tracks file moves very well

1. Using git for XWiki development
   1. Setup
      1. Install git using your package manager of choice
      2. Introduce yourself to git
      3. Configure global ignores
      4. Configure Shell
      5. Create a fork of a target module
      6. Clone the fork repository locally
      7. Fix bug, commit locally
      8. Pushing the changes to your fork
      9. Requesting for the change to be integrated back in the main repository
   2. Working with git
      1. Basic work with git branches and subversion commits
      2. Merging several local revisions into one big commit
      3. Working with the stash
      4. Resplitting several local changes into a few subversion commits
      5. Working with central branches
      6. Inner branches
         1. Checking out remote branches
         2. Merging changes between branches
      7. Different repositories for branches
   3. Helpful git commands
   4. References
   5. Understading git versions
   6. The repository, the index, and the working tree

Setup

The best way of working with XWiki over GitHub is to create an account on GitHub, fork one or more of the modules, and clone your fork locally on your computer. After committing some changes locally, push them to your fork on GitHub, and make a pull request to have the changes merged back into the official repository. Committers can clone directly the main repository and push changes to it, although that's not recommended.

Install git using your package manager of choice

Introduce yourself to git

This is also nice for command-line geeks:

Alternatively you can edit /.gitconfig (notice the nice aliases at the end):

Configure global ignores

Create a file named .gitignores, put it in your home folder (or anywhere else, but remember its path):

Then execute:

Configure Shell

Optionally you can configure your shell to display the current branch you're in (for ex by putting the following in your .bash_profile):

Create a fork of a target module

If you don't have an account on GitHub yet, register.

Visit the XWiki homepage and search for the right repository. For example, start typing admini to find the xwiki-extension-administration repository. Open it, and click on the fork button

Now you've forked the project into your very own repository, where you can commit your changes and make them available publicly.

Clone the fork repository locally

To actually be able to change files, you'll have to clone the repository locally, on your computer. To do this, open your fork repository (you should be already there if you just did the previous step, forking). This should look like https://github.com/<your username>/xwiki-extension-administration. There you should see the git repository URL which you should clone:

Copy this URL, open a console, go to your work directory and execute:

Fix bug, commit locally

Now edit the code, build, test, until the bug is fixed, or the new feature is implemented. You should have some changes to commit now. To check that, execute:

Or the shorter summary:

Now you can stage the changes, or just a subset of them, preparing a commit. The notion of "staging" will be explained later.

You can either add all the files directly:

Or you can pick which changes to commit interactively (useful when trying to split a larger changeset into smaller commits):

Now you can commit; execute the following command, and enter the commit message when prompted:

Pushing the changes to your fork

Congratulations, now you've got the bug fixed, but the commit is only on your computer, in the local repository. You need to push it to GitHub, in your own fork of the project. Since you got the local repository by cloning, it should already know what its origin is, and you should be able to push directly:

Now the commit should be visible on GitHub:

Requesting for the change to be integrated back in the main repository

The change is now publicly accessible, but it's not yet in the official distribution. You should request for it to be integrated back in the main repository by one of the core committers.

Now you can send a message to the committers, change which branches to be merged, which commits to send, and you can review the changes once again.

Committers will be notified of the request, and will be able to review the request:

Then the can merge your changes from their own local clone:

And now the pull request is automatically closed:

And the commit is part of the official repository:

More information about pull request on the official GitHub documentation.

Working with git

Subversion teaches us to work with one trunk, and to hate branches as much as possible, since it is always hard to merge changes. Git gives a different meaning to branches. A branch is not something hard to maintain, but something that should be easy to use. With git, short-lived branches should be created daily. This is a good thing to do since git by default works with the local repository, so all the branches are inside the same folder, on your disk.

In subversion, when working on a small issue, it is easy to:

• svn up (and have a clean up to date codebase)
• do a little changes (several minutes)
• make a quick build to make sure everything works
• svn commit
• pick another issue and start all over

The problem is that not all issues require this little work. I for one often find myself working on two or three long issues for several days in parallel, while fixing a few other minor issues inbetween. The problem is that it's always hard to separate commits and make a build just with the changes for one issue, so that work in progress on another unrelated issue doesn't break the build. When using subversion, I used to have two checkouts, one for work in progress, and a clean one for commits, and after fixing an issue, I would have to selectively copy files (or even just code blocks from files with changes from multiple issues) from the dirty to the clean repository. With git I can have a branch for every issue, each with just its changes, and easily switch between branches. Unlike subversion branches, which take a lot of space, git branches only require enough space to store the differences between the branch and the trunk, which is usually a few kilobytes. Let's take an example.

Basic work with git branches and subversion commits

Assume I want to fix the issue XWIKI-123. I start by updating the local repository:

I then create a branch for this issue:

git branch just creates the branch, while git checkout actually switches to it. git checkout -b both creates and switches to a new branch. I then start coding on it, building, coding, and so on. Then I get an urgent thing to debug, so I need to switch back to the master branch, without any of my changes. Since unaccounted changes don't belong to any branch, I must first bind the current files to the branch, then switch to the clean master branch:

-a means "commit all the changed files", while -m is the commit message. Any commit should have a meaningful commit message.

I then go on debugging, making changes, building and back again, until I spot the problem, which proves to be a bit hard to fix at the moment. So, I create a new issue, XWIKI-234, and move my local changes to a new branch, and leave it for a later fix (since I don't want all the debug code to hinder my development, and I don't want to drop any changes either):

When changing the local branch, any uncommitted changes will be moved to the new branch also. git add . adds all the changes in the local (and descendant) directories to the index. git commit commits the index to a new revision. Checking out the master branch will leave all the committed changes just to the XWIKI-234 branch, so the master branch will now have only the clean files. More about the index later.

Since I spent a lot of time on XWIKI-234, I am too tired to continue working on XWIKI-123, but I feel that I can take on a smaller issue, like XWIKI-100. So, I create a new branch, work on it, quickly commit, then delete the branch.

First I (1) update the local repository with the latest changes from the server, since others have been busy, too. Then, I (2) create a new branch for the issue, and switch to it. After I make the necessary changes, I (3) check the changes made to see what I modified, and to make sure that I didn't leave any debug code or garbage, I didn't introduce an obvious bug, or messed the codestyle. Then I (4) stage the changes to the index, for a commit, and (5) commit to the local repository. Nothing is sent to GitHub yet. I (6) switch to the main (master) branch, which is the usual bridge between the local and remote repositories, I (7) merge the change into the master branch, (8) remove the no longer useful topic branch, and finally (9) push the commit to the server.

So I go home, and when I come the next morning, I can check which branches are already open and continue working on one of them, or create other branches for other issues:

The first commands prints the current branch names: master, XWIKI-123 and XWIKI-234. The second changes to another (existing) branch. The third command pulls updates from the main branch, rebasing my previous work on top of the latest codebase. The fourth command resets the last commit on this branch, since it was a temporary one in order not to loose changes ("Work in progress", remember?).

Users that are not committers and want to add their contribution can create a pull request and select the XWIKI-100 branch as the source:
(1) git pull
(2) git checkout -b XWIKI-100
<modify files, build, ready to commit>
(3) git diff
(4) git add .
(5) git commit
(6) git push origin XWIKI-100:XWIKI-100
(7) git checkout master
Every new contribution should have it's own branch and no changes should be directly committed on the master branch because they will be listed when creating a pull request (for non committers).

Merging several local revisions into one big commit

Since git has a local repository, you can commit as many times as you want, so that you can have all versions of a file changed many times, with different algorithms and different approaches. The problem is that by default, git merge and git pull will keep each individual commit, which might be too much once you find the right solution, and don't want all the intermediary stages to be public. To workaround this problem, you can do this:

After (1) creating a new branch, edit and (2) commit changes to local revisions. At the end, (3) reset to the original version of the master branch, keeping the files and the index to their current status, meaning that the local repository will not have any additional commits compared to the central server, but all the changes will be staged for the following commit. After (4) committing locally, only one revision will be visible in this branch from now on.

Working with the stash

In subversion, changes can be committed, reverted, or saved into a patch. Apart from these options, git also allows changes to be stashed away, to be applied at a later moment. Think of the stash as a special virtual place for patches.

Assume I want to commit something, but I have some minor local changes that I don't want to revert, nor to commit, and creating a branch just for those is not worth the effort. The solution is simple: use the stash.

When trying (1) to push the changes to the central repository, git complains that there are local changes. (2) stash these changes away, so that the (3) commit succeeds. After that, the changes are (4) re-added to the current directory.

Resplitting several local changes into a few subversion commits

A more interesting use of the stash is this: Like the scenario above with merging several git revisions into one subversion commit, assume I want to create not one subversion commit, but several, out of a lot more git commits.

So, after a busy day of work, I (1) check what will get committed to the central server. 6 commits is a little too much for an issue, so I decide to trim it down. I (2) check each individual change to see what happened in there, and I decide that the first 3 revisions should become the first commit, the following 2 should become the second commit, and the last revision should become the third commit. So, I (3) check one last time that the grouped revisions should indeed become a single commit, then start preparing the split. From the top of the commit log, I (4) undo commits and (5) stash them away. git reset, a shortcut for git reset --mixed, keeps the files in the working directory as they are, but resets the index. Revisions can either be specified as the full SHA identifier (the long number), the most significant part of it (the short number, enough symbols to uniquely determine the revision, may be even as short as 2 or 3 symbols for a young repository), or relative to the current head (HEAD~ is the previous version, or the father, HEAD~~ or HEAD~2 is the grandfather, and so on). After stashing the 2 changesets, I (6) reset to the initial version, and start re-creating commits. Since the first 3 revisions were not stashed away, they are still in the working directory, so I can (7) commit it directly. Then, I (8) unstash each of the two stashed changes, and (9) commit them. Now, I can (10) check that indeed there are only 3 commits to push now, and (11) finally commit them to the central repository. This seems like a lot of work, but if we want to have a clean, meaningful history, it's worth the effort.

Working with central branches

When a version enters in the RC phase, it is branched (in the subversion meaning), so most changes will have to be integrated in two places, the trunk and a branch.

Inner branches

This means that the git repository tracks not just one remote trunk, but the trunk along with all the branches and tags. Switching between remote branches and mergin is easy, and it's all done in the same repository.

Checking out remote branches

If you cloned a root project URL, like git svn clone --username=jdoe --stdlayout https://svn.xwiki.org/svnroot/xwiki/platform/web/, then all the branches are already available. To check this, execute: git branch -r, which lists all the remote branches. To create a git branch tracking a subversion branch, do:

This creates a new git branch called "1.8", tracking the remote branch xwiki-web-1.8 (assuming the command is executed in a checkout of the web module). Since in subversion the difference between tags, branches and trunk is just a naming convention, all branches being equal, you can also create git branches tracking svn tags:

To make sure the local branch is tracking the correct remote branch, execute:

and check the URL:

Or simply:

To list all the available remote branches, execute:

Notice the old branches which no longer exist, like xwiki-web-1.6, and the virtual branches like trunk@3685, which are triggered by changes in the URL, when reorganizing the repository. You can prune these using this nice collection of bash scripts: http://github.com/offbytwo/git-svn-extensions/tree/master

Merging changes between branches

Assuming you have a master branch for the trunk, and a 1.9 branch for the xwiki-core-1.9 remote branch. The way I do it:

So, after making some changes, I (1) commit to the local repository and (2) to the remote repository, and remember the (svn) version number for the new commit. I (3) switch to the local 1.9 branch, which tracks the remote branch for 1.9, and merge the last commit from the master branch to the current (1.9) branch. git cherry-pick can also accept a different commit to merge, so if more than one commit must be merged, copy their numbers from git log master and use git cherry-pick 123abc456 to merge each such commit. Of course, if you've been nice and worked on a different branch, just replace master with the actual branch name.

Different repositories for branches

This means that you checked out just the trunk in a repository, and just a branch in another repository. You can either communicate using patches, or configure a connection between the two repositories.

[Comming soon]

Helpful git commands

git gui is a visual tool for working with git. It can be used to perform most common actions: manage branches, view history, commit, etc.

gitk is a nice visual tool for browsing the repository.

git citool is a visual tool for committing (to the local repository, not to SVN).

The manual pages are pretty nice, try man git-reset, man git-commit, man git-svn and others.

git reset goes back to a different version. It can work in different modes:

• git reset --soft is good for reverting a commit, but leaves files to their current state, and also keeps the index ready for a commit. In other words, it just resets the version number.
• git reset or git reset --mixed also resets the index.
• git reset --hard also reverts any uncommitted file changes. In other words, it resets both the index and the files. This completely looses changes, so be sure nothing important will be lost.

git log shows the full history. It can also show the history for just one file, and can show the log in many different formats, from one line summaries to full diffs.

git merge allows to merge two branches.

git cherry shows the differences between two branches, useful for detecting unmerged changes between the trunk and a branch.

git fetch allows to fetch changes from other repositories, either local (good when having different git folders for the trunk and a branch) or from other people (this is where the distributed aspect of git comes into play).

git config --global alias.st status, git config --global alias.ci commit and similar aliases allow to create shorter names for common commands.

git-svn-extensions provides a few useful methods, for creating and deleting remote tags and branches, for pruning defunct references from the local repository, for listing existing tags/branches and finding out the current local or remote branch.

References

There are lots of tutorials and documentation pages on the net:

• http://www.kernel.org/pub/software/scm/git/docs/everyday.html
• http://git-scm.com/course/svn.html
• http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html

Cheat sheets:

• http://refcardz.dzone.com/assets/download/refcard/f1f9176b6f1f4bd003841f13772c2426/rc094-010d-git_0.pdf
• http://jan-krueger.net/development/git-cheat-sheet-extended-edition

Strategies:

• http://nvie.com/posts/a-successful-git-branching-model/

Understading git versions

Subversion, being centralized, keeps a continuous incremental revision number, git uses SHA hashes as the version identifier, which allows to have non-linear histories, non-linear (not even tree-like) branches, and also increases the security. While these long strings can seem a bit hard to use at first, we live in a world where copy/paste is available everywhere (even on the iPhone), so it's not needed anymore to remember version numbers. Also, it is enough to enter just the first distinctive characters, enough to uniquely distinguish the revision. It's also possible to define aliases for revisions, and such aliases are available already: HEAD is the most recent committed version, trunk and xwiki-web-1.9 and tags/xwiki-web-1.8.4 are aliases for remote branches, relative references can be obtained with ~ and ^, like HEAD~ for the previous version and trunk~6 for 6 versions ago on trunk, and even @{5 minutes ago} for the commit from five minutes ago (this one doesn't work with all commands).

The repository, the index, and the working tree

The git repository is a local database, which doesn't contain any direct files. You can't find a file directly in there, like pom.xml, although you can extract a version of it using git commands. git doesn't work with files, but with content, so it can track a piece of text moved from one file to another, file renames, file copies, and so on. This database holds the entire history of the code, unlike a svn checkout, which only holds the most recent snapshot of the files.

Then, you checkout branches from different snapshots of the repository, into a working tree. There is only one working tree at a time (with a normal usage of git), the one currently found in the filesystem. This working tree is not part of the repository, it is just a view of data inside it, so you can do anything with it, nobody else will see these changes when communicating with your repository. If you want to push some of the changes into the repository, you must stage them into the index. The index is a temporary changeset sitting somewhere between the working tree and the repository, and this is what gets committed. Remember, git works with changes, not with files, so it can't commit files from the working tree.

So, to translate into commands:

• git checkout extracts a certain snapshot from the repository into files into the working directory.
• git add extracts changes between the working tree and the original repository snapshot into a staged commit.
• git commit makes these changes permanent, adding them to the repository and creating a new snapshot.