Ham is a Geek: January 2006

Distributed version control systems seem to be the fad these days. They are popular for good reason. These new breed of version control systems give developers the flexibility and ease of use that the traditional Concurrent Version control System (CVS) does not. While there have been some talk in the openacs community about Arch and many more whom I've had the privilege of chatting with praise its features, there doesn't seem to be any plans to make the switch and for good reason. I had the time and opportunity recently to review some distributed version control systems including Arch but found that it's not as easy as it looks, at least in the beginning. There's a learning curve and the habits that you form with prolonged use of CVS can be hard to break.

This article, however, won't be about Arch. It's about another distributed control system that is based on Subversion, it's called SVK.

When I set out on my search for "THE" version control system, I had a set of criteria in mind :

It has to be DISTRIBUTED. Although I said it's a fad, I think the advantages of a distributed version control system specially in a project which requires heavy collaboration with geographically diverse developers will make it a necessity as soon as people start shaking off CVS.

It has to be FLEXIBLE. I like to be able to pull bug fixes and changes from OpenACS CVS to several projects and vice versa.

It has to be EASY TO USE and LEARN. I want it to work like CVS minus the headaches.

It has to be PORTABLE. I want to be able to work on a project without necessarily being connected all the time to the internet. I can be mobile and bring a copy of the repository with me on my laptop and merge the work I did on it offline to the main repository later.

It has to SCALE. Scale here does not only mean that it should be able to handle a lot of developers but also be easy to use and maintain, even as the number of projects using the same code base increases. In short the effort involved in getting changes from the central repository to the mirrors and back should not increase too much as the number of projects increase.

The Scenario

In the succeeding article we will be creating a product called ecwork based on OpenACS. Its starting point will be OpenACS 5.1 stable. We want to create new packages and modify existing packages to create this new product. At the same time we want to contribute code back to the OpenACS community from lessons learned, code produced and from experiences developing ecwork.

Create a Central Repository

I used the term "Central Repository" for a lack of a better term to describe the father tree or the first repository from which other repositories will be mirrored.

Here's how to create an SVK repository for our ecwork product.

Checkout a copy of OpenACS. Change the revision (-r) depending on which branch you wish to work from or commit code to later on.

cvs -d :ext:cvs.openacs.org:/cvsroot co -r oacs-5-1 openacs-4

Create an SVK repository

svk depotmap --init

Relocate the repository to a more obvious less intrusive location

svk --relocate ~/.svk/local /svn/repos/svklocal

Import the OpenACS checkout into the repository

svk import -m "import openacs" //ecwork/trunk openacs-4/

SVK actually uses subversion underneath. So creating an SVK repository also creates a subversion repository. In short you can use Subversion commands with this central repository. You, therefore, have the option to use either a distributed version control with SVK, or central version control with Subversion.

New Project, New Mirror

When a new project comes along where we would like to use the same code base, we create a separate staging server and mirror the central repository there

In the staging server, create the local repository.

su - service_name
svk depotmap --init
svk --relocate ~/.svk/local /var/svk

Create a mirror of the remote repository (aka the central repository). There are several ways to access a remote repository, the most common is to use apache and expose the repository in a url. Alternatively a repository can also be accessed using svn+ssh but it is advised that users use public key authentication with ssh-agent caching their passphrase.

svk mirror http://ham.is-a-geek.com/svn/repos/svklocal/ecwork/trunk //ecwork-service_name/trunk

Verify if the mirror has been created

svk mirror --list

Sync up the mirror. Creating the mirror just creates a map between the local repository and the remote (central repository). Executing sync performs the actual replication or mirroring.

svk sync //ecwork-service_name/trunk

Create a local branch from the mirror

svk cp -m 'create local branch' //ecwork-service_name/trunk //ecwork-service_name/local

Checkout a copy from the local branch

svk checkout //ecwork-service_name/local /home/service_name/openacs

Configure your checkout to ignore CVS. We will discuss how to commit to CVS later.

cd openacs
svk -R ps svn:ignore 'CVS'

Use the checked out copy as your serverroot, modify your config.tcl accordingly.

A local branch is created so that developers who are familiar with svn can use svn commands or a GUI like TortoiseSVN to perform commits and updates on this local branch. See below how SVK makes it easy to merge the changes from this local branch back to the mirror repository and back to the central repository.

If the repository is accessible only via ssh here's the alternative way to create a mirror of the repository :

svk mirror svn+ssh://ham@ham.is-a-geek.com/svn/repos/svklocal/ecwork/trunk //ecwork-service_name/trunk

Notice that the URL (http://ham.is-a-geek.com/svn/repos/svklocal/ecwork/trunk) is substituted with the following notation

svn+ssh://username@hostname/repository_location

Types of Developers

Before anything else you should create your own local repository where you will be able to mirror the central repository if necessary. It's exactly the same steps as we did when we created the central repository. Here's a quick run down :

svk depotmap --init
svk --relocate ~/.svk/local /var/svk

You should now have an SVK repository in /var/svk. You can choose to relocate your repository elsewhere if you wish.

As a developer there are many ways to go about with development given the this setup. Let's enumerate first and then tackle them individually.

Type	Checkout from Repository/Branch	Description

Core Developer 2	Mirror Repository	Checkout is made directly from the mirror instead of a local branch. Changes here automatically are merged to the repository as soon as a commit is performed on the working copy.
Project Developer	Mirror Repository/Local Branch	If you are restricted to developing only on one project and it is likely that the changes you make will affect only that project. This does not mean, though that changes made by the project developer will not find its way to the central repository.
Offline Project Developer	Mirror Repository/Local Branch	You can mirror the central repository or mirror another mirror (from another project based on the central repository) into a local development box and disconnect from the internet.

Core Developer 1

Your working copy is a checkout from the central repository.
You can use Subversion GUI's like TortoiseSVN
Your changes are immediately available to all mirrored repositories as soon as a "svk sync" is executed on the mirrors
You need to be online all the time and have access to the server where central repository is housed to be able to commit and update your working copy
No need to perform svk sync as you are working from the central repository already

To checkout a working copy directly from the central repository :

svk checkout http://ham.is-a-geek.com/svn/repos/svklocal/ecwork/trunk ecwork

Core Developer 2

Your working copy is a checkout from one of the mirror repositories
You CAN NOT use subversion GUI's like TortoiseSVN because this breaks the mirror
Your changes are immediately merged back to the main repository on commit
You need to be online all the time and have access to the server where the mirror repository is housed to be able to commit and update your working copy
You still need to do svk sync to retrieve the changes made in the central repository to the mirror you are currently working on and vice versa

To checkout a copy from the mirror, ensure that the mirror also exposes its repository either via URL or SVN+SSH.

svk checkout http://service_name/var/svk/ecwork-service_name/trunk ecwork-service_name

Project Developer

Your working copy is a checkout of a local branch created from a mirror
You can use subversion GUI's like TortoiseSVN
Commits stay in the local branch until an smerge is done between mirror repository and local branch
You need to be online all the time and have access to the server where the mirror repository and local branch are housed unless both are in your local dev machine
You need to run svk sync to retrieve changes from the central repository

To checkout a copy from the local branch of a mirror repository

svk checkout http://service_name/var/svk/ecwork-service_name/local ecwork-service_name

Regular Version Control

At this point you should now have a checkout (working copy) of the source from any of the following sources.

Central Repository
Mirror Repository (project repository)
Local Branch (local branch of a project repository)

Svk, like all version control systems, has commands to add, remove, update and commit files to the repository. The syntax is very similar to CVS and Subversion so a developer who is familiar with any of the two will have no problems with these common tasks.

Add a File

To add a file into source control, change directory (cd) into your working copy and perform the following on a file that you want to add

svk add file_name.html

Remove a File

To remove a file from source control, change directory (cd) into your working copy and perform the following on a file you want to remove

svk rm file_name.html

Check Status of Files

In CVS, we usually perform a 'cvs -n update' to determine which files have been modified and which files have not been added to source control. In subversion and svk there is a separate command called 'status' that does this. So for SVK, if you want to determine the status of files. This can be executed on a filename, directory or the entire working copy.

svk status file_name.html

Update Working Copy

This is SVK's equivalent to cvs update.This can be executed on a filename, directory or the entire working copy.

svk up working_copy

View Logs

To view the associated logs of a file, similar to cvs log. This can be executed on a filename, directory or the entire working copy.

svk log file_name.html

Diff

This is the equivalent of cvs diff. This can be executed between files in a working copy, between two repositories or between a repository and a working copy.

svk diff file_name1.html file_name2.thml

Commit

This is the equivalent of cvs commit. This can be executed on a filename, directory or the entire working copy.

svk commit -m 'your comment here'

Getting Help

To find out more about the svk commands

svk help commands

To find out more about a specific command

svk help svk_command

Mirrors and Merging

This section is more for the benefit of Core Developer 2, Project Developer and Offline Project Developer. Core Developer 1 will only need to do an svk update to move changes from his/her working copy back to the central repository. For the rest, it is necessary to pull chanages from the central repository and push back changes from the mirror repository. This can be done selectively by using svk push or svk pull and automatically by using svk sync. However, if you are working as a Project Developer where you commit changes to a local branch, before you start moving stuff from the mirror repository to the central repository you need to merge the changes in the local branch to the mirrored repository first. Below are some typical scenarios related to repository mirroring and branch merging and how we solve them using SVK.

Branching in CVS has always been a night mare specially with code that have already been merged in but are getting remerged causing conflicts. With SVK and many other distriuted verson control systems that allow "cherry-picking" or "star-merge", earlier merges are noted and SVK remembers them all so that you don't get conflicts as a result of code already in the repository.

Scenario 1 : Pulling Updates from the Remote Repository

For Core Developer 2

Note that Core Developer 2 works on a checkout directly from the mirror. Simply do an svk sync to pull changes from the remote repository to the locally mirrored repository and then svk update your working copy

svk sync //ecwork-service_name/trunk
cd working_copy
svk update

For Project Developer

Note the Project Developer works on a checkout from a local branch created from the mirrored repository. You must first sync changes between the central and locally mirrored repository by doing an svk sync. Once the mirrored repository is updated you now must merge changes from the mirror to the local branch and then svk update the working copy

svk sync //ecwork-service_name/trunk
svk smerge -C //ecwork-service_name/trunk //ecwork-service_name/local # check if there are conflicts
svk smerge -l
//ecwork-service_name/trunk //ecwork-service_name/local # do the merge
cd working_copy
svk update

Scenario 2 : Pushing Local updates and changes to the Remote Repository

For Core Developer 2

Changes that Core Developer 2 has committed are automatically pushed to the central repository.

cd working_copy
svk commit -m 'your comments'

For Project Developer

Note that the Project Developer works on a checkout from a local branch created from the mirrored repository. Ensure that your working copy is up to date and you have committed all the files that need to be committed to the local branch. Then perform a sync to ensure that we have all the changes from the remote repository. Next do the merge to move changes from the local branch to the mirrored repository. Once the changes are in SVK will automatically push them for you to the central (remote repository).

cd working_copy
svk update
svk sync //ecwork-service_name/trunk
svk smerge -C //ecwork-service_name/local //ecwork-service_name/trunk #check if there are conflicts
svk smerge -l
//ecwork-service_name/local //ecwork-service_name/trunk #do the merge

General Workflow

The image below best illustrates the workflow of a typical Project Developer. An smerge from the local branch to the localy mirrored repository automatically pushes the changes back to the central (remote) repository.

Image courtesy of http://www.bieberlabs.com/wordpress/archives/2005/03/19/explaining-the-svk-workflow

The Big Picture

The image shows how svk works in a single code base multi-project set up. Project 1 uses a centralized method where developers checkout directly from the mirror or a local branch of the mirror. Project 2 uses a decentralized method where developers mirror the local branch of project 2 in their development envrionment.