One of the many things I enjoy taking care of as a sysadmin are the SVN repositories at our Bangalore office. These are large repositories, and have three known mirrors, one NFS share where each commit is incrementally dumped, as well as a secret mirror that no one else knows about ;)
Here are the steps that I follow when setting up an SVN mirror. I'll later post information on problems that one faces when setting up and maintaining SVN mirrors.
Very, Very, Important: When committing to an SVN mirror, you need to use TortoiseSVN 1.5.0 and no later. Any later revision may work, but may break during certain tasks. There is no such problem when committing to the main SVN server.
How SVN mirroring works (in a nutshell):
Users connect to mirror servers. mod_dav_svn serves content from the local system, and sends commits to the main server. The main server's post-commit scripts use svnsync to push commits to the mirrors via a protected URL that is writable only from the main server.
My instructions include integrating with LDAP on an Active Directory server.
Install CentOS 5.2 (I've tried FC8 and Ubuntu, and they have some wierd issue or the other with SVN mirroring. Nothing that I can definitely point out).
Uninstall all SVN the comes with the OS
Install Collabnet SVN client binaries
Install Collabnet SVN server binaries
symlink /opt/CollabNet_Subversion/bin/ into /usr/bin
Include the following httpd fragment on the main server's httpd.conf file. You could Include another file containing the following, too. This is what I do.
LoadModule dav_svn_module /etc/httpd/modules/mod_dav_svn.so
LoadModule authz_svn_module /etc/httpd/modules/mod_authz_svn.so
<Location /someproject>
DAV svn
SVNPath /repos/svn/repos/someproject
AuthzSVNAccessFile /repos/svn/access/someproject/svn_access.conf
AuthType Basic
AuthName "Active Directory LDAP Authentication"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPBindDN user@adserver.thoughtworks.com
AuthLDAPBindPassword somePassword
AuthLDAPURL "ldap://adserver.thoughtworks.com:389/ou=Principal,dc=Corp orate,dc=thoughtworks,dc=com?sAMAccountName?sub?(&(objectClass=user))"
require vaild-user
SVNPathAuthz off
</location>
/repos/svn/access/someproject/svn_access.conf
-------------------------------------------------------------
can_write_group=aduserA, aduserB,aduserC
read_only_group=aduserD,aduserE,aduserF
no_access_group=aduserG,aduserH,aduserJ
[repository:/]
@can_write_group=rw
@read_only_group=r
@no_access_group=
Create a repository as follows:
svnadmin create /repos/svn/repos/someproject
change permissions as follows
chmod -R g+w /repos/svn/repos/someproject
chown -R apache.apache /repos/svn/repos/someproject
symlink the collablnet svn modules
ln -s /opt/CollabNet_Subversion/modules/mod_dav_svn.so /etc/httpd/modules/mod_dav_svn.so
ln -s /opt/CollabNet_Subversion/modules/mod_authz_svn.so authz_svn_module /etc/httpd/modules/mod_authz_svn.so
reload httpd
service httpd reload
On the mirror server:
Follow instructions as above.
Modify the httpd Location to become
DAV svn
SVNPath /repos/svn/repos/someproject
SVNMasterURI
http://svnserver.thoughtworks.com/someproject AuthzSVNAccessFile /repos/svn/access/someproject/svn_access.conf
Also include the following snippet:
<location /someproject-sync>
DAV svn
SVNPath /repos/svn/repos/someproject
Order Deny,Allow
Deny from all
Allow from svnserver.thoughtworks.com
</location>
cat > /repos/svn/repos/someproject/hooks/pre-revprop-change << EOF
#!/bin/bash
exit 0
EOF
chmod +x /repos/s
Reload the httpd server
service httpd reload
Now, back on the main svn server
- initialize the SVN mirror on the mirror server
svnsync init
http://svnmirror.thoughtworks.com/someproject-sync http://svnserver.thoughtworks.com/someprojectEdit the post-commit hook to contain the following:
svnsync sync
http://svnmirror.thoughtworks.com/someproject-sync --source-username read_only_user --source-password valid_password
You can now either let the next post-commit update the mirror at one shot, or transport the data yourself and then update the mirror's metadata.
To populate the mirror automatically, simply commit to the main server once.
Populating the SVN mirror manually to seed it with data is the preferred approach, especially when the mirror is at a remote location.
Note down the properties of the mirror server
svn pl --revprop -r 0
http://svnmirror.thoughtworks.com/someproject-syncUnversioned properties on revision 0:
svn:sync-from-uuid
svn:sync-last-merged-rev
svn:date
svn:sync-from-url
Get each property:
svn pg svn:sync-from-uuid --revprop -r 0
http://svnmirror.thoughtworks.com/someproject-syncsvn pg svn:sync-last-merged-rev --revprop -r 0
http://svnmirror.thoughtworks.com/someproject-syncsvn pg svn:date --revprop -r 0
http://svnmirror.thoughtworks.com/someproject-syncsvn pg svn:sync-from-url --revprop -r 0
http://svnmirror.thoughtworks.com/someproject-sync1. Export the svn data
Assuming the latest revision is revision 32478 (an excellent case for populating data manually)
svnadmin dump /repos/svn/repos/someproject -r 1:32478 --incremental | gzip -9 > someproject_1_32748.gz
2. Transfer the file to the mirror server
scp someproject_1_32748.gz user@svnmirror.thoughtworks.com:
On the remote server:
3. Extract the archive file
gunzip someproject_1_32748.gz
4. Import the data into the repository
svnadmin load /repos/svn/repos/someproject < someproject_1_32748
Back on the main server
5. update the mirror repository's sync-last-merged-rev property
svn ps svn:sync-last-merged-rev 32748 --revprop -r 0
http://svnmirror.thoughtworks.com/someproject-sync6. Ensure that everything's fine
svnsync sync
http://svnmirror.thoughtworks.com/someproject-syncIf you face any problems, reply to this post and I'll see how I can help with my own experiences on solving those problems :)
Once I get SVN 1.5 based mirroring to work on Solaris 10, I'll post about that too. In my opinion, Solaris 10 + ZFS should be the best bet for SVN servers and mirrors, because there are some problems that even having a mirror will not help you solve quickly.
Some such problems are:
- what if the ext3 file system goes corrupt (I just dealt with a corrupted ext3 file system on an RHEL 5 box this weekend).
- what if someone makes a dangerous commit and you need to remove that commit before anyone else checks it out ?
one example would be checking in a massive delete/folder structure by mistake
another example would be checking in an infected file
yet another example would be checking in installers into the repository.
If you need to roll back quickly, you can do so if you have ZFS snapshots per post-commit, but not with other filesystems. With other file systems, you'd need to go through the painful steps of dumping all the commits, importing in data upto (but not including) the poison commit, and then pointing users to that repository.
A alternative would be to have another repository which is updated once every twenty commits or so. You would then have the opportunity to selectively push in commits and to then drop the most recent bad commit.
The best would of course be to use Mercurial and to ditch SVN altogether. This is not always feasible, though.
I'm told that there aren't adequate tooling to support Mercurial in Windows as much as there are for SVN. Next, our customers may be averse to moving to (and asking their other vendors to move to) a new tool (we should ask, though).