subversion

We have experienced some abuse of our subversion repository at work recently. Someone committed 400 MB of data all at once including many object files, libraries, and executables. I did not get very harsh with the person who did this. Because a) I have no objection to binaries in subversion in the first place, b) I don't really know what he's working on, c) disk space is cheap and we are no where near capacity, and d) his commit was still smaller than a few commits we had a long time ago (which were legit). Still, if you just allow people to commit whatever they want to your subversion repository, in the worst case, you could run out of disk space, necessitating an svn dump-and-load onto a new larger drive (pain). It would suck to have to do that just because some people were committing large binaries (without any legitimate reason to). There are other annoying consequences. Our tarball backups of svn currently fit on a DVD, which is cheap and easy, if we allowed this abuse to continue it would complicate our backup process.

What I wanted was a way to limit the commit size for certain users automatically. There did not seem to be any hooks out there to do this, so I wrote one.

Just paste the following into your pre-commit hook:

/svn/repos/hooks/check_txn_size.py "$REPOS" "$TXN" || exit 1

and paste the following into a check_txn_size.py file in your hooks directory and make it executable.

#!/usr/bin/env python
import sys,os,popen2
 
MAX_BYTES = 1024000
DEBUG = False
SVNLOOK = '/usr/bin/svnlook'
ALLOWED_USERS = ['david', 'ctang', 'vjain', 'mike', 'sbridges', 'tcirip']
ADMIN_EMAIL = '<a href="mailto:admin@company.com">admin@company.com</a>'
 
def printUsage():
  sys.stderr.write('Usage: %s "$REPOS" "$TXN" ' % sys.argv[0])
 
def getTransactionSize(repos, txn):
  txnRevPath = repos+'/db/transactions'+'/'+txn+'.txn'+'/rev'
  return os.stat(txnRevPath)[6]
 
def printDebugInfo(repos, txn):
  for root, dirs, files in os.walk(repos+'/db/transactions', topdown=False):
    sys.stderr.write(root+", filesize="+str(os.stat(root)[6])+"\n\n")
    for name in files:
      sys.stderr.write(name+", filesize="+str(os.stat(root+'/'+name)[6])+"\n")
 
def checkTransactionSize(repos, txn):
  size = getTransactionSize(repos, txn)
  if (size > MAX_BYTES):
    sys.stderr.write("Sorry, you are trying to commit %d bytes, which is larger than the limit of %d.\n" % (size, MAX_BYTES))
    sys.stderr.write("If you think you have a good reason to, email %s and ask for permission." % (ADMIN_EMAIL))
    sys.exit(1)
 
def getUser(repos, txn):
  cmd = SVNLOOK + " author " + repos + " -t " + txn
  out, x, y = popen2.popen3(cmd)
  cmd_out = out.readlines()
  return cmd_out[0][:-1]
 
if __name__ == "__main__":
  #Check that we got a repos and transaction with this script
  if len(sys.argv) != 3:
    printUsage()
    sys.exit(2)
  else:
    repos = sys.argv[1]
    txn = sys.argv[2]
 
  if DEBUG: printDebugInfo(repos, txn)
  user=getUser(repos, txn)
  if DEBUG: sys.stderr.write("User:"+user)
  if DEBUG:
    if (user in ALLOWED_USERS):
      sys.stderr.write(user+" in allowed users")
    else:
      sys.stderr.write(user+" not in allowed users")
  if (user not in ALLOWED_USERS):
    checkTransactionSize(repos, txn)

Here is how I upgrade my Drupal sites (which have their own customizations) to new versions of drupal core when they become available, as I mentioned in a previous post.

This section of the subversion book pretty much explains it. Here is specifically what I did for Drupal, assuming you already have some directories set up as follows:

/var/svn/repos/www/vendor/drupalcore
/var/svn/repos/www/vendor/drupalcore/4.7.2
/var/svn/repos/www/vendor/drupalcore/4.7.3
/var/svn/repos/www/vendor/drupalcore/4.7.4
/var/svn/repos/www/vendor/drupalcore/current
/var/svn/repos/www/drupal4.7

The /var/svn/repos/www/drupal4.7 is like my own drupal4.7 "trunk". The /var/svn/repos/www/vendor/drupalcore is where I track all the vendor sources.

Download the new drupal sources:

cd ~/downloads
wget <a href="http://ftp.osuosl.org/pub/drupal/files/projects/drupal-4.7.5.tar.gz

">http://ftp.osuosl.org/pub/drupal/files/projects/drupal-4.7.5.tar.gz
[/ge...
Run svn_load_dirs to load the new sources into current and create a tag of current called 4.7.5:

svn_load_dirs file:///var/svn/repos/www/vendor/drupalcore current ~/downloads/drupal-4.7.5 -t 4.7.5

Now go to a checked-out copy of /var/svn/repos/www/drupal4.7

cd /var/www/localhost/htdocs/sandbox
svn st
svn ci -m "commit some uncommitted changes that I had (might as well get these in)"

Merge in the changes between 4.7.4 and 4.7.5. This is the magic (almost as magical as svn-load-dirs):

svn merge file:///var/svn/repos/www/vendor/drupalcore/4.7.4 file:///var/svn/repos/www/vendor/drupalcore/current .

Now you will see a bunch of output similar to what you would get if you did an update. Look through the changes with svn diff |less. Especially the changelog, just to make sure you are actually upgrading from 4.7.4 4.7.5 and not something different (due to a typo in the previous svn commands above).
Make sure to test the codebase first. I backed up my database first, then went to <a href="http://localhost/update.php">http://localhost/update.php[/geshifilter-code] and made sure the upgrade worked. Very important step!
Commit it when you are happy:

svn ci -m "Merged differences between drupal core 4.7.4 and 4.7.5"

After this I logged in to my 3 drupal sites, backed up the db, ran svn up, ran <a href="http://localhost/update.php">http://localhost/update.php[/geshifilter-code], and checked that there were no errors and that the site loaded some pages with no trouble. If I notice a problem in a few days I can always revert to the database backup I made.

That's all there is to it to upgrade a Drupal site when a new Drupal tarball is released! As long as you track vendor sources in your own subversion repository, the rest is easy.

The advantage of this technique is that I can make changes to drupal core but I can still easily merge in changes from new drupal releases. I can also do this same thing with modules. I track the module sources in a directory structure like:

/var/svn/repos/www/vendor/drupalmodules/gmap
/var/svn/repos/www/vendor/drupalmodules/gmap/09_09_2006
/var/svn/repos/www/vendor/drupalmodules/gmap/10_24_2006
/var/svn/repos/www/vendor/drupalmodules/gmap/10_25_2006
/var/svn/repos/www/vendor/drupalmodules/gmap/current
/var/svn/repos/www/vendor/drupalmodules/flash_gallery
/var/svn/repos/www/vendor/drupalmodules/flash_gallery/11_01_2006
/var/svn/repos/www/vendor/drupalmodules/flash_gallery/current
/var/svn/repos/www/vendor/drupalmodules/location
/var/svn/repos/www/vendor/drupalmodules/location/10_20_2006
/var/svn/repos/www/vendor/drupalmodules/location/current

I made some changes to the gmap and location module. I have had to merge in upstream changes in the gmap module 3 times. You can perform the merge by providing the location of the old tag and the new tag (or just use current) and the destination of the merge is a working directory of the module in question's directory (within the drupal modules directory of your trunk).

I have also used this technique several times for the mediawiki site at work. I highly recommend doing this any time you modify upstream source code and want to get updates from upstream when they become available.

I was trying to do an svn copy from a specific server path into a local working copy. I basically wanted to copy over the vanilla mediawiki-1.8.1 sources from my repository into a local directory (called trunk). Here is what I did:

svn copy svn+<a href="ssh://david@server/svn/repos/Projects/wiki/mediawiki/1.8.1">ssh://david@server/svn/repos/Projects/wiki/mediawiki/1.8.1</a> ./trunk

You would think that this would copy the contents of the 1.8.1 directory into the trunk directory. Wrong; it copies the 1.8.1 directory and its contents into the trunk directory, thus creating a 1.8.1 directory inside trunk. It took me forever to figure out why it was doing this instead of what I wanted it to do (which is to copy the contents of 1.8.1, not the 1.8.1 directory itself).

The problem was that the trunk directory already existed. If I removed the trunk directory, then it creates the trunk directory and copies the contents of the 1.8.1 directory into trunk.

In case anyone is curious, what I am doing is following the instructions from the svn book on how to maintain vendor branches.