Simple Subversion Merges

Subversion merging can be a complex task, one that's decidedly terrifying at times. I'm typically always worried when I perform an svn merge, and so I always consult the red bean manual when I'm about to do one. In the last few weeks I've written a handful of emails explaining how to do it quickly and painlessly, so I thought it beneficial to detail how I do basic merging as a reminder to myself and hopefully as a helpful tutorial for someone looking to make it as painless as possible via a command line interface.

Let's begin with a scenario; one that should be quite familiar (This may be a "no shit" scenario for you, but I want to give us some context.): you're going to publish your website for the first time. You want to publish the site live but your client wants you to develop a new feature. You want to be able to update the production build with bug fixes while you work on the new feature. So, you create a branch in your subversion repository:

svn cp trunk branches/release-1.0_launch

The release-1.0_launch branch will be uploaded to your host (however you might be doing that; FTP, svn check-out, rsync, et al) and remain live while you develop your new features in trunk. That's all find and dandy, but say you fix a number of bugs or quickly have to add a feature to the release-1.0_launch branch. Instead of "hackily" making the same bug fixes in trunk and having to copy and paste the new features, you'll do a merge from the branch to the trunk.

The merging process is basically two steps:

  1. Make sure everything is up to date locally (svn update the repo).

  2. Use svn merge to merge the branch into your trunk.

Where it gets somewhat complex is you need to let subversion know how much you want to merge, i.e. you can merge a number of commits from your branch to your trunk. The syntax is straightforward:

svn merge -r{FROM_REVISION}:{TO_REVISION} branches/release-1.0_launch trunk

Let's say you want to merge all the commits to the branch into the trunk. You can use the --stop-on-copy flag to figure out the revision number was when you copied from trunk:

svn log branches/release-1.0_launch --stop-on-copy

The earliest log number (FROM_REVISION) is where you want to copy from through the most recent commit number (TO_REVISION), which you can use svn update to get. Before you even run merge, definitely make sure you do a svn merge with the --dry-run flag first to make sure nothing alarming will result.

Taking the revision number from the --stop-on-copy and the current HEAD revision from svn update, you'll merge from branch to trunk. For example assuming our FROM_REVISION was "123" and our TO_REVISION was "126"

svn merge -r123:126 --dry-run branches/release-1.0_launch trunk

This merges revisions 123 through the most recent (HEAD) from the branch to the trunk. The --dry-run flag will tell you what should happen. You'll need to remove that to actually perform a merge.

Once you think the files changed in the potential merge look good, remove the --dry-run flag and run it. Running svn status will tell you what's different in the trunk after the merge. You may have to resolve some conflicts with the files in your trunk. One really useful task I use all the time is to get a diff of the changes and pipe it to TextMate:

svn diff | mate

This will open the diff file with TextMate (assuming you've set up the mate command from Textmate > Help > Terminal Usage) with some lovely syntax highlighting; typically makes it much easier to take in the changes with color coding!

Once you've merged and resolved your conflicts, obviously you'll want to run any unit tests and check the project to make sure it's working as expected. After that, commit the trunk with a thoughtful message like "Merged -r123:126 from release-1.0_launch to trunk" so you can look back and see what revisions were specifically merged.

June 30th, 2011 | Permalink

Django "Stack" Install with ActivePython on Win/Mac

A current project involves Django, PostgreSQL, and git; three softwares I was pretty much unfamiliar with as a LAMP developer. After some poking around and initial love affair with Python and the idea of Django...well, Django is super-sweet on *nix OSs like Ubuntu, but not so fun to install and use on Windows and the "unix-like" Darwin-based Mac OS. So here's the start of my install guide for Windows. It can easily be applied to Mac since ActivePython is available for Mac as well.

So, the requirements: Python is easy to install; Django is easy to install; MySQL is generally pretty easy to install; the adapters that allow Python to talk to MySQL? Not so much. And what if you want to manage all those python modules you install? Or have virtual environments that match your target deployment environment? Enter ActivePython. ActivePython is essentially a distribution of Python and a number of other programs for managing python and python modules/package. I highly recommend you start with that:

Install ActivePython: http://www.activestate.com/activepython/

I just installed to C:\Python (/Python on Mac).

Now, the creators of the code I'm working with for this project use an application called "pip" (bundled with ActivePython) that allows you to install a list of dependancies from a standard requirements file. You can "install" packages, svn repositories, git repositories, and more. This is easily one of the coolest ways to distribute code I've ever seen. I fully expect this to be a standard development method in the next few years. Anyway, to import these files into pip require require you to have a command line version of git and subversion, so:

Install Git for windows

Options for the install:
- Install git as cmd line tool (not in bash shell)
- Checkout as-is, commit as Unix-style endings

Install Git for OSX

Install Subversion Command line client (CollabNet Subversion Command-Line Client v1.6.11 (for Windows))

On OSX you should have svn already and you shouldn't need to worry too much about line endings. Feel free to upgrade to the latest and greatest.

Now we have what we need to get going: from ActivePython; virtualenv, pip, and pypm; svn; and git.

Make sure you can run both. Open up a cmd prompt or Terminal on OSX:

$ git --version
git version 1.7.0.2.mysysgit.0

$ svn --version
svn, version 1.6.11

Ok, we're good to go. Switch to your drive root or wherever you want to create your virtual environments:

$ mkdir virtualenvs
$ cd virtualenvs

# create a virtual enviroment called "myenv"
$ virtualenv myenv

# OR, if you want the env to not inherit your globally installed python site packages
# which you probably want since installing mysql and postgre plugins in a virtual
# environment is tough since you have to compile them
$ virtualenv --no-site-packages myenv

# activate the environment (windows):
$ myenv\Scripts\activate.bat
(myenv)D:\virtualenvs>

# activate the environment (mac):
# note that scripts are created in "bin" instead of "Scripts"
$ source myenv/bin/activate
(myenv)virtualenvs $

Now running pip will use the pip created when the virtual environment was created, installing packages only in that working environment. So you can have different environments with different packages installed.

Let's install Django as a test first.

# latest
(myenv)virtualenvs $ pip install Django

# or install a specific version
(myenv)virtualenvs $ pip install Django==1.1
Downloading/unpacking Django==1.1
  Downloading Django-1.1.tar.gz (5.6Mb): 5.6Mb downloaded
  Running setup.py egg_info for package Django
Installing collected packages: Django
  Running setup.py install for Django
Successfully installed Django

Next step in my install process was getting the codebase from the client. They used github for version control (replacing client name with dummy copy):

# clone the git repository
(myenv)virtualenvs $ cd D:\Projects\Client\
(myenv)virtualenvs $ git clone git@github.com:client/repository.git

The client's repo had a requirments formatted file that I can then use to install with pip.

# back in our virtual environment, now install the requirements
# the -E flag tells pip which environment to install to.
(myenv)virtualenvs $ pip install -E myenv -r D:\Projects\Client\repository\requirements.txt

This is neat because pip's requirements format allows svn, git, and pypm (pythom package manager) hints in this file. It'll automatically pull in the dependancies using the method specified. Here's an example:

Django==1.1
-e svn+http://django-mptt.googlecode.com/svn/trunk@121#egg=django_mptt-0.3_pre-py2.5-dev
django-flatcontent==0.1.2
-e git://github.com/bartTC/django-frontendadmin.git@53d8ed1fdcd1ef466fb9c8b38ccb2abb77978b1e#egg=django_frontendadmin-0.5-py2.6-dev
django-markitup==0.5.2
django-haystack==1.0.1-final
Whoosh==0.3.9
textile

So we're installing Django 1.1, copying down an svn repo from googlecode, cloning a git repo from github and installing a number of python mods using PyPm. Pretty badass for one command!

If we were on unix or mac (with xcode installed) we could also use pip to compile the mysqldb module, but we have some binary install options on windows. This will install it for all python environments. Not sure how to use this to install to a specific virtual environment: Download MySQL-python-1.2.2.win32-py2.6.exe (1,022.8 KiB). Run the installer and select your ActivePython install when prompted. To test if it works, open cmd prompt and type

$ python
>>> import MySQLdb 

If there's no errors; hey, it worked. On MacOS, it's not quite as fun. I used Macports to download the MySQL5 headers, then I did the following:

# only download the package
(myenv)virtualenvs $ pip install --no-install MySQL-python

Then open the downloaded package and edit site.cfg and uncomment line 13 and add the path to your mysql_config (for me this ended up pointing to the bin folder and mysql_config I downloaded with Macports). After than you can run:

python setup.py build
python setup.py install

There's also a Windows port of the psycopg2 module (PostgreSQL for Python). We happen to be using this for the client project. You'll need to install Postgre 8.x first. Then install this guy. Not going to go into Postgre installation and whatnot since I only know a bit of the basics myself. It was enough to get my Django project up and running:

cd /Projects/Client/repository/web
python manage.py syncdb
pyhton manage.py runserver

Command line is fun. Now, getting WSGI to work with MAMP will be the real challenge.

May 14th, 2010 | Permalink

Getting Started with Custom Zend_Tool Resources

Recent versions of the Zend Framework come with a useful CLI (command line) tool for manipulating project structure and files called Zend_Tool. Assuming you use a standard framework layout, this tool can speed development by creating controllers, views, models, et al. I found this tool useful, but wanted to extends the tool to make it more custom to my liking. i.e. instead of

zf create controller Articles

outputting a controller that extends Zend_Controller_Action, I want to make a CMS controller builder; one that extends Typeoneerror_Cms_Controller_Crud, sets and creates a model that extends Typeoneerror_Db_ActiveRecord, etc. My intention at the beginning of this was to add a Provider that allowed me to type

zf create crud Articles

and set up all of this. After spending a few hours figuring out where everything was and how it worked, it's pretty clear that ZF Tool is still very much a "baby." Set-up is quite challenging, and you have to write quite a bit of code to make your own providers (providers define your command line actions) and contexts (contexts define resources and how to handle provider actions).  I thought I'd provide some basic steps to get started with your own resources.


Installing the zf tool.

First thing you need to do is install the zf.sh script. This is available in the "bin" directory of the release. I've got my own library with a subversion external to the latest release checked out in it, so I copied the "bin" file into the same directory. So locally I have something like:

/typeoneerror
    /bin
        /zf.php
        /zf.sh
    /library
        /Typeoneerror
        /Zend

Now, the zf scripts need to be in your unix include_path to run. Instead of doing that I added an alias to my .bash_profle that points to the shell script:

alias zf='/Users/ben/Documents/Codebase/taz/trunk/project/bin/zf.sh'

Also, make sure the zf script is executable:

chmod a+x /Users/ben/Documents/Codebase/taz/trunk/project/bin/zf.sh

Now if you run the following command, you should see the version output:

$ zf show version
Zend Framework Version: 1.10.2

In recent versions of zf tool, you have to create a storage directory and configuration file if you want to write any custom contexts or providers. Let's do that next. First run:

$ zf --setup storage-directory
Storage directory created at /Users/ben/.zf/

followed by:

$ zf --setup config-file
Config file written to /Users/ben/.zf.ini

If you open that up you should see that the include path to the Zend library folder has been added because it's where the zf script expects it to be ("../library"). Since my client code in Typeoneerror directory lives there as well, we should be set.

# Inside /Users/ben/.zf.ini
php.include_path = "/Users/ben/Documents/Codebase/taz/trunk/project/library:.:"


Custom Resources

Now, to begin creating custom providers, we first need a Manifest. A custom manifest tells the tool what providers to register. Let's create a manifest and a sample provider first:

# Manifest.php

require_once "Typeoneerror/Tool/Provider/Crud.php";

class Typeoneerror_Tool_Manifest implements Zend_Tool_Framework_Manifest_ProviderManifestable
{
    public function getProviders()
    {
        return array(
            new Typeoneerror_Tool_Provider_Crud()
        );
    }
}

All we implement in this file is the getProviders method which returns a list of instantiated providers.

# Crud.php

class Typeoneerror_Tool_Provider_Crud
extends Zend_Tool_Project_Provider_Abstract
implements Zend_Tool_Framework_Provider_Pretendable
{
    public function create($name = 'world')
    {
       $this->_registry->getResponse()
                       ->appendContent("Hello, {$name}!");
    }
}

Our sample defines a create method which simply echos out a "Hello" message
to the CLI output. Before you can actually use these new tools though, we
have to register them with the Reposity. Back to the command line:

$ zf enable config.manifest Typeoneerror_Tool_Manifest
Provider/Manifest 'Typeoneerror_Tool_Manifest' was enabled for usage with Zend Tool.

If you take another look at your config file you'll see something like:

php.include_path = "/Users/ben/Documents/Codebase/taz/trunk/project/library:.:"
basicloader.classes.0 = "Typeoneerror_Tool_Manifest"

Next register the Crud provider:

trunk $ zf enable config.provider Typeoneerror_Tool_Provider_Crud
Provider/Manifest 'Typeoneerror_Tool_Provider_Crud' was enabled for usage with Zend Tool.

And check the ini output again

php.include_path = "/Users/ben/Documents/Codebase/taz/trunk/project/library:.:"
basicloader.classes.0 = "Typeoneerror_Tool_Manifest"
basicloader.classes.1 = "Typeoneerror_Tool_Provider_Crud"

This tells the tool to load those classes for use. Theoretically, you could just add these manually to the ini file. Anyway, now we can run our tool!

$ zf create crud
Hello, world!

$ zf create crud Ben
Hello, Ben!

If you run the following you can now see that the Crud controller is registered
with the zf tool.

$ zf show manifest
type=Tool, clientName=all, providerName=Crud    : crud


Custom Output Providers and Contexts

Ok, great, now we can see how to register parts with the zf tool, but now you're wondering (probably)
"how do I create custom output?".

Well, you're going to need to create two classes: a Context (in most cases a "file context" or how to save the file and the code that will be injected into the file) and a Provider which is initialized from the CLI and uses Contexts to create the resources. I found this tutorial to be a good overview of those steps.

To make things easy for my crud controller provider, I simply copied Zend_Tool_Project_Context_Zf_ControllerFile (context) and Zend_Tool_Project_Provider_Controller (provider) and edited the code generation to suit my needs. You can download the sample files I created here. The Provider defines the "create" and "delete" functions which are accessed from the command line. The Provider checks to see if the Crud controller exists and if not, creates it using Zend_CodeGenerator (Typeoneerror_Tool_Context_CrudControllerFile :: getContents).

As I said at the beginning, custom output is a chore, but I feel like this has a load of potential. ZF 1.10 begins the support of "delete" methods with providers as well but they don't seem to be fully complete yet (I implemented a delete method in my crud provider but I'm unsure how to remove the line item in the .zfproject.xml manifest as of yet). You may find yourself manually editing your project manifest as you get going.

Good luck! Feel free to send me a message if you have any tips or questions about Zend_Tool.

March 7th, 2010 | Permalink

Adding CVS-style $Id$ tags to subversion commits

You can automatically add versioning data to your files when you commit with subversion. This is a super helpful way to quickly see when the last update was and which user committed the revision. For example, here's the header of one of my .as files:

/**
 * @author     Benjamin Borowski (ben.borowski@typeoneerror.com)
 * @copyright  Copyright (c) Typeoneerror Studios http://typeoneerror.com
 * @version    $Id: Fetch.as 98 2010-01-31 07:31:11Z ben $
 */

Above you can see version number, date commited and my name as the last person to commit changes to the repository for this file. Adding this "auto-tagging" is quite easy. You simply need to add svn:keywords to the file(s) you're going to commit.

To start, you can do the following (assuming "yourFileName" is the name of the file you want to add the Id keyword to):

$ svn propset svn:keywords Id yourFileName
property 'svn:keywords' set on 'yourFileName'
$ svn propget svn:keywords yourFileName
Id

Now you simply add the Id tag in your file, e.g.:

/**
 * $Id$
 */

Since the Id property is set, when subversion commits, you'll see it updates to the "version" tag in the earlier example.

Now that we know how that works, how about we configure our subversion installation to add this property to files whenever we "svn add" them to our project? Open up ~/.subversion/config in your favorite text editor. You're going to want to make sure the following options are set:

[miscellany]
enable-auto-props = yes

This turns on adding properties automatically when added to working copies. Now under [auto-props], add something like the following. Each of these is just a wildcarded file extension. You may not need all of these; these are just some of the common file-types I deal with:

*.as = svn:keywords=Id
*.css = svn:keywords=Id
*.cpp = svn:keywords=Id
*.email = svn:keywords=Id
*.h = svn:keywords=Id
*.ini = svn:keywords=Id
*.js = svn:keywords=Id
*.m = svn:keywords=Id
*.mm = svn:keywords=Id
*.mxml = svn:keywords=Id
*.php = svn:keywords=Id
*.phtml = svn:keywords=Id
*.pjs = svn:keywords=Id
*.xml = svn:keywords=Id

So now any time I add (for example) MyClass.as or MyViewController.m to a working copy, the Id property is automatically set. To wrap up, here's a bash function you can add to your .bash_profile to add the Id property recursively to files already under version control:

# add svn:keywords Id property recursively
function addSvnId
{
find . \( -name "*.as" -o \
-name "*.php" -o \
-name "*.xml" \) -exec svn propset svn:keywords Id {} \;
}

Of course you can add and remove new extensions. To use this, simply navigate to the working copy in question in Terminal and run:

$ addSvnId

Be careful as this is recursive so it will apply to all files with the specified extensions starting from the current directory. You can also accomplish the same tasks easily in a GUI subversion client such as Tortoise. Just right click the folder and edit the "Properties." Under the subversion tab, you can again, edit properties and select svn:keywords property and add Id to the text field. You can also apply it recursively.

February 3rd, 2010 | Permalink