Reducing CSS/Javascript Requests and File Size with Ant and YUI

With continuing reliance on front-end Javascript plugins and CSS, it's important to keep HTTP requests and file sizes smaller to reduce page load times. On this site, we use jQuery, jQuery Address, a custom application Javascript include, and other plugins. We also use a master CSS file as well as a reset file and other CSS additions for things like @media queries and the like. We can reduce all these file requests to one or two requests by combining them all into one file and minifying them.

We can use a compression tool such as YUI Compressor to handle the minifying, but actually doing the work to compile all those files together and run them through the compressor can be a time-consuming task to perform manually. It'd be great if we had a tool to do that for us. Enter Apache Ant.

Ant is a java-based build automation tool. It can be used to perform a variety of tasks and is ideally suited for building files for web applications. If you're on Mac OS X, Ant comes pre-installed (1.7.x on 10.5 and 1.8.x on 10.6). For other systems, check out the easy instructions here. YUI is a simple download and extraction from here.

All we need for a basic Ant task is a build file. Create a "build.xml" file in the root of your application. To start, you just need some basic markup. I added a property that contains the path to the YUI Compressor .jar file.

<?xml version="1.0" encoding="UTF-8"?>

<!-- project name, default target and base directory -->
<project name="typeoneerror" default="init" basedir=".">

    <!-- path to YUI Compressor -->
    <property name="yuic" location="/Users/typeoneerror/bin/yuicompressor-2.4.2/build/yuicompressor-2.4.2.jar"/>

    <!-- path to build resources -->
    <property name="css" location="public/css"/>
    <property name="js" location="public/js"/>
    <property name="ui_css" location="public/ui/css"/>
    <property name="ui_js" location="public/ui/js"/>

    <!-- default target -->
    <target name="init"/>
</project>

This defines our project build with a default target (one of the actions that will run when we build) called "init". Also, we've defined some properties (paths to YUI and to our CSS and Javascript folder) that we'll use later.

Next, create the first target:

<!--concatenate javascript-->
<target name="concat.js" description="Stick all the JS together">
     <concat destfile="build/application.js">
        <!-- first concat all the files in the /ui/js directory -->
        <filelist dir="${ui_js}" files="jquery.min.js,jquery.address.min.js"/>
        <!-- then add our application js file -->
        <filelist dir="${js}" files="application.js"/>
    </concat>
</target>

This target uses the concat process to concatenate a list of javascript files into a destfile (destination file) build/application.js. This combines everything we're using into one file. The next target will use that file and minify it:

<!--minify javascript-->
<target name="minify.js" depends="concat.js" description="Minify JavaScript using YUI Compressor">
    <apply executable="java" parallel="false">
        <!-- look for any javascript files in the build dir -->
        <fileset dir="build" includes="*.js"/>
        <!-- pass arguments to the yui program -->
        <arg line="-jar"/>
        <arg path="${yuic}"/>
        <srcfile/>
        <!-- output minified files to our js dir with .min.js extenstion -->
        <arg line="-o"/>
        <mapper type="glob" from="*.js" to="${js}/*.min.js"/>
        <targetfile/>
    </apply>
</target>

The depends property defines actions that need to be performed before running the target. Add the minify.js target to the depends property of our default action:

<target name="init" depends="minify.js"/>

When we run the build, init will be run and then minify.js. Of course, concat.js will be run before minify.js since minify.js depends on it.

We add another two targets to concat and minify our CSS. Add another target to the list of targets in the init target. Our final build file looks something like this:

<?xml version="1.0" encoding="UTF-8"?>

<!-- project name, default target and base directory -->
<project name="typeoneerror" default="init" basedir=".">

    <!-- path to YUI Compressor -->
    <property name="yuic" location="/Users/typeoneerror/bin/yuicompressor-2.4.2/build/yuicompressor-2.4.2.jar"/>

    <!-- path to build resources -->
    <property name="css" location="public/css"/>
    <property name="js" location="public/js"/>
    <property name="ui_css" location="public/ui/css"/>
    <property name="ui_js" location="public/ui/js"/>

    <!-- default target -->
    <target name="init" depends="minify.js,minify.css"/>

    <!--concatenate javascript-->
    <target name="concat.js" description="Stick all the JS together">
         <concat destfile="build/application.js">
            <!-- first concat all the files in the /ui/js directory -->
            <filelist dir="${ui_js}" files="jquery.min.js,jquery.address.min.js"/>
            <!-- then add our application js file -->
            <filelist dir="${js}" files="application.js"/>
        </concat>
    </target>

    <!--minify javascript-->
    <target name="minify.js" depends="concat.js" description="Minify JavaScript using YUI Compressor">
        <apply executable="java" parallel="false">
            <!-- look for any javascript files in the build dir -->
            <fileset dir="build" includes="*.js"/>
            <!-- pass arguments to the yui program -->
            <arg line="-jar"/>
            <arg path="${yuic}"/>
            <srcfile/>
            <!-- output minified files to our js dir with .min.js extenstion -->
            <arg line="-o"/>
            <mapper type="glob" from="*.js" to="${js}/*.min.js"/>
            <targetfile/>
        </apply>
    </target>

    <!--concatenate css-->
    <target name="concat.css" description="Stick all the CSS together">
         <concat destfile="build/core.css">
            <filelist dir="${ui_css}" files="reset.min.css"/>
            <filelist dir="${css}" files="screen.css,additions.css"/>
        </concat>
    </target>

    <!--minify concatenated css-->
    <target name="minify.css" depends="concat.css" description="Minify CSS using YUI Compressor">
        <apply executable="java" parallel="false">
            <fileset dir="build" includes="*.css"/>
            <arg line="-jar"/>
            <arg path="${yuic}"/>
            <srcfile/>
            <arg line="-o"/>
            <mapper type="glob" from="*.css" to="${css}/*.min.css"/>
            <targetfile/>
        </apply>
    </target>
</project>

Running an Ant build file is as simple as shelling into the directory via your Terminal or command line interface and entering ant. It automatically looks for a file called "build.xml". You can also grab Simon Gregory's Ant TextMate Bundle if you want to just hit Cmd+R to build in your TextMate project (awesome).

December 20th, 2010 | Permalink

Django "Stack" Install with ActivePython on Win/Mac

A current project involves Django, PostgreSQL, and git; three softwares I was pretty much unfamiliar with as a LAMP developer. After some poking around and initial love affair with Python and the idea of Django...well, Django is super-sweet on *nix OSs like Ubuntu, but not so fun to install and use on Windows and the "unix-like" Darwin-based Mac OS. So here's the start of my install guide for Windows. It can easily be applied to Mac since ActivePython is available for Mac as well.

So, the requirements: Python is easy to install; Django is easy to install; MySQL is generally pretty easy to install; the adapters that allow Python to talk to MySQL? Not so much. And what if you want to manage all those python modules you install? Or have virtual environments that match your target deployment environment? Enter ActivePython. ActivePython is essentially a distribution of Python and a number of other programs for managing python and python modules/package. I highly recommend you start with that:

Install ActivePython: http://www.activestate.com/activepython/

I just installed to C:\Python (/Python on Mac).

Now, the creators of the code I'm working with for this project use an application called "pip" (bundled with ActivePython) that allows you to install a list of dependancies from a standard requirements file. You can "install" packages, svn repositories, git repositories, and more. This is easily one of the coolest ways to distribute code I've ever seen. I fully expect this to be a standard development method in the next few years. Anyway, to import these files into pip require require you to have a command line version of git and subversion, so:

Install Git for windows

Options for the install:
- Install git as cmd line tool (not in bash shell)
- Checkout as-is, commit as Unix-style endings

Install Git for OSX

Install Subversion Command line client (CollabNet Subversion Command-Line Client v1.6.11 (for Windows))

On OSX you should have svn already and you shouldn't need to worry too much about line endings. Feel free to upgrade to the latest and greatest.

Now we have what we need to get going: from ActivePython; virtualenv, pip, and pypm; svn; and git.

Make sure you can run both. Open up a cmd prompt or Terminal on OSX:

$ git --version
git version 1.7.0.2.mysysgit.0

$ svn --version
svn, version 1.6.11

Ok, we're good to go. Switch to your drive root or wherever you want to create your virtual environments:

$ mkdir virtualenvs
$ cd virtualenvs

# create a virtual enviroment called "myenv"
$ virtualenv myenv

# OR, if you want the env to not inherit your globally installed python site packages
# which you probably want since installing mysql and postgre plugins in a virtual
# environment is tough since you have to compile them
$ virtualenv --no-site-packages myenv

# activate the environment (windows):
$ myenv\Scripts\activate.bat
(myenv)D:\virtualenvs>

# activate the environment (mac):
# note that scripts are created in "bin" instead of "Scripts"
$ source myenv/bin/activate
(myenv)virtualenvs $

Now running pip will use the pip created when the virtual environment was created, installing packages only in that working environment. So you can have different environments with different packages installed.

Let's install Django as a test first.

# latest
(myenv)virtualenvs $ pip install Django

# or install a specific version
(myenv)virtualenvs $ pip install Django==1.1
Downloading/unpacking Django==1.1
  Downloading Django-1.1.tar.gz (5.6Mb): 5.6Mb downloaded
  Running setup.py egg_info for package Django
Installing collected packages: Django
  Running setup.py install for Django
Successfully installed Django

Next step in my install process was getting the codebase from the client. They used github for version control (replacing client name with dummy copy):

# clone the git repository
(myenv)virtualenvs $ cd D:\Projects\Client\
(myenv)virtualenvs $ git clone git@github.com:client/repository.git

The client's repo had a requirments formatted file that I can then use to install with pip.

# back in our virtual environment, now install the requirements
# the -E flag tells pip which environment to install to.
(myenv)virtualenvs $ pip install -E myenv -r D:\Projects\Client\repository\requirements.txt

This is neat because pip's requirements format allows svn, git, and pypm (pythom package manager) hints in this file. It'll automatically pull in the dependancies using the method specified. Here's an example:

Django==1.1
-e svn+http://django-mptt.googlecode.com/svn/trunk@121#egg=django_mptt-0.3_pre-py2.5-dev
django-flatcontent==0.1.2
-e git://github.com/bartTC/django-frontendadmin.git@53d8ed1fdcd1ef466fb9c8b38ccb2abb77978b1e#egg=django_frontendadmin-0.5-py2.6-dev
django-markitup==0.5.2
django-haystack==1.0.1-final
Whoosh==0.3.9
textile

So we're installing Django 1.1, copying down an svn repo from googlecode, cloning a git repo from github and installing a number of python mods using PyPm. Pretty badass for one command!

If we were on unix or mac (with xcode installed) we could also use pip to compile the mysqldb module, but we have some binary install options on windows. This will install it for all python environments. Not sure how to use this to install to a specific virtual environment: Download MySQL-python-1.2.2.win32-py2.6.exe (1,022.8 KiB). Run the installer and select your ActivePython install when prompted. To test if it works, open cmd prompt and type

$ python
>>> import MySQLdb 

If there's no errors; hey, it worked. On MacOS, it's not quite as fun. I used Macports to download the MySQL5 headers, then I did the following:

# only download the package
(myenv)virtualenvs $ pip install --no-install MySQL-python

Then open the downloaded package and edit site.cfg and uncomment line 13 and add the path to your mysql_config (for me this ended up pointing to the bin folder and mysql_config I downloaded with Macports). After than you can run:

python setup.py build
python setup.py install

There's also a Windows port of the psycopg2 module (PostgreSQL for Python). We happen to be using this for the client project. You'll need to install Postgre 8.x first. Then install this guy. Not going to go into Postgre installation and whatnot since I only know a bit of the basics myself. It was enough to get my Django project up and running:

cd /Projects/Client/repository/web
python manage.py syncdb
pyhton manage.py runserver

Command line is fun. Now, getting WSGI to work with MAMP will be the real challenge.

May 14th, 2010 | Permalink

Getting Started with Custom Zend_Tool Resources

Recent versions of the Zend Framework come with a useful CLI (command line) tool for manipulating project structure and files called Zend_Tool. Assuming you use a standard framework layout, this tool can speed development by creating controllers, views, models, et al. I found this tool useful, but wanted to extends the tool to make it more custom to my liking. i.e. instead of

zf create controller Articles

outputting a controller that extends Zend_Controller_Action, I want to make a CMS controller builder; one that extends Typeoneerror_Cms_Controller_Crud, sets and creates a model that extends Typeoneerror_Db_ActiveRecord, etc. My intention at the beginning of this was to add a Provider that allowed me to type

zf create crud Articles

and set up all of this. After spending a few hours figuring out where everything was and how it worked, it's pretty clear that ZF Tool is still very much a "baby." Set-up is quite challenging, and you have to write quite a bit of code to make your own providers (providers define your command line actions) and contexts (contexts define resources and how to handle provider actions).  I thought I'd provide some basic steps to get started with your own resources.


Installing the zf tool.

First thing you need to do is install the zf.sh script. This is available in the "bin" directory of the release. I've got my own library with a subversion external to the latest release checked out in it, so I copied the "bin" file into the same directory. So locally I have something like:

/typeoneerror
    /bin
        /zf.php
        /zf.sh
    /library
        /Typeoneerror
        /Zend

Now, the zf scripts need to be in your unix include_path to run. Instead of doing that I added an alias to my .bash_profle that points to the shell script:

alias zf='/Users/ben/Documents/Codebase/taz/trunk/project/bin/zf.sh'

Also, make sure the zf script is executable:

chmod a+x /Users/ben/Documents/Codebase/taz/trunk/project/bin/zf.sh

Now if you run the following command, you should see the version output:

$ zf show version
Zend Framework Version: 1.10.2

In recent versions of zf tool, you have to create a storage directory and configuration file if you want to write any custom contexts or providers. Let's do that next. First run:

$ zf --setup storage-directory
Storage directory created at /Users/ben/.zf/

followed by:

$ zf --setup config-file
Config file written to /Users/ben/.zf.ini

If you open that up you should see that the include path to the Zend library folder has been added because it's where the zf script expects it to be ("../library"). Since my client code in Typeoneerror directory lives there as well, we should be set.

# Inside /Users/ben/.zf.ini
php.include_path = "/Users/ben/Documents/Codebase/taz/trunk/project/library:.:"


Custom Resources

Now, to begin creating custom providers, we first need a Manifest. A custom manifest tells the tool what providers to register. Let's create a manifest and a sample provider first:

# Manifest.php

require_once "Typeoneerror/Tool/Provider/Crud.php";

class Typeoneerror_Tool_Manifest implements Zend_Tool_Framework_Manifest_ProviderManifestable
{
    public function getProviders()
    {
        return array(
            new Typeoneerror_Tool_Provider_Crud()
        );
    }
}

All we implement in this file is the getProviders method which returns a list of instantiated providers.

# Crud.php

class Typeoneerror_Tool_Provider_Crud
extends Zend_Tool_Project_Provider_Abstract
implements Zend_Tool_Framework_Provider_Pretendable
{
    public function create($name = 'world')
    {
       $this->_registry->getResponse()
                       ->appendContent("Hello, {$name}!");
    }
}

Our sample defines a create method which simply echos out a "Hello" message
to the CLI output. Before you can actually use these new tools though, we
have to register them with the Reposity. Back to the command line:

$ zf enable config.manifest Typeoneerror_Tool_Manifest
Provider/Manifest 'Typeoneerror_Tool_Manifest' was enabled for usage with Zend Tool.

If you take another look at your config file you'll see something like:

php.include_path = "/Users/ben/Documents/Codebase/taz/trunk/project/library:.:"
basicloader.classes.0 = "Typeoneerror_Tool_Manifest"

Next register the Crud provider:

trunk $ zf enable config.provider Typeoneerror_Tool_Provider_Crud
Provider/Manifest 'Typeoneerror_Tool_Provider_Crud' was enabled for usage with Zend Tool.

And check the ini output again

php.include_path = "/Users/ben/Documents/Codebase/taz/trunk/project/library:.:"
basicloader.classes.0 = "Typeoneerror_Tool_Manifest"
basicloader.classes.1 = "Typeoneerror_Tool_Provider_Crud"

This tells the tool to load those classes for use. Theoretically, you could just add these manually to the ini file. Anyway, now we can run our tool!

$ zf create crud
Hello, world!

$ zf create crud Ben
Hello, Ben!

If you run the following you can now see that the Crud controller is registered
with the zf tool.

$ zf show manifest
type=Tool, clientName=all, providerName=Crud    : crud


Custom Output Providers and Contexts

Ok, great, now we can see how to register parts with the zf tool, but now you're wondering (probably)
"how do I create custom output?".

Well, you're going to need to create two classes: a Context (in most cases a "file context" or how to save the file and the code that will be injected into the file) and a Provider which is initialized from the CLI and uses Contexts to create the resources. I found this tutorial to be a good overview of those steps.

To make things easy for my crud controller provider, I simply copied Zend_Tool_Project_Context_Zf_ControllerFile (context) and Zend_Tool_Project_Provider_Controller (provider) and edited the code generation to suit my needs. You can download the sample files I created here. The Provider defines the "create" and "delete" functions which are accessed from the command line. The Provider checks to see if the Crud controller exists and if not, creates it using Zend_CodeGenerator (Typeoneerror_Tool_Context_CrudControllerFile :: getContents).

As I said at the beginning, custom output is a chore, but I feel like this has a load of potential. ZF 1.10 begins the support of "delete" methods with providers as well but they don't seem to be fully complete yet (I implemented a delete method in my crud provider but I'm unsure how to remove the line item in the .zfproject.xml manifest as of yet). You may find yourself manually editing your project manifest as you get going.

Good luck! Feel free to send me a message if you have any tips or questions about Zend_Tool.

March 7th, 2010 | Permalink