eZ Community » Blogs » Arne Bakkebo » Automatic deployment - QA#6

By

Arne Bakkebo

Automatic deployment - QA#6

Wednesday 21 August 2013 10:19:24 am

  • Currently 5 out of 5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

After a break during the summer holiday months, I'm back with another musing around quality software development. This time I'll talk a little about how to set up automatic deployment. Automatic deployment is an essential part of quality development, and it's also one of the simplest starting points for improvement. There is of course a range of ways to do automatic deployment, the following will describe how I have solved it. I have used this only on eZ Publish 4.x, but most of it should work with some adjustments on 5.x also.

[QA5 - Code review considerations]

The deployment job

In Making Waves we use Jenkins[1] as a deployment server. It is quite easy to set up and configure, you can set it up in ten minutes on your local machine for testing. If you haven't looked at it before, feel free to download and try it out today. And with some configuring it can allow you to deploy your source code changes by the click of a button.

Start by installing Jenkins, and then set up a new job on it. This is explained on the Jenkins website, I won't go into the details here. When editing the job, at "Source Code Management" we usually specify a Github repository, Subversion and others are also supported.

On "Build" I add an "Execute shell" option that will pack the source code with tar. This is done with the following code snippet:

 tar --exclude='./.git' --exclude='./deploy.tar.gz' -zcf deploy.tar.gz .

This is not required to make it work, you could transfer the source code as is, but packing it will make the transfer faster. On the first deploy this will also make the deploy fail, tar complaining that the file changed as we read it. The reason is that deploy.tar.gz does not exist, so when it's added tar ignores the exclude and interprets it as new files being added to the file list. I haven't found a way to avoid this, any tar experts out there with suggestions? Anyway, on following deploys, the file will exist, so it's not a practical problem.

You need to configure the server to deploy to centrally in the Jenkins "Configure System" menu. Generate a public/private key pair[2], add the private key to the server settings in Jenkins, and the public key to the file ~/.ssh/authorized_keys for the deploy user on the server. Make sure this file have access for only the owner of the file. This will allow Jenkins to automatically log in to this user when deploying, no passwords asked. I recommend that the user in question have access to as little as possible on the server, just enough to perform the deploy.

"Post-build Actions" is the settings that actually do the deployment work for us. Here I use "Send build artifacts over SSH", and select the previously mentioned configured server. "Transfer Set Source files" deploy.tar.gz and then "Exec command" to run a script on the server called deploy.sh. This script is responsible for setting up everything on the server.

Cache handling

One way of handling the eZ Publish cache is to clear all cache on every deploy. This, as we all know, is suboptimal, and may worst case crash the server. I have solved this by traversing the Jenkins log to generate a list of template files that have changed. I send this list to the server, and there remove the specific template cache.

In my Jenkins deploy job configuration, I add two "Execute shell" options with the following scripts:

 grep \.tpl ../builds/${BUILD_NUMBER}|>/changelog.xml | sed 's/.*\/\(.*\)\.tpl/\1-\*/' > template_changes.txt
 grep translation\.ts ../builds/${BUILD_NUMBER}|>/changelog.xml | sed  's/.*\/\(.*\/translation\.ts\)/\1/' > translation_changes.txt

The template_changes.txt and translation_changes.txt files will be in the Jenkins job workspace/ folder, and contains a list of respectively template files changes since last deploy and translations files changed since last deploy. As my previously mentioned tar operation is run after these files are generated, they will be included in the deploy.

Then in the deploy script on the server, I run the following:

 # Clear relevant compiled templates
 
 
template_list=`cat git/template_changes.txt`
if [ -d var/ezflow_site/cache/template/compiled ]; then
    cd var/ezflow_site/cache/template/compiled
    for i in $template_list; do
        echo Clearing compiled templates for $i
        rm -f $i
    done
    cd -
fi

This will remove any compiled templates relating to the templates that have been modified for this deploy. For translations it's a bit more complicated. I can clear the translations cache, but I can't know in which templates the translations are used so I might need to fix this manually after the deploy. In theory, it could be possible to find the modified translations strings, and detect the template files they belong to, but I do not think it's worth the work required. So as a result my translations clearing script will look like this:

 translation_list=`cat git/translation_changes.txt`
if [[ ! $translation_list = '' ]]; then
    echo 'Clearing translations cache (may require manual clearing)'
    $PHP bin/php/ezcache.php --clear-id=translation
fi

In the end I always clear content view, ini, cache block and override cache. These are light weight cache, so should be no problem unless it's a very high traffic site (in which case special precautions should be considered anyway):

 # Clear possible other lightweight cache that might need it
$PHP bin/php/ezcache.php --clear-tag=ini
$PHP bin/php/ezcache.php --clear-id=template-override,template-block,content

See my full server side deployment script further down.

INI settings

For setting up ini settings for the different environments, we use the extension NovenINIUpdate[3]. I've set this up on the Jenkins server, to run before the transfer to the different servers to deploy to. NovenINI runs as a command line eZ Publish script, so to run it on the Jenkins job, I needed to set up eZ Publish in relation to the job. Since we have a bunch of different deploy jobs on our Jenkins server, I set up eZ Publish centrally, and did some creative symlinking to it from each job. This is not strictly necessary, but it saves some disk space. :)

A Jenkins job is defined in a directory structure under Jenkins as a folder jobs/. Under jobs/ there's a folder for each job, named as your job name, and everything relating to your job is stored under this folder. Here you'll find config.xml for the configuration, a builds/ folder for the different builds that have been done, and workspace/ which is where Jenkins does the actual building of your source code.

So what you need is to add eZ Publish to the job folder, I did so under jobs/<job name>/ezpublish/. Then you need to symlink the eZ Publish extension and settings folders to the ../workspace/ folder, where Jenkins will set up your project repository files. Note that your repository need to include the NovenINIUpdate extension for this to work. A database is not required, NovenINIUpdate will run without it.

After eZ Publish is set up, back in the Jenkins job configuration I can add the following "Build" "Execute shell" script:

 cd ../ezpublish && /usr/bin/php extension/noveniniupdate/bin/php/noviniupdate.php --env=staging && cd -

This script will change directory to our new eZ Publish installation, run novenini to set correct ini settings on our repository files (provided you set up the symlinks correctly) and return Jenkins to the workspace/ folder again. Mission accomplished. After that, we can deploy code with the correct settings for the given environment (staging in this example).

Unit tests

I have chosen to run unit tests on the staging server from the deploy script, to avoid installing everything required on the Jenkins server. It's not a perfect solution, since the changes will be deployed even if a unit test fails. But Jenkins will display a nice, red button when a unit test fails, and this works well enough for us.

Server side deployment script

My complete server side deployment script looks something like this:

 #!/bin/bash
# Script to do an automatic deploy of project
# Expects to find a tar ball of code to deploy (script parameter 1)
# Will selectively clear required template cache
# Be aware: If translations files have been changed, you might need to clear all cache manually!
 
# Detection of template files in Jenkins:
# grep \.tpl ../builds/${BUILD_NUMBER}/changelog.xml | sed 's/.*\/\(.*\)\.tpl/\1-\*/' > template_changes.txt
# Detection of translation files in Jenkins:
#  grep translation\.ts ../builds/${BUILD_NUMBER}/changelog.xml | sed  's/.*\/\(.*\/translation\.ts\)/\1/' > translation_changes.txt
 
FILE=$1
PHP='/usr/bin/php'
 
# Aborting script if any command fails
set -e
cd ~
 
if [ ! -e $FILE ]; then
    echo "Deploy tar ball missing or unspecified!"
    exit 1
fi
 
# Unpack deployment package
echo "Unpacking files to deploy..."
rm -rf deploy_workspace previous_deploy_workspace
mkdir deploy_workspace
tar -zxf $FILE -C deploy_workspace
 
echo "Update live source code"
mv www/git previous_deploy_workspace
mv deploy_workspace www/git
# Restarting Apache to avoid PHP autoload problems
apachectl restart
 
# Clear relevant compiled templates
cd www
template_list=`cat git/template_changes.txt`
if [ -d var/ezflow_site/cache/template/compiled ]; then
    cd var/ezflow_site/cache/template/compiled
    for i in $template_list; do
        echo Clearing compiled templates for $i
        rm -f $i
    done
    cd -
fi
 
translation_list=`cat git/translation_changes.txt`
if [[ ! $translation_list = '' ]]; then
    echo 'Clearing translations cache (might require manual clearing)'
    $PHP bin/php/ezcache.php --clear-id=translation
fi
 
# Clear possible other lightweight cache that might need it
$PHP bin/php/ezcache.php --clear-tag=ini
$PHP bin/php/ezcache.php --clear-id=template-override,template-block,content
 
# Run unit tests (should be enabled on staging server only)

Deployment parameters

In the Jenkins job configuration you can add parameters to the deploy job, using the "This build is parameterized" setting. I have used this for a few different purposes. First off, I added a general parameter to the production deploy job for a single reason, namely to give the unfortunate developer that clicked the wrong deploy button a chance to reconsider. For staging server we do not need such a safe guard, so there the deploy is running on one click.

A second use of parameters is to give the developer an option of specifying which git tag or branch to deploy. I added a string parameter called GIT_TAG, and used the following value in the git configuration section at "Branch Specifier": $GIT_TAG.

Last one is to add checkboxes for different deploy options. I use these in the next section, when considering how much to update from production to staging environment.

Mirror prod to staging

In addition to deploying modified code, I found that Jenkins can also be used to run an update of staging data from production server. The advantage is that I seldom need to log in to the servers, I just use Jenkins for everything. And we know the update will be done in the same way every time, making it predictable.

The job configuration here need no repository access, and no build steps. All we do is run a script on the staging server, with a couple of checkbox "Choice" parameters. The parameters are called MIRROR_STORAGE and EZFIND_INDEX. $MIRROR_STORAGE is used to specify if the script should copy the storage folder from prod to stage, to update images. $EZFIND_INDEX is used to specify if the script should run eZ Find reindexing, to update the search. The reason to make these operations optional is that they take a lot of time. You might not want to do this on every update. I use the post processing sql script to empty notification tables, for instance, or set a different admin password on the staging environment.

The actual script looks as follows:

 #!/bin/bash
# Script to mirror database and storage from production server to stage server in a safe manner
 
# Two optional parameters:
# <mirror> - 'yes' to mirror storage/ folder (default is to prompt for it)
# <index> - 'yes' to update eZ Find search index (default is to prompt for it)
 
PHP='/local/bin/php-cli'
STAGE_DB_PASS='<stage_db_password'
PROD_DB_PASS='<prod_db_password'
 
mirror=$1
index=$2
 
# Aborting script if any command fails
set -e
 
echo ''
echo 'Updating staging server based on production data!'
echo 'Run from eZ Publish root folder!'
echo 'Make sure that all changes to staging database have been documented or mirrored to prod before running this!'
echo ''
echo 'Getting prod database...'
 
# Dump production database, backup stage database, and import prod in stage database
ssh user@production.server.net mysqldump -u<user> -p$PROD_DB_PASS <database> > latest_prod_database.sql
mysqldump -u<user> -p$STAGE_DB_PASS <database> > latest_stage_database.sql
echo 'Databases dumped, now importing data...'
mysql -u<user> -p$STAGE_DB_PASS <database> < latest_prod_database.sql
 
# Run post processing sql
mysql -u<database -p$STAGE_DB_PASS <database> < extension/<project>/sql/stage_post_processing.sql
$PHP bin/php/ezcache.php --clear-all
 
echo ''
if [[ $mirror = '' ]]; then
    read -p 'Do you wish to mirror storage/ folder (images etc.)? [yes/no]:' mirror
fi
 
if [[ $mirror = 'yes' ]]; then
    echo 'Mirroring storage/ folder...'
    rsync -r -l --exclude '/trashed' nsj-jenk@nsj-app01.osl.basefarm.net:www/var/ezflow_site/storage/ var/ezflow_site/storage
    chmod -R a+rwx var/ezflow_site/storage -f
    $PHP bin/php/ezcache.php --clear-all
fi
 
echo ''
if [[ $index = '' ]]; then
    read -p 'Do you wish to reindex eZ Find? [yes/no]:' index
fi
 
if [[ $index = 'yes' ]]; then
    echo 'Reindexing eZ Find...'
    $PHP extension/ezfind/bin/php/updatesearchindexsolr.php --php-exec=$PHP -s superadmin --clean --quiet
    $PHP bin/php/ezcache.php --clear-all
fi
 
# Cleanup
echo 'Packing database dumps...'
gzip -f latest_prod_database.sql
gzip -f latest_stage_database.sql

Summary

Jenkins have a high number of deploy job options, that can be used in an infinite number of ways. I would like to hear from you if you have set this up differently, or have set up other features. For more reading, there are many other possibilities in the Jenkins PHP site[3], as well as the PHP Quality Assurance tools site[4]. I'm considering adding Checkstyle for checking the php coding style, and PHP Mess Detector for analyzing the code structure, but haven't gotten around to this yet.

[1] Jenkins: http://jenkins-ci.org/
[2] Private/public key pairs: http://www.ece.uci.edu/~chou/ssh-key.html
[3] NovenINIUpdate: http://projects.ez.no/noveniniupdate
[4] Jenkins PHP templates: http://jenkins-php.org/
[5] PHP QA tools: http://phpqatools.org/

Proudly Developed with from