Amazon S3 plugin for Jenkins CI again

About once a year I revisit (link) this topic again (usually when the plugin causes trouble). Now I get this signature error

AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method., S3 Extended Request ID:..

The good news first:
The S3 plugin became mainstream, you can install it from the plugin page under Jenkins Administration | Plugin Manager. You dont need to build the plugin any longer by yourself and can skip the rest of this entry.

S3 Plugin

The long version:
It seems the error is caused by a ‘+’ sign in the access key troubling the encoding function used (see issue). The latest build (Sep 2012) should fix this problem.

If you want to build by yourself, you need to get the sourcecode from git and build the plugin file, beware as it requires Maven 3 now. Below instructions apply fro Ubuntu.

• sudo add-apt-repository http://ppa.launchpad.net/natecarlson/maven3/ubuntu
  from https://launchpad.net/~natecarlson/+archive/maven3
• sudo apt-get update
• apt-get install maven3
• Change to any folder and clone the plugin from https://github.com/jenkinsci/s3-plugin
  git clone https://github.com/jenkinsci/s3-plugin.git
• Build
  mvn3 install
• After a while of downloading dependencies you should get a hpi file for Jenkins

Upload plugin

 

 

Jenkins CI

I migrated (lie the majority if Hudson users) to Jenkins. The migration is easy and simple (backup, add jenkins deb in sources.list, install, the rest if fixed by Jenkins). All info for migrating users here. While Jenkins regained its frequent release cycle, former Hudson is still sitting on version 1.398.

Jenkins CI Book

I want to highlight the first (and only) book about Jenkins CI by John Smart (link): You can download it for free. Thanks !

Hudson Plugin’s of the day

I really enjoy using Hudson as CI platform every day. It is open, it is vastly extensible, has a great community support and not to forget the outstanding job by Kohsuke to create Hudson and lay out the foundation of this ecosystem. There is always more to learn, find new ways to do jobs with Hudson, maybe that is the reason why there is no book yet about Hudson. I just want to highlight 3 very simple plugins that make life easer:

• Build Timeout (link)
  I have some jobs that take a long time, mainly because they transfer artifacts to S3 and deploy applications to test and demo servers. Once in a while I run into timeouts or hanging batch scripts which will make the complete job haning. If you dont notice you start creating a build queue. This plugin terminates a job if it extends the configurable time limit.
  

  Build Timeout

• Time Stamper (link)
  With lon running jobs if would be helpful when which task started and how long it took to finish. The plugin does nothing but putting a timestamp in front of each line of the console output.
  

  Time Stamper

• FTP Publisher (link)
  For simple file transfers you might want to try this plugin instead of batch script coding the transfer.

Hudson saves Artifacts to Amazon S3

I recently start to work a lot with Amazon AWS by utilizing EC2 servers and using S3 as online storage. While you should do your maths using the servers to generously (it quickly becomes expensive if you create instances on the fly and keep them running), S3 usage is a rather innocent matter. Currently 1GB costs you 0.15 U$ per month (more details here), but please dont forget to look at data transfer charges as well (not longer free from November 2010). I use the service mainly for backup purpose, while I still rely on local backups once a month, the continuous backup happens on S3 (which might be safer than my backup harddisk). The data volume I transfer is small enough (not maintaining a filesharing server !) to keep the costs low.

I was looking for a way to transfer my Hudson artifacts to my S3 account and could not find a regular promoted plugin, almost creating a manual script as workaround, until I found a plugin, which you still need by yourself, but works perfectly. This tutorial (based on Ubuntu) is a hands-on walk-through from the scratch. Some steps are rather simple.

In this previous tutorial I described my approach of using artifacts as a backup media.

Update 2010-10-04: There is another updater in github who added functionality to the plugin: here the address git://github.com/d6y/hudson-s3.git

Quick Tutorial: Hudson and Artifacts

I really like using Hudson ! It is amazing what magic you can do with it, even some stuff is simple, like the “Archive the artifacts” feature. Dont you have the same situation: Some build running on some server will create an error that you can’t see in you current codebase. Would be nice to get exact the sourcecode and deployment that belongs to this build ? I keep the workspace (sourcecode excluding the build folder and .svn folder) with the 2 steps below.

Quick Tutorial: Netbeans + Selenium + Hudson (Part 2)

In part 1 of this tutorial (I recommend to walk through it first) we created a simple web application, set up a Hudson server to build it and test it with Selenium. But we left with one major flaw: Hudson build the war file, but the JUnit test the job triggered was tesing an application already deployed to a Glassfish instance. Today we wipe out this problem, setting up a proper sequence of building, deploying and testing the application.

Pre-Requirements (same as in part 1):

• Netbeans 6.9 (earlier versions certainly work as well)
• Glassfish
  A local Glassfish that comes with Netbeans is good enough. I am using a Glassfish 3.0.1 running in a VirtualBox.
• Hudson Server
  Download the war file from here. For the tutorial we will use a local standalone Hudson server.
• Selenium RC
  Download from here.
• Sourcecode Repository (SVN)
  In order to use Hudson, it is recommended to have a sourcecode repository like SVN to upload the code.

Tutorial:

Automatic Version Numbering in Web Applications with Hudson (Add-On)

In the 3 parts of the tutorial (part 1, part 2, part3) we setup a Netbeans/Web-Project/Hudson/ZK environment that creates and reads automatically version numbers. Unfortunately while building outside HUDSON, or better with your local Netbeans IDE, ANT can’t read the HUDSON variables, resulting in a Manifest like this:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.1
Created-By: 14.2-b01 (Sun Microsystems Inc.)
builduser: sven
version: ${env.BUILD_VERSION}
id: ${env.BUILD_ID}
tag: ${env.BUILD_TAG}
server: ${env.BUILD_URL}

And potentially the website will display ${env.BUILD_VERSION} as version. Not very professional. Of course a local build should not find its way to production system, but at least it should state it properly.

Automatic Version Numbering in Web Applications with Hudson (Part 3)

In this third and last part of the tutorial (part 1, part 2) we add the version reading feature to a ZK application, which is actually not a big difference to the regular jsp page that we created earlier.
It is also a good exercise how to add the ZK library to an existing web project.

Requirements:

• Netbeans 6.8
• Glassfish V3 Application Server
• Installed ZK 5.0 Plugin for Netbeans (link)

(Older Versions and non-EE6 work as well)

Automatic Version Numbering in Web Applications with Hudson (Part 2)

Lets continue with the application from part 1(link) wit a static MANIFEST.MF file to dynamically created attributes using the ANT manifest task.

Requirements for this tutorial

• Netbeans 6.8
• Glassfish V3
• Hudson (local running good enough, how-to’s on Hudson setup you find in my earlier Tutorials)

Tutorial Part 2

• Open the web application project from part 1
  

Automatic Version Numbering in Web Applications with Hudson (Part 1)

Creating a web application deployed to a multitude of servers (or even to 1 server only) will put you to the challenge to find out what are you actually looking at when opening its URL. At good old windows time you could “compile” a major/minor version string into the executable and at least check the exe-files properties. There is no equivalent if you open a website. You could choose to put a version manually at your page (the least advanced and error prone way) or you make your automatic build (CI) server (that also might deploy to test systems) to write a time and version stamp into a property file (which happens to be the MANIFEST.MF file which usually gets little to no attention) and make the web application self-aware of its version number and print it somewhere on the index page.
I came across a couple of online discussions with solutions using the InputStreamReader, but it didnt worked for me. Either it reads null or the manifest file of some other jar files in the classpath.

This tutorial cover 3 parts:

1. Create a web application that reads from the manifest file.
2. Extend the ant script and make Hudson adding the build number and ID into the war file.
3. Create a ZK application that reads from the manifest file.
   

Information on the manifest file

• The MANIFEST.MF is a file inside the META-INF directory which is optional. The Netbeans IDE creates it. Actuall ant is creating it. Guess it became de-facto standard without many people really looking at this file(s).
  By default it contains usually only one line “Manifest-Version: 1.0”
  References: link1, link2
• Infos on the ANT manifest task (link)

Requirements for this tutorial

• Netbeans 6.8
• Glassfish V3

  (earlier might work as well, but I did not test it)