It was a matter of time until Amazon AWS would react to the recent price reduction by Google. With effect of April 1st they cut prices for EC2 and S3 massively again (up to 65% on S3 and 40% on EC2). For now the customer is the winner, lets observe how it develops on the long term.
s3
Amazon S3 plugin for Jenkins CI again
About once a year I revisit (link) this topic again (usually when the plugin causes trouble). Now I get this signature error
AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method., S3 Extended Request ID:..
The good news first:
The S3 plugin became mainstream, you can install it from the plugin page under Jenkins Administration | Plugin Manager. You dont need to build the plugin any longer by yourself and can skip the rest of this entry.
The long version:
It seems the error is caused by a ‘+’ sign in the access key troubling the encoding function used (see issue). The latest build (Sep 2012) should fix this problem.
If you want to build by yourself, you need to get the sourcecode from git and build the plugin file, beware as it requires Maven 3 now. Below instructions apply fro Ubuntu.
- sudo add-apt-repository http://ppa.launchpad.net/natecarlson/maven3/ubuntu
from https://launchpad.net/~natecarlson/+archive/maven3 - sudo apt-get update
- apt-get install maven3
- Change to any folder and clone the plugin from https://github.com/jenkinsci/s3-plugin
git clone https://github.com/jenkinsci/s3-plugin.git - Build
mvn3 install - After a while of downloading dependencies you should get a hpi file for Jenkins
ZK goes EC2 (Part 3)
The third part of the tutorial where I improve a few things. I will not walk through the complete code but highlight a few important points and give you the complete sourcecode at the end.
To recap, my requirements:
- I want to allow users in my company to start and stop instances on their own without them login to AWS console.
- Only specific instances are available to them.
- Avoid using elastic IP’s (you pay for them if they are not assigned)
- Make it configurable
The improvements in this version:
- Remove the hardcoded access keys and place them encrypted in a properties file.
- Only instances that are not protected can be started or stopped.
- Update DynDNS entries from the application
- Some cosmetic cleanup of the control panel
Amazon S3 plugin for Jenkins CI
About one year back I found this plugin, that saves your build artifacts to Amazon S3 storage. Unfortunately the plugin does not properly work with the current Jenkins versions and the original plugin is not maintained. Luckily someone forked and updated the plugin at github. You need to build the plugin by yourself with the help of maven, you still can refer to my original post, just one change is required to make it work, the settings for your ~/.m2/settings/xml need to be changed according to the plugin page description. Please be patient, maven is going to download a lot !
<settings> <pluginGroups> <pluginGroup>org.jenkins-ci.tools</pluginGroup> </pluginGroups> <profiles> <!-- Give access to Jenkins plugins --> <profile> <id>jenkins</id> <activation> <activeByDefault>true</activeByDefault> <!-- change this to false, if you don't like to have it on per default --> </activation> <repositories> <repository> <id>maven.jenkins-ci.org</id> <url>http://maven.jenkins-ci.org/content/groups/artifacts/</url> </repository> </repositories> <pluginRepositories> <pluginRepository> <id>maven.jenkins-ci.org</id> <url>http://maven.jenkins-ci.org/content/groups/artifacts/</url> </pluginRepository> </pluginRepositories> </profile> </profiles> </settings>
Otherwise you get errors like
[INFO] Scanning for projects... Downloading: http://download.java.net/maven/2/org/jenkins-ci/plugins/plugin/1.434/plugin-1.434.pom [INFO] Unable to find resource 'org.jenkins-ci.plugins:plugin:pom:1.434' in repository java.net2 (http://download.java.net/maven/2) Downloading: http://repo1.maven.org/maven2/org/jenkins-ci/plugins/plugin/1.434/plugin-1.434.pom [INFO] Unable to find resource 'org.jenkins-ci.plugins:plugin:pom:1.434' in repository central (http://repo1.maven.org/maven2) [INFO] ------------------------------------------------------------------------ [ERROR] FATAL ERROR [INFO] ------------------------------------------------------------------------ [INFO] Error building POM (may not be this project's POM).
ZK goes EC2 (Part 2)
Part 1 of this tutorial we concluded with an web application that displays all our instances and their status. In this part we will add some more features to control our instances.
Prerequisites:
- The project and environment from part 1
Tutorial (complete sourcecode at the end):
- Display the region endpoints and allow to select different one
We hardcoded our endpoint in the first version, if you run instances across the globe in the various AWS datacentres (US, Ireland, Singapore, Tokyo) we need to switch the endpoint easily.
Lets add one more listbox, that we hide in a ZK popup (we could use a combo listbox, but for the sake of playing with all the available ZK components I use the popup). Same concept add a listbox in the zul and a EC2 region list and a list model with customer renderer in our controller. Continue reading
ZK goes EC2 (Part 1)
Certainly not a big deal to deploy a ZK web application to a Amazon EC2 instance, but I needed a simple application that allows my team to start/stop our EC2 instance that we use for testing and demo without logging into the AWS account or using the Firefox plugin (both giving too many rights and are too complex for some business users).
I created this app to give my users (tester and trainer) a chance to start the servers without logging into AWS. I share this because the are no samples in the SDK file that cover the EC2 instances in detail.
In part 1 of this tutorial we will create a ZK application that displays the status of our instances in a list.
In part 2 we will add the start-stop instance function, in part 3 we tinker with IP addresses and DynDNS domains and in part 4 we let our web application time scheduled (EJB Timer) control the instances automatically.
Pre-Requirements:
- Netbeans 6.9 or higher (download here)
- The ZK plugin (download here)
- Amazon AWS account (using the free tier you can stay even play for free). Login here
- The AWS Java SDK (download here)
Tutorial Part 1:
- Create a new ZK Web Application ‘ZKEC2CloudControl’
Creating an Ubuntu 10.04 AMI using a local VMWare
I am using Amazon EC2 and S3 now more often, and our architecture, development and deployment partially relies on Amazon. For example, we save artifacts from our Build-Server on S3 and deploy the application for trial and testing in the EC2 cloud. The level of control you have over your instances and buckets is just great, and new features (like VPC, SNS) are added frequently. The API allows me to remote control our infrastructure without using the browser.
No one can say, you wouldnt find a fitting Linux distribution on EC2. It seems there are myriads of AMI’s and almost all popular Linux distros are available for you to get started. But being a control-freak I prefer a slightly different approach. We create a virtual appliance in-house (our products runs out-of-the-box) and use the appliance for local development and tests. I maintain reference appliances knowing exactly which kernel and which packages are running. For large number deployment is essential that all instances are identical. Unfortunately there is no straightforward solution to “upload” your vmdk to EC2 (or any of the few other cloud/IAAS providers) that allow upload and expect it running (due to a couple of technical facts in the background that usually are transparent to a cloud user, eg. XEN specific kernels, etc).
Collecting some inputs and tutorials from various sources I tried to create my local UBUNTU 10.04 LTS server on VMWare Workstation (Player) and get it running as EC2 instance.
I summarize the process here.
Warning: I still face a major issue with the instance (created from the uploaded AMI) which can be started, but it is not possible to connect via SSH. I will updated this blog as soon I (hopefully with your help) find the solution.
Pre-Requirements:
- VMWare Workstation/Player or VirtualBox
It does not matter which tool you use because during the process with create a bundle “inside” the running server. We are not converting a vmdk file or similar (which is also possible)
For this tutorial I assume you have it downloaded and installed (There is a 30 day trial version of VMWare Workstation available with some more features than the player). - Ubuntu 10.04 Server LTS (or any other version, recommend 8.04 or later)
You installed the basic server as Virtual Machine and can login as root. The installation process is simple enough and not covered here. - AMAZON AWS account
You have an active AWS account with access to S3 and EC2.
Tutorial Part A (getting keys and certificates from Amazon AWS)
- Login into your AWS account and navigate to Account |Security Credentials
- Take note of your Access Key and Secret Access Key
Take note of your Account Number (at the top right under your name)
(one keypair should be created by default when you create an AWS account) - Create and Download X.509 Certificates
Please read the warning: The private key can only be created and downloaded 1 time ! Download both to your desktop.
1 Certificate File: cert-{some_random_key}.pem
1 Private Key File: pk-{some_random_key}.pem - Create a bucket in S3
Please note the bucket name must be unique worldwide. You can use something like “mycompanyname.images” or similar.
By default the bucket is private.
Tutorial Part B (Preparation of the Ubuntu Server)
I assume you already installed a Virtual Machine with Ubuntu Server 10.04 (without any extra packages). All steps performed as root user (via sudo or you “change” to root with sudo -i)
- Add a drive to the instance
Edit virtual machine settings | Add.. | Hard Disk | Create new virtual disk | SCSI | 10GB | Store as single file
- Power on the virtual machine
- Mount the additional harddisk
mkdir /disk2
mkfs -t ext2 /dev/sdb
mount /dev/sdb /disk2root@ubuntu:~# mkdir /disk2 root@ubuntu:~# mkfs -t ext2 /dev/sdb mke2fs 1.41.11 (14-Mar-2010) /dev/sdb is entire device, not just one partition! Proceed anyway? (y,n) y Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 655360 inodes, 2621440 blocks 131072 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=2684354560 80 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Writing inode tables: done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 25 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override. root@ubuntu:~# mount /dev/sdb /disk2 root@ubuntu:~# cd /disk2 root@ubuntu:/disk2# ls lost+found root@ubuntu:/disk2#
- Install SSH server
Otherwise we cant access the instance later. It is also easier to work with ssh session connect to our local instance.
apt-get install openssh-server - Install FTP Server
We need to transfer files to our instance.
apt-get install vsftpd
Remember to configure /etc/vsftpd.conf
write_enable=YES
local_enable=YES
and restart vsftpd
service vsftpd restart - Disable the firewall
ufw disable
We configure the firewall with Amazon console - Install the EC2 AMI Tools
apt-get install ec2-ami-tools - Transfer the 2 key files to /tmp
with ftp from your local machine/desktop. In /tmp it will not be bundled with your AMI later. - Delete network info
rm /etc/udev/rules.d/70-persistent-net.rules - Install ec2 kernel
Make sure the universe entry in /etc/apt/sources.list is enabled.
apt-get update
apt-get install linux-image-ec2
Do not restart !root@ubuntu:~# apt-get install linux-image-ec2 Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: linux-image-2.6.32-309-ec2 Suggested packages: fdutils linux-ec2-doc-2.6.32 linux-ec2-source-2.6.32 The following NEW packages will be installed: linux-image-2.6.32-309-ec2 linux-image-ec2 0 upgraded, 2 newly installed, 0 to remove and 30 not upgraded. Need to get 19.2MB of archives. After this operation, 57.6MB of additional disk space will be used. Do you want to continue [Y/n]? y Get:1 http://sg.archive.ubuntu.com/ubuntu/ lucid-updates/main linux-image-2.6.32-309-ec2 2.6.32-309.18 [19.2MB] Get:2 http://sg.archive.ubuntu.com/ubuntu/ lucid-updates/main linux-image-ec2 2.6.32.309.10 [3,276B] Fetched 19.2MB in 51s (375kB/s) Selecting previously deselected package linux-image-2.6.32-309-ec2. (Reading database ... 28138 files and directories currently installed.) Unpacking linux-image-2.6.32-309-ec2 (from .../linux-image-2.6.32-309-ec2_2.6.32-309.18_i386.deb) ... Done. Selecting previously deselected package linux-image-ec2. Unpacking linux-image-ec2 (from .../linux-image-ec2_2.6.32.309.10_i386.deb) ... Setting up linux-image-2.6.32-309-ec2 (2.6.32-309.18) ... Running depmod. update-initramfs: Generating /boot/initrd.img-2.6.32-309-ec2 Running postinst hook script /usr/sbin/update-grub. Generating grub.cfg ... Found linux image: /boot/vmlinuz-2.6.32-309-ec2 Found initrd image: /boot/initrd.img-2.6.32-309-ec2 Found linux image: /boot/vmlinuz-2.6.32-24-generic Found initrd image: /boot/initrd.img-2.6.32-24-generic Found linux image: /boot/vmlinuz-2.6.32-21-generic Found initrd image: /boot/initrd.img-2.6.32-21-generic Found memtest86+ image: /boot/memtest86+.bin done Setting up linux-image-ec2 (2.6.32.309.10) ...
Resulting boot directory
Do not reboot. The new default kernel is the ec2 kernel, the virtual machine will NOT boot anymore !
- Adjust default kernel in grub
Edit your /boot/grub/grub.cfg (This is not good practice because any update-grub trashes your manual changes!)... ### BEGIN /etc/grub.d/00_header ### if [ -s $prefix/grubenv ]; then load_env fi set default="2" if [ ${prev_saved_entry} ]; then set saved_entry=${prev_saved_entry} save_env saved_entry ... ### BEGIN /etc/grub.d/10_linux ### menuentry 'Ubuntu, with Linux 2.6.32-309-ec2' --class ubuntu --class gnu-linux --class gnu --class os { recordfail insmod ext2 set root='(hd0,1)' search --no-floppy --fs-uuid --set ab6ee13e-e9c8-4654-aad1-a94c69906e11 linux /boot/vmlinuz-2.6.32-309-ec2 root=UUID=ab6ee13e-e9c8-4654-aad1-a94c69906e11 ro find_preseed=/preseed.cfg noprompt quiet splash initrd /boot/initrd.img-2.6.32-309-ec2 } menuentry 'Ubuntu, with Linux 2.6.32-309-ec2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os { recordfail insmod ext2 set root='(hd0,1)' search --no-floppy --fs-uuid --set ab6ee13e-e9c8-4654-aad1-a94c69906e11 echo 'Loading Linux 2.6.32-309-ec2 ...' linux /boot/vmlinuz-2.6.32-309-ec2 root=UUID=ab6ee13e-e9c8-4654-aad1-a94c69906e11 ro single find_preseed=/preseed.cfg noprompt echo 'Loading initial ramdisk ...' initrd /boot/initrd.img-2.6.32-309-ec2 } menuentry 'Ubuntu, with Linux 2.6.32-24-generic' --class ubuntu --class gnu-linux --class gnu --class os { recordfail insmod ext2 set root='(hd0,1)' search --no-floppy --fs-uuid --set ab6ee13e-e9c8-4654-aad1-a94c69906e11 linux /boot/vmlinuz-2.6.32-24-generic root=UUID=ab6ee13e-e9c8-4654-aad1-a94c69906e11 ro find_preseed=/preseed.cfg noprompt quiet splash initrd /boot/initrd.img-2.6.32-24-generic } ...
Change the line set default=”0″ to a different kernel, in this case to “2” (count like 0,1,2 the menu entries)
Now you could reboot your virtual machine because it will boot the previous kernel (that one you configured in grub.cfg)
If you reboot please reset your network again (rm /etc/udev/rules.d/70-persistent-net.rules)BEFORE you create the bundle you must set the default to “0” ! Otherwise the ec2 instance will not start up and immediately terminate.
(afterwards you should set it back to “2” to continue your local virtual machine)Check the kernel:
user@ubuntu:~$ uname -a
Linux ubuntu 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:17:33 UTC 2010 i686 GNU/Linux - Find a kernel
You can choose a kernel to run on ec2, but the kernel is a location dependent AKI-ID.
The cloud market is very useful to find the right kernel
- Create a Bundle to upload
ec2-bundle-vol -c /tmp/cert-xxxxxxxxx.pem -k /tmp/pk-xxxxxxxxx.pem –user {account_number} -d /disk2 -r i386 –kernel aki-{kernel_id} –no-inherit
Use your Account number that you retrieved earlier from the AWS console and the 2 key files that you transfered to virtual machine.
Use the kernel ID that you looked up at the cloud market.
Depending on your hardware this process can easily take 20min and longer (my reference Intel Core 2 Duo 8600) !... root@ubuntu:/disk2# ec2-bundle-vol -c /tmp/cert-xxxxxx.pem -k /tmp/pk-xxxxxx.pem --user xxxxxx -d /disk2 -r i386 --kernel aki-70067822 --no-inherit Copying / into the image file /disk2/image... Excluding: /sys/kernel/debug /sys/kernel/security /sys / /proc /sys/fs/fuse/connections /dev/pts /dev /dev /media /mnt /proc /sys /etc/udev/rules.d/70-persistent-net.rules /etc/udev/rules.d/z25_persistent-net.rules /disk2/image /mnt/img-mnt 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0069198 s, 152 MB/s mke2fs 1.41.11 (14-Mar-2010) warning: Unable to get device geometry for /disk2/image Bundling image file... Splitting /disk2/image.tar.gz.enc... Created image.part.00 Created image.part.01 Created image.part.02 Created image.part.03 ... Created image.part.52 Created image.part.53 Created image.part.54 Created image.part.55 Created image.part.56 Created image.part.57 Generating digests for each part... Digests generated. Creating bundle manifest... ec2-bundle-vol complete.
- Upload the bundle
ec2-upload-bundle -b my.ubuntu.image -m /disk2/image.manifest.xml -a {your access key} -s {your secret access key} –part8
the –part parameter is optional in case your upload fails half way.
This depends very much on your uplink speed !
[codesyntax lang=”xml” title=”Bundle Upload”]
…
Uploaded image.part.55
Uploaded image.part.56
Uploaded image.part.57
Uploading manifest …
Uploaded manifest.
Bundle uploaded ompleted.
[/codesyntax] - Register AMI
Go to your AWS console and open the S3 folder to see the uploaded filesGoto the EC2 tab and select AMI
Register new AMI
Enter the path as {your bucket name}/image.manifest.xml - Create an instance and start it up
You should use a security group with the ports 22 open (and icmp if you want to ping)
Problem Solving
As I stated in the beginning, none of my instances really started up successfully due to various problems.
- After starting up the instance, I can ping it but any attempt to ssh fails with connection refused.
user@wanaka-ubuntu:~/Desktop/amazon$ ping ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com PING ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com (175.41.xxx.xxx) 56(84) bytes of data. 64 bytes from ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com (175.41.xxx.xxx): icmp_req=2 ttl=51 time=103 ms 64 bytes from ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com (175.41.xxx.xxx): icmp_req=3 ttl=51 time=39.7 ms ^C --- ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com ping statistics --- 7 packets transmitted, 6 received, 14% packet loss, time 6015ms rtt min/avg/max/mdev = 15.663/53.436/103.023/32.123 ms user@wanaka-ubuntu:~/Desktop/amazon$ ssh -i xxxx.pem root@ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com ssh: connect to host ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com port 22: Connection refused
Instance logfile see here.
- The instance is created with root device as instance store not EBS. I would prefer EBS !
Remarks and Outlook
- I believe cloud computing, despite not being a completely new concept (you remember dumb amber screens in the 70’s and 80’s) is still in its infancy today. Already very powerful, but we can expect more finetuning in the near future that will allows us to scale hardware on the fly for a running instance.
- We also should expect more standards that allow seamless excahnge of virtual instances between your local deployment and the cloud, or between cloud providers.
Hudson saves Artifacts to Amazon S3
I recently start to work a lot with Amazon AWS by utilizing EC2 servers and using S3 as online storage. While you should do your maths using the servers to generously (it quickly becomes expensive if you create instances on the fly and keep them running), S3 usage is a rather innocent matter. Currently 1GB costs you 0.15 U$ per month (more details here), but please dont forget to look at data transfer charges as well (not longer free from November 2010). I use the service mainly for backup purpose, while I still rely on local backups once a month, the continuous backup happens on S3 (which might be safer than my backup harddisk). The data volume I transfer is small enough (not maintaining a filesharing server !) to keep the costs low.
I was looking for a way to transfer my Hudson artifacts to my S3 account and could not find a regular promoted plugin, almost creating a manual script as workaround, until I found a plugin, which you still need by yourself, but works perfectly. This tutorial (based on Ubuntu) is a hands-on walk-through from the scratch. Some steps are rather simple.
In this previous tutorial I described my approach of using artifacts as a backup media.
Update 2010-10-04: There is another updater in github who added functionality to the plugin: here the address git://github.com/d6y/hudson-s3.git