AWS Services Inconsistency

AWS still puzzles me at times. While the range of products and services is so huge and there is hardly something you cant accomplish with AWS at hand, still you find little gaps which can annoy, see my earlier discussion around VPC and VPN.

SimpleDB

I though every service would be accessible through the admin console. While looking for an alternative to place log data I came across AWS SimpleDB, just to learn there is no function in the web admin console to control it. I even created a support issue believing the service was not enabled for my account. You have to go through local html files (aka scratchpad) to access the console.

AWS Admin Console

SimpleDB Console

 

AWS MFA

While I can handle the above case, I cant appreciate the approach AWS chooses with the MFA they actively promote. Is a very good feature and you can opt for a hardware based MFA or a bit simlpler, with the virtual device. But here comes the contradiction, due to geo restrictions you cant download the AWS MFA app to your Android device if you dont have an US AWS account. Seriously, how much sense does this make ? I confirmed this with the AWS support.

You could go for an alternative solution with the Google Authenticator, but this creates a dependency to another third part which I am not willing to add, my “contract” is between AWS and me.

Is it related to this: http://en.wikipedia.org/wiki/Export_of_cryptography_from_the_United_States ?

Hitting the wall with AWS and multiple VPN

We started to operate servers for our customers at AWS within one (root) account. Each customer environment is inside a separate VPC for maximum separation and ease of inter-instance access. Now with the second customer requiring a VPN connection I hit the wall with AWS the first time. You can’t have multiple VPC’s with VPN’s pointing to separate IP’s.
AWS support: “Unfortunately, due to routing restrictions in our regions we only allow a CGW IP to be used once per region.”

conflict among chosen gateway ip addresses

There is quite a number of discussions and threads around this, anyway without a straightforward technical solution (and installing software VPN is not suitable solution for me), I use a different workaround. I create additional accounts for every  customer and link them back to our root account for consolidated billing. As nice side-effect, billings becomes more transparent for every account linked to one of our customers.
More info: http://docs.aws.amazon.com/awsaccountbilling/latest/about/consolidatedbilling.html

Cloud Price War

It was a matter of time until Amazon AWS would react to the recent price reduction by Google. With effect of April 1st they cut prices for EC2 and S3 massively again (up to 65% on S3 and 40% on EC2). For now the customer is the winner, lets observe how it develops on the long term.

More info here..

Increase size and type of AWS EBS volume

I was offline for quite a while because shifting from one continent to another. But now regular posts should be rolling in again.

I am running a couple of instances in pre-production requirement mode and changed from a standard EBS volume to a IOPS volume for the DB instance or the volume with the DB files. I could not identify a reasonable increase of performance, maybe a misconception that IOPS volumes will boost performance, rather provide a defined and consistent random access I/O throughput. I must admit I did not use a value higher than 1000.

Billing IOPS

Some recommended reading:

• Increasing EBS Performance (by Amazon)
• Benchmarking (by Amazon)
• Even Stranger than Expected: a Systematic Look at EC2 I/O (by Scalyr)

I decided to return to a standard ESB volume for my database as its performance did not benefit from the IOPS type (the DB is not overly busy too).
You cant change type and size of an EBS volume on the fly.

Here the steps to achieve the same:

Enforce password for Ubuntu user on EC2 instances

Using linux (Ubuntu) instances on Amazon EC2 is a quite safe thing to do, at least measured by the security provided by the platform (security groups, ACL, physical security,..). I recommend reading their security site here. At the end of the day the server is only as secure as you configure it, if you choose to open all ports running services with their default configurations and password settings, Amazon can’t help you.

When connecting to a Ubuntu server with ssh you need to provide the keyfile (somekeyfile.pem) that you can download when creating the key pair.

Key file

This 2048 bit key is required to login as regular ubuntu user. What I dislike is the fact that this user can sudo all, so once someone manage to get into you user account, he has root access too. I recommend to set a password for the ubuntu user and change the sudoers configuration.

Change the password for user ubuntu

Open the sudoers include file

sudo vi /etc/suderos.d/90-cloudimg-ubuntu or sudo vi /etc/sudoers

change last line from

ubuntu  ALL=(ALL) NOPASSWD:ALL

to

ubuntu ALL=(ALL) ALL

Copy AMI between regions

Finally this feature is available and easy as the click of a button. While it was previously almost impossible and last year through snapshots only you can select any AMI and copy to another region. It makes my life much easier and I stop maintaining reference images for every region but can make use of one image only ! More info here.

AMI Transferral

Copy EC2 instance to another region

Is it finally possible ? While the AMI import tool is long awaited for but only available for Windows, it is rather a big hazzle to transfer manually (see this) any other OS ( my last attempt in 2010).

Today Amazon announced the EBS Snapshot Copy Feature (across regions). The intention is certainly to allow easy migration of data to another region, as you can copy the snapshot, create a volume and attach it to an instance. I was curious to try if I can migrate my Ubuntu instance to another region and it worked. You can use both command-line as well the AWS web admin.

• Create a snapshot of a volume in your source region
  

  Create Snapshot

Running Ubuntu 12.04 Desktop on EC2

To say it upfront: Usually there is no need to run an Ubuntu server with a desktop in the cloud. Whatever you do on the desktop you can do in a terminal too (assuming you dont want to use GIMP in the cloud). Here a little summary to get you started with a Precise Pangolin desktop running in the cloud.

Security: We will not use VNC, but NX. VNC is not secure (though can be tunnelled through SSH) and it works by sending compressed bitmaps of the screen, which is slower and less accurate than a NX server (X Server calls, Unix/Linux only)

Requirements: Amazon AWS account

Step-by-Step

• Log into your AWS account
• Optional: Create a security group with port 22 inbound only
  

  Security Group

  

  Port 22 only

Running EC2 spot instances

or ‘ Save ultimately more money with AWS’

I use EC2 instances for test, development, demo and also for deployment to production. Amazon offers different types of instances, ranging from a micro instance (613 MB Ram and 2 CPU units) to a full fledge Cluster Compute Quadruple Extra Large Instance (60GB RAM and 33 CPU units). Of course a different price and paid per hour usage, available anytime.

All on demand Linux instances (Singapore):

• Micro instance: U$ 0.02 per hour
• Medium instance: U$ 0.34 per hour
• High Mem/CPU instance: U$ 2.024 per hour

On top of this there are 3 different categories of instances (in contractual terms)

Some price comparison for a m1.Large instance we use for testing (7,5GB RAM and 4 CPU units)

• On Demand (any time without any contractual obligations, we are using them currently)
  $0.340 per Hour > 1 month U$ 244.80 (fulltime 24h)
• Reserved Instance (1 year term, one time payment U$ 276.00)
  U$ 0.196 per Hour > 1 month U$ 141.12 (3 months: U$ 699.36 vs on-demand U$ 734.40, 12 months: U$ 1969.44 vs. on-demand U$ 2937.60 = ~30% savings )
• Spot Instance (depends on availability, you bid on a price range, if price exceeds your limit your instance shuts down)
  U$ 0.04 per Hour (as of December 5th 2012) > 1 month U$ 28.80

The spot instance, almost at 10% of the on-demand price, is extremely attractive and I am using it as test server.
Not suitable for production or demo purpose though.

The reserved instance starts to break even after 3 months full-time usage !

In order not to pay for instances running idle (at night, weekend) they auto-shutdown and the user can start them in a self provision fashion (for test, demo or training).

Interesting enough, the price fluctuation is very different in the AWS regions. Lets look at a m1.large instance type in the Ireleand  versus Singapore datacentre.

AWS Ireland

 

AWS Singapore

Obviously Singapore customers are not into this bidding concept, it remains permanently at 4cts while for Ireland the price jumps up to several Dollars !

More information at:

• https://javadude.wordpress.com/2012/10/16/ec2-server-on-demand-without-elastic-ip/
• https://javadude.wordpress.com/2012/06/04/running-ftp-server-on-ec2-on-demand/
• http://aws.amazon.com/pricing/ec2/
• http://aws.amazon.com/ec2/instance-types/

S3 for Backup

I use S3 for all kind of backup purpose, assuming S3 will be always available and the data/files are secure. Amazon offers a encryption of storage, but at the end of the day they are in control and not you. This encryption is rendered useless the moment someone picks up your AWS keys (I am not jumping into more paranoid scenarios covered under the Patriot Act where the US government may access your data, since Amazon is an US based company). I prefer to encrypt at the source, here a 2-liner to add to your backup procedure (Linux). You can even combine this with a cronjob including the S3 transfer using the S3cmd toolkit.

Encrypt on the fly:
tar cz whichfile | openssl des3 -salt -out whichfile.tar.gz.enc -k mysecretpassword

Decrypt and uncompress:
openssl des3 -d -salt -in whichfile.tar.gz.enc -out whichfile.tar.gz -k mysecretpassword
tar -xf whichfile.tar.gz

More:

• www.openssl.org/docs/apps/openssl.html
• alan.blog-city.com/amazon_s3_encryption.htm
• s3tools.org/s3cmd