First hands-on Oracle Solaris 11

After Oracle took over SUN and they dropped OpenSolaris I stopped using or even looking at it. Now a customer is keen on going the Solaris way and I have a chance to get my hands on it (again). I dont have a SUN SPARC sitting here but Oracle continues to support i386 architecture and they have both the iso file with the text installer and  even a ready to run virtual appliance ready to download (link). Oracle released version 11 of Solaris on 2011 November 9. Did you know SUN’s first OS goes back to 1982 as Sun UNIX 0.7 ?

I am not sure what happen to Solaris Express 11 which disappeared completely from Oracle’s website. Will there be another Express version for “free”/development/evaluation use ?

Continue reading

Advertisements

Glassfish 3.1 – Clustering Tutorial Part2 (sessions)

In the previous part (link) I get you running with a simple Glassfish 3.1 cluster setup with 2 instances running on 2 nodes. Now we have a cluster setup and can deploy an application one time and it will run on both nodes, no big deal, it doesn’t get us anywhere. So in part 2 we will do some modifications to our virtualbox server setup and create a web application with sessions replicated to both instances.

Prerequisites:

  • The server and cluster setup from part 1 (link)

Preparation of host and Virtualbox guests:

In part 1 we used n1 and n2 as hostname, this creates trouble for this part. A key information for the sessions is the server-hostname (domain) and we cannot share sessions between totally different hosts.

  1. Update guest server hostname
    Change etc/hosts and /etc/hostname
    Server n1 becomes n1.test.com

    127.0.0.1    localhost
    127.0.1.1    n1.test.com
    127.0.0.1    n1.test.com
    192.168.56.102  n2.test.com
    

    Server n2 becomes n2.test.com

    127.0.0.1    localhost
    127.0.1.1    n2.test.com
    127.0.0.1    n2.test.com
    192.168.56.101  n1.test.com
    

    Remarks:

    • You can use any domain name (other than test.com, which I dont own btw).
    • The IP addresses must fit your own Virtualbox setup.
  2. Update your host
    The host running Virtualbox. Change /etc/hosts

    192.168.56.101  n1.test.com
    192.168.56.102  n2.test.com
    
  3. Check multicast
    The communication between the nodes is using multicast for the session replication. The Glassfish team gives us a tool to verify if your servers can “see” each other.
    Go to the bin folder of the Glassfish installation and execute ./asadmin validate-multicast on both nodes
    You should get feedback like this Continue reading

Glassfish 3.1 – Clustering Tutorial

Glassfish Clustering, after being absent from version 3, made its re-debut after 2.1 in the current version 3.1
I was eager to get my hands on and tried to make sense of some information from various sources (see reference).

Clustering is quite a sophisticated subject, which you dont need to cover during development time, but at some stage (deployment to production) you better off knowing how it works and verify your application runs in the clustering environment.

I compiled the most essential steps in this instant-15min-tutorial creating the most simple cluster: 2 nodes with 1 instance each, 1 node also runs the DAS.

Glassfish Cluster

Continue reading

Android goes x86 and Virtualbox

Didnt realize there are some guys porting the Android source to the x86 platform. You can install it (but it targets Asus EEE hardware) or just run it in a Virtualbox. Find out by yourself  and download the iso images here. Not sure how close the port is to the real thing, but the current unstable release covers Froyo ! I will also try to put it on a USB stick and try to start it on the new Asus EEE T101MT.

Android X86 in Virtualbox

Android X86 in Virtualbox

Creating an Ubuntu 10.04 AMI using a local VMWare

I am using Amazon EC2 and S3 now more often, and our architecture, development and deployment partially relies on Amazon. For example, we save artifacts from our Build-Server on S3 and deploy the application for trial and testing in the EC2 cloud. The level of control you have over your instances and buckets is just great, and new features (like VPC, SNS) are added frequently. The API allows me to remote control our infrastructure without using the browser.

No one can say, you wouldnt find a fitting Linux distribution on EC2. It seems there are myriads of AMI’s and almost all popular Linux distros are available for you to get started. But being a control-freak I prefer a slightly different approach. We create a virtual appliance in-house (our products runs out-of-the-box) and use the appliance for local development and tests. I maintain reference appliances knowing exactly which kernel and which packages are running. For large number deployment is essential that all instances are identical. Unfortunately there is no straightforward solution to “upload” your vmdk to EC2 (or any of the few other cloud/IAAS providers) that allow upload and expect it running (due to a couple of technical facts in the background that usually are transparent to a cloud user, eg. XEN specific kernels, etc).

Collecting some inputs and tutorials from various sources I tried to create my local UBUNTU 10.04 LTS server on VMWare Workstation (Player) and get it running as EC2 instance.

I summarize the process here.

Warning: I still face a major issue with the instance (created from the uploaded AMI) which can be started, but it is not possible to connect via SSH. I will updated this blog as soon I (hopefully with your help) find the solution.

Pre-Requirements:

  • VMWare Workstation/Player or VirtualBox
    It does not matter which tool you use because during the process with create a bundle “inside” the running server. We are not converting a vmdk file or similar (which is also possible)
    For this tutorial I assume you have it downloaded and installed (There is a 30 day trial version of VMWare Workstation available with some more features than the player).
  • Ubuntu 10.04 Server LTS (or any other version, recommend 8.04 or later)
    You installed the basic server as Virtual Machine and can login as root. The installation process is simple enough and not covered here.
  • AMAZON AWS account
    You have an active AWS account with access to S3 and EC2.

Tutorial Part A (getting keys and certificates from Amazon AWS)

  • Login into your AWS account and navigate to Account |Security Credentials
  • Take note of your Access Key and Secret Access Key
    Take note of your Account Number (at the top right under your name)
    (one keypair should be created by default when you create an AWS account) 

    AWS Access Keys

  • Create and Download X.509 Certificates
    Please read the warning: The private key can only be created and downloaded 1 time ! Download both to your desktop.
    1 Certificate File: cert-{some_random_key}.pem
    1 Private Key File: pk-{some_random_key}.pem 

    X.509 Certificates

  • Create a bucket in S3
    Please note the bucket name must be unique worldwide. You can use something like “mycompanyname.images” or similar.
    By default the bucket is private. 

    Create S3 bucket

Tutorial Part B (Preparation of the Ubuntu Server)
I assume you already installed a Virtual Machine with Ubuntu Server 10.04 (without any extra packages). All steps performed as root user (via sudo or you “change” to root with sudo -i)

  • Add a drive to the instance

    Virtual Machine

    Edit virtual machine settings | Add.. | Hard Disk | Create new virtual disk | SCSI | 10GB | Store as single file

    Virtual Machine Settings

  • Power on the virtual machine

    Virtual Machine

  • Mount the additional harddisk
    mkdir /disk2
    mkfs -t ext2 /dev/sdb
    mount /dev/sdb /disk2

    root@ubuntu:~# mkdir /disk2
    root@ubuntu:~# mkfs -t ext2 /dev/sdb
    mke2fs 1.41.11 (14-Mar-2010)
    /dev/sdb is entire device, not just one partition!
    Proceed anyway? (y,n) y
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    Stride=0 blocks, Stripe width=0 blocks
    655360 inodes, 2621440 blocks
    131072 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=2684354560
    80 block groups
    32768 blocks per group, 32768 fragments per group
    8192 inodes per group
    Superblock backups stored on blocks:
     32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
    
    Writing inode tables: done
    Writing superblocks and filesystem accounting information: done
    
    This filesystem will be automatically checked every 25 mounts or
    180 days, whichever comes first.  Use tune2fs -c or -i to override.
    root@ubuntu:~# mount /dev/sdb /disk2
    root@ubuntu:~# cd /disk2
    root@ubuntu:/disk2# ls
    lost+found
    root@ubuntu:/disk2#
    
  • Install SSH server
    Otherwise we cant access the instance later. It is also easier to work with ssh session connect to our local instance.
    apt-get install openssh-server
  • Install FTP Server
    We need to transfer files to our instance.
    apt-get install vsftpd
    Remember to configure /etc/vsftpd.conf
    write_enable=YES
    local_enable=YES
    and restart vsftpd
    service vsftpd restart
  • Disable the firewall
    ufw disable
    We configure the firewall with Amazon console
  • Install the EC2 AMI Tools
    apt-get install ec2-ami-tools
  • Transfer the 2 key files to /tmp
    with ftp from your local machine/desktop. In /tmp it will not be bundled with your AMI later.
  • Delete network info
    rm /etc/udev/rules.d/70-persistent-net.rules
  • Install ec2 kernel
    Make sure the universe entry in /etc/apt/sources.list is enabled.
    apt-get update
    apt-get install linux-image-ec2
    Do not restart !

    root@ubuntu:~# apt-get install linux-image-ec2
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following extra packages will be installed:
     linux-image-2.6.32-309-ec2
    Suggested packages:
     fdutils linux-ec2-doc-2.6.32 linux-ec2-source-2.6.32
    The following NEW packages will be installed:
     linux-image-2.6.32-309-ec2 linux-image-ec2
    0 upgraded, 2 newly installed, 0 to remove and 30 not upgraded.
    Need to get 19.2MB of archives.
    After this operation, 57.6MB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Get:1 http://sg.archive.ubuntu.com/ubuntu/ lucid-updates/main linux-image-2.6.32-309-ec2 2.6.32-309.18 [19.2MB]
    Get:2 http://sg.archive.ubuntu.com/ubuntu/ lucid-updates/main linux-image-ec2 2.6.32.309.10 [3,276B]
    Fetched 19.2MB in 51s (375kB/s)
    Selecting previously deselected package linux-image-2.6.32-309-ec2.
    (Reading database ... 28138 files and directories currently installed.)
    Unpacking linux-image-2.6.32-309-ec2 (from .../linux-image-2.6.32-309-ec2_2.6.32-309.18_i386.deb) ...
    Done.
    Selecting previously deselected package linux-image-ec2.
    Unpacking linux-image-ec2 (from .../linux-image-ec2_2.6.32.309.10_i386.deb) ...
    Setting up linux-image-2.6.32-309-ec2 (2.6.32-309.18) ...
    Running depmod.
    update-initramfs: Generating /boot/initrd.img-2.6.32-309-ec2
    Running postinst hook script /usr/sbin/update-grub.
    Generating grub.cfg ...
    Found linux image: /boot/vmlinuz-2.6.32-309-ec2
    Found initrd image: /boot/initrd.img-2.6.32-309-ec2
    Found linux image: /boot/vmlinuz-2.6.32-24-generic
    Found initrd image: /boot/initrd.img-2.6.32-24-generic
    Found linux image: /boot/vmlinuz-2.6.32-21-generic
    Found initrd image: /boot/initrd.img-2.6.32-21-generic
    Found memtest86+ image: /boot/memtest86+.bin
    done
    
    Setting up linux-image-ec2 (2.6.32.309.10) ...
    

    Resulting boot directory

    /boot

    Do not reboot. The new default kernel is the ec2 kernel, the virtual machine will NOT boot anymore !

  • Adjust default kernel in grub
    Edit your /boot/grub/grub.cfg (This is not good practice because any update-grub trashes your manual changes!)

    ...
    ### BEGIN /etc/grub.d/00_header ###
    if [ -s $prefix/grubenv ]; then
     load_env
    fi
    set default="2"
    if [ ${prev_saved_entry} ]; then
     set saved_entry=${prev_saved_entry}
     save_env saved_entry
    ...
    ### BEGIN /etc/grub.d/10_linux ###
    menuentry 'Ubuntu, with Linux 2.6.32-309-ec2' --class ubuntu --class gnu-linux --class gnu --class os {
     recordfail
     insmod ext2
     set root='(hd0,1)'
     search --no-floppy --fs-uuid --set ab6ee13e-e9c8-4654-aad1-a94c69906e11
     linux    /boot/vmlinuz-2.6.32-309-ec2 root=UUID=ab6ee13e-e9c8-4654-aad1-a94c69906e11 ro find_preseed=/preseed.cfg noprompt  quiet splash
     initrd    /boot/initrd.img-2.6.32-309-ec2
    }
    menuentry 'Ubuntu, with Linux 2.6.32-309-ec2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os {
     recordfail
     insmod ext2
     set root='(hd0,1)'
     search --no-floppy --fs-uuid --set ab6ee13e-e9c8-4654-aad1-a94c69906e11
     echo    'Loading Linux 2.6.32-309-ec2 ...'
     linux    /boot/vmlinuz-2.6.32-309-ec2 root=UUID=ab6ee13e-e9c8-4654-aad1-a94c69906e11 ro single find_preseed=/preseed.cfg noprompt
     echo    'Loading initial ramdisk ...'
     initrd    /boot/initrd.img-2.6.32-309-ec2
    }
    menuentry 'Ubuntu, with Linux 2.6.32-24-generic' --class ubuntu --class gnu-linux --class gnu --class os {
     recordfail
     insmod ext2
     set root='(hd0,1)'
     search --no-floppy --fs-uuid --set ab6ee13e-e9c8-4654-aad1-a94c69906e11
     linux    /boot/vmlinuz-2.6.32-24-generic root=UUID=ab6ee13e-e9c8-4654-aad1-a94c69906e11 ro find_preseed=/preseed.cfg noprompt  quiet splash
     initrd    /boot/initrd.img-2.6.32-24-generic
    }
    ...
    

    Change the line set default=”0″ to a different kernel, in this case to “2” (count like 0,1,2 the menu entries)
    Now you could reboot your virtual machine because it will boot the previous kernel (that one you configured in grub.cfg)
    If you reboot please reset your network again (rm /etc/udev/rules.d/70-persistent-net.rules)

    BEFORE you create the bundle you must set the default to “0” ! Otherwise the ec2 instance will not start up and immediately terminate.
    (afterwards you should set it back to “2” to continue your local virtual machine)

    Check the kernel:
    user@ubuntu:~$ uname -a
    Linux ubuntu 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:17:33 UTC 2010 i686 GNU/Linux

  • Find a kernel
    You can choose a kernel to run on ec2, but the kernel is a location dependent AKI-ID.
    The cloud market is very useful to find the right kernel
    Cloud Market
  • Create a Bundle to upload
    ec2-bundle-vol -c /tmp/cert-xxxxxxxxx.pem -k /tmp/pk-xxxxxxxxx.pem –user {account_number} -d /disk2 -r i386 –kernel aki-{kernel_id} –no-inherit
    Use your Account number that you retrieved earlier from the AWS console and the 2 key files that you transfered to virtual machine.
    Use the kernel ID that you looked up at the cloud market.
    Depending on your hardware this process can easily take 20min and longer (my reference Intel Core 2 Duo 8600) ! 

    ...
    root@ubuntu:/disk2# ec2-bundle-vol -c /tmp/cert-xxxxxx.pem -k /tmp/pk-xxxxxx.pem --user xxxxxx -d /disk2 -r i386 --kernel aki-70067822 --no-inherit
    Copying / into the image file /disk2/image...
    Excluding:
     /sys/kernel/debug
     /sys/kernel/security
     /sys
     /
     /proc
     /sys/fs/fuse/connections
     /dev/pts
     /dev
     /dev
     /media
     /mnt
     /proc
     /sys
     /etc/udev/rules.d/70-persistent-net.rules
     /etc/udev/rules.d/z25_persistent-net.rules
     /disk2/image
     /mnt/img-mnt
    1+0 records in
    1+0 records out
    1048576 bytes (1.0 MB) copied, 0.0069198 s, 152 MB/s
    mke2fs 1.41.11 (14-Mar-2010)
    warning: Unable to get device geometry for /disk2/image
    Bundling image file...
    Splitting /disk2/image.tar.gz.enc...
    Created image.part.00
    Created image.part.01
    Created image.part.02
    Created image.part.03
    ...
    Created image.part.52
    Created image.part.53
    Created image.part.54
    Created image.part.55
    Created image.part.56
    Created image.part.57
    Generating digests for each part...
    Digests generated.
    Creating bundle manifest...
    ec2-bundle-vol complete.
    
  • Upload the bundle
    ec2-upload-bundle -b my.ubuntu.image -m /disk2/image.manifest.xml -a {your access key} -s {your secret access key} –part8
    the –part parameter is optional in case your upload fails half way.
    This depends very much on your uplink speed !
    [codesyntax lang=”xml” title=”Bundle Upload”]

    Uploaded image.part.55
    Uploaded image.part.56
    Uploaded image.part.57
    Uploading manifest …
    Uploaded manifest.
    Bundle uploaded ompleted.
    [/codesyntax]
  • Register AMI
    Go to your AWS console and open the S3 folder to see the uploaded files 

    S3 bucket with uploaded image

    Goto the EC2 tab and select AMI
    Register new AMI
    Enter the path as {your bucket name}/image.manifest.xml

    Register new AMI

  • Create an instance and start it up

    New instance

    You should use a security group with the ports 22 open (and icmp if you want to ping)

    Security Group

Problem Solving
As I stated in the beginning, none of my instances really started up successfully due to various problems.

  • After starting up the instance, I can ping it but any attempt to ssh fails with connection refused.
    user@wanaka-ubuntu:~/Desktop/amazon$ ping ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com
    PING ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com (175.41.xxx.xxx) 56(84) bytes of data.
    64 bytes from ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com (175.41.xxx.xxx): icmp_req=2 ttl=51 time=103 ms
    64 bytes from ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com (175.41.xxx.xxx): icmp_req=3 ttl=51 time=39.7 ms
    ^C
    --- ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com ping statistics ---
    7 packets transmitted, 6 received, 14% packet loss, time 6015ms
    rtt min/avg/max/mdev = 15.663/53.436/103.023/32.123 ms
    user@wanaka-ubuntu:~/Desktop/amazon$ ssh -i xxxx.pem root@ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com
    ssh: connect to host ec2-175-41-xxx-xxx.ap-southeast-1.compute.amazonaws.com port 22: Connection refused
    

    Instance logfile see here.

  • The instance is created with root device as instance store not EBS. I would prefer EBS !

Remarks and Outlook

  • I believe cloud computing, despite not being a completely new concept (you remember dumb amber screens in the 70’s and 80’s) is still in its infancy today. Already very powerful, but we can expect more finetuning in the near future that will allows us to scale hardware on the fly for a running instance.
  • We also should expect more standards that allow seamless excahnge of virtual instances between your local deployment and the cloud, or between cloud providers.

How does Virtualbox help me in hacking with Netbeans, Glassfish and PostgreSQL ? (2)

Did you read yet Part 1 ? If not, I recommend to run through it here.

We started from the scratch with VirtualBox and created a new virtual machine running Ubuntu Server 9.10. The last step was to shutdown the virtual machine. Now it is time to clean up, update our server, finetune the configuration and install JDK and Glassfish.
Continue reading

How does Virtualbox help me in hacking with Netbeans, Glassfish and PostgreSQL ?

Using the Netbeans IDE together with Glassfish as application server is quite easy and comes out-of-the box with the Netbeans installer. To install PostgreSQL is not significantly harder to install on Linux (or Windows). Our team uses Ubuntu 9.10, Netbeans 6.8, Glassfish and PostgreSQL 8.4.2. If you work alone, it is usually no problem to develop and deploy locally but once you are embedded into a team, you better ensure that you deploy and test applications in the same environment. Very quick you will have Ubuntu updating the OS and PostgreSQL and someone might install Glassfish updates or even add optional modules, soon you have as many deployment environments as you have team-members (times 2). The straight forward answer would be to have all working with the same one server running the application server and the DB, but we still need the comfort of a local sandbox playground and in-dependency from being online or in the office. Certainly I do not advocate island development leading to a different codebase and varying DB’s. But after the tutorial you will agree with me our solution is a reasonable approach.

Our approach: We create a virtual server (with VirtualBox) that everyone is running on his/her desktop and also one instance on a central server. One team-member is responsible to create so-called raw virtual images of the server and documents new versions that are deployed to each desktop. Each team-member does nothing but starting the guest Server in VirtualBox (Glassfish and PostreSQL autostarting) and immediately can connect with Netbeans and pgadmin to AS and DB.

Virtual Ubuntu 9.10 Server

Not all developers are familar with setting up Ubuntu Server and VirtualBox, so I compiled this tutorial as a walk-through the necessary steps to get you started witthout knowing too much about  the OS nitty-gritty stuff. You want to concentrate on creating your application, not learning to hack the operating system (even I recommend to know as much as possible on Linux).
A remark on the naming: Your desktop or Notebook running VirtualBox is the HOST and the server or OS you run in the virtual machine is the GUEST.

Continue reading