We ran into some kind of concurrency problems with JDBC pools and thread where Glassfish appeared to be hanging. One approach is to create thread dumps for the JVM.
With Glassfish we have a few options:

1. jstack
   Run jps which returns you the list of applications running a JVM, choose the PID and execute
   jstack <PID> or jstack -F <PID> > td.log
2. asadmin
   Go the Glassfish_HOME/bin folder and execute
   ./asadmin --user admin generate-jvm-report --type=thread > threaddump.txt
3. kill -3 <PID>
   Supposed to create a dump in the default log folder of Glassfish. Doesnt work for me.

With the (or better more than 1) file at hand you can evaluate them by hand or use some of the tools around. I am still struggling to make the tool analyzing my dumps. They simply open the files like a editor.

tda tool

Some References

• Samurai Analyzer
• TDA
• IBM dev community tool
• http://planet.jboss.org/post/simple_tools_to_analyze_thread_dumps
• https://sites.google.com/site/threaddumps/java-thread-dumps
• http://architects.dzone.com/articles/how-analyze-java-thread-dumps
• http://www.javacodegeeks.com/2012/03/jvm-how-to-analyze-thread-dump.html

When connecting to a Ubuntu server with ssh you need to provide the keyfile (somekeyfile.pem) that you can download when creating the key pair.

Key file

This 2048 bit key is required to login as regular ubuntu user. What I dislike is the fact that this user can sudo all, so once someone manage to get into you user account, he has root access too. I recommend to set a password for the ubuntu user and change the sudoers configuration.

Change the password for user ubuntu

Open the sudoers include file

sudo vi /etc/suderos.d/90-cloudimg-ubuntu or sudo vi /etc/sudoers

change last line from

ubuntu  ALL=(ALL) NOPASSWD:ALL

to

ubuntu ALL=(ALL) ALL

With JMX (Java Management Extensions, Wikipedia) at hand, there are a few options to choose from.

JConsonsole

The graphical monitoring tool is great for local deployment, it allows you to connect to a JVM on the same host or a remote host. It creates line graphs for your for all relevant from the moment you connect, it is perfect to observe a server while you do some testing or other actions, though it does not record any values while you are not connected. I have a hard time to get it running on a remote server and I do not favour the ‘open’ approach (see previous blog entry) which allows anyone to access the JVM with the disabled authentication settings. I also had situations where the JVM was frozen and it was no longer possible to access the JVM for monitoring, here I would rather have snapshots before the problem started together with server.log.

JConsole

Glassfish Rest Interface

Note: You need to enable the areas you want to monitor with the admin console (or the asadmin command line) because per default all are OFF.

Glassfish Monitoring Settings

Then you access the monitoring values from the browser, curl or another application (authorization required!) through the URI, like https://localhost:4848/monitoring/domain/server/jvm/memory
This also very useful to explore the available values for the asadmin approach (next option)

Restful through browser

Restful through browser

asadmin commandline

You can access all these telemetry stuff also through the asadmin command line ($GLASSFISH_HOME/bin), this comes handy for administrator who want to run cronjobs or bash scripts to take snapshots.

asadmin

./asadmin –user myadminuser –passwordfile /opt/passwordfile.txt get –monitor server.jvm.thread-system.*
Note: you need to pass admin user and password, the passwordfile contains the password in clear text:
AS_ADMIN_PASSWORD=mysupersecretpassword

glassfish@ip-10-10-10-10:/opt/app-server/bin# ./asadmin --user admin --passwordfile ~/pwd.txt get --monitor server.jvm.memory.*
server.jvm.memory.committedheapsize-count-count = 3914924032
server.jvm.memory.committedheapsize-count-description = Amount of memory in bytes that is committed for the Java virtual machine to use
server.jvm.memory.committedheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.committedheapsize-count-name = CommittedHeapSize
server.jvm.memory.committedheapsize-count-starttime = 1364197189330
server.jvm.memory.committedheapsize-count-unit = bytes
server.jvm.memory.committednonheapsize-count-count = 259194880
server.jvm.memory.committednonheapsize-count-description = Amount of memory in bytes that is committed for the Java virtual machine to use
server.jvm.memory.committednonheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.committednonheapsize-count-name = CommittedNonHeapSize
server.jvm.memory.committednonheapsize-count-starttime = 1364197189330
server.jvm.memory.committednonheapsize-count-unit = bytes
server.jvm.memory.dotted-name = server.jvm.memory
server.jvm.memory.initheapsize-count-count = 4244635648
server.jvm.memory.initheapsize-count-description = Amount of memory in bytes that the Java virtual machine initially requests from the operating system for memory management
server.jvm.memory.initheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.initheapsize-count-name = InitialHeapSize
server.jvm.memory.initheapsize-count-starttime = 1364197189330
server.jvm.memory.initheapsize-count-unit = bytes
server.jvm.memory.initnonheapsize-count-count = 24313856
server.jvm.memory.initnonheapsize-count-description = Amount of memory in bytes that the Java virtual machine initially requests from the operating system for memory management
server.jvm.memory.initnonheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.initnonheapsize-count-name = InitialNonHeapSize
server.jvm.memory.initnonheapsize-count-starttime = 1364197189330
server.jvm.memory.initnonheapsize-count-unit = bytes
server.jvm.memory.maxheapsize-count-count = 3914924032
server.jvm.memory.maxheapsize-count-description = Maximum amount of memory in bytes that can be used for memory management
server.jvm.memory.maxheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.maxheapsize-count-name = MaxHeapSize
server.jvm.memory.maxheapsize-count-starttime = 1364197189330
server.jvm.memory.maxheapsize-count-unit = bytes
server.jvm.memory.maxnonheapsize-count-count = 587202560
server.jvm.memory.maxnonheapsize-count-description = Maximum amount of memory in bytes that can be used for memory management
server.jvm.memory.maxnonheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.maxnonheapsize-count-name = MaxNonHeapSize
server.jvm.memory.maxnonheapsize-count-starttime = 1364197189330
server.jvm.memory.maxnonheapsize-count-unit = bytes
server.jvm.memory.objectpendingfinalizationcount-count-count = 0
server.jvm.memory.objectpendingfinalizationcount-count-description = Approximate number of objects for which finalization is pending
server.jvm.memory.objectpendingfinalizationcount-count-lastsampletime = 1364268183797
server.jvm.memory.objectpendingfinalizationcount-count-name = ObjectsPendingFinalization
server.jvm.memory.objectpendingfinalizationcount-count-starttime = 1364197189330
server.jvm.memory.objectpendingfinalizationcount-count-unit = count
server.jvm.memory.usedheapsize-count-count = 1841024784
server.jvm.memory.usedheapsize-count-description = Amount of used memory in bytes
server.jvm.memory.usedheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.usedheapsize-count-name = UsedHeapSize
server.jvm.memory.usedheapsize-count-starttime = 1364197189330
server.jvm.memory.usedheapsize-count-unit = bytes
server.jvm.memory.usednonheapsize-count-count = 236054792
server.jvm.memory.usednonheapsize-count-description = Amount of used memory in bytes
server.jvm.memory.usednonheapsize-count-lastsampletime = 1364268183797
server.jvm.memory.usednonheapsize-count-name = UsedNonHeapSize
server.jvm.memory.usednonheapsize-count-starttime = 1364197189330
server.jvm.memory.usednonheapsize-count-unit = bytes

Command get executed successfully.

You can use wildcards, but be aware, server.* can create easily 50.000 + lines, even server.jvm.* 5.000 + lines.
Only one parameter can be passed.
An option is to snapshot this by a cronjob into a datetime stamped file every 1 or 5 minutes.

Further Options

• Implement your own monitoring as part of your deployment, make sure your persist into a database or logfiles, so the information is accessible even the JVM is down.
  Pro: You can implement to whatever extend you want.
  Con: Learning curve for JMX.

  http://weblogs.java.net/blog/kalali/archive/2010/04/06/manage-administrate-and-monitor-glassfish-v3-java-code-using-amx-jmx

  http://docs.oracle.com/cd/E19575-01/821-0031/gcgvu/index.html

• Use an existing solution that can be deployed into the application server
  The only application I could find is LightFish by Adam Bien

  http://lightfish.adam-bien.com/

Even the Glassfish Admin Console allows to change the ports (Configurations/Server Config/Network Config/Network Listener), certain server OS such as Ubuntu do not allow non-root users (you should run Glassfish as separate user !) to ports below 1024. We can achieve this by port rerouting with the iptables command (under Ubuntu)

iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 8181
iptables-save -c > /etc/iptables.rules
iptables-restore < /etc/iptables.rules

vi /etc/network/if-pre-up.d/iptablesload
#!/bin/sh
iptables-restore < /etc/iptables.rules
exit 0

Additionally you can get a proper SSL certificate to stop annoying the user with a no proper certificate warning. See previous tutorial here.

SSL Error (Chrome)

If you operate an enterprise application with a known URL to the users, unlike a regular website where the portal should be reached with regular http, I would completely disable regular http.

Disable http

Let me summarize the key steps here:

• Get yourself a GoDaddy account
• Order a standard SSL
  

  Standard SSL

   

• After payment you get 1 credit to use. Before we create the certificate we need to create a certificate request with the keytools for our glassfish domain
• Change domain masterpassword (not the admin password)
  The default password is ‘changeit’, better change that with /bin$ ./asadmin change-masterpassword yourdomain
• Go to you domain config folder ~/glassfish/domains/yourdomain/config$
• We will use the default keystore file in the config folder keystore.jks
• Optional: List the existing certifcates, there should be 2.
  keytool -list -keystore keystore.jks or keytool -list -v -keystore keystore.jks (more info)
  You should get something like this

  Keystore type: JKS
  Keystore provider: SUN
  
  Your keystore contains 2 entries
  
  glassfish-instance, Mar 28, 2011, PrivateKeyEntry,
  Certificate fingerprint (SHA1): AA:AA:AA:AA:AA:AA:BD:9D:85:5B:CF:AA:AA:86:BA:49:50:AA:AA:AA
  s1as, Mar 28, 2011, PrivateKeyEntry,
  Certificate fingerprint (SHA1): 95:B4:E4:55:7F:F6:AA:AA:AA:AA:3D:65:21:D1:1B:AA:AA:AA:AA:AA
  

• Create a key
  keytool -genkey -keysize 2048 -genkey -alias sub.whateverdomain.com -keyalg RSA  -keystore keystore.jks
  This differs from the previous command, you dont use the -dname option any longer, but the keytool will ask interactivelyImportant: It asks for your last and first name, this is the CN tag. You MUST use the domain name you register for (sub.whateverdomain.com). The alias I would choose the same.
  

  keytool

• Create a certificate request file
  keytool -certreq -alias sub.whateverdomain.com -keystore keystore.jks -file whateverdomain.csr
• Open the file with the editor
  

  Request File

   

• Now we need to go back to GoDaddy to create our certificate with the request string
  

  SSL Setup

  

  SSL Setup

   

• Enter the request string from the above file
  

  SSL Setup

   

• Choose validation method
  If you are the contact in the whois record, email verification is the easiest option.
  

  SSL Validation

  

  SSL Admin

   

• Confirm the email
  

  Validation

  

  Validation

   

• Go to the SSL admin page
  

  SSL Console

  Download the certificate
  Choose ‘other’
  

   

• Upload the 2 cert files from the zip to the domain config folder
  gd_bundle.crt
  sub.whateverdomain.com.crt
  and import them
  keytool -import -trustcacerts -alias root -file gd_bundle.crt -keystore keystore.jks
  keytool -import -trustcacerts -alias demo.airtilus.net -file sub.whateverdomain.com.crt -keystore keystore.jks
• Go to the Glassfish admin console
  

  Glassfish admin

  and adjust nickname (alias, here sub.whateverdomain.com) and the keystore file (here keystore.jks)

  

  http listener 2

   

• Restart is not required but I recommend.
• Optionally you can disable the http port 8080.
• Open any of your web applications at port 8181
  
  

  https

  

  https

   

• Voila !

Currently there are 2 books about D3 both from O’Reilly and both have a similar introductory focus.

Getting Started with D3

June 2012, 12.99 U$ (ebook)

The books does what its title promises, getting you started, It jumps right into D3 with sample applications and code. What I really like is the fact the author connects the visualizations to real life data (New York’s MTA transportation data) which makes the whole book more entertaining and tangible. It also provides a chapter about transition and interaction, even about layouts which make more exciting visualizations, like those we all know from the D3 websites sample page. Though it does not go into advanced details. At this reasonable price I would recommend the title.

Interactive Data Visualization for the Web

November 2012, 23.99 US (ebook)

This book is a bit more comprehensive than the first one, it starts with some more basic underlying technologies and provides the reader with an introduction to HTML, DOM, CSS and Javascripts. The chapters covering D3 are written lengthier providing slightly more details. It runs along the sample around a bar-charts and scatter-plots which turns dull after a while.  The early release I have seems to be incomplete, so I dont want to give a final verdict.

With D3 obviously getting more popular we will certainly see more books, hopefully covering advanced features and more visualization centric. I was asked if I like to write one but my D3 knowledge is way not comprehensive enough, I wish Mike Bostock would write one.

Post number 300 ! Thanks to the up to 1000 visitors a day.

Today Amazon announced the EBS Snapshot Copy Feature (across regions). The intention is certainly to allow easy migration of data to another region, as you can copy the snapshot, create a volume and attach it to an instance. I was curious to try if I can migrate my Ubuntu instance to another region and it worked. You can use both command-line as well the AWS web admin.

• Create a snapshot of a volume in your source region
  

  Create Snapshot

• Copy Snapshot
  Beware of the transfer charges (this is considered regular IN/OUT traffic!)
  

  Copy Snapshot

  

  Copy Snapshot

   

• Change to target region
• Create an instance of the same type as your source (not really important to be same sizing) with some local AMI
• Stop the new instance
• Detach the volume automatically created
  

  Detach Volume

• Create a volume from the copied snapshot
  Make sure it is in the same availability zone !
  

  Create Volume

• Attach it to the new instance as /dev/sda1
  

  Attach Volume

  

  Attach Volume

• Start instance et Voila !
  

  Server Log

I am not sure if there are any side-effects or if it work with other OS but my Ubuntu server boots as expected and works fine. Great !

Security: We will not use VNC, but NX. VNC is not secure (though can be tunnelled through SSH) and it works by sending compressed bitmaps of the screen, which is slower and less accurate than a NX server (X Server calls, Unix/Linux only)

Requirements: Amazon AWS account

Step-by-Step

• Log into your AWS account
• Optional: Create a security group with port 22 inbound only
  

  Security Group

  

  Port 22 only

• Optional: Create a new key pair and download it
  

  Keypair

• Find an AMI for your region (EBS recommended) at http://cloud-images.ubuntu.com/desktop/precise/current/
  I select an EBS 64 bit version for Singapore datacentre (ami-4695d614)
  

  Ubuntu AMI

• Create an instance
  Using the AMI we looked up and the security group and key pair we created. I recommend to choose a type at least m1.small, the micro instance not sufficient memory to run GNOME and other applications.
  

  New instance

  

  Instance Type

  

  Key Pair

  

  Security Group

• When instance is running connect with SSH
  eg. ssh -i desktopserver.pem ubuntu@ec2-xx-yy-zz.ap-southeast-1.compute.amazonaws.com
  (Dont forget to set the pem file permission to 400 !)
  I recommend to use an elastic IP and a dyndns entry if you want to use the instance long term.
  

  Running Instances

  

  SSH session

• Change to root user and install this package (NX server)

  sudo -i
  add-apt-repository ppa:freenx-team
  apt-get update
  aptitude install -y freenx
  

  The desktop package is already installed in this AMI, otherwise

  export DEBIAN_FRONTEND=noninteractive
  apt-get update
  apt-get install -y ubuntu-desktop
  

  Download the install script from https://launchpad.net/freenx-server with

  cd /var/lib/nxserver
  wget https://bugs.launchpad.net/freenx-server/+bug/576359/+attachment/1378450/+files/nxsetup.tar.gz
  gunzip *gz
  tar -xvf *tar
  

  Execute the script, I recommend to create a private key (not to share the key with the whole planet, even claimed to be secure)
  ./nxsetup –install

  root@ip-10-00-2-00:/var/lib/nxserver# ./nxsetup --install
  ------> It is recommended that you use the NoMachine key for
  easier setup. If you answer "y", FreeNX creates a custom
  KeyPair and expects you to setup your clients manually.
  "N" is default and uses the NoMachine key for installation.
  
  Do you want to use your own custom KeyPair? [y/N] y
  Setting up /etc/nxserver ...done
  Generating public/private dsa key pair.
  Your identification has been saved in /etc/nxserver/users.id_dsa.
  Your public key has been saved in /etc/nxserver/users.id_dsa.pub.
  The key fingerprint is:
  9b:0c:11:64:00:00:00:00:00:00:13:43:3a:61:fc:1b root@ip-10-148-2-238
  The key's randomart image is:
  +--[ DSA 1024]----+
  |   . +=+         |
  |    +.*.         |
  |      =o o       |
  |       .+        |
  |                 |
  |                 |
  +-----------------+
  Setting up /var/lib/nxserver/db ...done
  Setting up /var/log/nxserver.log ...done
  Adding user "nx" to group "utmp" ...done
  Setting up known_hosts and authorized_keys2 ...Unique key generated; your users must install
  
  /var/lib/nxserver/home/.ssh/client.id_dsa.key
  
  on their computers.
  done
  Setting up permissions ...done
  Setting up cups nxipp backend ...cp: `/usr/lib/cups/backend/ipp' and `/usr/lib/cups/backend/ipp' are the same file
  

• Copy the created key file to your local desktop
  cat /var/lib/nxserver/home/.ssh/client.id_dsa.key 

  root@ip-10-00-2-00:/var/lib/nxserver# cat /var/lib/nxserver/home/.ssh/client.id_dsa.key
  -----BEGIN DSA PRIVATE KEY-----
  MIIBuwIBAAKBgQClwGVwNjZophIgelkOtUIcZsuzvvmHEkGql27ctHj21/sb33BG
  ECHMJ/pwKkRlS4Xq9f9PvXj3QVW8yGOImM6oHHgKQze49JmSYDGfZ6o8UkNNQniL
  ....
  W6SJuJK2yX1OHZAY1AtRpaPkQcl3EtZqSdYzPf83Zz26zsbJ63QgFuhns/UahId2
  ...
  S2VvyiMh6rbJ0yoBwFxqi6Qc/NmgvIVT/6U6NTuMdLNMeE73EmvimwIVAJJVPYog
  c839CAl7vfZgpdZOSzZf
  -----END DSA PRIVATE KEY-----
  

• Important: NX cant handle the Gnome 3D desktop, you need to update with this, otherwise you get a Failed to load session “gnome-fallback”
  

  echo -e "\nCOMMAND_START_GNOME='gnome-session --session=ubuntu-2d'"|sudo tee -a /etc/nxserver/node.conf
  

• All Ubuntu AMI come with the user ubuntu and disabled password login, we need to change this:

  vi  /etc/ssh/sshd_config
  
  Change to yes
  # Change to no to disable tunnelled clear text passwords
  PasswordAuthentication yes
  
  and restart
  /etc/init.d/ssh restart
  
  set password
  passwd ubuntu
  

• Download and install a NX client from www.nomachine.com
  

  NX Client

   

• Run and Configure the client
  

  NX Client Config

  

  Key

   

• Connect to the instance et Voila !
  

  Connecting

  

  Desktop Session

I use EC2 instances for test, development, demo and also for deployment to production. Amazon offers different types of instances, ranging from a micro instance (613 MB Ram and 2 CPU units) to a full fledge Cluster Compute Quadruple Extra Large Instance (60GB RAM and 33 CPU units). Of course a different price and paid per hour usage, available anytime.

All on demand Linux instances (Singapore):

• Micro instance: U$ 0.02 per hour
• Medium instance: U$ 0.34 per hour
• High Mem/CPU instance: U$ 2.024 per hour

On top of this there are 3 different categories of instances (in contractual terms)

Some price comparison for a m1.Large instance we use for testing (7,5GB RAM and 4 CPU units)

• On Demand (any time without any contractual obligations, we are using them currently)
  $0.340 per Hour > 1 month U$ 244.80 (fulltime 24h)
• Reserved Instance (1 year term, one time payment U$ 276.00)
  U$ 0.196 per Hour > 1 month U$ 141.12 (3 months: U$ 699.36 vs on-demand U$ 734.40, 12 months: U$ 1969.44 vs. on-demand U$ 2937.60 = ~30% savings )
• Spot Instance (depends on availability, you bid on a price range, if price exceeds your limit your instance shuts down)
  U$ 0.04 per Hour (as of December 5th 2012) > 1 month U$ 28.80

The spot instance, almost at 10% of the on-demand price, is extremely attractive and I am using it as test server.
Not suitable for production or demo purpose though.

The reserved instance starts to break even after 3 months full-time usage !

In order not to pay for instances running idle (at night, weekend) they auto-shutdown and the user can start them in a self provision fashion (for test, demo or training).

Interesting enough, the price fluctuation is very different in the AWS regions. Lets look at a m1.large instance type in the Ireleand  versus Singapore datacentre.

AWS Ireland

 

AWS Singapore

Obviously Singapore customers are not into this bidding concept, it remains permanently at 4cts while for Ireland the price jumps up to several Dollars !

More information at:

• http://javadude.wordpress.com/2012/10/16/ec2-server-on-demand-without-elastic-ip/
• http://javadude.wordpress.com/2012/06/04/running-ftp-server-on-ec2-on-demand/
• http://aws.amazon.com/pricing/ec2/
• http://aws.amazon.com/ec2/instance-types/