Dead Glassfish Connections

Running a complex web application with sessions and connections to the DB, MQ and HTTP clients brings you sooner or later into a discussion or problem with connections that have not been properly terminated by Glassfish or the client. Depending on the time-out settings in the OS and in Glassfish you might face CLOSE_WAIT connections. What is causing this is topic by itself, also helping the system to get rid of it. At the end quite likely the problem lies somewhere in the sourcecode.

In my case we found a large number of CLOSE_WAITS to Glassfish with strange IP addresses (closed enterprise app but open to public web).

Here only some helpful scripts (Linux/Ubuntu):

Find the Glassfish PID

jps | grep ASMain | awk '{ print $1 }'

#get it into variable
pid=$(jps | grep ASMain | awk '{ print $1 }')

Show all socket connections

netstat -anp | grep ESTABLISHED
netstat -anp | grep CLOSE_WAIT

Show all connections to Glassfish

netstat -anp | grep CLOSE_WAIT| grep $(jps | grep ASMain | awk '{ print $1 }')
netstat -anp | grep ESTABLISHED| grep $(jps | grep ASMain | awk '{ print $1 }')

Still a bit hard to review

netstat

netstat

Now lets sort by IP and count it (ignoring the ports)

netstat -anp | grep CLOSE_WAIT| grep $(jps | grep ASMain | awk '{ print $1 }')|awk '{print $5}'|awk -F':' '{print $1}'|sort -k1|uniq -c

Creates a list like this (number of connections and the related ip address)

netstat summary

netstat summary

You can pack this int a cron job or even get it combined with a geolocation webservice to find the user.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s