Dead Glassfish Connections

Running a complex web application with sessions and connections to the DB, MQ and HTTP clients brings you sooner or later into a discussion or problem with connections that have not been properly terminated by Glassfish or the client. Depending on the time-out settings in the OS and in Glassfish you might face CLOSE_WAIT connections. What is causing this is topic by itself, also helping the system to get rid of it. At the end quite likely the problem lies somewhere in the sourcecode.

In my case we found a large number of CLOSE_WAITS to Glassfish with strange IP addresses (closed enterprise app but open to public web).

Here only some helpful scripts (Linux/Ubuntu):

Find the Glassfish PID

jps | grep ASMain | awk '{ print $1 }'

#get it into variable
pid=$(jps | grep ASMain | awk '{ print $1 }')

Show all socket connections

netstat -anp | grep ESTABLISHED
netstat -anp | grep CLOSE_WAIT

Show all connections to Glassfish

netstat -anp | grep CLOSE_WAIT| grep $(jps | grep ASMain | awk '{ print $1 }')
netstat -anp | grep ESTABLISHED| grep $(jps | grep ASMain | awk '{ print $1 }')

Still a bit hard to review



Now lets sort by IP and count it (ignoring the ports)

netstat -anp | grep CLOSE_WAIT| grep $(jps | grep ASMain | awk '{ print $1 }')|awk '{print $5}'|awk -F':' '{print $1}'|sort -k1|uniq -c

Creates a list like this (number of connections and the related ip address)

netstat summary

netstat summary

You can pack this int a cron job or even get it combined with a geolocation webservice to find the user.