Work in progress.

How is SSH used by GlassFish 3.1?

For most administrative operations the GlassFish Domain Admin Server (DAS) communicates directly with a running server instance, but for some operations, like creating or starting an instance, this is not possible. In those cases the DAS can use SSH to perform the needed operation on the remote systme.

What commands use SSH in GlassFish 3.1?

create-instance, start-instance, delete-instance, start-cluster

SSH is only needed if the instances being operated on are remote from the DAS.

What's a Node?

A Node represents a GlassFish installation on a host. It is a GlassFish configuration construct. Any system that will be hosting GlassFish instances must have a Node created for it. The system comes with one built-in node "localhost" that can be used for instances local to the DAS.

GlassFish 3.1 supports two types of nodes:

  • SSH nodes that are used for nodes that can be reached via SSH
  • CONFIG nodes that are used for nodes that are not reached via SSH (The "localhost" node is a CONFIG node)
How are Nodes created?

For SSH nodes you need to create them explicitly using create-noode-ssh.

CONFIG nodes can be created explicitly using create-node-config, or in some cases they can be auto-created when an instance is created using create-local-instance.

What happened to the node agent?

The node agent did not make the feature list for 3.1. A number of GlassFish customers had requested the ability to run GlassFish clusters without the node agent. So our emphasis was on providing the ability to do that, and then using the SSH support to provided some (but not all) of the capabilities that the v2 node agent provided.

I'm on Windows, how can I get an SSH provider?

We recommend Cygwin or MKS Toolkit. See section three at Using SSH to Manage Instance Lifecycles.

I just want to create a small cluster of instances on the same machine as the DAS. Do I need SSH? What's the easiest way to do that?

No SSH needed. Here is an example of a two instance cluster with both instances co-located with the DAS:

asadmin create-cluster c1
asadmin create-local-instance --cluster c1 instance1
asadmin create-local-instance --cluster c1 instance2
asadmin start-cluster c1
I don't what to use SSH. How can I create a cluster of remote instances?

If you do not want to use SSH for managing remote instances you can still create and manage them using the local version of asadmin commands. Also, due to an "auto-node creation" feature you don't have to explicitly create any nodes before hand. For example if you have this setup:

  • GlassFish installation on three systems: systemA, systemB, systemC
  • DAS running on systemA

Then you can create a cluster by doing this:

# On systemA
asadmin create-cluster c1
# Log into systemB
asadmin --host systemA create-local-instance --cluster c1 instanceB
asadmin start-local-instance instanceB
# Log into systemC
asadmin --host systemA create-local-instance --cluster c1 instanceC
asadmin start-local-instance instanceC

In this case you can't use start-cluster to start the cluster. You must use start-local-instance to start each instance (or use the asadmin create-service command to install the instance as a native service).

I can't get SSH public key authentication working. ssh(1) keeps asking for my password. What am I doing wrong?

It can be any number of issues, such as not having permissions to the private key file, an incorrect entry in the remote user's $HOME/.ssh/authorized_keys file, incorrect host name resolution, etc.  One option is to run the ssh client using the -v option for verbose debugging output (up to 3 -v's for increasing levels of detail) to help with locating the issue.  Another option is to run the sshd server with the -d option to get debugging output (up to 3 -d's for increasing level of detail).  

create-node-ssh is failing with "Illegal sftp packet len: 1651074913". What is going on?

Turns out this is a fairly common problem that can be encountered by sftp. It's caused by a startup file on the remote system (.bashrc, .profile, /etc/csh.chrc, .login, etc) that has a statement that outputs text messages on login (typically to be read by humans). This output corrupts the SFTP/SSH session since it is interpreted as a file-transfer protocol packet. Some details here: http://www.snailbook.com/faq/sftp-corruption.auto.html

Resolution is to make sure that none of your startup files display text for non-interactive shells. One way to test for an interactive shell is:

case $- in
*i*) # do things for interactive shell
;;
*) # do things for non-interactive shell
;;
esac
I'm using password authentication. How do I pass the SSH password to create-node-ssh?

You use the asadmin password file. To do this securely you should create a GlassFish password alias for the password first. See section 3.2 here: http://wikis.sun.com/display/GlassFish/3.1SSHAuthentication

I'm using public key authentication with an encrypted key. How do I pass the SSH key passphrase to create-node-ssh?

You use the asadmin password file. To do this securely you should create a GlassFish password alias for the key passphrase first. See section 2.2 here: http://wikis.sun.com/display/GlassFish/3.1SSHAuthentication

How many instances does start-cluster start in parallel?

start-cluster takes the size of the admin thread pool (50 by default) and divides that number in half to determine the max number of concurent instances it will start. So by default start-cluster will start up to 25 instances concurrently.