GMS Group Discovery in Supported Environments

General Architecture

When a GMS member is started, it must contact an existing GMS member in order to join the group (or if it is the first member started, recognize that it's the only one and make itself the group master). When multicast is available, the member can broadcast a request and this request will reach the other group members, if any. Without multicast, the member must be given a list of running group members to contact individually. This document is concerned with obtaining/generating the configuration information needed to be passed into GMS for group discovery, and not the mechanics of dynamic GMS group formation.

In the multicast case, GMS must obtain the multicast address and port from the GlassFish configuration. This information is already present in GlassFish 3.1. In the non-multicast case, GMS must be given a list of addresses of existing members. For this list, we are adding a configuration property. In this document, I'll call it the "bootstrap list" of members.

Multicast address/port notes:

• Must be the same for each instance in the cluster. (The multicast address is not the address of the instance, but is more like a GMS Topic or CB radio channel.)

Bootstrap list notes:

• It does not need to be a complete list of group members.
• It does not need to be a correct list of group members (an instance could fail while a new one is starting).
• It does not need to contain the group master. For 3.2, we will make GMS work so that master requests can be forwarded from a non-master to the master.
• The bootstrap list could contain address and port information, but we need to be able to work without a specified port (see below). For this reason, GMS needs a consistent port available for each instance. Port 9090... TODO

Notes on using the multicast or bootstrap list parameters:

• The multicast params (address/port) and bootstrap list info are mutually exclusive. It is an error to have both specified in domain.xml. GMS uses the presence of one or the other to know in what type of environment it will be running.
• If neither are specified, one is automatically generated by the gms-adapter code in GF before starting GMS. To be consistent with 3.1, the default will be multicast address/port.
• To specify that the bootstrap list should be created instead of the multicast information, we need to add something. The way generation works is dependent on the environment in which we're running (PaaS anywhere, ad-hoc only in EC2 – see below), so this could be specified along with the new option. For instance, a param on asadmin create-cluster such as:
  1. A --nomulticast param with no other information (or pass in an empty bootstrap list). In this case, gms-adapter could query some environment variable to determine whether running in EC2 or not. TBD if this is possible or not. It could also check if AWS credentials are present in some known location.
  2. A --memberlistgenerator <das|ec2> parameter that specifies the generator used to create the member bootstrap list.
  3. A memberlistgenerator property that can be passed in with the current --properties param.

(We're not married to any of the above names.)

Generating the Multicast or Bootstrap List Values

These sections document how the multicast information or existing member (bootstrap list) locations will be generated based on the type of cluster and network options. This information is generated by the gms-adapter code in GlassFish, and then is passed to GMS at startup. Note that the multicast option is the same for both cases. The information below is only generated if not specified by the user.

In an environment in which we cannot support generation of these properties, a user can still set a property manually to run in an environment that would otherwise be unsupported. For an example, see the the ad-hoc, no multicast case below.

PaaS Case (Clusters work with a DAS)

When multicast is available:

When asadmin create-cluster is run, these values are generated randomly by the gms-adapter code. The multicast address and port must be the same for all members in the group (instances/DAS in the cluster). These values should be different for different clusters. This is already the case in GlassFish 3.1.

When multicast is not available:

The cluster instances will be known by the DAS, and this information will be available to a starting instance through the domain configuration information. When a cluster instance is started, it will be synchronized with the DAS in order to have this information available. The gms-adapter can iterate through the nodes listed in the domain in order to obtain the instance addresses and create the list (either IP address or host names). The GMS listener for each instance will have to be running on a known port.

When this list is created and passed to GMS, it is not written back to the domain config. This would force the list to be static and stop generation going forward. Rather, the bootstrap list is generated each time an instance starts based on the current list of nodes in the cluster.

Ad-hoc Case (Clusters work with no DAS)

When multicast is available:

When asadmin create-cluster is run, these values are generated randomly by the gms-adapter code. The multicast address and port must be the same for all members in the group (instances/DAS in the cluster). These values should be different for different clusters. This is already the case in GlassFish 3.1.

When multicast is not available:

When running in EC2, including any EC2 subset of environments, we can use the AWS API for obtaining running instances belonging to the current user. Given a set of AWS credentials, the API allows access to all running instances or to specific sets of them, for instance within an AutoScale group (effectively, a cluster).

This is still being researched, but there may be EC2 environment specific ways to obtain this info as well without resorting to using the AWS API. For instance, environment variables accessible to GlassFish. Still researching this. There's also the option to manually store/retrieve this information from S3, but we shouldn't have to do this.

Similarly to the PaaS case, when this list is created and passed to GMS, it is not written back to the domain config. This would force the list to be static and stop generation going forward. Rather, the bootstrap list is generated each time an instance starts based on the current list of running instances obtained from EC2.

Note that we could have a dependency on the AWS credentials of the user, and there should hopefully be some standard way to access these within GlassFish. (Some AWS environments let the user set these as environment variables through the AWS console.)

For OVM, we cannot support generating the bootstrap list automatically if there is no DAS and no multicast.