GlassFish Server Open Source Edition 3.1 - Shoal Group Management Service (GMS) for Runtime Clustering Services

Scope of the Project

Shoal GMS is a clustering framework that provides infrastructure to build fault tolerance, reliability and availability.
The clustering framework provides the following functionalities to Glassfish services and user applications.

• GMS event notification
  • register callback for GMS event notification
  • GMS runtime notifies when changes in group membership occur.
• Cluster-wide Messaging
  • send a message to one, sublist or broadcast to all members of cluster
  • register message callback to process messages sent by other group members.
• GMS status methods
  • list of all members or just list of CORE members
  • request a member status on any member in cluster
• GMS Group Configuration
  • configure multicast address and port used by a group
  • ability to configure gms listener BIND_INTERFACE_ADDRESS(ip address of network interface) to use on a multihome machine

Recent enhancement to GMS is to have a pluggable transport and a Grizzly implementation of that transport.
GF HA subsystem is building on top of GMS clustering framework and its messaging.

group-management-service Feature Status from glassfish issue tracker features targeted for GFv3.1 milestone

Feature Overview

GMS-01 Shoal GMS over Grizzly implementation

• Description

Replace JXTA as the transport provider for GMS and provide an implementation over Grizzly using Grizzly as the transport provider for TCP and UDP messages. The basic implementation is being provided by Bongjae Chang, Shoal community member, but needs further enhancements to make it production quality.

• Sub-tasks
  • Tune default Grizzly NetworkManager parameters. These are settable via following GMS Property parameters. <b>MAX_POOLSIZE, CORE_POOLSIZE, KEEP_ALIVE_TIME, POOL_QUEUE_SIZE, HIGH_WATER_MARK, MAX_PARALLEL, WRITE_TIMEOUT, MAX_WRITE_SELECTOR_POOL_SIZE</b>. These parameters manage resources available to process incoming Grizzly processing.
  • The GMS over Grizzly implementation will be tested over several of the critical test scenarios to establish its viability and stability for usage in a production quality GlassFish deployment. Initial stability tests will be based on module level distributed tests.
  • Requires QE to port over earlier GMS tests that were based on Appserver and depended on Appserver asadmin commands et al and revert to a module level test as we did with initial GlassFish v2 testing.
  • Fix P1 and P2 type bugs
• Dependencies
  • Grizzly framework and util jars version 1.9.19
  • Add GMS messaging load testing since HA relying directly on GMS over Grizzly messaging. (in v2.1, HA used jxta messaging directly)

GMS-02 Integrate Shoal GMS into GF v3.1 using dev. stop-gap clustering cmds

• Sub-tasks
  • QE use the stop-gap clustering cmds/configurations so gms over grizzly testing in gf v3.1 does not have to wait till GMS-03 functionality complete. Should be simple replacement of stop-gap cluster command for equivalent asadmin cluster command when GMS-03 is ready.
  • Make GMS OSGi compliant - GMS was not an OSGi compliant module. Changes being incorporated to mavenize GMS to publish to maven repository, and use GF v3 maven commands to build an OSGi compliant module
  • Create a GMS Service in GlassFish that will act as a delegate to start and stop GMS module in each GF 3.1 instance

Create a GMS Service in GlassFish that will delegate start and stop operations on GMS module in each GF 3.1 instance. This is an @Startup class that will be part of the startup services in a clustered GF instance. Need to close on requirements with respect to when GMS module will be started - need architects to consider impact of introducing instability and unpredictability in group memberships with lazy startup guidelines. COMPLETED by Sheetal.

• Dependencies

GMS for JoinedAndReady (instance start and restart), failure detection notifications and all Messaging pertaining to session replication.
DOL (possibly for deployment framework support to correctly handle automatic timer creation) <br>

GMS-07 GMS over Grizzly virtual multicast

• Description

Multicast is not always enabled in all production environments. Additionally, it is not available in cload environments.
So this feature is desirable for those configurations.

Implement as a staic list of ip address and ports that make up cluster. no dynamic capabilities when multicast not enabled in network.

• issues
  • Yet another configuration to test. Really need to run all existing shoal tests with this configuration to be sure it is working.

GMS-09 Monitoring Stat Providers

• Description

Provide monitoring stats that will help with diagnosing GMS membership related issues.

One-pager / Functional Specification

• GMS one pager

Dev Tests

• Information on GMS junit dev tests posted on Dev Tests page
• TBD. Post how to run automated distributed shoal gms developer unit test will be added to Dev Tests page
• 14 junit tests. Run as part of "mvn install". Also, "ant run-junit-tests.
• distributed developer level tests (runnable with multiple instances on one machine AND distributed as one instance per machine.)
  • 4 gms event notification tests (join/joinAndReady/suspected/failure/plannedShutdown)
  • 2 gms messaging tests
• Automation of nightly running of shoal gms distributed developer level test (both on single machine and distributed) via hudson is in progress.

Quality

• Link to Test Plans

Documentation

• Link to Documentation

Milestone Schedule

References / Links

• GMS configuration in domain.xml

  Email Alias

• Please contact us at dev@shoal.java.net

gmsconfig_gfv3_1.pdf (application/x-download)
gms_gfv3_1_onepager.txt.pdf (application/pdf)
gmsconfig_gfv3_4.rtf.pdf (application/pdf)