GlassFish - World's first Java EE 7 Application Server


GlassFish Server Open Source Edition 3.1 - In-memory Session Replication: High Availability

Scope of the Project

The In-Memory replication feature allows session availability by replicating session state to other instances in the cluster. By replicating session data, when the instance that was serving the request fails, the session can be restored by retrieving the session data from the replica instance.

Also, instead of letting the containers to directly interact with the in-memory replication layer, a BackingStore SPI has been defined. The advantages of using this SPI are:
1. It defines a set of classes and interfaces that a service provider (file system, replication, database etc.) must implement
2. It allows the containers to interact with different kinds of stores (like in-memory replication, file system, database etc.)

In GlassFish Server Open Source Edition 3.1, we will implement the in-memory replication module as a service provider of BackingStore SPI.

High Level Features

Feature-ID Desired Improvement Priority Comments Milestone Eng Response
HA-1 Session Persistence BackingStore SPI port from v2 to 3.1 workspace P1 Port V2 SPI functionality. Milestone 1 YES
HA-2 Implement Replication Store in Project Shoal providing save(), remove() operations P1 Allows session data to be persisteed and removed from store Milestone 2 YES
HA-3 Implement load() operation to retrieve state from shoal replication store P1 Allows session data to be retrieved from ReplicationStore Milestone 3 YES
HA-4 Support load() and save() operations on a restarted instance P1 Allows session data to be retrieved from ReplicationStore after a server restart Milestone 4 YES
HA-5 Implement HA BackingStore SPI using shoal Replication store P1 Allows Containers to interact with the shoal Replication store using the SPI Milestone 2 YES
HA-6 Implement a No-Op HA BackingStore SPI P1 Allows Metro to operate outside of GlassFish. Other containers may also use this approach when run in non-cluster mode. Milestone 2 YES
HA-7 Retain v2.1 HA configuration elements and attributes in domain.xml P1 Allows QA test scripts to enable & disable HA Milestone 2 YES
HA-8 Improved logging for tracking replication messages P2 Provide better logging to track replication commands Milestone 3 YES
HA-9 Monitoring Stat Providers P2 Provide stats that be used for both monitoring and debugging Milestone 4 YES
HA-10 Performance improvement work P2 Improve performance Milestone 4 YES
HA-11 Implement BatchBackingStore for EJB Container P1 Feature parity Milestone 3 YES
HA-12 Implement Local caching of state for passivated EJB P1 Feature parity. Also, ensures that passivated EJB's state is available when a replica instance crashes Milestone 3 YES

High Level Overview of the project

The general approach is to replicate session state from each instance in a cluster to a back-up / replica. Unlike V2, where HTTP and EJB sessions were replicated to the neighboring instance (buddy), the replication module will rely on an external object called ReplicaSelector to pick a replica. The input to the ReplicaSelector will be the SessionKey and the ReplicaSelector will return an instance name (that is alive) to which the data must be replicated. Each backup instance will store the replicated data in-memory. Upon a failure the instance now servicing the request (after failover) will either already have the necessary data or it will use the ReplicaSelector to locate the replica to obtain and take ownership of the data.

Availability configuration will continue to work as it has for previous HA enabled releases. The existing persistence-type ("replicated") will continue to be supported. This will allow QA and performance tests to run as they do with V2.x.

The replication layer itself will use the GMS communication APIs for transporting the data. We will use GMS's send() API to replicate data to a replica instance.

The current plan is to leverage GMS for cluster group membership services including things like initial bootstrapping/startup and various cluster shape changes like instances being stopped and re-started, instances failing, new instances being added to the cluster, etc. Its is assumed that ReplicaSelector will register itself with GMS to know the current 'alive' members so that it can pick an 'alive' replica.

Changes to Various Containers:

1. EJB Container:

The EJB Container uses the store to (a) passivate EJBs when the cache overflows and (b) to checkpoint EJBs at the end of business method (or at the end of Tx). The EJB Container currently does not use the SPI to talk to a store. The EJB Container needs to use the BackingStore SPI to talk to in-memory replication layer. Since, the interface used by EJB Container is very similar to the BackingStore SPI, it should not be a major change for the container to switch to the SPI.

2. Web Container:

The Web Container will continue to use the same BackingStore SPI to save session state. The Web Container persists session data using one of the two approaches: (a) Full session: The entire session will be saved to the BackingStore and (b) Modified Attributes: Only those HTTP Session attributes that were either deleted / added / updated will be saved. We will continue to support both approaches.

3. Metro:

Metro will use the BackingStore SPI. It will save the following data in BackingStore: (a) Message, (b) SAAJMessage and (c) ProtocolSourceMessage. The above three messages will be serialized and the resulting byte[] will be persisted into BackingStore.

Feature Overview

HA-1 Session Persistence BackingStore SPI port from v2 to 3.1

  • Description

Provides a session persistence abstraction to containers allowing GlassFish containers to be agnostic to the actual session persistence implementation.

  • Sub-tasks

(a) Create an OSGi module that contains the SPI classes and interfaces.

(b) Make the BackingStoreFactory as a hk2 @Contract to allow containers to inject BackingStoreFactory.

(c) This Module will also contain a no-op BackingStoreFactory and a no-op BackingStore.

  • Owners: Mahesh
  • Dependencies: None

HA-2 Implement Backing Store in Project Shoal providing all of BackingStore operations

  • Description

The implementation will provide the basic load / save / remove operations and will use GMS APIs to replicate data to replica instance.

  • Sub-tasks

(a) Create a DataStore that will provide load / save / remove operations.

(b) The DataStore will internally use a ConsistentHash algorithm to select a replica for a given key. The DataStore will also use the Consistent hash algorithm (along with a GMS API to obtain the previous view of the cluster) to locate replica location after a failover.

(c) Implement various commands to replicate, remove and locate replica data.

(d) Use GMS APIs to react to Cluster Shape changes to implement a simple consistent hash algorithm.

  • Owners

Mahesh & Joe

  • Dependencies

(a) GMS APIs to provide cluster shape changes

(b) GMS APIs to provide send and receive replication data and various command data.

(a) GMS API to provide previous cluster state after a node failure / restart.

  • Estimate TBD

HA-3 Implement Backing Store in Project Shoal providing generic CRUD semantics

  • Description

Provides a backing store implementation with simple APIs to do CRUD operations and support querying. The store will be an OSGi module integrated into GlassFish periodically over the release time frame.

  • Sub-tasks

Implement save, load, findByCriteria, remove and support for partial updates such as updateTimeStamp of a session.

  • Owner

Mahesh

  • Dependencies

GMS for JoinedAndReady (instance start and restart), failure detection notifications and all Messaging pertaining to session replication.
DOL (possibly for deployment framework support to correctly handle automatic timer creation) <br>

  • Estimate TBD

HA-4 Monitoring Stat Providers

  • Description

Provide monitoring stats that will help with diagnosing session replication related issues for testers and end customers.

  • Sub-tasks

TBD

  • Dependencies
  • Owner

Mahesh

  • Estimate TBD

One-pager / Functional Specification

  • Coming soon

Dev Tests

  • 14 junit tests. Run as part of "mvn install".
  • Coming Soon - Information on HA dev tests will be posted on Dev Tests page

Quality

  • Link to Test Plans

Documentation

Milestone Schedule

Please refer to the HA one pager document

Task List

Task start date end date QA handoff Target Milestone Owner(s) Feature ID Status / Comments
Convert BackingStore SPI into an OSGi module   MS1 NO MS1 Mahesh HA-1 DONE.
Provide no-op BackingStore and BackingSoreFactory for Metro   MS2 NO MS2 Mahesh HA-6 Done. The persistence type for this BackingStoreFactory is 'noop'
Retain v2.1 HA configuration elements and attributes in domain.xml   6/21 NO MS2 Mahesh HA-7 Done. No configuration changes are expected between v2.1 and v3.1
Implement HA BackingStore SPI using shoal Replication store   MS2 NO MS2 Mahesh HA-5 Done. shoal-backing-store module acts as the adapter.
Add File based SPi implementation   MS2 NO MS2 Mahesh   Done. Ejb Container uses the file store through HA spi
Provide store and remove operations in SHOAL replication store   MS2 NO MS2 Mahesh HA-2 DONE. The shoal replication store already supports these operations.
Allow ReplicaSelector to listen for failure / join events   MS2 NO MS2 Mahesh Kannan HA-2 & HA-4 Done. Reacts to failure and join and ready events.
Implement a consistent hash algorithm to locate replica instance   MS3 NO MS3 Mahesh HA-2 & HA - 4 Done, but blocked by 12730 (which is an intermittent bug)
Implement load operation in SHOAL replication store   MS3 NO MS3 Mahesh HA-2 Done
Implement load() operation to retrieve state from store   MS3 NO MS3 Mahesh HA-3 Done. Again blocked by 12730
Implement updateTimeStamp method in BackingStore   MS4 NO MS4 ?? HA - 2 Done.
Implement local caching of state in shoal replication store   MS4 NO MS4 Mahesh HA - 12 Done
Support local caching of passivated EJBs. Ensure passivated EJB state is available after server crash.   MS4 NO MS4 Mahesh HA - 12 Started
Implement BatchBackingStore for EJB   MS4 NO MS4 Mahesh HA-11 Not started.
Work with Web container team & LB team to support replicaLocation cookie. Ensure better performance after server crash   MS4 Yes MS4 Mahesh HA-11 This is really a web container task. Working with Rajiv and Kshitiz
Ensure that updates / save occur in the correct order in replica cache   MS5 NO MS5 Joe Fialli HA-2 Not started.
Add support for batching replication messages using interceptors   MS5 NO MS5 Mahesh HA-10 Not Started. Moving this from MS4 to MS5 as this is really a perf related feature
Allow ReplicaSelector to rejoin events   MS5 NO MS5 Joe Fialli HA-2 & HA-4 Not yet started.
Improve log messages for easy debugging   MS5 NO MS5 Mahesh & Joe Fialli HA-8 Started. This is really an internal feature. Other modules will not directly depend on any of the log messages

References / Links

  • Functional spec:
  • GMS Functional spec:
  • SHOAL Replication store doc:

Email Alias