GlassFish Server Open Source Edition 3.1 - Dynamic Reconfiguration

This wiki page / document details how dynamic reconfiguration will work when cluster support is added to GlassFish 3.1.

• Jerome Dochez (dochez@dev.java.net)
• Vijay Ramachandran (vijaysr@dev.java.net)

Scope of the Project

Dynamic reconfiguration feature for clustering in GlassFish Server Open Source Edition 3.1 refers to the way CLI commands (for example, asadmin create-jdbc-resource some-options --target some-target) from user to DAS gets reflected in the server instance (if the user specified target is a server instance) or all server instances that are part of the cluster (if user specified target is a cluster).

High Level Overview of the Features

To be able to effect the user requested changes (like creating resources as mentioned above) on a server instance / or all instances of a cluster, the following steps have to happen.

1. Execute the command on DAS; ensure that the command goes through; If the command succeeds on DAS, then
2. Replicate the same command on the target server instance or all server instances that are part of a specified cluster; report progress status back to client
3. Ensure that all replicated commands go through fine. If there are errors, report back errors to user while marking instances where the command failed for restart; Handle all error scenarios; Differentiate complete failure of a command (which may trigger a rollback) from a partial failure (the command failed in a small subset of target only which may not require a rollback); In failure scenarios, ensure that information on what failed is clearly available
4. Ensure that, when commands are replicated successfully on all instances, subsequent restart of any instance does not result in unnecessary synchronization (one scenario is the mismatch of timestamps of domain.xml between DAS and instance)
5. Take care of supplemental commands (these are commands that cannot be replicated just as they were done in DAS. For example creation of references in instances will be done as a supplemental command to the deploy command - click here for more information on this example).
6. From a user perspective, the behavior from CLI/console should remain the same except for the need to specify the target cluster / instance for the command. The CLI/console will continue to talk to DAS (just as in GlassFish v3) and DAS will take decision on whether to replicate the command or not (depending on the command, the RuntimeType specified etc.) The command replication infrastructure will ensure that the instances obey only the commands from DAS and no direct commands to instances from anywhere else will be obeyed.

The first step listed above is already happening in GlassFish 3.0 and it is common whether we are in a clustered environment or a stand alone environment. Since the rest of the steps listed above are essentially kicked off after step #1 is completed successfully, rest of this document will only go into the details of the step 2 onwards.

Implementation details

TBD : API reference below is internal javadoc - got to point it to an external javadoc - how?
A new set of annotations and interfaces have been added to GlassFish 3.1 towards implementation of the dynamic reconfiguration feature. The implementation approach is as follows :

Hooking into GlassFish 3.0

This section gives a high level view of how new code for dynamic reconfiguration will hook into / affect existing (GlassFish 3.0) command execution code on the server side.

• Every CLI command should annotate itself with @ExecuteOn. It should also specify the RuntimeType of this command as part of @ExecuteOn. If the command is expected to be executed only on DAS, RuntimeType should be DAS. If the command has to executed on DAS and replicated on applicable instances, RuntimeType for @ExecuteOn of this command should be set to DAS and INSTANCE.
• If a command does not have @ExecuteOn, then the implementation will consider that command to have RuntimeType set to DAS and INSTANCE. This way all existing non-local commands in GlassFish 3.0 will not have to be changed and will be executed on DAS and all applicable server instances.
• The RemoteCommand facility, currently used for communication between user/cli and DAS, will be used (after some refactoring) for communication between DAS and instances for command replication.
• The GlassfishClusterExecutor will plugged into the current server-side command execution as follows :
  • Currently, a non-local CLI command reaches the server through the AdminAdapter and, after authentication and other checks, the command flows into CommanRunnerImpl.doCommand where parameters are injected and the actual command is executed.
  • Once the command is executed and is successful, instead of returning and if the server environment is RunType.DAS, we will give control to the ClusterExecutor which will take over control. Details of ClusterExecutor are present in the next section.

Command Replication using ClusterExecutor

• GlassFish 3.1 will have an implementation of ClusterExecutor interface, the GlassfishClusterExecutor. This will be the default ClusterExecutor implementation that will be used if @ExecuteOn is not specified with a ClusterExecutor type. (Which in turn means, GlassFishClusterExecutor will be used for doing command replication for all GlassFish3.0 remote commands).
• As detailed in the previous section, control comes to the ClusterExecutor only for DAS - not on the instances.
• It will be the job of the cluster executor to
  • decode the target type and extract all instances on which the command has to be replicated
  • change the parameters (if required)
  • Add DAS timestamp as part of the parameters and ensure that instance has updated timestamp of its domain.xml
  • send the command across to all instances
  • collect progress status / result from instances to which command was sent
  • Take care of errors (more details on this in the next section)
  • report success / error to the user
  • Upon successful completion, execute supplemental commands
  • In case of failures, set restart required state as appropriate
  • In case of failures, if the command is an UndoableCommand, try to undo the command

Supplemental commands

Supplemental commands are commands that are specified by the user to be executed upon successful completion of a CLI command.For example, deployment in a cluster will be using supplemental command to deploy the bits in various instances as explained here. The GlassfishClusterExecutor will look up the habitat to see if supplemental commands have been registered for the current CLI command.

• If supplemental commands are specified for execution before CLI execution (by using Supplemental.Before, then the supplemental command will be executed on all nodes and the primary CLI command will be executed if and only if the supplemental command's FailurePolicy is FailurePolicy.Ignore or FailurePolicy.Warn.
• The behaviour of supplemental commands that are specified to be executed after that primary CLI command (and the effect of supplemental command execution on overall command replication result) is detailed in the table in the next section.

Undoable commands

Undoable commands give the command implementor a way to rollback a failed command execution. The table in the next section details the behaviour of command replication if the command implementor chooses to implement the UndoableCommand interface.

Command replication results and action taken

The table below gives details on the action taken upon completion of command replication for various scenarios

Securing DAS-Instance communcations

TBD : How DAS will authenticate itself with instance while doing command replication; Do we use SSL by default for communicating with instances or the way admin adapter is configured; Currently investigation various possibilities

Milestone Schedule

Note: for deliverables that have a link to an issue i the status column, see IssueTracker for the status.

Task List

Demo

Command replication demo for MS1

Documentation

• Info for CLI Owner/Developer

Dev Tests

• Developer Tests

References / Links

GlassFish Server Open Source Edition 3.1 Plan
GlassFish Server Open Source Edition 3.1 Clustering design Spec

Email Alias (Where Discussions are Taking Place)

• Please contact us at admin@glassfish.java.net