GlassFish User-Managed Clusters Design Spec - take 1 (obsolete)This is the design specification for the user-managed clusters feature for GlassFish. Authors
IntroductionIn the GlassFish 3.1 release, clusters and clustered instances are managed by a domain administration server (DAS). The DAS is required to create, list, start, stop, and delete a cluster and the instances that are in a cluster. Configuration information is synchronized with members of the cluster by the DAS. The user-managed cluster feature provides the ability to create a cluster of instances without using a DAS. Responsibility for managing the cluster and its instances rests with the user, either by manually updating the configuration of each instance or by providing an external software system that does this management. Since the user provides the management for the cluster, the feature is called user-managed clusters. The purpose of this document is to describe the design for the feature. This includes the following sections:
Additional high level information about the requirements and design, such as packaging, i18n/l10n impact, etc. is available in the one-pager/project page for this feature. That information is not duplicated in this document. RequirementsThe requirements for this feature are specified on the one-pager/project page for this feature. General DesignThe basic idea for this feature is to allow a server running with the RuntimeType INSTANCE to be configured so that it accepts and processes asadmin requests directly rather than only accepting them from the DAS. The user or a user-provided software system will need to submit the commands to the instance. This basic idea has an impact across several areas of GlassFish and dealing with this impact is the subject of the remaining sections in this design. Domains, Clusters, and InstancesWhen using the user-managed cluster feature, a domain can contain exactly one cluster, i.e., there is a 1-1 mapping between domains and clusters. The domain name and the cluster name are the same. As with current instances with a RuntimeType of INSTANCE, the data for the instance is stored in the nodes directory structure. Typically, this directory structure is found under as-install/glassfish/nodes, but the directory structure may be elsewhere as indicated by the --nodedir option for various commands. As with other instances that are created on the localhost, the default node name for instances in a user-managed cluster is localhost-domainname, e.g., if a user-managed cluster name is "cluster1", then the domain name is also "cluster1" and the node name for instances in that cluster by default is "localhost-cluster1". Since there is no DAS, the node directory structure for a user-managed cluster node does not contain an "agent" subdirectory nor does it have a das.properties file. Thus, the software can distinguish between a node directory for DAS-managed instances and one for user-managed cluster instances based on the presence of an agent subdirectory or the das.properties file. This feature does not guarantee that the configuration for instances in a cluster stays in sync with each other. It is the responsibility of the user or software provided by the user to do this. When GlassFish instances communicate with one another, the software must handle potential errors caused by having configuration data, including the list of deployed applications, that is not in sync. Configuration changes to remove isDas() callsThe usages of isDas() are quite numerous and varied.
Option 2 would introduce too many new configuration attributes because the usages are so many and varied. Decision: use option 3 for this For the isDas() calls that do apply to a user-managed instance, we could
For Option 1 or 2, the code (not isDas()) would ask the question it wants the answer to. The RuntimeType or AdminService type defines the basic state of the server - such as whether it's user-managed or DAS-managed. Then the code can interpret those facts and draw conclusions based on those facts - such as whether to accept admin requests from non-DAS origins. This approach allows the config to remain more stable over time while over successive releases the code can make more and more sophisticated decisions based on that stable config, as needs dictate. Option 3 introduces probably at least 8 new attributes. (More if isDas() is removed completely for the cases where isDas() does not apply to user-managed instances). These 8 new attributes help to define the state of the server using a more fine-grained approach than Option 1 or 2. The attributes for DAS and a user-managed instance would be "true", whereas the attributes for a DAS-managed instance would be "false". The advantage is the flexibility to configure the system differently for its specific needs, but this flexibility and fine-grained approach may not really be needed. Also, in future releases, we may have to add more attributes for new features. Decision: implement mainly option 2b, but with a slight variation. The Server config bean will be modified to have duck-typed methods that answer the various questions that are asked by the code, such as isAdminEnabled, generateArtifacts, etc. The implementation of these methods will use the type attribute for the <server> element to determine how to answer the question. Also, we will not implement admin console running on a user-managed instance. Analysis of isDas()Cluster GetHealthCommand uses !env.isDas() to allow the command to run only on DAS. User-managed instance should be able to run get-health. The get-health command should be executed on those servers which have knowledge about all instances in a cluster, not just itself. A new duck-typed method on Server config is required: isOmniscient Some commands (CreateInstanceCommand, ListInstancesCommand, RestartInstanceCommand, StartClusterCommand, StopInstanceCommand) use env.isDas() to allow the command to run only on DAS. No changes are required in the command. However, !env.isDas() could be removed because the command already has defined @ExecuteOn(RuntimeType.DAS). StartInstanceCommand and StopClusterCommand also use env.isDas() to only run on DAS. These could be replaced with @ExecuteOn(RuntimeType.DAS). SupplementalCommandExecutorImpl uses serverEnv.isDas(), serverEnv.isInstance(), and a command's RuntimeType to determine if the command can be executed. No change required. GMS GMSAdapterImpl uses isDas to initialize health history of the cluster. A user-managed instance may need to initialize the health history as well. Since the server needs the health history of all the instances in the cluster, the duck-typed method on Server config, isOmniscient, can be used. This method returns true when server type="das" or type="user-managed-instance", and false when server type="instance". GMSAdapterImpl uses isDas to determine the member type as spectator for DAS or core for non-DAS. No change required. GMSAdapterImpl uses isDAS to determine whether it is a bootstrapping node, where DAS is a bootstrapping node. A bootstrapping node refers to a node that was used to bootstrap finding the cluster when multicast is not enabled. This is currently not being used in 3.x. The isDAS call may not be the only way to determine if a self-managed member is considered a bootstrap node. But at this time we are not using the concept at all, so it is okay to just comment that info out and we will work on it when implementing non-multicast support. (from Joe F.) GMSAdapterService uses env.isDas() to determine whether to create a GMSAdapter for all clusters on DAS that have GMS enabled, or just for the cluster of the current instance. No change required. GMSAdapterServer uses env.isDas() in the config listener changed method, to have DAS join a gms-enabled cluster on create-cluster event, and to have DAS leave a gms-enabled cluster on delete-cluster event. This does not apply to user-managed instances. No changes required. HealthHistory uses server.isDas() to skip creation of health history for the DAS. No changes required. Config API Server uses env.isDas() during _unregister-instance to prepare information about the cluster for the supplemental command. No changes required. ConfigRefValidator uses server.isDas() to validate that config-ref for "server-config" cannot be changed on DAS, and no other config can use "server-config. No changes required. PortManager uses server.isDas() to help create the local server list. If the server is DAS, then we can add it to the serversOnHost list. No need to check if it's local. No changes required. ServerHelper uses server.isDas() and env.isDas() in the getAdminHost() method to return the admin host using System.getProperty("com.sun.aas.hostName"); if the server is DAS. No change required. Connectors Connector classes (and JDBCResourceManager) use isDas() to ask 'Does this server have information about all targets (clusters/instances)? If yes, then the code can check if a resource is referenced by more than one target. This is useful for validating if a resource may be deleted or not. A resource may not be deleted from a target, if it is also referenced by another target. This is also used in ConnectorRuntime so it can do a data source lookup on DAS. Since there is only one user-managed cluster per domain, there is only one target available, which is the user-managed cluster. These validations do not apply to a user-managed instance. No change required. However, isDas() could be removed, because when a resource is referenced by more than one target, the command fails on DAS and will never be replicated to any of the targets so the isDas() is not needed. Deployment Partial Deployment GlassFish 3.1 allows partial deployment in a cluster. Partial deployment means the deployment did not fully succeed, e.g. it failed loading on one or more instances and the application was not written to the instance's domain.xml. Deployment commands, DeleteApplicationRefCommand, DisableCommand, and UndeployCommand use isDas() to allow the clean-up of partial deployments, while also allowing the command to fail on the DAS if an application is not already registered on DAS. For example, if an application is partially deployed when a user runs delete-application-ref, the application is first removed successfully from the DAS domain.xml. Then on the instance where the application is not registered, isDas() prevents the command from failing on the instance and returns without trying to remove the application from domain.xml. In this case, isDas() is asking 'Should we fail this command if the application is not registered?' For a User-Managed instance, these commands should also fail if the application is not registered. To allow the commands, DeleteApplicationRefCommand, DisableCommand, and UndeployCommand to fail, we can remove isDas() in the places where it is asking 'Should we fail this command if the application is not registered?' and add set the commands Failure Policy to Ignore if the command is being replicated. @ExecuteOn(...., ifFailure=FailurePolicy.Ignore, ...) This means all failures will be ignored when replicating this command. Replication Deployment classes use isDas() to determine whether to replicate a command. In this case, isDas(), is asking 'Can this server replicate commands?'. Replication does not exist in a User-Managed Cluster, so no code change is required here. DisableCommand DisableCommand uses isDas() to send an APPLICATION_DISABLED event on DAS so proper clean up can be triggered. The JPADeployer listens for the APPLICATION_DISABLED event, so it can close all the EntityManagerFactory(s) created for the application. A user-managed instance should also send an APPLICATION_DISABLED event so the EMFs can be closed by the JPADeployer. A new duck-type method on Server can be used isJava2dbEnabled, which returns true or false depending on the value of the new Server type attribute: das, instance, user-managed-instance. DeleteApplicationRefCommad if (server.isDas()) {
commandParams.origin = Origin.unload;
} else {
// delete application ref on instance
// is essentially an undeploy
commandParams.origin = Origin.undeploy;
}
In this case, isDas() is asking 'Can this server manage applications it's not running?'. A User-Managed Instance does not manage applications it is not running. No change is required. AutoDeployService isDas() is used to only start the auto deployer if the server is DAS. isDas() is asking 'Should we enable the auto deployer?'. A user-managed instance may also want auto deployer to be started. EJB Container Deployment EjbContainerUtilImpl uses !isDas() to set _doDBReadBeforeTimeout = true. In this case, !isDas() is asking 'Is _doDBReadBeforeTimeout default true?' On a clustered instance, the default is true, so a user-managed instance default should also be true. No change is required. EjbContainerUtilImpl uses isDas() to ask 'Can this server manage applications it's not running?', and sets the origin deployment parameter accordingly to deploy or load. A user-managed instance does not manage applications it's not running. No change is required. EjbDeployer uses isDas() to ask 'Does this server need to be cleaned of any files and artifacts (CMP beans, timers) that were created during the execution of the prepare method?'. A user-managed instance may also need to be cleaned of files and artifacts. A new duck-typed method, createsEjbFilesAndArtifacts, can be added to EjbContainer config. This method returns true when server type="das" or "user-managed-instance",and false when type="instance". EjbDeployer uses isDas() to ask 'Does this server require EJB artifacts to be generated?' Currently, artifacts are only generated when being deployed on DAS. A user-managed instance may also want EJB artifacts to be generated. The new duck-typed method, createsEJbFilesAndArtifacts, could be used. Timers EJBTimerService uses !(deploy && isDas()) based timers will be created. And no timers will be scheduled. If it is called from deploy on a non-clustered instance, both persistent and non-persistent timers will be created. Otherwise, only non-persistent timers are created by this method. A user-managed cluster can have a clustered deployment so non-persistent timers should not be created on clustered deploy of a user-managed instance either. Since the negative is used '!', no change is required to allow the same behavior on a user-managed instance. DistributedEJBTimerServiceImpl uses !isDas() to ask 'Is the server required to 1) register for Planned Shutdown event, 2) set DB read before timeout to true, 3) register for transaction recovery events?'. Distributed ReadOnlyBeanMessageCallBack uses !ejbContainerUtil.isDas() to ask 'Does this server require to 1) register as GMS adapter Message Listener 2) set as DistributedReadOnlyBeanNotifier ?'. JMS JMSDestination uses isDas() to help determine the CommandTarget type: CONFIG, DAS, STAND_ALONE_INSTANCE, CLUSTER. If the CommandTarget is DAS, the default address list is used as the connection URL to get configured instance of MQ-RA, which is then used to obtain the JMXServiceURL/JMXServiceURLList. A user-managed instance would be similar to a target of clustered instance so no change necessary. JMSPing uses !targetServer.isDas() to ask 'Is this server a stand-alone or clustered instance?' If yes, then get the JMS host and port. No change is required, but !targetServer.isDas() could be replaced with targetServer.isInstance(). JMSConfigListener uses thisServer.isDas() to ask 'Does this server not need to update the cluster broker list on the active JMS resource adapter when there is a config change event on the Server config?' MQAddressList uses isDAS(targetName) to ask 'Does this server require the default setup to concatenate all JMSHosts in a JMSService to create the address list?'. A user-managed instance would use the setup for clusters. No change required. Kernel CommandRunnerImpl CommandRunnerImpl uses env.isDas() to validate the command target types. Since a user-managed instance can execute a command, it should also validate the command target types. A new duck-type method on AdminService config is required: isFullAccessEnabled. CommandRunnerImpl uses env.isDas() to check that the server matches if the RuntimeType is DAS, or the command target is DOMAIN. A user-managed instance would run the command for serverEnv.isInstance() && runtimeTypes.contains(RuntimeType.INSTANCE). No change required. CommandRunnerImpl uses env.isDas() to ask 'Does this server require replication?'. A user-managed instance does not require replication. No changes required. DyanmicInterceptor DynamicInterceptor uses MbeanService.getInstance().isDas() to add "server" as a target. No change required. StopDomainCommand StopDomainCommand uses !env.isDas() to fail the stop-domain command if it's not run on DAS. stop-domain should also fail on a user-managed instance. No change required. AdminConsoleAdapter AdminConsoleAdapter uses !env.isDas() to return if it's not running on DAS. Admin Console will not run on a user-managed instance, so no change is required. ApplicationConfigListener ApplicationConfigListener uses server.isDas() to help determine if the current instance matches the target. No change required. ApplicationLifecycle ApplicationLifecycle uses env.isDas() to set the target to "server" if the target is "domain" and we are running in DAS. ApplicationLifecycle uses env.isDas() to load system applications on DAS. A user-managed instance does not have system applications. No change required. ApplicationLoaderService ApplicationLoaderService uses env.isDas() to load system applications on DAS. A user-managed instance does not have system applications. No change required. ApplicationLoaderService uses env.isDas()to (partially) load the application on DAS so the application information is available on DAS. No change required. ApplicationLoaderService uses env.isDas() to send a disable event on the DAS when the application is not loaded on DAS. A user-managed instance should also send a disable event. Then on the disable event, the EntityManagerFactory(s) can be closed by the JPADeployer. The new duck-typed method, isJava2dbEnabled can be used. AppServerStartup AppServerStartup uses env.isDas() during shutdown to execute stop-domain for DAS, and _stop-instance for non-DAS. A user-managed instance would require _stop-instance. No changes required. Logging LogManagerService, DeleteLogLevel, ListLogAttributes, ListLoggerLevels, SetLogLevel, and LogFilter use isDas() to get/delete/update the DAS logging properties. The logging.properties for a user-managed instance is in the same location for a DAS-managed instance: AS_INSTALL\nodes\nodename\instancename\config\instancename-clustername\logging.properties. No changes required. CollectLogFiles uses targetServer.isDas() to determine how to collect the log files for DAS. The targetServer.isInstance() is used to collect log file for a stand-alone instance. The 'else' condition collects log files for a cluster. A user-managed cluster can be a target of collect-log-files, not a user-managed instance. No change required. LogFilter uses targetServer.isDas() to determine how to get the log records and where to store them. It's used by Admin Tool Log Viewer Front End to filter the records. A user-managed instance will not have Admin Console running it so no change is required. Monitoring MonitoringReporter (get command) uses isDas() to determine whether to run locally and how to build the pattern. No change needed if the get command also runs locally on a user-managed instance and follows the same dotted pattern as a DAS-managed instance. Persistence JPADeployer uses isDas() in multiple places to ask 'Does this server execute java2db?'. If yes, the following type of actions are performed. 1) execute java2db on DAS If user-managed instance needs to do these actions as well, a new duck-typed method on Server config is required, such as, isJava2dbEnabled. Security CreateFileUser CreateFileUser uses isDas() to ask 'Should we fail this command when there is a BadRealmException that is not due to a user already existing in a shared keyfile?'. In case of the user-managed instances (without a DAS), if the keyfile is shared, as in the previous case, no error would be thrown in the first instance where the command is executed. However we could afford to ignore the error for all the other user-managed instances. However we cannot be sure how to identify the first case of execution and not ignore the error. For a user-managed instance it cannot ignore the BadRealmException. If create-file-user is run on a user-managed instance, it will not be replicated to the other instances because replication doesn't exist in user-managed clusters. If the user tries to run create-file-user on a second user-managed instance, they would get the error if the user has already been created in the shared keyfile. A new duck-type method on ? config is required, ? . AdminAdapter AdminAdapter uses !env.isDas() during admin request authentication to ask 'Is it ok to invoke req.getUserPrincipal() to get the SSL Principal?'. final Principal sslPrincipal = ! env.isDas() ||
Boolean.getBoolean(DAS_LOOK_FOR_CERT_PROPERTY_NAME) ? req.getUserPrincipal() : null;
return authenticator.loginAsAdmin(user, password, as.getAuthRealmName(),
req.getRemoteHost(), authRelatedHeaders(req), sslPrincipal);
A user-managed instance will also invoke req.getUserPrincipal(). Also, with more recent changes to Grizzly, this check is going to be removed by another project. No change required. AdminAdapter uses env.isDas() to determine which authorization failure to report during authentication when access is NONE. if (env.isDas()) {
reportAuthFailure(res, report, "adapter.auth.userpassword",
"Invalid user name or password",
HttpURLConnection.HTTP_UNAUTHORIZED,
"WWW-Authenticate", "BASIC");
} else {
reportAuthFailure(res, report, "adapter.auth.notOnInstance",
"Configuration access to an instance is not allowed; please connect to the domain admin server instead to make configuration changes",
HttpURLConnection.HTTP_FORBIDDEN);
}
A user-managed instance should also report the same authorization failure as DAS when access is NONE. A new duck-typed method on AdminService config is required: isFullAccessEnabled. GenericAdminAccessController The GenericAdminAccessController.chooseAccess method grants Access.FULL to a server only if serverEnv.isDas is true. This will be changed to use a isFullAccessEnabled duck-typed method on AdminService config. This method returns true when Server type="das" or type="user-managed-instance", and false when type="instance". GenericAdminAuthenticator GenericAdminAuthenticator uses serverEnv.isDas() to return the access to be granted the authenticated user. If the current node is not the DAS, then we grant only monitoring access. 3.1 does not permit full admin access to instances. If the current node is the DAS, then we grant full admin access only if the request came from the same host or if secure admin is enabled. The GenericAdminAuthenticator.authenticateAsTrustedSender method is only used for communication between a DAS and an instance, so this would never be used for the user-managed cluster case. So the call to isDas at line 312 doesn't need to be changed, and there doesn't have to be any configuration changes related to this. ServerRemoteAdminCommand uses serverEnv.isDas() in the getCertAlias() method to determine how to get the appropriate alias for the DAS or the instance. The default alias for DAS is "s1as". The default alias for non-DAS is "glassfish-instance". No changes required. ResourceUtil uses server.isDas() in getTargetsReferringResourceRef(String refName) to add DAS to the list of targets if DAS also has the specified resource ref. No change required. Web Services MetroContainer uses isDas() in deployWsTxServices(String target) to ask 'Does this server manage applications it's not running?'. The deployment parameters, origin and target, are set accordingly. Analysis of isInstance()CommandThreadPool uses serverEnv.isInstance() to check if it is not DAS and to return without creating the command pool for executing commands. A user-managed instance can execute commands so it would also need the command pool. The new duck-typed method on AdminService isFullAccessEnabled, may be used. RestAdapter uses serverEnvironment.isInstance() to only allow GET requests on an instance that is not DAS, and returns a FORBIDDEN error for other requests. A user-managed instance should be allowed SET requests, same as DAS. The new duck-typed method on AdminService, isFullAccessEnabled, may be used. StopInstanceInstanceCommand (_stop-instance) and RestartInstanceInstanceCommand (_restart-instance), which is called by stop-local-instance and restart-local-instance, uses !env.isInstance() to only allow the command to be executed on a DAS-managed instance. This validation can be removed, since the class has already defined @ExecuteOn(RuntimeType.INSTANCE). _stop-instance needs to also be allowed to run on a user-managed instance. Since RuntimeType.INSTANCE includes both DAS and user-managed instances, no change is required. Command Changes |
