Admin Infrastructure Functional Specifications

Back to V3 Functional Specifications
[

|(TableOfContentstitle='TableofContents')]

0. Important Notes

• This one-pager is a bit unusual as it covers a lot of territory and has contributions from multiple authors. The layout is a bit different from the usual format. E.g. section 6 normally contains the schedule. What we've done instead is put the schedule details right in with the appropriate section 4 area.

1. Introduction

This document lists the tasks to be done in the administration infrastructure area (configuration, pluggability, admin CLI, public management infrastructure/API, administrative security etc.) for GlassFish V3 Final. It should be used as a guideline to plan the deliverables and test specifications. Every task has been articulated in sufficient detail. Ideally, some kind of design document should be created for the functionality being delivered for this release.

1.1. Project/Component Working Name:

Admin Infrastructure

1.2. Name(s) and e-mail address of Document Author(s)/Supplier:

The individuals who contributed to this document are:

Kedar Mhaswade (km@dev.java.net)
Byron Nevins (bnevins@dev.java.net)
Nandini Ektare (ne110415@dev.java.net)

1.3. Date of This Document:

March 2009

2. Project Summary

2.1. Project Description:

This project deals with facilities of GlassFish v3 that relate to the administration of the server. Deployment is available as a separate spec and is not covered here. The goal is to provide facilities such that users are able to manage instances of GlassFish V3. Most of the management operations relevant to web application deployment and management from GlassFish V2 are carried forward. There are a few new management operations which are explicitly called out in this specification.

2.2. Risks and Assumptions:

3. Problem Summary

3.1. Problem Area:

Refer to V3OverviewFunctionalSpec

3.2. Justification:

Refer to V3OverviewFunctionalSpec

4. Technical Description:

4.1. Details:

4.1.1 Core Configuration Tasks

4.1.1.1 Provide basic configuration validation support

• Define the validation framework.

New: JSR 303 Bean Validation API (javax.validation.*) will be used for all config validations. The implementation is provided through integration of Hibernate Validator. (See the references below for more info on the JSR and its implementation.) The framework defines standard basic validation annotations that can be used to validate object fields (config properties/attributes) and also has extensions to provide custom object level validations. Validations will be provided at both these levels. GlassFish V2 used regex for certain field level validations. The JSR provides a standard @Pattern annotation that suits this kind of validation and that shall be used.

• Define data-type based validation (application of 1. above for data-type validation). This is required because our configuration interfaces entertain only strings for attributes for any type of data.

The validation support through the Bean Validation API supports annotations that will take care of this requirement. For example. Custom annotations can be added through the constraints extension mechanism in the API to take care of any specific validation criteria.

• Define additional annotation types or enhance the org.jvnet.hk2.config.Attribute annotation to include medadata about:
  • data-type - actual data type of a variable. Provide support for int, java.lang.Integer, boolean, java.lang.Boolean.
  • one-of - enumerated values. range - specifies ranges.

The Bean Validation API has basic annotations to cover the above metadata enhancements. The support can be further extended through the mechanism to add additional constraint annotations

4.1.1.2 Provide a configuration option to turn validation on or off.

• Provide a boolean attribute, validate on admin-service element. (TBD: Jerome's comments: Not sure if this a good idea to support)

Every time the configuration of objects is changed in memory (via admin interfaces), this flag determines whether validation checks apply. If the flag is turned off, no validation checks are performed while changing the configuration objects.

Final Decision (ASARCH April 13th, 2009):
No such support will be provided.

4.1.1.3 Create a DTD-like document from configuration.

(Jerome's comments: We don't need this for 3.0 release)

(June's comments: The doc team needs this info in some form in order to document the domain.xml file. It doesn't have to be DTD-like. If it's Javadoc-like, that would work too. And what about QA? Don't they need something that tells them everything domain.xml can contain and whether it's well-formed? Furthermore, what about users? Some of them will want to edit domain.xml but won't want to have to read the source code to do it.)

Final Decision (ASARCH April 13th, 2009):
1. Provide javadoc-like documentation to be release to the community directly and for doc team's reference.
2. Implement verify-domain-xml CLI to make sure domain.xml is validated and verified on demand.

4.1.1.4 Provide a new attribute named version on the root element of the domain.xml.

• Modify the configuration interface Domain.java to have read-only @Attribute named version.

Since there is no schema document that validates the domain.xml, it is not possible to identify which release of GlassFish v3 a particular domain.xml corresponds to. This is mostly from the point of view of future provision. This will become prominent once we start thinking of upgrading GlassFish v3 Prelude domains.
The value of this should match the Version of the product as returned by the fullVersion field of com.sun.appserv.server.util.Version class.

Final Decision (ASARCH April 13th, 2009):
Let the version at the top level remain as is, but do not use it in any decision making because ina partially-upgradable system like v3, the top level version does not really help in making meaningful upgrade decisions based on that version

4.1.1.5 Incorporate new Grizzly configuration.

1. Move and rename (to org.glassfish) the following interfaces to glassfish-api for configuration (the idea is that all pluggable modules share these interfaces in their configuration):
   1. Property.java – A generic interface with a name and a value. Denotes the <property> element in XML.
   2. PropertyBag.java – A generic interface to hold <property> elements.
2. Incorporate new Grizzly configuration as Java interfaces in org.glassfish.grizzly.config package (and its subpackages).

• This is rather risky, but we can't do anything better.
• Open Issue: OI-0.

4.1.1.6 Minify the domain.xml.

1. Since this is a web-specific release, the default domain.xml should be modified to remove any extraneous elements.

As far as possible there should be one domain.xml template for all the distributions of GlassFish v3 Prelude.
This requirement is tightly coupled with the requirement to split config beans and not keep them all together in Base GF. Each module contains its config beans. On the first loading of a new module(/container) GF v3 provide a mechanism to detect any default configs or minimal initial configs supported by the module(/container) and merge this new default config with the domain.xml.
(Note: The present mechanism to do so is as follows:
Every container has a Sniffer provided by the module(/container).
The Sniffer interface includes a callback API : public Module[] setup(String containerHome, Logger logger)
This is called everytime the container is loaded in memory and gives the container the opportunity to provide its initial configuration through the parseInhabitants() API implemented by the module provider.)

4.1.1.7 Provide remaining dynamic reconfiguration support.

1. Provide an API to access the changes to configuration that are persisted to domain.xml but not applied to the runtime.
2. Provide support (and API) so that admin interfaces can indicate if the runtime requires a restart of the server to (possibly) apply the changes that are not applied. (e.g. some changes to the JVM options like -Xmx, port being occupied etc.)

4.1.1.8 Provide admin traffic on a separate port called admin-port, which is 4848 by default (just like GlassFish V1, V2).

1. Ensure that admin operations are available only on admin-port by default. Admin-port is a required interface on the installer as well as asadmin create-domain command. The default value for the port is 4848.
2. Modify asadmin stop-domain command logic to specify what the admin-port is. The name of the admin http-listener is admin-listener.
3. See more details here

• Benefits of separating the admin-port from user port by default are obvious (compatibility). Also, name-space collision is disliked by users.
• Supporting administration both ports is rather tricky. It may so happen that users want administration to be available on secure channel while user traffic on application-specific channel.
• Assess the impact of Grizzly Config task on this.

4.1.1.9 Document configuration Pluggability

1. Document the manual steps about how a particular module can use the configuration annotations and domain.xml to plug in itself into v3 Prelude runtime. Administration interface to seamlessly do this is out of scope of this release. Users are expected to modify the domain.xml by hand to provide the XML structure for their configuration that needs to get injected.

4.1.1.10 Support password aliasing tokens.

See Admin Security Section.

4.1.1.11 Specify the complete domain.xml persistence policy.

1. It is absolutely important to save a backup of domain.xml since it is modified frequently.
2. We have seen issues when domain.xml is corrupt and there is no backup copy available.
3. The best strategy seems to be making a change to a backup file and then making an atomic move to the real file if writing to backup succeeds. This is not easy to do on Windows.

4.1.2 Command Line Interface

CLI for V3 Final is exhaustively documented separately:

• List of CLI Commands
• CLI Functional Specifications

4.1.3 Platform Services

V2 introduced support for setting up GlassFish as a service on Solaris and Windows. The main functional design will remain the same. We will introduce the same command name, create-service, into V3.

In keeping with the philosophy of minimizing the introduction of native code into V3, we are planning on leveraging JNA for this project. Naturally all services are platform-dependent and native so there is no way to avoid it.

asadmin create-service, an existing command in V2, will be ported to V3 as a starting point. It currently works with Solaris and SMF. It will be expanded to work with multiple platforms.

4.1.3.1 SMF

Much work was put into documenting SMF support for V2. Some useful SMF documents have links at the end of this document. SMF is an important feature. E.g. it improves OpenSolaris adoption for GlassFish. These two documents are especially important:

• Design document from V2
• SMF FAQ from V2

4.1.3.2 Windows Services

• Windows: A easy-to-use Windows Service will help adoption of GlassFish. It will be flexible enough for use as a Service for other java programs like JavaDB.

4.1.3.3 Schedule

Platform Services will be completed in mid July for the V3 Final Release

4.1.4 Server start and restart

Server restart has been a highly desired feature for a long time. The Admin Console Team is especially excited about this feature so that a user can easily restart the server after making configuration changes that require a restart. The user will be able to restart remotely.

There are several ways to implement server restart. All such ideas of course require some sort of process to be running on the server's behalf so that it can restart the server. Two different approaches for implementing the behavior are discussed here. The two approaches have wildly differing levels of \

• development cost
• ease of use
• reliability

4.1.4.1 Asadmin as a Watchdog

• development cost: low
• ease of use: medium
• reliability: high
• This solution has been implemented and is available in V3 as of April 3, 2009

This solution leverages an already existing process and behavior. This is overwhelmingly our choice for V3 final.

Background

asadmin start-domain is the supported way to start the server in an external process. It will start the server in a new JVM and exit. If you add the --verbose option then asadmin starts the server and remains running waiting for the server to exit. asadmin holds a reference to the Java Process object of the server's JVM. It also has a console window. It is called "verbose" because it echos all log messages to its console window. If the console window is disposed of (e.g. by entering ^C) then not only is the asadmin process killed but the server's JVM is also destroyed.

Goals

• Add a new option to asadmin start-domain. Namely the --watchdog option.
• start the domain on the domain machine with a watchdog process running
• The watchdog does not require a console window but will have one if --verbose is also chosen.
• A new CLI Command will be added – restart-domain.

Details

• The user starts the domain with asadmin start-domain --watchdog
• If verbose is not also set, asadmin does not directly start the domain. Instead it creates and starts another JVM which in turn runs start-domain. The original asadmin then exits.
  • Why? This is how we get rid of the unwanted console window.
• If verbose is set then we do everything in this asadmin process.
• The user runs the asadmin restart-domain command, which makes the JVM process exit with integer value 10.
• asadmin sees the special return value and starts up the server again with the original arguments.

There are 3 ways to restart a server:

1. REST command
2. asadmin restart-domain
3. Admin Console GUI action, probably a button which eventually calls

AMXRoot.getInstance().getDomainRoot().executeREST(DomainRoot.RESTART_DOMAIN);

If the startup fails for any reason in this restart mode, launcher gives up and exits with whatever exit code server process exits with. The launcher clearly logs/displays the errors so that the corrective actions can be taken. For this release, no retry attempts have been specified. This facility is used to do a clean implementation of domain restart from the admin console/CLI.

Notes

• If the server won't stop properly via a System.exit(10) then a restart is not attempted.
• killing the server JVM from the OS itself results in an exit value of 1 - so no restart is attempted
• We purposely don't want to add a state to asadmin where it will always try to restart – e.g. asadmin start-domain --alwaysrestart
• The new CLI command, restart-domain should be very easy to implement. It is exactly the same as stop-domain with one difference. It returns 10 instead of 0.
• Starting the domain with asadmin --watchdog will not use a console window. This is important for users. We have learned in past incarnations of AppServer that some users really hate having unnecessary open console windows around. With verbose set to off the console windows would be useless anyway. If they want a console window they simply run in watchdog and verbose modes.
• Fairly exhaustive timing tests were done. If the user starts the domain with verbose and watchdog on - there is zero time impact relative to starting the domain non-verbose and non-watchdog. On the other hand there is a quite large cost for running the watchdog in a separate window – more than 2 seconds of startup time. This is something we can look at more closely post milestone 2.

Documentation Ideas/Requirements:

1. User must start the domain with the --watchdog option to enable restarts
2. If the previous step is forgotten then a call to the remote command, RestartDomainCommand, will stop the domain and emit a warning log message.
3. The --watchdog option must be added to the CLI man page for start-domain
4. A new man page must be created for the new remote command – RestartDomainCommand (asadmin restart-domain)
5. --watchdog is by default active.

QA Testing Suggestions:

• after running this: asadmin start-domain --watchdog
  • call asadmin stop-domain. The domain should stop and not restart.
  • Kill the domain externally via, say, the kill command. The domain should NOT restart.
  • Kill the console where asadmin start-domain --watchdog is running by entering ^C. The domain should stop and not restart.
  • Run jps then asadmin restart-domain then jps again. You should see the server running with some pid, then it should reappear in the list with a different pid. The console window should say "Restarting Domain..."
  • Repeat the previous step over and over and over.
• asadmin start-domain --verbose
  • Restarting should NOT work. It should only work with the --watchdog option
• asadmin start-domain --verbose --watchdog
  • This should work ''exactly' like * asadmin start-domain --watchdog except that log messages are printed to the console.

Timing Results

Invasive testing. I added code to AsadminMain, Launcher, AppServerStartup as follows

1. AsadminMain immediately gets the clock time and sets a System Property
2. Launcher starts the server JVM with that System Propertyas a "-D" argument
3. AppServerStartup gets the clock time when all services are started, subtracts the time sent in from AsadminMain
4. the resulting time is logged.

4.1.4.2 ALTERNATE IMPLEMENTATION – Server Reincarnation

• development cost: very high
• ease of use: very high
• reliability: medium

In this solution the server itself starts a separate JVM process that will run asadmin start-domain. The running server starts the new process when it is near death. Asadmin will be given the process ID (pid) of the running server's JVM. Asadmin waits for the operating system to verify that the process has died.

Notes

• Guaranteeing the running server's demise is the sticky and expensive part of this solution but without it it will be far less reliable.
• Windows assigns pid's in a much less clear way than *NIX. What if Windows reuses the dead server's pid right away before asadmin has had a chance to check? Asadmin will wait for that new random process to stop.
• There is a requirement for native code. The plan would be to use JNA. Nevertheless it is still native code.
• This solution is great from a user's perspective because the server did not have to be started in any special way
• All running servers are capable of restarting remotely or locally

4.1.4.3 Schedule

Because of agile deadlines and the relative simplicity of the solution in 4.1.4.1 we went ahead and completely implemented debugged and tested it in advance of the asarch approval process. I.e. the SCF is only 1 week after the review. Assuming approval it will be ready for V3 Milestone 2.

4.1.5 Server stop

It is possible for a server to run wild and not be stoppable in the usual controlled way. I.e. asadmin asks the server to stop. This is not possible if, say, the admin listener can't communicate for some reason or a non-daemon thread in the server refuses to stop.
A forced kill has been working and available in V3 since TP2 as follows:

• start the server with asadmin start-domain --verbose
  • Enter ^C at the asadmin console and it will kill the server.
• New forced kill scenario is to start the server with asadmin start-domain --watchdog
  • In this mode there is no console. Kill the asadmin process and it will guarantee stopping the server as well.
• The above is the only pure Java way that we can guarantee server shutdown. I.e. we must have an extra running JVM that holds a reference to the server's Java process object. This object is only valid in the JVM that started the server. In the default case that JVM is long gone at runtime.
• The server can not just kill itself with a System.exit or an abort call because it may be totally incommunicado.

4.1.6 Backup Restore *** STRETCH GOAL ***

New additions to Backup Restore functionality include:

• Ability to configure automatic backups
• The configuration will be persisted in domain.xml
• Backup of a running server will be supported for the first time (see next bullet)
• GlassFish-Admin Quiescence will be developed. I.e. all Admin commands will be stopped before a backup is allowed in a running server.

4.2. Bug/RFE Number(s):

4.3. In Scope:

4.4. Out of Scope:

4.5. Interfaces:

4.5.1 Exported Interfaces

Disclose all interfaces that this project exports.

• Interface:
• Stability: Committed
• Former Stability (if changing):
• Comments:

4.5.2 Imported interfaces

None

4.5.3 Other interfaces (Optional)

None

4.6. Doc Impact:

The existence of the Administration Reference, which documents the domain.xml file, depends on the existence of reference information about what domain.xml can contain that is outside the source code, such as a DTD file or Javadoc pages.

4.7. Admin/Config Impact:

4.8. HA Impact:

4.9. I18N/L10N Impact:

Not applicable as special requirement. All externally visible strings should be externalized.

4.10. Packaging & Delivery:

4.11. Security Impact:

4.12. Compatibility Impact

4.12.1 Open Issues

4.13. Dependencies:

Not applicable.

5. Reference Documents:

• HK2 Overview
• Separating Admin and User HTTP Traffic
• List of CLI Commands
• CLI Functional Specifications
• Password Aliasing Document
• SMF
• JSR 303
• Hibernate Validator
• SMF Design Doc for V2
• SMF FAQ from V2
• SMF Reference Document
• SMF Reference Document from Liane Praza
• SMF Domain-service-smf.xml.template
• SMF NodeAgent-service-smf.xml.template
• SMF service_bundle.dtd
• Watchdog and Start Domain Timing Data

6. Schedule:

6.1. Projected Availability:

smf-lisa.pdf (application/pdf)
SMF_design.html.pdf (application/pdf)