9.1Changes

  • asadmin no longer uses any scripts (i.e. startserv) to start and stop servers
  • the startserv scripts are now fully functional and supported for use from a commandline
  • the startserv scripts get the username and passwords directly from the user
  • Any server can be started from arbitrary java code directly (look in the startserv script to see how)
  • The domains are relocatable.
  • Shutdown has been dramatically improved. It used to require between 2 and 6 external processes (for non-native and native respectively) to be started. Now it doesn't start any external processes so it is much faster.
  • To compare the new launchhing times with the old you can start servers with the current build and then repeat with build 22
  • To compare new with sort-of old, set OLD_LAUNCHER=true in your OS environment. In this case, the scripts will run in multiple processes as before but the scripts are pointing at new code instead of old code.

Coming Soon

  • Native launching fix
  • I implemented changes that call ProcessLauncher directly from the asadmin VM rather than going through startserv script
  • No security information (admin user and password, master password) is transmitted
  • All previous behavior is intact. The same asadmin command is used to run Before and After. The only difference is that the After version sets -DNEW_LAUNCHER=true which the new code looks for.

Before

C:\jws\glassfish\admin\servermgmt>asadmin start-domain
Starting Domain domain1, please wait.
Log redirected to c:\ee\domains\domain1\logs\server.log.
Domain domain1 is ready to receive client requests. Additional services are being started in background.
Domain [domain1] is running [Sun Java System Application Server Platform Edition 9.1 (build )] with its configuration and logs at: [c:\ee\domains].
Admin Console is available at [http://localhost:4848].
Use the same port [4848] for "asadmin" commands.
User web applications are available at these URLs:
[http://localhost:8080 https://localhost:8181 ].
Following web-contexts are available:
[/web1 /asadmin ].
Standard JMX Clients (like JConsole) can connect to JMXServiceURL:
[service:jmx:rmi:///jndi/rmi://hp:8686/jmxrmi] for domain management purposes.
Domain listens on at least following ports for connections:
[8080 8181 4848 3700 3820 3920 8686 ].


Time to start domain: 15.8 seconds.

After

C:\jws\glassfish\admin\servermgmt>cli start-domain
Starting Domain domain1, please wait.
Log redirected to c:\ee\domains\domain1\logs\server.log.
Domain domain1 is ready to receive client requests. Additional services are being started in background.
Domain [domain1] is running [Sun Java System Application Server Platform Edition 9.1 (build )] with its configuration and logs at: [c:\ee\domains].
Admin Console is available at [http://localhost:4848].
Use the same port [4848] for "asadmin" commands.
User web applications are available at these URLs:
[http://localhost:8080 https://localhost:8181 ].
Following web-contexts are available:
[/web1 /asadmin ].
Standard JMX Clients (like JConsole) can connect to JMXServiceURL:
[service:jmx:rmi:///jndi/rmi://hp:8686/jmxrmi] for domain management purposes.
Domain listens on at least following ports for connections:
[8080 8181 4848 3700 3820 3920 8686 ].


Time to start domain: 1.5 seconds.

The following table shows timing results of New code versus Old code. Quick refers to the quick-startup option. Six trials were done and an average calculated. The data was generated on a Solaris 10 system with this command:
time asadmin start-domain (all readings in seconds)
note: Script means running the startserv script itself, i.e. no asadmin involvement.

Version Quick Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 Average
Old Yes 8.823 7.498 8.731 6.811 9.111 7.480 8.076
New Yes 6.193 6.031 6.050 6.532 6.331 5.854 6.165
Old No 25.214 24.811 25.371 25.261 24.992 25.281 25.155
New No 24.452 25.081 25.021 24.501 24.471 25.041 24.761
Script No 2.01 1.997 2.803 2.184 2.341 2.119 2.242

The following table is from data collected from a Windows XP desktop system (1 GB RAM)
time asadmin start-domain

Version Quick Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 Average
Old Yes 18.146 17.618 12.787 17.634 12.842 18.370 16.233
New Yes 10.226 10.334 10.343 10.346 10.321 15.367 11.156
Script Yes 3.708 4.393 3.721 3.713 3.700 3.702 3.823

Background

Starting and Stopping Server Instances

Ref: http://wikis.sun.com/download/attachments/211392120/admin-fs.html#pl

The one-pager simplifies the process of starting and stopping servers. This article goes into a bit more depth.

A Process View of Starting (non-native)

  1. Proc#1 VM#1 asadmin start-domain
  2. Proc#2 asadmin runs an external process for domain-dir/bin/startserv.bat
  3. Proc#3 VM#2 com.sun.enterprise.tools.launcher.ProcessLauncher
  4. Proc#4 VM#3 com.sun.enterprise.server.PELaunch (this is the instance's VM)

A Process View of Stopping (non-native)

  1. Proc#1 VM#1 asadmin stop-domain
  2. Proc#2 asadmin runs an external process for domain-dir/bin/stopserv.bat
  3. Proc#3 VM#2 com.sun.enterprise.tools.launcher.ProcessLauncher
  4. Proc#4 VM#3 com.sun.enterprise.server.PELaunch!! Stopping an Instance

Startup

The startup procedure is quite complex. It is extremely difficult to debug because of the extra JVM in the middle. The startserv script is potentially confusing to a user and is vulnerable if edited.

For V2, the middle JVM will be eliminated. The instance JVM will be started directly by asadmin or by other running Java programs.
Requirements

  1. Startserv script must work all by itself. I.e. running startserv script with no args will start appserver
  2. All current functionality, including native launcher, must stay exactly as is. I.e. the new startup methods are used when a new special argument is given to asadmin
  3. appserver can be started by any java program
  4. appserver can be started by AMX
  5. Threaddumps must be supported
  6. Nodeagent will use the new classes for starting instances
    Tasks

1. Take over Middle VM responsibilities
The startserv script that lives under the domain directory has a few pieces of vital information that are domain-specific (the name of the domain, directory locations, etc.). ProcessLauncher (the middle VM) also assembles lots of information from domain.xml in order to create a proper java command for starting appserver. We will need to duplicate some of this code in new classes, and add code to the new classes for faster domain.xml parsing.

2. Modify startserv script
Startserv script must be carefully modified. One will be able to simply run it with no arguments and appserver will start. Conversely it will still work as before for backward compatability. Note that running the script stand-alone will result in one VM for the new launch functionality followed by the final vm with the appserver.
Tasks Details

  1. Copy ProcessLauncher.java in toto to a new java file
  2. Change the new java class to get information from domain.xml directly instead of from system properties

Shutdown

This is a ripe area for improvement. The stopping of instances is far too complex. The starting code is super-complex and needs to be because the instance has to run in an elaborately configured JVM environment. E.g. special JVM options have to be parsed from domain.xml and used to start the instance-JVM. Stopping the instance, conversely, should be much simpler since its job is to shutdown running objects. It is not neccessary and a waste of time and resources to build intricately configured JVMs to stop an instance.

This is a classic example of questionable code reuse. The super-complex starting code was reused to stop instances. Here is an example of how this approach can bite: https://github.com/javaee/glassfish/issues/949.

We propose simplifying stop and undoing the common code between start and stop.

VM Creation command issued by Startserv script

Note that all system properties are passed through to the final AS JVM except:

  1. In Windows, paths with are changed to /
  2. The lines starting with *** are missing
"C:\jdk\bin\java" 
-cp 
"c:\ee\lib\appserv-rt.jar";"c:\ee\lib\appserv-admin.jar";"c:\ee\lib\javaee.jar" 
*** -DCATALINA_OPTS="" %%
-Dcom.sun.aas.configRoot="c:/ee/config" 
-Dcom.sun.aas.installRoot="c:\ee" 
-Dcom.sun.aas.instanceName="server" 
-Dcom.sun.aas.instanceRoot="C:\ee\domains\domain1"  
*** -Dcom.sun.aas.launcherReturn="return" 
-Dcom.sun.aas.processLauncher="SE" 
*** -Dcom.sun.aas.processName="as9-server" 
-Dcom.sun.aas.promptForIdentity="true" 
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory 
-Ddomain.name="domain1" 
-Djava.util.logging.manager=com.sun.enterprise.server.logging.ServerLogManager 
-DJAVA_HOME="C:\jdk" 
com.sun.enterprise.tools.launcher.ProcessLauncher start debug verbose

VM Creation command issued by ProcessLauncher

C:/jdk\bin\java
-client
-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009
-Xmx512m
-XX:NewRatio=2
-Dcom.sun.aas.classloader.appserverChainJars.ee=
-Dcom.sun.aas.classloader.appserverChainJars=admin-cli.jar,admin-cli-ee.jar,dbschema.jar,j2ee-svc.jar
-Dcom.sun.aas.classloader.optionalOverrideableChain.ee=
-Dcom.sun.aas.classloader.optionalOverrideableChain=webservices-rt.jar,webservices-tools.jar,appserv-ws.jar,commons-logging.jar,commons-launcher.jar
-Dcom.sun.aas.classloader.serverClassPath.ee=c:/ee/hadb/4.5.0-9/lib/hadbjdbc4.jar,c:/ee/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,c:/ee/hadb/4.5.0-9/lib/dbstate.jar,c:/ee/hadb/4.5.0-9/lib/hadbm.jar,c:/ee/hadb/4.5.0-9/lib/hadbmgt.jar,c:/ee/lib/SUNWmfwk/lib/mfwk_instrum_tk.jar
-Dcom.sun.aas.classloader.serverClassPath=c:/ee/lib/install/applications/jmsra/imqjmsra.jar,c:/ee/imq/lib/jaxm-api.jar,c:/ee/imq/lib/fscontext.jar,c:/ee/imq/lib/imqbroker.jar,c:/ee/imq/lib/imqjmx.jar,c:/ee/lib/ant/lib/ant.jar,c:/ee/lib/SUNWjdmk/5.1/lib/jdmkrt.jar
-Dcom.sun.aas.classloader.sharedChainJars.ee=appserv-se.jar,appserv-ee.jar,c:/ee/hadb/4.5.0-9/lib/dbstate.jar,c:/ee/hadb/4.5.0-9/lib/hadbjdbc4.jar,jgroups-all.jar,c:/ee/lib/SUNWmfwk/lib/mfwk_instrum_tk.jar
-Dcom.sun.aas.classloader.sharedChainJars=javaee.jar,C:/jdk/lib/tools.jar,install/applications/jmsra/imqjmsra.jar,commons-launcher.jar,c:/ee/imq/lib/jaxm-api.jar,c:/ee/imq/lib/fscontext.jar,c:/ee/imq/lib/imqbroker.jar,c:/ee/imq/lib/imqjmx.jar,c:/ee/imq/lib/imqxm.jar,c:/ee/lib/ant/lib/ant.jar,webservices-rt.jar,webservices-tools.jar,appserv-ws.jar,mail.jar,jsf-api.jar,jsf-impl.jar,appserv-jstl.jar,appserv-env.jar,jmxremote_optional.jar,c:/ee/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,commons-logging.jar,activation.jar,appserv-rt.jar,appserv-admin.jar,appserv-cmp.jar
-Dcom.sun.aas.ClassPathPrefix=c:/ee/lib/webservices-rt.jar;c:/ee/lib/webservices-tools.jar
-Dcom.sun.aas.ClassPathSuffix=
-Dcom.sun.aas.configName=server-config
-Dcom.sun.aas.configRoot=c:/ee/config
-Dcom.sun.aas.defaultLogFile=C:/ee/domains/domain1/logs/server.log
-Dcom.sun.aas.domainName=domain1
-Dcom.sun.aas.installRoot=c:/ee
-Dcom.sun.aas.instanceName=server
-Dcom.sun.aas.instanceRoot=C:/ee/domains/domain1
-Dcom.sun.aas.jdwpOptions=transport=dt_socket,server=y,suspend=n,address=9009
-Dcom.sun.aas.processLauncher=SE
-Dcom.sun.aas.promptForIdentity=true
-Dcom.sun.aas.ServerClassPath=
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dcom.sun.enterprise.overrideablejavaxpackages=javax.xml.bind,javax.help,javax.portlet
-Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar
-Dcom.sun.enterprise.taglisteners=jsf-impl.jar
-Dcom.sun.enterprise.websvc.useJAXWSRearch=true
-Ddomain.name=domain1
-Djava.endorsed.dirs=c:/ee/lib/endorsed
-Djava.ext.dirs=C:/jdk/jre/lib/ext;C:/ee/domains/domain1/lib/ext;c:/ee/javadb/lib
-Djava.library.path=c:\ee\lib;c:\ee\lib;C:\jdk\bin;.;C:\WINDOWS\system32;C:\WINDOWS;c:\ee\lib;c:\ee\bin;c:\jdk\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;c:\bin;c:\bnbin;C:\DEV\MKS\bin;C:\DEV\MKS\bin\x11;C:\DEV\MKS\mksnt;c:\apps\cvs\bin;c:\batch;C:\Program Files\Common Files\Roxio Shared\DLLShared;C:\Apps\Executive Software\Diskeeper;c:\ee\bin;C:\dev\vc6\common\Tools\WinNT;C:\dev\vc6\common\MSDev98\Bin;C:\dev\vc6\common\Tools;C:\dev\vc6\VC98\bin;c:\dev\maven\bin;C:\Apps\Executive Software\Diskeeper;c:\dev\ant\bin;c:\temp\ee\bin
-Djava.security.auth.login.config=C:/ee/domains/domain1/config/login.conf
-Djava.security.policy=C:/ee/domains/domain1/config/server.policy
-Djava.util.logging.manager=com.sun.enterprise.server.logging.ServerLogManager
-Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
-Djavax.net.ssl.keyStore=C:/ee/domains/domain1/config/keystore.jks
-Djavax.net.ssl.trustStore=C:/ee/domains/domain1/config/cacerts.jks
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Djmx.invoke.getters=true
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-cp
;c:/ee/lib/appserv-launch.jar
com.sun.enterprise.server.PELaunch
start
debug

Timing Research

Time to start PE domain on my feeble HP system (Windows) via CLI

10.9, 14.4, 14.4, 10.3, 13.8,15.0,15.3,14.4,15.3,14.6,9.9

Time to start PE domain on my feeble HP system (Windows) via new improved startserv script

6.0, 5.8, 5.9, 5.9, 5.8, 5.9, 5.9, 5.8, 5.8, 5.9, 5.9

  • I implemented changes that call ProcessLauncher directly from the asadmin VM rather than going through startserv script
  • No security information (admin user and password, master password) is transmitted
  • All previous behavior is intact. The same asadmin command is used to run Before and After. The only difference is that the After version sets -DNEW_LAUNCHER=true which the new code looks for.

Before

C:\jws\glassfish\admin\servermgmt>asadmin start-domain
Starting Domain domain1, please wait.
Log redirected to c:\ee\domains\domain1\logs\server.log.
Domain domain1 is ready to receive client requests. Additional services are being started in background.
Domain [domain1] is running [Sun Java System Application Server Platform Edition 9.1 (build )] with its configuration and logs at: [c:\ee\do
mains].
Admin Console is available at [http://localhost:4848].
Use the same port [4848] for "asadmin" commands.
User web applications are available at these URLs:
[http://localhost:8080 https://localhost:8181 ].
Following web-contexts are available:
[/web1 /asadmin ].
Standard JMX Clients (like JConsole) can connect to JMXServiceURL:
[service:jmx:rmi:///jndi/rmi://hp:8686/jmxrmi] for domain management purposes.
Domain listens on at least following ports for connections:
[8080 8181 4848 3700 3820 3920 8686 ].


Time to start domain: 15.8 seconds.

After

C:\jws\glassfish\admin\servermgmt>cli start-domain
Starting Domain domain1, please wait.
Log redirected to c:\ee\domains\domain1\logs\server.log.
Domain domain1 is ready to receive client requests. Additional services are being started in background.
Domain [domain1] is running [Sun Java System Application Server Platform Edition 9.1 (build )] with its configuration and logs at: [c:\ee\d
mains].
Admin Console is available at [http://localhost:4848].
Use the same port [4848] for "asadmin" commands.
User web applications are available at these URLs:
[http://localhost:8080 https://localhost:8181 ].
Following web-contexts are available:
[/web1 /asadmin ].
Standard JMX Clients (like JConsole) can connect to JMXServiceURL:
[service:jmx:rmi:///jndi/rmi://hp:8686/jmxrmi] for domain management purposes.
Domain listens on at least following ports for connections:
[8080 8181 4848 3700 3820 3920 8686 ].


Time to start domain: 1.5 seconds.

The following table shows timing results of New code versus Old code. Quick refers to the quick-startup option. Six trials were done and an average calculated. The data was generated on a Solaris 10 system with this command:
time asadmin start-domain (all readings in seconds)
note: Script means running the startserv script itself, i.e. no asadmin involvement.

Version Quick Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 Average
Old Yes 8.823 7.498 8.731 6.811 9.111 7.480 8.076
New Yes 6.193 6.031 6.050 6.532 6.331 5.854 6.165
Old No 25.214 24.811 25.371 25.261 24.992 25.281 25.155
New No 24.452 25.081 25.021 24.501 24.471 25.041 24.761
Script No 2.01 1.997 2.803 2.184 2.341 2.119 2.242

The following table is from data collected from a Windows XP desktop system (1 GB RAM)
time asadmin start-domain

Version Quick Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 Average
Old Yes 18.146 17.618 12.787 17.634 12.842 18.370 16.233
New Yes 10.226 10.334 10.343 10.346 10.321 15.367 11.156
Script Yes 3.708 4.393 3.721 3.713 3.700 3.702 3.823