Admin Developer Tests (devtests) Developer tests for administration functionality, especially tests that run asadmin commands, currently exist in two places, as we migrate from one framework to another. Information about the old location is further down on the page in a section called "Old Admin Devtest Development". Information about developing new tests immediately follows this introduction. Developer tests are integration tests developed by developers. These tests depend on a fully built distribution consisting of many modules, unlike unit tests that only depend on a single module. The source code for the developer tests is centrally located, apart from any specific modules. The new devtests are based on the TestNG framework and they reside within the GlassFish source code directory tree. This means that as GlassFish is branched, the devtests will automatically be branched with the code. No more having to have separate branches for devtests. The way that tests are counted is different between the old and new framework. With the old framework, every call to "report" is consisted a test and counted. With the TestNG framework, only the test methods are counted, even if a test method has many assert statements in it. So keep this in mind when converting tests; the number of tests will appear to go down. Workspace Info Tests that depend only on nucleus features are here: main/nucleus/tests/admin To checkout the tests, just following the regular procedure for checking out GlassFish, which includes nucleus. Adding a New Test Tests are contained within Java files in the org.glassfish.nucleus.admin package. To add a test to an existing class, just add a method to the class. To add a new suite of tests in new class, just create a new class within this package (or any other package within the module). Copy an existing test class to get started. When a new test is added, there is no need to edit the POM file or any other file within the module. The devtests make use of a utility module (utils-ng) for some useful functions when writing tests. These include, for example, the "nadmin" method which is used to run the nadmin (aka asadmin) command. Look at the NucleusTestUtils class for more details. Instructions to Run the Tests To run the tests, build nucleus and then do:
cd main/nucleus/tests/admin
mvn test
This will run the tests against the nucleus distribution that is staged in main/distributions/nucleus. To run the tests against an installation of your choice, use the nucleus.home property setting, as in:
cd main/nucleus/tests/admin
mvn -Dnucleus.home=$HOME/test/glassfish3 test
The nucleus.home can point to any distribution of GlassFish based on nucleus. Running a subset of the Tests To run a subset of the tests, use the -Dtest property. The test property is a list of test classes to run. See the pom.xml file for an example of how to run a subset of the tests. Hudson Jobs (Oracle internal) The "nucleus-admin-devtests-trunk" job on the hudson-sca internal hudson server runs the nucleus admin devtests after every build:
Workspace Info If you have an extra hour or three and want to see everything (50,000+ directories) do this:
*svn co https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests* To get just exactly what you need for our devtests run the following (Windows version) __
md \gf\v2\appserv-tests
cd \gf\v2\appserv-tests
svn co https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/lib https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/config https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/util
svn co https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/devtests/admin/cli devtests\admin\cli
cd ..
svn co -N https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests
__ Mac/Unix version __
mkdir -p gf/v2/appserv-tests
cd gf/v2/appserv-tests
svn co https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/lib https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/config https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/util
svn co https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests/devtests/admin/cli devtests/admin/cli
cd ..
svn co -N https://svn.java.net/svn/glassfish~svn/trunk/v2/appserv-tests
__ Byron's List of Tips and Gotchas Gotchas
- the report(string, boolean) does some pretty unexpected stuff:
- If the string has a space in it – everything before the first space is thrown away!!
- If you have 100 tests all with the same string – it will show up as one test. Only the very first test will count as pass fail. The other 99 tests and their results are totally ignored.
- June 10 – There is now a base class, AdminBaseDevTest, that has report functions that solve these two problems. All admin tests should extend this base class.
Tips
- If you run asadmin(String label, String... args) it will return a boolean and print everything out. The text is gone.
- If you run asadminWithOutput(String label, String... args) then the output text will not be printed. But the Object it returns has all of the output and your boolean result. You can use the output for more sophisticated testing. I added this capability June 3,2010
- Example I create 10 instances. Then I run list-instances. Then delete the 10 instances and run list-instances again. list-instances always return success. Were ALL 10 really deleted? An easy way to find out is to check the return string for "Nothing to list."
1) the get command is laborious and nasty to parse from devtests. Let your baseclass take the drudgery out of it for you like so: report("i3-server-element-uses-24850", doesGetMatch(
"servers.server.i3.system-property.ASADMIN_LISTENER_PORT.value",
"24850")); – here it verifies that the value of that key is "24850". 1B) or you can use the get method: String s = get("servers.server.i3.system-property.ASADMIN_LISTENER_PORT.value"); 2) I added a "server" that will run for the duration of the JVM with a port that you specify. It is a daemon so it will die when the tests are done.
– I use it to clog up a port to test how the port-assigning code reacts runFakeServerDaemon(24851); June 22 I can't stand the tens of thousands of lines of junk printed out anymore! I added a verbose option. By default the devtests for everyone else will be as chatty as ever. For our tests they will be non-verbose by default. I hardcoded a call to setVerbose(false) in our base class. If you want to help – then figure out a nice way to do it via -D commandline args. What it does: It only prints out the commands that officially fail. You will see a dramatic change in the amount of output. Important: If you simply set AS_LOGFILE=your-path then EVERY CLI command and result will be saved there. So you can always locate all the commands that were run. THis technique is also handy with QL tests. October 27, 2011 I have a handy dandy script I use on UNIX and Windows that checks out the devtests, 3.1.2 branch and the trunk all into one tree. I'm putting it here for safe-keeping
svn=https://svn.java.net/svn/glassfish~svn
cd /export/home/bnlocal
svn co -N $svn gf
cd gf
svn co -N $svn/trunk
svn co -N $svn/branches
cd trunk
svn co -N $svn/trunk/v2
cd v2
svn co -N $svn/trunk/v2/appserv-tests
cd appserv-tests
svn co $svn/trunk/v2/appserv-tests/lib
svn co $svn/trunk/v2/appserv-tests/config
svn co $svn/trunk/v2/appserv-tests/util
svn co $svn/trunk/v2/appserv-tests/devtests/admin/cli devtests/admin/cli
svn co $svn/trunk/v2/appserv-tests/devtests/cluster
cd /export/home/bnlocal/gf
svn co $svn/trunk/main trunk/main
cd /export/home/bnlocal/gf/branches
svn co $svn/branches/3.1.2
Instructions to run the devtests There is a build file which will build and run all the tests. All you need are these 4 basic steps. These steps are for running the tests on the trunk.
cd appserv-tests/devtests/admin/cli
export APS_HOME=<appserv-tests-dir>
export S1AS_HOME=<directory where glassfish is installed> if I unzip glassfish.zip into $HOME/test, S1AS_HOME=$HOME/test/glassfishv3/glassfish
ant all
__Results are stored in $APS_HOME/test_results.html __ Instructions to run a subset of the tests
ant -Dteststorun=[cluster|instance|...] all
The value to use (cluster, instance, etc), should match a target name in build.xml. Instructions to run more tests based on the num_tests parameter The num_tests property determines how many instances are created in the instances test. The default value is 2.
Adding a new test Adding a new test is easy.
For easy reporting and simplicity there is a separate class for each suite of tests.
Under admin/cli/src/admin there are different classes like ClusterTest, InstanceTest, etc. The easiest way to add tests is to add them to one of the existing classes. The "run" method of the test must be modified to actually call your test method. Another option is to create a new class, say NodeTest, for which you want to add tests. Add a target to the build.xml to run the new tests, and add the test to the list of dependencies for the "cli" test. Copy one of the existing test files, such as InstanceTest.java.
@Override
protected String getTestName() {
return "cluster";
}
@Override
protected String getTestDescription() {
return "Unit test for create/delete/list cluster";
}
Now to add new tests, all you need is to add a line like this in the run method or define a separate method for your test.
report("create-cluster", asadmin("create-cluster","cl1"));
The following shows an example of adding system properties,passing other params to the cli command
report("create-cluster-system-props", asadmin("create-cluster",
"--systemproperties" ,"foo=bar",
"cl4"));
An example of how to use AsadminReturn to check for system properties in domain.xml
AsadminReturn ret = asadminWithOutput("get", "clusters.cluster.cl4.system-property.foo.name");
boolean success = ret.outAndErr.indexOf("clusters.cluster.cl4.system-property.foo.name=foo") >= 0;
report("check-cluster-syspropname", success);
ret = asadminWithOutput("get", "clusters.cluster.cl4.system-property.foo.value");
success = ret.outAndErr.indexOf("clusters.cluster.cl4.system-property.foo.value=bar") >= 0;
report("check-cluster-syspropvalue", success);
An example on how to use XPath apis to check for certain values
//evaluate using xpath that there are 3 elements in the domain.xml
String xpathExpr = "count"+"("+"/domain/clusters/cluster"+")";
Object o = evalXPath(xpathExpr, XPathConstants.NUMBER);
System.out.println ("No of cluster elements in cluster: "+o);
if (o instanceof Double) {
report ("evaluation-xpath-create-cluster",o.equals(new Double("3")));
} else {
report ("evaluation-xpath-create-cluster",false);
}
Some tests will require some cleanup at the end to ensure the next round is successful like deleting clusters, instances etc. Take a look at the AdminBaseDevTest for useful methods that can be used in tests. Finally commit the files The whole source code of a test is here https://glassfish-svn.java.net/source/browse/glassfish-svn/trunk/v2/appserv-tests/devtests/admin/cli/src/admin/ClusterTest.java There is a file created called test.out which will show how the commands got executed. Here is how a sample file looks :
asadmin --host localhost --port 4848 --user admin --interactive=false --echo=true --terse=false create-cluster cl1
Command create-cluster executed successfully.
asadmin --host localhost --port 4848 --user admin --interactive=false --echo=true --terse=false create-cluster --config cl1-config cl2
Command create-cluster executed successfully.
asadmin --host localhost --port 4848 --user admin --interactive=false --echo=true --terse=false create-cluster cl1
Command create-cluster failed.
org.glassfish.api.admin.CommandException: remote failure: Exception while adding the new configuration org.jvnet.hk2.config.TransactionFailure: A Cluster instance with a "cl1" name already exist in the configuration : org.jvnet.hk2.config.TransactionFailure: A Cluster instance with a "cl1" name already exist in the configuration
asadmin --host localhost --port 4848 --user admin --interactive=false --echo=true --terse=false create-cluster --config junk-config cl3
Command create-cluster failed.
Just-In-Time Debugging Our DevTests are all-or-none. One test failing is just exactly as bad as 100 tests failing. I added a feature that makes it a bit easier to debug problems. The monitoring devtests ALWAYS run with jpda debugging turned on (suspend = "n", of course!!). The moment ANY test fails the screen is cleared and output is displayed that describes what to do. The code will sit and wait 60 seconds for you to attach your debugger to port 9999. It even tells you how to force it to stop waiting early so you can start debugging – by setting a static variable to true. This will add one minute to every failed test so I suppose this could create hudson issues. If there are many failures then we'd wait one minute for each. You must do something to enable the JIT debugging. You can set either an environmental variable or System Property as below – and then and only then will the JIT Debugging become enabled. export APS_WAIT=true
set APS_WAIT=true -DAPS_WAIT="true" Now if we get some weird issue where Hudson fails but your environment succeeds – then simply change hudson.sh and set the variable and then you can debug at the exact right moment on the Hudson machine! – Currently this only applies to monitoring devtests but it can be used for all devtests. The main work is adding the right jvmargs to the bazillion different ant targets in build.xml A very useful technique is to attach and then set a breakpoint in the code right after the failed test (look in the call stack). Once it hits, I can leave the debugger and I now have full access to the server(s) in exactly the state I want. I can run plenty of manual tests directly with the server (what does get -m return? Which instances are really running? etc., etc.) |
