Secure Administration One Pager

(template version 1.91)

Introduction

Project/Component Working Name:

Secure Administration

Name(s) and e-mail address of Document Author(s)/Supplier:

Tim Quinn: tim dot quinn at oracle dot com

Date of This Document:

07/21/2010

Project Summary

Project Description:

Administrators should be able to use secure administration, which means:

• Admin messages between admin clients and the DAS, between the DAS and remote instances, and between local admin clients and instances are encrypted using SSL/TLS.
• The sender and receiver of admin messages authenticate to each other.
• Credentials or other sensitive information sent over the network are always encrypted.

Admin clients can be the asadmin command line tool, the admin console, IDEs, browsers, and any client using the REST interface.

This one-pager describes our plans for securing that traffic.

See below for a longer, more detailed technical description.

Risks and Assumptions:

• Grizzly support for
  • port unification
  • redirection (http -> https)
  • SSL/TLS
  • returning Principal associated with request

• Performance of SSL/TLS between DAS and instances

Problem Summary

Problem Area:

Administrators need to be able to secure the admin traffic among the runtime components: admin clients, the DAS, and remote instances. Specific requirements:

• Imposing a heightened level of security is optional.

When the administrator enables elevated admin security:

• Both communicating parties authenticate to each other.
• No credentials are ever sent in the clear.
• GlassFish helps prevent DAS-to-DAS and instance-to-instance traffic and carefully restricts admin client-to-instance traffic.

Justification:

Some users will want or need to operate in a secure environment, in which they are confident that rogue users or processes cannot intercept or corrupt admin traffic or impersonate legitimate GlassFish components.

Technical Description:

User Experience:

The secure admin feature allows an administrator to secure all communication among the domain admin server, any remote instances, and admin clients such as the asadmin utility, the admin console, and REST clients.

Enabling and disabling secure admin has these main effects:

Secure admin is a domain-wide setting. It affects the DAS and all instances and all admin clients.

If you use xxx-local-instance commands to set up remote instances, plan to either leave admin security disabled or enable it before you create or start instances and leave it that way. Although GlassFish will let you switch back and forth, doing so creates a cumbersome situation. (details below) If, on the other hand, you use xxx-instance-ssh commands to manage remote instances, you can enable and disable secure admin at will without problem although this is not recommended for security reasons.

New CLI Commands

Note: Make sure only the domain admin server, and no instances, are running when you run these commands.

enable-secure-admin

enable-secure-admin
[--adminalias=alias (default s1as)]
[ --instancealias=alias (default glassfish-instance)]

Most developers will use enable-secure-admin without options.

Restart the domain admin server immediately after enabling secure admin.

Out-of-the-box, GlassFish uses self-signed certificates to identify the DAS and the instances to each other and to clients.  You can have GlassFish use your own certificates for this purpose by first adding your certificates to the GlassFish keystore and truststore, then running enable-secure-admin and specifying the aliases for your certificates.

disable-secure-admin

disable-secure-admin

If you have previously enabled secure admin, this disables secure admin, restoring the out-of-the-box configuration.

Default Admin Account

If only one admin account exists in the admin realm, GlassFish treats that account as the current default admin account. In this case, when you run an asadmin command you do not need to specify the username. If the password for that username is not empty you do need to specify it, typically using the --passwordfile option or letting asadmin prompt you for it.

If multiple admin accounts exist then GlassFish does not recognize any admin username as the default. You must specify a valid username and its associated password (if the associated password is not empty) when you use asadmin.

Out-of-the-Box

Secure admin is disabled. The default admin account is username "admin" with an empty password.

Accepted and Rejected Operations if Secure Admin is Disabled

[1] This can happen only if you created the local instance while secure admin was enabled and then disabled secure admin.  Don't do that.

Prompting about and Remembering Certificates

The self-signed certificates that GlassFish uses are not trustworthy, because no certificate authority vouches for the authenticity of the certificate.  If you enable secure admin and then contact the DAS using an admin client, that client will detect that the cert is not inherently trustworthy.  Browsers will warn you, let you view the certificate, and then ask you to reject the cert, accept it once, or accept it indefinitely.  The first time the asadmin program sees a given untrusted cert it will display the cert and let you accept it or reject it.  If you accept it, asadmin will also accept that cert in the future. (same as in GlassFish 2.x).

Some asadmin commands - such as run-script - can contact an instance directly to retrieve information (but not to make configuration changes).  The instances do not use the same cert as the DAS, so in such cases you will be prompted again for the instance cert this time.

The asadmin program saves certs you accept in the file .asadmintruststore in your log-in default directory.  You do not normally need to work with the file directly, but if you delete it or move it asadmin will prompt you again when it sees untrusted certs.

Technical Details:

Two-step process to Authenticate Admin Requests

Every admin request arrives to the DAS as an HTTP or HTTPS request. Grizzly routes each request to the proper Grizzly adapter. Examples are

• the AdminAdapter, which handles requests of the form http(s)://das:port/__asadmin, which includes requests from the asadmin utility
• the REST adapter, which handles http(s)://das:port/monitoring requests.

These adapters delegate the authentication task to an implementation of AdminAuthenticator. This class considers several factors in deciding what access should be granted:

• whether the request is local or remote
• whether the request used DAS or instance certificate authentication
• whether the current server is the DAS or an instance
• whether secure admin is enabled
• whether the username and password are valid in the admin realm

The AdminAuthenticator lets the invoking adapter know what kind of access to allow using a fairly coarse approach:

It is up to the invoking adapter to enforce any access restriction. For example, in GlassFish 3.1 the AdminAdapter has no way to enforce monitoring-only access, so it rejects requests that are eligible only for monitoring access. This would happen, for example, if a user accidentally sent a request directly to an instance.

The REST adapter, on the other hand, can tell easily whether a given request is for monitoring or management and so can accept or reject each request depending on the level of access returned from the AdminAdapter.

With this division of labor, the AdminAuthenticator, to which all adapters delegate, does the bulk of the decision-making so the logic resides in one place, not duplicated among the adapters.

Authenticator Logic

Currently, the class GenericAdminAuthenticator is the implementation for AdminAuthenticator. This class basically does the following to authenticate an incoming request or connection and decide what level of access to grant:

if (request uses locally-provisioned password) {
    return FULL access;
}

if (    (request came to an instance from the DAS using the DAS cert )
     || (request came to the DAS from an instance using the instance cert [1])
     || (request came from the DAS or an instance using a special marker [2]) 
     || (request triggered by the DAS but submitted from another process [3]) ) {
    return FULL access;
}

if (request has a valid admin user/password) {
    if (current server is the DAS) {
        if (   secure admin is enabled
            || request came from current host ) {
            return FULL access;
        }
    } 
    // This is an instance OR secure admin is off OR the request is remote
    return MONITORING access;
}

// username/password absent from the admin realm
return NO access;

1 In GlassFish 3.1 instances do not send requests to the DAS but the architecture and the code is in place to support that.

2 If secure admin is disabled, the DAS uses a special marking technique so an instance can identify the DAS as the origin of the request.

3 Either on the same system as the DAS in another process or on a non-DAS system using ssh.

Identifying Requests that Originate from the DAS

If secure admin is enabled, this is easy: The SSL certificate-based authentication presents the DAS certificate to the instance. The instance compares the Principal associated with the certificate against the Principal in its truststore stored with the DAS alias.

If secure admin is disabled, the DAS adds a header to the HTTP request that indicates the DAS is the origin. This is not secure but it is convenient.

We do not turn on SSL for the instance admin port if secure admin is disabled. Some components (such as REST and monitoring scripting) connect to the instance admin port for read-only access. If we used SSL all the time on that port then users of those components would be prompted with the instance's cert upon the first connection. We decided to avoid that extra step in the initial user experience.

Running GlassFish out-of-the-box

The DAS admin realm contains a single user, the default admin user with "admin" as the username and an empty password. Admin clients provide empty credentials or none at all, so all are authenticated and authorized as that default admin user.

None of the participants (clients, DAS, or instances) encrypts network messages.

Running GlassFish with secure admin

In the steady state the DAS and one or more remote instances are installed, initialized, and running, and admin clients are installed. The administrator has specified two aliases corresponding to certificates in the keystore and truststore: one which the DAS will use for authenticating itself in admin traffic, and one which the instances will use for authenticating.

The DAS and instances always identify themselves using SSL/TLS certificate authentication, whether as senders or receivers of admin requests.

Valid admin clients always identify themselves using an HTTP Authorization header (with one exception, noted below).

For the DAS or an instance to accept an admin request, it

• establishes the identity of the requester (authentication), and
• makes sure that requester is allowed to perform admin tasks (authorization).

Remote admin clients (asadmin, admin console, browsers, and IDEs) send a username and password in the HTTP Authorization header (as in GlassFish 2). The receiving DAS makes sure those credentials appear in its admin realm, authenticating and authorizing at once.

A locally-running asadmin (that is, connecting to an instance on the same host) authenticates and authorizes to the co-located instance using the locally-provisioned password as described here.

A server (DAS or instance) sending a request will authenticate itself at the SSL/TLS level using a certificate.

The AdminAdapter tries to identify the sender of the request in this order:

• Check for the locally-provisioned password in the HTTP Authorization header. If it matches, accept and execute the request.
• Check for the SSL/TLS cert Principal from the Grizzly request. If the DAS receives a request authenticated with the instance certificate, or if an instance receives a request authenticated with the DAS certificate, then the receiver accepts and executes the request.
• DAS only
  • If the credentials from the HTTP Authorization header appear in the admin realm, accept and execute the request.
  • If the HTTP request contains no Authorization header but one is required, or if the request contains such a header but there is no corresponding entry in the admin realm, the AdminAdapter rejects the request with 401 - NotAuthorized and returns a WWW-Authenticate challenge header with the response.

For this to work, the receiving processes need to have certain data available:

We accomplish this by storing both private keys in the DAS keystore, both certs in the DAS truststore, and the alias for each in the domain.xml config. Then normal instance creation operations (create-instance over ssh, create-local-instance) will bring the up-to-date keystore, truststore, and configuration to each instance. (See also the bootstrapping discussion below.)

Note that secure admin affects the entire domain: the DAS and all instances.

Controlling and Recording Secure Admin Settings

This feature adds new subelements to <domain> in domain.xml.

<domain>
  <secure-admin enabled="true/false">
    <secure-admin-principal type="admin/instance" alias="xxx"/>
  </secure-admin>
...

Normally, with secure admin enabled, we'll have one <secure-admin-principal> for type "admin" (DAS) and one for type "instance." This configuration exists on both the DAS and the instances. The DAS will use the alias associated with the <secure-admin-principal type="instance"> element to check incoming requests that use SSL/TLS cert authentication. The instances will use the alias associated with the <secure-admin-principal type="admin"> element to check incoming requests with cert authentication. (Over time we might want to support multiple aliases of each type. The proposed changes to the config would permit this although the current proposal does not include commands or command options to do so.)

This feature also adds two new commands for controlling secure admin settings.

enable-secure-admin

enable-secure-admin
[--adminalias=alias (default s1as)]
[ --instancealias=alias (default glassfish-instance)]

The enable-secure-admin command will:

• Add new or modify existing <secure-admin> and <secure-admin-principal> elements under <domain>. It will set the <secure-admin enabled> attribute to "true."
• Create the truststore if it is missing.
• If the keystore and truststore do not contain a cert for the instance alias, then generate the instance self-signed key pair and add the private key to the keystore and the public cert to the truststore.
• If the truststore does not contain a cert for the DAS alias, add the DAS cert from the keystore to the truststore.
• Adjust all configurations in the domain, including default-config, creating or updating <secure-admin> and changing the Grizzly settings:
  • Enable SSL/TLS, using the specified --adminalias value in the DAS's admin listener and the --instancealias value in the instances' admin listeners.
  • Set up Grizzly port unification, redirection, and client-auth=want.

If the <secure-admin> fragment already exists in domain.xml when the user runs enable-secure-admin then the alias values in the <secure-admin-principal> elements are changed only if the user specifies --adminalias or --instancealias on the command.

The enable-secure-admin command implementation will send the hidden _instance-enable-secure-admin command to all non-DAS targets in the domain. This command performs the same configuration changes as enable-secure-admin does on the DAS.

A server restart is required to change the Grizzly adapter behavior, which will also sync the restarted instances which will deliver updated keystore and truststore files (if they have in fact changed).

disable-secure-admin

disable-secure-admin

This command:

• Sets the enabled attribute of <secure-admin> to false.
• Turns off Grizzly's use of SSL/TLS in the Grizzly configuration.
• Turns off Grizzly redirection from http to https.
• Turns off Grizzly port unification for the admin listener port.

Detailed Use Cases

Next are detailed use cases, showing the moving parts at work. Note that in any use case that ends up sending an https request SSL/TLS negotiation occurs within the SSL infrastructure between the sender and the listener. The negotiation is not shown explicitly in each use case; it is summarized in a separate section. Also, except where noted the credentials which the user supplies are assumed to be valid administrator credentials for the DAS.

Remote Client to DAS

asadmin --secure=false --user me --passwordfile myFile.txt cmd ...

The user submits a command, not asking to secure the messages using SSL/TLS encryption, and specifying credentials (--user and --passwordfile).

Note that asadmin upgrades to use https - because of the DAS redirection - despite the user's --secure=false. As it does so, it sees that the new protocol is https so it proactively sends the HTTP Authorization header along with the redirected request.

asadmin --secure=true --user me --passwordfile myFile.txt cmd...

In this case asadmin connects using https (the user specified --secure=true) and so sends the HTTP header immediately.

asadmin --secure=true cmd...

asadmin --secure=false cmd...

Browser access to the DAS - user enters http://das:4848/...

SSL/TLS negotiation

This happens any time a client contacts the DAS using https as part of establishing the secure connection.

By default, the DAS uses a self-signed cert for authentication. Because a self-signed cert lacks a cert chain to a trusted cert authority, neither browsers nor asadmin will automatically accept them as trusted. Rather, the user is warned, shown information about the cert, and asked if he/she wants to trust the other party.

[
[
  Version: V3
  Subject: CN=timothy-quinns-macbook-pro-2.local, OU=Sun GlassFish Enterprise Server, O=Sun Microsystems, L=Santa Clara, ST=California, C=US
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 1024 bits
  modulus: 145793420626688566088394158159638779041277831140455913481435972977559795017671574823864315131750418659965821779796460080673147008571695264778429964788011972328001078278671878387307747704372091832930835291346891002043684898471960200484166079708458546502354191033994197133166962960373293979307533373507167827001
  public exponent: 65537
  Validity: [From: Tue May 04 12:32:11 CDT 2010,
               To: Fri May 01 12:32:11 CDT 2020]
  Issuer: CN=timothy-quinns-macbook-pro-2.local, OU=Sun GlassFish Enterprise Server, O=Sun Microsystems, L=Santa Clara, ST=California, C=US
  SerialNumber: [    4be05a1b]

Certificate Extensions: 1
[1]: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 6F 61 EA F4 1A F0 21 DB   B6 B0 70 0B 10 08 32 15  oa....!...p...2.
0010: 40 BA D2 BC                                        @...
]
]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 03 3A 70 C2 26 20 78 B5   DF 60 C2 5B C1 20 EA 5D  .:p.& x..`.[. .]
0010: D6 29 0A 65 4F 08 41 6A   10 BC 5A 5C A7 2B CA A0  .).eO.Aj..Z\.+..
0020: 9D 75 3C 02 D8 41 D5 10   95 CC FB 0B FB 17 88 2E  .u<..A..........
0030: 54 05 97 08 22 96 D9 A7   BB EB 70 6F 87 8B 52 88  T...".....po..R.
0040: EB E2 E0 3B F2 F4 84 07   E1 50 87 5E 51 F3 A1 BF  ...;.....P.^Q...
0050: 85 A2 DE 60 93 26 75 1F   FD B1 18 3E 4F DB 48 E1  ...`.&u....>O.H.
0060: DA 10 CC 8C 8D 7A 01 B5   62 4E EA 8D 0C 8E A3 7C  .....z..bN......
0070: E5 26 60 E0 BF 3F 14 BF   02 E3 4F FC AF 76 B6 8D  .&`..?....O..v..

]
Do you trust the above certificate [y|n] -->

Note that if the user agrees to trust the cert, then asadmin stores the cert into the user's private asadmin truststore (~/.asadminstruststore) and will accept that cert going forward. Browsers tend to ask the user to reject the cert, accept it for the current access, or accept it indefinitely.

Unsecured, anonymous administrator login

If the administrator does not enable elevated security, then Grizzly on the DAS and on the instances is not configured for SSL/TLS and the admin security realm contains no entries for administrator users.

Local asadmin client to instance

Parts of clustering require a user to use asadmin with the instance on the same node. This uses the locally-provisioned password mechanism . Briefly, during instance start-up the instance generates a local password, storing it in memory and also writing it to a protected file on disk. The asadmin program reads the password from the protected file and sends it with an empty username for HTTP authorization. The instance compares the password sent from asadmin with its in-memory copy of the password and, if they agree, treats the client as a fully-privileged administrative user.

DAS to Instance, Instance to DAS

The DAS and instances uses SSL mutual authentication. That is, the DAS trusts a cert offered by the instance and the instance trusts a cert offered by the DAS. The configuration's <secure-admin> fragment gives the DAS and instance aliases, so the receiving AdminAdapter can check to see if the requester used cert-based authentication and, if so, if the corresponding Principal matches the one fetched from the truststore using the correct alias.

Guarding against Unwanted Connections

DAS-to-DAS, instance-to-instance

As discussed in the bootstrapping section below, the DAS and the instances will have copies of the same truststores which contains the public cert of the DAS and the separate public cert that is used by all instances.

DAS-to-other-DAS communication will not be authenticated because each different DAS will have its own self-signed cert that will not be in the truststore of the other DAS.

DAS-to-self communication is unlikely but could conceivably happen if the user incorrectly configures the admin listener port for an instance on the same host so it is the same as for the DAS. Similarly, instance-to-instance traffic is unlikely but possible if the user incorrectly configures listener ports for instances on the same host. To prevent both of these problems, AdminAdapter will check to make sure that if the client has authenticated using SSL/TLS client authentication that the Principal associated with the remote client is not the same as the current process. That is, the DAS will make sure that the Principal is not itself. Similarly, each instance will make sure that the client is not an instance. (This is done easily because the instances share the same self-signed cert and therefore are mapped to the same Principal.) Both cases are handled by making sure that the connecting Principal is not the running Principal.

Remote client-to-instance

An earlier use case described how asadmin can connect to a local instance. Remote clients will be unable to connect directly to instances. If the user on host "foo" runs a local command but specifies a remote instance on host "bar", asadmin on foo will read and send that locally-provisioned password. The instance on "bar" will have a different locally-provisioned password and so the authentication attempt will fail. Further, AdminAdapter on instances will have no admin realm to use in verifying HTTP Authorization-based authentication. So a user on "foo" will not be able to run a remote command targeting an instance on "bar."

Bootstrapping and Set-up

Three main areas need to be handled as part of "bootstrapping" a remote instance:

• configuring Grizzly,
• placing the keystore and truststore in the remote instance's file system, and
• storing the DAS cert Principal in the instance's admin realm.

Configuring Grizzly

The Grizzly config on the DAS and each instance are identical with one difference: the alias to be used for SSL/TLS authentication, shown below as ${aliasName}:

• Enable port unification (so http, https handled by the same port) on DAS
• Redirect "http://xxx:${adminPort}" -> "https://xxx:${adminPort}"
• Configure SSL client-auth=want
• Configure SSL to use the admin keystore and the alias "${aliasName}" (for the self-signed cert) for authenticating itself. The administrator can load his or her own cert into the keystore under a different alias and configure Grizzly to use that alias instead.
• Configure SSL to use the admin truststore

Building GlassFish and Creating a Domain

• (as in GF 2 and 3.0) DAS self-signed key pair - Private key already present in the keystore under the alias "s1as" when the user installs GlassFish or creates a new domain.
• Instance self-signed cert - Add to the GlassFish build and create-domain logic to create a second self-signed key pair and place its private key also into the keystore under the alias "gf-instance."
• Insert the public keys for the DAS self-signed cert and the instance self-signed cert into a new truststore file.

Creating a New Instance

Remotely using asadmin create-instance and ssh

1. create-instance command is remote, so the user authenticates to the DAS.
2. create-instance runs the hidden _register-instance command. (no contact yet between the new instance and the DAS)
3. create-instance uses ssh to the remote system to run the _create-local-filesystem command, a local command that therefore does not itself require authentication to the DAS. As part of this, create-instance sends data through the ssh connection, readable from System.in by _create-local-filesystem:

• the DAS keystore and truststore contents,
• the configuration (the portion of domain.xml) that this new instance will use as its configuration (primarily to carry the Grizzly config for security),
• DAS Principal name.

_create-local-filesystem then stores the keystore and truststore contents it reads into the keystore and truststore files in the instance's file system, the configuration into the instance's copy of domain.xml, and the DAS Principal name into the file-based admin realm.

Locally using asadmin create-local-instance

1. Because create-local-instance is a remote command, the user authenticates to the DAS.
2. {[create-local-instance}} sends, in its response, the keystore and truststore files, the domain.xml - or at least the portion of it relevant to the instance, and the DAS Principal name.  asadmin stores these files in the correct places in the instance's file system courtesy of the payload handling framework.

Starting the new instance

Whether created by create-instance or create-local-instance, by the time the new instance starts and contacts the DAS to sync, the keystore and truststore files, the admin realm, and the relevant part of domain.xml are correct. The instance can mutually authenticate with the DAS using certs so that the initial message and all subsequent ones are secure.

Upgrade

Upgrading a multi-instance GlassFish 2 installation to GlassFish 3.1 is similar to the bootstrapping problem.

Upgrading the DAS

Upgrade must:

1. Infer if the GlassFish 2 installation being upgraded is already configured for v2 secure communication.  If so, continue. Testing for the presence of

   <configs>
     <config name="server-config...>
       <http-service>
         <http-listener id="admin-listener"...>
           <ssl ...>

   should be sufficient to detect a secure GlassFish 2 set-up. The <ssl cert-nickname> value should be used as the DAS alias for secure-admin.

2. Run the enable-secure-admin command (which will update the Grizzly config for the DAS configuration (name = "server-config") and all other configs.) Here is where the ssl cert-nickname is used to set the adminalias value.

Upgrading the instances

Upgrade must:

1. Deliver the updated config (in domain.xml) to the instances.
2. Deliver the updated keystore, truststore, and admin realm to the instances.

This will happen as a result of starting the remote instances, part of which is to sync with the DAS thereby retrieving the updated domain.xml.

Bug/RFE Number(s):

Security for DAS/instance command protocol: 12405
Enable secure communication during upgrade: 12739

In Scope:

Out of Scope:

Interfaces:

Interfaces may be commands, files, directory structure, ports,
DTD/Schema, tools, APIs, CLIs, etc.
Note: In lieu of listing the interfaces in the one pager, providing
a link to another specification which defines the interfaces
is acceptable.

Public Interfaces

New command to enable/disable secure admin traffic.

Private Interfaces

This work introduces no new private interfaces that are externally observable.

Deprecated/Removed Interfaces:

None.

Doc Impact:

• Administration Guide
• possibly Quick-start Guide?

We will need to include instructions an administrator can follow for "tightening up" the admin security.

Admin/Config Impact:

Config to support secure admin traffic uses existing configurable items: Grizzly settings, adding admin users, etc.

HA Impact:

Affects some of the implementation of some cluster commands:

• create-instance
• create-remote-instance
• _create-local-filesystem

I18N/L10N Impact:

No impact.

Packaging, Delivery & Upgrade:

Packaging

No new IPS packages.

Delivery

No impact on product installation.

Upgrade and Migration:

See the earlier technical discussion about upgrade.

Security Impact:

No explicit impact on security, except for the configuration settings already described.

Compatibility Impact

No earlier public interfaces changed.

Dependencies:

Internal Dependencies

• Grizzly - support for port unification, redirection, SSL/TLS

External Dependencies

None

Testing Impact:

Existing tests that exercise remote instance and cluster operations can be used to exercise this feature. We can enable secure admin traffic and then run those same tests. The results should not vary.

Additional tests would be valuable to make sure forbidden connections (e.g., remote admin client -> instance) are prevented.

Reference Documents:

Schedule:

Projected Availability:

• Initially integrated (may not be feature complete): 3.1 M 4
• Feature complete (ready for handoff to QA) Ideally 3.1 M 4
• At production quality level 3.1 M 5

AdminTrafficSecurity-asarch-2010-aug-11.pdf (application/pdf)