One pager for GlassFish 3.2 Virtualization The IAAS management service (IMS) is designed so it has no understanding of what Java EE Service types are or what a GlassFish cluster is. It's basically a stateless service that gets asked to provision virtual machines and allow access to low level virtual machines management like start/stop. The IMS is a layered approach where responsibilities are divided by three components :
- IMS core which contains the implementation of the IAAS interface(s) to be used by the other GlassFish components like the orchestrator. This layer has no understanding of machine topology or virtualization solutions. It is configured with 1 to n server pools definitions that are used to delegate the real work of virtual machines allocation or lifecyle.
- Server pool master is responsible for providing a unified API to allocate or retrieve virtual machines from a set of resources it manages. Resources like machines can be added to a server pool to form a physical ServerPool or the ServerPool can act as a proxy to a virtualization management solution like OVM, EC2. ServerPools are configured for a particular virtualization solution but they delegate all the low level native virtualization
- Virtualization plugin is implementing low level access to the virtualization solution, we are planning to deliver OVM, libvirt and virtual box plugins.
Let's have a look at the different combinations : 
- Case A :
In case A, a separate machine is acting as the group master, the IMS is configured with a remote group access configuration to interface with this remote group master. The IMS forwards request for creating virtual machines to the group master that use its configured machines to allocate VM and return the information to the IMS.
- Case B :
Similar case, the group master is now co-located with the IMS, the only configuration found is the group definition and the IMS+GroupMaster is using a virtualization plugin to interface the low level virtual machine handling.
- Case C :
In case C, the group master is still co-located but the virtualization plugin is just interfacing with a public cloud infrastructure. The group definition does not contain any machine definitions, it's left to the public cloud how virtual machines are allocated. This is called an abstract group (better name ?). Example of such deployments would be EC2.
- Case D :
IAAS The IAAS layer is responsible for allocating virtual machines, managing the virtual machine (start, stop) and finally destroy permanently virtual machines. Virtual Machine creation A virtual machine creation will necessitate a virtual machine template that will be copied over to create the virtual machine disk as well as runtime properties that are specified by the caller and will be passed to the virtual machine instance. This allow for customizing the virtual machine with domain specific information. Such information can be retrieved by software installed in the template (like an init.d script) and can be used to further customize the virtual machine instance.
/**
* Defines the Infrastructure Management Service
* @author Jerome Dochez
*/
@Contract
public interface IAAS extends Iterable<ServerPool> {
/**
* Returns the runtime information for the serverPool identified by its name.
* @param groupName the serverPool name
* @return the serverPool runtime information or null if not found.
*/
ServerPool byName(String groupName);
/**
* Allocate a virtual machine based on the provided template.
*
* @param order the allocation constraints for the virtual machine allocation
* @param listeners list of synchronous {@link Listener} to register before starting any allocation
* work.
* @return a {@link ListenableFuture} to get asynchronous phases completion notification and the
* {@link VirtualMachine} instance upon allocation completion.
* @throws VirtException when the virtual machine creation failed.
*/
ListenableFuture<AllocationPhase, VirtualMachine> allocate(VMOrder order,
List<Listener<AllocationPhase>> listeners)
throws VirtException;
/**
* Allocate a virtual machine based on the provided template.
*
* @param strategy strategy to allocate the virtual machines within the machine pools.
* @param order the allocation constraints for the virtual machine allocation
* @return a {@link ListenableFuture} to get asynchronous phases completion notification and the
* {@link VirtualMachine} instance upon allocation completion.
* @param listeners list of synchronous {@link Listener} to register before starting any allocation
* work.
* @throws VirtException when the virtual machine creation failed.
*/
ListenableFuture<AllocationPhase, VirtualMachine> allocate(AllocationStrategy strategy,
VMOrder order,
List<Listener<AllocationPhase>> listeners)
throws VirtException;
}
A small variant of the API will be to provide an AllocationStrategy instance to decide in which server pools the virtual machines need to be instantiated. Further customization can be provided by returning a ServerPoolAllocationStrategy to customize how virtual machine are allocated within a server pool (to avoid co-location for instance) AllocationStrategy A set of allocation strategy instances will be available for lookup in the hk2 registry, users can define their own. We will use a default one that allocates each virtual machine on a single hardware instance following a round-robin algorithm. Default allocation strategy. The default virtual machine allocation strategy will be a simple round-robin based on ServerPools and Machines within each pool. Let's take a few examples on a environment with 3 groups (A, B, C) where Group A and B have 2 machines, and group C has 1 machine : Allocation of 5 virtual machines :
Group A |
Group A |
Group B |
Group B |
Group C |
Machine A |
MachineB |
Machine A |
Machine B |
Machine A |
1 |
2 |
3 |
4 |
5 |
Allocation of 3 virtual machines
Group A |
Group A |
Group B |
Group B |
Group C |
Machine A |
MachineB |
Machine A |
Machine B |
Machine A |
1 |
2 |
3 |
|
|
Allocation of 8 virtual machines
Group A |
Group A |
Group B |
Group B |
Group C |
Machine A |
MachineB |
Machine A |
Machine B |
Machine A |
1,6 |
2,7 |
3,8 |
4 |
5 |
Allocation of 13 virtual machines
Group A |
Group A |
Group B |
Group B |
Group C |
Machine A |
MachineB |
Machine A |
Machine B |
Machine A |
1,6,11 |
2,7,12 |
3,8,13 |
4,9 |
5,10 |
Note that the above order is the consideration order for machines to run a particular virtual machine. If the machine resources have already been exhausted, the algorithm will automatically move to the next selectable machine in the order. Machine resources will be represented by processor utilization rate and memory available. So for example, say you want to allocate 3 virtual machines and Machine A of Group B is already fully utilized, the allocation will become : Allocation of 3 virtual machines
Group A |
Group A |
Group B |
Group B |
Group C |
Machine A |
MachineB |
Machine A |
Machine B |
Machine A |
1 |
2 |
|
3 |
|
It is possible to provide a specific allocation strategy when requesting virtual machines by implementing the following interface (final interface tbd) :
/**
* An allocation strategy is responsible for allocating virtual machines given
* a set of server pools and machines within these server pools.
*
* @author Jerome Dochez
*/
public interface AllocationStrategy {
/**
* allocate a virtual machine within the provided server pools, notifying of progress all listeners
*
* @param serverPools the server pools elligible to run the virtual machine
* @param order the virtual machine requested characteristics
* @param listeners listeners to register before any work is performed
* @return a listenable future implementation that can be used to monitor progress and get the
* result of the allocation as {@link VirtualMachine} instance.
* @throws VirtException if the allocation failed.
*/
ListenableFuture<AllocationPhase, VirtualMachine> allocate(Collection<ServerPool> serverPools,
VMOrder order,
List<Listener<AllocationPhase>> listeners)
throws VirtException;
/**
* Returns the server pool allocation strategy for a particular server pool instance.
* @param serverPool the server pool instance
* @return the server pool allocation strategy
*/
ServerPoolAllocationStrategy getServerPoolStrategy(ServerPool serverPool);
}
If the allocation strategy cannot allocate all requested virtual machines because hardware resources are fully utilized, the returned allocated pools may be smaller than the requested provisioning request. In such case, it is the responsibility of the caller to decide if the reduced return is acceptable or not. If the reduced outcome is not acceptable, it is also the responsibility of the caller to delete all allocated virtual machines. VMOrder A VMOrder instance contains the characteristics of the virtual machine provisioning request :
- virtual machine attributes like memory, number of processors allocated
- template to duplicate for each virtual machine
- group affinities where these virtual machines should allocated in priority
- list of existing virtual machines to no co-locate with.
- a set of properties (key-values) to pass to the virtual machine so it can configure itself.
/**
* Virtual machine procurement order.
* @author Jerome Dochez
*/
public class VMOrder {
final TemplateInstance template;
final VirtualCluster targetCluster;
final List<ServerPool> groups = new ArrayList<ServerPool>();
final List<VirtualMachine> noColocationList = new ArrayList<VirtualMachine>();
final Properties vmProps = new Properties();
public VMOrder(TemplateInstance template, VirtualCluster targetCluster) {
this.template = template;
this.targetCluster = targetCluster;
}
/**
* Returns the virtual cluster this allocation is targeted to
* @return the target virtual cluster
*/
public VirtualCluster getTargetCluster() {
return targetCluster;
}
/**
* Returns the requested characteristics for the virtual machines
* @return a set of static virtual machine characteristics to use when
* allocating the new virtual machines or null if the template characteristics
* should apply.
*/
public StaticVirtualMachineInfo getCharacteristics() {
return null;
}
/**
* Specifies the serverPool in which the number of virtual machines should be allocated.
* If no serverPool is specified, it's left to the Infrastructure Management Service to
* decide in which groups those Virtual Machines will be allocated.
*
* @param groups desired serverPool instance
* @return itself
*/
public VMOrder in(ServerPool... groups) {
this.groups.addAll(Arrays.asList(groups));
return this;
}
/**
* Specifies the virtual machines that should not be co-located on the same hardware
* with the new allocated virtual machines. This is particularly useful when willing
* to allocate replication instances for existing virtual machines so they do not end
* up being running on the same hardware resource.
*
* @param vms list of virtual machines to not co-locate with.
* @return itself.
*/
public VMOrder noColocationWith(VirtualMachine... vms) {
this.noColocationList.addAll(Arrays.asList(vms));
return this;
}
/**
* Returns the properties for a specific virtual machine allocation. These properties
* will be passed to the virtual machine through a provider specific mechanism. Such
* properties can be used by the virtual machine to configure itself.
*
* @return the virtual machine properties
*/
public Properties getVirtualMachineProperties() {
return vmProps;
}
/**
* Returns the groups this set of virtual machine allocations should be allocated into
*
* @return the groups we should use to allocate the virtual machines if possible.
*/
public Collection<ServerPool> affinities() {
return Collections.unmodifiableList(groups);
}
/**
* Returns a list of virtual machine we would like the new virtual machines to not be
* co-located with (meaning not running on the same hardware resource).
*
* @return a list of virtual machines to no co-locate new ones with.
*/
public List<VirtualMachine> separateFrom() {
return Collections.unmodifiableList(noColocationList);
}
/**
* Returns the template associated with the virtual machine order.
*
* @return the template to use for the virtual machines allocation.
*/
public TemplateInstance getTemplate() {
return template;
}
}
Customization Properties When creating or rebooting a virtual machine, a number of properties are passed from the IAAS layer to the virtual machine instance. How such properties are passed is a virtualization provider specific. The list of properties passed that can be used within the virtual machine is :
Property Name |
Description |
Cluster |
Name of the GlassFish cluster this instance will belong too |
Instance |
Name of the intended GlassFish instance |
IMSPort |
Port used by the IMS (by default, it's the DAS administration port |
IMSMachine |
IP address used by the IMS (by default, it's the DAS |
DAS |
DAS machine name |
DASAddress |
DAS IP Address |
DASPort |
DAS Administrative Port |
AuthToken |
Authentication token to talk to the DAS |
Some virtualization technologies will have the ability to add more properties which are private contracts between the virt provider and the template. For instance, libvirt also adds those properties :
Property Name |
Description |
UserName |
User name that will be running GlassFish instance in this virtual machine |
UserId |
User id |
GroupId |
User's group id |
Any virtual machine specific properties passed to the VMOrder instance (see above) will also be stored in the customization properties. Virtual Machine Once a virtual machine is created, it's possible to get some runtime information like its IP address or performs some life-cycle operations like starting, stopping and destroying.
/**
* Defines a Virtual machine
* @author Jerome Dochez
*/
public interface VirtualMachine {
/**
* Returns the machine's name
* @return machine's name as it is registered in the configuration
*/
String getName();
/**
* Returns the IP address of the machine, which can be varying at runtime
*
* @return the machine's IP address
*/
String getAddress();
/**
* Sets the IO address of the machine, usually performed by a back end
* operation.
* @param address the new IP address
*/
void setAddress(String address);
/**
* Starts the virtual machine
*
* @throws VirtException if the request to start failed.
*/
void start() throws VirtException;
/**
* Suspend the virtual machine
* @throws VirtException if the request to suspend failed
*/
void suspend() throws VirtException;
/**
* Resumes the virtual machine.
*
* @throws VirtException if the request to resume failed
*/
void resume() throws VirtException;
/**
* Stops a virtual machine.
*
* @throws VirtException if the request to stop failed
*/
void stop() throws VirtException;
/**
* Deletes the virtual machine and all associated storage
*
* @throws VirtException if the deletion failed
*/
void delete() throws VirtException;
/**
* Returns the current machine information
* @return the machine's current information
*/
VirtualMachineInfo getInfo();
}
Virtual Machine information To get runtime information about a particular virtual machine, the VirtualMachineInfo instance for that virtual machine instance must be retrieved.
/**
* Returns the virtual machine information
* @author Jerome Dochez
*/
public interface VirtualMachineInfo extends StaticVirtualMachineInfo {
/**
* Returns the maximum memory allocated to this virtual machine.
*
* @return the virtual machine maximum memory.
*/
long maxMemory() throws VirtException;
/**
* Returns the current memory used by this virtual machine
* @return the virtual machine current memory usage
*/
long memory() throws VirtException;
/**
* Returns the CPU used by this virtual machine
* @return the CPU time used by this VM.
* @throws VirtException
*/
long cpuTime() throws VirtException;
/**
* Returns the machine's state
* @return the machine's state
*
* @throws VirtException if the machine's state cannot be obtained
*/
Machine.State getState() throws VirtException;
/**
* Registers a memory changes listener
* @param ml the memory listener instance
* @param delay notification interval for memory changes polling.
* @param unit the time unit to express delay
*/
void registerResourcesListener(VMResourcesListener ml, long delay, TimeUnit unit);
/**
* Un-registers a memory changes listener
* @param ml, the listener to un-register.
*/
void unregisterResourcesListener(VMResourcesListener ml);
}
Monitoring code will be able to register listeners to virtual machines to be notified at interval of the memory consumption of the virtual machine.
public interface VMResourcesListener {
public void notified(VirtualMachine vm, long memory, long cpuTime);
}
Templates Templates are virtual machines disks pre-loaded with an operating system and a glassfish installation (tbd if we eventually support the install-node command in templates). Templates are used to create a virtual machine by copying the template to a disk storage, customize the template and eventually run it. Templates content A template must contain the following pages : glassfish, jdk6. Find below an example of the template installation based on an ubuntu operating system :
sudo apt-get -y -q install wget
sudo apt-get -y -q install sun-java6-jdk
sudo apt-get -y -q install unzip
sudo apt-get -y -q install openssh-server
sudo apt-get clean
The template boot must perform a number of tasks that are dependent on the virtualization solution. As an example, in kvm a template must have a script responsible for providing the virtual machine IP address to the DAS at each startup. Template configuration Each template implements 1 to many service type. Each service type will carry some configuration that will be used to communicate with the instance one it is up (like its ports, installation directory). How this information is stored is not yet fully determined : it will be using domain.xml configuration or it might be using the template configuration file it is extensible. Java EE Service configuration Each template will provide the following configuration :
- glassfish installation directory
- user credential
ServerPool A ServerPool defines an interface to allocate virtual machine based on a registered template. Several ServerPools can be registered to a GlassFish configuration, each group representing a boundary, either physical like a set of machines, either functional like a particular virtualization implementation. The IAAS layer internally delegates all lifecycles calls to ServerPools implementation. There are three types of ServerPools possible : The IMS only deals with ServerPool instances, it is immaterial if those instances are abstract ServerPools, physical ServerPools or remote ServerPools. A ServerPool uses a plugin to interface with the virtualization implementation. Abstract ServerPool An abstract ServerPool is an interface to an existing virtualization implementation that does not need to manage machines, resource pools (like storage) and the relationship between them. It has the ability to create virtual machines based on templates, and provide basic lifecyle for such virtual machines but it offers very little control over where such virtual machines are allocation and the physical attributes of the bare metal machines running the virtual machines. An example of an abstract group is EC2 where GlassFish does not need to be concerned about the hardware available to run virtual machines.
/**
* Abstract a set of servers than can be used to provide {@link VirtualMachine}
*
* @author Jerome Dochez
*/
public interface ServerPool {
/**
* Returns the configuration for this server pool.
*
* @return the server pool's configuration.
*/
ServerPoolConfig getConfig();
void setConfig(ServerPoolConfig config);
/**
* Returns this pool's name.
* @return this pool's name
*/
String getName();
/**
* Returns all allocated virtual machine in this server pool
* @return the list of allocated virtual machines
*/
Collection<VirtualMachine> getVMs() throws VirtException;
/**
* Returns an allocated virtual machine in this server pool using its name.
* @param name virtual machine name
* @return virtual machine instance if found or null otherwise.
* @throws VirtException if the vm cannot be obtained
*/
VirtualMachine vmByName(String name) throws VirtException;
/**
* Allocates number of virtual machines on any machine belonging to this server pool, each virtual machine
* should be based on the provided template.
* @param template template for the virtual machines
* @param cluster the virtual cluster instance to allocated virtual machines for
* @return Listenable future for the VirtualMachine instance
* @throws VirtException when the virtual machine creation failed.
*/
ListenableFuture<AllocationPhase, VirtualMachine> allocate(TemplateInstance template, VirtualCluster cluster) throws VirtException;
/**
* Returns the list of templates installed.
* @return list of installed templates or empty list if none
*/
List<VMTemplate> getInstalledTemplates();
/**
* Install a template in this server pool
* @param template to install
*/
void install(VMTemplate template);
}
ServerPool
/**
* Abstract a set of servers than can be used to provide {@link VirtualMachine}
*
* @author Jerome Dochez
*/
public interface ServerPool {
/**
* Returns the configuration for this server pool.
*
* @return the server pool's configuration.
*/
ServerPoolConfig getConfig();
void setConfig(ServerPoolConfig config);
/**
* Returns this pool's name.
* @return this pool's name
*/
String getName();
/**
* Returns all allocated virtual machine in this server pool
* @return the list of allocated virtual machines
*/
Collection<VirtualMachine> getVMs() throws VirtException;
/**
* Returns an allocated virtual machine in this server pool using its name.
* @param name virtual machine name
* @return virtual machine instance if found or null otherwise.
* @throws VirtException if the vm cannot be obtained
*/
VirtualMachine vmByName(String name) throws VirtException;
/**
* Allocates number of virtual machines on any machine belonging to this server pool, each virtual machine
* should be based on the provided template.
* @param template template for the virtual machines
* @param cluster the virtual cluster instance to allocated virtual machines for
* @return Listenable future for the VirtualMachine instance
* @throws VirtException when the virtual machine creation failed.
*/
ListenableFuture<AllocationPhase, VirtualMachine> allocate(TemplateInstance template, VirtualCluster cluster) throws VirtException;
/**
* Returns the list of templates installed.
* @return list of installed templates or empty list if none
*/
List<VMTemplate> getInstalledTemplates();
/**
* Install a template in this server pool
* @param template to install
*/
void install(VMTemplate template);
}
Physical ServerPool A physical ServerPool is an abstraction on a set of machines that use the same virtualization technology (in reality they only need to use the same plugin implementation to communicate). Such machines can be grouped by geography, nothing prevents the registration of several groups using the same underlying technology. On top of the interface described above, ServerPool membership is handled by the physical ServerPool so machines can be attached to group using the machine's IP address or DNS name or Mac address (subnet requirements apply).
/**
* Represents a serverPool of machines. How machines are allocated to groups is defined
* by the user.
*
* @author Jerome Dochez
*/
@Contract
public interface PhysicalServerPool extends ServerPool {
/**
* Returns the currently attached machines to this serverPool
*
* @return iterable of machines attached to this serverPool
*/
Iterable<? extends Machine> machines();
/**
* Returns a @see Machine instance which name is equal to the provided name
* @param name name of the machine looked up
* @return the @see Machine instance if found or null otherwise
*/
Machine byName(String name);
/**
* Returns the size of this serverPool
*
* @return size of the serverPool.
*/
int size();
}
ServerPool Management ServerPools must be defined in the domain.xml configuration, a number of methods are used to alter this configuration.
create-server-pool --virtname=virtname --subnet=subnet --portname=portname name
delete-server-pool name
list-server-pools
Notes : virtName is the virtualization solution used by the ServerPool, subnet is the subnet of this ServerPool, and portname is the port used by the IAAS to communicate with ServerPool members. ServerPool default User Machines attached to a ServerPool need a user id to connect to through SSH. ServerPools can define a default user id for all machines within the ServerPool. This might move into the create-server-pool command.
create-server-pool-user --pool=poolid [--authmethod=authmethod] --userid=userid --groupid=groupid name
to do : delete-group-user Machine Machines can be added to a ServerPool using the create-machine command. Each machine can have it's own emulator (paravirtualized versus hardware accelerated), defines where templates and virtual machines disks should be stored. A machine is identified by its name within a group and can be accessed through its mac-address or its network DNS name. A machine will also have the following properties :
- maximum processor utilization rate (80%)
- maximum memory utilization rate (80%)
create-machine [--emulator=emulator] [--templateslocation=templateslocation] [--mac=mac] --serverPool=poolName [--networkname=networkname] [--diskslocation=diskslocation] [--maxcpu=maxcpu] [--maxmem=maxmemory] name
delete-machine --pool=group name
list-machines --pool=group
User User id used to ssh into the machine when overriding the ServerPool's default user. Again this might move to the create-machine command.
create-machine-user --groupid=groupid [--authmethod=authmethod] --userid=userid --machine=machine --pool=poolName name
Remote ServerPool access A remote group happens when the server pool master is located on a different machine than the IMS. This is particularly useful when the virtualization infrastructure is owned by a different IT group. The APIs are similar, the remoting part of the call for a remote group is left to implementation details (most likely to use the asadmin commands infrastructure). more work needed to express a remote group access : URL ? machine name ? address ? port ?
create-group-manager --url=url name
|