GlassFish - World's first Java EE 7 Application Server


Minutes of meeting LB team 2007-06-26

Minutes of meeting LB team 2007-06-26

Larry is worried about the overall latency requirements (see Larry's email). Specifically:

  • waiting for ack is expensive, so we should explore whether we really want to avoid replication failures and to what cost.
  • moving active objects is expensive, so we should try to see if we can have an architecture where we can keep ongoing sessions sticky once they have moved, but still route new sessions to new/restored node.

The GMS distributed hash can be used to provide an override mechanism for ongoing (sticky) sessions.

Migrating active sessions

After some discussion on how data/user centric routing works the tentative conclusion is that we probably can not implement the proposal from Larry/Jan since the LB can not see if the request is for a new session of for an ongoing session. It is the application that decides this in the @SipApplicationKey method and it depends on the fact whether the SAS for the key generated by this method is already in progress or not.

So (in my personal interpretation) this would mean that in the LB we would have to somehow map hashkeys to SASids and have some sort of feedback (via the hashmap) of all the still ongoing sip application sessions on the 'failover node'. Only once an ongoing session has stopped (e.g., based on timeout of the SAS) would the new requests with this hashkey be routed to the restored node. This does not seem really feasible to me, but the LB team will investigate whether there is some equivalent solution.

  • Reply by Joel on 072707 10:08 CET: Yes, we will look into it and see what can be done. I envision something like that if a certain hash key value has been tied to a session which is found to have been migrated, then that specific hash key value will be sticky to a certain AS instance instead of be used as input to consistent hashing. There is one corner case here though. What if at a later point (remember that sessions can last for days in the presence application), the second AS instance fails?
    • Reply by Erik van der Velden on 072707 15:59 CET: I do not get this. At the failover you would have to know which hashkeys refer to ongoing sessions on the failed node. As you mentioned in another mail, there will be relation between the session replication framework, the semantics of the replicated items (sip application session id) and the relation with the hashkey. This would be quite difficult to achieve. And how would you eventually go back to the normal situation, which would mean that you have to remove those hashkeys from your 'consistent hash exclusion table' when the corresponding sip application session is invalidated. If you never remove any items from the exclusion table, then it would grow at each failover, leading to an unbalanced load and potential problems with multiple failures.
      • Reply by Joel on 072707 16:53 CET: As I said, We shall see what can be done. This was just a rough idea, it might not be a feasible solution at all.

If we assume that the migration of active sessions is needed we have two choices

  • a forced migration (called backflush by Larry). This might be tricky since you have to know which part of the data to 'flush back'.
  • a lazy migration based on incoming request routed by the LB to the restored node.

The latter would probably already work. Any session that did fail over during the time the node was down will have become active on another instance and (this is important) would have created a replica with the same version as the result of a load request on the failover node. When the LB routes requests in the session back to the restored node the load request will active the session in the restored node based on the load request. The last replica would be removed and a new replica would be made. This is all part of the current replicaiton framework.

The only drawback is that the previous active version of the data is not removed during the second load request. However, the embedded version information will ensure that that active version becomes stale.

  • Reply by Joel on 072707 10:08 CET: One caveat though: timers will still be active for it. Thus, it is possible that a timer fires in this stale version and the application is invoke and does something causing the session to be mutated and suddenly this instance is the latest version!
    • Reply by Erik van der Velden on 072707 15:59 CET: Correct. I forgot about this. This is the primary reason why you want to avoid this. The other drawbacks mentioned below are just secondary.

This should work in most situations, but has two drawbacks:

  • some additional data is kept on the system that is superfluous.
  • some request for the data can not detect the staleness of the data due to the fact that there is no version information in the request (e.g., if requested by the utils.)

It was suggested by Ericsson that we could remove any (duplicate) active session during the load request, just as all replicas are removed to avoid these two problems.

With the suggested solution active session migration might be possible, but still the impact on latency etc has to be investigated and any alterantive solution that avoids the migration should be explored.

  • Reply by Joel on 072707 10:08 CET: As I commented to your previous mail. It struck me that we might not be able to migrate active sessions, since that must be treated as a session invalidation on the node where it going to be deleted. According to JSR-289 (section 6.1.2.2.2) it is not allowed to invalidate a SipApplicationSession until all protocol sessions have been invalidated and for SIP that means that all ongoing transactions must have finished. This can be really tricky!
    • Reply by Erik van der Velden on 072707 15:59 CET: I do not see what migration has to do with invalidation. You need to call sessionWillPassivate() on the old one and sessionDidActivate() on the new one. You move including any protocol session children. There is the question of concurrent access that plays an important role during migration. What if at the moment of migration, because a new request is routed to the restored instance, something is still accessing the session on the failover instance. We should wait until the doXXX() returns before migrating the session in that case, I guess (this locking issue is another reason to avoid migration).
      • Reply by Joel on 072707 16:53 CET: Yeah, you are right about that in regard to informing the application. But I think the JSR-289 is a little but confusing about this (6.4.3 and 6.2.4.1). I don't know if they have thought about this in JSR-289. I mean, they have specified some support for distributed systems in section 6.4.3, but I beleive they have not thought about ongoing transactions. If you migrate a SipApplicationSession containing SipSessions which have ongoing transactions the related transactionm must be removed as well and the transaction will be lost (maybe that is acceptable and then this is not a problem).
        • Reply by Joel on 072707 17:39 CET: After thinking of it a bit more, I have come to the conclusion that it should be acceptable to lose your transactions at recovery. I mean, we do not give any guarantees that transactions are kept when a node fails, consequently it should be OK to not guarantee that transactions are kept at recovery after fail-over.

Access via the SipSessionsUtil

The SipSessionUtil allows access to the SipApplicationSession using the SASid that is obtained in any way (out-of-band).

This can be used to access the SAS from, e.g., an EJB.

It is impossible to route the requests in such a way that these request only occur on the instances where the SAS is already available.

There are several possibilities:

  • The data is access remotely
  • The active session is migrated to the node where the request is done (and migrated back when normal traffic occurs on the session).
  • We simply do not comply to the spec, e.g., we do not guarantee that the correct session is returned, or that a session is returned at all.

If we decide to implement this we also have to think on how to accomplish locking semantics, which makes it even more tricky.