Agenda 1. Introduction
2. FSD questions
3. Continuation Introduction Since not all the people in the call where acquainted, we had a short introduction round. It was also decided to ramp up the technical discussions. We probably need more coordination then just one technical conference per week. The idea is to continue with some more detailed technical discussions in a smaller group (Larry, Jan, Binod and Erik) and have more intensive e-mail discussions.
FSD questions Erik started on an FSD, during which he came up with some questions. Some of these questions were addressed, but due to lack of time not all of them.
unreliable transport The transport is not really unreliable. However, due to performance reasons the implementation does NOT wait for the ack to be received, so it can happen that replications fail. Indeed for the case of CompositeMetaData? this could lead to 'out-of-sync' data. Some observations:
- The use of CompositeMetaData? is not recommended by SUN. The data decomposition is something that should show the need for the CompositeMetaData?, so this should be done fairly soon.
- If SimpleMetaData? is used, the data will be in sync at the first successful replication
- The chances that data is out of sync at the moment of 'restore' of the replica cache is something that should be taken into account for the reliability figures. Taking the total picture into account it might be perfectly acceptable to have some unsuccessful restores.
- If we are willing to take the performance impact, it might be an idea to wait for the ack anyway.
Transactions Transaction IDs is not something that is supported by SUN, at the moment. Larry is even hesitant to recommend using this. Also currently there can already be an issue with transactions between EJB stateful sessions and HTTP sessions that depend on each other. Once in every while, in the tests SUN sees some problems that might be related to incomplete 'transactions'. Again, it is a question of the calculations and the acceptable reliability requirements if this is OK or not. Also, it was stated that we should distinguish the two alternatives;
- HADB which does offer transactions and real replication (also in case of multiple faillures)
- session replication which has a better performance, is easier to setup, but is less reliable.
Border between temporary failure and cluster reshape Terminology; cluster reshape is not an event, it is an activity (or rather a set of activities). There are GSM events; join and re-join that can be the trigger of the reshape. There are three ways to trigger:
- Join or re-join events from GMS in combination with a home-grown notification on JXTA (join ready). The latter is needed, but not part of GMS. The Join and ReJoin? come in the beginning, the JoinReady? is sent when the instance is ready with its initialisation and can receive the replication messages.
- Planned shutdown. This is typically not received from GMS, but is based on JXTA pipes that are being closed.
Multiple failures After a reshape where a node is removed, it will happen that some data is only available in the replica cache. It will be activated/restored when a new event is received that needs this data, only then will it be replicated again, and be redundantly available. Between the time of the instance being removed and the first use of the data, the system is susceptible to multiple failures. This issue is related to the SSA timer problem, if there is no active copy of the data no timer functionality will be enabled. The solution to both problems could be to do a sort of 'reverse repair'. This would mean that after the reshape the data that is available only in the replica cache will be moved to the active cache (and hence triggers normal replication mechanism again and activating the timer functionality). Probably the easiest is to do this on the clockwise neighbor of the removed instance...This is still to be discussed. Another alternative to combat the multiple failure problem is to use the HADB, although this probably does not help with the timer functionality.
How are events handled during temporary failure Larry told us that in the case of HTTP, the request is moved to another instance in the case that connection can not be made. Retry is very tricky and is an option in the load balancer. Retry should only be used in case of idempotent requests (if 'at-least once' semantics apply). It is actually more of a question for the Load Balancer and Joel will continue this discussion off-line via e-mail.
Continuation The time ran out and we did not manage to complete the list of questions. It was decided to continue the discussion on monday in a limited conference call (Binod, Larry, Jan, Erik). Action Items No action items have been identified yet. |
