Abstract
Clustering for High Availability & Scalability is one of the main requirements of any enterprise deployment. This is also true for WSO2's open source SOA middleware products. However, configuring a cluster can be tricky. This article explains in detail the clustering configuration language used by WSO2 Carbon based products. If you haven't already read "Introduction to WSO2 Carbon Clustering", I'd recommend you to read that article first.Table of Contents
Introduction
As we have seen in[1], Apache Axis2 [2] is the base of WSO2 Carbon [3]. Hence, the clustering functionality of Apache Axis2 are inherited by WSO2 Carbon. In[1], we already looked at the concepts governing WSO2 Carbon clustering. In this article we will see how clustering has to be configured in the axis2.xml file.The axis2.xml clustering configuration section has 6 main parts:
- Clustering Agent
- Parameters
- State Management
- Node Management
- Group Managment
- Members
1. Clustering Agent
The ClusteringAgent is responsible for initializing all clustering related functionality of a Carbon member or node. Typically, the initialization of a node in a cluster is handled here. It is also responsible for getting a node to join the cluster. In the default Carbon/Axis2 clustering implementation, which is based on Apache Tribes [4], we use org.apache.axis2.clustering.tribes.TribesClusteringAgent.2. ClusteringAgent Parameters
In order to initialize the Clustering Agent, we require some parameters. Here are the parameters that are used in the current implementation:AvoidInitiation
This parameter indicates whether the cluster has to be automatically initalized when the AxisConfiguration is built. If set to "true", the initialization will not be done at that stage, and some other party will have to explicitly initialize the cluster. In WSO2 Carbon, we have set this to "true", since Carbon will decide when the cluster has to be initialized. In this case, the cluster is initialized after the server has completely started and the transports have been started.membershipScheme
The membership scheme used in this setup. The only values supported at the moment are "multicast" and "wka".1. multicast - membership is automatically discovered using multicasting. In order for this to work, multicasting should be allowed in the network and all members in the cluster should use the same multicast address (as defined by the mcastAddress parameter) and the exact multicast port (as defined by the mcastPort parameter).
2. wka - Well-Known Address based multicasting. Membership is discovered with the help of one or more nodes running at a Well-Known Address. New members joining a cluster will first connect to a well-known node, register with the well-known node and get the membership list from it. When new members join, one of the well-known nodes will notify the others in the group. When a member leaves the cluster or is deemed to have left the cluster, it will be detected by the Group Membership Service (GMS) using a TCP ping mechanism. WKA-based membership is necessary when multicast based membership discovery is not possible. For example, on Amazon EC2, multicasting is not allowed and there is no control over the IP address assigned to EC2 instances. Hence, in such a scenario, WKA-based membership discovery tends to be used.
domain
The clustering domain/group. Note that the words "domain" and "group" are used synonymously. There will not be any interference between nodes in different groups. Messages received from members outside the group will generally be ignored. However, special messages, such as cluster management messages or membership messages from members outside the group will be allowed.synchronizeAll
When a Web service request is received, and processed, should we update the states of all members in the cluster prior to the response being sent to the client? If the value of this parameter is set to "true", the response to the client will be sent only after all members have been updated. Obviously, this can be time consuming. In some cases, where this overhead may not be acceptable, in which case the value of this parameter should be set to "false". The risk in this case will be that all members will not be in the same state when the response is sent to the client. This condition will have to be handled by the user. In summary, this parameter defines whether we should synchronize the state of all members in the group before we send the response to the client or not.maxRetries
This defines the maximum number of times we need to retry to send a message to a particular member, before giving up and considering that node to be faulty. After maxTries number of retries, we give up.mcastAddress
The multicast address to be used. This parameter will only be taken into consideration when the membershipScheme is set to "multicast".mcastPort
The multicast port to be used. This parameter will only be taken into consideration when the membershipScheme is set to "multicast".mcastFrequency
The frequency of sending membership multicast messages. The value should be specified in milliseconds. This parameter will only be taken into consideration when the membershipScheme is set to "multicast".memberDropTime
The time interval within which, if a member does not respond, the member is deemed to have left the group. The value should be specified in milliseconds.mcastBindAddress
The IP address of the network interface to which the multicasting has to be bound to. Multicasting would be done using this interface. This parameter will only be taken into consideration when the membershipScheme is set to "multicast". Note that this can be different from the localMemberHost parameter. So we can listen for point-to-point messages on the localMemberHost network interface while listening for multicast messages on the network interface bound to the mcastBindAddress.localMemberHost
The host name or IP address of this member. This is the IP address advertised by this member when it joins the group and sends messages. This should be set to a valid value other than localhost or 127.0.0.1. In most cases, it would suffice to set it to the IP address bound to the network interface which is used for communicating with members in the group.localMemberPort
The TCP port used by this member. This is the port through which other members will contact this member.preserveMessageOrder
Preserve message ordering. This will be done according to sender order.atmostOnceMessageSemantics
Maintain atmost-once message processing semantics.properties
Properties specific to this member. These properties are are simply name-value pairs. When a member joins groups, these properties are bound to this member so that other members in the group can detect these properties. In the case of WSO2 Carbon, two member properties "backendServerURL" and "mgtConsoleURL" have been specified. These member properties are used for cluster management. The backendServerURL property is used to connect to the backend management Web services of this member using the front-end of the cluster manager. Similarly, if this member wishes to expose its own management console, it can do so by providing the "mgtConsoleURL" property. The backendServerURL property is specified using the <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/> entry. Here, ${hostName} signifies the "hostName" member property & ${httpsPort} denotes the "httpsPort" member property. "hostName" & "httpsPort" are two implicit member properties. "httpPort" is another implicit member property. Properties that have been previously declared can be used in subsequent property definitions. For example, we may define a new property as follows: <property name="foo" value="${backendServerULR}/foo"/>3. State Management
The stateManager element needs to be enabled if you require to synchronize state across members in a cluster group. An implementation of the org.apache.axis2.clustering.state.StateManager needs to be provided as the value of the class attribute. In the default Apache Tribes based implementation, we provide the org.apache.axis2.clustering.state.DefaultStateManager class. The user can choose not to replicate certain properties. This is done by providing a property name pattern. For example, specifying <exclude name="local_*"/> will exclude all properties having the names prefixed with local_ from replication, <exclude name="*_local"/> will exclude all properties having the name suffixed with _local from replication. It follows that <exclude name="*"/> excludes all properties from replication. Excluding all properties from replication may be useful when one needs to replicate only the properties in certain contexts. For example, if a user wishes to replicate properties only in the serviceContext, under the exclusion entries of org.apache.axis2.context.ConfigurationContext & org.apache.axis2.context.ServiceGroupContext, the <exclude name="*"/> entry should be added.The exclusion patters under the defaults element will be applicable to all contexts. Hence, <exclude name="local_*"/> under the "defaults" element means that all the properties prefixed with local_ in all the properties having names prefixed with local_ will be excluded from replication. Shown below is a sample "stateManager" entry.
<stateManager class="org.apache.axis2.clustering.state.DefaultStateManager"
enable="true">
<replication>
<defaults>
<exclude name="local_*"/>
<exclude name="LOCAL_*"/>
</defaults>
<context class="org.apache.axis2.context.ConfigurationContext">
<exclude name="UseAsyncOperations"/>
<exclude name="SequencePropertyBeanMap"/>
</context>
<context class="org.apache.axis2.context.ServiceGroupContext">
<exclude name="my.sandesha.*"/>
</context>
<context class="org.apache.axis2.context.ServiceContext">
<exclude name="my.sandesha.*"/>
</context>
</replication>
</stateManager>
4. Node Management
The nodeManager element needs to be enabled in order to have node management functionality. An implementation of the org.apache.axis2.clustering.NodeManager interface needs to be provided as the value of the class attribute.<nodeManager class="org.apache.axis2.clustering.management.DefaultNodeManager"
enable="true"/>
5. Group Management
When this member is deployed as a cluster manager, the groupManagement element needs to be enabled. A group management agent, which is an instance of the org.apache.axis2.clustering.management.GroupManagementAgent interface, needs to be specified for each group (applicationDomain) that is being managed. The example configuration below shows how cluster management has been enabled for two groups; group1 and group2.<groupManagement enable="true">
<applicationDomain name="group1"
description="This is the first group"
agent="org.apache.axis2.clustering.management.DefaultGroupManagementAgent"/>
<applicationDomain name="group2"
description="This is the second group"
agent="org.apache.axis2.clustering.management.DefaultGroupManagementAgent"/>
</groupManagement>
6. (Static) Members
The members element is used for specifying static or well-known members. The hostName and the primary port of these members need to be specified. the sample member configuration below shows two static members with different IP addresses and ports:<members>
<member>
<hostName>10.100.1.202</hostName>
<port>4000</port>
</member>
<member>
<hostName>10.100.1.154</hostName>
<port>4001</port>
</member>
</members>
Full Configuration
Shown below is a typical clustering configuration in WSO2 Carbon.<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="false">
<parameter name="AvoidInitiation">true</parameter>
<parameter name="membershipScheme">multicast</parameter>
<parameter name="domain">wso2.carbon.domain</parameter>
<parameter name="synchronizeAll">true</parameter>
<parameter name="maxRetries">10</parameter>
<parameter name="mcastAddress">228.0.0.4</parameter>
<parameter name="mcastPort">45564</parameter>
<parameter name="mcastFrequency">500</parameter>
<parameter name="memberDropTime">3000</parameter>
<parameter name="mcastBindAddress">127.0.0.1</parameter>
<parameter name="localMemberHost">127.0.0.1</parameter>
<parameter name="localMemberPort">4000</parameter>
<parameter name="preserveMessageOrder">true</parameter>
<parameter name="atmostOnceMessageSemantics">true</parameter>
<parameter name="properties">
<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
</parameter>
<members>
<member>
<hostName>127.0.0.1</hostName>
<port>4000</port>
</member>
<member>
<hostName>127.0.0.1</hostName>
<port>4001</port>
</member>
</members>
<groupManagement enable="false">
<applicationDomain name="apache.axis2.application.domain"
description="Axis2 group"
agent="org.apache.axis2.clustering.management.DefaultGroupManagementAgent"/>
</groupManagement>
<nodeManager class="org.apache.axis2.clustering.management.DefaultNodeManager"
enable="true"/>
<stateManager class="org.apache.axis2.clustering.state.DefaultStateManager"
enable="true">
<replication>
<defaults>
<exclude name="local_*"/>
<exclude name="LOCAL_*"/>
</defaults>
<context class="org.apache.axis2.context.ConfigurationContext">
<exclude name="UseAsyncOperations"/>
<exclude name="SequencePropertyBeanMap"/>
</context>
<context class="org.apache.axis2.context.ServiceGroupContext">
<exclude name="my.sandesha.*"/>
</context>
<context class="org.apache.axis2.context.ServiceContext">
<exclude name="my.sandesha.*"/>
</context>
</replication>
</stateManager>
</clustering>
Conclusion
In this article, we had a detailed look at the clustering configuration language used in WSO2 Carbon and Apache Axis2. Knowledge of these configuration parameters are essential when it comes to configuring a cluster in an enterprise deployment setup.References
- Introduction to WSO2 Carbon Clustering, http://wso2.org/library/articles/introduction-wso2-carbon-clustering
- Apache Axis2, http://ws.apache.org/axis2/
- WSO2 Carbon, https://wso2.org/projects/carbon/
- Apache Tribes, http://tomcat.apache.org/tomcat-6.0-doc/tribes/introduction.html
No comments:
Post a Comment