NG FP3
For additional technical information about Check Point products, consult Check Points
SecureKnowledge database at
http://support.checkpoint.com/kb/
Preface
Introduction
This document explores various technical aspects of the Cluster Control Protocol as utilized by ClusterXL. Although
parts of the Cluster Control Protocol are also used by OPSEC High Availability products, this aspect will not be covered.
This document is not meant as an installation guide, and assumes the reader has a working knowledge of the ClusterXL
product.
Overview
The introduction of ClusterXL into an existing network, often implies that certain changes to that network be made. Such
alterations are needed to accommodate both the clustered topology, as well as the Cluster Control Protocol itself.
Understanding these changes is important for the purposes of planning, implementation, monitoring, and troubleshooting.
Moreover, adequate comprehension of the ClusterXL decision making process is needed. Otherwise, there will be no
context in which to place observed cluster behavior.
Therefore, the enclosed sections will explore the following topics:
Implementation Planning
Cluster Control Protocol Overview
Cluster Control Protocol Logic
Implementation Planning
NOTE: Due to the enhanced Hot/Standby configuration available in FP3, Legacy HA will not be covered in the
following sections.
Switch Support
The Cluster Control Protocol used by both New Mode, and Load Sharing configurations, makes use of layer two
multicast. In keeping with multicast standards, this multicast address is used only as the destination, and is used in all
CCP packets sent on "non-secured" interfaces.
A layer two switch connected to non-secured interfaces, must be capable of forwarding multicast packets to ports within
that VLAN. It is acceptable that the switch forward such traffic to all ports within the given VLAN. However, it is
considered more efficient to forward to only those ports connecting cluster members.
The steps needed to enable multicast support will vary according to the switch vendor, and model. Please check your
switch documentation for details.
If the connecting switch is incapable of forwarding multicast, CCP can be changed to use broadcast instead. To toggle
between these two modes use the command
(mode survives a reboot):
'cphaconf set_ccp broadcast/multicast'
VLAN Configuration
It is not recommended to connect the non-secured interfaces of multiple clusters to the same VLAN. Doing so will cause
the connecting switch ports to flap. If such a need exists, a separate VLAN, and/or switch will be needed for each cluster.
A HotFix is also available from Check Point support which allows this configuration to be supported in FP3.
Connecting together the secured interfaces of multiple clusters is also not recommended for the same reason. While the
above mentioned HotFix may be used, there are additional concerns with this configuration which make it currently
unsupportable. Therefore, it is best to connect the secured interfaces of a given cluster via a crossover link when possible,
or to an isolated VLAN.
IP Address Migration
It is reasonable to assume that many ClusterXL installs will be either for a new VPN-1/FireWall-1 cluster, or to replace a
different clustering solution. However, it is also reasonable to assume that many will be to provide high availability to an
existing single gateway configuration.
In the latter case, existing NAT, and IPSec connections will need to be altered to accommodate the new clustered
landscape. Therefore, it is recommended to take the existing IP addresses from the current gateway, and make these the
cluster owned VIP's, or cluster addresses when feasible. Doing so will avoid altering current IPSec endpoint identities, as
well keep Hide NAT configurations the same in many cases.
SmartCenter/CMA Location
A SmartCenter/CMA Server, can install a Security Policy to one or more clusters with only a single install action. It does
so by installing the Security Policy to each cluster member, using the general tab IP of each cluster member object as the
recipient. This is true regardless of the IP address(es) of the cluster object itself.
This design affords a great level of flexibility when using either New Mode, or Load Sharing configurations, as the
SmartCenter/CMA Server can now reside on any given IP segment. One only needs to ensure that the general tab IP
address of the cluster member object is reachable. If not, simply choose one which will be accessible to the
SmartCenter/CMA Server.
Load Sharing
Important factors to consider while planning for a Load Sharing implementation are:
Switch Support
The Cluster Control Protocol used by both New Mode, and Load Sharing configurations, makes use of layer two
multicast. In keeping with multicast standards, this multicast address is used only as the destination, and is used in all
CCP packets sent on "non-secured" interfaces.
A layer two switch connected to non-secured interfaces, must be capable of forwarding multicast packets to ports within
that VLAN. It is acceptable that the switch forward such traffic to all ports within the given VLAN. However, it is
considered more efficient to forward to only those ports connecting cluster members.
The steps needed to enable multicast support will vary according to the switch vendor, and model. Please check your
switch documentation for details.
If the connecting switch is incapable of forwarding multicast, CCP can be changed to use broadcast instead. To toggle
between these two modes use the command
(mode survives a reboot):
'cphaconf set_ccp broadcast/multicast'
Router Support
In addition to the use of multicast by CCP, Load Sharing associates a multicast MAC for each configured cluster IP. This
design ensures that traffic destined to the cluster is received by all members.
Therefore, ARP replies sent by a cluster member will indicate that the unicast cluster IP, is reachable via a multicast
MAC. Some routing devices are incapable of receiving such ARP replies. For instance, all versions of Cisco IOS do not
include such support. In such cases, adding a static ARP entry for the cluster IP on the routing device will solve the issue.
Even still there are some routers such as Extreme routers, Avia routers, and some Nortel models (Passport 1200 or XLR)
which will not accept this type of static ARP entry. For those cases, ClusterXL FP4 will introduce a new mode of
operation referred to as Pivot mode. Pivot mode operates as a Load Sharing cluster, but without the need of multicast for
the cluster addresses.
VLAN Configuration
It is not recommended to connect the non-secured interfaces of multiple clusters to the same VLAN. Doing so will cause
the connecting switch ports to flap. If such a need exists, a separate VLAN, and/or switch will be needed for each cluster.
A HotFix is also available from Check Point support which allows this configuration to be supported.
Connecting together the secured interfaces of multiple clusters is also not recommended for the same reason. While the
above mentioned HotFix may be used, there are additional concerns with this configuration which make it currently
unsupportable. Therefore, it is best to connect the secured interfaces of a given cluster via a crossover link when possible,
or to an isolated VLAN.
IP Address Migration
It is reasonable to assume that many ClusterXL installs will be either for a new VPN-1/FireWall-1 cluster, or to replace a
different clustering solution. However, it is also reasonable to assume that many will be to provide high availability to an
existing single gateway configuration.
In the ladder case, existing NAT, and IPSec connections will need to be altered to accommodate the new clustered
landscape. Therefore, it is recommended to take the existing IP addresses from the current gateway, and make these the
cluster owned VIP's, or cluster addresses when feasible. Doing so will avoid altering current IPSec endpoint identities, as
well keep Hide NAT configurations the same in many cases.
SmartCenter/CMA Location
A SmartCenter/CMA Server, can install a Security Policy to one or more clusters with only a single install action. It does
so by installing the Security Policy to each cluster member, using the general tab IP of each cluster member object as the
recipient. This is true regardless of the IP address(es) of the cluster object itself.
This design affords a great level of flexibility when using either New Mode, or Load Sharing configurations, as the
SmartCenter/CMA Server can reside on any given IP segment. One only needs to ensure that the general tab IP address
of the cluster member object is reachable. If not, simply choose one which will be accessible to the SmartCenter/CMA
Server.
Message Types
Below is a complete listing of the possible CCP message types with description:
CCP Transmission
Non-Secured Interfaces
For interfaces not defined as secured (non synchronization interfaces), CCP transmits it's packets by default with layer two
multicast. The addressable fields are as follows:
As an example, lets assume a scenario in which New Mode is being used. On a given segment, the cluster IP is
10.3.220.103/27. CCP packets sent by the highest priority machine will look like this:
0:0:0:0:fe:0 1:0:5e:3:dc:67 ip 78: 0.0.0.0 > 10.3.220.96
Here, 1:0:5e:3:dc:67 corresponds to the destination MAC, and indicates that the OID is multicast (1:0:5e:), with the rest
corresponding to the last three octets of the cluster address.
The last octet of the source address 0:0:0:0:fe:0, indicates the Machine ID of the transmitting member, in this case this
primary. In case of a second, or third member, this source address will reflect the members priority such as:
0:0:0:0:fe:1
0:0:0:0:fe:2
Using this design, both the destination MAC, and destination IP address will change per IP segment, according to both
the cluster, and network address.
Secured Interfaces
For interfaces defined as secured (synchronization interfaces), CCP transmits by default as follows:
Port usage
CCP uses UDP as the transmission protocol, with both the source and destination port set to 8116. This is true
irrespective of the interface type.
Figure 1.1
Topology Legend
Primary Reboot
1. As the primary goes down, it changes it's state to "Down/dead", and announces this as part of FWHA_MY_STATE.
2. This triggers the secondary to prepare itself to become the active member. It does so by sending as series of gratuitous
ARP's for both it's physical IP, and cluster IP for each clustered segment. This will update all necessary hosts/routers on
each segment with the relevant updated MAC address information.
3. The secondary now moves to the "Active/Active-Attention" state, and assumes responsibility for processing all
connections.
4. Though the primary is now considered "Down/dead", it will still be able to send/receive CCP packets until its'
interfaces are brought down. Once this occurs, the secondary, which is now in the "Active/Active-Attention" state, will
make notice of the fact that no CCP packets are being received.
5. The secondary will do several things in an effort to ascertain why no CCP packets are being received. First, it sends
FWHA_IF_PROB_REQ packets on all segments in which no CCP packets have been heard. This is to illicit a response
from any member capable of responding. The secondary will ARP on each segment for IP's belonging to that segment,
and ping those hosts which respond.
The pings will continue as long as we cannot identify by other means (i.e. CCP packets) that the interface is alive. This
will happen when there are N cluster members, and N-1 of them are down. When more than two members are present,
such pings will only be issued if all other cluster members do not respond to CCP probing.
6.Once the primary has rebooted, but before the policy is loaded, is will begin sending FWHA_IFCONF_REPLY packets
regularly. It does so without knowing what Cluster it belongs to, so a random ID is used.
7. After the Primary learns the cluster ID, it begins announcing FWHA_MY_STATE. The primary at this stage
announces itself as "Down/Dead".
8. The primary fetches the policy from another cluster member if possible, otherwise from the management server.
9. The primary now initiates full synchronization on the secured interface via the FW1 protocol
10. Once synchronization is complete, the primary moves to the "Ready" state.
11. The secondary acknowledges this by moving to the "Standby" state.
12.Once this state has been acknowledged, the primary issues gratuitous ARP's for both the physical, and cluster IP for
each segment, and now moves to the "Active/Active-Attention" state.
Dual Failure
This scenario assumes a dual failure by both cluster members of the secured (synchronization) interface connected via a
crossover link.
1. Since it is assumed that the secured interfaces are connected via a crossover link, the failure of one interface will bring
down the line protocol of the other resulting in a dual failure. Once this occurs, both members will become aware of this
fact via CCP. Both members will announce as part of FWHA_MY_STATE that N-1 interfaces are up.
2. Since both members have suffered the loss of a single interface, a decision must be made as to what action to take
next. Bringing both members down will result in a total failure, but some level of disturbance as already occurred. The
solution is for the highest priority member to remain in the "Active/Active Attention" state, and for the secondary to
report itself as "Down/Dead".
3. The necessary state changes are made, and announced as part of FWHA_MY_STATE.
4. Upon recovery of the secured link, the secondary will resume the "Standby" status.
Load Sharing
The events carried out by CCP during various failures in Load Sharing mode, closely resembles those covered thus far in
the previous New Mode section. For this reason, a complete analysis will not be given. However, there are some
important differences which should be noted.
1. As opposed to New Mode, all members of a Load Sharing cluster will remain in the "Active/Active Attention" state
during normal operation.
2. Upon the failure of a member, that members state will be changed to "Down/Dead", while all other members will
remain in "Active/Active Attention", and continure to process connections.
3. In New Mode, for the purposes of packet forwarding, each cluster address is associated with the corresponding
physical MAC address of the active member. For this reason, it is necessary to issue gratuitous ARP's during a failure.
This is not to be confused with the multicast MAC used by CCP for message transmission.
However, in Load Sharing mode, the multicast MAC used per segment by CCP, is also used as the MAC address for the
purposes of packet forwarding. This is necessary to ensure that each cluster member receives every packet. Therefore,
there will be no issuance of gratuitous ARP's for any cluster address during a failure.
4. Although CCP will advertise the configured priority of the sending cluster member, these priority labels do not dictate
a level of seniority in Load Sharing during normal operation as they do in New Mode configurations.