Lab: Part 6 – Configure NetScaler 11 High Availability (HA Pair)

In this post, we will configure our NetScaler virtual appliances for High Availability. 

More from the Lab!

 

Introduction

In the previous post, we reviewed the architecture of Citrix Netscaler and installed two standalone virtual appliances (VPX). The next step is to configure High Availability with these two VPX. You will learn the best practices to set up HA smoothly, to increase resiliency for your services.

What is High Availability?

Today’s businesses need access to their systems and services all the time. Globalization has made the workplace a 24/7 environment. People work from home, multiple locations, and everyday offices–they cannot afford any downtime caused by maintenance, human error, or hardware failure.

The best practice while building your infrastructure is to make sure that your system is available all the time. One solution to achieve this is through configuring High Availability (HA). This will eliminate a single point of failure, as the service will remain available for your users even if one of the appliances is down. It will also allow you to schedule maintenance for your devices without impacting production.

In the Netscaler world, an HA pair is composed of at least two appliances/nodes (up to 64) permanently exchanging UDP heartbeat messages.

At the same time, there is always only one node in charge of all traffic; this is the primary node. It actively accepts connections, manages servers, and manages all shared IP addresses.

The node not in charge is called the secondary node, and monitors the health of the primary node to make sure that the service is up and running. If there are issues, a failover will occur and the primary role will be transferred to the secondary node.

NetScaler HA Architecture

Below is the generic NetScaler HA Architecture:

All IP addresses are said to be floating IP addresses and are shared across the members of the HA pair except the NSIPs which are unique to each appliance.
Only the primary NetScaler(NetScaler 1 in the schema) is in use. The secondary NetScaler is in stand-by mode and is waiting for the primary node to fail. In others words, all the traffic goes through the primary node.

The two nodes are exchanging packets together (HA sync) for health monitoring.

ARP/GARP

Network devices including NetScalers use ARP (Address Resolution Protocol) to find MAC addresses of others devices located on the LAN.

For example, you have a virtual IP address (VIP) owned by a NetScaler. When a request to this VIP arrives to the gateway, it will check its own ARP cache (where are located previously resolved IP addresses) for the corresponding MAC address. If the cache is empty, the gateway will send an ARP broadcast packet on the LAN. The device with the corresponding IP address will reply to the gateway with an unicast packet containing its MAC address. Then the gateway will put this value into its ARP cache.

You can see the NetScaler ARP table with the command below:

or this one:

You can clear the NetScaler ARP table:

or remove a specific IP:

In HA environments and especially after a failover, this could cause issues. The previous MAC address in the gateway ARP cache table is the MAC address of the previous NetScaler appliance.

To fix this issue, the new primary appliance will send GARP (Gratuitous ARP) broadcast packets for all NetScaler HA owned IP addresses (VIPs, NSIPs, etc). The gateway will received these packets and update its ARP cache table.

There is few moment when GARP packets are sent:

• NetScaler startup (for MIP and SNIP)
• Creation of a new LB server
• Add new ip
• Set new vLAN
• Failover

You can only disable GARP on VIP:

In others words, ARP packets are sent when another device is requesting information and GARP packets are sent on-demand when a change is occurring in the configuration.

Netscaler HA Settings

Node States

• STAYPRIMARY: This option forces the NetScaler appliance to stay in the primary mode.
• STAYSECONDARY: This option forces the NetScaler appliance to stay in the secondary mode.
• ENABLED: This is the default option. This option enables the NetScaler appliance of the high availability pair to fail over based on the high availability events.
• DISABLED: This option disables the high availability engine.

In the lab, both HA nodes are configured by default (HA Enabled).

Fail-Safe

Fail-safe mode ensures that one node is always primary when both nodes failed the health check. This is to ensure that when a node is only partially available, backup methods are enabled to handle traffic as best as possible. The HA fail-safe mode needs to be configure on each node.

Recommendation: Enable fail-safe on both nodes.

HA behavior

Below is the behavior of the HA resulting of the nodes states.

Fail-Safe disabled

NS01 (Primary)	NS02 (Secondary)	Nodes states	HA state
Down (failed last)	Down (failed first)	NS01 - Secondary NS02 - Secondary	HA is DOWN
Down (failed first)	Down (failed last)	NS01 - Secondary NS02 - Secondary	HA is DOWN
UP	UP	NS01 - Primary NS02 - Secondary	HA is UP
UP	Down	NS01 - Primary NS02 - Secondary	HA is UP
Down	UP	NS01 - Secondary NS02 - Primary	HA is UP
Down	UP (Stay Secondary)	NS01 - Secondary NS02 - Secondary	HA is DOWN

Fail-Safe enabled

NS01 (Primary)	NS02 (Secondary)	Nodes states	HA state
Down (failed last)	Down (failed first)	NS01 - Primary NS02 - Secondary	HA is UP
Down (failed first)	Down (failed last)	NS01 - Secondary NS02 - Primary	HA is UP
UP	UP	NS01 - Primary NS02 - Secondary	HA is UP
UP	Down	NS01 - Primary NS02 - Secondary	HA is UP
Down	UP	NS01 - Secondary NS02 - Primary	HA is UP
Down	UP (Stay Secondary)	NS01 - Primary NS02 - Secondary	HA is UP

Default Configuration

• The primary node is in charge of all traffic.
• Both nodes have their own NSIPs but share everything else including VIPs, SNIPs and MIPs.
• Management over the SNIP addresses is recommended.
• Changes made on the primary node are replicated to the secondary node. (Enabled by default)
• Heartbeat(Hello interval) is 200ms.  
  • UDP Packets on port 3003.
  • Dead interval is 3secs.
  • Failover occurs when 3 secs of heartbeat packets are missed.
• Fail-safe is disabled by default and is configured independently on each node.
• By default, communications are not secure:
  • HA pair synchronization occurs on port 3010 (TCP).
  • Secure HA pair synchronization occurs on port 3008 (TCP).
  • Commands propagation occurs on port 3011 (TCP).
  • Secure Commands propagation occurs on port 3009 (TCP).

Lab NetScaler HA Architecture

Configure NetScaler High Availability

Requirements

NetScaler model, version and licenses must be the same on all the appliances.

RPC password must be the same.

Management must be done on the primary node only.

Make sure that the two Netscalers are configured with a unique NSIP.

Also make sure that your appliances are not part of any HA configuration:

This command should have no result.

Make sure that all interfaces are up and running.

There is more requirements available here.

Configuration

On NS02, type the following command:

This command will force NS02 to stay the secondary node in the HA pair.

On NS01, type the following command:

Then configure the sync between the nodes.

On NS02, type the following command:

Then you can show the HA configuration with the command below:

NS01 is configured as Primary and NS02 is configured as Secondary.

Note that the sync state of the Secondary NetScaler is SUCCESS.

Type the following command on NS02 to enable the HA node:

Verify the configuration:

Verify the configuration with the GUI:

Connect to http://192.168.199 and check the configuration.

This node is primary in the HA pair.

Connect to http://192.168.200 and check the configuration.

NetScaler is warning you that this appliance is the secondary node and nothing will be saved or propagated.

This node is secondary in the HA pair.

Test failover

On NS02, type the following command:

Note that NS01 is the Primary and NS02 is the secondary node.

Now type the command below to force the failover:

Then, show the HA configuration again:

Note that NS01 is now the Secondary node and NS02 is the now the Primary node.

Before the next step, save your NS config on both appliances:

Shutdown NS02:

On NS01, type the following command:

The previous Primary node(NS02) is now unreachable, so the role has been automatically transferred to NS01.

Additional configuration

Configure default route

The configuration below will allow the Netscalers to resolve Internet DNS.

Configure SUBNET IP for management

Connect to the primary node, and add a new SNIP.

Go to NetScaler -> System -> Network > IPs -> IPV4 and select the new IP and click Edit.

And then select Enable Management Access.

Open your browser and type http://192.168.1.201. As SNIP addresses are shared between HA nodes, this address will always redirect to the primary NetScaler.

Force HA pair sync

You can force the synchronization of the HA pair with the following command:

Secure HA communication

Connect to the Primary node and type the following command:

The setting is currently configured on OFF and communication is not secure between the two nodes.

You can change the RPC password by typing the following commands on both nodes:

I recommend to change the password before enabling the Secure RPC.

To make sure that the communication is OK between the two nodes, type the following command:

Sync State should be ENABLED on the primary node and SUCCESS on the secondary node.

Reconfigure NTP server

You can reconfigure the ntp server for the HA pair with the command below:

To make sure that NTPd is running:

Force NTP update

In the next post, we will review how to update the firmware of the nodes configured in HA.

More from the Lab!