The Never-Ending Citrix Local Host Cache Story

Citrix Local Host Cache (LHC) evolution overview. Falkor knows it! 

CitrixGuru’s op-ed pieces have received a lot of excellent engagement and comments from our readers in recent months. I’d like to keep the dialogue going with a review of the evolution of the Citrix Local Host Cache (LHC) over the years. What was once a no-brainer has since gone through an ebb and flow of changes, discontinuations, and resurgences. The latest chapter in the LHC saga was in December 2016 when Citrix claimed to bring back the functionality as users once knew it. Instead, there are still gaps with LHC working the way it should.

Good old LHC

First, let’s do a quick recap of the LHC timeline.

Local Host Cache was a core feature of the Independent Management Architecture (IMA) that was introduced with Citrix Metaframe XP 1.0 in 2001, and is still used today with Citrix XenApp 6.5. LHC helped avoid downtimes caused by a failure to contact the remote SQL data store, but did not allow administrators to benefit from all of the features during an outage. LHC also aimed to improve the performance of app enumerations by locally caching a subset of the farm configuration, Windows domain trust relationships, and apps’ settings on each XenApp server.

Technically the LHC was a simple Microsoft Access database (Imalhc.mdb) stored by default in the <ProgramFiles>\Citrix\Independent Management Architecture folder. The content was reloaded at the start of the machine and kept up to date by the IMA service, which pulled the remote data store every 30 minutes. If the data store was unreachable, the LHC contained enough information about the farm to allow normal operations. LHC evolved a lot over the years and allowed SQL downtimes for an indefinite period of time in its last release with XenApp 6.5. However, no new static information could be published or added to the farm, until the farm data store was reachable and operational again.

At this time, LHC architecture was an extremely simple and easy-to-manage solution that allowed administrators to perform maintenance on the SQL infrastructure without impacting the current XenApp production. Citrix provided many command line tools to interact with the LHC, making the life of Citrix administrators easier.

The disappearance of LHC

With the release of the awful version 7.0 of XenApp in 2013 and the move to XenDesktop FlexCast Management Architecture (FMA), Citrix decided to remove the Local Host Cache feature–and many others–without offering any other alternative. To be fair, Citrix converged XenApp into XenDesktop, which was already using the FMA design since the version 5 and without Local Cache Host equivalent.  This decision immediately made the SQL infrastructure a critical piece of any XenApp implementation. Any downtime on the SQL infrastructure would immediately cause a downtime for new sessions on the XenApp infrastructure as well. It could also have some side effects with the old Citrix Web Interface. I remember having extreme slowness caused by the XenDesktop/XenApp 7 controllers not being able to process XML requests and impacting others Citrix infrastructures.

Citrix recommends having a highly-available SQL infrastructure to host XenApp and XenDesktop databases. There are multiple options available to achieve that by following high availability best practices from Microsoft, such as SQL Server Clustered Instances, SQL Server Mirroring and SQL Server 2012 AlwaysOn Availability Groups. SQL Express is supported by Citrix but does not offer HA capabilities and therefore is not recommended for large deployments.

While you can successfully implement HA for your SQL infrastructure, it does not necessarily mean that you will avoid downtimes, as many components are to be considered.  Network issues and interruptions may prevent Delivery Controllers from accessing the databases, resulting in users not being able to connect to their applications or desktops. I have a good example of this: a few years ago, I had an outage on XenDesktop where all SQL servers part of our cluster were unreachable from the Citrix servers, the SQL service itself was fine but the communication was not possible resulting on users not able to broker new sessions.

The pseudo renaissance of LHC with Connection Leasing

I’m sure you understand by now that having dependencies on a separate infrastructure is not good. Facing a storm of complaints, Citrix also started–finally!–to listen to its customers and released XenDesktop 7.6 in Sept 2014 with the Connection Leasing (CL) feature enabled by default.

Unfortunately, CL was no LHC and was a poor equivalent limited to frequently used and assigned applications/desktops (up to 2 weeks by default). For users not using Citrix frequently or using pooled desktops, CL was completely useless and did not resolve anything. There are also many limitations: load management, workspace control, power actions were not supported, for example.  Also, CL did not completely remove downtimes as there were 2 brief intervals during which users could not connect or reconnect: (1) from the time the database becomes unavailable to when the Controller enters leased connection mode, and (2) from the time the Controller changes from leased connection mode to when database access is fully restored and the VDAs have re-registered.

Citrix is already planning to get rid of Connection Leasing by not supporting this feature after the next Long Term Service Release (LTSR). I would not recommend to implement an already-dead feature.

The return of LHC

Citrix came up with a milestone achievement with its new idea as part of the XenDesktop 7.12 release in Dec 2016. This time, they claimed to bring back all the Local Host Cache (LHC) features from XenApp 6.5, even adding few improvements to make it more reliable. LHC feature is offered for Cloud and On Premises implementations along Connection Leasing in 7.12, but is considered the primary mechanism to allow connection brokering operations when database connectivity to the site database is disrupted. Surprisingly, Local Host Cache feature is disabled by default. We can expect Citrix to enable that feature by default in the next version.

Local Host Cache is the most comprehensive high availability feature in XenApp and XenDesktop. It is a more powerful alternative to the connection leasing feature that was introduced in XenApp 7.6. – Citrix

When installing XenDesktop 7.12 and up, a SQL Express instance(LocalDB) will be installed locally (C:\Windows\ServiceProfiles\NetworkService\HaDatabaseName.mdf) on each Delivery Controller to store the Local Host Cache. For those familiar with XenDesktop 7, the Broker Service is still available and referenced as the Principal Broker and still manages VDA registrations and brokering during normal operations. This service also checks on the remote database to make sure that it is online.  

The Config Synchronizer Service (CSS) and the Secondary Brokering Service are now part of the installation. CSS takes care of the synchronization between the remote database and the Local Host Cache(LocalDB). The Secondary Brokering Service takes over from the Principal Broker when an outage is detected and does all registration and brokering operations.

There are many serious limitations with this version of LHC. Let’s talk first about the LocalDB, which is a runtime version of SQL Server with a specific licensing that limits the usage of four cores or a single socket. We cannot really blame Citrix for that one as it is a limitation imposed by Microsoft. Pooled desktops are not supported, which is a huge downside and no change can be made to the farm (assignments, publications, power actions, etc), you cannot even open the consoles (Director & Studio) and PowerShell. There is also no control over the LHC election process and only a single Delivery Controller will take care of all VDA registrations and broker sessions for the whole zone during an outage. The election does not take into account broker resources, so administrators need to make sure that any controller in the zone can take care of all the registrations during an outage. Citrix recommends to limit LHC to farms with less than 5000 VDA, but does not enforce that value.

Most importantly, it is only a one-way communication between the LHC and the remote SQL database. In other words, nothing is saved during an outage as the LocalDB is nothing less than a snapshot of the SQL database and cannot be modified.

You would think that this new version of the Local Host Cache would assure you zero downtime, but you would be wrong. There is also a delay before users can actually connect. When the remote database goes down, all Delivery Controllers stop responding to enumeration requests from StoreFront. It takes time for the Principal Broker Service to detect the loss of connectivity with the data store(up to 1 minute) and switch over to the Secondary Broker Service of the elected Deliver Controller. Then VDAs still have to re-register to the newly and ONLY elected Delivered Controller. It can result in users not having icons in StoreFront or in users not able to start new sessions for a short period of time.

Wrap Up

In conclusion, it took Citrix almost 4 years to deliver a somewhat equivalent of the good old Local Host Cache for XenDesktop 7.x. The database is not a single point of failure anymore in a XenDesktop/XenApp deployment. However, customers with large deployments are not supported with this version of the Local Host Cache and some of the -HUGE- limitations can discourage you from using that feature. Far from perfect but going in the right direction, there is still a lot of work for Citrix to match IMA capabilities and to be able to claim that FMA fully supports database disconnections. 

Useful links

• Connection Leasing
• SQL considerations
• Local Host Cache (up to XenApp 6.5)
• Local Host Cache (XenDesktop 7.12)
• Local Host Cache considerations (XenDesktop 7.12)