I've had a lot of requests to explain the Citrix XML Black hole and how Citrix Netscaler can resolve these issues by load balancing the XML Broker service. This document will help explain the issues and explain how to resolve them using Citrix Netscaler.
When XENAPP servers become highly utilized, the XML service stops responding and ultimately, application enumeration stops working until the servers utilization returns to normal or the server is rebooted.
The XML Black hole was a phrase coined a few years back to describe the following scenario:
- An XML Broker, IMA, or Termsrv error occurs on the XML Broker (XENAPP Server)
- Web Interface is Able to Query the XML broker
- The XML service is not able to query the IMA
The result is the Web Interface receives valid XML data without any published applications for the user. Web Interface treats this scenario as a success and does not remove the server from load balancing. Users that logon to Web Interface may or may not receive published applications in their list.
Using Health Monitoring and recovery is known to help resolve these issues, especially in smaller farms. Larger farms should consider using Citrix Netscaler to load balance and monitor the XML Broker service.
- The first step is to create a new published application in the farm. This application will be used by the netscaler to monitor and verify that the application is properly resolved.. I usually publish Notepad.exe and name the published application XML-Service.
- You must ensure that at least one user has rights to see the application. Usually a service account is sufficient (Thereby keeping the application from actually being used by end users).
- You should also ensure that the published applications has all servers in the farm listed.
- On the Netscaler, add a server entity for each XML Broker.
- Next define a Service for each server or Service Group using http and the port number of the XML Broker services. Bind the servers to the services.
- Create a new http Vserver using a new IP address and the port of the XML Broker. Bind your services.
- Verify that the services are up, then create a new monitor. For type, select CITRIX-XML-BROKER.
- Leave the Interval, RTO and Down Time as it's defaults, then click on Special Parameters and add the name of the published application in the Application Name section, i.e. XML-Service.
- Next, bind the monitor to the services or service group and verify that the services are still alive.
- Using the AMC on the Web Interface server, go to manage server farms and remove the servers listed, then add the IP address of the VServer.
NOTE: I've noticed that in larger farms, the enumeration process can slow down, this is due to Web Interface trying to send a NetBios Packet to the Vserver. To resolve this, simply go into the NIC card properties and disable NetBios over TCP/IP.That's it... The Netscaler will then poll the XML Broker services every 5 seconds. If the XML service does not respond with the Published Application, the Service will be taken out of load balance.
NOTE: While I've seen this solution have great success for many companies, it is important to test this out before implementing it. Create a new Web Interface site etc. before going into full production.