Event ID 3961 - Data Collector is out of memory message

User report they are unable to launch applications from web Interface. They get a resource is unavailable message.

Note : In our environment the ZDC and Backup ZDC are set at the XML brokers for web interface.

The Data collector showed this eventlog entry
Event ID 3961
Source : IMAService
The Data collector is out of memory, and the dynamic data store data might be out of sync. Please elect a new Data collector and make sure you have enough memory on the new data collector

Web Interface servers showed this eventlog entry
Event ID: 0
All of the configured XML Services for farm "PS4" failed to respond to this XML transaction. [Log ID: eecdd2dc]


A Restart of IMA service on the XML brokers/ZDC's resolved the issue.

Looking at the memory metrics for the IMA Service it had grown to use 2GB of memory. The server has 4GB of memory installed but 2Gb is around the limit of addressable space for a 32 bit process. This large increase in IMA memory use is being investigated by ourselves and citrix with the ise of their monitoring tools.

to troubleshoot the issue we have run GLFLAGS.EXE /i ImaSrv.exe
restart IMA service
Install USERDUMP tool from windows resource kit. Set this to monitor IMASRV.EXE
Run USERDUMP.EXE PID dump of IMA
Run UMDH.EXE -P:PID -f:c:\alloc.txt

Repeat taking the dumps as memory useage increases

Full instructions are in citrix articles:
ctx106970
ctx120596