The problem comes from a TL maintenance patches that were applied after installation of the SCSP agent. This caused a mismatch of supported IPS driver on the system and resulted in the behavior seen of a) Empty process list in the driver, b) memory errors in errpt, and the GAI bloat, and the hang condition as seen in the dump that was received.
The oslevel command shows it's AIX 6.1 TL7, however the version of the bos.mp64 package (the kernel unix file for AIX 64-bit) has 3 versions, with the committed /active/ version being older than then newly applied package version (shown with 'A' state below). This type of partial install state may indicate an error or intentional staging of packages when installing/updating software on the system.
From lslpp output:
18.104.22.168 C F Base Operating System 64-bit
22.214.171.124 A F Base Operating System 64-bit
126.96.36.199 A F Base Operating System 64-bit
At the time SCSP was installed the TL was at a lower level requiring a different version of the sisips driver to be installed. This brought to light an issue with the preventative steps we take to ensure an up-to-date driver is staged in the system /usr/lib/drivers directory. During install and boot time, we attempt to copy the appropriate driver file to the /usr/lib/drivers directory — however if the /opt filesystem is not accessible at the time (as is often the case during boot), the driver file is never updated. This opens the possibility for the driver to become out of sync with the current kernel Tech Level of the system after updating the Technology Level of the system.
Fix the driver file mismatch by doing either:
A) Re-install the 5.2.8 MP4 agent
B) Run the command "/etc/methods/sisipsctrl –dm" to ensure correct version of the driver is installed in /usr/lib/drivers. Verify the detected OS Tech Level is "7" (should match-up with TL from "oslevel –r").
Reboot to have the correct driver in effect.
# sum /usr/lib/drivers/sisips
55891 1939 /usr/lib/drivers/sisips
AIX and SCSP.