Monday, December 31, 2012

All Management Servers Resource Pool Unavailable Hack/Fix

I have seen a lot of posts regarding the "All Management Servers Resource Pool Unavailable" error since SCOM 2012 RC, and I am still seeing numerous posts. I have also not seen anything in SP1 regarding fixing this error. The "All Management Servers Resource Pool Unavailable" error seems to cause trouble where no trouble really exists.

This is what I have found under normal circumstances:

  1. Everything is fine
  2. Then, you get the error
  3. You might get a couple of other errors, saying stuff doesn't work
  4. The agents on the affected management server all serve up a false heartbeat failure
  5. The agents fail to a different management server
  6. The heartbeat failures resolve
  7. The affected management server says, "oh, I'm fine now, turns out nothing was really wrong with me"
  8. The agents change back to their primary management server
  9. Everything is now fine again...except you now have inaccurate availability, a bunch of auto-resolved false alerts, unnecessary state stages, unnecessary e-mails, and probably an upset server team who received those e-mails.
  10. Then, it all happens over again...
A Microsoft article existed at one point, which provided a registry change to workaround/fix the issue, but then the article disappeared. If you make this registry change on all of your management servers, it might just fix the issue. It fixed it for me, as well as other people. As far as I know, this change is unsupported, but it is also easily reversible.

Here is the fix/workaround:

Some Notes First:
  1. As always, before making any changes, back up your registry and SCOM databases.
  2. Make this change on ONE Management Server at a time, and give the management server some time to recover after the restart - about 15 minutes should do.
  3. As far as I know, Microsoft does not support this, so use at your own risk.
The Actual Steps:
  1. Open you Registry editor (Start > Run > regedit)
  2. HKLM\SYSTEM\CurrentControlSet\services\HealthService\Parameters
  3. Select the PoolManager Folder/Key. (If it does not exist, create it under Parameters)
  4. Create 2 new D-Words
    1. PoolLeaseRequestPeriodSeconds
    2. Give it a Value of 600 (Decimal)
    3. PoolNetworkLatencySeconds
    4. Give it a Value of 120 (Decimal)
  5. Restart the Management Server
  6. Give it Time to Recover
  7. Repeat on all Management Servers
  8. Let things calm down (probably an hour or more)
Good luck!

6 comments:

  1. Life is all about timing... the unreachable becomes reachable, the unavailable become available, the unattainable... attainable. Have the patience, wait it out It's all about timing. See the link below for more info.


    #unavailable
    www.ufgop.org

    ReplyDelete
  2. Legend! I have spent the last two days trying to find the fix for this. This fix also applies to SCOM 2016.

    If you're ever in Sydney I'll buy you a beer.

    ReplyDelete
  3. hi,

    I tried your solution and after reboot, i closed the alert manually from SCOM dashboard. However, I'm still receiving these alerts (emails) after every 2 minutes. Can you help me ?
    Resolution state: Closed Alert: All Management Servers Pool Unavailable.

    ReplyDelete
  4. It works for me, Thank you very much !!!

    ReplyDelete
  5. Aw, this was a really nice post. In idea I wish to put in writing like this additionally – taking time and precise effort to make an excellent article… but what can I say… I procrastinate alot and by no means seem to get something done. usa online casino

    ReplyDelete
  6. I have made these registry entries already but still see this behavior in SCOM 2012 R2 UR 14.

    Did someone tried increasing the time ?

    ReplyDelete