SharePoint 2010 - Search Error - The search service is not able to connect to the machine that hosts the administration component

Problem:


We recently ran across an issue that prevented the SharePoint Server 2010 from starting with the below exception:

"The search service is not able to connect to the machine that hosts the administration component"

When I clicked on any link in the left navigation in search administration interface inside Central Administration, I was greeted with the below exception:

The application pool in IIS that the search service uses is found in stopped state. It stops automatically as soon as it is started.

Explanation:

After hours of research, we have found that it was all happening due to an AD policy set in domain controller that was not allowing certain service accounts from being able to start services and application pools

Here are the steps we followed to fix this issue:

• First we did a complete rebuild including the OS
• Then the search service worked but a service account was not able to start search service and the Web Services Default App Pool.. (sp2010sa account in our case)
• We then updated the group policies in AD to allow the service accounts to have "logon as a batch" job rights
• We had a problem in pushing the group policy to client(s). For that change to take place in the servers in the domain, we had to take the servers out of the domain and re add them..
• So we removed computer(s) from domain, computer account was not created in the correct Organizational Unit(OU). New policy was forced as an update on the computer. So we removed the computer from the domain and rejoined to the domain..
• In this process SharePoint Config DB got corrupted and so CA was gone
• So we uninstalled SP 2010 again, removed all the SP databases and re-installed SP 2010

Summary:

In short, before installing SharePoint, make sure that you follow the least privileged approach and create all required service accounts. This article here  describes it very well.

Make sure that the managed accounts used such as SharePoint Search, SharePoint App Pool and the SharePoint Farm Account have "logon as a batch" job rights.

Server(s) can be removed from domain but computer accounts should not be removed. If computer account(s) are removed, the identifier(s) will be deleted and therefore the identifiers change the next time when the servers are added again to the domain controller. SharePoint looks for the identifiers in config database and since it could not find the correct identifiers, CA would not work.

If you have to apply new policy such as "logon as a service", make sure to place the computer(s) in correct Organizational Unit(OU) and remove the computer(s) from domain. The policy is then reset and it assigns the correct "logon as a service" credentials.

This wont be the case in most installations.
In our case, we ran into issues due to incorrect / corrupt AD structure / domain controller.

I am not a network specialist but if there is a way to ensure that all the AD group policies are being applied properly prior to installing SharePoint, you would save yourself from the above mentioned hassles.