When I went to deploy the code, which had already been successfully deployed in 1 WFE Dev environment, I noticed my solutions were getting stuck in “Deploying” status.
I verified that I did see dlls deployed by the solutions in the GAC for all my 4 servers, but the solution in Central Admin did not seem to know that the solutions had completed.
My coworker, Blake, pointed out that the server time on my WFEs was in the wrong time zone and that he’d had issues with SharePoint before when servers were out of sync on time. I checked and all my WFE’s were on the same incorrect time, but my SQL servers were on the right time. A ha!
I went and changed the server time zone to the correct time and restarted the SharePoint Timer Service on all 4 servers. Then I retracted all my code, manually deleted .dlls from the GAC of all servers (had to use gacutil, or I get an Access Denied error), and then re-added & redeployed the solutions. My timer job time was still wrong, but the solutions completed deploying to all 4 servers this time!
Anyone know how to reset the SharePoint time?
Followup: We had a MS tech guy in for a risk assessment and we mentioned this issue he said to go ahead and rebuild the farm. GAHHH!
So, lesson learned: Make sure your servers are ALL on the same time zone before you install SharePoint!! It is now on the infrastructure teams check off list.