Three years ago I experienced a problem where delivery websites were not getting their indexes updated following content changes in the authoring environment. We went around the houses and openned a ticket with Sitecore support. After some back and forth they suggested we check that the web and database server clocks are synchronised as this can cause issues, “you have difference about 3 seconds between the time stamps, which is very critical for scaling functionality”.
We synchronised the clocks and everything started working correctly. Back then the importance of the server time on each of your instances was not documented anywhere. Although these days it is still only mentioned in the docs relating to xDB as you can lose data.
Jump forward to present day on a Sitecore upgrade and again I was experiencing intimitent indexing issues. After much investigation and trawling through logs I remember the project from my distant past (those who know me know my memory is on a par with a goldfish*).
I checked the server clocks and sure enough one of them was 15 seconds out compared with the other two. Once again after synchronising them the problems all disappeared.
Everyone I discuss this with is suprised that something so trivial causes such a big problem and without being documented anywhere. If you delve under the bonnet at the reasons this is an issue it makes more sense, but it doesn’t stop people spending days looking at the application for answers when it is something as simple as timing.
*I know that despite popular belief, goldfish don’t have a 3 second memory