That happens to me too.
It appears to be random, I haven't found any pattern in it and reloading the game doesn't seem to work (I think)