Fix a bug introduced in Synapse 1.76 where 5s delays would occasionally occur in deployments using workers.