Federation Sender#

The Federation Sender is responsible for sending Persistent Data Units (PDUs) +and Ephemeral Data Units (EDUs) to other homeservers using +the /send Federation API.

How do PDUs get sent?

The Federation Sender is made aware of new PDUs due to FederationSender.notify_new_events. +When the sender is notified about a newly-persisted PDU that originates from this homeserver +and is not an out-of-band event, we pass the PDU to the _PerDestinationQueue for each +remote homeserver that is in the room at that point in the DAG.

Per-Destination Queues

There is one PerDestinationQueue per ‘destination’ homeserver. +The PerDestinationQueue maintains the following information about the destination:

whether the destination is currently in catch-up mode (see below);
a queue of PDUs to be sent to the destination; and
a queue of EDUs to be sent to the destination (not considered in this section).

Upon a new PDU being enqueued, attempt_new_transaction is called to start a new +transaction if there is not already one in progress.

Transactions and the Transaction Transmission Loop

Each federation HTTP request to the /send endpoint is referred to as a ‘transaction’. +The body of the HTTP request contains a list of PDUs and EDUs to send to the destination.

The Transaction Transmission Loop (_transaction_transmission_loop) is responsible +for emptying the queued PDUs (and EDUs) from a PerDestinationQueue by sending +them to the destination.

There can only be one transaction in flight for a given destination at any time. +(Other than preventing us from overloading the destination, this also makes it easier to +reason about because we process events sequentially for each destination. +This is useful for Catch-Up Mode, described later.)

The loop continues so long as there is anything to send. At each iteration of the loop, we:

dequeue up to 50 PDUs (and up to 100 EDUs).
make the /send request to the destination homeserver with the dequeued PDUs and EDUs.
if successful, make note of the fact that we succeeded in transmitting PDUs up to +the given stream_ordering of the latest PDU by
if unsuccessful, back off from the remote homeserver for some time. +If we have been unsuccessful for too long (when the backoff interval grows to exceed 1 hour), +the in-memory queues are emptied and we enter Catch-Up Mode, described below.

Catch-Up Mode

When the PerDestinationQueue has the catch-up flag set, the Catch-Up Transmission Loop +(_catch_up_transmission_loop) is used in lieu of the regular _transaction_transmission_loop. +(Only once the catch-up mode has been exited can the regular tranaction transmission behaviour +be resumed.)

Catch-Up Mode, entered upon Synapse startup or once a homeserver has fallen behind due to +connection problems, is responsible for sending PDUs that have been missed by the destination +homeserver. (PDUs can be missed because the PerDestinationQueue is volatile — i.e. resets +on startup — and it does not hold PDUs forever if /send requests to the destination fail.)

The catch-up mechanism makes use of the last_successful_stream_ordering column in the +destinations table (which gives the stream_ordering of the most recent successfully +sent PDU) and the stream_ordering column in the destination_rooms table (which gives, +for each room, the stream_ordering of the most recent PDU that needs to be sent to this +destination).

Each iteration of the loop pulls out 50 destination_rooms entries with the oldest +stream_orderings that are greater than the last_successful_stream_ordering. +In other words, from the set of latest PDUs in each room to be sent to the destination, +the 50 oldest such PDUs are pulled out.

These PDUs could, in principle, now be directly sent to the destination. However, as an +optimisation intended to prevent overloading destination homeservers, we instead attempt +to send the latest forward extremities so long as the destination homeserver is still +eligible to receive those. +This reduces load on the destination in aggregate because all Synapse homeservers +will behave according to this principle and therefore avoid sending lots of different PDUs +at different points in the DAG to a recovering homeserver. +This optimisation is not currently valid in rooms which are partial-state on this homeserver, +since we are unable to determine whether the destination homeserver is eligible to receive +the latest forward extremities unless this homeserver sent those PDUs — in this case, we +just send the latest PDUs originating from this server and skip this optimisation.

Whilst PDUs are sent through this mechanism, the position of last_successful_stream_ordering +is advanced as normal. +Once there are no longer any rooms containing outstanding PDUs to be sent to the destination +that are not already in the PerDestinationQueue because they arrived since Catch-Up Mode +was enabled, Catch-Up Mode is exited and we return to _transaction_transmission_loop.

A note on failures and back-offs

If a remote server is unreachable over federation, we back off from that server, +with an exponentially-increasing retry interval. +Whilst we don’t automatically retry after the interval, we prevent making new attempts +until such time as the back-off has cleared. +Once the back-off is cleared and a new PDU or EDU arrives for transmission, the transmission +loop resumes and empties the queue by making federation requests.

If the backoff grows too large (> 1 hour), the in-memory queue is emptied (to prevent +unbounded growth) and Catch-Up Mode is entered.

It is worth noting that the back-off for a remote server is cleared once an inbound +request from that remote server is received (see notify_remote_server_up). +At this point, the transaction transmission loop is also started up, to proactively +send missed PDUs and EDUs to the destination (i.e. you don’t need to wait for a new PDU +or EDU, destined for that destination, to be created in order to send out missed PDUs and +EDUs).

+ +