summary refs log tree commit diff
path: root/synapse/handlers/federation.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Remove the deprecated BaseHandler. (#11005)Patrick Cloke2021-10-081-4/+2
| | | | | | | | The shared ratelimit function was replaced with a dedicated RequestRatelimiter class (accessible from the HomeServer object). Other properties were copied to each sub-class that inherited from BaseHandler.
* Strip "join_authorised_via_users_server" from join events which do not need ↵Patrick Cloke2021-09-301-2/+7
| | | | | | | it. (#10933) This fixes a "Event not signed by authorising server" error when transition room member from join -> join, e.g. when updating a display name or avatar URL for restricted rooms.
* Split `event_auth.check` into two parts (#10940)Richard van der Hoff2021-09-291-12/+18
| | | | | | | | | | | | | Broadly, the existing `event_auth.check` function has two parts: * a validation section: checks that the event isn't too big, that it has the rught signatures, etc. This bit is independent of the rest of the state in the room, and so need only be done once for each event. * an auth section: ensures that the event is allowed, given the rest of the state in the room. This gets done multiple times, against various sets of room state, because it forms part of the state res algorithm. Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think that makes everything hard to follow. Instead, we split the function in two and call each part separately where it is needed.
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-1/+1
|
* Use `RoomVersion` objects (#10934)Richard van der Hoff2021-09-291-20/+26
| | | Various refactors to use `RoomVersion` objects instead of room version identifiers.
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-241-1/+1
|
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-231-1/+1
|
* Factor out a separate `EventContext.for_outlier` (#10883)Richard van der Hoff2021-09-221-5/+4
| | | | | | Constructing an EventContext for an outlier is actually really simple, and there's no sense in going via an `async` method in the `StateHandler`. This also means that we can resolve a bunch of FIXMEs.
* Ensure we mark sent knocks as outliers (#10873)Richard van der Hoff2021-09-221-0/+7
|
* Require type hints in the handlers module. (#10831)Patrick Cloke2021-09-201-130/+0
| | | | | | | Adds missing type hints to methods in the synapse.handlers module and requires all methods to have type hints there. This also removes the unused construct_auth_difference method from the FederationHandler.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-2/+2
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Populate `rooms.creator` field for easy lookup (#10697)Eric Eastwood2021-09-011-0/+1
| | | | | | Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values
* Split `FederationHandler` in half (#10692)Richard van der Hoff2021-08-261-1765/+22
| | | The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
* Make `backfill` and `get_missing_events` use the same codepath (#10645)Richard van der Hoff2021-08-261-233/+40
| | | Given that backfill and get_missing_events are basically the same thing, it's somewhat crazy that we have entirely separate code paths for them. This makes backfill use the existing get_missing_events code, and then clears up all the unused code.
* Split `on_receive_pdu` in half (#10640)Richard van der Hoff2021-08-191-98/+138
| | | Here we split on_receive_pdu into two functions (on_receive_pdu and process_pulled_event), rather than having both cases in the same method. There's a tiny bit of overlap, but not that much.
* Extract `_resolve_state_at_missing_prevs` (#10624)Richard van der Hoff2021-08-191-105/+124
| | | This is a follow-up to #10615: it takes the code that constructs the state at a backwards extremity, and extracts it to a separate method.
* Refactor `on_receive_pdu` code (#10615)Richard van der Hoff2021-08-181-134/+137
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * drop room pdu linearizer sooner No point holding onto it while we recheck the db * move out `missing_prevs` calculation we're going to need `missing_prevs` whatever we do, so we may as well calculate it eagerly and just update it if it gets outdated. * Add another `if missing_prevs` condition this should be a no-op, since all the code inside the block already checks `if missing_prevs` * reorder if conditions This shouldn't change the logic at all. * Push down `min_depth` read No point reading it from the database unless we're going to use it. * Collect the sent_to_us_directly code together Move the remaining `sent_to_us_directly` code inside the `if sent_to_us_directly` block. * Properly separate the `not sent_to_us_directly` branch Since the only way this second block is now reachable is if we *didn't* go into the `sent_to_us_directly` branch, we can replace it with a simple `else`. * changelog
* Stop setting the outlier flag for things that aren't (#10614)Richard van der Hoff2021-08-171-7/+2
| | | | | Marking things as outliers to inhibit pushes is a sledgehammer to crack a nut. Move the test further down the stack so that we just inhibit the thing we want.
* Clean up some logging in the federation event handler (#10591)Richard van der Hoff2021-08-161-28/+24
| | | | | | | | | | | | | | | | | | | * Include outlier status in `str(event)` In places where we log event objects, knowing whether or not you're dealing with an outlier is super useful. * Remove duplicated logging in get_missing_events When we process events received from get_missing_events, we log them twice (once in `_get_missing_events_for_pdu`, and once in `on_receive_pdu`). Reduce the duplication by removing the logging in `on_receive_pdu`, and ensuring the call sites do sensible logging. * log in `on_receive_pdu` when we already have the event * Log which prev_events we are missing * changelog
* Clean up federation event auth code (#10539)Richard van der Hoff2021-08-061-52/+66
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * drop old-room hack pretty sure we don't need this any more. * Remove incorrect comment about modifying `context` It doesn't look like the supplied context is ever modified. * Stop `_auth_and_persist_event` modifying its parameters This is only called in three places. Two of them don't pass `auth_events`, and the third doesn't use the dict after passing it in, so this should be non-functional. * Stop `_check_event_auth` modifying its parameters `_check_event_auth` is only called in three places. `on_send_membership_event` doesn't pass an `auth_events`, and `prep` and `_auth_and_persist_event` do not use the map after passing it in. * Stop `_update_auth_events_and_context_for_auth` modifying its parameters Return the updated auth event dict, rather than modifying the parameter. This is only called from `_check_event_auth`. * Improve documentation on `_auth_and_persist_event` Rename `auth_events` parameter to better reflect what it contains. * Improve documentation on `_NewEventInfo` * Improve documentation on `_check_event_auth` rename `auth_events` parameter to better describe what it contains * changelog
* Add support for MSC2716 marker events (#10498)Eric Eastwood2021-08-041-6/+113
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
* Make historical events discoverable from backfill for servers without any ↵Eric Eastwood2021-07-281-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | scrollback history (MSC2716) (#10245) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Only connect base insertion event to prev_event_ids Per discussion with @erikjohnston, https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org * Make it possible to get the room_version with txn * Allow but ignore historical events in unsupported room version See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489 We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either. Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`. * Move to unique index syntax See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509 * High-level document how the insertion->chunk lookup works * Remove create_event fallback for room_versions See https://github.com/matrix-org/synapse/pull/10245/files#r677641879 * Use updated method name
* Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke2021-07-261-10/+44
| | | | (#10254)
* Port the ThirdPartyEventRules module interface to the new generic interface ↵Brendan Abolivier2021-07-201-2/+2
| | | | | (#10386) Port the third-party event rules interface to the generic module interface introduced in v1.37.0
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-1/+1
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-161-11/+11
|
* Fix a number of logged errors caused by remote servers being down. (#10400)Erik Johnston2021-07-151-9/+16
|
* Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-011-13/+23
| | | Instead of mixing them with user authentication methods.
* Return errors from `send_join` etc if the event is rejected (#10243)Richard van der Hoff2021-06-241-7/+39
| | | Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
* Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-241-126/+51
| | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
* Send out invite rejections and knocks over federation (#10223)Richard van der Hoff2021-06-231-0/+14
| | | | | ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.
* Check third party rules before persisting knocks over federation (#10212)Andrew Morgan2021-06-211-2/+2
| | | | | An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules. This PR corrects that by performing the TPAR check *before* persisting the event.
* update black to 21.6b0 (#10197)Marcus2021-06-171-1/+1
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* Add fields to better debug where events are being soft_failed (#10168)Eric Eastwood2021-06-171-3/+18
| | | Follow-up to https://github.com/matrix-org/synapse/pull/10156#discussion_r650292223
* Remove the experimental flag for knocking and use stable prefixes / ↵Patrick Cloke2021-06-151-4/+2
| | | | | | | endpoints. (#10167) * Room version 7 for knocking. * Stable prefixes and endpoints (both client and federation) for knocking. * Removes the experimental configuration flag.
* Add metrics to track how often events are `soft_failed` (#10156)Eric Eastwood2021-06-111-0/+7
| | | | | | | | | | | Spawned from missing messages we were seeing on `matrix.org` from a federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770. The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066 where the message and join event race and the message is `soft_failed` before the `join` event reaches the remote federated server. Less soft_failed events = better and usually this should only trigger for events where people are doing bad things and trying to fuzz and fake everything.
* Implement knock feature (#6739)Sorunome2021-06-091-3/+183
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Handle /backfill returning no events (#10133)Erik Johnston2021-06-081-13/+25
| | | Fixes #10123
* Don't try and backfill the same room in parallel. (#10116)Erik Johnston2021-06-041-0/+8
| | | | | If backfilling is slow then the client may time out and retry, causing Synapse to start a new `/backfill` before the existing backfill has finished, duplicating work.
* Limit number of events in a replication request (#10118)Erik Johnston2021-06-041-2/+3
| | | Fixes #9956.
* add a cache to have_seen_event (#9953)Richard van der Hoff2021-06-011-5/+7
| | | Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
* Limit the number of events sent over replication when persisting events. ↵Brendan Abolivier2021-05-271-7/+10
| | | | (#10082)
* Refactor checking restricted join rules (#10007)Patrick Cloke2021-05-181-20/+9
| | | | | To be more consistent with similar code. The check now automatically raises an AuthError instead of passing back a boolean. It also absorbs some shared logic between callers.
* Improve performance of backfilling in large rooms. (#9935)Erik Johnston2021-05-101-69/+54
| | | | | | We were pulling the full auth chain for the room out of the DB each time we backfilled, which can be *huge* for large rooms and is totally unnecessary.
* Don't set the external cache if its been done recently (#9905)Erik Johnston2021-05-051-1/+3
|
* Check for space membership during a remote join of a restricted room (#9814)Patrick Cloke2021-04-231-9/+35
| | | | | | When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-1/+1
|
* Separate creating an event context from persisting it in the federation ↵Patrick Cloke2021-04-141-65/+113
| | | | | | handler (#9800) This refactoring allows adding logic that uses the event context before persisting it.
* Revert "Check for space membership during a remote join of a restricted ↵Patrick Cloke2021-04-141-142/+70
| | | | | | | | room. (#9763)" This reverts commit cc51aaaa7adb0ec2235e027b5184ebda9b660ec4. The PR was prematurely merged and not yet approved.
* Check for space membership during a remote join of a restricted room. (#9763)Patrick Cloke2021-04-141-70/+142
| | | | | | | When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-1/+1
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Add type hints to the federation handler and server. (#9743)Patrick Cloke2021-04-061-80/+81
|
* Make RateLimiter class check for ratelimit overrides (#9711)Erik Johnston2021-03-301-1/+1
| | | | | | | This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited. We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits. Fixes #9663
* Optimise missing prev_event handling (#9601)Richard van der Hoff2021-03-151-21/+131
| | | | | | | | | | | | | | | | | | | | | | | | | | | Background: When we receive incoming federation traffic, and notice that we are missing prev_events from the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events, we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote server for the state at those prev_events, and then we may need to then ask the remote server for any events in that state which we don't already have, as well as the auth events for those missing state events, so that we can auth them. This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the observation that we are currently loading all of those auth events into memory at the start of the operation, but we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're actually going to need, but it doesn't.) The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about 60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache churn. (NB I've already tripled the size of the cache from its default of 10K). Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different (ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting that we clean it up.
* Improve logging when processing incoming transactions (#9596)Richard van der Hoff2021-03-121-46/+16
| | | Put the room id in the logcontext, to make it easier to understand what's going on.
* Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-3/+3
| | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-41/+61
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Remove dead handled_events set in invite_join (#9394)Andrew Morgan2021-02-121-6/+0
| | | | | This PR removes a set that was created and [initially used](https://github.com/matrix-org/synapse/commit/1d2a0040cff8d04cdc7d7d09d8f04a5d628fa9dd#diff-0bc92da3d703202f5b9be2d3f845e375f5b1a6bc6ba61705a8af9be1121f5e42R435-R436), but is no longer today. May help cut down a bit on the time it takes to accept invites.
* Honour ratelimit flag for application services for invite ratelimiting (#9302)Erik Johnston2021-02-031-1/+3
|
* Ratelimit invites by room and target user (#9258)Erik Johnston2021-01-291-0/+4
|
* Precompute joined hosts and store in Redis (#9198)Erik Johnston2021-01-261-0/+5
|
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-111-1/+1
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-1/+1
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Consistently use room_id from federation request body (#8776)Richard van der Hoff2020-11-191-5/+5
| | | | | | | | | | | | | * Consistently use room_id from federation request body Some federation APIs have a redundant `room_id` path param (see https://github.com/matrix-org/matrix-doc/issues/2330). We should make sure we consistently use either the path param or the body param, and the body param is easier. * Kill off some references to "context" Once upon a time, "rooms" were known as "contexts". I think this kills of the last references to "contexts".
* Generalise _maybe_store_room_on_invite (#8754)Andrew Morgan2020-11-131-4/+6
| | | | | | | | | There's a handy function called maybe_store_room_on_invite which allows us to create an entry in the rooms table for a room and its version for which we aren't joined to yet, but we can reference when ingesting events about. This is currently used for invites where we receive some stripped state about the room and pass it down via /sync to the client, without us being in the room yet. There is a similar requirement for knocking, where we will eventually do the same thing, and need an entry in the rooms table as well. Thus, reusing this function works, however its name needs to be generalised a bit. Separated out from #6739.
* Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-231-7/+7
|
* Move third_party_rules check to event creation timeRichard van der Hoff2020-10-131-44/+2
| | | | | Rather than waiting until we handle the event, call the ThirdPartyRules check when we fist create the event.
* Remove redundant calls to third_party_rules in `on_send_{join,leave}`Richard van der Hoff2020-10-131-19/+1
| | | | | There's not much point in calling these *after* we have decided to accept them into the DAG.
* Fix message duplication if something goes wrong after persisting the event ↵Erik Johnston2020-10-131-3/+6
| | | | | (#8476) Should fix #3365.
* Remove stream ordering from Metadata dict (#8452)Richard van der Hoff2020-10-051-0/+3
| | | | | | | | There's no need for it to be in the dict as well as the events table. Instead, we store it in a separate attribute in the EventInternalMetadata object, and populate that on load. This means that we can rely on it being correctly populated for any event which has been persited to the database.
* Move `resolve_events_with_store` into StateResolutionHandlerRichard van der Hoff2020-09-291-5/+8
|
* Mypy fixes for `synapse.handlers.federation` (#8422)Richard van der Hoff2020-09-291-4/+9
| | | For some reason, an apparently unrelated PR upset mypy about this module. Here are a number of little fixes.
* A pair of tiny cleanups in the federation request code. (#8401)Richard van der Hoff2020-09-281-1/+1
|
* Add EventStreamPosition type (#8388)Erik Johnston2020-09-241-6/+10
| | | | | | | | | | | | | | The idea is to remove some of the places we pass around `int`, where it can represent one of two things: 1. the position of an event in the stream; or 2. a token that partitions the stream, used as part of the stream tokens. The valid operations are then: 1. did a position happen before or after a token; 2. get all events that happened before or after a token; and 3. get all events between two tokens. (Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
* Merge tag 'v1.20.0rc5' into developPatrick Cloke2020-09-181-8/+57
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.20.0rc5 (2020-09-18) ============================== In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3. Features -------- - Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343)) Bugfixes -------- - Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342)) - Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349)) Internal Changes ---------------- - Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
| * Intelligently select extremities used in backfill. (#8349)Erik Johnston2020-09-181-8/+57
| | | | | | | | | | | | | | | | | | Instead of just using the most recent extremities let's pick the ones that will give us results that the pagination request cares about, i.e. pick extremities only if they have a smaller depth than the pagination token. This is useful when we fail to backfill an extremity, as we no longer get stuck requesting that same extremity repeatedly.
* | Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | | | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* | Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-1/+1
| | | | | | | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* | Add experimental support for sharding event persister. Again. (#8294)Erik Johnston2020-09-141-14/+30
| | | | | | | | | | | | This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
* | Clean up `Notifier.on_new_room_event` code path (#8288)Erik Johnston2020-09-101-3/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it. This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens. The valid operations here are: 1. Is a position before or after a token 2. Fetching all events between two tokens 3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C. Future PR will change the token type to a dedicated type.
* | Remove some unused distributor signals (#8216)Patrick Cloke2020-09-091-42/+1
| | | | | | | | | | Removes the `user_joined_room` and stops calling it since there are no observers. Also cleans-up some other unused signals and related code.
* | Fixup pusher pool notifications (#8287)Erik Johnston2020-09-091-1/+1
| | | | | | | | | | `pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed. I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
* | Revert "Fixup pusher pool notifications"Erik Johnston2020-09-091-1/+1
| | | | | | | | This reverts commit e7fd336a53a4ca489cdafc389b494d5477019dc0.
* | Fixup pusher pool notificationsErik Johnston2020-09-091-1/+1
|/
* Revert "Add experimental support for sharding event persister. (#8170)" (#8242)Brendan Abolivier2020-09-041-30/+14
| | | | | | | * Revert "Add experimental support for sharding event persister. (#8170)" This reverts commit 82c1ee1c22a87b9e6e3179947014b0f11c0a1ac3. * Changelog
* Fix typing for `@cached` wrapped functions (#8240)Erik Johnston2020-09-031-5/+5
| | | This requires adding a mypy plugin to fiddle with the type signatures a bit.
* Add experimental support for sharding event persister. (#8170)Erik Johnston2020-09-021-14/+30
| | | | | | This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
* Convert additional database code to async/await. (#8195)Patrick Cloke2020-08-281-2/+2
|
* Define StateMap as immutable and add a MutableStateMap type. (#8183)Patrick Cloke2020-08-281-6/+14
|
* Add type hints for state. (#8140)Patrick Cloke2020-08-241-4/+6
|
* Convert events worker database to async/await. (#8071)Patrick Cloke2020-08-181-11/+5
|
* Convert synapse.api to async/await (#8031)Patrick Cloke2020-08-061-1/+1
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-1/+1
|
* Convert a synapse.events to async/await. (#7949)Patrick Cloke2020-07-271-1/+1
|
* Remove hacky error handling for inlineDeferreds. (#7950)Patrick Cloke2020-07-271-9/+5
|
* Fix up types and comments that refer to Deferreds. (#7945)Patrick Cloke2020-07-241-3/+5
|
* Fix deprecation warning: import ABC from collections.abc (#7892)Karthikeyan Singaravelan2020-07-201-1/+1
|
* Reject attempts to join empty rooms over federation (#7859)Richard van der Hoff2020-07-161-2/+13
| | | | | | We shouldn't allow others to make_join through us if we've left the room; reject such attempts with a 404. Fixes #7835. Fixes #6958.
* Fix resync remote devices on receive PDU in worker mode. (#7815)Erik Johnston2020-07-101-8/+19
| | | | | | The replication client requires that arguments are given as keyword arguments, which was not done in this case. We also pull out the logic so that we can catch and handle any exceptions raised, rather than leaving them unhandled.
* Fix recursion error when fetching auth chain over federation (#7817)Erik Johnston2020-07-101-12/+37
| | | | | | | | | | | | | | | When fetching the state of a room over federation we receive the event IDs of the state and auth chain. We then fetch those events that we don't already have. However, we used a function that recursively fetched any missing auth events for the fetched events, which can lead to a lot of recursion if the server is missing most of the auth chain. This work is entirely pointless because would have queued up the missing events in the auth chain to be fetched already. Let's just diable the recursion, since it only gets called from one place anyway.
* Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-1/+1
| | | ... instead of duplicating `config.signing_key[0]` everywhere
* Merge branch 'master' into developPatrick Cloke2020-07-021-3/+3
|\
| * Correctly handle outliers as prev events over federationErik Johnston2020-07-021-3/+3
| |
* | Add early returns to `_check_for_soft_fail` (#7769)Richard van der Hoff2020-07-011-64/+55
| | | | | | | | my editor was complaining about unset variables, so let's add some early returns to fix that and reduce indentation/cognitive load.
* | Type checking for `FederationHandler` (#7770)Richard van der Hoff2020-07-011-17/+30
| | | | | | fix a few things to make this pass mypy.
* | Yield during large v2 state res. (#7735)Erik Johnston2020-06-241-0/+1
| | | | | | | | | | | | | | | | | | | | | | State res v2 across large data sets can be very CPU intensive, and if all the relevant events are in the cache the algorithm will run from start to finish within a single reactor tick. This can result in blocking the reactor tick for several seconds, which can have major repercussions on other requests. To fix this we simply add the occaisonal `sleep(0)` during iterations to yield execution until the next reactor tick. The aim is to only do this for large data sets so that we don't impact otherwise quick resolutions.=
* | Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-5/+4
| |
* | Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-12/+10
| |
* | Add option to enable encryption by default for new rooms (#7639)Andrew Morgan2020-06-101-2/+10
|/ | | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/2431 Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used. Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637 Based on #7637
* Fix exceptions when fetching events from a down host. (#7622)Erik Johnston2020-06-031-1/+1
| | | We already caught some exceptions, but not all.
* Add option to move event persistence off master (#7517)Erik Johnston2020-05-221-6/+10
|
* Add ability to wait for replication streams (#7542)Erik Johnston2020-05-221-10/+23
| | | | | | | The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room). Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on. People probably want to look at this commit by commit.
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-0/+7
| | | | These are business as usual errors, rather than stuff we want to log at error.
* Convert federation handler to async/await. (#7459)Patrick Cloke2020-05-111-18/+14
|
* async/await is_server_admin (#7363)Andrew Morgan2020-05-011-11/+10
|
* Convert some of the federation handler methods to async/await. (#7338)Patrick Cloke2020-04-241-25/+24
|
* Rewrite prune_old_outbound_device_pokes for efficiency (#7159)Richard van der Hoff2020-03-301-23/+2
| | | | make sure we clear out all but one update for the user
* Store room version on invite (#6983)Richard van der Hoff2020-02-261-0/+12
| | | | | When we get an invite over federation, store the room version in the rooms table. The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere...
* Remove redundant store_room call (#6979)Richard van der Hoff2020-02-241-23/+0
| | | | | `_process_received_pdu` is only called by `on_receive_pdu`, which ignores any events for unknown rooms, so this is redundant.
* Upsert room version when we join over federation (#6968)Richard van der Hoff2020-02-241-10/+12
| | | | | | | | This is intended as a precursor to storing room versions when we receive an invite over federation, but has the happy side-effect of fixing #3374 at last. In short: change the store_room with try/except to a proper upsert which updates the right columns.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-9/+9
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Limit the number of events that can be requested when backfilling events (#6864)Patrick Cloke2020-02-061-0/+4
| | | Limit the maximum number of events requested when backfilling events.
* pass room version into FederationClient.send_join (#6854)Richard van der Hoff2020-02-061-2/+1
| | | | ... which allows us to sanity-check the create event.
* Merge pull request #6823 from matrix-org/rav/redact_changes/5Richard van der Hoff2020-02-061-6/+2
|\ | | | | pass room versions around
| * Pass room version object into `FederationClient.get_pdu`Richard van der Hoff2020-02-051-6/+2
| |
* | Merge tag 'v1.10.0rc2' into developErik Johnston2020-02-061-14/+60
|\ \ | |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.10.0rc2 (2020-02-06) ============================== Bugfixes -------- - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844)) - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848)) Internal Changes ---------------- - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
| * Check sender_key matches on inbound encrypted events. (#6850)Erik Johnston2020-02-051-13/+59
| | | | | | If they don't then the device lists are probably out of sync.
| * Fix detecting unknown devices from remote encrypted events. (#6848)Erik Johnston2020-02-041-1/+1
| | | | | | | | | | | | | | | | We were looking at the wrong event type (`m.room.encryption` vs `m.room.encrypted`). Also fixup the duplicate `EvenTypes` entries. Introduced in #6776.
* | make FederationHandler.send_invite asyncRichard van der Hoff2020-02-031-3/+2
| |
* | make FederationHandler.on_get_missing_events asyncRichard van der Hoff2020-02-031-5/+4
| |
* | make FederationHandler.user_joined_room asyncRichard van der Hoff2020-02-031-3/+3
| |
* | make FederationHandler._clean_room_for_join asyncRichard van der Hoff2020-02-031-4/+4
| |
* | make FederationHandler._notify_persisted_event asyncRichard van der Hoff2020-02-031-4/+6
| |
* | make FederationHandler.persist_events_and_notify asyncRichard van der Hoff2020-02-031-10/+10
| |
* | make FederationHandler._make_and_verify_event asyncRichard van der Hoff2020-02-031-5/+10
| |
* | make FederationHandler.do_remotely_reject_invite asyncRichard van der Hoff2020-02-031-6/+7
| |
* | make FederationHandler._check_for_soft_fail asyncRichard van der Hoff2020-02-031-13/+9
| |
* | make FederationHandler._persist_auth_tree asyncRichard van der Hoff2020-02-031-11/+7
| |
* | make FederationHandler.do_invite_join asyncRichard van der Hoff2020-02-031-16/+14
| |
* | make FederationHandler.on_event_auth asyncRichard van der Hoff2020-02-031-5/+4
| |
* | make FederationHandler.on_exchange_third_party_invite_request asyncRichard van der Hoff2020-02-031-14/+12
| |
* | make FederationHandler.construct_auth_difference asyncRichard van der Hoff2020-02-031-3/+4
| |
* | make FederationHandler._update_context_for_auth_events asyncRichard van der Hoff2020-02-031-10/+10
| |
* | make FederationHandler._update_auth_events_and_context_for_auth asyncRichard van der Hoff2020-02-031-20/+21
| |
* | make FederationHandler.do_auth asyncRichard van der Hoff2020-02-031-10/+14
| |
* | make FederationHandler._prep_event asyncRichard van der Hoff2020-02-031-23/+10
| |
* | make FederationHandler._handle_new_event asyncRichard van der Hoff2020-02-031-6/+7
| |
* | make FederationHandler._handle_new_events asyncRichard van der Hoff2020-02-031-8/+6
| |
* | make FederationHandler.on_make_leave_request asyncRichard van der Hoff2020-02-031-13/+10
| |
* | make FederationHandler.on_send_leave_request asyncRichard van der Hoff2020-02-031-5/+3
| |
* | make FederationHandler.on_make_join_request asyncRichard van der Hoff2020-02-031-13/+10
| |
* | make FederationHandler.on_invite_request asyncRichard van der Hoff2020-02-031-5/+4
| |
* | make FederationHandler.on_send_join_request asyncRichard van der Hoff2020-02-031-9/+7
| |
* | make FederationHandler.on_query_auth asyncRichard van der Hoff2020-02-031-7/+6
|/
* pass room_version into compute_event_signature (#6807)Richard van der Hoff2020-01-311-1/+4
|
* Merge pull request #6820 from matrix-org/rav/get_room_version_idRichard van der Hoff2020-01-311-9/+9
|\ | | | | Make `get_room_version` return a RoomVersion object
| * s/get_room_version/get_room_version_id/Richard van der Hoff2020-01-311-9/+9
| | | | | | | | | | ... to make way for a forthcoming get_room_version which returns a RoomVersion object.
* | Fix bug with getting missing auth event during join 500'ed (#6810)Erik Johnston2020-01-311-1/+5
|/
* pass room version into FederationHandler.on_invite_request (#6805)Richard van der Hoff2020-01-301-3/+3
|
* Resync remote device list when detected as stale. (#6786)Erik Johnston2020-01-301-2/+16
|
* Detect unknown remote devices and mark cache as stale (#6776)Erik Johnston2020-01-281-0/+20
| | | | We just mark the fact that the cache may be stale in the database for now.
* Pass room version object into event_auth.check and check_redaction (#6788)Richard van der Hoff2020-01-281-7/+11
| | | | | | | These are easier to work with than the strings and we normally have one around. This fixes `FederationHander._persist_auth_tree` which was passing a RoomVersion object into event_auth.check instead of a string.
* Add `rooms.room_version` column (#6729)Erik Johnston2020-01-271-15/+50
| | | This is so that we don't have to rely on pulling it out from `current_state_events` table.
* Add StateMap type alias (#6715)Erik Johnston2020-01-161-6/+4
|
* Fix conditions failing if min_depth = 0Brendan Abolivier2020-01-071-2/+2
| | | | This could result in Synapse not fetching prev_events for new events in the room if it has missed some events.
* Merge branch 'master' into developRichard van der Hoff2019-12-201-1/+4
|\
| * Fix exceptions when attempting to backfill (#6576)Richard van der Hoff2019-12-201-1/+4
| | | | | | Fixes #6575
* | Change EventContext to use the Storage class (#6564)Erik Johnston2019-12-201-7/+7
| |
* | Merge release-v1.7.1 into developRichard van der Hoff2019-12-181-0/+1
|\|
| * Exclude rejected state events when calculating state at backwards extrems ↵Richard van der Hoff2019-12-161-1/+1
| | | | | | | | | | (#6527) This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
| * Persist auth/state events at backwards extremities when we fetch them (#6526)Richard van der Hoff2019-12-161-163/+80
| | | | | | | | The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
| * sanity-checking for events used in state res (#6531)Richard van der Hoff2019-12-161-0/+1
| | | | | | | | | | When we perform state resolution, check that all of the events involved are in the right room.
| * Check the room_id of events when fetching room state/auth (#6524)Richard van der Hoff2019-12-161-24/+54
| | | | | | | | | | | | | | | | | | | | | | When we request the state/auth_events to populate a backwards extremity (on backfill or in the case of missing events in a transaction push), we should check that the returned events are in the right room rather than blindly using them in the room state or auth chain. Given that _get_events_from_store_or_dest takes a room_id, it seems clear that it should be sanity-checking the room_id of the requested events, so let's do it there.
| * Add `include_event_in_state` to _get_state_for_room (#6521)Richard van der Hoff2019-12-161-18/+21
| | | | | | | | | | | | Make it return the state *after* the requested event, rather than the one before it. This is a bit easier and requires fewer calls to get_events_from_store_or_dest.
| * Move get_state methods into FederationHandler (#6503)Richard van der Hoff2019-12-161-6/+95
| | | | | | | | | | This is a non-functional refactor as a precursor to some other work.
* | Exclude rejected state events when calculating state at backwards extrems ↵Richard van der Hoff2019-12-161-1/+1
| | | | | | | | | | (#6527) This fixes a weird bug where, if you were determined enough, you could end up with a rejected event forming part of the state at a backwards-extremity. Authing that backwards extrem would then lead to us trying to pull the rejected event from the db (with allow_rejected=False), which would fail with a 404.
* | Persist auth/state events at backwards extremities when we fetch them (#6526)Richard van der Hoff2019-12-161-167/+80
| | | | | | The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
* | sanity-checking for events used in state res (#6531)Richard van der Hoff2019-12-131-0/+1
| | | | | | | | | | | | | | When we perform state resolution, check that all of the events involved are in the right room.
* | Check the room_id of events when fetching room state/auth (#6524)Richard van der Hoff2019-12-121-23/+51
| | | | | | | | | | | | | | | | | | | | When we request the state/auth_events to populate a backwards extremity (on backfill or in the case of missing events in a transaction push), we should check that the returned events are in the right room rather than blindly using them in the room state or auth chain. Given that _get_events_from_store_or_dest takes a room_id, it seems clear that it should be sanity-checking the room_id of the requested events, so let's do it there.
* | Add `include_event_in_state` to _get_state_for_room (#6521)Richard van der Hoff2019-12-111-22/+28
| | | | | | | | | | Make it return the state *after* the requested event, rather than the one before it. This is a bit easier and requires fewer calls to get_events_from_store_or_dest.
* | convert to async: FederationHandler._process_received_pduRichard van der Hoff2019-12-111-11/+10
| | | | | | | | also fix user_joined_room to consistently return deferreds
* | convert to async: FederationHandler._get_state_for_roomRichard van der Hoff2019-12-111-21/+21
| | | | | | | | ... and _get_events_from_store_or_dest
* | convert to async: FederationHandler.on_receive_pduRichard van der Hoff2019-12-111-27/+22
| | | | | | | | | | | | | | and associated functions: * on_receive_pdu * handle_queued_pdus * get_missing_events_for_pdu
* | Convert federation backfill to asyncRichard van der Hoff2019-12-111-25/+22
| | | | | | | | | | | | | | | | | | PaginationHandler.get_messages is only called by RoomMessageListRestServlet, which is async. Chase the code path down from there: - FederationHandler.maybe_backfill (and nested try_backfill) - FederationHandler.backfill
* | Clean up some logging (#6515)Richard van der Hoff2019-12-111-18/+19
| | | | | | | | This just makes some of the logging easier to follow when things start going wrong.
* | Prevent redacted events from appearing in message search (#6377)Andrew Morgan2019-12-111-2/+5
| |
* | Prevent message search in upgraded rooms we're not in (#6385)Andrew Morgan2019-12-111-2/+2
| |
* | Move get_state methods into FederationHandler (#6503)Richard van der Hoff2019-12-101-6/+95
|/ | | | | This is a non-functional refactor as a precursor to some other work.
* Stronger typing in the federation handler (#6480)Richard van der Hoff2019-12-051-24/+57
| | | | | replace the event_info dict with an attrs thing
* Sanity-check the rooms of auth events before pulling them in. (#6472)Richard van der Hoff2019-12-051-9/+25
|
* get rid of (most of) have_events from ↵Richard van der Hoff2019-12-041-38/+24
| | | | | | | | | | | | | _update_auth_events_and_context_for_auth (#6468) have_events was a map from event_id to rejection reason (or None) for events which are in our local database. It was used as filter on the list of event_ids being passed into get_events_as_list. However, since get_events_as_list will ignore any event_ids that are unknown or rejected, we can equivalently just leave it to get_events_as_list to do the filtering. That means that we don't have to keep `have_events` up-to-date, and can use `have_seen_events` instead of `get_seen_events_with_rejection` in the one place we do need it.
* Add ephemeral messages support (MSC2228) (#6409)Brendan Abolivier2019-12-031-0/+8
| | | | | | | | Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are: * the feature is hidden behind a configuration flag (`enable_ephemeral_messages`) * self-destruction doesn't happen for state events * only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one) * doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
* Propagate reason in remotely rejected invitesErik Johnston2019-11-281-2/+2
|
* Merge pull request #6358 from matrix-org/babolivier/message_retentionBrendan Abolivier2019-11-271-2/+2
|\ | | | | Implement message retention policies (MSC1763)
| * Merge branch 'develop' into babolivier/message_retentionBrendan Abolivier2019-11-261-3/+8
| |\
| * | Fix 3PID invite exchangeBrendan Abolivier2019-11-191-1/+1
| | |
| * | Implement per-room message retention policiesBrendan Abolivier2019-11-041-1/+1
| | |
* | | remove confusing fixmeRichard van der Hoff2019-11-261-6/+0
| | |
* | | Merge remote-tracking branch 'origin/develop' into rav/event_auth/4Richard van der Hoff2019-11-181-2/+2
|\ \ \ | | |/ | |/|
| * | Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-2/+2
| | |
* | | Use get_events_as_list rather than lots of calls to get_eventRichard van der Hoff2019-11-081-16/+8
| | | | | | | | | | | | It's more efficient and clearer.
* | | Update some docstrings and commentsRichard van der Hoff2019-11-081-8/+31
| | |
* | | Simplify _update_auth_events_and_context_for_authRichard van der Hoff2019-11-081-11/+9
|/ / | | | | | | | | move event_key calculation into _update_context_for_auth_events, since it's only used there.
* | Fix bug which caused rejected events to be stored with the wrong room state ↵Richard van der Hoff2019-11-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | (#6320) Fixes a bug where rejected events were persisted with the wrong state group. Also fixes an occasional internal-server-error when receiving events over federation which are rejected and (possibly because they are backwards-extremities) have no prev_group. Fixes #6289.
* | Add some checks that we aren't using state from rejected events (#6330)Richard van der Hoff2019-11-051-1/+5
|/ | | | | | | | | | | | | * Raise an exception if accessing state for rejected events Add some sanity checks on accessing state_group etc for rejected events. * Skip calculating push actions for rejected events It didn't actually cause any bugs, because rejected events get filtered out at various later points, but there's not point in trying to calculate the push actions for a rejected event.
* Factor out an _AsyncEventContextImpl (#6298)Richard van der Hoff2019-11-011-19/+19
| | | | | | The intention here is to make it clearer which fields we can expect to be populated when: notably, that the _event_type etc aren't used for the synchronous impl of EventContext.
* Depublish a room from the public rooms list when it is upgraded (#6232)Andrew Morgan2019-11-011-1/+29
|
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-4/+5
| | | * update version of black and also fix the mypy config being overridden
* Merge pull request #6294 from matrix-org/erikj/add_state_storageErik Johnston2019-10-311-9/+10
|\ | | | | Add StateGroupStorage interface
| * Port to use state storageErik Johnston2019-10-301-9/+10
| |
* | Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-16/+20
|/ | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-301-3/+19
|\ | | | | | | erikj/split_out_persistence_store
| * Fix log line that was printing undefined value (#6278)Andrew Morgan2019-10-301-1/+1
| |
| * Improve signature checking on some federation APIs (#6262)Richard van der Hoff2019-10-281-2/+18
| | | | | | | | | | Make sure that we check that events sent over /send_join, /send_leave, and /invite, are correctly signed and come from the expected servers.
* | Use new EventPersistenceStoreErik Johnston2019-10-231-1/+2
|/
* Remove Auth.check method (#6217)Richard van der Hoff2019-10-181-3/+4
| | | This method was somewhat redundant, and confusing.
* rip out some unreachable codeRichard van der Hoff2019-10-171-102/+0
| | | | The only possible rejection reason is AUTH_ERROR, so all of this is unreachable.
* TypoBrendan Abolivier2019-10-041-1/+1
|
* LintBrendan Abolivier2019-10-041-3/+1
|
* Incorporate reviewBrendan Abolivier2019-10-041-14/+8
|
* Add test caseBrendan Abolivier2019-10-031-1/+1
|
* LintBrendan Abolivier2019-10-021-1/+3
|
* Don't 500 code when trying to exchange a revoked 3PID inviteBrendan Abolivier2019-10-021-2/+13
| | | | | | | | While this is not documented in the spec (but should be), Riot (and other clients) revoke 3PID invites by sending a m.room.third_party_invite event with an empty ({}) content to the room's state. When the invited 3PID gets associated with a MXID, the identity server (which doesn't know about revocations) sends down to the MXID's homeserver all of the undelivered invites it has for this 3PID. The homeserver then tries to talk to the inviting homeserver in order to exchange these invite for m.room.member events. When one of the invite is revoked, the inviting homeserver responds with a 500 error because it tries to extract a 'display_name' property from the content, which is empty. This might cause the invited server to consider that the server is down and not try to exchange other, valid invites (or at least delay it). This fix handles the case of revoked invites by avoiding trying to fetch a 'display_name' from the original invite's content, and letting the m.room.member event fail the auth rules (because, since the original invite's content is empty, it doesn't have public keys), which results in sending a 403 with the correct error message to the invited server.
* Remove origin parameter from add_display_name_to_third_party_invite and add ↵Andrew Morgan2019-09-111-1/+6
| | | | | | | params to docstring (#6010) Another small fixup noticed during work on a larger PR. The `origin` field of `add_display_name_to_third_party_invite` is not used and likely was just carried over from the `on_PUT` method of `FederationThirdPartyInviteExchangeServlet` which, like all other servlets, provides an `origin` argument. Since it's not used anywhere in the handler function though, we should remove it from the function arguments.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Implement a structured logging output system. (#5680)Amber Brown2019-08-281-2/+3
|
* Merge pull request #5790 from matrix-org/erikj/groups_request_errorsErik Johnston2019-08-011-0/+3
|\ | | | | Handle RequestSendFailed exception correctly in more places.
| * Don't log as exception when failing durig backfillErik Johnston2019-07-301-0/+3
| |
* | Room Complexity Client Implementation (#5783)Amber Brown2019-07-301-0/+25
|/
* Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatchRichard van der Hoff2019-07-261-2/+35
|\ | | | | Log when we receive a /make_* request from a different origin
| * Log when we receive a /make_* request from a different originRichard van der Hoff2019-07-261-2/+35
| |
* | Replace returnValue with return (#5736)Amber Brown2019-07-231-42/+40
|/
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-21/+22
|
* Run Black. (#5482)Amber Brown2019-06-201-521/+363
|
* Merge pull request #5042 from matrix-org/erikj/fix_get_missing_events_errorErik Johnston2019-06-191-9/+19
|\ | | | | Handle the case of `get_missing_events` failing
| * Fix loglineErik Johnston2019-06-181-1/+1
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-06-181-132/+289
| |\ | | | | | | | | | erikj/fix_get_missing_events_error
| * | Handle the case of `get_missing_events` failingErik Johnston2019-04-101-9/+20
| | | | | | | | | | | | | | | | | | | | | Currently if a call to `/get_missing_events` fails we log an exception and stop processing the top level event we received over federation. Instead let's try and handle it sensibly given it is a somewhat expected failure mode.
* | | Add some logging to 3pid invite sig verification (#5015)David Baker2019-06-181-8/+38
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | I had to add quite a lot of logging to diagnose a problem with 3pid invites - we only logged the one failure which isn't all that informative. NB. I'm not convinced the logic of this loop is right: I think it should just accept a single valid signature from a trusted source rather than fail if *any* signature is invalid. Also it should probably not skip the rest of middle loop if a check fails? However, I'm deliberately not changing the logic here.
* | Merge pull request #5464 from matrix-org/erikj/3pid_remote_invite_stateErik Johnston2019-06-171-6/+0
|\ \ | | | | | | Fix 3PID invite room state over federation.
| * | Fix 3PID invite room state over federation.Erik Johnston2019-06-141-6/+0
| | | | | | | | | | | | | | | | | | | | | Fixes that when a user exchanges a 3PID invite for a proper invite over federation it does not include the `invite_room_state` key. This was due to synapse incorrectly sending out two invite requests.
* | | Add plugin APIs for implementations of custom event rules.Brendan Abolivier2019-06-141-2/+66
|/ /
* | Add logging when request fails and clarify we ignore errors.Erik Johnston2019-06-051-4/+12
| |
* | Fix handling of failures when calling /event_auth.Erik Johnston2019-06-031-12/+38
| | | | | | | | | | | | | | | | | | | | When processing an incoming event over federation, we may try and resolve any unexpected differences in auth events. This is a non-essential process and so should not stop the processing of the event if it fails (e.g. due to the remote disappearing or not implementing the necessary endpoints). Fixes #3330
* | Simplifications and comments in do_auth (#5227)Richard van der Hoff2019-05-231-120/+181
| | | | | | | | | | I was staring at this function trying to figure out wtf it was actually doing. This is (hopefully) a non-functional refactor which makes it a bit clearer.
* | Exclude soft-failed events from fwd-extremity candidates. (#5146)Richard van der Hoff2019-05-211-1/+6
|/ | | | | | | | When considering the candidates to be forward-extremities, we must exclude soft failures. Hopefully fixes #5090.
* Collect room-version variations into one place (#4969)Richard van der Hoff2019-04-011-8/+5
| | | | Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.
* Factor out soft fail checksErik Johnston2019-03-081-5/+22
|
* Implement soft failErik Johnston2019-03-061-1/+76
|