summary refs log tree commit diff
path: root/synapse/handlers/message.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add some tracing spans to give insight into local joins (#13439)Shay2022-08-031-6/+9
|
* Return 404 or member list when getting joined_members after leaving (#13374)andrew do2022-08-031-2/+4
| | | | | | Signed-off-by: Andrew Doh <andrewddo@gmail.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrewm@element.io> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Implement MSC3848: Introduce errcodes for specific event sending failures ↵Will Hunt2022-07-271-2/+11
| | | | | (#13343) Implements MSC3848
* Faster room joins: avoid blocking when pulling events with missing prevs ↵Sean Quah2022-07-261-0/+4
| | | | | | | | | (#13355) Avoid blocking on full state in `_resolve_state_at_missing_prevs` and return a new flag indicating whether the resolved state is partial. Thread that flag around so that it makes it into the event context. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Rate limit joins per-room (#13276)David Robertson2022-07-191-0/+11
|
* Don't pull out full state when sending dummy events (#13310)Erik Johnston2022-07-181-7/+1
|
* Don't pull out state in `compute_event_context` for unconflicted state (#13267)Erik Johnston2022-07-141-1/+6
|
* Handle race between persisting an event and un-partial stating a room (#13100)Sean Quah2022-07-051-26/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Whenever we want to persist an event, we first compute an event context, which includes the state at the event and a flag indicating whether the state is partial. After a lot of processing, we finally try to store the event in the database, which can fail for partial state events when the containing room has been un-partial stated in the meantime. We detect the race as a foreign key constraint failure in the data store layer and turn it into a special `PartialStateConflictError` exception, which makes its way up to the method in which we computed the event context. To make things difficult, the exception needs to cross a replication request: `/fed_send_events` for events coming over federation and `/send_event` for events from clients. We transport the `PartialStateConflictError` as a `409 Conflict` over replication and turn `409`s back into `PartialStateConflictError`s on the worker making the request. All client events go through `EventCreationHandler.handle_new_client_event`, which is called in *a lot* of places. Instead of trying to update all the code which creates client events, we turn the `PartialStateConflictError` into a `429 Too Many Requests` in `EventCreationHandler.handle_new_client_event` and hope that clients take it as a hint to retry their request. On the federation event side, there are 7 places which compute event contexts. 4 of them use outlier event contexts: `FederationEventHandler._auth_and_persist_outliers_inner`, `FederationHandler.do_knock`, `FederationHandler.on_invite_request` and `FederationHandler.do_remotely_reject_invite`. These events won't have the partial state flag, so we do not need to do anything for then. The remaining 3 paths which create events are `FederationEventHandler.process_remote_join`, `FederationEventHandler.on_send_membership_event` and `FederationEventHandler._process_received_pdu`. We can't experience the race in `process_remote_join`, unless we're handling an additional join into a partial state room, which currently blocks, so we make no attempt to handle it correctly. `on_send_membership_event` is only called by `FederationServer._on_send_membership_event`, so we catch the `PartialStateConflictError` there and retry just once. `_process_received_pdu` is called by `on_receive_pdu` for incoming events and `_process_pulled_event` for backfill. The latter should never try to persist partial state events, so we ignore it. We catch the `PartialStateConflictError` in `on_receive_pdu` and retry just once. Refering to the graph of code paths in https://github.com/matrix-org/synapse/issues/12988#issuecomment-1156857648 may make the above make more sense. Signed-off-by: Sean Quah <seanq@matrix.org>
* Don't process /send requests for users who have hit their ratelimit (#13134)Shay2022-06-301-0/+3
|
* Decouple `synapse.api.auth_blocking.AuthBlocking` from ↵Quentin Gliech2022-06-141-2/+2
| | | | `synapse.api.auth.Auth`. (#13021)
* Uniformize spam-checker API, part 4: port other spam-checker callbacks to ↵David Teller2022-06-131-7/+5
| | | | | return `Union[Allow, Codes]`. (#12857) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Merge branch 'rav/simplify_event_auth_interface' into developRichard van der Hoff2022-06-131-20/+3
|\
| * Remove redundant `room_version` param from `check_auth_rules_from_context`Richard van der Hoff2022-06-121-19/+2
| | | | | | | | It's now implied by the room_version property on the event.
| * Remove `room_version` param from `validate_event_for_room_version`Richard van der Hoff2022-06-121-1/+1
| | | | | | | | | | | | | | Instead, use the `room_version` property of the event we're validating. The `room_version` was originally added as a parameter somewhere around #4482, but really it's been redundant since #6875 added a `room_version` field to `EventBase`.
* | Faster joins: add issue links to the TODOs (#13004)Richard van der Hoff2022-06-091-0/+1
|/ | | | ... to help us keep track of these things
* Reduce the amount of state we pull from the DB (#12811)Erik Johnston2022-06-061-1/+3
|
* Wait for lazy join to complete when getting current state (#12872)Erik Johnston2022-06-011-1/+1
|
* Fix 404 on `/sync` when the last event is a redaction of an unknown/purged ↵Richard van der Hoff2022-06-011-37/+77
| | | | | | | | | | | event (#12905) Currently, we try to pull the event corresponding to a sync token from the database. However, when we fetch redaction events, we check the target of that redaction (because we aren't allowed to send redactions to clients without validating them). So, if the sync token points to a redaction of an event that we don't have, we have a problem. It turns out we don't really need that event, and can just work with its ID and metadata, which sidesteps the whole problem.
* Merge branch 'master' into developErik Johnston2022-05-311-8/+20
|\
| * Fix import in module_api module and docs on the new check_event_for_spam ↵Brendan Abolivier2022-05-311-4/+15
| | | | | | | | | | signature (#12918) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Rename storage classes (#12913)Erik Johnston2022-05-311-13/+17
| |
* | Uniformize spam-checker API, part 3: Expand check_event_for_spam with the ↵David Teller2022-05-301-0/+15
| | | | | | | | | | ability to return additional fields (#12846) Signed-off-by: David Teller <davidt@element.io>
* | Pull out less state when handling gaps mk2 (#12852)Erik Johnston2022-05-261-3/+37
| |
* | Fix up `state_store` naming (#12871)Erik Johnston2022-05-251-5/+5
|/
* Uniformize spam-checker API, part 2: check_event_for_spam (#12808)David Teller2022-05-231-5/+6
| | | Signed-off-by: David Teller <davidt@element.io>
* Update EventContext `get_current_event_ids` and `get_prev_event_ids` to ↵Shay2022-05-201-3/+11
| | | | accept state filters and update calls where possible (#12791)
* Consolidate logic for parsing relations. (#12693)Patrick Cloke2022-05-161-18/+12
| | | | | | | | | | | | | Parse the `m.relates_to` event content field (which describes relations) in a single place, this is used during: * Event persistence. * Validation of the Client-Server API. * Fetching bundled aggregations. * Processing of push rules. Each of these separately implement the logic and each made slightly different assumptions about what was valid. Some had minor / potential bugs.
* Remove unneeded `ActionGenerator` class. (#12691)Patrick Cloke2022-05-111-2/+4
| | | | It simply passes through to `BulkPushRuleEvaluator`, which can be called directly instead.
* Refactor `EventContext` (#12689)Erik Johnston2022-05-101-1/+5
| | | | | | | | | | Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing. The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used. Part of #12684
* Update mypy to 0.950 and fix complaints (#12650)David Robertson2022-05-061-1/+2
|
* remove constantly lib use and switch to enums. (#12624)andrew do2022-05-041-2/+2
|
* Fix missing sync events during historical batch imports (#12319)Nick Mills-Barrett2022-04-131-9/+5
| | | | | | | | Discovered after much in-depth investigation in #12281. Closes: #12281 Closes: #3305 Signed off by: Nick Mills-Barrett nick@beeper.com
* Remove references to unstable identifiers from MSC3440. (#12382)Patrick Cloke2022-04-121-4/+1
| | | | | Removes references to unstable thread relation, unstable identifiers for filtering parameters, and the experimental config flag.
* Refactor and convert `Linearizer` to async (#12357)Sean Quah2022-04-051-1/+1
| | | | | | | | | | | Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Remove unused `auth_event_ids` argument plumbing (#12304)Eric Eastwood2022-03-291-8/+0
| | | | | | | | | | | | | | Follow-up to https://github.com/matrix-org/synapse/pull/12083 Since we are now using the new `state_event_ids` parameter to do all of the heavy lifting. We can remove any spots where we plumbed `auth_event_ids` just for MSC2716 things in https://github.com/matrix-org/synapse/pull/9247/files. Removing `auth_event_ids` from following functions: - `create_and_send_nonmember_event` - `_local_membership_update` - `update_membership` - `update_membership_locked`
* Refactor `create_new_client_event` to use a new parameter, ↵Eric Eastwood2022-03-251-16/+47
| | | | | | | `state_event_ids`, which accurately describes the usage with MSC2716 instead of abusing `auth_event_ids` (#12083) Spawned from https://github.com/matrix-org/synapse/pull/10975#discussion_r813183430 Part of [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716)
* Support stable identifiers for MSC3440: Threading (#12151)Patrick Cloke2022-03-101-1/+4
| | | | The unstable identifiers are still supported if the experimental configuration flag is enabled. The unstable identifiers will be removed in a future release.
* Limit the size of the aggregation_key (#12101)Erik Johnston2022-03-031-0/+3
| | | There's no reason to let people use long keys.
* Faster joins: persist to database (#12012)Richard van der Hoff2022-03-011-0/+2
| | | | | | | | | | | | When we get a partial_state response from send_join, store information in the database about it: * store a record about the room as a whole having partial state, and stash the list of member servers too. * flag the join event itself as having partial state * also, for any new events whose prev-events are partial-stated, note that they will *also* be partial-stated. We don't yet make any attempt to interpret this data, so API calls (and a bunch of other things) are just going to get incorrect data.
* Improve exception handling for concurrent execution (#12109)Richard van der Hoff2022-03-011-2/+2
| | | | | | | | | | | | | | | * fix incorrect unwrapFirstError import this was being imported from the wrong place * Refactor `concurrently_execute` to use `yieldable_gather_results` * Improve exception handling in `yieldable_gather_results` Try to avoid swallowing so many stack traces. * mark unwrapFirstError deprecated * changelog
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-2/+2
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Minor typing fixes (#12034)Richard van der Hoff2022-02-211-4/+6
| | | | | | These started failing in https://github.com/matrix-org/synapse/pull/12031... I'm a bit mystified by how they ever worked.
* Fix historical messages backfilling in random order on remote homeservers ↵Eric Eastwood2022-02-071-4/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#11114) Fix https://github.com/matrix-org/synapse/issues/11091 Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`) 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`) - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)). - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort. - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date. - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order. 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls. - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793 Before | After --- | --- ![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png) #### Why aren't we sorting topologically when receiving backfill events? > The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway. > > As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering. > > -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138 See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
* Bundle aggregations outside of the serialization method. (#11612)Patrick Cloke2022-01-071-1/+1
| | | | | | | | This makes the serialization of events synchronous (and it no longer access the database), but we must manually calculate and provide the bundled aggregations. Overall this should cause no change in behavior, but is prep work for other improvements.
* Do not attempt to bundled aggregations for /members and /state. (#11623)Patrick Cloke2021-12-291-3/+1
| | | | Both of those APIs return state events, which will not have bundled aggregations added anyway.
* Do not bundle aggregations for APIs which shouldn't include them. (#11592)Patrick Cloke2021-12-201-1/+3
| | | | | And make bundling aggregations opt-in, instead of opt-out to avoid having APIs to include extraneous data (and being much heavier than necessary).
* Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-141-7/+6
|
* Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-101-6/+18
| | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* Include bundled aggregations in /sync and related fixes (#11478)Patrick Cloke2021-12-061-7/+1
| | | | | | | | Due to updates to MSC2675 this includes a few fixes: * Include bundled aggregations for /sync. * Do not include bundled aggregations for /initialSync and /events. * Do not bundle aggregations for state events. * Clarifies comments and variable names.
* Refactor the code to inject bundled relations during serialization. (#11408)Patrick Cloke2021-11-231-1/+1
|
* Do not allow MSC3440 threads to fork threads (#11161)Patrick Cloke2021-11-181-6/+48
| | | | | | | | | | | Adds validation to the Client-Server API to ensure that the potential thread head does not relate to another event already. This results in not allowing a thread to "fork" into other threads. If the target event is unknown for some reason (maybe it isn't visible to your homeserver), but is the target of other events it is assumed that the thread can be created from it. Otherwise, it is rejected as an unknown event.
* Rename to more clear `get_insertion_event_id_by_batch_id` (MSC2716) (#11244)Eric Eastwood2021-11-081-1/+1
| | | | | `get_insertion_event_by_batch_id` -> `get_insertion_event_id_by_batch_id` Split out from https://github.com/matrix-org/synapse/pull/11114
* Add remaining type hints to `synapse.events`. (#11098)Patrick Cloke2021-11-021-4/+10
|
* Implement an `on_new_event` callback (#11126)Brendan Abolivier2021-10-261-3/+6
| | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Resolve and share `state_groups` for all historical events in batch ↵Eric Eastwood2021-10-131-23/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#10975) Resolve and share `state_groups` for all historical events in batch. This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch. This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room. For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch. --- How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example: **Before** (new `state_group` for every event 😬, very inefficient): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10 create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11 create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12 create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13 create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14 ``` **After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10 create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10 create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10 create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10 create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10 ```
* Remove the deprecated BaseHandler. (#11005)Patrick Cloke2021-10-081-5/+2
| | | | | | | | The shared ratelimit function was replaced with a dedicated RequestRatelimiter class (accessible from the HomeServer object). Other properties were copied to each sub-class that inherited from BaseHandler.
* Merge tag 'v1.44.0rc2' into developOlivier Wilkinson (reivilibre)2021-09-301-9/+4
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc2 (2021-09-30) ============================== Bugfixes -------- - Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error. ([\#10938](https://github.com/matrix-org/synapse/issues/10938)) - Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944)) Improved Documentation ---------------------- - Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))
| * Fix event context for outliers in important MSC2716 spot (#10938)Eric Eastwood2021-09-301-9/+4
| | | | | | | | | | | | | | | | | | Fix event context for outlier causing failures in all of the MSC2716 Complement tests. The `EventContext.for_outlier` refactor happened in https://github.com/matrix-org/synapse/pull/10883 and this spot was left out.
* | Split `event_auth.check` into two parts (#10940)Richard van der Hoff2021-09-291-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | Broadly, the existing `event_auth.check` function has two parts: * a validation section: checks that the event isn't too big, that it has the rught signatures, etc. This bit is independent of the rest of the state in the room, and so need only be done once for each event. * an auth section: ensures that the event is allowed, given the rest of the state in the room. This gets done multiple times, against various sets of room state, because it forms part of the state res algorithm. Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think that makes everything hard to follow. Instead, we split the function in two and call each part separately where it is needed.
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-6/+8
| |
* | Use `RoomVersion` objects (#10934)Richard van der Hoff2021-09-291-6/+21
| | | | | | Various refactors to use `RoomVersion` objects instead of room version identifiers.
* | Ensure `(room_id, next_batch_id)` is unique to avoid cross-talk/conflicts ↵Eric Eastwood2021-09-281-0/+34
|/ | | | | | | | between batches (MSC2716) (#10877) Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) Part of https://github.com/matrix-org/synapse/issues/10737
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-241-4/+4
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-231-2/+2
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-231-2/+2
|
* Rename MSC2716 things from `chunk` to `batch` to match `/batch_send` ↵Eric Eastwood2021-09-211-1/+1
| | | | | | | | endpoint (#10838) See https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684574497 Dropping support for older MSC2716 room versions so we don't have to worry about supporting both chunk and batch events.
* Require type hints in the handlers module. (#10831)Patrick Cloke2021-09-201-8/+12
| | | | | | | Adds missing type hints to methods in the synapse.handlers module and requires all methods to have type hints there. This also removes the unused construct_auth_difference method from the FederationHandler.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-2/+2
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Move `maybe_kick_guest_users` out of `BaseHandler` (#10744)Richard van der Hoff2021-09-061-2/+25
| | | This is part of my ongoing war against BaseHandler. I've moved kick_guest_users into RoomMemberHandler (since it calls out to that handler anyway), and split maybe_kick_guest_users into the two places it is called.
* Allow room creator to send MSC2716 related events in existing room versions ↵Eric Eastwood2021-09-041-3/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#10566) * Allow room creator to send MSC2716 related events in existing room versions Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869 Restoring `get_create_event_for_room_txn` from, https://github.com/matrix-org/synapse/pull/10245/commits/44bb3f0cf5cb365ef9281554daceeecfb17cc94d * Add changelog * Stop people from trying to redact MSC2716 events in unsupported room versions * Populate rooms.creator column for easy lookup > From some [out of band discussion](https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`). > > https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642 * Remove and switch away from get_create_event_for_room_txn * Fix no create event being found because no state events persisted yet * Fix and add tests for rooms creator bg update * Populate rooms.creator field for easy lookup Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values * Add changelog * Fix usage * Remove extra delta already included in #10697 * Don't worry about setting creator for invite * Only iterate over rows missing the creator See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898 * Use constant to fetch room creator field See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029 * More protection from other random types See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853 * Move new background update to end of list See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181 * Fix query casing * Fix ambiguity iterating over cursor instead of list Fix `psycopg2.ProgrammingError: no results to fetch` error when tests run with Postgres. ``` SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room ``` --- We use `txn.fetchall` because it will return the results as a list or an empty list when there are no results. Docs: > `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used: > > https://www.psycopg.org/docs/cursor.html#cursor-iterable And I'm guessing iterating over a raw cursor does something weird when there are no results. --- Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530 ``` tests.test_visibility.FilterEventsForServerTestCase.test_large_room =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column self.get_success( File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success return self.successResultOf(d) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf self.fail( twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead: Traceback (most recent call last): File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback self._startRunCallbacks(fail) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks self._runCallbacks() File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks current.result = callback( # type: ignore[misc] File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult current_context.run(_inlineCallbacks, r, gen, status) --- <exception caught here> --- File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks result = current_context.run( File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update await self._do_background_update(desired_duration_ms) File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update items_updated = await update_handler(progress, batch_size) File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column end = await self.db_pool.runInteraction( File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction result = await self.runWithConnection( File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection return await make_deferred_yieldable( File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks current.result = callback( # type: ignore[misc] File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda> d.addCallback(lambda x: function(*args, **kwargs)) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise raise exception.with_traceback(traceback) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func return func(db_conn, *args, **kwargs) File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction r = func(cursor, *args, **kwargs) File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn for room_id, event_json in txn: psycopg2.ProgrammingError: no results to fetch ``` * Move code not under the MSC2716 room version underneath an experimental config option See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909 * Add ordering to rooms creator background update See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277 * Add comment to better document constant See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458 * Use constant field
* Merge pull request from GHSA-3x4c-pq33-4w3qreivilibre2021-08-311-3/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add some tests to characterise the problem Some failing. Current states: RoomsMemberListTestCase test_get_member_list ... [OK] test_get_member_list_mixed_memberships ... [OK] test_get_member_list_no_permission ... [OK] test_get_member_list_no_permission_former_member ... [OK] test_get_member_list_no_permission_former_member_with_at_token ... [FAIL] test_get_member_list_no_room ... [OK] test_get_member_list_no_permission_with_at_token ... [FAIL] * Correct the tests * Check user is/was member before divulging room membership * Pull out only the 1 membership event we want. * Update tests/rest/client/v1/test_rooms.py Co-authored-by: Erik Johnston <erik@matrix.org> * Fixup tests (following apply review suggestion) Co-authored-by: Erik Johnston <erik@matrix.org>
* Port the ThirdPartyEventRules module interface to the new generic interface ↵Brendan Abolivier2021-07-201-4/+4
| | | | | (#10386) Port the third-party event rules interface to the generic module interface introduced in v1.37.0
* Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-161-11/+7
|
* Add base starting insertion event when no chunk ID is provided (MSC2716) ↵Eric Eastwood2021-07-081-0/+8
| | | | | | | | | (#10250) * Add base starting insertion point when no chunk ID is provided This is so we can have the marker event point to this initial insertion event and be able to traverse the events in the first chunk.
* Rebuild event auth when rebuilding an event after a call to a ↵Brendan Abolivier2021-07-081-4/+6
| | | | | `ThirdPartyEventRules` module (#10316) Because modules might send extra state events when processing an event (e.g. matrix-org/synapse-dinsic#100), and in some cases these extra events might get dropped if we don't recalculate the initial event's auth.
* Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-011-3/+6
| | | Instead of mixing them with user authentication methods.
* Correct type hints for synapse.event_auth. (#10253)Patrick Cloke2021-06-301-0/+7
|
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-221-5/+99
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Implement knock feature (#6739)Sorunome2021-06-091-10/+20
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Run cache_joined_hosts_for_event in background (#9951)Erik Johnston2021-05-121-5/+40
|
* Always cache 'event_to_prev_state_group' (#9950)Erik Johnston2021-05-071-6/+7
| | | Fixes regression in send PDU times introduced in #9905.
* Don't set the external cache if its been done recently (#9905)Erik Johnston2021-05-051-4/+30
|
* Use get_current_users_in_room from store and not StateHandler (#9910)Erik Johnston2021-05-051-1/+1
|
* Add missing type hints to handlers and fix a Spam Checker type hint. (#9896)Patrick Cloke2021-04-291-7/+17
| | | | | The user_may_create_room_alias method on spam checkers declared the room_alias parameter as a str when in reality it is passed a RoomAlias object.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-3/+8
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff2021-03-301-1/+1
| | | | | | | `room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
* Add configs to make profile data more private (#9203)AndrewFerr2021-02-191-1/+7
| | | | | | | Add off-by-default configuration settings to: - disable putting an invitee's profile info in invite events - disable profile lookup via federation Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-11/+23
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add an admin API to get the current room state (#9168)Travis Ralston2021-02-021-1/+1
| | | | | This could arguably replace the existing admin API for `/members`, however that is out of scope of this change. This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
* Precompute joined hosts and store in Redis (#9198)Erik Johnston2021-01-261-0/+42
|
* Try and drop stale extremities. (#8929)Erik Johnston2020-12-181-1/+1
| | | | If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-111-1/+1
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Add admin API for logging in as a user (#8617)Erik Johnston2020-11-171-11/+10
|
* Block clients from sending server ACLs that lock the local server out. (#8708)Erik Johnston2020-11-031-0/+3
| | | Fixes #4042
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-281-3/+2
|\ | | | | Fix serialisation errors when using third-party event rules.
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-281-3/+2
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Abstract code for stripping room state into a separate method (#8671)Andrew Morgan2020-10-271-28/+7
| | | | | | | | | | | | | | | | | | | | | | This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change. For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included. The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event. This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem. In essence this is just moving some inline code to a reusable store method.
* | Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-231-1/+1
|/
* Fix modifying events in `ThirdPartyRules` modules (#8564)Richard van der Hoff2020-10-161-1/+6
| | | EventBuilder.build wants auth events these days
* Merge pull request #8535 from matrix-org/rav/third_party_events_updatesRichard van der Hoff2020-10-151-8/+71
|\ | | | | Support modifying event content from ThirdPartyRules modules
| * Allow ThirdPartyRules modules to replace event contentRichard van der Hoff2020-10-131-2/+62
| | | | | | | | Support returning a new event dict from `check_event_allowed`.
| * Move third_party_rules check to event creation timeRichard van der Hoff2020-10-131-8/+11
| | | | | | | | | | Rather than waiting until we handle the event, call the ThirdPartyRules check when we fist create the event.
* | Simplify `_locally_reject_invite`Richard van der Hoff2020-10-131-2/+20
| | | | | | | | | | Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.
* | Remove redundant `token_id` parameter to create_eventRichard van der Hoff2020-10-131-5/+3
|/ | | | this is always the same as requester.access_token_id.
* Move additional tasks to the background worker, part 4 (#8513)Patrick Cloke2020-10-131-9/+9
|
* Fix message duplication if something goes wrong after persisting the event ↵Erik Johnston2020-10-131-8/+40
| | | | | (#8476) Should fix #3365.
* Allow modules to create and send events into rooms (#8479)Andrew Morgan2020-10-091-6/+5
| | | | | This PR allows Synapse modules making use of the `ModuleApi` to create and send non-membership events into a room. This can useful to have modules send messages, or change power levels in a room etc. Note that they must send event through a user that's already in the room. The non-membership event limitation is currently arbitrary, as it's another chunk of work and not necessary at the moment.
* Remove the deprecated Handlers object (#8494)Patrick Cloke2020-10-091-2/+2
| | | All handlers now available via get_*_handler() methods on the HomeServer.
* update wordingRichard van der Hoff2020-10-071-2/+3
|
* kill off `send_nonmember_event`Richard van der Hoff2020-10-051-49/+25
| | | | This is now redundant, and we can just call `handle_new_client_event` directly.
* pull up event.sender assertionRichard van der Hoff2020-10-051-4/+4
|
* Move shadow-ban check down into `handle_new_client_event`.Richard van der Hoff2020-10-051-8/+24
|
* De-duplicate duplicate handlingRichard van der Hoff2020-10-051-18/+23
| | | | | move the "duplicate state event" handling down into `handle_new_client_event` where it can be shared between multiple call paths.
* Remove stream ordering from Metadata dict (#8452)Richard van der Hoff2020-10-051-1/+3
| | | | | | | | There's no need for it to be in the dict as well as the events table. Instead, we store it in a separate attribute in the EventInternalMetadata object, and populate that on load. This means that we can rely on it being correctly populated for any event which has been persited to the database.
* Add EventStreamPosition type (#8388)Erik Johnston2020-09-241-3/+3
| | | | | | | | | | | | | | The idea is to remove some of the places we pass around `int`, where it can represent one of two things: 1. the position of an event in the stream; or 2. a token that partitions the stream, used as part of the stream tokens. The valid operations are then: 1. did a position happen before or after a token; 2. get all events that happened before or after a token; and 3. get all events between two tokens. (Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
* Factor out `_send_dummy_event_for_room` (#8370)Richard van der Hoff2020-09-231-48/+54
| | | this makes it possible to use from the manhole, and seems cleaner anyway.
* Add experimental support for sharding event persister. Again. (#8294)Erik Johnston2020-09-141-6/+8
| | | | | | This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
* Make `StreamToken.room_key` be a `RoomStreamToken` instance. (#8281)Erik Johnston2020-09-111-0/+1
|
* Clean up `Notifier.on_new_room_event` code path (#8288)Erik Johnston2020-09-101-4/+0
| | | | | | | | | | | | | The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it. This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens. The valid operations here are: 1. Is a position before or after a token 2. Fetching all events between two tokens 3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C. Future PR will change the token type to a dedicated type.
* Fixup pusher pool notifications (#8287)Erik Johnston2020-09-091-1/+1
| | | | | `pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed. I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
* Revert "Fixup pusher pool notifications"Erik Johnston2020-09-091-1/+1
| | | | This reverts commit e7fd336a53a4ca489cdafc389b494d5477019dc0.
* Fixup pusher pool notificationsErik Johnston2020-09-091-1/+1
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-2/+2
|
* Revert "Add experimental support for sharding event persister. (#8170)" (#8242)Brendan Abolivier2020-09-041-8/+6
| | | | | | | * Revert "Add experimental support for sharding event persister. (#8170)" This reverts commit 82c1ee1c22a87b9e6e3179947014b0f11c0a1ac3. * Changelog
* Add experimental support for sharding event persister. (#8170)Erik Johnston2020-09-021-6/+8
| | | | | | This is *not* ready for production yet. Caveats: 1. We should write some tests... 2. The stream token that we use for events can get stalled at the minimum position of all writers. This means that new events may not be processed and e.g. sent down sync streams if a writer isn't writing or is slow.
* Convert additional databases to async/await part 2 (#8200)Patrick Cloke2020-09-011-10/+3
|
* Remove unused parameter from, and add safeguard in, get_room_data (#8174)Andrew Morgan2020-08-261-8/+12
| | | | | | Small cleanup PR. * Removed the unused `is_guest` argument * Added a safeguard to a (currently) impossible code path, fixing static checking at the same time.
* Do not allow send_nonmember_event to be called with shadow-banned users. (#8158)Patrick Cloke2020-08-251-5/+34
|
* Stop shadow-banned users from sending non-member events. (#8142)Patrick Cloke2020-08-241-0/+10
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-191-2/+3
|
* Convert events worker database to async/await. (#8071)Patrick Cloke2020-08-181-3/+3
|
* Fix the return type of send_nonmember_events. (#8112)Patrick Cloke2020-08-181-1/+1
|
* Return the previous stream token if a non-member event is a duplicate. (#8093)Patrick Cloke2020-08-181-10/+15
|
* Run `remove_push_actions_from_staging` in foreground (#8081)Richard van der Hoff2020-08-131-3/+1
| | | | | | | If we got an error persisting an event, we would try to remove the push actions asynchronously, which would lead to a 'Re-starting finished log context' warning. I don't think there's any need for this to be asynchronous.
* Add type hints to handlers.message and events.builder (#8067)Erik Johnston2020-08-121-9/+13
|
* Add an assertion on prev_events in create_new_client_event (#8041)Richard van der Hoff2020-08-101-0/+9
| | | | | | I think this would have caught all the cases in https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes more sense here than a 403
* Convert synapse.api to async/await (#8031)Patrick Cloke2020-08-061-1/+1
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-1/+1
|
* Remove hacky error handling for inlineDeferreds. (#7950)Patrick Cloke2020-07-271-12/+7
|
* Convert the message handler to async/await. (#7884)Patrick Cloke2020-07-221-136/+152
|
* Convert _base, profile, and _receipts handlers to async/await (#7860)Patrick Cloke2020-07-171-2/+6
|
* Generate real events when we reject invites (#7804)Richard van der Hoff2020-07-091-7/+16
| | | | | | | | Fixes #2181. The basic premise is that, when we fail to reject an invite via the remote server, we can generate our own out-of-band leave event and persist it as an outlier, so that we have something to send to the client.
* Convert directory handler to async/await (#7727)Patrick Cloke2020-06-221-1/+3
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-3/+1
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-4/+4
|
* Performance improvements and refactor of Ratelimiter (#7595)Andrew Morgan2020-06-051-1/+0
| | | | | | | | | | While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both: * Rather undocumented, and * causing a *lot* of config checks This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. Best to be reviewed commit-by-commit.
* Optimise some references to hs.config (#7546)Richard van der Hoff2020-05-221-3/+5
| | | These are surprisingly expensive, and we only really need to do them at startup.
* Add option to move event persistence off master (#7517)Erik Johnston2020-05-221-5/+7
|
* Add ability to wait for replication streams (#7542)Erik Johnston2020-05-221-11/+25
| | | | | | | The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room). Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on. People probably want to look at this commit by commit.
* Omit displayname or avatar_url if they aren't set instead of returning null ↵Aaron Raimist2020-05-191-2/+6
| | | | | | | | | (#7497) Per https://github.com/matrix-org/matrix-doc/issues/1436#issuecomment-410089470 they should be omitted instead of returning null or "". They aren't marked as required in the spec. Fixes https://github.com/matrix-org/synapse/issues/7333 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Allow censoring of events to happen on workers. (#7492)Erik Johnston2020-05-131-2/+0
| | | This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
* Add a configuration setting for the dummy event threshold (#7422)Brendan Abolivier2020-05-071-1/+3
| | | Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.
* async/await is_server_admin (#7363)Andrew Morgan2020-05-011-44/+39
|
* Use the proper error code when a canonical alias that does not exist is ↵Patrick Cloke2020-03-231-19/+38
| | | | used. (#7109)
* Don't filter out dummy events when we're checking the visibility of stateBrendan Abolivier2020-03-111-1/+1
|
* Validate the alt_aliases property of canonical alias events (#6971)Patrick Cloke2020-03-031-3/+44
|
* Port PresenceHandler to async/await (#6991)Erik Johnston2020-02-261-3/+2
|
* Add `allow_departed_users` param to `check_in_room_or_world_readable`Richard van der Hoff2020-02-191-4/+8
| | | | | | | ... and set it everywhere it's called. while we're here, rename it for consistency with `check_user_in_room` (and to help check that I haven't missed any instances)
* make FederationHandler.send_invite asyncRichard van der Hoff2020-02-031-3/+2
|
* s/get_room_version/get_room_version_id/Richard van der Hoff2020-01-311-3/+5
| | | | | ... to make way for a forthcoming get_room_version which returns a RoomVersion object.
* Pass room version object into event_auth.check and check_redaction (#6788)Richard van der Hoff2020-01-281-2/+6
| | | | | | | These are easier to work with than the strings and we normally have one around. This fixes `FederationHander._persist_auth_tree` which was passing a RoomVersion object into event_auth.check instead of a string.
* Remove unused hashes and depths from create_event paramsRichard van der Hoff2020-01-061-16/+5
|
* Remove unused hashes and depths from create_new_client_event paramsRichard van der Hoff2020-01-061-12/+14
|
* replace get_prev_events_and_hashes_for_room with get_prev_events_for_room in ↵Richard van der Hoff2020-01-061-9/+3
| | | | create_new_client_event
* rename get_prev_events_for_room to get_prev_events_and_hashes_for_roomRichard van der Hoff2020-01-061-2/+4
| | | | ... to make way for a new method which just returns the event ids
* Change EventContext to use the Storage class (#6564)Erik Johnston2019-12-201-5/+5
|
* Prevent redacted events from appearing in message search (#6377)Andrew Morgan2019-12-111-2/+3
|
* Pull out room_invite_state_types config option once.Erik Johnston2019-12-091-1/+3
| | | | Pulling things out of config is currently surprisingly expensive.
* Add ephemeral messages support (MSC2228) (#6409)Brendan Abolivier2019-12-031-1/+122
| | | | | | | | Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are: * the feature is hidden behind a configuration flag (`enable_ephemeral_messages`) * self-destruction doesn't happen for state events * only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one) * doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
* Discard retention policies when retrieving stateBrendan Abolivier2019-11-281-1/+1
| | | | | | | | Purge jobs don't delete the latest event in a room in order to keep the forward extremity and not break the room. On the other hand, get_state_events, when given an at_token argument calls filter_events_for_client to know if the user can see the event that matches that (sync) token. That function uses the retention policies of the events it's given to filter out those that are too old from a client's view. Some clients, such as Riot, when loading a room, request the list of members for the latest sync token it knows about, and get confused to the point of refusing to send any message if the server tells it that it can't get that information. This can happen very easily with the message retention feature turned on and a room with low activity so that the last event sent becomes too old according to the room's retention policy. An easy and clean fix for that issue is to discard the room's retention policies when retrieving state.
* Implement per-room message retention policiesBrendan Abolivier2019-11-041-2/+2
|
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-6/+8
| | | * update version of black and also fix the mypy config being overridden
* Merge pull request #6294 from matrix-org/erikj/add_state_storageErik Johnston2019-10-311-4/+6
|\ | | | | Add StateGroupStorage interface
| * Port to use state storageErik Johnston2019-10-301-4/+6
| |
* | Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-1/+1
|/ | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Use new EventPersistenceStoreErik Johnston2019-10-231-1/+2
|
* Fix dummy event insertion consent bug (#6053)Neil Johnson2019-09-261-27/+72
| | | Fixes #5905
* Fix commentsErik Johnston2019-09-111-1/+1
| | | Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fix how we check for self redactionErik Johnston2019-09-111-4/+18
|
* Allow use of different ratelimits for admin redactions.Erik Johnston2019-09-111-1/+7
| | | | | This is useful to allow room admins to quickly deal with a large number of abusive messages.
* Apply suggestions from code reviewWill Hunt2019-08-231-2/+2
| | | Co-Authored-By: Erik Johnston <erik@matrix.org>
* Exempt bot usersHalf-Shot2019-08-231-1/+1
|
* Return user_type in get_user_by_idHalf-Shot2019-08-231-1/+4
|
* Deny redaction of events in a different room.Erik Johnston2019-07-311-1/+3
| | | | | We already correctly filter out such redactions, but we should also deny them over the CS API.
* Stop trying to fetch events with event_id=None. (#5753)Richard van der Hoff2019-07-241-1/+7
| | | | | | | `None` is not a valid event id, so queuing up a database fetch for it seems like a silly thing to do. I considered making `get_event` return `None` if `event_id is None`, but then its interaction with `allow_none` seemed uninituitive, and strong typing ftw.
* Replace returnValue with return (#5736)Amber Brown2019-07-231-17/+15
|
* Ignore redactions of m.room.create events (#5701)Richard van der Hoff2019-07-171-9/+24
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Run Black. (#5482)Amber Brown2019-06-201-158/+100
|
* Run as background process and fix commentsErik Johnston2019-06-191-2/+5
|
* Add experimental option to reduce extremities.Erik Johnston2019-06-181-1/+71
| | | | | | | Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.
* Add plugin APIs for implementations of custom event rules.Brendan Abolivier2019-06-141-2/+12
|
* Merge pull request #5220 from matrix-org/erikj/dont_bundle_live_eventsErik Johnston2019-05-241-0/+3
|\ | | | | Don't bundle aggregations with events in /sync or /events or state queries
| * Don't bundle aggs for /state and /members etc APIsErik Johnston2019-05-241-0/+3
| |
* | Fix wordsErik Johnston2019-05-211-3/+3
| |
* | Block attempts to annotate the same event twiceErik Johnston2019-05-201-1/+15
|/
* Allow client event serialization to be asyncErik Johnston2019-05-141-3/+4
|
* Do checks on aliases for incoming m.room.aliases events (#5128)Brendan Abolivier2019-05-081-0/+30
| | | | | Follow-up to #5124 Also added a bunch of checks to make sure everything (both the stuff added on #5124 and this PR) works as intended.
* Fix grammar and document get_current_users_in_room (#4998)Andrew Morgan2019-04-031-1/+1
|
* Collect room-version variations into one place (#4969)Richard van der Hoff2019-04-011-2/+5
| | | | Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.
* Revert spurious deleteErik Johnston2019-03-201-0/+4
|
* Use flagsErik Johnston2019-03-201-13/+5
|
* Move requester check into assert_accepted_privacy_policyErik Johnston2019-03-201-5/+8
|
* Only require consent for events with an associated requestErik Johnston2019-03-201-1/+5
| | | | | | | | | There are a number of instances where a server or admin may puppet a user to join/leave rooms, which we don't want to fail if the user has not consented to the privacy policy. We fix this by adding a check to test if the requester has an associated access_token, which is used as a proxy to answer the question of whether the action is being done on behalf of a real request from the user.
* fix test_auto_create_auto_join_where_no_consent (#4886)Neil Johnson2019-03-191-3/+10
|
* Fix incorrect log about not persisting duplicate state event. (#4776)Richard van der Hoff2019-03-011-3/+4
| | | We were logging this when it was not true.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-01-291-28/+6
|\ | | | | | | erikj/redactions_eiah
| * Correctly set context.app_serviceErik Johnston2019-01-291-0/+2
| |
| * Refactor event building into EventBuilderErik Johnston2019-01-291-29/+5
| | | | | | | | | | This is so that everything is done in one place, making it easier to change the event format based on room version
* | Implement rechecking of redactionsErik Johnston2019-01-291-1/+5
|/
* Merge pull request #4494 from matrix-org/erikj/fixup_event_validatorErik Johnston2019-01-291-1/+3
|\ | | | | Split up event validation between event and builder
| * Split up event validation between event and builderErik Johnston2019-01-281-1/+3
| | | | | | | | | | | | | | | | | | The validator was being run on the EventBuilder objects, and so the validator only checked a subset of fields. With the upcoming EventBuilder refactor even fewer fields will be there to validate. To get around this we split the validation into those that can be run against an EventBuilder and those run against a fully fledged event.
* | Correctly use default room version if none is setErik Johnston2019-01-281-2/+2
| |
* | Pass through room version to event authErik Johnston2019-01-251-1/+6
|/
* Require event format version to parse or create eventsErik Johnston2019-01-251-1/+9
|
* Revert "Require event format version to parse or create events"Erik Johnston2019-01-251-9/+1
|
* Require event format version to parse or create eventsErik Johnston2019-01-231-1/+9
|
* Log some bits about event creation (#4121)Richard van der Hoff2018-10-311-0/+3
| | | | I found these helpful in debugging my room upgrade tests.
* Refactor state group lookup to reduce DB hits (#4011)Erik Johnston2018-10-251-12/+8
| | | | | | | | Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.
* Remove redundant run_as_background_process() from pusherpoolRichard van der Hoff2018-10-221-1/+1
| | | | | | | `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.
* Avoid reraise, to improve stacktracesRichard van der Hoff2018-10-011-12/+13
|
* Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updatesRichard van der Hoff2018-08-171-5/+2
|\ | | | | Fix logcontexts for running pushers
| * Fix logcontexts for running pushersRichard van der Hoff2018-08-171-5/+2
| | | | | | | | | | | | | | First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.
* | block event creation and room creation on hitting resource limitsNeil Johnson2018-08-161-1/+5
|/
* speed up /members and add at= and membership params (#3568)Matthew Hodgson2018-08-151-12/+76
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
|
* Use new helper base class for ReplicationSendEventRestServletErik Johnston2018-07-311-7/+4
|
* Merge branch 'develop' into erikj/client_apis_moveErik Johnston2018-07-241-12/+19
|\
| * Fix missing attributes on workers.Erik Johnston2018-07-231-2/+3
| | | | | | | | | | This was missed during the transition from attribute to getter for getting state from context.
| * Use new gettersErik Johnston2018-07-231-10/+16
| |
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-07-231-2/+2
|\| | | | | | | erikj/client_apis_move
| * Combine Limiter and LinearizerRichard van der Hoff2018-07-201-2/+2
| | | | | | | | | | Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.
| * Improvements to the LimiterRichard van der Hoff2018-07-201-1/+1
| | | | | | | | | | * give them names, to improve logging * use a deque rather than a list for efficiency
* | Move PaginationHandler to its own fileErik Johnston2018-07-201-240/+2
| |
* | Move check_in_room_or_world_readable to AuthErik Johnston2018-07-201-34/+6
| |
* | Split MessageHandler into read only and writersErik Johnston2018-07-181-131/+150
|/ | | | | This will let us call the read only parts from workers, and so be able to move some APIs off of master, e.g. the `/state` API.
* s/becuase/because/gMatthew Hodgson2018-07-101-1/+1
|
* run isortAmber Brown2018-07-091-13/+10
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-3/+2
|
* allow non-consented users to still part rooms (to let us autopart them)Matthew Hodgson2018-06-251-2/+6
|
* Remove all global reactor imports & pass it around explicitly (#3424)Amber Brown2018-06-251-2/+2
|
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-221-0/+1
|
* Remove run_on_reactor (#3395)Amber Brown2018-06-141-3/+1
|
* Exempt AS-registered users from doing gdprRichard van der Hoff2018-05-291-0/+3
|
* Misc. py3 fixesAdrian Tschira2018-05-241-4/+5
| | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uriRichard van der Hoff2018-05-231-1/+4
|\
| * Support for putting %(consent_uri)s in messagesRichard van der Hoff2018-05-231-1/+4
| | | | | | | | | | Make it possible to put the URI in the error message and the server notice that get sent by the server
* | Use the localpart in the consent uriRichard van der Hoff2018-05-231-1/+3
|/ | | | ... because it's shorter.
* Reject attempts to send event before privacy consent is givenRichard van der Hoff2018-05-221-1/+85
| | | | | | Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.