summary refs log tree commit diff
path: root/synapse/handlers (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix performance of device lists in `/key/changes` and sliding sync (#17537) HEAD developErik Johnston2024-08-092-205/+209
| | | | | | We do this by reusing the code from sync v2. Reviewable commit-by-commit. The function `get_user_ids_changed` has been rewritten entirely, so I would recommend not looking at the diff.
* SSS: Implement PREVIOUSLY room tracking (#17535)Erik Johnston2024-08-081-16/+52
| | | | | | Implement tracking of rooms that have had updates that have not been sent down to clients. Simplified Sliding Sync (SSS)
* Sliding Sync: Use `stream_ordering` based timeline pagination for ↵Eric Eastwood2024-08-075-38/+115
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | incremental sync (#17510) Use `stream_ordering` based `timeline` pagination for incremental `/sync` in Sliding Sync. Previously, we were always using a `topological_ordering` but we should only be using that for historical scenarios (initial `/sync`, newly joined, or haven't sent the room down the connection before). This is slightly different than what the [spec suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing) > Events are ordered in this API according to the arrival time of the event on the homeserver. This can conflict with other APIs which order events based on their partial ordering in the event graph. This can result in duplicate events being received (once per distinct API called). Clients SHOULD de-duplicate events based on the event ID when this happens. But we've had a [discussion below in this PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569) and this matches what Sync v2 already does and seems like it makes sense. Created a spec issue https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this. Related issues: - https://github.com/matrix-org/matrix-spec/issues/1917 - https://github.com/matrix-org/matrix-spec/issues/852 - https://github.com/matrix-org/matrix-spec-proposals/pull/4033
* Sliding Sync: Add more tracing (#17514)Eric Eastwood2024-08-061-239/+327
| | | | | | | | | | | | Spawning from looking at a couple traces and wanting a little more info. Follow-up to github.com/element-hq/synapse/pull/17501 The changes in this PR allow you to find slow Sliding Sync traces ignoring the `wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user` operation with `RESULT.result=true` indicating that it actually returned non-empty results. If you want to find traces for your own user, you can use `RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`
* SS: Reset connection if token is unrecognized (#17529)Erik Johnston2024-08-061-0/+27
| | | | | | | | | | | | | | This triggers the client to start a new sliding sync connection. If we don't do this and the client asks for the full range of rooms, we end up sending down all rooms and their state from scratch (which can be very slow) This causes things like https://github.com/element-hq/element-x-ios/issues/3115 after we restart the server --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Add typing notification extension (MSC3961) (#17505)Eric Eastwood2024-07-313-1/+93
| | | | | | | [MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Sliding Sync: Update filters to be robust against remote invite rooms (#17450)Eric Eastwood2024-07-302-38/+351
| | | | | | | | | | | | | | | Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to be robust when dealing with remote invite rooms in Sliding Sync. Part of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Follow-up to https://github.com/element-hq/synapse/pull/17434 We now take into account current state, fallback to stripped state for invite/knock rooms, then historical state. If we can't determine the info needed to filter a room (either from state or stripped state), it is filtered out.
* Sliding Sync: Add receipts extension (MSC3960) (#17489)Eric Eastwood2024-07-302-61/+211
| | | | | | | [MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add some more opentracing to sliding sync (#17501)Erik Johnston2024-07-301-3/+15
| | | This will make it easier to see what it is doing in jaeger.
* Fix `failures` property in `/keys/query` (#17499)Richard van der Hoff2024-07-301-8/+18
| | | | Fixes: https://github.com/element-hq/synapse/issues/17498 Fixes: https://github.com/element-hq/element-web/issues/27867
* Only send rooms with updates down sliding sync (#17479)Erik Johnston2024-07-301-2/+49
| | | | | | | | | | | | Rather than always including all rooms in range. Also adds a pre-filter to rooms that checks the stream change cache to see if anything might have happened. Based on #17447 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Track whether we have sent rooms down to clients (#17447)Erik Johnston2024-07-291-36/+314
| | | | | | | | | | | | | | | | The basic idea is that we introduce a new token for a sliding sync connection, which stores the mapping of room to room "status" (i.e. have we sent the room down?). This token allows us to handle duplicate requests properly. In future it can be used to store more "per-connection" information safely. In future this should be migrated into the DB, so its important that we try to reduce the number of syncs where we need to update the per-connection information. In this PoC this only happens when we: a) send down a set of room for the first time, or b) we have previously sent down a room and there are updates but we are not sending the room down the sync (due to not falling in a list range) Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Add Account Data extension (MSC3959) (#17477)Eric Eastwood2024-07-241-0/+138
| | | | | | Extensions based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Sliding Sync: ensure bump stamp ignores backfilled events (#17478)Erik Johnston2024-07-241-2/+8
| | | | | | Backfill events have a negative stream ordering, and so its not useful to use to compare with other (positive) stream orderings. Plus, the Rust SDK currently assumes `bump_stamp` is positive.
* Use a new token format for sliding sync (#17452)Erik Johnston2024-07-241-12/+18
| | | | | | | This is in preparation for adding per-connection state. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Speed up SS room sorting (#17468)Erik Johnston2024-07-231-22/+21
| | | | | | | We do this by bulk fetching the latest stream ordering. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fixed presence results not returning offline users on initial sync (#17231)Michael Hollister2024-07-231-1/+5
| | | | | | | | | | | | | | | | This is to address an issue in which `m.presence` results on initial sync are not returning entries of users who are currently offline. The original behaviour was from https://github.com/element-hq/synapse/issues/1535 This change is useful for applications that use the presence system for tracking user profile information/updates (e.g. https://github.com/element-hq/synapse/pull/16992 or for profile status messages). This is gated behind a new configuration option to avoid performance impact for applications that don't need this, as a pragmatic solution for now.
* Sliding Sync: Add E2EE extension (MSC3884) (#17454)Eric Eastwood2024-07-222-19/+105
| | | | | Spec: [MSC3884](https://github.com/matrix-org/matrix-spec-proposals/pull/3884) Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* SS: Implement `$ME` support (#17469)Erik Johnston2024-07-221-1/+5
| | | `$ME` can be used as a substitute for the requester's user ID.
* Speed up room keys query by using read/write lock (#17461)Erik Johnston2024-07-221-9/+9
| | | | Linaerizing all access slows things down when devices try and fetch lots of keys on login
* Generate room sync data concurrently (#17458)Erik Johnston2024-07-191-2/+10
| | | This is also what we do for standard `/sync`.
* Improve default_power_level_content_override documentation (#17451)Ben Banfield-Zanin2024-07-181-0/+2
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Add `m.room.create` to default bump event types (#17453)Eric Eastwood2024-07-181-0/+1
| | | | | | Add `m.room.create` to default bump event types This probably helps when no messages have been sent in the room and it was just created.
* Fix bug where sync could get stuck when using workers (#17438)Erik Johnston2024-07-151-2/+9
| | | | This is because we serialized the token wrong if the instance map contained entries from before the minimum token.
* Add room subscriptions to Sliding Sync `/sync` (#17432)Eric Eastwood2024-07-151-115/+288
| | | | | | | | | | | | | Add room subscriptions to Sliding Sync `/sync` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Currently, you can only subscribe to rooms you have had *any* membership in before. In the future, we will allow `world_readable` rooms to be subscribed to without joining.
* Add `is_dm` room field to Sliding Sync `/sync` (#17429)Eric Eastwood2024-07-111-29/+46
| | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `heroes` and room summary fields to Sliding Sync `/sync` (#17419)Eric Eastwood2024-07-111-83/+197
| | | | | | | Additional room summary fields: `joined_count`, `invited_count` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Fix filtering room types on remote rooms (#17434)Erik Johnston2024-07-111-12/+10
| | | | | | We can only fetch room types for rooms the server is in, so we need to only filter rooms that we're joined to. Also includes a perf fix to bulk fetch room types.
* Handle to-device extensions to Sliding Sync (#17416)Erik Johnston2024-07-101-1/+102
| | | | | | | Implements MSC3885 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Add `rooms` `name` and `avatar` to Sliding Sync `/sync` (#17418)Eric Eastwood2024-07-091-55/+96
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `rooms.bump_stamp` to Sliding Sync `/sync` for easier client-side ↵Eric Eastwood2024-07-081-19/+56
| | | | | | | | | | | | | | | sorting (#17395) `bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves. Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies). In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types. --- In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for. The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622
* Upon deactivation, forget all of the user's rooms (#17400)Travis Ralston2024-07-051-0/+4
| | | | | | This can help ensure that the rooms are eventually purged if the other local users also forget them. Synapse already clears some of the room information as part of the `_background_remove_left_rooms` background task, but this doesn't catch `events`, `event_json`, etc.
* Add `rooms.required_state` to Sliding Sync `/sync` (#17342)Eric Eastwood2024-07-042-63/+391
| | | Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.
* Add `room_types`/`not_room_types` filtering to Sliding Sync `/sync` (#17337)Eric Eastwood2024-07-021-5/+30
| | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Return some room data in Sliding Sync `/sync` (#17320)Eric Eastwood2024-07-021-152/+490
| | | | | | - Timeline events - Stripped `invite_state` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `is_invite` filtering to Sliding Sync `/sync` (#17335)Eric Eastwood2024-06-241-3/+16
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add support for MSC3823 - Account Suspension Part 2 (#17255)Shay2024-06-241-0/+11
|
* Add `is_encrypted` filtering to Sliding Sync `/sync` (#17281)Eric Eastwood2024-06-172-99/+36
| | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `stream_ordering` sort to Sliding Sync `/sync` (#17293)Eric Eastwood2024-06-173-34/+136
| | | Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
* Include user membership on events (#17282)Richard van der Hoff2024-06-138-18/+0
| | | | | MSC4115 has now completed FCP, so we can enable it by default and switch to the stable identifier.
* Add `is_dm` filtering to Sliding Sync `/sync` (#17277)Eric Eastwood2024-06-131-6/+112
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Fix `newly_left` rooms not appearing if we returned early (Sliding Sync) ↵Eric Eastwood2024-06-131-13/+13
| | | | | | | | | | | (#17301) Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync) The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`. Found while working on https://github.com/element-hq/synapse/pull/17293
* Add `event.internal_metadata.instance_name` (#17300)Eric Eastwood2024-06-131-0/+1
| | | | | | | | | | Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`. `instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`. This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293 Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see https://github.com/element-hq/synapse/pull/17187/commits/09609cb0dbca3a4cfd9fbf90cc962e765ec469c0#diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410
* Reorganize Pydantic models and types used in handlers (#17279)Eric Eastwood2024-06-103-176/+5
| | | | | | Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues. - Moved Pydantic parsing models to `synapse/types/rest` - Moved handler types to `synapse/types/handlers`
* Add debug logging for when room keys are uploaded, including whether they ↵reivilibre2024-06-071-0/+18
| | | | | | | | | | | are replacing other room keys. (#17266) Fixes: #17013 Add logging for whether room keys are replaced This is motivated by the Crypto team who need to diagnose crypto issues. The existing opentracing logging is not enough because it is not enabled for all users.
* Add Sliding Sync `/sync` endpoint (initial implementation) (#17187)Eric Eastwood2024-06-062-8/+621
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering). Rooms appear in the Sliding sync response based on: - `invite`, `join`, `knock`, `ban` membership events - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`) - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`) - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range. ### Example request `POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync` ```json { "lists": { "foo-list": { "ranges": [ [0, 99] ], "sort": [ "by_notification_level", "by_recency", "by_name" ], "required_state": [ ["m.room.join_rules", ""], ["m.room.history_visibility", ""], ["m.space.child", "*"] ], "timeline_limit": 100 } } } ``` Response: ```json { "next_pos": "s58_224_0_13_10_1_1_16_0_1", "lists": { "foo-list": { "count": 1, "ops": [ { "op": "SYNC", "range": [0, 99], "room_ids": [ "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server" ] } ] } }, "rooms": {}, "extensions": {} } ```
* Handle OTK uploads off master (#17271)Erik Johnston2024-06-061-29/+55
| | | And fallback keys uploads. Only device keys need handling on master
* Don't try and resync devices for down hosts (#17273)Erik Johnston2024-06-061-6/+18
| | | | It's just a waste of time if we won't even query the remote host as its marked as down.
* Always return OTK counts (#17275)Erik Johnston2024-06-061-4/+29
| | | Broke in https://github.com/element-hq/synapse/pull/17215
* Use fully-qualified `PersistedEventPosition` when returning `RoomsForUser` ↵Eric Eastwood2024-06-045-69/+8
| | | | | | | (#17265) Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation. Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
* Fix bug where typing replication breaks (#17252)Erik Johnston2024-05-311-3/+3
| | | | This can happen on restarts of the service, due to old rooms being pruned.
* Fix logging errors when receiving invalid User ID for key querys (#17250)Erik Johnston2024-05-311-0/+5
|
* In sync wait for worker to catch up since token (#17215)Erik Johnston2024-05-301-0/+35
| | | | | | | Otherwise things will get confused. An alternative would be to make sure that for lagging stream we don't return anything (and make sure the returned next_batch token doesn't go backwards). But that is a faff.
* Fix deduplicating of membership events to not create unused state groups. ↵Erik Johnston2024-05-302-35/+32
| | | | | | | | | | | | | (#17164) We try and deduplicate in two places: 1) really early on, and 2) just before we persist the event. The first case was broken due to it occuring before the profile information was added, and so it thought the event contents were different. The second case did catch it and handle it correctly, however doing so creates a redundant state group leading to bloat. Fixes #3791
* Don't invalidate all `get_relations_for_event` on history purge (#17083)Erik Johnston2024-05-291-1/+1
| | | | This is a tree cache already, so may as well move the room ID to the front and use that
* Ignore attempts to send to-device messages to bad users (#17240)Erik Johnston2024-05-291-0/+7
| | | | | | | | Currently sending a to-device message to a user ID with a dodgy destination is accepted, but then ends up spamming the logs when we try and send to the destination. An alternative would be to reject the request, but I'm slightly nervous that could break things.
* Handle duplicate OTK uploads racing (#17241)Erik Johnston2024-05-291-33/+45
| | | Currently this causes one of then to 500.
* Add Sliding Sync `/sync/e2ee` endpoint for To-Device messages (#17167)Eric Eastwood2024-05-231-10/+237
| | | | | | | | | | | This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa. Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought. Based on: - https://github.com/matrix-org/matrix-spec-proposals/pull/3575 - https://github.com/matrix-org/matrix-spec-proposals/pull/3885 - https://github.com/matrix-org/matrix-spec-proposals/pull/3884
* Log exceptions when failing to auto-join new user according to the ↵reivilibre2024-05-221-1/+1
| | | | | | | `auto_join_rooms` option. (#17176) Would have been useful for tracking down #16878. Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
* Reduce work of calculating outbound device pokes (#17211)Erik Johnston2024-05-221-0/+7
|
* Bring auto-accept invite logic into Synapse (#17147)devonh2024-05-211-1/+1
| | | | | | | | | | | | | | This PR ports the logic from the [synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite) module into synapse. I went with the naive approach of injecting the "module" next to where third party modules are currently loaded. If there is a better/preferred way to handle this, I'm all ears. It wasn't obvious to me if there was a better location to add this logic that would cleanly apply to all incoming invite events. Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter errors.
* Improve perf of sync device lists (#17216)Erik Johnston2024-05-212-35/+25
| | | | | | | | Re-introduces #17191, and includes #17197 and #17214 The basic idea is to stop calling `get_rooms_for_user` everywhere, and instead use the table `device_lists_changes_in_room`. Commits reviewable one-by-one.
* Refactor `SyncResultBuilder` assembly to its own function (#17202)Eric Eastwood2024-05-161-116/+148
| | | | | | We will re-use `get_sync_result_builder(...)` in https://github.com/element-hq/synapse/pull/17167 Split out from https://github.com/element-hq/synapse/pull/17167
* Fix `joined_rooms`/`joined_room_ids` usage (#17208)Eric Eastwood2024-05-161-1/+1
| | | | | | | | This change was introduced in https://github.com/element-hq/synapse/pull/17203 But then https://github.com/element-hq/synapse/pull/17207 was reverted which brought back usage `joined_rooms` that needed to be updated. Wasn't caught because `develop` wasn't up to date before merging.
* Rename to be obvious: `joined_rooms` -> `joined_room_ids` (#17203)Eric Eastwood2024-05-161-2/+2
| | | Split out from https://github.com/element-hq/synapse/pull/17167
* Removed `request_key` from the `SyncConfig` (moved outside as its own ↵Eric Eastwood2024-05-161-3/+3
| | | | | | | | | function parameter) (#17201) Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) so it doesn't have to flow into `_generate_sync_entry_for_xxx` methods. This way we can separate the concerns of caching from generating the response and reuse the `_generate_sync_entry_for_xxx` functions as we see fit. Plus caching doesn't really have anything to do with the config of sync. Split from https://github.com/element-hq/synapse/pull/17167 Spawning from https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
* Revert "Improve perf of sync device lists" (#17207)Erik Johnston2024-05-161-6/+31
| | | Reverts element-hq/synapse#17191
* Fix bug where push rules would be empty in `/sync` (#17142)Erik Johnston2024-05-161-12/+8
| | | | | | Fixes #16987 Some old accounts seem to have an entry in global account data table for push rules, which we should ignore
* Refactor Sync handler to be able to return different sync responses ↵Eric Eastwood2024-05-161-7/+58
| | | | | | | | | | | | | | (`SyncVersion`) (#17200) Refactor Sync handler to be able to be able to return different sync responses (`SyncVersion`). Preparation to be able support sync v2 and a new Sliding Sync `/sync/e2ee` endpoint which returns a subset of sync v2. Split upon request: https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279 Split from https://github.com/element-hq/synapse/pull/17167 where we will add `SyncVersion.E2EE_SYNC` and a new type of sync response.
* Improve perf of sync device lists (#17191)Erik Johnston2024-05-141-31/+6
| | | | | It's almost always more efficient to query the rooms that have device list changes, rather than looking at the list of all users whose devices have changed and then look for shared rooms.
* Allows CAS SSO flow to provide user IDs composed of numbers only (#17098)Aurélien Grimpard2024-05-141-0/+5
|
* Fix undiscovered linter errors (#17166)devonh2024-05-081-3/+11
| | | | | | Linter errors are showing up in #17147 that are unrelated to that PR. The errors do not currently show up on develop. This PR aims to resolve the linter errors separately from #17147.
* Add support for MSC3823 - Account Suspension (#17051)Shay2024-05-011-0/+30
|
* Apply user `email` & `picture` during OIDC registration if present & ↵devonh2024-04-291-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | selected (#17120) This change will apply the `email` & `picture` provided by OIDC to the new user account when registering a new user via OIDC. If the user is directed to the account details form, this change makes sure they have been selected before applying them, otherwise they are omitted. In particular, this change ensures the values are carried through when Synapse has consent configured, and the redirect to the consent form/s are followed. I have tested everything manually. Including: - with/without consent configured - allowing/not allowing the use of email/avatar (via `sso_auth_account_details.html`) - with/without automatic account detail population (by un/commenting the `localpart_template` option in synapse config). ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
* Add support for MSC4115 (#17104)Richard van der Hoff2024-04-298-7/+40
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Ensure that incoming to-device messages are not dropped (#17127)Richard van der Hoff2024-04-291-0/+3
| | | | | | | | | | | | | | | | | | | | ... when workers are unreachable, etc. Fixes https://github.com/element-hq/synapse/issues/17117. The general principle is just to make sure that we propagate any exceptions to the JsonResource, so that we return an error code to the sending server. That means that the sending server no longer considers the message safely sent, so it will retry later. In the issue, Erik mentions that an alternative solution would be to persist the to-device messages into a table so that they can be retried. This might be an improvement for performance, but even if we did that, we still need this mechanism, since we might be unable to reach the database. So, if we want to do that, it can be a later follow-up. --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Redact membership events if the user requested erasure upon deactivating ↵Till2024-04-251-1/+12
| | | | | (#17076) Fixes #15355 by redacting all membership events before leaving rooms.
* bugfix: make msc3967 idempotent (#16943)Kegan Dougal2024-04-151-0/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MSC3967 was updated recently to make it more robust to network failures: > there is an existing cross-signing master key and it exactly matches the cross-signing master key provided in the request body. If there are any additional keys provided in the request (self signing key, user signing key) they MUST also match the existing keys stored on the server. In other words, the request contains no new keys. If there are new keys, UIA MUST be performed. https://github.com/matrix-org/matrix-spec-proposals/blob/hughns/device-signing-upload-uia/proposals/3967-device-signing-upload-uia.md#proposal This covers the case where the 200 OK is lost in transit so the client retries the upload, only to then get UIA'd. Complement tests: https://github.com/matrix-org/complement/pull/713 - passing example https://github.com/element-hq/synapse/actions/runs/7976948122/job/21778795094?pr=16943#step:7:8820 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: reivilibre <oliverw@matrix.org>
* Also check if first event matches the last in prev batch (#17066)Erik Johnston2024-04-091-7/+13
| | | | | Refinement of #17064 cc @richvdh
* Fix PR #16677, a parameter was missing in a function call (#17033)Mathieu Velten2024-04-091-0/+1
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fixups to #17064 (#17065)Erik Johnston2024-04-081-0/+3
| | | | | Forget a line, and an empty batch is trivially linear. c.f. #17064
* Add back fast path for non-gappy syncs (#17064)Erik Johnston2024-04-081-0/+36
| | | | | | | | | | | PR #16942 removed an invalid optimisation that avoided pulling out state for non-gappy syncs. This causes a large increase in DB usage. c.f. #16941 for why that optimisation was wrong. However, we can still optimise in the simple case where the events in the timeline are a linear chain without any branching/merging of the DAG. cc. @richvdh
* Fix bug in calculating state for non-gappy syncs (#16942)Richard van der Hoff2024-04-041-54/+37
| | | | | | | | | Unfortunately, the optimisation we applied here for non-gappy syncs is not actually valid. Fixes https://github.com/element-hq/synapse/issues/16941. ~~Based on https://github.com/element-hq/synapse/pull/16930.~~ Requires https://github.com/matrix-org/sytest/pull/1374.
* `/sync`: fix bug in calculating `state` response (#16930)Richard van der Hoff2024-04-041-41/+13
| | | | | | | Fix a long-standing issue which could cause state to be omitted from the sync response if the last event was filtered out. Fixes: https://github.com/element-hq/synapse/issues/16928
* Fix bug in `/sync` response for archived rooms (#16932)Richard van der Hoff2024-04-041-14/+107
| | | | | | | | | | | | This PR fixes a very, very niche edge-case, but I've got some more work coming which will otherwise make the problem worse. The bug happens when the syncing user leaves a room, and has a sync filter which includes "left" rooms, but sets the timeline limit to 0. In that case, the state returned in the `state` section is calculated incorrectly. The fix is to pass a token corresponding to the point that the user leaves the room through to `compute_state_delta`.
* Fixups to new push stream (#17038)Erik Johnston2024-03-281-2/+4
| | | Follow on from #17037
* Add support for moving `/push_rules` off of main process (#17037)Erik Johnston2024-03-281-3/+16
|
* Add OIDC config to add extra parameters to the authorize URL (#16971)Mathieu Velten2024-03-221-6/+14
|
* Do not refuse to set read_marker if previous event_id is in wrong room (#16990)SpiritCroc2024-03-211-2/+2
|
* Fix reject knocks on deactivating account (#17010)Hanadi2024-03-211-10/+15
|
* OIDC: try to JWT decode userinfo response if JSON parsing failed (#16972)Mathieu Velten2024-03-211-4/+28
|
* Update power level default for public rooms (#16907)Shay2024-03-191-1/+1
|
* Improve event validation (#16908)Shay2024-03-192-1/+24
| | | As the title states.
* Pass module API to OIDC mapping provider (#16974)Mathieu Velten2024-03-191-3/+14
| | | | As done for SAML mapping provider, let's pass the module API to the OIDC one so the mapper can do more logic in its code.
* `/sync`: Fix edge-case in calculating the "device_lists" response (#16949)Richard van der Hoff2024-03-141-2/+9
| | | | | Fixes https://github.com/element-hq/synapse/issues/16948. If the `join` and the `leave` are in the same sync response, we need to count them as a "left" user.
* Split up `SyncHandler.compute_state_delta` (#16929)Richard van der Hoff2024-03-141-145/+236
| | | | | | This is a huge method, which melts my brain. This is a non-functional change which lays some groundwork for future work in this area.
* Improve lock performance when a lot of locks are waiting (#16840)Mathieu Velten2024-03-141-6/+9
| | | | | | | | | | | | When a lot of locks are waiting for a single lock, notifying all locks independently with `call_later` on each release is really costly and incurs some kind of async contention, where the CPU is spinning a lot for not much. The included test is taking around 30s before the change, and 0.5s after. It was found following failing tests with https://github.com/element-hq/synapse/pull/16827.
* Bump mypy from 1.5.1 to 1.8.0 (#16901)dependabot[bot]2024-03-132-3/+3
|
* Bump black from 23.10.1 to 24.2.0 (#16936)dependabot[bot]2024-03-1311-49/+51
|
* Don't lock up when joining large rooms (#16903)Erik Johnston2024-02-201-9/+17
| | | | Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
* Add a config to not send out device list updates for specific users (#16909)Erik Johnston2024-02-131-2/+11
| | | | | | | | | List of users not to send out device list updates for when they register new devices. This is useful to handle bot accounts. This is undocumented as its mostly a hack to test on matrix.org. Note: This will still send out device list updates if the device is later updated, e.g. end to end keys are added.
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-2347-0/+60
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Allow room creation but not publishing to continue if room publication rules ↵Shay2024-01-221-4/+2
| | | | | | | | | | | | | are violated when creating a new room. (#16811) Prior to this PR, if a request to create a public (public as in published to the rooms directory) room violated the room list publication rules set in the [config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules), the request to create the room was denied and the room was not created. This PR changes the behavior such that when a request to create a room published to the directory violates room list publication rules, the room is still created but the room is not published to the directory.
* Faster load recents for sync (#16783)Erik Johnston2024-01-101-6/+8
| | | This hopefully reduces the amount of state we need to keep in memory
* Pull less state out if we fail to backfill (#16788)Erik Johnston2024-01-101-9/+12
| | | | | | | | | | | Sometimes we fail to fetch events during backfill due to missing state, and we often end up querying the same bad events periodically (as people backpaginate). In such cases its likely we will continue to fail to get the state, and therefore we should try *before* loading the state that we have from the DB (as otherwise it's wasted DB and memory). --------- Co-authored-by: reivilibre <oliverw@matrix.org>
* Reduce amount of state pulled out when querying federation hierachy (#16785)Erik Johnston2024-01-101-1/+11
| | | | | | | | | | | There are two changes here: 1. Only pull out the required state when handling the request. 2. Change the get filtered state return type to check that we're only querying state that was requested --------- Co-authored-by: reivilibre <oliverw@matrix.org>
* Faster partial join to room with complex auth graph (#7)Erik Johnston2024-01-101-49/+30
| | | | | | | | Instead of persisting outliers in a bunch of batches, let's just do them all at once. This is fine because all `_auth_and_persist_outliers_inner` is doing is checking the auth rules for each event, which requires the events to be topologically sorted by the auth graph.
* Filter out rooms from the room directory being served to other homeservers ↵reivilibre2024-01-081-50/+127
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | when those rooms block that homeserver by their Access Control Lists. (#16759) The idea here being that the directory server shouldn't advertise rooms to a requesting server is the requesting server would not be allowed to join or participate in the room. <!-- Fixes: # <!-- --> <!-- Supersedes: # <!-- --> <!-- Follows: # <!-- --> <!-- Part of: # <!-- --> Base: `develop` <!-- git-stack-base-branch:develop --> <!-- This pull request is commit-by-commit review friendly. <!-- --> <!-- This pull request is intended for commit-by-commit review. <!-- --> Original commit schedule, with full messages: <ol> <li> Pass `from_federation_origin` down into room list retrieval code </li> <li> Don't cache /publicRooms response for inbound federated requests </li> <li> fixup! Don't cache /publicRooms response for inbound federated requests </li> <li> Cap the number of /publicRooms entries to 100 </li> <li> Simplify code now that you can't request unlimited rooms </li> <li> Filter out rooms from federated requests that don't have the correct ACL </li> <li> Request a handful more when filtering ACLs so that we can try to avoid shortchanging the requester </li> </ol> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Fix linting (#16780)Erik Johnston2024-01-051-1/+11
| | | Introduced in #16762
* Simplify internal metadata class. (#16762)Erik Johnston2024-01-053-5/+22
| | | | | | | | | We remove these fields as they're just duplicating data the event already stores, and (for reasons :shushing_face:) I'd like to simplify the class to only store simple types. I'm not entirely convinced that we shouldn't instead add helper methods to the event class to generate stream tokens, but I don't really think that's where they belong either
* Add recursion_depth to /relations if recursing (#16775)David Baker2024-01-041-0/+4
| | | | | This is an extra response parameter just added to MSC3981. In the current impl, the recursion depth is always 3, so this just returns a static 3 if the recurse parameter is supplied.
* Merge remote-tracking branch 'gitlab/clokep/license-license' into new_developErik Johnston2023-12-1350-522/+800
|\
| * Update license headersPatrick Cloke2023-11-2150-522/+800
| |
* | Fix upgrading a room without `events` field in power levels (#16725)David Robertson2023-12-051-1/+1
| |
* | Implement MSC4069: Inhibit profile propagation (#16636)Travis Ralston2023-12-041-2/+8
| | | | | | MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/4069
* | ModuleAPI SSO auth callbacks (#15207)Andrew Yasinishyn2023-12-012-0/+24
| | | | | | Signed-off-by: Andrii Yasynyshyn yasinishyn.a.n@gmail.com
* | Drop unused tables & unneeded access token ID for events. (#16522)Patrick Cloke2023-12-011-6/+2
| |
* | Reduce DB load when forget on leave setting is disabled (#16668)Erik Johnston2023-11-291-3/+8
| | | | | | | | | | * Reduce DB load when forget on leave setting is disabled * Newsfile
* | Ignore `encryption_enabled_by_default_for_room_type` for notices room (#16677)Mathieu Velten2023-11-281-1/+7
|/
* Reduce task concurrency (#16656)Erik Johnston2023-11-171-1/+1
|
* Speed up deleting device messages (#16643)Erik Johnston2023-11-161-3/+5
| | | Keeping track of a lower bound of stream ID where we've deleted everything below makes the queries much faster. Otherwise, every time we scan for rows to delete we'd re-scan across all the rows that have previously deleted (until the next table VACUUM).
* Speed up persisting large number of outliers (#16649)Erik Johnston2023-11-161-11/+7
| | | Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
* Add an Admin API to temporarily grant the ability to update an existing ↵David Robertson2023-11-151-7/+13
| | | | cross-signing key without UIA (#16634)
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-154-8/+8
|
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-092-4/+5
|
* Return attrs for more media repo APIs. (#16611)Patrick Cloke2023-11-092-7/+10
|
* Remove remaining usage of cursor_to_dict. (#16564)Patrick Cloke2023-10-313-36/+35
|
* Merge pull request from GHSA-mp92-3jfm-3575Patrick Cloke2023-10-312-0/+9
|
* Claim local one-time-keys in bulk (#16565)David Robertson2023-10-301-0/+10
| | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Claim fallback keys in bulk (#16570)David Robertson2023-10-301-0/+14
|
* Ensure local invited & knocking users leave before purge. (#16559)Patrick Cloke2023-10-271-3/+4
| | | | | This is mostly useful for federated rooms where some users would get stuck in the invite or knock state when the room was purged from their homeserver.
* Fix cross-worker ratelimiting (#16558)Erik Johnston2023-10-271-16/+57
| | | c.f. #16481
* Add new module API for adding custom fields to events `unsigned` section ↵Erik Johnston2023-10-276-18/+20
| | | | (#16549)
* Remove more usages of cursor_to_dict. (#16551)Patrick Cloke2023-10-262-12/+12
| | | Mostly to improve type safety.
* Add a new module API to update user presence state. (#16544)Patrick Cloke2023-10-263-33/+49
| | | | | | | | | | This adds a module API which allows a module to update a user's presence state/status message. This is useful for controlling presence from an external system. To fully control presence from the module the presence.enabled config parameter gains a new state of "untracked" which disables internal tracking of presence changes via user actions, etc. Only updates from the module will be persisted and sent down sync properly).
* Convert simple_select_list and simple_select_list_txn to return lists of ↵Patrick Cloke2023-10-262-6/+3
| | | | | tuples (#16505) This should use fewer allocations and improves type hints.
* Allow multiple workers to write to receipts stream. (#16432)Erik Johnston2023-10-254-30/+40
| | | Fixes #16417
* Mark sync as limited if there is a gap in the timeline (#16485)Erik Johnston2023-10-191-8/+44
| | | | | | | | This splits thinsg into two queries, but most of the time we won't have new event backwards extremities so this shouldn't actually add an extra RTT for the majority of cases. Note this removes the check for events with no prev events, but that was part of MSC2716 work that has since been removed.
* Improve performance of delete device messages query (#16492)Mathieu Velten2023-10-181-0/+2
|
* Convert DeviceLastConnectionInfo to attrs. (#16507)Patrick Cloke2023-10-171-16/+7
| | | To improve type safety & memory usage.
* Convert state delta processing from a dict to attrs. (#16469)Patrick Cloke2023-10-164-78/+73
| | | For improved type checking & memory usage.
* Remove useless async job to delete device messages on sync (#16491)Mathieu Velten2023-10-161-22/+0
|
* Revert "Drop unused tables & unneeded access token ID for events. (#16268)" ↵Patrick Cloke2023-10-121-2/+6
| | | | | | | | (#16465) This reverts commit cabd57746004fe2dacc11aa8d373854a3d25e306. There are additional usages of these tables which need to be removed first.
* Convert user_get_threepids response to attrs. (#16468)Patrick Cloke2023-10-113-5/+7
| | | This improves type annotations by not having a dictionary of Any values.
* Bump ruff from 0.0.290 to 0.0.292 (#16449)dependabot[bot]2023-10-095-8/+0
| | | | | | | | | | | | | | | | | | | | | | | | | * Bump ruff from 0.0.290 to 0.0.292 Bumps [ruff](https://github.com/astral-sh/ruff) from 0.0.290 to 0.0.292. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/BREAKING_CHANGES.md) - [Commits](https://github.com/astral-sh/ruff/compare/v0.0.290...v0.0.292) --- updated-dependencies: - dependency-name: ruff dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Fix up lint --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Apply join rate limiter outside the lineariser (#16441)David Robertson2023-10-061-20/+23
|
* Drop unused tables & unneeded access token ID for events. (#16268)Patrick Cloke2023-10-061-6/+2
| | | | Drop the event_txn_id table and the tables related to MSC2716, which is no longer supported in Synapse.
* Stop sending incorrect knock_state_events. (#16403)Patrick Cloke2023-10-061-11/+2
| | | | | | | | | Synapse was incorrectly implemented with a knock_state_events property on some APIs (instead of knock_room_state). This was correct in Synapse 1.70.0, but *both* fields were sent to also be compatible with Synapse versions expecting the wrong field. Enough time has passed that only the correct field needs to be included/handled.
* Factor out `MultiWriter` token from `RoomStreamToken` (#16427)Erik Johnston2023-10-054-6/+5
|
* Some refactors around receipts stream (#16426)Erik Johnston2023-10-043-19/+16
|
* mypy plugin to check `@cached` return types (#14911)David Robertson2023-10-021-2/+2
| | | | | | | | Co-authored-by: David Robertson <davidr@element.io> Co-authored-by: Patrick Cloke <patrickc@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org> Assert that the return type of callables wrapped in @cached and @cachedList are cachable (aka immutable).
* Add a cache around server ACL checking (#16360)Patrick Cloke2023-09-262-0/+11
| | | | | * Pre-compiles the server ACLs onto an object per room and invalidates them when new events come in. * Converts the server ACL checking into Rust.
* Reduce calls to `send_presence_to_destinations` (#16385)Erik Johnston2023-09-261-15/+18
|
* Convert more cached return values to immutable types (#16356)Patrick Cloke2023-09-202-3/+13
|
* Return immutable objects for cachedList decorators (#16350)Patrick Cloke2023-09-196-33/+37
|
* Make cached account data/tags/admin types immutable (#16325)Patrick Cloke2023-09-182-20/+25
|
* Return an immutable value from get_latest_event_ids_in_room. (#16326)Patrick Cloke2023-09-181-5/+3
|
* Additional validation of receipts (#16327)Patrick Cloke2023-09-181-2/+24
| | | | Reject invalid receipts with a reasonable error message & expands tests for receipts.
* Set email charset as utf-8 rather than utf8 (#16329)José Joaquín Atria2023-09-181-2/+2
|
* Add automatic purge after all users forget a room (#15488)Mathieu Velten2023-09-153-390/+281
| | | | | | Also add restore of purge/shutdown rooms after a synapse restart. Co-authored-by: Eric Eastwood <erice@matrix.org> Co-authored-by: Erik Johnston <erikj@matrix.org>
* Speed up deleting to-device messages task (#16318)Erik Johnston2023-09-141-13/+14
|
* Refactor `get_user_by_id` (#16316)Erik Johnston2023-09-143-31/+26
|
* Fix using dehydrated devices (MSC2697) & refresh tokens (#16288)Hanadi2023-09-131-3/+4
| | | | | Refresh tokens were not correctly moved to the rehydrated device (similar to how the access token is currently handled). This resulted in invalid refresh tokens after rehydration.
* Don't schedule an async task on every sync (#16312)Erik Johnston2023-09-131-11/+26
|
* Fix deleting device inbox when using background worker (#16311)Erik Johnston2023-09-131-31/+31
| | | | | Introduced in #16240 The action for the task was only defined on the "master" handler, rather than the base worker one.
* Add the List-Unsubscribe header for notification emails. (#16274)Patrick Cloke2023-09-111-1/+9
| | | | | | | | Adds both the List-Unsubscribe (RFC2369) and List-Unsubscribe-Post (RFC8058) headers to push notification emails, which together should: * Show an "Unsubscribe" link in the MUA UI when viewing Synapse notification emails. * Enable "one-click" unsubscribe (the user never leaves their MUA, which automatically makes a POST request to the specified endpoint).
* Filter out down hosts when retrying fetching device lists (#16298)Erik Johnston2023-09-111-1/+14
|
* Recheck if remote device is cached before requesting it (#16252)Erik Johnston2023-09-071-6/+15
| | | | This fixes a bug where we could get stuck re-requesting the device over replication again and again.
* Handle "registration_enabled" parameter for CAS (#16262)Aurélien Grimpard2023-09-061-0/+2
| | | | Similar to OIDC, CAS providers can now disable registration such that only existing users are able to login via SSO.
* Allow modules to delete rooms. (#15997)reivilibre2023-09-062-3/+19
| | | | | | | | | | | | | | | * Allow user_id to be optional for room deletion * Add module API method to delete a room * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Don't worry about the case block=True && requester_user_id is None --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Delete device messages asynchronously and in staged batches (#16240)Mathieu Velten2023-09-063-6/+62
|
* Time out busy presence status & test multi-device busy (#16174)Patrick Cloke2023-09-051-1/+18
| | | | | | | Add a (long) timeout to when a "busy" device is considered not online. This does *not* match MSC3026, but is a reasonable thing for an implementation to do. Expands tests for the (unstable) busy presence with multiple devices.
* Track presence state per-device and combine to a user state. (#16066)Patrick Cloke2023-09-051-47/+232
| | | | | | | | | | | Tracks presence on an individual per-device basis and combine the per-device state into a per-user state. This should help in situations where a user has multiple devices with conflicting status (e.g. one is syncing with unavailable and one is syncing with online). The tie-breaking is done by priority: BUSY > ONLINE > UNAVAILABLE > OFFLINE
* Bump twisted from 22.10.0 to 23.8.0 (#16235)dependabot[bot]2023-09-051-6/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Bump twisted from 22.10.0 to 23.8.0 Bumps [twisted](https://github.com/twisted/twisted) from 22.10.0 to 23.8.0. - [Release notes](https://github.com/twisted/twisted/releases) - [Changelog](https://github.com/twisted/twisted/blob/trunk/NEWS.rst) - [Commits](https://github.com/twisted/twisted/compare/twisted-22.10.0...twisted-23.8.0) --- updated-dependencies: - dependency-name: twisted dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Fix types * Fix lint * Newsfile --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Don't wake up destination transaction queue if they're not due for retry. ↵Erik Johnston2023-09-044-27/+36
| | | | (#16223)
* Add last_seen_ts to the admin users API (#16218)Mathieu Velten2023-09-041-0/+1
|
* Stop writing to the event_txn_id table (#16175)Patrick Cloke2023-08-301-13/+0
|
* Describe which rate limiter was hit in logs (#16135)David Robertson2023-08-305-26/+17
|
* Track currently syncing users by device for presence (#16172)Patrick Cloke2023-08-291-55/+100
| | | | | | | Refactoring to use both the user ID & the device ID when tracking the currently syncing users in the presence handler. This is done both locally and over replication. Note that the device ID is discarded but will be used in a future change.
* Bump ruff from 0.0.277 to 0.0.286 (#16198)dependabot[bot]2023-08-291-1/+1
| | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Support IPv6-only SMTP servers (#16155)Evilham2023-08-291-17/+11
| | | | Use Twisted HostnameEndpoint to connect to SMTP servers (instead of connectTCP/connectSSL) which properly supports IPv6-only servers.
* Pass the device ID around in the presence handler (#16171)Patrick Cloke2023-08-283-11/+45
| | | | | | Refactoring to pass the device ID (in addition to the user ID) through the presence handler (specifically the `user_syncing`, `set_state`, and `bump_presence_active_time` methods and their replication versions).
* Combine logic about not overriding BUSY presence. (#16170)Patrick Cloke2023-08-281-92/+63
| | | | | | | | | | | | | Simplify some of the presence code by reducing duplicated code between worker & non-worker modes. The main change is to push some of the logic from `user_syncing` into `set_state`. This is done by passing whether the user is setting the presence via a `/sync` with a new `is_sync` flag to `set_state`. If this is `true` some additional logic is performed: * Don't override `busy` presence. * Update the `last_user_sync_ts`. * Never update the status message.
* Add configuration setting for CAS protocol version (#15816)Aurélien Grimpard2023-08-241-1/+5
|
* Fix typechecking with twisted trunk (#16121)David Robertson2023-08-241-16/+16
|
* Only lock when we're backfilling (#16159)Erik Johnston2023-08-232-146/+156
|
* Disable caching in /sync corner case (#16080)Matthew Ma2023-08-171-10/+10
| | | Fixes #15502
* Rename pagination&purge locks and add comments explaining them (#16112)Mathieu Velten2023-08-164-13/+22
|
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-154-11/+16
|
* Allow customizing IdP name and icon for SAML and CAS (#16094)Gabriel Rodríguez2023-08-112-10/+12
|
* Clean-up presence code (#16092)Patrick Cloke2023-08-101-94/+75
| | | | | | | | Misc. clean-ups to: * Use keyword arguments. * Return early (reducing indentation) of some functions. * Removing duplicated / unused code. * Use wrap_as_background_process.
* Implements admin API to lock an user (MSC3939) (#15870)Mathieu Velten2023-08-102-1/+5
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-082-16/+11
|
* Fix deletion for Dehydrated Devices (#16046)Shay2023-08-041-0/+16
|
* Stabilize support for MSC3970: updated transaction semantics (scope to ↵Patrick Cloke2023-08-041-7/+5
| | | | | | | `device_id`) (#15629) For now this maintains compatible with old Synapses by falling back to using transaction semantics on a per-access token. A future version of Synapse will drop support for this.
* Trim whitespace when setting display names (#16031)Mohit Rathee2023-08-011-1/+1
|
* Combine duplicated code for calculating an event ID from a txn ID (#16023)Patrick Cloke2023-07-312-32/+35
| | | | Refactoring related to stabilization of MSC3970, refactor to combine code which has the same logic.
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-314-25/+414
| | | | | (#15791) c.f. #13476
* Fix 404 on /profile when the display name is empty but not the avatar (#16012)Mathieu Velten2023-07-271-1/+1
|
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-242-3/+109
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Resync stale devices in background (#15975)Erik Johnston2023-07-241-1/+8
| | | | This is so we don't block responding to federation transaction while we try and fetch the device lists.
* Speed up updating state in large rooms (#15971)Erik Johnston2023-07-201-5/+4
| | | This should speed up updating state in rooms with lots of state.
* Reduce the amount of state we pull out (#15968)Erik Johnston2023-07-203-12/+13
|
* Support room version 11 (#15912)Patrick Cloke2023-07-184-6/+6
| | | | | And fix a bug in the implementation of the updated redaction format (MSC2174) where the top-level redacts field was not properly added for backwards-compatibility.
* Add a locality to a few presence metrics (#15952)Jason Little2023-07-181-13/+24
|
* Better warning in logs when we fail to fetch an alias (#15922)Eric Eastwood2023-07-111-1/+3
| | | | | | | | | | | | | **Before:** ``` Error retrieving alias ``` **After:** ``` Error retrieving alias #foo:bar -> 401 Unauthorized ``` *Spawning from creating the [manual testing strategy for the outbound federation proxy](https://github.com/matrix-org/synapse/pull/15773).*
* Add + as an allowed character for Matrix IDs (MSC4009) (#15911)Patrick Cloke2023-07-113-13/+6
|
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-165-796/+34
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* Fix joining rooms through aliases where the alias server isn't a real ↵Tulir Asokan2023-06-141-1/+1
| | | | homeserver (#15776)
* Avoid backfill when we already have messages to return (#15737)Eric Eastwood2023-06-131-22/+115
| | | | | We now only block the client to backfill when we see a large gap in the events (more than 2 events missing in a row according to `depth`), more than 3 single-event holes, or not enough messages to fill the response. Otherwise, we return the messages directly to the client and backfill in the background for eventual consistency sake. Fix https://github.com/matrix-org/synapse/issues/15696
* Backfill in the background if we're doing it "just because" (#15710)Eric Eastwood2023-06-091-4/+14
| | | Fix https://github.com/matrix-org/synapse/issues/15702
* Check required power levels earlier in createRoom handler. (#15695)Grant McLean2023-06-071-14/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Check required power levels earlier in createRoom handler. - If a server was configured to reject the creation of rooms with E2EE enabled (by specifying an unattainably high power level for "m.room.encryption" in default_power_level_content_override), the 403 error was not being triggered until after the room was created and before the "m.room.power_levels" was sent. This allowed a user to access the partially-configured room and complete the setup of E2EE and power levels manually. - This change causes the power level overrides to be checked earlier and the request to be rejected before the user gains access to the room. - A new `_validate_room_config` method is added to contain checks that should be run before a room is created. - The new test case confirms that a user request is rejected by the new validation method. Signed-off-by: Grant McLean <grant@catalyst.net.nz> * Add a changelog file. * Formatting fix for black. * Remove unneeded line from test. --------- Signed-off-by: Grant McLean <grant@catalyst.net.nz>
* Remove some unused `server_name` fields (#15723)Sean Quah2023-06-064-4/+0
| | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Some house keeping on `maybe_backfill()` functions (#15709)Eric Eastwood2023-06-051-0/+17
|
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-026-22/+14
| | | | `profiles` and `user_filters` (#15649)
* Add a catch-all * to the supported relation types when redacting (#15705)Mathieu Velten2023-06-021-5/+11
| | | This is an update to MSC3912 implementation
* Add stricter mypy options (#15694)Patrick Cloke2023-05-312-3/+3
| | | | Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any, and disallow_incomplete_defs.
* Refactor config to be an experimental featureHugh Nimmo-Smith2023-05-301-2/+2
| | | | Also enforce you can't combine it with incompatible config options
* Disable account related endpoints when using OAuth delegationQuentin Gliech2023-05-301-1/+7
|
* Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-241-8/+62
| | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* Add requesting user id parameter to key claim methods in ↵Shay2023-05-241-1/+2
| | | | `TransportLayerClient` (#15663)
* Consolidate logic to check for deactivated users. (#15634)Patrick Cloke2023-05-232-25/+8
| | | | | | | This moves the deactivated user check to the method which all login types call. Additionally updates the application service tests to be more realistic by removing invalid tests and fixing server names.
* Trace how many new events from the backfill response we need to process (#15633)Eric Eastwood2023-05-191-0/+5
| | | You can kinda derive this information from how many `_process_pulled_event` spans there are but it would be nice to quickly glance.
* Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-193-12/+10
| | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* Do not allow deactivated users to login with JWT. (#15624)Patrick Cloke2023-05-191-0/+118
| | | | | To improve the organization of this code it moves the JWT login checks to a separate handler and then fixes the bug (and a deprecation warning).
* Handle missing previous read marker event. (#15464)Nick Mills-Barrett2023-05-181-4/+14
| | | | | If the previous read marker is pointing to an event that no longer exists (e.g. due to retention) then assume that the newly given read marker is newer.
* Error if attempting to set m.push_rules account data, per MSC4010. (#15555)Patrick Cloke2023-05-092-11/+17
| | | | | m.push_rules, like m.fully_read, is a special account data type that cannot be set using the normal /account_data endpoint. Return an error instead of allowing data that will not be used to be stored.
* Use account data constants in more places. (#15554)Patrick Cloke2023-05-092-8/+9
|
* Check appservices for devices during a /user/devices query. (#15539)Patrick Cloke2023-05-051-0/+28
| | | | | | | | MSC3984 proxies /keys/query requests to appservices, but servers will can also requests devices / keys from the /user/devices endpoint. The formats are close enough that we can "proxy" that /user/devices to appservices (by calling /keys/query) and then change the format of the returned data before returning it over federation.
* Implement MSC4009 to widen the allowed Matrix ID grammar (#15536)Patrick Cloke2023-05-052-15/+18
| | | | | Behind a configuration flag this adds + to the list of allowed characters in Matrix IDs. The main feature this enables is using full E.164 phone numbers as Matrix IDs.
* Clean up and clarify "Create or modify Account" Admin API documentation (#15544)Andrew Morgan2023-05-051-2/+2
|
* Factor out an `is_mine_server_name` method (#15542)Sean Quah2023-05-056-8/+13
| | | | | | | | | | | | Add an `is_mine_server_name` method, similar to `is_mine_id`. Ideally we would use this consistently, instead of sometimes comparing against `hs.hostname` and other times reaching into `hs.config.server.server_name`. Also fix a bug in the tests where `hs.hostname` would sometimes differ from `hs.config.server.server_name`. Signed-off-by: Sean Quah <seanq@matrix.org>
* Move ThirdPartyEventRules into module_api/callbacks (#15535)Andrew Morgan2023-05-049-19/+28
|
* Update the base rules to remove the dont_notify action. (MSC3987) (#15534)Patrick Cloke2023-05-041-0/+2
| | | | | A dont_notify action is a no-op (and coalesce is undefined). These are both considered no-ops by the spec, per MSC3987 and the predefined push rules were updated to remove dont_notify from the list of actions.
* Add config option to forget rooms automatically when users leave them (#15224)Sean Quah2023-05-032-22/+154
| | | | | This is largely based off the stats and user directory updater code. Signed-off-by: Sean Quah <seanq@matrix.org>
* Initial implementation of MSC3981: recursive relations API (#15315)Patrick Cloke2023-05-021-0/+3
| | | | | | | | | | | Adds an optional keyword argument to the /relations API which will recurse a limited number of event relationships. This will cause the API to return not just the events related to the parent event, but also events related to those related to the parent event, etc. This is disabled by default behind an experimental configuration flag and is currently implemented using prefixed parameters.
* Add support for claiming multiple OTKs at once. (#15468)Patrick Cloke2023-04-272-16/+29
| | | | | | | MSC3983 provides a way to request multiple OTKs at once from appservices, this extends this concept to the Client-Server API. Note that this will likely be spit out into a separate MSC, but is currently part of MSC3983.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-261-6/+2
|
* Add unstable /keys/claim endpoint which always returns fallback keys. (#15462)Patrick Cloke2023-04-252-15/+68
| | | | | | | | | | | | | It can be useful to always return the fallback key when attempting to claim keys. This adds an unstable endpoint for `/keys/claim` which always returns fallback keys in addition to one-time-keys. The fallback key(s) are not marked as "used" unless there are no corresponding OTKs. This is currently defined in MSC3983 (although likely to be split out to a separate MSC). The endpoint shape may change or be requested differently (i.e. a keyword parameter on the current endpoint), but the core logic should be reasonable.
* Experimental support for MSC3970: per-device transaction IDs (#15318)Quentin Gliech2023-04-252-12/+59
|
* Remove legacy code of single user device resync api (#15418)Alok Kumar Singh2023-04-213-64/+22
| | | | | * Removed single-user resync usage and updated it to use multi-user counterpart Signed-off-by: Alok Kumar Singh alokaks601@gmail.com
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-187-26/+44
|
* Delete pushers after calling on_logged_out module hook on device delete (#15410)Mathieu Velten2023-04-141-2/+4
|
* Implement MSC2175: remove the creator field from create events. (#15394)Patrick Cloke2023-04-063-12/+22
|
* Don't keep old stream_ordering_to_exterm around (#15382)Erik Johnston2023-04-061-0/+10
|
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+3
|
* Fix a rare bug where initial /syncs would fail (#15383)Sean Quah2023-04-041-5/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change fixes a rare bug where initial /syncs would fail with a `KeyError` under the following circumstances: 1. A user fast joins a remote room. 2. The user is kicked from the room before the room's full state has been synced. 3. A second local user fast joins the room. 4. Events are backfilled into the room with a higher topological ordering than the original user's leave. They are assigned a negative stream ordering. It's not clear how backfill happened here, since it is expected to be equivalent to syncing the full state. 5. The second local user leaves the room before the room's full state has been synced. The homeserver does not complete the sync. 6. The original user performs an initial /sync with lazy_load_members enabled. * Because they were kicked from the room, the room is included in the /sync response even though the include_leave option is not specified. * To populate the room's timeline, `_load_filtered_recents` / `get_recent_events_for_room` fetches events with a lower stream ordering than the leave event and picks the ones with the highest topological orderings (which are most recent). This captures the backfilled events after the leave, since they have a negative stream ordering. These events are filtered out of the timeline, since the user was not in the room at the time and cannot view them. The sync code ends up with an empty timeline for the room that notably does not include the user's leave event. This seems buggy, but at least we don't disclose events the user isn't allowed to see. * Normally, `compute_state_delta` would fetch the state at the start and end of the room's timeline to generate the sync response. Since the timeline is empty, it fetches the state at `min(now, last event in the room)`, which corresponds with the second user's leave. The state during the entirety of the second user's membership does not include the membership for the first user because of partial state. This part is also questionable, since we are fetching state from outside the bounds of the user's membership. * `compute_state_delta` then tries and fails to find the user's membership in the auth events of timeline events. Because there is no timeline event whose auth events are expected to contain the user's membership, a `KeyError` is raised. Also contains a drive-by fix for a separate unlikely race condition. Signed-off-by: Sean Quah <seanq@matrix.org>
* Revert pruning of old devices (#15360)Erik Johnston2023-03-312-51/+3
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* Implement MSC3984 to proxy /keys/query requests to appservices. (#15321)Patrick Cloke2023-03-302-0/+77
| | | | | If enabled, for users which are exclusively owned by an application service then the appservice will be queried for devices in addition to any information stored in the Synapse database.
* Fix spinloop during partial state sync when a prev event is in backoff (#15351)Sean Quah2023-03-302-25/+35
| | | | | | | | Previously, we would spin in a tight loop until `update_state_for_partial_state_event` stopped raising `FederationPullAttemptBackoffError`s. Replace the spinloop with a wait until the backoff period has expired. Signed-off-by: Sean Quah <seanq@matrix.org>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-09-20 01:19:43 +0000