summary refs log tree commit diff
path: root/synapse/handlers/sliding_sync.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* SSS: Implement PREVIOUSLY room tracking (#17535)Erik Johnston2024-08-081-16/+52
| | | | | | Implement tracking of rooms that have had updates that have not been sent down to clients. Simplified Sliding Sync (SSS)
* Sliding Sync: Use `stream_ordering` based timeline pagination for ↵Eric Eastwood2024-08-071-3/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | incremental sync (#17510) Use `stream_ordering` based `timeline` pagination for incremental `/sync` in Sliding Sync. Previously, we were always using a `topological_ordering` but we should only be using that for historical scenarios (initial `/sync`, newly joined, or haven't sent the room down the connection before). This is slightly different than what the [spec suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing) > Events are ordered in this API according to the arrival time of the event on the homeserver. This can conflict with other APIs which order events based on their partial ordering in the event graph. This can result in duplicate events being received (once per distinct API called). Clients SHOULD de-duplicate events based on the event ID when this happens. But we've had a [discussion below in this PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569) and this matches what Sync v2 already does and seems like it makes sense. Created a spec issue https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this. Related issues: - https://github.com/matrix-org/matrix-spec/issues/1917 - https://github.com/matrix-org/matrix-spec/issues/852 - https://github.com/matrix-org/matrix-spec-proposals/pull/4033
* Sliding Sync: Add more tracing (#17514)Eric Eastwood2024-08-061-239/+327
| | | | | | | | | | | | Spawning from looking at a couple traces and wanting a little more info. Follow-up to github.com/element-hq/synapse/pull/17501 The changes in this PR allow you to find slow Sliding Sync traces ignoring the `wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user` operation with `RESULT.result=true` indicating that it actually returned non-empty results. If you want to find traces for your own user, you can use `RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`
* SS: Reset connection if token is unrecognized (#17529)Erik Johnston2024-08-061-0/+27
| | | | | | | | | | | | | | This triggers the client to start a new sliding sync connection. If we don't do this and the client asks for the full range of rooms, we end up sending down all rooms and their state from scratch (which can be very slow) This causes things like https://github.com/element-hq/element-x-ios/issues/3115 after we restart the server --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Add typing notification extension (MSC3961) (#17505)Eric Eastwood2024-07-311-0/+81
| | | | | | | [MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Sliding Sync: Update filters to be robust against remote invite rooms (#17450)Eric Eastwood2024-07-301-37/+348
| | | | | | | | | | | | | | | Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to be robust when dealing with remote invite rooms in Sliding Sync. Part of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Follow-up to https://github.com/element-hq/synapse/pull/17434 We now take into account current state, fallback to stripped state for invite/knock rooms, then historical state. If we can't determine the info needed to filter a room (either from state or stripped state), it is filtered out.
* Sliding Sync: Add receipts extension (MSC3960) (#17489)Eric Eastwood2024-07-301-60/+208
| | | | | | | [MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add some more opentracing to sliding sync (#17501)Erik Johnston2024-07-301-3/+15
| | | This will make it easier to see what it is doing in jaeger.
* Only send rooms with updates down sliding sync (#17479)Erik Johnston2024-07-301-2/+49
| | | | | | | | | | | | Rather than always including all rooms in range. Also adds a pre-filter to rooms that checks the stream change cache to see if anything might have happened. Based on #17447 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Track whether we have sent rooms down to clients (#17447)Erik Johnston2024-07-291-36/+314
| | | | | | | | | | | | | | | | The basic idea is that we introduce a new token for a sliding sync connection, which stores the mapping of room to room "status" (i.e. have we sent the room down?). This token allows us to handle duplicate requests properly. In future it can be used to store more "per-connection" information safely. In future this should be migrated into the DB, so its important that we try to reduce the number of syncs where we need to update the per-connection information. In this PoC this only happens when we: a) send down a set of room for the first time, or b) we have previously sent down a room and there are updates but we are not sending the room down the sync (due to not falling in a list range) Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Add Account Data extension (MSC3959) (#17477)Eric Eastwood2024-07-241-0/+138
| | | | | | Extensions based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Sliding Sync: ensure bump stamp ignores backfilled events (#17478)Erik Johnston2024-07-241-2/+8
| | | | | | Backfill events have a negative stream ordering, and so its not useful to use to compare with other (positive) stream orderings. Plus, the Rust SDK currently assumes `bump_stamp` is positive.
* Use a new token format for sliding sync (#17452)Erik Johnston2024-07-241-12/+18
| | | | | | | This is in preparation for adding per-connection state. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Speed up SS room sorting (#17468)Erik Johnston2024-07-231-22/+21
| | | | | | | We do this by bulk fetching the latest stream ordering. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Sliding Sync: Add E2EE extension (MSC3884) (#17454)Eric Eastwood2024-07-221-15/+92
| | | | | Spec: [MSC3884](https://github.com/matrix-org/matrix-spec-proposals/pull/3884) Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* SS: Implement `$ME` support (#17469)Erik Johnston2024-07-221-1/+5
| | | `$ME` can be used as a substitute for the requester's user ID.
* Generate room sync data concurrently (#17458)Erik Johnston2024-07-191-2/+10
| | | This is also what we do for standard `/sync`.
* Add `m.room.create` to default bump event types (#17453)Eric Eastwood2024-07-181-0/+1
| | | | | | Add `m.room.create` to default bump event types This probably helps when no messages have been sent in the room and it was just created.
* Fix bug where sync could get stuck when using workers (#17438)Erik Johnston2024-07-151-2/+9
| | | | This is because we serialized the token wrong if the instance map contained entries from before the minimum token.
* Add room subscriptions to Sliding Sync `/sync` (#17432)Eric Eastwood2024-07-151-115/+288
| | | | | | | | | | | | | Add room subscriptions to Sliding Sync `/sync` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Currently, you can only subscribe to rooms you have had *any* membership in before. In the future, we will allow `world_readable` rooms to be subscribed to without joining.
* Add `is_dm` room field to Sliding Sync `/sync` (#17429)Eric Eastwood2024-07-111-29/+46
| | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `heroes` and room summary fields to Sliding Sync `/sync` (#17419)Eric Eastwood2024-07-111-83/+197
| | | | | | | Additional room summary fields: `joined_count`, `invited_count` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Fix filtering room types on remote rooms (#17434)Erik Johnston2024-07-111-12/+10
| | | | | | We can only fetch room types for rooms the server is in, so we need to only filter rooms that we're joined to. Also includes a perf fix to bulk fetch room types.
* Handle to-device extensions to Sliding Sync (#17416)Erik Johnston2024-07-101-1/+102
| | | | | | | Implements MSC3885 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Add `rooms` `name` and `avatar` to Sliding Sync `/sync` (#17418)Eric Eastwood2024-07-091-55/+96
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `rooms.bump_stamp` to Sliding Sync `/sync` for easier client-side ↵Eric Eastwood2024-07-081-19/+56
| | | | | | | | | | | | | | | sorting (#17395) `bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves. Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies). In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types. --- In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for. The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622
* Add `rooms.required_state` to Sliding Sync `/sync` (#17342)Eric Eastwood2024-07-041-53/+379
| | | Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.
* Add `room_types`/`not_room_types` filtering to Sliding Sync `/sync` (#17337)Eric Eastwood2024-07-021-5/+30
| | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Return some room data in Sliding Sync `/sync` (#17320)Eric Eastwood2024-07-021-152/+490
| | | | | | - Timeline events - Stripped `invite_state` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `is_invite` filtering to Sliding Sync `/sync` (#17335)Eric Eastwood2024-06-241-3/+16
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `is_encrypted` filtering to Sliding Sync `/sync` (#17281)Eric Eastwood2024-06-171-3/+23
| | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `stream_ordering` sort to Sliding Sync `/sync` (#17293)Eric Eastwood2024-06-171-29/+129
| | | Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
* Add `is_dm` filtering to Sliding Sync `/sync` (#17277)Eric Eastwood2024-06-131-6/+112
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Fix `newly_left` rooms not appearing if we returned early (Sliding Sync) ↵Eric Eastwood2024-06-131-13/+13
| | | | | | | | | | | (#17301) Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync) The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`. Found while working on https://github.com/element-hq/synapse/pull/17293
* Reorganize Pydantic models and types used in handlers (#17279)Eric Eastwood2024-06-101-172/+3
| | | | | | Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues. - Moved Pydantic parsing models to `synapse/types/rest` - Moved handler types to `synapse/types/handlers`
* Add Sliding Sync `/sync` endpoint (initial implementation) (#17187)Eric Eastwood2024-06-061-0/+610
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering). Rooms appear in the Sliding sync response based on: - `invite`, `join`, `knock`, `ban` membership events - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`) - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`) - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range. ### Example request `POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync` ```json { "lists": { "foo-list": { "ranges": [ [0, 99] ], "sort": [ "by_notification_level", "by_recency", "by_name" ], "required_state": [ ["m.room.join_rules", ""], ["m.room.history_visibility", ""], ["m.space.child", "*"] ], "timeline_limit": 100 } } } ``` Response: ```json { "next_pos": "s58_224_0_13_10_1_1_16_0_1", "lists": { "foo-list": { "count": 1, "ops": [ { "op": "SYNC", "range": [0, 99], "room_ids": [ "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server" ] } ] } }, "rooms": {}, "extensions": {} } ```
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-09-19 23:48:06 +0000