| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\ |
|
| | | |
|
| |\ \ |
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is in response to issue #17473.
Not all the necessary handlers to deal with media requests are started
now when configuring synapse to use a media worker as per the [example
config](https://element-hq.github.io/synapse/latest/workers.html#synapseappmedia_repository).
The new media endpoints introduced with authenticated media fall under
the `client` & `federation` handlers in synapse.
This PR starts up handlers for the new media endpoints if a worker has
been configured with only the `media` resource type.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |\| |
|
| | |
| |
| |
| |
| |
| | |
Implement tracking of rooms that have had updates that have not been
sent down to clients.
Simplified Sliding Sync (SSS)
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
incremental sync (#17510)
Use `stream_ordering` based `timeline` pagination for incremental
`/sync` in Sliding Sync. Previously, we were always using a
`topological_ordering` but we should only be using that for historical
scenarios (initial `/sync`, newly joined, or haven't sent the room down
the connection before).
This is slightly different than what the [spec
suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing)
> Events are ordered in this API according to the arrival time of the
event on the homeserver. This can conflict with other APIs which order
events based on their partial ordering in the event graph. This can
result in duplicate events being received (once per distinct API
called). Clients SHOULD de-duplicate events based on the event ID when
this happens.
But we've had a [discussion below in this
PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569)
and this matches what Sync v2 already does and seems like it makes
sense. Created a spec issue
https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this.
Related issues:
- https://github.com/matrix-org/matrix-spec/issues/1917
- https://github.com/matrix-org/matrix-spec/issues/852
- https://github.com/matrix-org/matrix-spec-proposals/pull/4033
|
| | |
| |
| |
| | |
c.f.
https://github.com/element-hq/synapse/pull/17529#discussion_r1705780925
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Spawning from looking at a couple traces and wanting a little more info.
Follow-up to github.com/element-hq/synapse/pull/17501
The changes in this PR allow you to find slow Sliding Sync traces ignoring the
`wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user`
operation with `RESULT.result=true` indicating that it actually returned non-empty results.
If you want to find traces for your own user, you can use
`RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This triggers the client to start a new sliding sync connection. If we
don't do this and the client asks for the full range of rooms, we end up
sending down all rooms and their state from scratch (which can be very
slow)
This causes things like
https://github.com/element-hq/element-x-ios/issues/3115 after we restart
the server
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| | | |
|
| | | |
|
| | | |
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| | |
[MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
| | | |
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to
be robust when dealing with remote invite rooms in Sliding Sync.
Part of
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
Follow-up to https://github.com/element-hq/synapse/pull/17434
We now take into account current state, fallback to stripped state
for invite/knock rooms, then historical state. If we can't determine
the info needed to filter a room (either from state or stripped state),
it is filtered out.
|
| | |
| |
| |
| |
| |
| |
| | |
[MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
| | |\ |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
I also update the tests and HTTP Proxy code to fix it for this new
Twisted release.
Pulls in fix for
https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
| |\ \ \ |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | |/ / |
|
| |\| | |
|
| | | |
| | |
| | | |
This will make it easier to see what it is doing in jaeger.
|
| | | |
| | |
| | |
| | | |
Fixes: https://github.com/element-hq/synapse/issues/17498
Fixes: https://github.com/element-hq/element-web/issues/27867
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Rather than always including all rooms in range.
Also adds a pre-filter to rooms that checks the stream change cache to
see if anything might have happened.
Based on #17447
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
The basic idea is that we introduce a new token for a sliding sync
connection, which stores the mapping of room to room "status" (i.e. have
we sent the room down?). This token allows us to handle duplicate
requests properly. In future it can be used to store more
"per-connection" information safely.
In future this should be migrated into the DB, so its important that we
try to reduce the number of syncs where we need to update the
per-connection information. In this PoC this only happens when we: a)
send down a set of room for the first time, or b) we have previously
sent down a room and there are updates but we are not sending the room
down the sync (due to not falling in a list range)
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| | | | |
|
| | | | |
|
| |\ \ \
| | | |
| | | |
| | | | |
'origin/madlittlemods/sliding-sync-receipts-extension' into erikj/ss_hacks
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | |/ / |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| |\ \ \ |
|
| | |\ \ \
| | | | |
| | | | |
| | | | | |
erikj/ss_noop_rooms
|
| | | | | | |
|
| | | |\| | |
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Extensions based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Backfill events have a negative stream ordering, and so its not useful
to use to compare with other (positive) stream orderings.
Plus, the Rust SDK currently assumes `bump_stamp` is positive.
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | |/ / / |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| |\| | | |
|
| | |\| | |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This is in preparation for adding per-connection state.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| | |\ \ \ |
|
| | | | | | |
|
| | | |\| | |
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
We do this by bulk fetching the latest stream ordering.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
v1.109.0. (#17428)
Introduced in: #17215
This caused us a minor bit of grief as the volume of logs produced was
much higher than normal
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
This is to address an issue in which `m.presence` results on initial
sync are not returning entries of users who are currently offline.
The original behaviour was from
https://github.com/element-hq/synapse/issues/1535
This change is useful for applications that use the
presence system for tracking user profile information/updates (e.g.
https://github.com/element-hq/synapse/pull/16992 or for profile status
messages).
This is gated behind a new configuration option to avoid performance
impact for applications that don't need this, as a pragmatic solution
for now.
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Spec: [MSC3884](https://github.com/matrix-org/matrix-spec-proposals/pull/3884)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
| | | | | |
| | | | |
| | | | | |
`$ME` can be used as a substitute for the requester's user ID.
|
| | | | | |
| | | | |
| | | | |
| | | | | |
Linaerizing all access slows things down when devices try and fetch lots
of keys on login
|
| | | | | | |
|
| | | | | |
| | | | |
| | | | | |
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| | |\| | | |
|
| | | |\ \ \ |
|
| | | | | | | |
|
| | | | | | | |
|
| | |\ \ \ \ \ |
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | | |
|
| | | | | | | |
| | | | | | |
| | | | | | | |
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| |\ \ \ \ \ \ \ |
|
| | | |_|_|_|/ /
| |/| | | | | |
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
As part of the rollout of
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md)
this PR adds support for designating authenticated media and ensuring
that authenticated media is not served over unauthenticated endpoints.
|
| | | |_|_|/ /
| |/| | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
As it gets used in sliding sync.
We basically invalidate it in all the same places as
`get_rooms_for_user`. Most of the changes are due to needing the
arguments you pass in to be hashable (which lists aren't)
|
| | | |/ / /
| |/| | |
| | | | | |
This is also what we do for standard `/sync`.
|
| | | | | |
| | | | |
| | | | |
| | | | | |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Add `m.room.create` to default bump event types
This probably helps when no messages have been sent in the room and it
was just created.
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
The spec specifically mentions `stream_ordering` but that's a Synapse specific concept. In any case, the essence of the spec is basically the first 5 members of the room which `stream_ordering` accomplishes.
Split off from https://github.com/element-hq/synapse/pull/17419#discussion_r1671342794
## Spec compliance
> This should be the first 5 members of the room, **ordered by stream ordering**, which are joined or invited. The list must never include the client’s own user ID. When no joined or invited members are available, this should consist of the banned and left users.
>
> *-- https://spec.matrix.org/v1.10/client-server-api/#_matrixclientv3sync_roomsummary*
Related to https://github.com/matrix-org/matrix-spec/issues/1334
|
| | | | | | |
|
| | | | | | |
|
| | | | | |
| | | | |
| | | | |
| | | | | |
This reverts commit d8265b223f0ad96a47d4cb9a85d831c404008c6d.
|
| | | | | | |
|
| |\ \ \ \ \ |
|
| | | |/ / /
| |/| | | |
|
| | | | | | |
|
| |\| | | | |
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | | | | | |
|
| | | |/ /
| |/| | |
|
| | | | | |
|
| | |/ / |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Prior to this PR, remote downloads which did not provide a
`content-length` were decremented from the remote download ratelimiter
at the max allowable size, leading to excessive ratelimiting - see
https://github.com/element-hq/synapse/issues/17394.
This PR adds a linearizer to limit concurrent remote downloads to 6 per
IP address, and decrements remote downloads without a `content-length`
from the ratelimiter *after* the download is complete and the response
length is known.
Also adds logic to ensure that responses with a known length respect the
`max_download_size`.
|
| | | |
| | |
| | |
| | | |
This is because we serialized the token wrong if the instance map
contained entries from before the minimum token.
|
| | | |
| | |
| | |
| | |
| | | |
This removes the `enable_media_repo` attribute on the server config in
favour of always using the `can_load_media_repo` in the media config.
This should avoid issues like in #17420 in the future
|
| |\| | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Add room subscriptions to Sliding Sync `/sync`
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
Currently, you can only subscribe to rooms you have had *any* membership
in before.
In the future, we will allow `world_readable` rooms to be subscribed to
without joining.
|
| | | | |
|
| |\| | |
|
| | | |
| | |
| | |
| | |
| | | |
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
Additional room summary fields: `joined_count`, `invited_count`
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
| | | |
| | |
| | |
| | |
| | |
| | | |
We can only fetch room types for rooms the server is in, so we need to
only filter rooms that we're joined to.
Also includes a perf fix to bulk fetch room types.
|
| | | |
| | |
| | |
| | |
| | | |
This looks like a copy/paste error: the function doesn't reject
anything, but instead allows the action count to go through regardless.
The remainder of the function's documentation appears correct.
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| |\ \ \ |
|
| | |/ / |
|
| |\ \ \ |
|
| | |/ / |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| |/ / |
|
| | |
| |
| |
| |
| |
| |
| | |
Implements MSC3885
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| |\| |
|
| | |
| |
| | |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |/
|
| |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
| |
|
| |
Broke in #17381
|
| |
|
|
|
| |
We need it to work on workers and allow guest access.
Broke by #17392
|
| |
|
|
|
|
|
|
|
| |
We don't necessarily have `instance_name` for old events (before we
support multiple event persisters). We treat those as if the
`instance_name` was "master".
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sorting (#17395)
`bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves.
Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies).
In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types.
---
In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for.
The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622
|
| |
|
|
|
|
|
|
| |
Previous: https://github.com/element-hq/synapse/pull/17082
Fixes https://github.com/element-hq/synapse/issues/17402
See https://github.com/element-hq/synapse/issues/17402 for context
**Blocked on https://github.com/element-hq/synapse/pull/17388**
(required for spec compliance)
|
| |
|
|
| |
This makes it easier to go through an internal endpoint instead of the
public facing URL when introspecting tokens, reducing latency.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
`_matrix/client/v1/media/thumbnail` endpoint (#17388)
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)
added the endpoints `_matrix/federation/v1/media/thumbnail` and the
authenticated `_matrix/client/v1/media/thumbnail`.
This PR implements those endpoints, along with stabilizing
`_matrix/client/v1/media/config` and
`_matrix/client/v1/media/preview_url`.
Complement tests are at
https://github.com/matrix-org/complement/pull/728
|
| | |
|
| |
|
| |
Based on #17392
|
| |
|
|
|
|
|
| |
Follows on from @H-Shay's great work at
https://github.com/matrix-org/synapse/pull/15344 and MSC4026.
Also enables its use for MSC3881, mainly as an easy but concrete example
of how to use it.
|
| |
|
|
|
|
| |
This can help ensure that the rooms are eventually purged if the other
local users also forget them. Synapse already clears some of the room
information as part of the `_background_remove_left_rooms` background
task, but this doesn't catch `events`, `event_json`, etc.
|
| |
|
| |
Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.
|
| |
|
|
|
| |
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
| |
|
|
|
|
| |
- Timeline events
- Stripped `invite_state`
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| | |
Otherwise they are unbounded.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
| |
| |
| |
| |
| | |
Fix bug added in #17386, where we accidentally used `room_key` for the
receipts stream. See first commit.
Reviewable commit-by-commit
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.
Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).
We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.
Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).
We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
|
| | |
| |
| |
| |
| |
| |
| | |
Otherwise they are unbounded.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |/
|
|
|
|
|
|
|
|
| |
Fix bug where we don't get new to-device from remote if they resent a
message we've already persisted and have recorded in the DB twice.
`device_federation_inbox` table doesn't have a unique index, and so we
can race and store an entry in there twice. If we do so then
`simple_select_one_txn` will throw an error due to the query returning
more than one row. We should add an unique index, but it doesn't really
matter so lets just handle the case of multiple rows correctly for now.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Fixes up #17333, where we failed to actually send less data (the
`DISTINCT` didn't work due to `stream_id` being different).
We fix this by making it so that every device list outbound poke for a
given user ID has the same stream ID. We can't change the query to only
return e.g. max stream ID as the receivers look up the destinations to
send to by doing `SELECT WHERE stream_id = ?`
|
| |
|
|
|
|
| |
Reintroduces https://github.com/element-hq/synapse/pull/17333
Turns out the reason for revert was down two master instances running
|
| |
|
|
|
|
| |
Reverts element-hq/synapse#17333
It looks like master was still sending out replication RDATA with the
old format... somehow
|
| |
|
| |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
| |
|
|
|
| |
Correct event content field is `EventContentFields.ROOM_TYPE` (`type`) :white_check_mark: , not `room_type` :x:
Spec: https://spec.matrix.org/v1.10/client-server-api/#mroomcreate
|
| |
|
|
|
|
|
| |
This is #17291 (which got reverted), with some added fixups, and change
so that tests actually pick up the error.
The problem was that we were not calculating any new chain IDs due to a
missing `not` in a condition.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Reduce the replication traffic of device lists, by not sending every
destination that needs to be sent the device list update over
replication. Instead a "hosts to send to have been calculated"
notification over replication, and then federation senders read the
destinations from the DB.
For non federation senders this should heavily reduce the impact of a
user in many large rooms changing a device.
|
| |
|
|
|
|
|
|
|
|
| |
The parse_integer function was previously made to reject negative values by
default in https://github.com/element-hq/synapse/pull/16920, but the
documentation stated otherwise. This fixes the documentation and also:
- Removes explicit negative=False parameters from call sites.
- Brings the negative default of parse_integer_from_args in alignment with
parse_integer.
|
| |
|
|
|
|
| |
This reverts commit bdf82efea505c488953b46eb681b5a63c4e9655d (#17291)
This seems to have stopped persisting auth chains for new events, and so
is causing state res to fall back to the slow methods
|
| |
|
| |
As the API is slightly incompatible.
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
| | |
|
| |
|
|
|
| |
We calculate the auth chain links outside of the main persist event
transaction to ensure that we do not block other event sending during
the calculation.
|
| |
|
|
| |
Co-authored-by: Erik Johnston <erikj@element.io>
|
| | |
|
| |
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
| |
|
|
| |
This adds support for the `via` query parameter from
https://github.com/matrix-org/matrix-spec-proposals/pull/4156.
|
| |
|
|
| |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
| |
|
| |
Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
Rather than forcing the server operator to apply the SQL manually.
This should be safe, as there should be only one writer for these
sequences.
|
| | |
| |
| |
| |
| |
| | |
Per MSC3967, which is now stable, we should not require UIA when
uploading cross-signing keys for the first time.
Fixes: #17227
|
| | |
| |
| |
| |
| | |
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
|
| | |
| |
| | |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
(#17301)
Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`.
Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync)
The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`.
Found while working on https://github.com/element-hq/synapse/pull/17293
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`.
`instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`.
This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293
Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see https://github.com/element-hq/synapse/pull/17187/commits/09609cb0dbca3a4cfd9fbf90cc962e765ec469c0#diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
last event (#17295)
PR where this was introduced: https://github.com/matrix-org/synapse/pull/14817
### What does this affect?
`get_last_event_in_room_before_stream_ordering(...)` is used in Sync v2 in a lot of different state calculations.
`get_last_event_in_room_before_stream_ordering(...)` is also used in `/rooms/{roomId}/members`
|
| | |
| |
| |
| |
| | |
This clarifies in the comments that the MSC is being used in matrix.org
See #17270
|
| | |
| |
| |
| |
| |
| |
| |
| | |
https://github.com/matrix-org/matrix-spec-proposals/pull/4151
This is intended to be enabled by default for immediate use. When FCP is
complete, the unstable endpoint will be dropped and stable endpoint
supported instead - no backwards compatibility is expected for the
unstable endpoint.
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If the stream ID in the unconverted table is ahead of the device lists
ID gen, then it can break all /sync requests that had an ID from ahead
of the table.
The fix is to make sure we add the unconverted table to the list of
tables we check at start up.
Broke in https://github.com/element-hq/synapse/pull/17229
|
| | |
| |
| | |
Broke in https://github.com/element-hq/synapse/pull/17215
|
| | |
| |
| |
| |
| |
| | |
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.
- Moved Pydantic parsing models to `synapse/types/rest`
- Moved handler types to `synapse/types/handlers`
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
`ControlVarUsedAfterBlockViolation` `WPS441`) (#17272)
Fix loop var being used outside block.
Before this change, we were always using the last room_id's retention policy for all events being filtered.
I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning!
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
are replacing other room keys. (#17266)
Fixes: #17013
Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.
The existing opentracing logging is not enough because it is not enabled
for all users.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).
Rooms appear in the Sliding sync response based on:
- `invite`, `join`, `knock`, `ban` membership events
- Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
- `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
- In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.
### Example request
`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`
```json
{
"lists": {
"foo-list": {
"ranges": [ [0, 99] ],
"sort": [ "by_notification_level", "by_recency", "by_name" ],
"required_state": [
["m.room.join_rules", ""],
["m.room.history_visibility", ""],
["m.space.child", "*"]
],
"timeline_limit": 100
}
}
}
```
Response:
```json
{
"next_pos": "s58_224_0_13_10_1_1_16_0_1",
"lists": {
"foo-list": {
"count": 1,
"ops": [
{
"op": "SYNC",
"range": [0, 99],
"room_ids": [
"!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
]
}
]
}
},
"rooms": {},
"extensions": {}
}
```
|
| | |
| |
| | |
And fallback keys uploads. Only device keys need handling on master
|
| | |
| |
| |
| | |
It's just a waste of time if we won't even query the remote host as its
marked as down.
|
| | |
| |
| | |
Broke in https://github.com/element-hq/synapse/pull/17215
|
| | | |
|
| | |
| |
| | |
c.f. #16675
|
| |/
|
|
|
|
|
| |
(#17265)
Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation.
Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
|
| |
|
|
| |
This can happen on restarts of the service, due to old rooms being
pruned.
|
| | |
|
| |
|
|
|
| |
This was broken by the sentry 2.0 upgrade
Broke in v1.108.0
|
| |
|
|
|
|
|
| |
Otherwise things will get confused.
An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
(#17164)
We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.
The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.
Fixes #3791
|
| |
|
|
|
| |
(#17229)
Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`, which is safer.
|
| |
|
|
| |
We started ensuring we only insert valid destinations:
https://github.com/element-hq/synapse/pull/17240
|
| |
|
|
|
|
|
|
| |
Fixes up #17239
We need to keep the spam check within the `try/except` block. Also makes
it so that we don't enter the top span twice.
Also also ensures that we get the right thumbnail length.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is a problem with `StreamIdGenerator` where it can go backwards
over restarts when a stream ID is requested but then not inserted into
the DB. This is problematic if we want to land #17215, and is generally
a potential cause for all sorts of nastiness.
Instead of trying to fix `StreamIdGenerator`, we may as well move to
`MultiWriterIdGenerator` that does not suffer from this problem (the
latest positions are stored in `stream_positions` table). This involves
adding SQLite support to the class.
This only changes id generators that were already using
`MultiWriterIdGenerator` under postgres, a separate PR will move the
rest of the uses of `StreamIdGenerator` over.
|
| |
|
|
| |
This is a tree cache already, so may as well move the room ID to the
front and use that
|
| |
|
|
| |
We relax this as there are use cases where this is safe, though it is
still highly recommended that people avoid using it.
|
| |
|
|
|
|
|
|
| |
Currently sending a to-device message to a user ID with a dodgy
destination is accepted, but then ends up spamming the logs when we try
and send to the destination.
An alternative would be to reject the request, but I'm slightly nervous
that could break things.
|
| |
|
| |
Currently this causes one of then to 500.
|
| |
|
|
|
|
|
| |
When a module rejects a piece of media we end up trying to close the
same logging context twice.
Instead of fixing the existing code we refactor to use an async context
manager, which is easier to write correctly.
|
| |
|
|
|
|
|
|
|
|
| |
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md)
adds new media endpoints under `_matrix/client`. This PR adds the
`/preview_url`, `/config`, and `/thumbnail` endpoints. `/download` will
be added in a follow-up PR once the work for the federation `/download`
endpoint is complete (see
https://github.com/element-hq/synapse/pull/17172).
Should be reviewable commit-by-commit.
|
| |
|
|
|
|
|
|
|
|
|
| |
This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa.
Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought.
Based on:
- https://github.com/matrix-org/matrix-spec-proposals/pull/3575
- https://github.com/matrix-org/matrix-spec-proposals/pull/3885
- https://github.com/matrix-org/matrix-spec-proposals/pull/3884
|
| |
|
|
|
|
|
| |
`auto_join_rooms` option. (#17176)
Would have been useful for tracking down #16878.
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
usage. (#17219)
The log format is the same as the request log format, except:
- fields that are specific to HTTP requests have been removed
- the task's params are included at the end of the log line.
These log lines are emitted:
- when the task function finishes — both completion and failure (and I
suppose it is possible for a task to become schedulable again?)
- every 5 minutes whilst it is running
Closes #17217.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR ports the logic from the
[synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite)
module into synapse.
I went with the naive approach of injecting the "module" next to where
third party modules are currently loaded. If there is a better/preferred
way to handle this, I'm all ears. It wasn't obvious to me if there was a
better location to add this logic that would cleanly apply to all
incoming invite events.
Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter
errors.
|
| |
|
|
|
|
|
|
| |
Re-introduces #17191, and includes #17197 and #17214
The basic idea is to stop calling `get_rooms_for_user` everywhere, and
instead use the table `device_lists_changes_in_room`.
Commits reviewable one-by-one.
|
| |
|
| |
This helps prevent clients from "tight-looping" retrying their request.
|
| |
|
|
|
|
| |
We will re-use `get_sync_result_builder(...)` in
https://github.com/element-hq/synapse/pull/17167
Split out from https://github.com/element-hq/synapse/pull/17167
|
