| Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
Follow-up to https://github.com/element-hq/synapse/pull/17631 and
https://github.com/element-hq/synapse/pull/17632 to fix-up
https://github.com/element-hq/synapse/pull/17599
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
tables (#17635)
Fix outlier re-persisting causing problems with sliding sync tables
Follow-up to https://github.com/element-hq/synapse/pull/17512
When running on `matrix.org`, we discovered that a remote invite is
first persisted as an `outlier` and then re-persisted again where it is
de-outliered. The first the time, the `outlier` is persisted with one
`stream_ordering` but when persisted again and de-outliered, it is
assigned a different `stream_ordering` that won't end up being used.
Since we call `_calculate_sliding_sync_table_changes()` before
`_update_outliers_txn()` which fixes this discrepancy (always use the
`stream_ordering` from the first time it was persisted), we're working
with an unreliable `stream_ordering` value that will possibly be unused
and not make it into the `events` table.
|
|
This reverts commit
https://github.com/element-hq/synapse/commit/ab414f2ab8a294fbffb417003eeea0f14bbd6588.
Introduced in https://github.com/element-hq/synapse/pull/17512
|
|
This reverts commit ab414f2ab8a294fbffb417003eeea0f14bbd6588.
Introduced in https://github.com/element-hq/synapse/pull/17599
|
|
Based on #17600
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
This is so that we can cache it.
We also move the sliding sync types to
`synapse/types/handlers/sliding_sync.py`. This is mainly in-prep for
#17599 to avoid circular imports.
The only change in behaviour is that
`RoomSyncConfig.combine_sync_config(..)` now returns a new room sync
config rather than mutating in-place.
Reviewable commit-by-commit.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
Pre-populate room data for quick filtering/sorting in the Sliding Sync
API
Spawning from
https://github.com/element-hq/synapse/pull/17450#discussion_r1697335578
This PR is acting as the Synapse version `N+1` step in the gradual
migration being tracked by
https://github.com/element-hq/synapse/issues/17623
Adding two new database tables:
- `sliding_sync_joined_rooms`: A table for storing room meta data that
the local server is still participating in. The info here can be shared
across all `Membership.JOIN`. Keyed on `(room_id)` and updated when the
relevant room current state changes or a new event is sent in the room.
- `sliding_sync_membership_snapshots`: A table for storing a snapshot of
room meta data at the time of the local user's membership. Keyed on
`(room_id, user_id)` and only updated when a user's membership in a room
changes.
Also adds background updates to populate these tables with all of the
existing data.
We want to have the guarantee that if a row exists in the sliding sync
tables, we are able to rely on it (accurate data). And if a row doesn't
exist, we use a fallback to get the same info until the background
updates fill in the rows or a new event comes in triggering it to be
fully inserted. This means we need a couple extra things in place until
we bump `SCHEMA_COMPAT_VERSION` and run the foreground update in the
`N+2` part of the gradual migration. For context on why we can't rely on
the tables without these things see [1].
1. On start-up, block until we clear out any rows for the rooms that
have had events since the max-`stream_ordering` of the
`sliding_sync_joined_rooms` table (compare to max-`stream_ordering` of
the `events` table). For `sliding_sync_membership_snapshots`, we can
compare to the max-`stream_ordering` of `local_current_membership`
- This accounts for when someone downgrades their Synapse version and
then upgrades it again. This will ensure that we don't have any
stale/out-of-date data in the
`sliding_sync_joined_rooms`/`sliding_sync_membership_snapshots` tables
since any new events sent in rooms would have also needed to be written
to the sliding sync tables. For example a new event needs to bump
`event_stream_ordering` in `sliding_sync_joined_rooms` table or some
state in the room changing (like the room name). Or another example of
someone's membership changing in a room affecting
`sliding_sync_membership_snapshots`.
1. Add another background update that will catch-up with any rows that
were just deleted from the sliding sync tables (based on the activity in
the `events`/`local_current_membership`). The rooms that need
recalculating are added to the
`sliding_sync_joined_rooms_to_recalculate` table.
1. Making sure rows are fully inserted. Instead of partially inserting,
we need to check if the row already exists and fully insert all data if
not.
All of this extra functionality can be removed once the
`SCHEMA_COMPAT_VERSION` is bumped with support for the new sliding sync
tables so people can no longer downgrade (the `N+2` part of the gradual
migration).
<details>
<summary><sup>[1]</sup></summary>
For `sliding_sync_joined_rooms`, since we partially insert rows as state
comes in, we can't rely on the existence of the row for a given
`room_id`. We can't even rely on looking at whether the background
update has finished. There could still be partial rows from when someone
reverted their Synapse version after the background update finished, had
some state changes (or new rooms), then upgraded again and more state
changes happen leaving a partial row.
For `sliding_sync_membership_snapshots`, we insert items as a whole
except for the `forgotten` column ~~so we can rely on rows existing and
just need to always use a fallback for the `forgotten` data. We can't
use the `forgotten` column in the table for the same reasons above about
`sliding_sync_joined_rooms`.~~ We could have an out-of-date membership
from when someone reverted their Synapse version. (same problems as
outlined for `sliding_sync_joined_rooms` above)
Discussed in an [internal
meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7)
</details>
### TODO
- [x] Update `stream_ordering`/`bump_stamp`
- [x] Handle remote invites
- [x] Handle state resets
- [x] Consider adding `sender` so we can filter `LEAVE` memberships and
distinguish from kicks.
- [x] We should add it to be able to tell leaves from kicks
- [x] Consider adding `tombstone` state to help address
https://github.com/element-hq/synapse/issues/17540
- [x] We should add it `tombstone_successor_room_id`
- [x] Consider adding `forgotten` status to avoid extra
lookup/table-join on `room_memberships`
- [x] We should add it
- [x] Background update to fill in values for all joined rooms and
non-join membership
- [x] Clean-up tables when room is deleted
- [ ] Make sure tables are useful to our use case
- First explored in
https://github.com/element-hq/synapse/compare/erikj/ss_use_new_tables
- Also explored in
https://github.com/element-hq/synapse/commit/76b5a576eb363496315dfd39510cad7d02b0fc73
- [x] Plan for how can we use this with a fallback
- See plan discussed above in main area of the issue description
- Discussed in an [internal
meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7)
- [x] Plan for how we can rely on this new table without a fallback
- Synapse version `N+1`: (this PR) Bump `SCHEMA_VERSION` to `87`. Add
new tables and background update to backfill all rows. Since this is a
new table, we don't have to add any `NOT VALID` constraints and validate
them when the background update completes. Read from new tables with a
fallback in cases where the rows aren't filled in yet.
- Synapse version `N+2`: Bump `SCHEMA_VERSION` to `88` and bump
`SCHEMA_COMPAT_VERSION` to `87` because we don't want people to
downgrade and miss writes while they are on an older version. Add a
foreground update to finish off the backfill so we can read from new
tables without the fallback. Application code can now rely on the new
tables being populated.
- Discussed in an [internal
meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.hh7shg4cxdhj)
### Dev notes
```
SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase
SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase
```
```
SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.handlers.test_sliding_sync.FilterRoomsTestCase
```
Reference:
- [Development docs on background updates and worked examples of gradual
migrations
](https://github.com/element-hq/synapse/blob/1dfa59b238cee0dc62163588cc9481896c288979/docs/development/database_schema.md#background-updates)
- A real example of a gradual migration:
https://github.com/matrix-org/synapse/pull/15649#discussion_r1213779514
- Adding `rooms.creator` field that needed a background update to
backfill data, https://github.com/matrix-org/synapse/pull/10697
- Adding `rooms.room_version` that needed a background update to
backfill data, https://github.com/matrix-org/synapse/pull/6729
- Adding `room_stats_state.room_type` that needed a background update to
backfill data, https://github.com/matrix-org/synapse/pull/13031
- Tables from MSC2716: `insertion_events`, `insertion_event_edges`,
`insertion_event_extremities`, `batch_events`
- `current_state_events` updated in
`synapse/storage/databases/main/events.py`
---
```
persist_event (adds to queue)
_persist_event_batch
_persist_events_and_state_updates (assigns `stream_ordering` to events)
_persist_events_txn
_store_event_txn
_update_metadata_tables_txn
_store_room_members_txn
_update_current_state_txn
```
---
> Concatenated Indexes [...] (also known as multi-column, composite or
combined index)
>
> [...] key consists of multiple columns.
>
> We can take advantage of the fact that the first index column is
always usable for searching
>
> *--
https://use-the-index-luke.com/sql/where-clause/the-equals-operator/concatenated-keys*
---
Dealing with `portdb` (`synapse/_scripts/synapse_port_db.py`),
https://github.com/element-hq/synapse/pull/17512#discussion_r1725998219
---
<details>
<summary>SQL queries:</summary>
Both of these are equivalent and work in SQLite and Postgres
Options 1:
```sql
WITH data_table (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) AS (
VALUES (
?, ?, ?,
(SELECT membership FROM room_memberships WHERE event_id = ?),
(SELECT stream_ordering FROM events WHERE event_id = ?),
{", ".join("?" for _ in insert_values)}
)
)
INSERT INTO sliding_sync_non_join_memberships
(room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)})
SELECT * FROM data_table
WHERE membership != ?
ON CONFLICT (room_id, user_id)
DO UPDATE SET
membership_event_id = EXCLUDED.membership_event_id,
membership = EXCLUDED.membership,
event_stream_ordering = EXCLUDED.event_stream_ordering,
{", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)}
```
Option 2:
```sql
INSERT INTO sliding_sync_non_join_memberships
(room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)})
SELECT
column1 as room_id,
column2 as user_id,
column3 as membership_event_id,
column4 as membership,
column5 as event_stream_ordering,
{", ".join("column" + str(i) for i in range(6, 6 + len(insert_keys)))}
FROM (
VALUES (
?, ?, ?,
(SELECT membership FROM room_memberships WHERE event_id = ?),
(SELECT stream_ordering FROM events WHERE event_id = ?),
{", ".join("?" for _ in insert_values)}
)
) as v
WHERE membership != ?
ON CONFLICT (room_id, user_id)
DO UPDATE SET
membership_event_id = EXCLUDED.membership_event_id,
membership = EXCLUDED.membership,
event_stream_ordering = EXCLUDED.event_stream_ordering,
{", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)}
```
If we don't need the `membership` condition, we could use:
```sql
INSERT INTO sliding_sync_non_join_memberships
(room_id, membership_event_id, user_id, membership, event_stream_ordering, {", ".join(insert_keys)})
VALUES (
?, ?, ?,
(SELECT membership FROM room_memberships WHERE event_id = ?),
(SELECT stream_ordering FROM events WHERE event_id = ?),
{", ".join("?" for _ in insert_values)}
)
ON CONFLICT (room_id, user_id)
DO UPDATE SET
membership_event_id = EXCLUDED.membership_event_id,
membership = EXCLUDED.membership,
event_stream_ordering = EXCLUDED.event_stream_ordering,
{", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)}
```
</details>
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Erik Johnston <erik@matrix.org>
|
|
|
|
Regressed in #17543.
The `max_download_size` config is not available on workers that don't
load the media repo.
Besides, we should honour the max_size param that was passed into the
function.
|
|
|
|
|
|
When returning receipts in sliding sync for initial rooms we should
always include our own receipts in the room (even if they don't match
any timeline events).
Reviewable commit-by-commit.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
Otherwise we use the `expected_size` from the initial federation
request, which might be far too low.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Erik Johnston <erikj@element.io>
|
|
Move calculating of the room lists out of the core handler. This should
make it easier to switch things around to start using the tables in
#17512.
This is just moving code between files and methods.
Reviewable commit-by-commit
|
|
|
|
|
|
|
|
`hash_password` now actually accepts password from stdin. The `getpass`
reads from TTY, and does NOT accept stdin in any way.
The manpage has been updated to reflect that.
|
|
|
|
|
|
|
|
|
|
|
|
The idea is to engineer it so that the vast majority of the rooms can
stay in the cache, so we can just ignore them.
|
|
The decorator assumed we were always wrapping function methods
|
|
|
|
|
|
|
|
|
|
|
|
|
|
That file was getting long.
The changes are non functional, and simply split things up into:
- the main class
- the connection store
- the extensions
- the types
|
|
|
|
|
|
Let's only pull out the events we care about. Note that the index isn't
necessary here, as postgres is happy to scan the set of rooms for the
events.
|
|
This should let us see how quickly we actually process things in
practice.
|
|
This supersedes #17503, given the per-connection state is being heavily
rewritten it felt easier to recreate the PR on top of that work.
This correctly handles the case of timeline limits going up and down.
This does not handle changes in `required_state`, but that can be done
as a separate PR.
Based on #17575.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
This is to make initial sliding sync a bit faster
|
|
down. (#17575)
Add connection tracking to the receipts extension.
Based on #17574
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
This is some prep work ahead of correctly tracking receipts, where we
will also want to track the room status in terms of last receipt we had
sent down.
Essentially, we add two classes `PerConnectionState` and a mutable
version, and then operate on those.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
Results in:
```
AssertionError: null
File "synapse/http/server.py", line 332, in _async_render_wrapper
callback_return = await self._async_render(request)
File "synapse/http/server.py", line 544, in _async_render
callback_return = await raw_callback_return
File "synapse/federation/transport/server/_base.py", line 369, in new_func
response = await func(
File "synapse/federation/transport/server/federation.py", line 826, in on_GET
await self.media_repo.get_local_media(
File "synapse/media/media_repository.py", line 473, in get_local_media
await respond_with_multipart_responder(
File "synapse/media/_base.py", line 353, in respond_with_multipart_responder
assert content_length is not None
```
|
|
This stops people from getting half way through a step and it failing
due to the github token having expired (this happens to me every damn
time).
|
|
Follow on from #17537.
This is just adding a batched lookup function (you might want to hide
whitespace in the diff).
|
|
So that clients can check for support. Note that if the feature is only
enabled for some users, the `/versions` request must be authenticated to
pick up that SSS is enabled for the user
|
|
|
|
`old_verify_keys` isn't marked as required in
https://spec.matrix.org/v1.11/server-server-api/#get_matrixkeyv2server
and there's no functional difference between an empty object and
omitting the object, so I don't think there's any reason synapse should
explode when the field is omitted.
|
|
Follows on from #17567
|
|
Follow on from #17558
Basically, we want to reduce the number of threads we want to use at a
time, i.e. reduce the number of threads that are paused/blocked. We do
this by returning from the thread when the consumer pauses the producer,
rather than pausing in the thread.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
(#17538)
Previously, we just had very basic partial room exclusion based on
whether we were lazy-loading room members. Now with this PR, we added
`must_await_full_state(...)` with rules to check if we have a we're only
requesting `required_state` which is completely satisfied even with
partial state.
Partially-stated rooms should have all state events except for remote
membership events so if we require a remote membership event anywhere,
then we need to return `True`.
|
|
|
|
|
|
|
|
This is really spurious and causes a lot of spam. I don't think there is
a use for it even at DEBUG level.
|
|
Regression from #17558
|
|
|
|
We do this by reading from a threadpool, rather than blocking the main
thread.
This is broadly what we do in the [S3 storage
provider](https://github.com/matrix-org/synapse-s3-storage-provider/blob/main/s3_storage_provider.py#L234)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
We do this by reusing the code from sync v2.
Reviewable commit-by-commit. The function `get_user_ids_changed` has
been rewritten entirely, so I would recommend not looking at the diff.
|
|
This is in response to issue #17473.
Not all the necessary handlers to deal with media requests are started
now when configuring synapse to use a media worker as per the [example
config](https://element-hq.github.io/synapse/latest/workers.html#synapseappmedia_repository).
The new media endpoints introduced with authenticated media fall under
the `client` & `federation` handlers in synapse.
This PR starts up handlers for the new media endpoints if a worker has
been configured with only the `media` resource type.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
`get_connection_with_tls_context` (#17536)
|
|
Implement tracking of rooms that have had updates that have not been
sent down to clients.
Simplified Sliding Sync (SSS)
|
|
incremental sync (#17510)
Use `stream_ordering` based `timeline` pagination for incremental
`/sync` in Sliding Sync. Previously, we were always using a
`topological_ordering` but we should only be using that for historical
scenarios (initial `/sync`, newly joined, or haven't sent the room down
the connection before).
This is slightly different than what the [spec
suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing)
> Events are ordered in this API according to the arrival time of the
event on the homeserver. This can conflict with other APIs which order
events based on their partial ordering in the event graph. This can
result in duplicate events being received (once per distinct API
called). Clients SHOULD de-duplicate events based on the event ID when
this happens.
But we've had a [discussion below in this
PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569)
and this matches what Sync v2 already does and seems like it makes
sense. Created a spec issue
https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this.
Related issues:
- https://github.com/matrix-org/matrix-spec/issues/1917
- https://github.com/matrix-org/matrix-spec/issues/852
- https://github.com/matrix-org/matrix-spec-proposals/pull/4033
|
|
|
|
|
|
c.f.
https://github.com/element-hq/synapse/pull/17529#discussion_r1705780925
|
|
Spawning from looking at a couple traces and wanting a little more info.
Follow-up to github.com/element-hq/synapse/pull/17501
The changes in this PR allow you to find slow Sliding Sync traces ignoring the
`wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user`
operation with `RESULT.result=true` indicating that it actually returned non-empty results.
If you want to find traces for your own user, you can use
`RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This triggers the client to start a new sliding sync connection. If we
don't do this and the client asks for the full range of rooms, we end up
sending down all rooms and their state from scratch (which can be very
slow)
This causes things like
https://github.com/element-hq/element-x-ios/issues/3115 after we restart
the server
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
Added `_assertTimelineEqual(...)` because I got fed up trying to
understand the crazy diffs from the standard
`self.assertEqual(...)`/`self.assertListEqual(...)`
Before:
```
[FAIL]
Traceback (most recent call last):
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync
self.assertListEqual(
File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual
self.assertSequenceEqual(list1, list2, msg, seq_type=list)
File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual
self.fail(msg)
twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU']
First differing element 0:
'$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA'
'$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E'
- ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA',
- '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E',
? ^
+ ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E',
? ^
- '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM']
+ '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA',
+ '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU']
```
After:
```
[FAIL]
Traceback (most recent call last):
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync
self._assertTimelineEqual(
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual
self._assertListEqual(
File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual
self.fail(f"{diff_message}\n{message}")
twisted.trial.unittest.FailTest: Items must
Expected items to be in actual ('?' = missing expected items):
[
(10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4
(11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5
? (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join
]
Actual ('+' = found expected items):
[
+ (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5
+ (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4
(9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3
]
```
|
|
[MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
|
Split and move Sliding Sync tests so we have some more sane test file
sizes
|
|
|
|
Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to
be robust when dealing with remote invite rooms in Sliding Sync.
Part of
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
Follow-up to https://github.com/element-hq/synapse/pull/17434
We now take into account current state, fallback to stripped state
for invite/knock rooms, then historical state. If we can't determine
the info needed to filter a room (either from state or stripped state),
it is filtered out.
|
|
[MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
|
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
|
|
I also update the tests and HTTP Proxy code to fix it for this new
Twisted release.
Pulls in fix for
https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
|
|
|
|
|
I also update the tests and HTTP Proxy code to fix it for this new
Twisted release.
Pulls in fix for
https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
|
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
This will make it easier to see what it is doing in jaeger.
|
|
Fixes: https://github.com/element-hq/synapse/issues/17498
Fixes: https://github.com/element-hq/element-web/issues/27867
|
|
Rather than always including all rooms in range.
Also adds a pre-filter to rooms that checks the stream change cache to
see if anything might have happened.
Based on #17447
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
The basic idea is that we introduce a new token for a sliding sync
connection, which stores the mapping of room to room "status" (i.e. have
we sent the room down?). This token allows us to handle duplicate
requests properly. In future it can be used to store more
"per-connection" information safely.
In future this should be migrated into the DB, so its important that we
try to reduce the number of syncs where we need to update the
per-connection information. In this PoC this only happens when we: a)
send down a set of room for the first time, or b) we have previously
sent down a room and there are updates but we are not sending the room
down the sync (due to not falling in a list range)
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
`SlidingSyncBase.do_sync(...)` (pt. 2) (#17482)
`SlidingSyncBase.do_sync()` for tests was first introduced in
https://github.com/element-hq/synapse/pull/17452
Part 1: https://github.com/element-hq/synapse/pull/17481
|
|
(#17481)
`SlidingSyncBase` for tests was first introduced in
https://github.com/element-hq/synapse/pull/17452
Part 2: https://github.com/element-hq/synapse/pull/17482
|
|
allowed_local_3pids (#17476)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
Extensions based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
|
Backfill events have a negative stream ordering, and so its not useful
to use to compare with other (positive) stream orderings.
Plus, the Rust SDK currently assumes `bump_stamp` is positive.
|
|
This is in preparation for adding per-connection state.
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
Update debian template - new link to the delegation docs
|
|
|
|
|
|
We do this by bulk fetching the latest stream ordering.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
v1.109.0. (#17428)
Introduced in: #17215
This caused us a minor bit of grief as the volume of logs produced was
much higher than normal
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
|
This is to address an issue in which `m.presence` results on initial
sync are not returning entries of users who are currently offline.
The original behaviour was from
https://github.com/element-hq/synapse/issues/1535
This change is useful for applications that use the
presence system for tracking user profile information/updates (e.g.
https://github.com/element-hq/synapse/pull/16992 or for profile status
messages).
This is gated behind a new configuration option to avoid performance
impact for applications that don't need this, as a pragmatic solution
for now.
|
|
Bumps [hiredis](https://github.com/redis/hiredis-py) from 2.3.2 to
3.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/redis/hiredis-py/releases">hiredis's
releases</a>.</em></p>
<blockquote>
<h2>3.0.0</h2>
<h1>Changes</h1>
<h2>Breaking Changes</h2>
<ul>
<li>Return Redis sets as Python lists (<a
href="https://redirect.github.com/redis/hiredis-py/issues/189">#189</a>)</li>
</ul>
<h2>🐛 Bug Fixes</h2>
<ul>
<li>Return Redis sets as Python lists (<a
href="https://redirect.github.com/redis/hiredis-py/issues/189">#189</a>)</li>
</ul>
<h2>Contributors</h2>
<p>We'd like to thank all the contributors who worked on this
release!</p>
<p><a href="https://github.com/gerzse"><code>@gerzse</code></a></p>
<h2>2.4.0</h2>
<h1>Changes</h1>
<h2>🧰 Maintenance</h2>
<ul>
<li>Fix small typo (<a
href="https://redirect.github.com/redis/hiredis-py/issues/192">#192</a>)</li>
<li>Quote version for Python setup action in CI (<a
href="https://redirect.github.com/redis/hiredis-py/issues/191">#191</a>)</li>
<li>Fix building the wheel for windows (<a
href="https://redirect.github.com/redis/hiredis-py/issues/190">#190</a>)</li>
<li>pack: Replace sdsalloc.h with alloc.h (<a
href="https://redirect.github.com/redis/hiredis-py/issues/159">#159</a>)</li>
<li>Bump black from 22.3.0 to 24.3.0 (<a
href="https://redirect.github.com/redis/hiredis-py/issues/185">#185</a>)</li>
<li>Removing python 3.7 trove (<a
href="https://redirect.github.com/redis/hiredis-py/issues/181">#181</a>)</li>
<li>Badge for latest released on Pypi (<a
href="https://redirect.github.com/redis/hiredis-py/issues/182">#182</a>)</li>
<li>Sync license in metadata with LICENSE file (<a
href="https://redirect.github.com/redis/hiredis-py/issues/183">#183</a>)</li>
</ul>
<h2>Contributors</h2>
<p>We'd like to thank all the contributors who worked on this
release!</p>
<p><a href="https://github.com/Apteryks"><code>@Apteryks</code></a>, <a
href="https://github.com/ArtemIsmagilov"><code>@ArtemIsmagilov</code></a>,
<a href="https://github.com/chayim"><code>@chayim</code></a>, <a
href="https://github.com/dependabot"><code>@dependabot</code></a>, <a
href="https://github.com/dependabot"><code>@dependabot</code></a>[bot],
<a href="https://github.com/gerzse"><code>@gerzse</code></a> and <a
href="https://github.com/shadchin"><code>@shadchin</code></a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/redis/hiredis-py/commit/c1eefbdb76614435f7433207bf385ba8cb930b60"><code>c1eefbd</code></a>
Return Redis sets as Python lists (<a
href="https://redirect.github.com/redis/hiredis-py/issues/189">#189</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/a94bb447173c60b90709a9ed117b3a5b699ff795"><code>a94bb44</code></a>
Version 2.4.0 (<a
href="https://redirect.github.com/redis/hiredis-py/issues/193">#193</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/7792dd23384aa7b00761df51d8011fe24fa33143"><code>7792dd2</code></a>
Fix a typo in the README file (<a
href="https://redirect.github.com/redis/hiredis-py/issues/192">#192</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/01fa2fd6f123e3424ffa00a647f2bf83d48543be"><code>01fa2fd</code></a>
Quote version for Python setup action in CI (<a
href="https://redirect.github.com/redis/hiredis-py/issues/191">#191</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/4c970a336567223573c700a44e405a0c263a85fa"><code>4c970a3</code></a>
Fix building the wheel for windows (<a
href="https://redirect.github.com/redis/hiredis-py/issues/190">#190</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/f4dd0814c16dc9a8efa72434101d49c97778c830"><code>f4dd081</code></a>
pack: Replace sdsalloc.h with alloc.h (<a
href="https://redirect.github.com/redis/hiredis-py/issues/159">#159</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/e70af5b94f0881f23f2fb35417bb2e0d0b792f53"><code>e70af5b</code></a>
Bump black from 22.3.0 to 24.3.0 (<a
href="https://redirect.github.com/redis/hiredis-py/issues/185">#185</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/cc239705fb64f92c6ac3aff36679a300662e0ca7"><code>cc23970</code></a>
Removing Python 3.7 trove (<a
href="https://redirect.github.com/redis/hiredis-py/issues/181">#181</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/64e3394548fe670e7853a2407799e13daa4bf2cb"><code>64e3394</code></a>
Badge for latest released on Pypi (<a
href="https://redirect.github.com/redis/hiredis-py/issues/182">#182</a>)</li>
<li><a
href="https://github.com/redis/hiredis-py/commit/ba18089d74f991b9b9e7ac9c17f85432f50d0048"><code>ba18089</code></a>
Sync license in metadata with LICENSE file (<a
href="https://redirect.github.com/redis/hiredis-py/issues/183">#183</a>)</li>
<li>See full diff in <a
href="https://github.com/redis/hiredis-py/compare/v2.3.2...v3.0.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
Bumps [pyopenssl](https://github.com/pyca/pyopenssl) from 24.1.0 to
24.2.1.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pyca/pyopenssl/blob/main/CHANGELOG.rst">pyopenssl's
changelog</a>.</em></p>
<blockquote>
<h2>24.2.1 (2024-07-20)</h2>
<p>Backward-incompatible changes:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</p>
<p>Deprecations:
^^^^^^^^^^^^^</p>
<p>Changes:
^^^^^^^^</p>
<ul>
<li>Fixed changelog to remove sphinx specific restructured text
strings.</li>
</ul>
<h2>24.2.0 (2024-07-20)</h2>
<p>Backward-incompatible changes:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</p>
<p>Deprecations:
^^^^^^^^^^^^^</p>
<ul>
<li>Deprecated <code>OpenSSL.crypto.X509Req</code>,
<code>OpenSSL.crypto.load_certificate_request</code>,
<code>OpenSSL.crypto.dump_certificate_request</code>. Instead,
<code>cryptography.x509.CertificateSigningRequest</code>,
<code>cryptography.x509.CertificateSigningRequestBuilder</code>,
<code>cryptography.x509.load_der_x509_csr</code>, or
<code>cryptography.x509.load_pem_x509_csr</code> should be used.</li>
</ul>
<p>Changes:
^^^^^^^^</p>
<ul>
<li>Added type hints for the <code>SSL</code> module.
<code>[#1308](https://github.com/pyca/pyopenssl/issues/1308)
<https://github.com/pyca/pyopenssl/pull/1308></code>_.</li>
<li>Changed <code>OpenSSL.crypto.PKey.from_cryptography_key</code> to
accept public and private EC, ED25519, ED448 keys.
<code>[#1310](https://github.com/pyca/pyopenssl/issues/1310)
<https://github.com/pyca/pyopenssl/pull/1310></code>_.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pyca/pyopenssl/commit/8dd94578656d47f854c087783b72431aaeb73d8e"><code>8dd9457</code></a>
24.2.1 (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1320">#1320</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/19f093e0c3db5d7354df0a5256344d9ebc5d931f"><code>19f093e</code></a>
make changelog vanilla rst (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1319">#1319</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/e265b2867bbc4f45a14b31affe4a0be710d2b293"><code>e265b28</code></a>
Prepare for 24.2.0 release (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1318">#1318</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/6943ee524e511138dd78fde6000e6f8e2bdd4b0c"><code>6943ee5</code></a>
Deprecate CSR support in pyOpenSSL (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1316">#1316</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/01b9b5637318bde2490e3f8bb243723546dbcc62"><code>01b9b56</code></a>
Add more type definitions for <code>SSL</code> module, check with mypy
(<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1313">#1313</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/cdcb48baf7d5dd0f040f244934c2967b04b9f4bc"><code>cdcb48b</code></a>
Prune redundant <code>:rtype:</code> from SSL module (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1315">#1315</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/b86914d37fc6c0e138e2a0f733dd65dec2efc96c"><code>b86914d</code></a>
Fix <code>ruff</code> invocation (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1314">#1314</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/caa1ab3ac5c5096b20b69fac2e42cb67c6f716d1"><code>caa1ab3</code></a>
Update changelog for PR <a
href="https://redirect.github.com/pyca/pyopenssl/issues/1308">#1308</a>
and <a
href="https://redirect.github.com/pyca/pyopenssl/issues/1310">#1310</a>
(<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1311">#1311</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/9a2105501f601798bb2fb1928cee5ed8b21e32e4"><code>9a21055</code></a>
Allow loading EC, ED25519, ED448 public keys from cryptography (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1310">#1310</a>)</li>
<li><a
href="https://github.com/pyca/pyopenssl/commit/9eaa1073621dd46b3e721183efc318c95aba0d74"><code>9eaa107</code></a>
Add type annotations for the <code>SSL</code> module (<a
href="https://redirect.github.com/pyca/pyopenssl/issues/1308">#1308</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/pyca/pyopenssl/compare/24.1.0...24.2.1">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
Spec: [MSC3884](https://github.com/matrix-org/matrix-spec-proposals/pull/3884)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|
`$ME` can be used as a substitute for the requester's user ID.
|
|
Linaerizing all access slows things down when devices try and fetch lots
of keys on login
|
|
|
|
|
|
As part of the rollout of
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md)
this PR adds support for designating authenticated media and ensuring
that authenticated media is not served over unauthenticated endpoints.
|
|
As it gets used in sliding sync.
We basically invalidate it in all the same places as
`get_rooms_for_user`. Most of the changes are due to needing the
arguments you pass in to be hashable (which lists aren't)
|
|
This is also what we do for standard `/sync`.
|
|
|
|
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
Reverts element-hq/synapse#17448
We hit a bug when deploying with synctl:
```
Traceback (most recent call last):
File "/home/synapse/env-python311/bin/synctl", line 33, in <module>
sys.exit(load_entry_point('matrix-synapse', 'console_scripts', 'synctl')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python311/bin/synctl", line 25, in importlib_load_entry_point
return next(matches).load()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
module = import_module(match.group('module'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/synapse/src/synapse/_scripts/synctl.py", line 37, in <module>
from synapse.config import find_config_files
File "/home/synapse/src/synapse/config/__init__.py", line 22, in <module>
from ._base import ConfigError, find_config_files
File "/home/synapse/src/synapse/config/_base.py", line 49, in <module>
import pkg_resources
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3282, in <module>
@_call_aside
^^^^^^^^^^^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3266, in _call_aside
f(*args, **kwargs)
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3295, in _initialize_master_working_set
working_set = _declare_state('object', 'working_set', WorkingSet._build_master())
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 589, in _build_master
ws.require(__requires__)
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 926, in require
needed = self.resolve(parse_requirements(requirements))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 787, in resolve
dist = self._resolve_dist(
^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 816, in _resolve_dist
env = Environment(self.entries)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1014, in __init__
self.scan(search_path)
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1046, in scan
for dist in find_distributions(item):
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2091, in find_on_path
yield from factory(fullpath)
^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2183, in resolve_egg_link
return next(dist_groups, ())
^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2179, in <genexpr>
resolved_paths = (
^
File "/home/synapse/env-python311/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2167, in non_empty_lines
for line in _read_utf8_with_fallback(path).splitlines():
^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name '_read_utf8_with_fallback' is not defined
```
|
|
Add `m.room.create` to default bump event types
This probably helps when no messages have been sent in the room and it
was just created.
|
|
The spec specifically mentions `stream_ordering` but that's a Synapse specific concept. In any case, the essence of the spec is basically the first 5 members of the room which `stream_ordering` accomplishes.
Split off from https://github.com/element-hq/synapse/pull/17419#discussion_r1671342794
## Spec compliance
> This should be the first 5 members of the room, **ordered by stream ordering**, which are joined or invited. The list must never include the client’s own user ID. When no joined or invited members are available, this should consist of the banned and left users.
>
> *-- https://spec.matrix.org/v1.10/client-server-api/#_matrixclientv3sync_roomsummary*
Related to https://github.com/matrix-org/matrix-spec/issues/1334
|
|
|
|
|
|
|
|
Prior to this PR, remote downloads which did not provide a
`content-length` were decremented from the remote download ratelimiter
at the max allowable size, leading to excessive ratelimiting - see
https://github.com/element-hq/synapse/issues/17394.
This PR adds a linearizer to limit concurrent remote downloads to 6 per
IP address, and decrements remote downloads without a `content-length`
from the ratelimiter *after* the download is complete and the response
length is known.
Also adds logic to ensure that responses with a known length respect the
`max_download_size`.
|
|
|
|
This fell out of the authenticated media work - this bit of code masked
a bug but does not break anything when removed, so probably should be
removed.
|
|
This is because we serialized the token wrong if the instance map
contained entries from before the minimum token.
|
|
|
|
|
|
|
|
|
|
|
|
This removes the `enable_media_repo` attribute on the server config in
favour of always using the `can_load_media_repo` in the media config.
This should avoid issues like in #17420 in the future
|
|
Add room subscriptions to Sliding Sync `/sync`
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
Currently, you can only subscribe to rooms you have had *any* membership
in before.
In the future, we will allow `world_readable` rooms to be subscribed to
without joining.
|
|
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
|
Additional room summary fields: `joined_count`, `invited_count`
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
|
We can only fetch room types for rooms the server is in, so we need to
only filter rooms that we're joined to.
Also includes a perf fix to bulk fetch room types.
|
|
This looks like a copy/paste error: the function doesn't reject
anything, but instead allows the action count to go through regardless.
The remainder of the function's documentation appears correct.
|
|
Added RHEL/Rocky install instructions (PyPI). Instructions cover
versions 8 and 9 which are the only supported ones - except for RHEL7
which is now on extended life cycle support phase.
Large part of the guide is for installing Python 3.11 or 3.12. RHEL8
ships with Python 3.6 and RHEL9 ships with 3.9. Newer Python versions
can be installed easily as they don't interfere with OS software that
still relies on the default Python version.
I was first planning to add prerequisites part to the prerequisites
section and then install instructions on the top of the page but that
section is for pre-built packages so it just didn't sound right. So I
just dumped everything to the PyPI section of the page. But suggestions
to change are welcome.
I also didn't combine these with Fedora section. I haven't tested those
packages on RHEL and Fedora ships with Python 3.12 out-of-box.
|
|
Signed-off-by: Joe Groocock <me@frebib.net>
|
|
|
|
|
|
Implements MSC3885
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
|
|
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|
notes (#17421)
|
|
|
|
|
|
Co-authored-by: Andrew Morgan <andrewm@element.io>
|
|
Broke in #17381
|
|
|
|
We need it to work on workers and allow guest access.
Broke by #17392
|
|
|
|
|
|
We don't necessarily have `instance_name` for old events (before we
support multiple event persisters). We treat those as if the
`instance_name` was "master".
---------
Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
|
|
sorting (#17395)
`bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves.
Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies).
In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types.
---
In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for.
The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622
|
|
|
|
|
|
|
|
|
|
Previous: https://github.com/element-hq/synapse/pull/17082
Fixes https://github.com/element-hq/synapse/issues/17402
See https://github.com/element-hq/synapse/issues/17402 for context
**Blocked on https://github.com/element-hq/synapse/pull/17388**
(required for spec compliance)
|
|
This makes it easier to go through an internal endpoint instead of the
public facing URL when introspecting tokens, reducing latency.
|
|
|
|
`_matrix/client/v1/media/thumbnail` endpoint (#17388)
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)
added the endpoints `_matrix/federation/v1/media/thumbnail` and the
authenticated `_matrix/client/v1/media/thumbnail`.
This PR implements those endpoints, along with stabilizing
`_matrix/client/v1/media/config` and
`_matrix/client/v1/media/preview_url`.
Complement tests are at
https://github.com/matrix-org/complement/pull/728
|
|
|
|
Based on #17392
|
|
Follows on from @H-Shay's great work at
https://github.com/matrix-org/synapse/pull/15344 and MSC4026.
Also enables its use for MSC3881, mainly as an easy but concrete example
of how to use it.
|
|
|
|
This can help ensure that the rooms are eventually purged if the other
local users also forget them. Synapse already clears some of the room
information as part of the `_background_remove_left_rooms` background
task, but this doesn't catch `events`, `event_json`, etc.
|
|
I don't think this warrants a changelog?
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
|
|
|
Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.
|
|
extension (#17399)
Changelog entries only get merged if they have the same content and
extension
See
https://github.com/element-hq/synapse/pull/17301#discussion_r1665387218
|
|
|
|
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
|
- Timeline events
- Stripped `invite_state`
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|
|
|
Otherwise they are unbounded.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
Fix bug added in #17386, where we accidentally used `room_key` for the
receipts stream. See first commit.
Reviewable commit-by-commit
|
|
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.
Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).
We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
|
|
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.
Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).
We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
|
|
If we leave the `.so` in place it causes the tests to fail, as it gets
picked up (instead of the newly built .so) and so fails with mismatched
GLIBC errors.
|
|
Sid now defaults to python3.12, and our pinned version of cffi (1.5.1)
does not have wheels for 3.12. This installing cffi to fail as we did
not have the correct libs installed to build from source.
|
|
|
|
|
|
|
|
|
|
Otherwise they are unbounded.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
Follow up to https://github.com/element-hq/synapse/pull/17363, so we can
detect issues with the RST file early on.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
|
Fix bug where we don't get new to-device from remote if they resent a
message we've already persisted and have recorded in the DB twice.
`device_federation_inbox` table doesn't have a unique index, and so we
can race and store an entry in there twice. If we do so then
`simple_select_one_txn` will throw an error due to the query returning
more than one row. We should add an unique index, but it doesn't really
matter so lets just handle the case of multiple rows correctly for now.
|
|
|
|
As per
https://github.com/sphinx-doc/sphinx/issues/3921#issuecomment-315581557,
we need double underscores.
Running `rst2html README.rst > /dev/null` found some more warnings.
|
|
As per
https://github.com/sphinx-doc/sphinx/issues/3921#issuecomment-315581557,
we need double underscores.
Running `rst2html README.rst > /dev/null` found some more warnings.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Simple typo in the docs
|
|
Fixes up #17333, where we failed to actually send less data (the
`DISTINCT` didn't work due to `stream_id` being different).
We fix this by making it so that every device list outbound poke for a
given user ID has the same stream ID. We can't change the query to only
return e.g. max stream ID as the receivers look up the destinations to
send to by doing `SELECT WHERE stream_id = ?`
|
|
Reintroduces https://github.com/element-hq/synapse/pull/17333
Turns out the reason for revert was down two master instances running
|
|
Reverts element-hq/synapse#17333
It looks like master was still sending out replication RDATA with the
old format... somehow
|
|
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: reivilibre <oliverw@matrix.org>
|
|
A simple change to update the docs where default values were missing.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Kim Brose <2803622+HarHarLinks@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
Correct event content field is `EventContentFields.ROOM_TYPE` (`type`) :white_check_mark: , not `room_type` :x:
Spec: https://spec.matrix.org/v1.10/client-server-api/#mroomcreate
|
|
This is #17291 (which got reverted), with some added fixups, and change
so that tests actually pick up the error.
The problem was that we were not calculating any new chain IDs due to a
missing `not` in a condition.
|
|
|
|
Reduce the replication traffic of device lists, by not sending every
destination that needs to be sent the device list update over
replication. Instead a "hosts to send to have been calculated"
notification over replication, and then federation senders read the
destinations from the DB.
For non federation senders this should heavily reduce the impact of a
user in many large rooms changing a device.
|
|
The parse_integer function was previously made to reject negative values by
default in https://github.com/element-hq/synapse/pull/16920, but the
documentation stated otherwise. This fixes the documentation and also:
- Removes explicit negative=False parameters from call sites.
- Brings the negative default of parse_integer_from_args in alignment with
parse_integer.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This reverts commit bdf82efea505c488953b46eb681b5a63c4e9655d (#17291)
This seems to have stopped persisting auth chains for new events, and so
is causing state res to fall back to the slow methods
|
|
As the API is slightly incompatible.
|
|
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
|
|
|
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
|
|
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
|
We calculate the auth chain links outside of the main persist event
transaction to ensure that we do not block other event sending during
the calculation.
|
|
Co-authored-by: Erik Johnston <erikj@element.io>
|
|
|
|
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
|
This adds support for the `via` query parameter from
https://github.com/matrix-org/matrix-spec-proposals/pull/4156.
|
|
|
|
We tried to configure rooms `exclude_rooms_from_sync`. If we do not
quote we get an error.
The example should be valid.
|
|
This changes the release artefacts workflow to use `macos-12` runners
instead of `macos-11`, as the latter will be fully deprecated in a few
days.
This also updates `cibuildwheel` to a newer version, as it would not
'repair' the macOS wheels correctly
The difference is that now instead of outputting a macOS 11+ compatible
wheel, we output a macOS 12+ compatible one. This is fine, as macOS 11
is considered EOL since September 2023.
We can also expect that macOS 12 will be considered EOL in September
2024, as Apple usually supports the last 3 macOS version, and macOS 15
is scheduled to be released around that time.
|
|
|
|
|
|
|
