| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
|
|
| |
This can help ensure that the rooms are eventually purged if the other
local users also forget them. Synapse already clears some of the room
information as part of the `_background_remove_left_rooms` background
task, but this doesn't catch `events`, `event_json`, etc.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I don't think this warrants a changelog?
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
|
|
|
| |
Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.
|
|
|
|
|
|
|
|
|
|
| |
extension (#17399)
Changelog entries only get merged if they have the same content and
extension
See
https://github.com/element-hq/synapse/pull/17301#discussion_r1665387218
|
|
|
|
|
| |
Based on
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync
|
|
|
|
|
|
| |
- Timeline events
- Stripped `invite_state`
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|\ |
|
| | |
|
|\| |
|
| |
| |
| |
| |
| |
| |
| | |
Otherwise they are unbounded.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
| |
| |
| |
| |
| | |
Fix bug added in #17386, where we accidentally used `room_key` for the
receipts stream. See first commit.
Reviewable commit-by-commit
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.
Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).
We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
|
| |
| |
| |
| |
| | |
Sid now defaults to python3.12, and our pinned version of cffi (1.5.1)
does not have wheels for 3.12. This installing cffi to fail as we did
not have the correct libs installed to build from source.
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes https://github.com/element-hq/synapse/issues/17274, hopefully.
Basically, old versions of Synapse could advance streams without
persisting anything in the DB (fixed in #17229). On restart those
updates would get lost, and so the position of the stream would revert
to an older position. If this happened across an upgrade to a later
Synapse version which included #17215, then sync could get blocked
indefinitely (until the stream advanced to the position in the token).
We fix this by bounding the stream positions we'll wait for to the
maximum position of the underlying stream ID generator.
|
| |
| |
| |
| |
| | |
If we leave the `.so` in place it causes the tests to fail, as it gets
picked up (instead of the newly built .so) and so fails with mismatched
GLIBC errors.
|
| |
| |
| |
| |
| |
| |
| | |
Otherwise they are unbounded.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Follow up to https://github.com/element-hq/synapse/pull/17363, so we can
detect issues with the RST file early on.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fix bug where we don't get new to-device from remote if they resent a
message we've already persisted and have recorded in the DB twice.
`device_federation_inbox` table doesn't have a unique index, and so we
can race and store an entry in there twice. If we do so then
`simple_select_one_txn` will throw an error due to the query returning
more than one row. We should add an unique index, but it doesn't really
matter so lets just handle the case of multiple rows correctly for now.
|
|\| |
|
| | |
|
| |
| |
| |
| |
| |
| | |
As per
https://github.com/sphinx-doc/sphinx/issues/3921#issuecomment-315581557,
we need double underscores.
Running `rst2html README.rst > /dev/null` found some more warnings.
|
| | |
|
|/
|
|
|
|
| |
As per
https://github.com/sphinx-doc/sphinx/issues/3921#issuecomment-315581557,
we need double underscores.
Running `rst2html README.rst > /dev/null` found some more warnings.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
| |
Fixes up #17333, where we failed to actually send less data (the
`DISTINCT` didn't work due to `stream_id` being different).
We fix this by making it so that every device list outbound poke for a
given user ID has the same stream ID. We can't change the query to only
return e.g. max stream ID as the receivers look up the destinations to
send to by doing `SELECT WHERE stream_id = ?`
|
|
|
|
|
|
| |
Reintroduces https://github.com/element-hq/synapse/pull/17333
Turns out the reason for revert was down two master instances running
|
|
|
|
|
|
| |
Reverts element-hq/synapse#17333
It looks like master was still sending out replication RDATA with the
old format... somehow
|
|
|
| |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: reivilibre <oliverw@matrix.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A simple change to update the docs where default values were missing.
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Kim Brose <2803622+HarHarLinks@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
|
| |
Correct event content field is `EventContentFields.ROOM_TYPE` (`type`) :white_check_mark: , not `room_type` :x:
Spec: https://spec.matrix.org/v1.10/client-server-api/#mroomcreate
|
|
|
|
|
|
|
| |
This is #17291 (which got reverted), with some added fixups, and change
so that tests actually pick up the error.
The problem was that we were not calculating any new chain IDs due to a
missing `not` in a condition.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Reduce the replication traffic of device lists, by not sending every
destination that needs to be sent the device list update over
replication. Instead a "hosts to send to have been calculated"
notification over replication, and then federation senders read the
destinations from the DB.
For non federation senders this should heavily reduce the impact of a
user in many large rooms changing a device.
|
|
|
|
|
|
|
|
|
|
| |
The parse_integer function was previously made to reject negative values by
default in https://github.com/element-hq/synapse/pull/16920, but the
documentation stated otherwise. This fixes the documentation and also:
- Removes explicit negative=False parameters from call sites.
- Brings the negative default of parse_integer_from_args in alignment with
parse_integer.
|
|
|
|
|
|
| |
This reverts commit bdf82efea505c488953b46eb681b5a63c4e9655d (#17291)
This seems to have stopped persisting auth chains for new events, and so
is causing state res to fall back to the slow methods
|
|
|
| |
As the API is slightly incompatible.
|
|
|
|
| |
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
| |
|
|
|
|
| |
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
|
|
|
|
| |
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
|
|
|
|
| |
We calculate the auth chain links outside of the main persist event
transaction to ensure that we do not block other event sending during
the calculation.
|
|
|
|
| |
Co-authored-by: Erik Johnston <erikj@element.io>
|
| |
|
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
|
|
|
| |
This adds support for the `via` query parameter from
https://github.com/matrix-org/matrix-spec-proposals/pull/4156.
|
|
|
|
|
| |
We tried to configure rooms `exclude_rooms_from_sync`. If we do not
quote we get an error.
The example should be valid.
|
|
|
|
| |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|
|
| |
Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
|
|
|
|
|
|
| |
Per MSC3967, which is now stable, we should not require UIA when
uploading cross-signing keys for the first time.
Fixes: #17227
|
|
|
|
|
| |
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
|
|
|
| |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
|
|
|
|
|
|
|
|
|
|
|
| |
(#17301)
Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`.
Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync)
The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`.
Found while working on https://github.com/element-hq/synapse/pull/17293
|
|
|
|
|
|
|
|
|
|
| |
Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`.
`instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`.
This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293
Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see https://github.com/element-hq/synapse/pull/17187/commits/09609cb0dbca3a4cfd9fbf90cc962e765ec469c0#diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410
|
|
|
|
|
|
|
|
|
|
|
| |
last event (#17295)
PR where this was introduced: https://github.com/matrix-org/synapse/pull/14817
### What does this affect?
`get_last_event_in_room_before_stream_ordering(...)` is used in Sync v2 in a lot of different state calculations.
`get_last_event_in_room_before_stream_ordering(...)` is also used in `/rooms/{roomId}/members`
|
|
|
|
| |
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
|
|
|
| |
This clarifies in the comments that the MSC is being used in matrix.org
See #17270
|
|
|
|
|
|
|
|
| |
https://github.com/matrix-org/matrix-spec-proposals/pull/4151
This is intended to be enabled by default for immediate use. When FCP is
complete, the unstable endpoint will be dropped and stable endpoint
supported instead - no backwards compatibility is expected for the
unstable endpoint.
|
|
|
|
|
|
| |
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.
- Moved Pydantic parsing models to `synapse/types/rest`
- Moved handler types to `synapse/types/handlers`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`ControlVarUsedAfterBlockViolation` `WPS441`) (#17272)
Fix loop var being used outside block.
Before this change, we were always using the last room_id's retention policy for all events being filtered.
I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning!
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
are replacing other room keys. (#17266)
Fixes: #17013
Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.
The existing opentracing logging is not enough because it is not enabled
for all users.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).
Rooms appear in the Sliding sync response based on:
- `invite`, `join`, `knock`, `ban` membership events
- Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
- `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
- In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.
### Example request
`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`
```json
{
"lists": {
"foo-list": {
"ranges": [ [0, 99] ],
"sort": [ "by_notification_level", "by_recency", "by_name" ],
"required_state": [
["m.room.join_rules", ""],
["m.room.history_visibility", ""],
["m.space.child", "*"]
],
"timeline_limit": 100
}
}
}
```
Response:
```json
{
"next_pos": "s58_224_0_13_10_1_1_16_0_1",
"lists": {
"foo-list": {
"count": 1,
"ops": [
{
"op": "SYNC",
"range": [0, 99],
"room_ids": [
"!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
]
}
]
}
},
"rooms": {},
"extensions": {}
}
```
|
|
|
| |
And fallback keys uploads. Only device keys need handling on master
|
|
|
|
| |
It's just a waste of time if we won't even query the remote host as its
marked as down.
|
|
|
| |
Broke in https://github.com/element-hq/synapse/pull/17215
|
| |
|
|
|
| |
c.f. #16675
|
|
|
|
|
|
|
| |
(#17265)
Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation.
Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
|
| |
|
|
|
|
| |
This can happen on restarts of the service, due to old rooms being
pruned.
|
| |
|
|
|
|
|
| |
This was broken by the sentry 2.0 upgrade
Broke in v1.108.0
|
|
|
|
|
|
|
| |
Otherwise things will get confused.
An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#17164)
We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.
The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.
Fixes #3791
|
|
|
|
|
| |
(#17229)
Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`, which is safer.
|
|
|
|
| |
We started ensuring we only insert valid destinations:
https://github.com/element-hq/synapse/pull/17240
|
|
|
|
|
|
|
|
| |
Fixes up #17239
We need to keep the spam check within the `try/except` block. Also makes
it so that we don't enter the top span twice.
Also also ensures that we get the right thumbnail length.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is a problem with `StreamIdGenerator` where it can go backwards
over restarts when a stream ID is requested but then not inserted into
the DB. This is problematic if we want to land #17215, and is generally
a potential cause for all sorts of nastiness.
Instead of trying to fix `StreamIdGenerator`, we may as well move to
`MultiWriterIdGenerator` that does not suffer from this problem (the
latest positions are stored in `stream_positions` table). This involves
adding SQLite support to the class.
This only changes id generators that were already using
`MultiWriterIdGenerator` under postgres, a separate PR will move the
rest of the uses of `StreamIdGenerator` over.
|
|
|
|
| |
This is a tree cache already, so may as well move the room ID to the
front and use that
|
|
|
|
| |
We relax this as there are use cases where this is safe, though it is
still highly recommended that people avoid using it.
|
|
|
|
|
|
|
|
| |
Currently sending a to-device message to a user ID with a dodgy
destination is accepted, but then ends up spamming the logs when we try
and send to the destination.
An alternative would be to reject the request, but I'm slightly nervous
that could break things.
|
|
|
| |
Currently this causes one of then to 500.
|
|
|
|
|
|
|
| |
When a module rejects a piece of media we end up trying to close the
same logging context twice.
Instead of fixing the existing code we refactor to use an async context
manager, which is easier to write correctly.
|
|
|
|
|
|
|
|
|
|
| |
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md)
adds new media endpoints under `_matrix/client`. This PR adds the
`/preview_url`, `/config`, and `/thumbnail` endpoints. `/download` will
be added in a follow-up PR once the work for the federation `/download`
endpoint is complete (see
https://github.com/element-hq/synapse/pull/17172).
Should be reviewable commit-by-commit.
|
|
|
|
|
|
|
|
|
|
|
| |
This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa.
Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought.
Based on:
- https://github.com/matrix-org/matrix-spec-proposals/pull/3575
- https://github.com/matrix-org/matrix-spec-proposals/pull/3885
- https://github.com/matrix-org/matrix-spec-proposals/pull/3884
|
|
|
|
|
|
|
| |
`auto_join_rooms` option. (#17176)
Would have been useful for tracking down #16878.
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
usage. (#17219)
The log format is the same as the request log format, except:
- fields that are specific to HTTP requests have been removed
- the task's params are included at the end of the log line.
These log lines are emitted:
- when the task function finishes — both completion and failure (and I
suppose it is possible for a task to become schedulable again?)
- every 5 minutes whilst it is running
Closes #17217.
---------
Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
| |
Update OIDC documentation: by default Matrix doesn't query userinfo endpoint, then claims should be put on id_token.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR ports the logic from the
[synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite)
module into synapse.
I went with the naive approach of injecting the "module" next to where
third party modules are currently loaded. If there is a better/preferred
way to handle this, I'm all ears. It wasn't obvious to me if there was a
better location to add this logic that would cleanly apply to all
incoming invite events.
Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter
errors.
|
|
|
|
|
|
|
|
| |
Re-introduces #17191, and includes #17197 and #17214
The basic idea is to stop calling `get_rooms_for_user` everywhere, and
instead use the table `device_lists_changes_in_room`.
Commits reviewable one-by-one.
|
| |
|
|
|
| |
This helps prevent clients from "tight-looping" retrying their request.
|
| |
|
|
|
|
|
|
| |
We will re-use `get_sync_result_builder(...)` in
https://github.com/element-hq/synapse/pull/17167
Split out from https://github.com/element-hq/synapse/pull/17167
|
|
|
|
|
|
|
|
| |
This change was introduced in
https://github.com/element-hq/synapse/pull/17203
But then https://github.com/element-hq/synapse/pull/17207 was reverted
which brought back usage `joined_rooms` that needed to be updated.
Wasn't caught because `develop` wasn't up to date before merging.
|
|
|
| |
Split out from https://github.com/element-hq/synapse/pull/17167
|
|
|
|
|
|
|
|
|
| |
function parameter) (#17201)
Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) so it doesn't have to flow into `_generate_sync_entry_for_xxx` methods. This way we can separate the concerns of caching from generating the response and reuse the `_generate_sync_entry_for_xxx` functions as we see fit. Plus caching doesn't really have anything to do with the config of sync.
Split from https://github.com/element-hq/synapse/pull/17167
Spawning from https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
|
|
|
|
| |
(#17195)
|
|
|
| |
Reverts element-hq/synapse#17191
|
|
|
|
|
|
| |
Fixes #16987
Some old accounts seem to have an entry in global account data table for
push rules, which we should ignore
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(`SyncVersion`) (#17200)
Refactor Sync handler to be able to be able to return different sync
responses (`SyncVersion`). Preparation to be able support sync v2 and a
new Sliding Sync `/sync/e2ee` endpoint which returns a subset of sync
v2.
Split upon request:
https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
Split from https://github.com/element-hq/synapse/pull/17167 where we
will add `SyncVersion.E2EE_SYNC` and a new type of sync response.
|
|
|
|
| |
documentation (#17199)
|
|
|
|
| |
The sliding sync proxy (amongst other things) use literal json blobs as
filters, and repeatedly validating them takes a bunch of CPU.
|
|
|
|
| |
For large accounts waking up all the relevant notifier streams can cause
pauses of the reactor.
|
|
|
|
|
| |
It's almost always more efficient to query the rooms that have device
list changes, rather than looking at the list of all users whose devices
have changed and then look for shared rooms.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This is to allow clients to query the configured federation whitelist.
Disabled by default.
---------
Co-authored-by: Devon Hudson <devonhudson@librem.one>
Co-authored-by: devonh <devon.dmytro@gmail.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
|
| |
See #17177.
I'm an idiot and moved them to the wrong store :facepalm:
|
|
|
|
|
|
| |
If room publication is disabled then creating public rooms on workers
would not work.
Introduced in #16811.
|
|
|
|
| |
"push_rules" stream name (#17171)
|
| |
|
|
|
|
|
|
| |
Linter errors are showing up in #17147 that are unrelated to that PR.
The errors do not currently show up on develop.
This PR aims to resolve the linter errors separately from #17147.
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
|
|
|
|
|
|
| |
This version change requires a migration to a new API. See
https://pyo3.rs/v0.21.2/migration#from-020-to-021
This will fix the annoying warnings added when using the recent rust
nightly:
> warning: non-local `impl` definition, they should be avoided as they
go against expectation
|
| |
|
| |
|
|\ |
|
| | |
|
|/ |
|
|
|
|
|
|
|
| |
When there have been lots of changes compared with the number of
entities, we can do a fast(er) path.
Locally I ran some benchmarking, and the comparison seems to give the
best determination of which method we use.
|
|
|
| |
Annoyingly mypy didn't pick up this typo.
|
|
|
| |
Broke in #17073
|
| |
|
|
|
| |
upgrade.md: Bump minimum Rust version to 1.66.0 (element-hq#17079)
|
|
|
|
|
|
| |
This is an alternative to
https://github.com/matrix-org/matrix-rust-sdk/issues/3340 where we don't
need to change our CI setup too much in the Rust SDK repository, and
still can test MSC3266.
|
| |
|
|
|
|
|
|
| |
It is broken, and we only seemed to have been building it for the
federation sender.
Closes https://github.com/element-hq/synapse/issues/16804
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
|
|
|
| |
Follow-up to #17137 and
https://github.com/matrix-org/complement/pull/722
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
selected (#17120)
This change will apply the `email` & `picture` provided by OIDC to the
new user account when registering a new user via OIDC. If the user is
directed to the account details form, this change makes sure they have
been selected before applying them, otherwise they are omitted. In
particular, this change ensures the values are carried through when
Synapse has consent configured, and the redirect to the consent form/s
are followed.
I have tested everything manually. Including:
- with/without consent configured
- allowing/not allowing the use of email/avatar (via
`sso_auth_account_details.html`)
- with/without automatic account detail population (by un/commenting the
`localpart_template` option in synapse config).
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
... when workers are unreachable, etc.
Fixes https://github.com/element-hq/synapse/issues/17117.
The general principle is just to make sure that we propagate any
exceptions to the JsonResource, so that we return an error code to the
sending server. That means that the sending server no longer considers
the message safely sent, so it will retry later.
In the issue, Erik mentions that an alternative solution would be to
persist the to-device messages into a table so that they can be retried.
This might be an improvement for performance, but even if we did that,
we still need this mechanism, since we might be unable to reach the
database. So, if we want to do that, it can be a later follow-up.
---------
Co-authored-by: Erik Johnston <erik@matrix.org>
|
|
|
|
|
| |
Pretty straightforward. 😄
Fixes #17021
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
|
|
|
| |
`/_synapse/admin/v1/federation/destinations/<destination>/rooms` (#17077)
|
|
|
|
|
|
| |
This makes it easy to store UNIX sockets with correct permissions. Those
would be located in /run/synapse which is the directory used in many
examples in Synapse configuration manual. Additionally, the directory
and sockets are deleted when Synapse is shut down.
|
| |
|
| |
|
|
|
|
|
| |
Keep the existing endpoint for backwards compatibility
Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
|
|
|
|
|
|
| |
(#17000)
Fixes https://github.com/element-hq/synapse/issues/16999 and
https://github.com/element-hq/element-android/pull/8729 by returning the
search terms as search highlights.
|
| |
|
|
|
|
|
| |
(#17076)
Fixes #15355 by redacting all membership events before leaving rooms.
|
|
|
|
|
|
| |
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
Co-authored-by: Hugh Nimmo-Smith <hughns@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
|
| |
This should fix CI on `develop`. Broke in
https://github.com/element-hq/synapse/commit/0fe9e1f7dafa80f3e02762f7ae75cefee5b3316c,
presumably due to a `mypy` dependency upgrade.
|
|
|
|
| |
Co-authored-by: Mathieu Velten <mathieu.velten@beta.gouv.fr>
Co-authored-by: Olivier D <odelcroi@gmail.com>
|
|
|
| |
Documentation fix.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds functions to transform a Twisted request to the
`http::Request`, and then to send back an `http::Response` through it.
It also imports the SynapseError exception so that we can throw that
from Rust code directly
Example usage of this would be:
```rust
use crate::http::{http_request_from_twisted, http_response_to_twisted, HeaderMapPyExt};
fn handler(twisted_request: &PyAny) -> PyResult<()> {
let request = http_request_from_twisted(twisted_request)?;
let ua: headers::UserAgent = request.headers().typed_get_required()?;
if whatever {
return Err((crate::errors::SynapseError::new(
StatusCode::UNAUTHORIZED,
"Whatever".to_owned
"M_UNAUTHORIZED",
None,
None,
)));
}
let response = Response::new("hello".as_bytes());
http_response_to_twisted(twisted_request, response)?;
Ok(())
}
```
|
|
|
| |
Introduced in #17032
|
|
|
|
|
|
|
| |
This adds support for MSC4108 via delegation, similar to what has been done for MSC3886
---------
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
MSC3967 was updated recently to make it more robust to network failures:
> there is an existing cross-signing master key and it exactly matches
the cross-signing master key provided in the request body. If there are
any additional keys provided in the request (self signing key, user
signing key) they MUST also match the existing keys stored on the
server. In other words, the request contains no new keys. If there are
new keys, UIA MUST be performed.
https://github.com/matrix-org/matrix-spec-proposals/blob/hughns/device-signing-upload-uia/proposals/3967-device-signing-upload-uia.md#proposal
This covers the case where the 200 OK is lost in transit so the client
retries the upload, only to then get UIA'd.
Complement tests: https://github.com/matrix-org/complement/pull/713 -
passing example
https://github.com/element-hq/synapse/actions/runs/7976948122/job/21778795094?pr=16943#step:7:8820
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: reivilibre <oliverw@matrix.org>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Resurrecting https://github.com/matrix-org/synapse/pull/13918.
This should reduce IOPs incurred by joining to the events table to
lookup stream ordering, which happens in many receipt handling code
paths. Like the previous PR I believe sufficient time has passed between
the original migration in DB schema 72 and now to merge this as-is. It's
highly unlikely that both the migration is still ongoing AND (active)
users still have any receipts prior to that date.
In the unlikely event there is a receipt without a populated
`event_stream_ordering` synapse will behave just as it does now when
receipts exist for events that don't (yet): for push action calculation
the receipts are just ignored.
I've removed the validation on event IDs as this is already covered
here:
https://github.com/element-hq/synapse/blob/59ceabcb9798793cd4312fdbcced4e612aeda84d/synapse/handlers/receipts.py#L189-L192
|
|
|
|
|
|
| |
`ITransport.abortConnection` isn't a thing, but
`HTTPChannel.forceAbortClient` calls it, so lets just use that
Fixes https://github.com/element-hq/synapse/issues/16728
|
| |
|
|
|
|
|
|
|
| |
See
[MSC4010](https://github.com/matrix-org/matrix-spec-proposals/pull/4010),
but this is pretty much just removing an experimental flag.
Part of #17021
|
|
|
|
|
|
|
| |
See
[MSC3981](https://github.com/matrix-org/matrix-spec-proposals/pull/3981),
this pretty much just removes flags though.
Part of #17021
|
|
|
|
|
| |
Refinement of #17064
cc @richvdh
|
|
|
|
| |
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
| |
This should have been in #17045. Whoops.
|
|
|
|
|
| |
Forget a line, and an empty batch is trivially linear.
c.f. #17064
|
|
|
|
|
|
|
|
|
|
|
| |
PR #16942 removed an invalid optimisation that avoided pulling out state
for non-gappy syncs. This causes a large increase in DB usage. c.f.
#16941 for why that optimisation was wrong.
However, we can still optimise in the simple case where the events in
the timeline are a linear chain without any branching/merging of the
DAG.
cc. @richvdh
|
|
|
|
|
|
|
| |
Before we were pulling out *all* read receipts for a user for every
event we pushed. Instead let's only pull out the relevant receipts.
This also pulled out the event rows for each receipt, causing load on
the events table.
|
|
|
|
|
|
|
|
|
| |
Unfortunately, the optimisation we applied here for non-gappy syncs is
not actually valid.
Fixes https://github.com/element-hq/synapse/issues/16941.
~~Based on https://github.com/element-hq/synapse/pull/16930.~~
Requires https://github.com/matrix-org/sytest/pull/1374.
|
|
|
|
|
|
|
| |
Fix a long-standing issue which could cause state to be omitted from the
sync response if the last event was filtered out.
Fixes: https://github.com/element-hq/synapse/issues/16928
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR fixes a very, very niche edge-case, but I've got some more work
coming which will otherwise make the problem worse.
The bug happens when the syncing user leaves a room, and has a sync
filter which includes "left" rooms, but sets the timeline limit to 0. In
that case, the state returned in the `state` section is calculated
incorrectly.
The fix is to pass a token corresponding to the point that the user
leaves the room through to `compute_state_delta`.
|
|
|
| |
This was causing sequential scans when using refresh tokens.
|
|\ |
|
| | |
|
| |
| |
| | |
Since these queries are duplicated in two places.
|
| |
| |
| | |
Follow on from #17037
|
|/ |
|
|\ |
|
| |
| |
| |
| |
| | |
Requests may require a User-Agent header, and the change in #16972
accidentally removed it, resulting in requests getting rejected causing
login to fail.
|
|/ |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#16925)
Fixes https://github.com/element-hq/synapse/issues/16680, as well as a
related bug, where servers which we had *never* successfully sent an
event to would not be retried.
In order to fix the case of pending to-device messages, we hook into the
existing `wake_destinations_needing_catchup` process, by extending it to
look for destinations that have pending to-device messages. The
federation transmission loop then attempts to send the pending to-device
messages as normal.
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
| |
When running unit tests, we patch the database connection pool so that
it runs queries "synchronously". This is ok, except that if any queries
are launched before we do the patching, those queries get left in limbo
and never complete.
To fix this, let's change the way we do the switcheroo, by patching out
the method which creates the connection pool in the first place.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
when it's already set (#16985)
I have a use case where I'd like the Synapse image to start up a
postgres instance that I can use, but don't want to force Synapse to use
postgres as well.
This commit prevents postgres from being started when it has already
been explicitly enabled elsewhere.
|
| |
|
|
|
| |
As the title states.
|
|
|
|
| |
As done for SAML mapping provider, let's pass the module API to the OIDC
one so the mapper can do more logic in its code.
|
|
|
|
|
| |
This is needed, because the netaddr package removed support for the
implicit prefix form in version 1.0.0:
https://github.com/netaddr/netaddr/pull/360
|
|
|
|
| |
State *before* an event is different to state *after* that event, and
people tend to assume the wrong one.
|
|
|
|
|
| |
Fixes https://github.com/element-hq/synapse/issues/16948. If the `join`
and the `leave` are in the same sync response, we need to count them as
a "left" user.
|
|
|
|
|
|
| |
This is a huge method, which melts my brain.
This is a non-functional change which lays some groundwork for future
work in this area.
|
|
|
|
|
|
| |
Fixes #16964
This adds a proper sorter for versions which takes into account semantic
versions, rather than just relying on localeCompare.
|
|
|
|
| |
comma splice
"rollback" is a noun
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a lot of locks are waiting for a single lock, notifying all locks
independently with `call_later` on each release is really costly and
incurs some kind of async contention, where the CPU is spinning a lot
for not much.
The included test is taking around 30s before the change, and 0.5s
after.
It was found following failing tests with
https://github.com/element-hq/synapse/pull/16827.
|
|
|
|
|
|
|
|
|
|
|
| |
Background: we have a `matrixdotorg/synapse-workers` docker image, which
is intended for running multiple workers within the same container. That
image includes a `prefix-log` script which, for each line printed to
stdout or stderr by one of the processes, prepends the name of the
process.
This commit disables buffering in that script, so that lines are logged
quickly after they are printed. This makes it much easier to understand
the output, since they then come out in a natural order.
|
|
|
|
| |
A new poetry-core version was released. See if CI is happy. Required for
the latest Fedora Synapse package.
|
| |
|
|\ |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This PR aims to fix #16895, caused by a regression in #7 and not fixed
by #16903. The PR #16903 only fixes a starvation issue, where the CPU
isn't released. There is a second issue, where the execution is blocked.
This theory is supported by the flame graphs provided in #16895 and the
fact that I see the CPU usage reducing and far below the limit.
Since the changes in #7, the method `check_state_independent_auth_rules`
is called with the additional parameter `batched_auth_events`:
https://github.com/element-hq/synapse/blob/6fa13b4f927c10b5f4e9495be746ec28849f5cb6/synapse/handlers/federation_event.py#L1741-L1743
It makes the execution enter this if clause, introduced with #15195
https://github.com/element-hq/synapse/blob/6fa13b4f927c10b5f4e9495be746ec28849f5cb6/synapse/event_auth.py#L178-L189
There are two issues in the above code snippet.
First, there is the blocking issue. I'm not entirely sure if this is a
deadlock, starvation, or something different. In the beginning, I
thought the copy operation was responsible. It wasn't. Then I
investigated the nested `store.get_events` inside the function `update`.
This was also not causing the blocking issue. Only when I replaced the
set difference operation (`-` ) with a list comprehension, the blocking
was resolved. Creating and comparing sets with a very large amount of
events seems to be problematic.
This is how the flamegraph looks now while persisting outliers. As you
can see, the execution no longer locks up in the above function.
![output_2024-02-28_13-59-40](https://github.com/element-hq/synapse/assets/13143850/6db9c9ac-484f-47d0-bdde-70abfbd773ec)
Second, the copying here doesn't serve any purpose, because only a
shallow copy is created. This means the same objects from the original
dict are referenced. This fails the intention of protecting these
objects from mutation. The review of the original PR
https://github.com/matrix-org/synapse/pull/15195 had an extensive
discussion about this matter.
Various approaches to copying the auth_events were attempted:
1) Implementing a deepcopy caused issues due to
builtins.EventInternalMetadata not being pickleable.
2) Creating a dict with new objects akin to a deepcopy.
3) Creating a dict with new objects containing only necessary
attributes.
Concluding, there is no easy way to create an actual copy of the
objects. Opting for a deepcopy can significantly strain memory and CPU
resources, making it an inefficient choice. I don't see why the copy is
necessary in the first place. Therefore I'm proposing to remove it
altogether.
After these changes, I was able to successfully join these rooms,
without the main worker locking up:
- #synapse:matrix.org
- #element-android:matrix.org
- #element-web:matrix.org
- #ecips:matrix.org
- #ipfs-chatter:ipfs.io
- #python:matrix.org
- #matrix:matrix.org
|
|/ |
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since Synapse 1.76.0, any module which registers a `on_new_event`
callback would brick the ability to join remote rooms.
This is because this callback tried to get the full state of the room,
which would end up in a deadlock.
Related:
https://github.com/matrix-org/synapse-auto-accept-invite/issues/18
The following module would brick the ability to join remote rooms:
```python
from typing import Any, Dict, Literal, Union
import logging
from synapse.module_api import ModuleApi, EventBase
logger = logging.getLogger(__name__)
class MyModule:
def __init__(self, config: None, api: ModuleApi):
self._api = api
self._config = config
self._api.register_third_party_rules_callbacks(
on_new_event=self.on_new_event,
)
async def on_new_event(self, event: EventBase, _state_map: Any) -> None:
logger.info(f"Received new event: {event}")
@staticmethod
def parse_config(_config: Dict[str, Any]) -> None:
return None
```
This is technically a breaking change, as we are now passing partial
state on the `on_new_event` callback.
However, this callback was broken for federated rooms since 1.76.0, and
local rooms have full state anyway, so it's unlikely that it would
change anything.
|
| |
|
|
|
|
| |
Co-authored-by: reivilibre <oliverw@matrix.org>
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|\ |
|
| | |
|
|/ |
|
|
|
|
| |
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
|
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
| |
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
|
|
|
|
| |
This adds a counter `synapse_emails_sent_total` for emails sent. They
are broken down by `type`, which are `password_reset`, `registration`,
`add_threepid`, `notification` (matching the methods of `Mailer`).
|
|\ |
|
| | |
|
| |
| |
| |
| |
| | |
We do this by adding support to the LRU cache for "extra indices" based
on the cached value. This allows us to efficiently map from room ID to
the cached events and only invalidate those.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
List of users not to send out device list updates for when they register
new devices. This is useful to handle bot accounts.
This is undocumented as its mostly a hack to test on matrix.org.
Note: This will still send out device list updates if the device is
later updated, e.g. end to end keys are added.
|
|\| |
|
| |
| |
| |
| |
| |
| |
| |
| | |
This basically reverts a change that was in
https://github.com/element-hq/synapse/pull/16833, where we reduced the
batching.
The smaller batching can cause performance issues on busy servers and
databases.
|
|/
|
|
|
| |
Otherwise if we've stacked a bunch of requests for the keys of a server,
we'll end up sending lots of concurrent requests for its keys,
needlessly.
|
| |
|
|
|
| |
Now that the MSC3981 has passed FCP
|
|
|
|
| |
ICU is an optional dependency and also a pain to install. Mention that
you can just not install it and still get a working installation.
|
|
|
|
| |
Some links seemed to be incorrect (vector-im/sygnal and vector-im/sytest
have never been A Thing iirc) so pointed them back to matrix-org/*).
|
| |
|
| |
|
|
|
|
|
| |
Introduced in #16833
Fixes #16844
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
breaking changes. (#16847)
Partially reverts #16796
This is causing errors of the form:
```
Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
```
for the debs and wheels stages.
There were breaking changes that weren't included in the dependabot
changelog (:/):
https://github.com/actions/upload-artifact#breaking-changes
<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `release-v1.100` <!-- git-stack-base-branch:release-v1.100 -->
<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->
Original commit schedule, with full messages:
<ol>
<li>
Downgrade the `upload-artifact` and `download-artifact` actions to v3
</li>
</ol>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
| |
This helps with bot accounts with lots of non-e2e devices.
The change is basically to change the order of the join for the case of
using `INNER JOIN`
|
|
|
|
|
| |
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pulled out of #16803 since the drive-by cleanup was maybe not as
drive-by as I had hoped.
<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->
<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->
Original commit schedule, with full messages:
<ol>
<li>
Add a --generate-only option
</li>
</ol>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|