| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
| |
This version change requires a migration to a new API. See
https://pyo3.rs/v0.21.2/migration#from-020-to-021
This will fix the annoying warnings added when using the recent rust
nightly:
> warning: non-local `impl` definition, they should be avoided as they
go against expectation
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| |/ |
|
| |
|
|
|
|
|
| |
When there have been lots of changes compared with the number of
entities, we can do a fast(er) path.
Locally I ran some benchmarking, and the comparison seems to give the
best determination of which method we use.
|
| |
|
| |
Annoyingly mypy didn't pick up this typo.
|
| |
|
| |
Broke in #17073
|
| | |
|
| |
|
| |
upgrade.md: Bump minimum Rust version to 1.66.0 (element-hq#17079)
|
| |
|
|
|
|
| |
This is an alternative to
https://github.com/matrix-org/matrix-rust-sdk/issues/3340 where we don't
need to change our CI setup too much in the Rust SDK repository, and
still can test MSC3266.
|
| | |
|
| |
|
|
|
|
| |
It is broken, and we only seemed to have been building it for the
federation sender.
Closes https://github.com/element-hq/synapse/issues/16804
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
|
|
| |
Follow-up to #17137 and
https://github.com/matrix-org/complement/pull/722
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
selected (#17120)
This change will apply the `email` & `picture` provided by OIDC to the
new user account when registering a new user via OIDC. If the user is
directed to the account details form, this change makes sure they have
been selected before applying them, otherwise they are omitted. In
particular, this change ensures the values are carried through when
Synapse has consent configured, and the redirect to the consent form/s
are followed.
I have tested everything manually. Including:
- with/without consent configured
- allowing/not allowing the use of email/avatar (via
`sso_auth_account_details.html`)
- with/without automatic account detail population (by un/commenting the
`localpart_template` option in synapse config).
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
... when workers are unreachable, etc.
Fixes https://github.com/element-hq/synapse/issues/17117.
The general principle is just to make sure that we propagate any
exceptions to the JsonResource, so that we return an error code to the
sending server. That means that the sending server no longer considers
the message safely sent, so it will retry later.
In the issue, Erik mentions that an alternative solution would be to
persist the to-device messages into a table so that they can be retried.
This might be an improvement for performance, but even if we did that,
we still need this mechanism, since we might be unable to reach the
database. So, if we want to do that, it can be a later follow-up.
---------
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
|
|
| |
Pretty straightforward. 😄
Fixes #17021
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
|
|
| |
`/_synapse/admin/v1/federation/destinations/<destination>/rooms` (#17077)
|
| |
|
|
|
|
| |
This makes it easy to store UNIX sockets with correct permissions. Those
would be located in /run/synapse which is the directory used in many
examples in Synapse configuration manual. Additionally, the directory
and sockets are deleted when Synapse is shut down.
|
| | |
|
| | |
|
| |
|
|
|
| |
Keep the existing endpoint for backwards compatibility
Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
|
|
|
|
|
| |
(#17000)
Fixes https://github.com/element-hq/synapse/issues/16999 and
https://github.com/element-hq/element-android/pull/8729 by returning the
search terms as search highlights.
|
| | |
|
| |
|
|
|
| |
(#17076)
Fixes #15355 by redacting all membership events before leaving rooms.
|
| |
|
|
|
|
| |
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
Co-authored-by: Hugh Nimmo-Smith <hughns@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
| |
This should fix CI on `develop`. Broke in
https://github.com/element-hq/synapse/commit/0fe9e1f7dafa80f3e02762f7ae75cefee5b3316c,
presumably due to a `mypy` dependency upgrade.
|
| |
|
|
| |
Co-authored-by: Mathieu Velten <mathieu.velten@beta.gouv.fr>
Co-authored-by: Olivier D <odelcroi@gmail.com>
|
| |
|
| |
Documentation fix.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [X] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds functions to transform a Twisted request to the
`http::Request`, and then to send back an `http::Response` through it.
It also imports the SynapseError exception so that we can throw that
from Rust code directly
Example usage of this would be:
```rust
use crate::http::{http_request_from_twisted, http_response_to_twisted, HeaderMapPyExt};
fn handler(twisted_request: &PyAny) -> PyResult<()> {
let request = http_request_from_twisted(twisted_request)?;
let ua: headers::UserAgent = request.headers().typed_get_required()?;
if whatever {
return Err((crate::errors::SynapseError::new(
StatusCode::UNAUTHORIZED,
"Whatever".to_owned
"M_UNAUTHORIZED",
None,
None,
)));
}
let response = Response::new("hello".as_bytes());
http_response_to_twisted(twisted_request, response)?;
Ok(())
}
```
|
| |
|
| |
Introduced in #17032
|
| |
|
|
|
|
|
| |
This adds support for MSC4108 via delegation, similar to what has been done for MSC3886
---------
Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
MSC3967 was updated recently to make it more robust to network failures:
> there is an existing cross-signing master key and it exactly matches
the cross-signing master key provided in the request body. If there are
any additional keys provided in the request (self signing key, user
signing key) they MUST also match the existing keys stored on the
server. In other words, the request contains no new keys. If there are
new keys, UIA MUST be performed.
https://github.com/matrix-org/matrix-spec-proposals/blob/hughns/device-signing-upload-uia/proposals/3967-device-signing-upload-uia.md#proposal
This covers the case where the 200 OK is lost in transit so the client
retries the upload, only to then get UIA'd.
Complement tests: https://github.com/matrix-org/complement/pull/713 -
passing example
https://github.com/element-hq/synapse/actions/runs/7976948122/job/21778795094?pr=16943#step:7:8820
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Resurrecting https://github.com/matrix-org/synapse/pull/13918.
This should reduce IOPs incurred by joining to the events table to
lookup stream ordering, which happens in many receipt handling code
paths. Like the previous PR I believe sufficient time has passed between
the original migration in DB schema 72 and now to merge this as-is. It's
highly unlikely that both the migration is still ongoing AND (active)
users still have any receipts prior to that date.
In the unlikely event there is a receipt without a populated
`event_stream_ordering` synapse will behave just as it does now when
receipts exist for events that don't (yet): for push action calculation
the receipts are just ignored.
I've removed the validation on event IDs as this is already covered
here:
https://github.com/element-hq/synapse/blob/59ceabcb9798793cd4312fdbcced4e612aeda84d/synapse/handlers/receipts.py#L189-L192
|
| |
|
|
|
|
| |
`ITransport.abortConnection` isn't a thing, but
`HTTPChannel.forceAbortClient` calls it, so lets just use that
Fixes https://github.com/element-hq/synapse/issues/16728
|
| | |
|
| |
|
|
|
|
|
| |
See
[MSC4010](https://github.com/matrix-org/matrix-spec-proposals/pull/4010),
but this is pretty much just removing an experimental flag.
Part of #17021
|
| |
|
|
|
|
|
| |
See
[MSC3981](https://github.com/matrix-org/matrix-spec-proposals/pull/3981),
this pretty much just removes flags though.
Part of #17021
|
| |
|
|
|
| |
Refinement of #17064
cc @richvdh
|
| |
|
|
| |
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
| |
This should have been in #17045. Whoops.
|
| |
|
|
|
| |
Forget a line, and an empty batch is trivially linear.
c.f. #17064
|
| |
|
|
|
|
|
|
|
|
|
| |
PR #16942 removed an invalid optimisation that avoided pulling out state
for non-gappy syncs. This causes a large increase in DB usage. c.f.
#16941 for why that optimisation was wrong.
However, we can still optimise in the simple case where the events in
the timeline are a linear chain without any branching/merging of the
DAG.
cc. @richvdh
|
| |
|
|
|
|
|
| |
Before we were pulling out *all* read receipts for a user for every
event we pushed. Instead let's only pull out the relevant receipts.
This also pulled out the event rows for each receipt, causing load on
the events table.
|
| |
|
|
|
|
|
|
|
| |
Unfortunately, the optimisation we applied here for non-gappy syncs is
not actually valid.
Fixes https://github.com/element-hq/synapse/issues/16941.
~~Based on https://github.com/element-hq/synapse/pull/16930.~~
Requires https://github.com/matrix-org/sytest/pull/1374.
|
| |
|
|
|
|
|
| |
Fix a long-standing issue which could cause state to be omitted from the
sync response if the last event was filtered out.
Fixes: https://github.com/element-hq/synapse/issues/16928
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This PR fixes a very, very niche edge-case, but I've got some more work
coming which will otherwise make the problem worse.
The bug happens when the syncing user leaves a room, and has a sync
filter which includes "left" rooms, but sets the timeline limit to 0. In
that case, the state returned in the `state` section is calculated
incorrectly.
The fix is to pass a token corresponding to the point that the user
leaves the room through to `compute_state_delta`.
|
| |
|
| |
This was causing sequential scans when using refresh tokens.
|
| |\ |
|
| | | |
|
| | |
| |
| | |
Since these queries are duplicated in two places.
|
| | |
| |
| | |
Follow on from #17037
|
| |/ |
|
| |\ |
|
| | |
| |
| |
| |
| | |
Requests may require a User-Agent header, and the change in #16972
accidentally removed it, resulting in requests getting rejected causing
login to fail.
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#16925)
Fixes https://github.com/element-hq/synapse/issues/16680, as well as a
related bug, where servers which we had *never* successfully sent an
event to would not be retried.
In order to fix the case of pending to-device messages, we hook into the
existing `wake_destinations_needing_catchup` process, by extending it to
look for destinations that have pending to-device messages. The
federation transmission loop then attempts to send the pending to-device
messages as normal.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
When running unit tests, we patch the database connection pool so that
it runs queries "synchronously". This is ok, except that if any queries
are launched before we do the patching, those queries get left in limbo
and never complete.
To fix this, let's change the way we do the switcheroo, by patching out
the method which creates the connection pool in the first place.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
when it's already set (#16985)
I have a use case where I'd like the Synapse image to start up a
postgres instance that I can use, but don't want to force Synapse to use
postgres as well.
This commit prevents postgres from being started when it has already
been explicitly enabled elsewhere.
|
| | |
|
| |
|
| |
As the title states.
|
| |
|
|
| |
As done for SAML mapping provider, let's pass the module API to the OIDC
one so the mapper can do more logic in its code.
|
| |
|
|
|
| |
This is needed, because the netaddr package removed support for the
implicit prefix form in version 1.0.0:
https://github.com/netaddr/netaddr/pull/360
|
| |
|
|
| |
State *before* an event is different to state *after* that event, and
people tend to assume the wrong one.
|
| |
|
|
|
| |
Fixes https://github.com/element-hq/synapse/issues/16948. If the `join`
and the `leave` are in the same sync response, we need to count them as
a "left" user.
|
| |
|
|
|
|
| |
This is a huge method, which melts my brain.
This is a non-functional change which lays some groundwork for future
work in this area.
|
| |
|
|
|
|
| |
Fixes #16964
This adds a proper sorter for versions which takes into account semantic
versions, rather than just relying on localeCompare.
|
| |
|
|
| |
comma splice
"rollback" is a noun
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
When a lot of locks are waiting for a single lock, notifying all locks
independently with `call_later` on each release is really costly and
incurs some kind of async contention, where the CPU is spinning a lot
for not much.
The included test is taking around 30s before the change, and 0.5s
after.
It was found following failing tests with
https://github.com/element-hq/synapse/pull/16827.
|
| |
|
|
|
|
|
|
|
|
|
| |
Background: we have a `matrixdotorg/synapse-workers` docker image, which
is intended for running multiple workers within the same container. That
image includes a `prefix-log` script which, for each line printed to
stdout or stderr by one of the processes, prepends the name of the
process.
This commit disables buffering in that script, so that lines are logged
quickly after they are printed. This makes it much easier to understand
the output, since they then come out in a natural order.
|
| |
|
|
| |
A new poetry-core version was released. See if CI is happy. Required for
the latest Fedora Synapse package.
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This PR aims to fix #16895, caused by a regression in #7 and not fixed
by #16903. The PR #16903 only fixes a starvation issue, where the CPU
isn't released. There is a second issue, where the execution is blocked.
This theory is supported by the flame graphs provided in #16895 and the
fact that I see the CPU usage reducing and far below the limit.
Since the changes in #7, the method `check_state_independent_auth_rules`
is called with the additional parameter `batched_auth_events`:
https://github.com/element-hq/synapse/blob/6fa13b4f927c10b5f4e9495be746ec28849f5cb6/synapse/handlers/federation_event.py#L1741-L1743
It makes the execution enter this if clause, introduced with #15195
https://github.com/element-hq/synapse/blob/6fa13b4f927c10b5f4e9495be746ec28849f5cb6/synapse/event_auth.py#L178-L189
There are two issues in the above code snippet.
First, there is the blocking issue. I'm not entirely sure if this is a
deadlock, starvation, or something different. In the beginning, I
thought the copy operation was responsible. It wasn't. Then I
investigated the nested `store.get_events` inside the function `update`.
This was also not causing the blocking issue. Only when I replaced the
set difference operation (`-` ) with a list comprehension, the blocking
was resolved. Creating and comparing sets with a very large amount of
events seems to be problematic.
This is how the flamegraph looks now while persisting outliers. As you
can see, the execution no longer locks up in the above function.
Second, the copying here doesn't serve any purpose, because only a
shallow copy is created. This means the same objects from the original
dict are referenced. This fails the intention of protecting these
objects from mutation. The review of the original PR
https://github.com/matrix-org/synapse/pull/15195 had an extensive
discussion about this matter.
Various approaches to copying the auth_events were attempted:
1) Implementing a deepcopy caused issues due to
builtins.EventInternalMetadata not being pickleable.
2) Creating a dict with new objects akin to a deepcopy.
3) Creating a dict with new objects containing only necessary
attributes.
Concluding, there is no easy way to create an actual copy of the
objects. Opting for a deepcopy can significantly strain memory and CPU
resources, making it an inefficient choice. I don't see why the copy is
necessary in the first place. Therefore I'm proposing to remove it
altogether.
After these changes, I was able to successfully join these rooms,
without the main worker locking up:
- #synapse:matrix.org
- #element-android:matrix.org
- #element-web:matrix.org
- #ecips:matrix.org
- #ipfs-chatter:ipfs.io
- #python:matrix.org
- #matrix:matrix.org
|
| |/ |
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since Synapse 1.76.0, any module which registers a `on_new_event`
callback would brick the ability to join remote rooms.
This is because this callback tried to get the full state of the room,
which would end up in a deadlock.
Related:
https://github.com/matrix-org/synapse-auto-accept-invite/issues/18
The following module would brick the ability to join remote rooms:
```python
from typing import Any, Dict, Literal, Union
import logging
from synapse.module_api import ModuleApi, EventBase
logger = logging.getLogger(__name__)
class MyModule:
def __init__(self, config: None, api: ModuleApi):
self._api = api
self._config = config
self._api.register_third_party_rules_callbacks(
on_new_event=self.on_new_event,
)
async def on_new_event(self, event: EventBase, _state_map: Any) -> None:
logger.info(f"Received new event: {event}")
@staticmethod
def parse_config(_config: Dict[str, Any]) -> None:
return None
```
This is technically a breaking change, as we are now passing partial
state on the `on_new_event` callback.
However, this callback was broken for federated rooms since 1.76.0, and
local rooms have full state anyway, so it's unlikely that it would
change anything.
|
| | |
|
| |
|
|
| |
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |\ |
|
| | | |
|
| |/ |
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
|
| |
|
|
|
| |
This adds a counter `synapse_emails_sent_total` for emails sent. They
are broken down by `type`, which are `password_reset`, `registration`,
`add_threepid`, `notification` (matching the methods of `Mailer`).
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| | |
We do this by adding support to the LRU cache for "extra indices" based
on the cached value. This allows us to efficiently map from room ID to
the cached events and only invalidate those.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
List of users not to send out device list updates for when they register
new devices. This is useful to handle bot accounts.
This is undocumented as its mostly a hack to test on matrix.org.
Note: This will still send out device list updates if the device is
later updated, e.g. end to end keys are added.
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This basically reverts a change that was in
https://github.com/element-hq/synapse/pull/16833, where we reduced the
batching.
The smaller batching can cause performance issues on busy servers and
databases.
|
| |/
|
|
|
| |
Otherwise if we've stacked a bunch of requests for the keys of a server,
we'll end up sending lots of concurrent requests for its keys,
needlessly.
|
| | |
|
| |
|
| |
Now that the MSC3981 has passed FCP
|
| |
|
|
| |
ICU is an optional dependency and also a pain to install. Mention that
you can just not install it and still get a working installation.
|
| |
|
|
| |
Some links seemed to be incorrect (vector-im/sygnal and vector-im/sytest
have never been A Thing iirc) so pointed them back to matrix-org/*).
|
| | |
|
| | |
|
| |
|
|
|
| |
Introduced in #16833
Fixes #16844
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
breaking changes. (#16847)
Partially reverts #16796
This is causing errors of the form:
```
Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run
```
for the debs and wheels stages.
There were breaking changes that weren't included in the dependabot
changelog (:/):
https://github.com/actions/upload-artifact#breaking-changes
<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `release-v1.100` <!-- git-stack-base-branch:release-v1.100 -->
<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->
Original commit schedule, with full messages:
<ol>
<li>
Downgrade the `upload-artifact` and `download-artifact` actions to v3
</li>
</ol>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| | |
|
| |
|
|
|
|
| |
This helps with bot accounts with lots of non-e2e devices.
The change is basically to change the order of the join for the case of
using `INNER JOIN`
|
| |
|
|
|
| |
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pulled out of #16803 since the drive-by cleanup was maybe not as
drive-by as I had hoped.
<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->
<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->
Original commit schedule, with full messages:
<ol>
<li>
Add a --generate-only option
</li>
</ol>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
|
| |
More info [here](https://www.nginx.com/blog/nginx-plus-r30-released/).
Nginx threw error's at me when I used all the options of the doc
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
are violated when creating a new room. (#16811)
Prior to this PR, if a request to create a public (public as in
published to the rooms directory) room violated the room list
publication rules set in the
[config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules),
the request to create the room was denied and the room was not created.
This PR changes the behavior such that when a request to create a room
published to the directory violates room list publication rules, the
room is still created but the room is not published to the directory.
|
| |
|
|
|
| |
The ecosystem e.g. regex crate, have bumped up their MSRV to 1.65.0,
which was released Nov 2022. In line with our policy, let's bump to
match.
|
| | |
|
| |
|
| |
This happened during the migration
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
| |
The current query supports passing in a list of users, which generates a
query using `user_id = ANY(..)`. This is generates a less efficient
query plan that is notably slower than a simple `user_id = ?` condition.
Note: The new function is mostly a copy and paste and then a
simplification of the existing function.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The crux of the change is to try and make the queries simpler and pull
out fewer rows. Before, there were quite a few joins against subqueries,
which caused postgres to pull out more rows than necessary.
Instead, let's simplify the query and do some of the filtering out in
Python instead, letting Postgres do better optimizations now that it
doesn't have to deal with joins against subqueries.
Review note: this is a complete rewrite of the function, so not sure how
useful the diff is.
---------
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
In previous versions of authlib using `client_secret_basic` without a
`client_secret` would result in an invalid auth header. Since authlib
1.3 it throws an exception.
The configuration may be accepted in by very lax servers, so we don't
want to deny it outright. Instead, let's default the
`client_auth_method` to `none`, which does the right thing. If the
config specifies `client_auth_method` and no `client_secret` then that
is going to be bogus and we should reject it
|
| |
|
| |
This hopefully reduces the amount of state we need to keep in memory
|
| |
|
|
|
|
|
|
|
|
|
| |
Sometimes we fail to fetch events during backfill due to missing state,
and we often end up querying the same bad events periodically (as people
backpaginate). In such cases its likely we will continue to fail to get
the state, and therefore we should try *before* loading the state that
we have from the DB (as otherwise it's wasted DB and memory).
---------
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
| |
There are two changes here:
1. Only pull out the required state when handling the request.
2. Change the get filtered state return type to check that we're only
querying state that was requested
---------
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
| |
Otherwise for users with large numbers of devices this can cause a lot
of woe.
|
| |
|
|
| |
Otherwise if you hit the `Enable auto-merge` button and the linting
fails the PR is still aut-merged.
|
| |
|
| |
Since we don't require one anymore.
|
| | |
|
| |
|
|
|
|
|
|
| |
Instead of persisting outliers in a bunch of batches, let's just do them
all at once.
This is fine because all `_auth_and_persist_outliers_inner` is doing is
checking the auth rules for each event, which requires the events to be
topologically sorted by the auth graph.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
when those rooms block that homeserver by their Access Control Lists. (#16759)
The idea here being that the directory server shouldn't advertise rooms
to a requesting server is the requesting server would not be allowed to
join or participate in the room.
<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->
<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->
Original commit schedule, with full messages:
<ol>
<li>
Pass `from_federation_origin` down into room list retrieval code
</li>
<li>
Don't cache /publicRooms response for inbound federated requests
</li>
<li>
fixup! Don't cache /publicRooms response for inbound federated requests
</li>
<li>
Cap the number of /publicRooms entries to 100
</li>
<li>
Simplify code now that you can't request unlimited rooms
</li>
<li>
Filter out rooms from federated requests that don't have the correct ACL
</li>
<li>
Request a handful more when filtering ACLs so that we can try to avoid
shortchanging the requester
</li>
</ol>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
There are a couple of things we need to be careful of here:
1. The current python code does no validation when loading from the DB,
so we need to be careful to ignore such errors (at least on jki.re there
are some old events with internal metadata fields of the wrong type).
2. We want to be memory efficient, as we often have many hundreds of
thousands of events in the cache at a time.
---------
Co-authored-by: Quentin Gliech <quenting@element.io>
|
| |
|
| |
Introduced in #16762
|
| |
|
|
|
|
|
|
|
| |
We remove these fields as they're just duplicating data the event
already stores, and (for reasons :shushing_face:) I'd like to simplify
the class to only store simple types.
I'm not entirely convinced that we shouldn't instead add helper methods
to the event class to generate stream tokens, but I don't really think
that's where they belong either
|
| |
|
| |
Signed-off-by: Gaël Goinvic <gaelg@element.io>
|
| |
|
|
|
| |
This is an extra response parameter just added to MSC3981. In the
current impl, the recursion depth is always 3, so this just returns a
static 3 if the recurse parameter is supplied.
|
| |
|
|
|
| |
Closes #16370
Signed-off-by: Adam Jedrzejewski <adamjedrzejewski@icloud.com>
|
| |
|
|
|
|
| |
Previously, the response status of `HTMLResource` was hardcoded as
`200`. However, for proper redirection after the user verifies their
email, we require the status to be `302`. This PR addresses that issue
by using `code` as response status.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Added in https://github.com/matrix-org/synapse/pull/16533, this workflow
was intended to be run once to add the version picker to all historical
versions of the https://matrix-org.github.io/synapse documentation
website.
Note that the latest version of the docs built from this repo now exist
at https://element-hq.github.io/synapse/.
The workflow has been run successfully and the version picker was added
to the documentation. Thus we can now delete this workflow.
---
Note: Do not confuse this PR with
https://github.com/matrix-org/synapse/issues/9453. This PR was made
while we were populating this repo with "Dummy issues" after the
changeover from matrix-org/synapse to element-hq/synapse - therefore
referencing this PR may cause some confusion.
|
| |
|
| |
Signed-off-by: Christian Lölkes <christian.loelkes@gmail.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Closes:
- https://github.com/matrix-org/synapse/issues/10397
- #10397
An administrator should know whether he wants to set a password or not.
There are many uses cases where a blank password is required.
- Use of only some users with SSO.
- Use of bots with password, users with SSO
|
| |
|
| |
Signed-off-by: Fredrik Lanker <fredrik@lanker.se>
|
| |
|
|
| |
At least for vscode this allows click through / type checking / syntax
highlighting.
|
| |
|
| |
I accidentally broke it during the move by removing a trailing new line.
|
| |
|
|
| |
(#16738)
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Co-authored-by: Fabian Klemp <fabian.klemp@frequentis.com>
|
| | |
|
| |
|
|
| |
Co-authored-by: Quentin Gliech <quenting@element.io>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
/_synapse/admin/v1/federation/destinations (#16729)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
one. This should allow use of Postgres logical replication. (take 2, now with no added deadlocks!) (#16658)
* Add `ALTER TABLE ... REPLICA IDENTITY ...` for individual tables
We can't combine them into one file as it makes it likely to hit a deadlock
if Synapse is running, as it only takes one other transaction to access two
tables in a different order to the schema delta.
* Add notes
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Re-introduce REPLICA IDENTITY test
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
| |
(#16700)
|
| |
|
|
| |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
| |
MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/4069
|
| |
|
| |
Signed-off-by: Andrii Yasynyshyn yasinishyn.a.n@gmail.com
|
| | |
|
| |
|
|
|
|
|
|
| |
poetry-core 1.8.x includes a fix which properly moves the generate
synapse_rust.abi3.so file to the synapse directory when using an
editable install.
Without this change developers are left with a confusing experience
of the synapse.synapse_rust module not being found after installation.
|
| | |
|
| |
|
|
|
|
| |
Implement MSC3860 to follow redirects for federated media downloads.
Note that the Client-Server API doesn't support this (yet) since the media
repository in Synapse doesn't have a way of supporting redirects.
|
| |
|
|
|
| |
* Reduce DB load when forget on leave setting is disabled
* Newsfile
|
| |
|
| |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
| |
Silly query planner
|
| | |
|
| | |
|
| |
|
|
| |
These are not useful and make it difficult to search for
table definitions, etc.
|
| | |
|
| |
|
|
|
|
|
|
| |
* Move media retention tests out of rest tests
AFAICS this doesn't make any HTTP requests and so it ought not to belong
in `tests.rest`.
* Changelog
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
(#16672)
* Describe `insert_client_ip`
* Pull out client_ips and MAU tracking to BaseAuth
* Define HAS_AUTHLIB once in tests
sick of copypasting
* Track ips and token usage when delegating auth
* Test that we track MAU and user_ips
* Don't track `__oidc_admin`
|
| |
|
| |
Signed-off-by: Charles Wright <cvwright@futo.org>
|
| |
|
| |
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
|
| |
|
|
|
|
|
| |
* Admin API for server notice: disable rate limit for all calls
* Add changelog
* Update changelog.d/16670.bugfix
|
| | |
|
| | |
|
| |
|
| |
Now that we're reducing concurrency (#16656), this is more important.
|
| | |
|
| |
|
| |
What it says on the tin
|
| |
|
| |
Follow on from #16640
|
| | |
|
| | |
|
| |
|
|
|
| |
replica identities, then add more replica identities. (#16647)" (#16652)
This reverts commit 830988ae72d63bbb67d2020a3f221664f3f456ee.
|
| |
|
|
|
| |
implicit one. This should allow use of Postgres logical replication. (#16456)" (#16651)
This reverts commit 69afe3f7a0d89f3422ddbd3aa16bc9bbc01056eb.
|
| |
|
| |
Keeping track of a lower bound of stream ID where we've deleted everything below makes the queries much faster. Otherwise, every time we scan for rows to delete we'd re-scan across all the rows that have previously deleted (until the next table VACUUM).
|
| |
|
| |
Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
|
| |
|
|
|
| |
If a worker reconnects to Redis we send out the current positions of all our streams. However, if we're also trying to send out a backlog of RDATA at the same time then we can end up sending a `POSITION` with the current token *before* we've sent all the RDATA before the current token.
This doesn't cause actual bugs as the receiving servers see the POSITION, fetch the relevant rows from the DB, and then ignore the old RDATA as they come in. However, this is inefficient so it'd be better if we didn't send out-of-order positions
|
| |
|
|
| |
We may receive `POSITION` commands where we already know that worker has
advanced past that position, so there is no point in handling it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
identities, then add more replica identities. (#16647)
* Fix the CI query that did not detect all cases of missing primary keys
* Add more missing REPLICA IDENTITY entries
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
| |
cross-signing key without UIA (#16634)
|
| |
|
| |
Support asynchronous uploads as defined in MSC2246.
|
| | |
|
| | |
|
| |
|
|
| |
`/_synapse/admin/v1/rooms/<room_id>/timestamp_to_event` (#16631)
|
| |
|
|
| |
The statements are already executed within a transaction thus a table
level lock is unnecessary.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
one. This should allow use of Postgres logical replication. (#16456)
* Add Postgres replica identities to tables that don't have an implicit one
Fixes #16224
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Move the delta to version 83 as we missed the boat for 82
* Add a test that all tables have a REPLICA IDENTITY
* Extend the test to include when indices are deleted
* isort
* black
* Fully qualify `oid` as it is a 'hidden attribute' in Postgres 11
* Update tests/storage/test_database.py
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add missed tables
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| |
|
| |
To avoid asserting the type of the database connection.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
identifiers for Application Service users when deactivated and would not emit a compliant response. (#16617)
* Don't skip unbinding 3PIDs and returning success status when deactivating AS user
Fixes #16608
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| | |
|
| |
|
|
|
| |
This takes advantage of the new bulk method in more places to
invalidate caches for many keys at once (and then to stream that
over replication).
|
| | |
|
| | |
|
| |
|
|
| |
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
|
| |
|
|
|
|
| |
(#16609)
simple_update_many_txn had a bug in it which would cause each
update to be applied twice.
|
| |
|
|
|
|
| |
If simple_{insert,upsert,update}_many_txn is called without any data
to modify then return instead of executing the query.
This matches the behavior of simple_{select,delete}_many_txn.
|
| |
|
|
| |
Expand tests for the simple_* database methods, additionally
test against both PostgreSQL and SQLite variants.
|
| |
|
|
|
|
|
|
| |
Fetch information needed for push rule evaluation in parallel.
Ideally this would use query pipelining, but this is not
available in psycopg2.
Due to the database thread pool this may result in little
to no parallelization.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Previously only Twisted's EPollReactor was compatible with the
reactor timing metric, notably not working when asyncio was used.
After this change, the following configurations support the reactor
timing metric:
* poll, epoll, or select reactors
* asyncio reactor with a poll, epoll, select, /dev/poll, or kqueue event loop.
|
| |
|
|
|
|
|
|
|
|
|
| |
The event persistence code used to handle multiple rooms
at a time, but was simplified to only ever be called with a
single room at a time (different rooms are now handled in
parallel). The code is still generic to multiple rooms causing
a lot of work that is unnecessary (e.g. unnecessary loops, and
partitioning data by room).
This strips out the ability to handle multiple rooms at once, greatly
simplifying the code.
|
| |
|
|
|
| |
Just to standardize on the normal helpers, it might also have
a slight perf improvement on PostgreSQL which will now use
`ANY (?)` instead of `IN (?, ?, ...)`.
|
| | |
|
| |
|
|
|
| |
getfullargspec is relatively expensive and the results will
not change between calls, so precalculate it outside the
wrapper.
|
| | |
|
| | |
|
| |
|
|
| |
This reverts commit 4724a6ded136c727bcff5082b94c2b3d6355e908.
|
| | |
|
| |
|
|
| |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| |
|
|
| |
We can bail early if the from token is greater than or equal to the
current token.
|
| | |
|
| |
|
|
|
| |
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Robertson <davidr@element.io>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* complement: enable dirty runs
* Add changelog
* Set a low connpool limit when running in Complement
Dirty runs can cause many containers to be running concurrently,
which seems to easily exhaust resources on the host. The increased
speedup from dirty runs also seems to use more db connections on
workers, which are misconfigured currently to have
`SUM(workers * cp_max) > max_connections`, causing
```
FATAL: sorry, too many clients already
```
which results in tests failing.
* Try p=2 concurrency to restrict slowness of servers which causes partial state join tests to flake
* Debug logging
* Only run flakey tests
* Only adjust connection pool limits in worker mode
* Move cp vars to somewhere where they get executed in CI
* Move cp values back to where they actually work
* Debug logging
* Try p=1 to see if this makes worker mode happier
* Remove debug logging
|
