| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| |
|
|
| |
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
... when workers are unreachable, etc.
Fixes https://github.com/element-hq/synapse/issues/17117.
The general principle is just to make sure that we propagate any
exceptions to the JsonResource, so that we return an error code to the
sending server. That means that the sending server no longer considers
the message safely sent, so it will retry later.
In the issue, Erik mentions that an alternative solution would be to
persist the to-device messages into a table so that they can be retried.
This might be an improvement for performance, but even if we did that,
we still need this mechanism, since we might be unable to reach the
database. So, if we want to do that, it can be a later follow-up.
---------
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
|
|
| |
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
|
| | |
|
| | |
|
| |
|
| |
Mostly to improve type safety.
|
| |
|
|
|
|
|
|
|
| |
Synapse was incorrectly implemented with a knock_state_events
property on some APIs (instead of knock_room_state). This was
correct in Synapse 1.70.0, but *both* fields were sent to also be
compatible with Synapse versions expecting the wrong field.
Enough time has passed that only the correct field needs to be
included/handled.
|
| |
|
|
|
| |
* Pre-compiles the server ACLs onto an object per room and
invalidates them when new events come in.
* Converts the server ACL checking into Rust.
|
| |
|
|
| |
(#16223)
|
| | |
|
| |
|
|
| |
Python 3.8 provides a native AsyncMock, we can replace the
homegrown version we have.
|
| |
|
|
|
|
|
| |
Synapse routing of outbound federation traffic (#15806)
`matrix://` is a registered specced scheme nowadays and doesn't make sense for
our internal to Synapse use case anymore. ([discussion]
(https://github.com/matrix-org/synapse/pull/15773#discussion_r1227598679))
|
| | |
|
| |
|
|
| |
Avoid renaming configuration settings for now and rename internal code
to use blocklist and allowlist instead.
|
| |
|
|
|
|
|
| |
(#15625)
Synapse will no longer send (or respond to) the unstable flags
for faster joins. These were only available behind a configuration
flag and handled in parallel with the stable flags.
|
| | |
|
| |
|
|
| |
* Reproduce bad scenario in test
* Avoid catchup optimisation for partial state rooms
|
| | |
|
| |
|
|
| |
Adds a return type to HomeServerTestCase.make_homeserver and deal
with any variables which are no longer Any.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Make tests.federation pass mypy
* Untyped defs in tests.federation.transport
* test methods return None
* Remaining type hints in tests.federation
* Changelog
* Avoid an uncessary type-ignore
|
| | |
|
| |
|
|
|
| |
Serving partial join responses is no longer experimental. They will only be served under the stable identifier if the the undocumented config flag experimental.msc3706_enabled is set to true.
Synapse continues to request a partial join only if the undocumented config flag experimental.faster_joins is set to true; this setting remains present and unaffected.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Also use stable name in SendJoinResponse struct
follow-up to #14832
* Changelog
* Fix a rename I missed
* Run black
* Update synapse/federation/federation_client.py
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
* Use new query param when requesting a partial join
* Read new query param when serving partial join
* Provide new field names when serving partial joins
* Read new field names from partial join response
* Changelog
|
| |
|
|
| |
Use the newer foo_instances configuration instead of the
deprecated flags to enable specific features (e.g. start_pushers).
|
| |
|
| |
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Include the thread_id field when sending read receipts over
federation. This might result in the same user having multiple
read receipts per-room, meaning multiple EDUs must be sent
to encapsulate those receipts.
This restructures the PerDestinationQueue APIs to support
multiple receipt EDUs, queue_read_receipt now becomes linear
time in the number of queued threaded receipts in the room for
the given user, it is expected this is a small number since receipt
EDUs are sent as filler in transactions.
|
| |
|
|
|
|
|
|
|
|
|
| |
* Pull out hero selection logic
* Include heroes in partial join response's state
* Changelog
* Fixup trial test
* Remove TODO
|
| | |
|
| |
|
|
|
|
|
|
|
| |
from `destination` pattern (#14096)
1. `federation_client.timestamp_to_event(...)` now handles all `destination` looping and uses our generic `_try_destination_list(...)` helper.
2. Consistently handling `NotRetryingDestination` and `FederationDeniedError` across `get_pdu` , backfill, and the generic `_try_destination_list` which is used for many places we use this pattern.
3. `get_pdu(...)` now returns `PulledPduInfo` so we know which `destination` we ended up pulling the PDU from
|
| |
|
|
|
|
|
|
|
| |
This implements a fake OIDC server, which intercepts calls to the HTTP client.
Improves accuracy of tests by covering more internal methods.
One particular example was the ID token validation, which previously mocked.
This uncovered an incorrect dependency: Synapse actually requires at least
authlib 0.15.1, not 0.14.0.
|
| |
|
|
|
| |
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
|
| |
|
|
| |
(#14065)
|
| |
|
|
|
|
|
|
|
| |
Because we're doing the recording in `_check_sigs_and_hash_for_pulled_events_and_fetch` (previously named `_check_sigs_and_hash_and_fetch`), this means we will track signature failures for `backfill`, `get_room_state`, `get_event_auth`, and `get_missing_events` (all pulled event scenarios). And we also record signature failures from `get_pdu`.
Part of https://github.com/matrix-org/synapse/issues/13700
Part of https://github.com/matrix-org/synapse/issues/13676 and https://github.com/matrix-org/synapse/issues/13356
This PR will be especially important for https://github.com/matrix-org/synapse/pull/13816 so we can avoid the costly `_get_state_ids_after_missing_prev_event` down the line when `/messages` calls backfill.
|
| |
|
|
| |
Updates the `/receipts` endpoint and receipt EDU handler to parse a
`thread_id` from the body and insert it in the database.
|
| |
|
|
| |
other than just servlet methods. (#13662)
|
| |
|
|
|
| |
`get_current_hosts_in_room` (#13605)
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Update `get_pdu` to return the untouched, pristine `EventBase` as it was originally seen over federation (no metadata added). Previously, we returned the same `event` reference that we stored in the cache which downstream code modified in place and added metadata like setting it as an `outlier` and essentially poisoned our cache. Now we always return a copy of the `event` so the original can stay pristine in our cache and re-used for the next cache call.
Split out from https://github.com/matrix-org/synapse/pull/13205
As discussed at:
- https://github.com/matrix-org/synapse/pull/13205#discussion_r918365746
- https://github.com/matrix-org/synapse/pull/13205#discussion_r918366125
Related to https://github.com/matrix-org/synapse/issues/12584. This PR doesn't fix that issue because it hits [`get_event` which exists from the local database before it tries to `get_pdu`](https://github.com/matrix-org/synapse/blob/7864f33e286dec22368dc0b11c06eebb1462a51e/synapse/federation/federation_client.py#L581-L594).
|
| |
|
|
| |
(#13318)
|
| | |
|
| | |
|
| |
|
|
| |
(#13255)
|
| |
|
|
|
|
|
| |
* Reword failure message about `await_result=False`
* Use `reactor.advance()` instead of `reactor.pump()`
* Raise `AssertionError`s ourselves
* Un-instance method `_test_disconnect`
* Replace `ThreadedMemoryReactorClock` with `MemoryReactorClock`
|
| |
|
|
|
| |
federation (#12964)
Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
|
| | |
|
| |
|
| |
Instead of hard-coding strings in many places.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
RFC7230 (#12774)
The main differences are:
- values with delimiters (such as colons) should be quoted, so always
quote the origin, since it could contain a colon followed by a port
number
- should allow more than one space after "X-Matrix"
- quoted values with backslash-escaped characters should be unescaped
- names should be case insensitive
|
| |
|
|
|
|
|
|
| |
`BaseFederationServlet` wraps its endpoints in a bunch of async code
that has not been vetted for compatibility with cancellation.
Fail CI if a `@cancellable` flag is applied to a federation endpoint.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`BaseFederationServlet`s (#12699)
Both `RestServlet`s and `BaseFederationServlet`s register their handlers
with `HttpServer.register_paths` / `JsonResource.register_paths`. Update
`JsonResource` to respect the `@cancellable` flag on handlers registered
in this way.
Although `ReplicationEndpoint` also registers itself using
`register_paths`, it does not pass the handler method that would have the
`@cancellable` flag directly, and so needs separate handling.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
|
|
|
| |
When configuring the return values of mocks, prefer awaitables from
`make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable
once, so it is inappropriate for a mock to return the same `Deferred`
multiple times.
Also update `run_in_background` to support functions that return
arbitrary awaitables.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
| |
MSC2314 has now been closed, so we're backing out its implementation, which
originally happened in #6176.
Unfortunately it's not a direct revert, as that PR mixed in a bunch of
unrelated changes to tests etc.
|
| |
|
| |
Broke in #12365
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a first step in dealing with #7721.
The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:
1. we can defer calculating the set of remote servers that need to be poked about the change; and
2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.
However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled.
There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).
Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
|
| | |
|
| | |
|
| |
|
|
| |
transactions (#12098)
|
| | |
|
| |
|
|
|
|
|
| |
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
|
| |
|
|
| |
This is an endpoint that we have server-side support for, but no client-side support. It's going to be useful for resyncing partial-stated rooms, so let's introduce it.
|
| |
|
| |
Part of my work on #11249: add code to handle the new fields added in MSC3706.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* Make `get_auth_chain_ids` return a Set
It has a set internally, and a set is often useful where it gets used, so let's
avoid converting to an intermediate list.
* Minor refactors in `on_send_join_request`
A little bit of non-functional groundwork
* Implement MSC3706: partial state in /send_join response
|
| |
|
|
|
|
|
|
|
|
|
|
| |
If we prepopulate the test homeserver with a key for a remote homeserver, we
can make federation requests to it without having to stub out the
authenticator. This has two advantages:
* means that what we are testing is closer to reality (ie, we now have
complete tests for the incoming-request-authorisation flow)
* some tests require that other objects be signed by the remote server (eg,
the event in `/send_join`), and doing that would require a whole separate
set of mocking out. It's much simpler just to use real keys.
|
| |
|
|
| |
Invites and knocks will now include the topic in the stripped state
send to clients before joining the room.
|
| | |
|
| | |
|
| |
|
| |
This would have caught the bug #11438 introduced in #11217 and fixed in #11439.
|
| |
|
|
|
| |
Fix some harmless errors from background processes (mostly
due to awaiting Mock objects) that occurred in the Synapse
logs during unit tests.
|
| | |
|
| |
|
| |
The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
|
| | |
|
| |
|
| |
Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
|
| |
|
|
|
|
|
| |
endpoints. (#10167)
* Room version 7 for knocking.
* Stable prefixes and endpoints (both client and federation) for knocking.
* Removes the experimental configuration flag.
|
| |
|
|
|
|
| |
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403
Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* tests for push rule pattern matching
* tests for acl pattern matching
* factor out common `re.escape`
* Factor out common re.compile
* Factor out common anchoring code
* add word_boundary support to `glob_to_regex`
* Use `glob_to_regex` in push rule evaluator
NB that this drops support for character classes. I don't think anyone ever
used them.
* Improve efficiency of globs with multiple wildcards
The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.
* add assertion on regex pattern
* Fix mypy
* Simplify glob_to_regex
* Inline the glob_to_regex helper function
Signed-off-by: Dan Callahan <danc@element.io>
* Moar comments
Signed-off-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.
Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
|
| |
|
|
|
|
|
|
|
|
| |
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.
Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| |
|
|
|
|
|
|
|
|
| |
The `RoomDirectoryFederationTests` tests were not being run unless explicitly called as an `__init__.py` file was not present in `tests/federation/transport/`. Thus the folder was not a python module, and `trial` did not look inside for any test cases to run. This was found while working on #6739.
This PR adds a `__init__.py` and also fixes the test in a couple ways:
- Switch to subclassing `unittest.FederatingHomeserverTestCase` instead, which sets up federation endpoints for us.
- Supply a `federation_auth_origin` to `make_request` in order to more act like the request is coming from another server, instead of just an unauthenicated client requesting a federation endpoint.
I found that the second point makes no difference to the test passing, but felt like the right thing to do if we're testing over federation.
|
| |
|
|
| |
This was never used, so let's get rid of it.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix _set_destination_retry_timings
This came about because the code assumed that retry_interval
could not be NULL — which has been challenged by catch-up.
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.20.0rc3 (2020-09-11)
==============================
Bugfixes
--------
- Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
|
| | | |
|
| | | |
|
| | | |
|
| |/
|
| |
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Fixes #7901.
Signed-off-by: Niklas Tittjung <nik_t.01@web.de>
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Fixes https://github.com/matrix-org/synapse/issues/2431
Adds config option `encryption_enabled_by_default_for_room_type`, which determines whether encryption should be enabled with the default encryption algorithm in private or public rooms upon creation. Whether the room is private or public is decided based upon the room creation preset that is used.
Part of this PR is also pulling out all of the individual instances of `m.megolm.v1.aes-sha2` into a constant variable to eliminate typos ala https://github.com/matrix-org/synapse/pull/7637
Based on #7637
|
| |
|
|
|
|
|
|
| |
@uhoreg has confirmed these were both typos. They are only in comments and tests though, rather than anything critical.
Introduced in:
* https://github.com/matrix-org/synapse/pull/7157
* https://github.com/matrix-org/synapse/pull/5726
|
| |
|
|
|
|
|
| |
The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room).
Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on.
People probably want to look at this commit by commit.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Occasionally we could get a federation device list update transaction which
looked like:
```
[
{'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D2', 'prev_id': [], 'stream_id': 12, 'deleted': True}},
{'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D1', 'prev_id': [12], 'stream_id': 11, 'deleted': True}},
{'edu_type': 'm.device_list_update', 'content': {'user_id': '@user:test', 'device_id': 'D3', 'prev_id': [11], 'stream_id': 13, 'deleted': True}}
]
```
Having `stream_ids` which are lower than `prev_ids` looks odd. It might work
(I'm not actually sure), but in any case it doesn't seem like a reasonable
thing to expect other implementations to support.
|
| |
|
|
| |
make sure we clear out all but one update for the user
|
| | |
|
| |
|
|
|
| |
this is never set to anything other than "test", and is a source of unnecessary
boilerplate.
|
| |
|
|
| |
Ensure good comprehension hygiene using flake8-comprehensions.
|
| |
|
|
|
|
|
| |
... and use it in places where it's trivial to do so.
This will make it easier to pass room versions into the FrozenEvent
constructors.
|
| |
|
|
| |
Ensure that the the default settings for the room directory are that the it is hidden from public view by default.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Rate-limit outgoing read-receipts as per #4730.
|
| | |
|
| | |
|
| |
|
|
|
| |
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
setup_test_homeserver function in utils.
|
| |
|
|
| |
from a remote server if we are missing some of them
|
| | |
|
| |
|
|
| |
retries
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
Conflicts:
synapse/federation/replication.py
|
| | |
| |
| |
| | |
schema
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| |
|
|
| |
from the transaction itself
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
Conflicts:
synapse/app/homeserver.py
|
| | |
| |
| |
| | |
than passing MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES as a fake query string parameter
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
| |
TestCase; set up logging in ONE PLACE ONLY
|
| |
|
|
| |
hasn't been incorporated in time for launch.
|
| |
|
|
|
|
| |
than /matrix to make it easier for existing websites to mount a HS in their namespace without collisions.
perl -pi -e 's#/matrix#/_matrix#g' ./cmdclient/console.py ./docs/client-server/howto.rst ./docs/client-server/specification.rst ./docs/client-server/swagger_matrix/directory ./docs/client-server/swagger_matrix/events ./docs/client-server/swagger_matrix/login ./docs/client-server/swagger_matrix/presence ./docs/client-server/swagger_matrix/profile ./docs/client-server/swagger_matrix/registration ./docs/client-server/swagger_matrix/rooms ./docs/server-server/specification.rst ./graph/graph.py ./jsfiddles/create_room_send_msg/demo.js ./jsfiddles/event_stream/demo.js ./jsfiddles/example_app/demo.js ./jsfiddles/register_login/demo.js ./jsfiddles/room_memberships/demo.js ./synapse/api/urls.py ./tests/federation/test_federation.py ./tests/handlers/test_presence.py ./tests/handlers/test_typing.py ./tests/rest/test_events.py ./tests/rest/test_presence.py ./tests/rest/test_profile.py ./tests/rest/test_rooms.py ./webclient/components/fileUpload/file-upload-service.js ./webclient/components/matrix/matrix-service.js
|
| | |
|
| |
|
|
| |
resource rather than an entire server
|
| |
|
|
| |
resource_for_federation or resource_for_client depending on what is being tested.
|
| | |
|
| |
|
|
| |
handling Queries on the Federation's increasingly-inaccurately-named ReplicationLayer
|
| | |
|
| |
|
|
| |
copyrighter.pl whilst we're at it
|
| |
|
