summary refs log tree commit diff
path: root/tests/handlers (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add tests for restoring the presence state after a restart. (#16151)Patrick Cloke2023-08-231-0/+116
|
* Clean-up presence tests (#16158)Patrick Cloke2023-08-221-92/+37
| | | Reduce duplicated code & remove unused variables.
* Add an admin endpoint to allow authorizing server to signal token ↵Shay2023-08-221-1/+33
| | | | revocations (#16125)
* Fix user directory test for deactivated support user. (#16157)Patrick Cloke2023-08-221-8/+11
| | | | Support users should not be added to the user directory after being deactivated.
* MSC3861: allow impersonation by an admin using a query param (#16132)Mathieu Velten2023-08-181-0/+35
|
* Cache token introspection response from OIDC provider (#16117)Shay2023-08-171-0/+62
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-081-4/+5
|
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-311-0/+74
| | | | | (#15791) c.f. #13476
* Fix 404 on /profile when the display name is empty but not the avatar (#16012)Mathieu Velten2023-07-271-0/+10
|
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-241-1/+98
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-184-4/+12
| | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Mark get_user_in_directory private since only used in tests (#15884)Mathieu Velten2023-07-121-9/+9
|
* Add + as an allowed character for Matrix IDs (MSC4009) (#15911)Patrick Cloke2023-07-111-6/+5
|
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-104-12/+4
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Federation outbound proxy (#15773)Eric Eastwood2023-07-054-4/+12
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Fix forgotten rooms missing in initial sync (#15815)Nicolas Werner2023-06-211-0/+21
| | | | | | | | If you leave a room and forget it, then rejoin it, the room would be missing from the next initial sync. fixes #13262 Signed-off-by: Nicolas Werner <n.werner@famedly.com>
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-081-1/+1
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-021-20/+8
| | | | `profiles` and `user_filters` (#15649)
* Enforce that an admin token also has the basic Matrix API scopeQuentin Gliech2023-05-301-1/+25
|
* Reject tokens with multiple device scopesQuentin Gliech2023-05-301-1/+28
|
* Handle errors when introspecting tokensQuentin Gliech2023-05-301-0/+35
| | | | | This returns a proper 503 when the introspection endpoint is not working for some reason, which should avoid logging out clients in those cases.
* Disable incompatible Admin API endpointsQuentin Gliech2023-05-301-0/+19
|
* Make AS tokens work & allow ASes to /registerQuentin Gliech2023-05-301-2/+2
|
* Refactor config to be an experimental featureHugh Nimmo-Smith2023-05-301-6/+9
| | | | Also enforce you can't combine it with incompatible config options
* Disable account related endpoints when using OAuth delegationQuentin Gliech2023-05-301-3/+177
|
* Actually enforce guest + return www-authenticate headerHugh Nimmo-Smith2023-05-301-3/+40
|
* Initial tests for OAuth delegationHugh Nimmo-Smith2023-05-301-0/+345
|
* Make the api.auth.Auth a ProtocolQuentin Gliech2023-05-301-2/+2
|
* Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-241-0/+95
| | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* Add requesting user id parameter to key claim methods in ↵Shay2023-05-241-1/+15
| | | | `TransportLayerClient` (#15663)
* Consolidate logic to check for deactivated users. (#15634)Patrick Cloke2023-05-231-38/+21
| | | | | | | This moves the deactivated user check to the method which all login types call. Additionally updates the application service tests to be more realistic by removing invalid tests and fixing server names.
* Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-191-1/+1
| | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* Check appservices for devices during a /user/devices query. (#15539)Patrick Cloke2023-05-051-1/+134
| | | | | | | | MSC3984 proxies /keys/query requests to appservices, but servers will can also requests devices / keys from the /user/devices endpoint. The formats are close enough that we can "proxy" that /user/devices to appservices (by calling /keys/query) and then change the format of the returned data before returning it over federation.
* Implement MSC4009 to widen the allowed Matrix ID grammar (#15536)Patrick Cloke2023-05-051-0/+13
| | | | | Behind a configuration flag this adds + to the list of allowed characters in Matrix IDs. The main feature this enables is using full E.164 phone numbers as Matrix IDs.
* Add config option to forget rooms automatically when users leave them (#15224)Sean Quah2023-05-031-0/+11
| | | | | This is largely based off the stats and user directory updater code. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add support for claiming multiple OTKs at once. (#15468)Patrick Cloke2023-04-271-18/+14
| | | | | | | MSC3983 provides a way to request multiple OTKs at once from appservices, this extends this concept to the Client-Server API. Note that this will likely be spit out into a separate MSC, but is currently part of MSC3983.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-261-14/+12
|
* Add unstable /keys/claim endpoint which always returns fallback keys. (#15462)Patrick Cloke2023-04-251-6/+235
| | | | | | | | | | | | | It can be useful to always return the fallback key when attempting to claim keys. This adds an unstable endpoint for `/keys/claim` which always returns fallback keys in addition to one-time-keys. The fallback key(s) are not marked as "used" unless there are no corresponding OTKs. This is currently defined in MSC3983 (although likely to be split out to a separate MSC). The endpoint shape may change or be requested differently (i.e. a keyword parameter on the current endpoint), but the core logic should be reasonable.
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-181-1/+1
|
* Convert async to normal tests in `TestSSOHandler` (#15433)Dirk Klimpel2023-04-141-4/+4
| | | | | * Convert async to normal tests in `TestSSOHandler` * newsfile
* Revert pruning of old devices (#15360)Erik Johnston2023-03-312-2/+2
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* Implement MSC3984 to proxy /keys/query requests to appservices. (#15321)Patrick Cloke2023-03-301-1/+120
| | | | | If enabled, for users which are exclusively owned by an application service then the appservice will be queried for devices in addition to any information stored in the Synapse database.
* Add the ability to enable/disable registrations when in the OIDC flow (#14978)Warren Bailey2023-03-301-1/+16
| | | Signed-off-by: Warren Bailey <warren@warrenbailey.net>
* Delete stale non-e2e devices for users, take 3 (#15183)Erik Johnston2023-03-292-2/+2
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Implement MSC3983 to proxy /keys/claim queries to appservices. (#15314)Patrick Cloke2023-03-281-1/+75
| | | | | | Experimental support for MSC3983 is behind a configuration flag. If enabled, for users which are exclusively owned by an application service then the appservice will be queried for one-time keys *if* there are none uploaded to Synapse.
* Refresh remote profiles that have been marked as stale, in order to fill the ↵reivilibre2023-03-161-2/+185
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | user directory. [rei:userdirpriv] (#14756) * Scaffolding for background process to refresh profiles * Add scaffolding for background process to refresh profiles for a given server * Implement the code to select servers to refresh from * Ensure we don't build up multiple looping calls * Make `get_profile` able to respect backoffs * Add logic for refreshing users * When backing off, schedule a refresh when the backoff is over * Wake up the background processes when we receive an interesting state event * Add tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Add comment about 1<<62 --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Batch up storing state groups when creating new room (#14918)Shay2023-02-242-9/+19
|
* Add information on uploaded media to user export command. (#15107)Dirk Klimpel2023-02-231-0/+29
|
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-222-2/+0
|
* Fix a bug introduced in Synapse v1.74.0 where searching with colons when ↵reivilibre2023-02-201-0/+7
| | | | | | using ICU for search term tokenisation would fail with an error. (#15079) Co-authored-by: David Robertson <davidr@element.io>
* Add account data to export command (#14969)Dirk Klimpel2023-02-171-0/+27
| | | | | | | | | | | * Add account data to to export command * newsfile * remove not needed function * update newsfile * adopt #14973
* Use mypy 1.0 (#15052)David Robertson2023-02-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Add final type hint to tests.unittest. (#15072)Patrick Cloke2023-02-1411-78/+93
| | | | Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.
* Faster joins: don't stall when a user joins during a fast join (#14606)Mathieu Velten2023-02-101-20/+20
| | | | | | | | | | | | | | | | Fixes #12801. Complement tests are at https://github.com/matrix-org/complement/pull/567. Avoid blocking on full state when handling a subsequent join into a partial state room. Also always perform a remote join into partial state rooms, since we do not know whether the joining user has been banned and want to avoid leaking history to banned users. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Add a class UnpersistedEventContext to allow for the batching up of storing ↵Shay2023-02-091-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | state groups (#14675) * add class UnpersistedEventContext * modify create new client event to create unpersistedeventcontexts * persist event contexts after creation * fix tests to persist unpersisted event contexts * cleanup * misc lints + cleanup * changelog + fix comments * lints * fix batch insertion? * reduce redundant calculation * add unpersisted event classes * rework compute_event_context, split into function that returns unpersisted event context and then persists it * use calculate_context_info to create unpersisted event contexts * update typing * $%#^&* * black * fix comments and consolidate classes, use attr.s for class * requested changes * lint * requested changes * requested changes * refactor to be stupidly explicit * clearer renaming and flow * make partial state non-optional * update docstrings --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Add more missing type hints to tests. (#15028)Patrick Cloke2023-02-081-2/+2
|
* Faster joins: Refactor handling of servers in room (#14954)Sean Quah2023-02-032-2/+2
| | | | | | | | | | | | | | Ensure that the list of servers in a partial state room always contains the server we joined off. Also refactor `get_partial_state_servers_at_join` to return `None` when the given room is no longer partial stated, to explicitly indicate when the room has partial state. Otherwise it's not clear whether an empty list means that the room has full state, or the room is partial stated, but the server we joined off told us that there are no servers in the room. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add more user information to export-data command. (#14894)Dirk Klimpel2023-02-011-0/+60
| | | | | | * The user's profile information. * The user's devices. * The user's connections / IP address information.
* Fix type hints in typing edu unit tests (#14886)Andrew Morgan2023-01-261-40/+59
|
* Faster joins: Avoid starting duplicate partial state syncs (#14844)Sean Quah2023-01-201-1/+111
| | | | | | | | | | | | | | | | | | Currently, we will try to start a new partial state sync every time we perform a remote join, which is undesirable if there is already one running for a given room. We intend to perform remote joins whenever additional local users wish to join a partial state room, so let's ensure that we do not start more than one concurrent partial state sync for any given room. ------------------------------------------------------------------------ There is a race condition where the homeserver leaves a room and later rejoins while the partial state sync from the previous membership is still running. There is no guarantee that the previous partial state sync will process the latest join, so we restart it if needed. Signed-off-by: Sean Quah <seanq@matrix.org>
* Support RFC7636 PKCE in the OAuth 2.0 flow. (#14750)Patrick Cloke2023-01-041-8/+144
| | | | | | | PKCE can protect against certain attacks and is enabled by default. Support can be controlled manually by setting the pkce_method of each oidc_providers entry to 'auto' (default), 'always', or 'never'. This is required by Twitter OAuth 2.0 support.
* Add missing type hints to tests.handlers. (#14680)Patrick Cloke2022-12-1619-373/+524
| | | And do not allow untyped defs in tests.handlers.
* Revert the deletion of stale devices due to performance issues. (#14662)reivilibre2022-12-121-32/+1
|
* Limit the number of devices we delete at once (#14649)Erik Johnston2022-12-091-0/+31
|
* Delete stale non-e2e devices for users, take 2 (#14595)Erik Johnston2022-12-091-1/+1
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Improve logging and opentracing for to-device message handling (#14598)Richard van der Hoff2022-12-061-1/+6
| | | | | | | A batch of changes intended to make it easier to trace to-device messages through the system. The intention here is that a client can set a property org.matrix.msgid in any to-device message it sends. That ID is then included in any tracing or logging related to the message. (Suggestions as to where this field should be documented welcome. I'm not enthusiastic about speccing it - it's very much an optional extra to help with debugging.) I've also generally improved the data we send to opentracing for these messages.
* Merge branch 'release-v1.73' into developDavid Robertson2022-12-011-1/+1
|\
| * Revert "POC delete stale non-e2e devices for users (#14038)" (#14582)David Robertson2022-11-291-1/+1
| |
* | Modernize unit tests configuration settings for workers. (#14568)realtyem2022-12-013-5/+11
|/ | | | Use the newer foo_instances configuration instead of the deprecated flags to enable specific features (e.g. start_pushers).
* POC delete stale non-e2e devices for users (#14038)Erik Johnston2022-11-291-1/+1
| | | | | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Use `device_one_time_keys_count` to match MSC3202 (#14565)Andrew Ferrazzutti2022-11-281-2/+2
| | | | | | | | | | | | | | | | | | | * Use `device_one_time_keys_count` to match MSC3202 Rename the `device_one_time_key_counts` key in responses to `device_one_time_keys_count` to match the name specified by MSC3202. Also change related variable/class names for consistency. Signed-off-by: Andrew Ferrazzutti <andrewf@element.io> * Update changelog.d/14565.misc * Revert name change for `one_time_key_counts` key as this is a different key altogether from `device_one_time_keys_count`, which is used for `/sync` instead of appservice transactions. Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
* Add support for handling avatar with SSO login (#13917)Ashish Kumar2022-11-251-0/+145
| | | | | | | | This commit adds support for handling a provided avatar picture URL when logging in via SSO. Signed-off-by: Ashish Kumar <ashfame@users.noreply.github.com> Fixes #9357.
* Add a type hint for `get_device_handler()` and fix incorrect types. (#14055)Patrick Cloke2022-11-221-6/+13
| | | | | This was the last untyped handler from the HomeServer object. Since it was being treated as Any (and thus unchecked) it was being used incorrectly in a few places.
* Fix an invalid comparison of `UserPresenceState` to `str` (#14393)Andrew Morgan2022-11-161-6/+35
|
* Check appservice user interest against the local users instead of all users ↵Eric Eastwood2022-10-271-5/+157
| | | | (`get_users_in_room` mis-use) (#13958)
* fix broken avatar checks when server_name contains a port (#13927)Ashish Kumar2022-10-261-0/+49
| | | | | | Fixes check_avatar_size_and_mime_type() to successfully update avatars on homeservers running on non-default ports which it would mistakenly treat as remote homeserver while validating the avatar's size and mime type. Signed-off-by: Ashish Kumar ashfame@users.noreply.github.com
* Save login tokens in database (#13844)Quentin Gliech2022-10-261-57/+78
| | | | | | | | | | | * Save login tokens in database Signed-off-by: Quentin Gliech <quenting@element.io> * Add upgrade notes * Track login token reuse in a Prometheus metric Signed-off-by: Quentin Gliech <quenting@element.io>
* Refactor OIDC tests to better mimic an actual OIDC provider. (#13910)Quentin Gliech2022-10-251-330/+250
| | | | | | | | | This implements a fake OIDC server, which intercepts calls to the HTTP client. Improves accuracy of tests by covering more internal methods. One particular example was the ID token validation, which previously mocked. This uncovered an incorrect dependency: Synapse actually requires at least authlib 0.15.1, not 0.14.0.
* Avoid checking the event cache when backfilling events (#14164)Andrew Morgan2022-10-181-1/+104
|
* Stop getting missing `prev_events` after we already know their signature is ↵Eric Eastwood2022-10-151-2/+199
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | invalid (#13816) While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place. Related to - https://github.com/matrix-org/synapse/issues/13622 - https://github.com/matrix-org/synapse/pull/13635 - https://github.com/matrix-org/synapse/issues/13676 Part of https://github.com/matrix-org/synapse/issues/13356 Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures. With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now. For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761 To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid. - `backfill` - `outgoing-federation-request` `/backfill` - `_check_sigs_and_hash_and_fetch` - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill - ❗ `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - ❗ `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - `_process_pulled_events` - `_process_pulled_event` for each validated event - ❗ Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it - `_get_state_ids_after_missing_prev_event` - `outgoing-federation-request` `/state_ids` - ❗ `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again - ❗ `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
* Fix a bug where redactions were not being sent over federation if we did not ↵Shay2022-10-111-7/+11
| | | | have the original event. (#13813)
* Optimise get_rooms_for_user (drop with_stream_ordering) (#13787)Nick Mills-Barrett2022-09-291-0/+1
|
* Persist CreateRoom events to DB in a batch (#13800)Shay2022-09-282-3/+11
|
* typing: check origin server of typing event against room's servers (#13830)Mathieu Velten2022-09-261-0/+4
| | | | | | | This is also using the partial state approximation if needed so we do not block here during a fast join. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Accept & store thread IDs for receipts (implement MSC3771). (#13782)Patrick Cloke2022-09-231-0/+1
| | | | Updates the `/receipts` endpoint and receipt EDU handler to parse a `thread_id` from the body and insert it in the database.
* Faster Remote Room Joins: tell remote homeservers that we are unable to ↵reivilibre2022-09-231-1/+1
| | | | authorise them if they query a room which has partial state on our server. (#13823)
* Port the push rule classes to Rust. (#13768)Erik Johnston2022-09-201-15/+12
|
* Avoid putting rejected events in room state (#13723)Sean Quah2022-09-161-0/+399
| | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Keep track when we try and fail to process a pulled event (#13589)Eric Eastwood2022-09-141-0/+222
| | | | | | | | | | | | | | We can follow-up this PR with: 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700 Fix https://github.com/matrix-org/synapse/issues/13621 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
* Fix bug in device list caching when remote users leave rooms (#13749)Sean Quah2022-09-141-1/+7
| | | | | | | | | | | | When a remote user leaves the last room shared with the homeserver, we have to mark their device list as unsubscribed, otherwise we would hold on to a stale device list in our cache. Crucially, the device list would remain cached even after the remote user rejoined the room, which could lead to E2EE failures until the next change to the remote user's device list. Fixes #13651. Signed-off-by: Sean Quah <seanq@matrix.org>
* Remove support for unstable private read receipts (#13653)Šimon Brandner2022-09-011-35/+13
| | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Fix that user cannot `/forget` rooms after the last member has left (#13546)Dirk Klimpel2022-08-301-2/+91
|
* `synapse.api.auth.Auth` cleanup: make permission-related methods use ↵Quentin Gliech2022-08-221-3/+5
| | | | | | | | | `Requester` instead of the `UserID` (#13024) Part of #13019 This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it. It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
* Register homeserver modules when creating test homeserver (#13558)Andrew Morgan2022-08-192-18/+0
|
* Reduce the number of tests using TCP replication. (#13543)Patrick Cloke2022-08-191-2/+2
| | | | Uses Redis replication in additional test cases (instead of TCP replication). A small step towards dropping TCP replication.
* Fix validation problem that occurs when a user tries to deactivate their ↵reivilibre2022-08-191-0/+15
| | | | account or change their password. (#13563)
* Make push rules use proper structures. (#13522)Erik Johnston2022-08-161-16/+17
| | | | | | | | | | | | This improves load times for push rules: | Version | Time per user | Time for 1k users | | -------------------- | ------------- | ----------------- | | Before | 138 µs | 138ms | | Now (with custom) | 2.11 µs | 2.11ms | | Now (without custom) | 49.7 ns | 0.05 ms | This therefore has a large impact on send times for rooms with large numbers of local users in the room.
* Support stable identifiers for MSC2285: private read receipts. (#13273)Šimon Brandner2022-08-051-16/+42
| | | | | This adds support for the stable identifiers of MSC2285 while continuing to support the unstable identifiers behind the configuration flag. These will be removed in a future version.
* Use literals in place of `HTTPStatus` constants in tests (#13463)Dirk Klimpel2022-08-053-5/+3
|
* Faster Room Joins: don't leave a stuck room partial state flag if the join ↵reivilibre2022-08-011-1/+121
| | | | fails. (#13403)
* Refactor `_resolve_state_at_missing_prevs` to return an `EventContext` (#13404)Sean Quah2022-08-011-4/+11
| | | | | | | | Previously, `_resolve_state_at_missing_prevs` returned the resolved state before an event and a partial state flag. These were unwieldy to carry around would only ever be used to build an event context. Build the event context directly instead. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add missing type hints for tests.unittest. (#13397)Patrick Cloke2022-07-271-10/+2
|
* Faster room joins: avoid blocking when pulling events with missing prevs ↵Sean Quah2022-07-261-0/+1
| | | | | | | | | (#13355) Avoid blocking on full state in `_resolve_state_at_missing_prevs` and return a new flag indicating whether the resolved state is partial. Thread that flag around so that it makes it into the event context. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Support Implicit TLS for sending emails (#13317)Jan Schär2022-07-251-1/+56
| | | | | | | | | | Previously, TLS could only be used with STARTTLS. Add a new option `force_tls`, where TLS is used from the start. Implicit TLS is recommended over STARTLS, see https://datatracker.ietf.org/doc/html/rfc8314 Fixes #8046. Signed-off-by: Jan Schär <jan@jschaer.ch>
* Rate limit joins per-room (#13276)David Robertson2022-07-191-0/+290
|
* Use HTTPStatus constants in place of literals in tests. (#13297)Dirk Klimpel2022-07-151-20/+21
|
* Async get event cache prep (#13242)Nick Mills-Barrett2022-07-151-1/+1
| | | | | Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)
* Federation Sender & Appservice Pusher Stream Optimisations (#13251)Nick Mills-Barrett2022-07-151-8/+8
| | | | | | | | | | | | | * Replace `get_new_events_for_appservice` with `get_all_new_events_stream` The functions were near identical and this brings the AS worker closer to the way federation senders work which can allow for multiple workers to handle AS traffic. * Pull received TS alongside events when processing the stream This avoids an extra query -per event- when both federation sender and appservice pusher process events.
* Rip out auth-event reconciliation code (#12943)Richard van der Hoff2022-07-141-139/+1
| | | | | | | There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution. This isn't specced, and there's general agreement we shouldn't be doing it. However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.
* Rename test case method to `add_hashes_and_signatures_from_other_server` ↵David Robertson2022-07-122-4/+4
| | | | (#13255)
* Fix inconsistencies in event validation (#13088)Richard van der Hoff2022-06-172-5/+10
|
* Fix typechecks against twisted trunk (#13061)David Robertson2022-06-151-4/+4
|
* Move the "email unsubscribe" resource, refactor the macaroon generator & ↵Quentin Gliech2022-06-141-4/+3
| | | | | | | | | | | | | | | | | | | | | simplify the access token verification logic. (#12986) This simplifies the access token verification logic by removing the `rights` parameter which was only ever used for the unsubscribe link in email notifications. The latter has been moved under the `/_synapse` namespace, since it is not a standard API. This also makes the email verification link more secure, by embedding the app_id and pushkey in the macaroon and verifying it. This prevents the user from tampering the query parameters of that unsubscribe link. Macaroon generation is refactored: - Centralised all macaroon generation and verification logic to the `MacaroonGenerator` - Moved to `synapse.utils` - Changed the constructor to require only a `Clock`, hostname, and a secret key (instead of a full `Homeserver`). - Added tests for all methods.
* Decouple `synapse.api.auth_blocking.AuthBlocking` from ↵Quentin Gliech2022-06-143-3/+3
| | | | `synapse.api.auth.Auth`. (#13021)
* Replace noop background updates with DELETE. (#12954)Patrick Cloke2022-06-131-28/+0
| | | | Removes the `register_noop_background_update` and deletes the background updates directly in a delta file.
* Stop depending on `room_id` to be returned for children state in the ↵Patrick Cloke2022-06-101-1/+1
| | | | | | | | | | | hierarchy response. (#12991) The `room_id` field was removed from MSC2946 before it was accepted. It was initially kept for backwards compatibility and should be removed now that the stable form of the API is used. This change only stops Synapse from validating that it is returned, a future PR will remove returning it as part of the response.
* Consolidate the logic of delete_device/delete_devices. (#12970)Patrick Cloke2022-06-071-2/+2
| | | | | | | | By always using delete_devices and sometimes passing a list with a single device ID. Previously these methods had gotten out of sync with each other and it seems there's little benefit to the single-device variant.
* Reduce state pulled from DB due to sending typing and receipts over ↵Erik Johnston2022-06-061-2/+4
| | | | | federation (#12964) Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
* Reduce the amount of state we pull from the DB (#12811)Erik Johnston2022-06-061-1/+2
|
* Wait for lazy join to complete when getting current state (#12872)Erik Johnston2022-06-013-4/+8
|
* Remove remaining bits of groups code. (#12936)Patrick Cloke2022-06-012-4/+0
| | | | | | * Update worker docs to remove group endpoints. * Removes an unused parameter to `ApplicationService`. * Break dependency between media repo and groups. * Avoid copying `m.room.related_groups` state events during room upgrades.
* Rename storage classes (#12913)Erik Johnston2022-05-314-12/+19
|
* Additional constants for EDU types. (#12884)Patrick Cloke2022-05-273-27/+41
| | | Instead of hard-coding strings in many places.
* Remove unstable APIs for /hierarchy. (#12851)Patrick Cloke2022-05-261-10/+10
| | | | Removes the unstable endpoint as well as a duplicated field which was modified during stabilization.
* Pull out less state when handling gaps mk2 (#12852)Erik Johnston2022-05-261-1/+5
|
* Fix up `state_store` naming (#12871)Erik Johnston2022-05-251-2/+2
|
* Add some type hints to `event_federation` datastore (#12753)Dirk Klimpel2022-05-181-0/+1
| | | Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
* Discard null-containing strings before updating the user directory (#12762)David Robertson2022-05-181-0/+28
|
* Remove code which updates `application_services_state.last_txn` (#12680)Richard van der Hoff2022-05-171-10/+0
| | | | This column is unused as of #12209, so let's stop writing to it.
* Track in memory events using weakrefs (#10533)Erik Johnston2022-05-171-0/+1
|
* Avoid unnecessary copies when filtering private read receipts. (#12711)Šimon Brandner2022-05-161-38/+26
| | | | | | A minor optimization to avoid unnecessary copying/building identical dictionaries when filtering private read receipts. Also clarifies comments and cleans-up some tests.
* Refactor `EventContext` (#12689)Erik Johnston2022-05-101-1/+3
| | | | | | | | | | Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing. The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used. Part of #12684
* Support MSC3266 room summaries over federation (#11507)DeepBlueV7.X2022-05-051-0/+26
| | | | Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
* Use `private` instead of `hidden` in MSC2285 related code. (#12635)Šimon Brandner2022-05-051-16/+16
|
* Use `getClientAddress` instead of `getClientIP`. (#12599)Patrick Cloke2022-05-043-4/+4
| | | | | getClientIP was deprecated in Twisted 18.4.0, which also added getClientAddress. The Synapse minimum version for Twisted is currently 18.9.0, so all supported versions have the new API.
* Implement changes to MSC2285 (hidden read receipts) (#12168)Šimon Brandner2022-05-041-53/+76
| | | | | * Changes hidden read receipts to be a separate receipt type (instead of a field on `m.read`). * Updates the `/receipts` endpoint to accept `m.fully_read`.
* Remove unstable/unspecced login types. (#12597)Patrick Cloke2022-05-041-3/+1
| | | | | | * `m.login.jwt`, which was never specced and has been deprecated since Synapse 1.16.0. (`org.matrix.login.jwt` can be used instead.) * `uk.half-shot.msc2778.login.application_service`, which was stabilized as part of the Matrix spec v1.2 release.
* Use constants for receipt types in tests. (#12582)Šimon Brandner2022-04-281-19/+19
|
* Fix logging of incorrect status codes for disconnected requests (#12580)Sean Quah2022-04-282-2/+26
| | | | | | | | | | | | | | The status code of requests must always be set, regardless of client disconnection, otherwise they will always be logged as 200!. Broken for `respond_with_json` in f48792eec43f893f4f893ffdcbf00f8958b6f6b5. Broken for `respond_with_json_bytes` in 3e58ce72b42f2ae473c1e76a967548cd6fa7e2e6. Broken for `respond_with_html_bytes` in ea26e9a98b0541fc886a1cb826a38352b7599dbe. Signed-off-by: Sean Quah <seanq@element.io>
* Prefer `make_awaitable` over `defer.succeed` in tests (#12505)Sean Quah2022-04-274-28/+25
| | | | | | | | | | | When configuring the return values of mocks, prefer awaitables from `make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable once, so it is inappropriate for a mock to return the same `Deferred` multiple times. Also update `run_in_background` to support functions that return arbitrary awaitables. Signed-off-by: Sean Quah <seanq@element.io>
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-6/+3
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Bound ephemeral events by key (#12544)Nick Mills-Barrett2022-04-261-0/+82
| | | | Co-authored-by: Brad Murray <bradtgmurray@gmail.com> Co-authored-by: Andrew Morgan <andrewm@element.io>
* Prevent a sync request from removing a user's busy presence status (#12213)David Baker2022-04-131-0/+79
| | | | | | | | | | In trying to use the MSC3026 busy presence status, the user's status would be set back to 'online' next time they synced. This change makes it so that syncing does not affect a user's presence status if it is currently set to 'busy': it must be removed through the presence API. The MSC defers to implementations on the behaviour of busy presence, so this ought to remain compatible with the MSC.
* Add type hints for `tests/unittest.py`. (#12347)Richard van der Hoff2022-04-014-7/+13
| | | In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
* Remove redundant `get_success` calls in test code (#12346)Richard van der Hoff2022-04-013-17/+14
| | | There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
* Raise an exception when getting state at an outlier (#12191)Richard van der Hoff2022-04-011-18/+54
| | | | | | It seems like calling `_get_state_group_for_events` for an event where the state is unknown is an error. Accordingly, let's raise an exception rather than silently returning an empty result.
* Optimise `_get_state_after_missing_prev_event`: use `/state` (#12040)Richard van der Hoff2022-04-011-0/+225
| | | If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-301-1/+120
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Bump `black` and `click` versions (#12320)David Robertson2022-03-291-3/+3
|
* Always allow the empty string as an avatar_url. (#12261)David Robertson2022-03-251-0/+6
| | | | | Hopefully this fixes #12257. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add type hints to tests files. (#12256)Dirk Klimpel2022-03-211-15/+20
|
* Add type hints to more tests files. (#12240)Dirk Klimpel2022-03-173-26/+42
|
* Add type hints to some tests/handlers files. (#12224)Dirk Klimpel2022-03-155-126/+155
|
* Add some type hints to the tests.handlers module. (#12207)Patrick Cloke2022-03-114-59/+71
|
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-1/+0
|
* Do not return allowed_room_ids from /hierarchy response. (#12175)Patrick Cloke2022-03-081-0/+3
| | | | | This field is only to be used in the Server-Server API, and not the Client-Server API, but was being leaked when a federation response was used in the /hierarchy API.
* Rename various ApplicationServices interested methods (#11915)Andrew Morgan2022-03-031-14/+42
|
* Make get_room_version use cached get_room_version_id. (#11808)lukasdenk2022-03-021-1/+4
|
* Remove the unstable `/spaces` endpoint. (#12073)Patrick Cloke2022-02-281-117/+2
| | | | | | | | ...and various code supporting it. The /spaces endpoint was from an old version of MSC2946 and included both a Client-Server and Server-Server API. Note that the unstable /hierarchy endpoint (from the final version of MSC2946) is not yet removed.
* Replace assertEquals and friends with non-deprecated versions. (#12092)Patrick Cloke2022-02-289-100/+100
|
* Add support for MSC3202: sending one-time key counts and fallback key usage ↵reivilibre2022-02-241-4/+190
| | | | | states to Application Services. (#11617) Co-authored-by: Erik Johnston <erik@matrix.org>
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-2318-37/+39
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Allow modules to set a display name on registration (#12009)Brendan Abolivier2022-02-171-30/+93
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Adds misc missing type hints (#11953)Patrick Cloke2022-02-111-7/+2
|
* Add a callback to allow modules to deny 3PID (#11854)Brendan Abolivier2022-02-081-1/+75
| | | | | Part of the Tchap Synapse mainlining. This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
* Add a background database update to purge account data for deactivated ↵reivilibre2022-02-021-0/+106
| | | | users. (#11655)
* Expose the registered device ID from the `register_appservice_user` test ↵reivilibre2022-02-021-2/+4
| | | | helper. (#11615)
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-11/+270
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Configurable limits on avatars (#11846)Brendan Abolivier2022-01-281-2/+92
| | | | | | Only allow files which file size and content types match configured limits to be set as avatar. Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
* Add a module callback to set username at registration (#11790)Brendan Abolivier2022-01-261-2/+77
| | | | | | This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step). This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not. Co-authored-by: David Robertson <davidr@element.io>
* Remove account data (including client config, push rules and ignored users) ↵reivilibre2022-01-241-0/+219
| | | | | upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Correctly await on_logged_out callbacks (#11786)Brendan Abolivier2022-01-201-1/+27
|
* Fix a bug that corrupted the cache of federated space hierarchies (#11775)Sean Quah2022-01-201-2/+90
| | | | `FederationClient.get_room_hierarchy()` caches its return values, so refactor the code to avoid modifying the returned room summary.
* Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)Patrick Cloke2022-01-071-0/+32
| | | | | | | By returning all of the m.space.child state of the space, not just the first 50. The number of rooms returned is still capped at 50. For the federation API this implies that the requesting server will need to individually query for any other rooms it is not joined to.
* Test that bans win a join against a race when computing `/sync` response ↵David Robertson2022-01-071-3/+94
| | | | (#11701)
* Cache empty responses from `/user/devices` (#11587)David Robertson2022-01-051-0/+96
| | | If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
* Refactor the way we set `outlier` (#11634)Richard van der Hoff2022-01-051-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * `_auth_and_persist_outliers`: mark persisted events as outliers Mark any events that get persisted via `_auth_and_persist_outliers` as, well, outliers. Currently this will be a no-op as everything will already be flagged as an outlier, but I'm going to change that. * `process_remote_join`: stop flagging as outlier The events are now flagged as outliers later on, by `_auth_and_persist_outliers`. * `send_join`: remove `outlier=True` The events created here are returned in the result of `send_join` to `FederationHandler.do_invite_join`. From there they are passed into `FederationEventHandler.process_remote_join`, which passes them to `_auth_and_persist_outliers`... which sets the `outlier` flag. * `get_event_auth`: remove `outlier=True` stop flagging the events returned by `get_event_auth` as outliers. This method is only called by `_get_remote_auth_chain_for_event`, which passes the results into `_auth_and_persist_outliers`, which will flag them as outliers. * `_get_remote_auth_chain_for_event`: remove `outlier=True` we pass all the events into `_auth_and_persist_outliers`, which will now flag the events as outliers. * `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter This param is now never set to True, so we can remove it. * `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param This is no longer set anywhere, so we can remove it. * `get_pdu`: remove unused `outlier` parameter ... and chase it down into `get_pdu_from_destination_raw`. * `event_from_pdu_json`: remove redundant `outlier` param This is never set to `True`, so can be removed. * changelog * update docstring
* Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-101-0/+103
| | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi2021-12-091-5/+25
|
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-064-33/+188
| | | As a step towards allowing back-channel logout for OIDC.
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-29/+65
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Fix checking whether a room can be published on creation. (#11392)Patrick Cloke2021-11-191-33/+62
| | | | | | | | If `room_list_publication_rules` was configured with a rule with a non-wildcard alias and a room was created with an alias then an internal server error would have been thrown. This fixes the error and properly applies the publication rules during room creation.
* Keep fallback key marked as used if it's re-uploaded (#11382)Hubert Chathi2021-11-191-1/+31
|
* Rename `get_access_token_for_user_id` method to ↵reivilibre2021-11-171-5/+5
| | | | `create_access_token_for_user_id` (#11369)
* Avoid sharing room hierarchy responses between users (#11355)Sean Quah2021-11-161-0/+55
| | | | | Different users may be allowed to see different rooms within a space, so sharing responses between users is inadvisable.
* Misc typing fixes for tests, part 2 of N (#11330)David Robertson2021-11-161-3/+6
|
* Add support for the stable version of MSC2778 (#11335)Tulir Asokan2021-11-151-1/+4
| | | | | | | | | * Add support for the stable version of MSC2778 Signed-off-by: Tulir Asokan <tulir@maunium.net> * Expect m.login.application_service in login and password provider tests Signed-off-by: Tulir Asokan <tulir@maunium.net>
* Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-091-2/+6
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-2/+3
| | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Include cross-signing signatures when syncing remote devices for the first ↵Erik Johnston2021-11-091-0/+151
| | | | | | | time (#11234) When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results. c.f. #11159
* Add a linearizer on (appservice, stream) when handling ephemeral events. ↵Nick Barrett2021-11-031-0/+51
| | | | | (#11207) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-281-2/+33
| | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-271-0/+31
| | | Fixes: #9346
* Don't set new room alias before potential 403 (#10930)AndrewFerr2021-10-251-1/+101
| | | | | | Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Fix adding excluded users to the private room sharing tables when joining a ↵David Robertson2021-10-211-14/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | room (#11143) * We only need to fetch users in private rooms * Filter out `user_id` at the top * Discard excluded users in the top loop We weren't doing this in the "First, if they're our user" branch so this is a bugfix. * The caller must check that `user_id` is included This is in the docstring. There are two call sites: - one in `_handle_room_publicity_change`, which explicitly checks before calling; - and another in `_handle_room_membership_event`, which returns early if the user is excluded. So this change is safe. * Test joining a private room with an excluded user * Tweak an existing test * Changelog * test docstring * lint
* Don't remove local users from dir when the leave their last room (#11103)David Robertson2021-10-181-0/+50
|
* Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-151-35/+107
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix 500 error on `/messages` when we accumulate more than 5 backward ↵Eric Eastwood2021-10-141-0/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | extremities (#11027) Found while working on the Gitter backfill script and noticed it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390 When there are more than 5 backward extremities for a given depth, backfill will throw an error because we sliced the extremity list to 5 but then try to iterate over the full list. This causes us to look for state that we never fetched and we get a `KeyError`. Before when calling `/messages` when there are more than 5 backward extremities: ``` Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper callback_return = await self._async_render(request) File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render callback_return = await raw_callback_return File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET msgs = await self.pagination_handler.get_messages( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages await self.hs.get_federation_handler().maybe_backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill return await self._maybe_backfill_inner(room_id, current_depth, limit) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner likely_extremeties_domains = get_domains_from_state(states[e_id]) KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl' ```
* Merge remote-tracking branch 'origin/release-v1.45' into developDavid Robertson2021-10-131-7/+58
|\
| * Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-131-7/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-131-26/+197
|/ | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Fix overwriting profile when making room public (#11003)David Robertson2021-10-081-2/+69
| | | This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
* Don't alter directory entries for local users when setting a per-room ↵David Robertson2021-10-071-0/+34
| | | | | nickname (#11002) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-062-18/+42
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Consistently exclude from user_directory (#10960)David Robertson2021-10-041-12/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-041-2/+2
|
* Clean-up registration tests (#10945)Patrick Cloke2021-09-301-37/+52
| | | | Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent to properly configure auto-join rooms.
* Refactor user directory tests (#10935)David Robertson2021-09-301-194/+89
| | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-7/+7
|
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-243-8/+10
|
* Factor out common code for persisting fetched auth events (#10896)Richard van der Hoff2021-09-241-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Factor more stuff out of `_get_events_and_persist` It turns out that the event-sorting algorithm in `_get_events_and_persist` is also useful in other circumstances. Here we move the current `_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`, and then factor the sorting part out to `_auth_and_persist_fetched_events`. * `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment `get_event_auth` returns events with the outlier flag already set, so this is redundant (though we need to update a test where `get_event_auth` is mocked). * `_get_remote_auth_chain_for_event`: move existing-event tests earlier Move a couple of tests outside the loop. This is a bit inefficient for now, but a future commit will make it better. It should be functionally identical. * `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events` We can use the same codepath for persisting the events fetched as part of an auth chain as for those fetched individually by `_get_events_and_persist` for building the state at a backwards extremity. * `_get_remote_auth_chain_for_event`: use a dict for efficiency `_auth_and_persist_fetched_events` sorts the events itself, so we no longer need to care about maintaining the ordering from `get_event_auth` (and no longer need to sort by depth in `get_event_auth`). That means that we can use a map, making it easier to filter out events we already have, etc. * changelog * `_auth_and_persist_fetched_events`: improve docstring
* Improve typing in user_directory files (#10891)David Robertson2021-09-241-2/+3
| | | | | | | | | | | * Improve typing in user_directory files This makes the user_directory.py in storage pass most of mypy's checks (including `no-untyped-defs`). Unfortunately that file is in the tangled web of Store class inheritance so doesn't pass mypy at the moment. The handlers directory has already been mypyed. Co-authored-by: reivilibre <olivier@librepush.net>
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+1
|
* Fix reactivated users not being added to the user directory (#10782)David Robertson2021-09-231-1/+41
| | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add type hints for event streams. (#10856)Patrick Cloke2021-09-212-9/+39
|
* Always add local users to the user directory (#10796)David Robertson2021-09-211-2/+5
| | | | | | | | | | | | | | | | | | | | | | | It's a simplification, but one that'll help make the user directory logic easier to follow with the other changes upcoming. It's not strictly required for those changes, but this will help simplify the resulting logic that listens for `m.room.member` events and generally make the logic easier to follow. This means the config option `search_all_users` ends up controlling the search query only, and not the data we store. The cost of doing so is an extra row in the `user_directory` and `user_directory_search` tables for each local user which - belongs to no public rooms - belongs to no private rooms of size ≥ 2 I think the cost of this will be marginal (since they'll already have entries in `users` and `profiles` anyway). As a small upside, a homeserver whose directory was built with this change can toggle `search_all_users` without having to rebuild their directory. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Easy refactors of the user directory (#10789)David Robertson2021-09-101-3/+3
| | | No functional changes here. This came out as I was working to tackle #5677
* Pull out encrypted_by_default tests from user_directory tests (#10752)David Robertson2021-09-062-95/+109
|
* Ignore rooms with unknown room versions in the spaces summary. (#10727)Patrick Cloke2021-09-011-0/+25
| | | | This avoids breaking the entire endpoint if a room with an unsupported room version is encountered.
* Consider the `origin_server_ts` of the `m.space.child` event when ordering ↵Patrick Cloke2021-09-011-5/+13
| | | | | | | | | rooms. (#10730) This updates the ordering of the returned events from the spaces summary API to that defined in MSC2946 (which updates MSC1772). Previously a step was skipped causing ordering to be inconsistent with clients.
* Merge remote-tracking branch 'origin/release-v1.41' into developRichard van der Hoff2021-08-271-0/+112
|\
| * Fix incompatibility with Twisted < 21. (#10713)Richard van der Hoff2021-08-271-0/+112
| | | | | | | | | | | | | | Turns out that the functionality added in #10546 to skip TLS was incompatible with older Twisted versions, so we need to be a bit more inventive. Also, add a test to (hopefully) not break this in future. Sadly, testing TLS is really hard.
* | Split `FederationHandler` in half (#10692)Richard van der Hoff2021-08-262-6/+14
| | | | | | The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
* | Do not include rooms with an unknown room version in a sync response. (#10644)Patrick Cloke2021-08-191-6/+131
|/ | | | A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-179-13/+9
|
* Experimental support for MSC3266 Room Summary API. (#10394)Michael Telatynski2021-08-161-15/+93
|
* Support federation in the new spaces summary API (MSC2946). (#10569)Patrick Cloke2021-08-161-123/+169
|
* Handle string read receipt data (#10606)Šimon Brandner2021-08-161-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Test that we handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add changelog for #10606 Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add docs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Ignore malformed RRs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Only surround hidden = ... Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Remove unnecessary argument Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Update changelog.d/10606.bugfix Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Update the pagination parameter name based on MSC2946 review. (#10579)Patrick Cloke2021-08-111-7/+7
|
* Allow requesting the summary of a space which is joinable. (#10580)Patrick Cloke2021-08-111-2/+26
| | | | | | | As opposed to only allowing the summary of spaces which the user is already in or has world-readable visibility. This makes the logic consistent with whether a space/room is returned as part of a space and whether a space summary can start at a space.
* Fix type hints in space summary tests. (#10575)Patrick Cloke2021-08-111-6/+5
| | | And ensure that the file is checked via mypy.
* Add local support for the new spaces summary endpoint (MSC2946) (#10549)Patrick Cloke2021-08-101-107/+279
| | | | | This adds support for the /hierarchy endpoint, which is an update to MSC2946. Currently this only supports rooms known locally to the homeserver.
* Fix an edge-case with invited rooms over federation in the spaces summary. ↵Patrick Cloke2021-08-101-19/+87
| | | | | | | (#10560) If a room which the requesting user was invited to was queried over federation it will now properly appear in the spaces summary (instead of being stripped out by the requesting server).
* Do not remove `status_msg` when user going offline (#10550)Dirk Klimpel2021-08-091-2/+161
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Merge branch 'release-v1.40' into developBrendan Abolivier2021-08-091-6/+6
|\
| * Support MSC3289: Room version 8 (#10449)Patrick Cloke2021-08-091-6/+6
| | | | | | This adds support for MSC3289: room version 8. This is room version 7 + MSC3083.
* | Refactoring before implementing the updated spaces summary. (#10527)Patrick Cloke2021-08-051-78/+107
| | | | | | | | | | This should have no user-visible changes, but refactors some pieces of the SpaceSummaryHandler before adding support for the updated MSC2946.
* | Only return an appservice protocol if it has a service providing it. (#10532)Will Hunt2021-08-051-1/+121
|/ | | | | | If there are no services providing a protocol, omit it completely instead of returning an empty dictionary. This fixes a long-standing spec compliance bug.
* Fix backfilled events being rejected for no `state_groups` (#10439)Eric Eastwood2021-07-291-0/+131
| | | | | | | | | | | | | Reproducible on a federated homeserver when there is a membership auth event as a floating outlier. Then when we try to backfill one of that persons messages, it has missing membership auth to fetch which caused us to mistakenly replace the `context` for the message with that of the floating membership `outlier` event. Since `outliers` have no `state` or `state_group`, the error bubbles up when we continue down the persisting route: `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` Call stack: ``` backfill _auth_and_persist_event _check_event_auth _update_auth_events_and_context_for_auth ```
* Merge tag 'v1.39.0rc3' into developErik Johnston2021-07-281-5/+15
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
| * Always communicate device OTK counts to clients (#10485)Andrew Morgan2021-07-271-5/+15
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Support for MSC2285 (hidden read receipts) (#10413)Šimon Brandner2021-07-281-0/+294
|/ | | Implementation of matrix-org/matrix-doc#2285
* Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke2021-07-131-15/+176
| | | | | | | | | | Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
* [pyupgrade] `tests/` (#10347)Jonathan de Jong2021-07-131-1/+1
|
* Additional unit tests for spaces summary. (#10305)Patrick Cloke2021-07-121-1/+203
|
* Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-081-193/+10
| | | Fixes #9602
* Ignore EDUs for rooms we're not in (#10317)Dagfinn Ilmari Mannsåker2021-07-061-0/+37
|
* Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-011-2/+2
| | | Instead of mixing them with user authentication methods.
* Do not recurse into non-spaces in the spaces summary. (#10256)Patrick Cloke2021-06-291-23/+25
| | | | | Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.
* Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-241-1/+1
| | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-14 13:41:35 +0000