summary refs log tree commit diff
path: root/tests/handlers (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Prefer `make_awaitable` over `defer.succeed` in tests (#12505)Sean Quah2022-04-274-28/+25
| | | | | | | | | | | When configuring the return values of mocks, prefer awaitables from `make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable once, so it is inappropriate for a mock to return the same `Deferred` multiple times. Also update `run_in_background` to support functions that return arbitrary awaitables. Signed-off-by: Sean Quah <seanq@element.io>
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-6/+3
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Bound ephemeral events by key (#12544)Nick Mills-Barrett2022-04-261-0/+82
| | | | Co-authored-by: Brad Murray <bradtgmurray@gmail.com> Co-authored-by: Andrew Morgan <andrewm@element.io>
* Prevent a sync request from removing a user's busy presence status (#12213)David Baker2022-04-131-0/+79
| | | | | | | | | | In trying to use the MSC3026 busy presence status, the user's status would be set back to 'online' next time they synced. This change makes it so that syncing does not affect a user's presence status if it is currently set to 'busy': it must be removed through the presence API. The MSC defers to implementations on the behaviour of busy presence, so this ought to remain compatible with the MSC.
* Add type hints for `tests/unittest.py`. (#12347)Richard van der Hoff2022-04-014-7/+13
| | | In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
* Remove redundant `get_success` calls in test code (#12346)Richard van der Hoff2022-04-013-17/+14
| | | There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
* Raise an exception when getting state at an outlier (#12191)Richard van der Hoff2022-04-011-18/+54
| | | | | | It seems like calling `_get_state_group_for_events` for an event where the state is unknown is an error. Accordingly, let's raise an exception rather than silently returning an empty result.
* Optimise `_get_state_after_missing_prev_event`: use `/state` (#12040)Richard van der Hoff2022-04-011-0/+225
| | | If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-301-1/+120
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Bump `black` and `click` versions (#12320)David Robertson2022-03-291-3/+3
|
* Always allow the empty string as an avatar_url. (#12261)David Robertson2022-03-251-0/+6
| | | | | Hopefully this fixes #12257. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add type hints to tests files. (#12256)Dirk Klimpel2022-03-211-15/+20
|
* Add type hints to more tests files. (#12240)Dirk Klimpel2022-03-173-26/+42
|
* Add type hints to some tests/handlers files. (#12224)Dirk Klimpel2022-03-155-126/+155
|
* Add some type hints to the tests.handlers module. (#12207)Patrick Cloke2022-03-114-59/+71
|
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-1/+0
|
* Do not return allowed_room_ids from /hierarchy response. (#12175)Patrick Cloke2022-03-081-0/+3
| | | | | This field is only to be used in the Server-Server API, and not the Client-Server API, but was being leaked when a federation response was used in the /hierarchy API.
* Rename various ApplicationServices interested methods (#11915)Andrew Morgan2022-03-031-14/+42
|
* Make get_room_version use cached get_room_version_id. (#11808)lukasdenk2022-03-021-1/+4
|
* Remove the unstable `/spaces` endpoint. (#12073)Patrick Cloke2022-02-281-117/+2
| | | | | | | | ...and various code supporting it. The /spaces endpoint was from an old version of MSC2946 and included both a Client-Server and Server-Server API. Note that the unstable /hierarchy endpoint (from the final version of MSC2946) is not yet removed.
* Replace assertEquals and friends with non-deprecated versions. (#12092)Patrick Cloke2022-02-289-100/+100
|
* Add support for MSC3202: sending one-time key counts and fallback key usage ↵reivilibre2022-02-241-4/+190
| | | | | states to Application Services. (#11617) Co-authored-by: Erik Johnston <erik@matrix.org>
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-2318-37/+39
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Allow modules to set a display name on registration (#12009)Brendan Abolivier2022-02-171-30/+93
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Adds misc missing type hints (#11953)Patrick Cloke2022-02-111-7/+2
|
* Add a callback to allow modules to deny 3PID (#11854)Brendan Abolivier2022-02-081-1/+75
| | | | | Part of the Tchap Synapse mainlining. This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
* Add a background database update to purge account data for deactivated ↵reivilibre2022-02-021-0/+106
| | | | users. (#11655)
* Expose the registered device ID from the `register_appservice_user` test ↵reivilibre2022-02-021-2/+4
| | | | helper. (#11615)
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-11/+270
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Configurable limits on avatars (#11846)Brendan Abolivier2022-01-281-2/+92
| | | | | | Only allow files which file size and content types match configured limits to be set as avatar. Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
* Add a module callback to set username at registration (#11790)Brendan Abolivier2022-01-261-2/+77
| | | | | | This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step). This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not. Co-authored-by: David Robertson <davidr@element.io>
* Remove account data (including client config, push rules and ignored users) ↵reivilibre2022-01-241-0/+219
| | | | | upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Correctly await on_logged_out callbacks (#11786)Brendan Abolivier2022-01-201-1/+27
|
* Fix a bug that corrupted the cache of federated space hierarchies (#11775)Sean Quah2022-01-201-2/+90
| | | | `FederationClient.get_room_hierarchy()` caches its return values, so refactor the code to avoid modifying the returned room summary.
* Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)Patrick Cloke2022-01-071-0/+32
| | | | | | | By returning all of the m.space.child state of the space, not just the first 50. The number of rooms returned is still capped at 50. For the federation API this implies that the requesting server will need to individually query for any other rooms it is not joined to.
* Test that bans win a join against a race when computing `/sync` response ↵David Robertson2022-01-071-3/+94
| | | | (#11701)
* Cache empty responses from `/user/devices` (#11587)David Robertson2022-01-051-0/+96
| | | If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
* Refactor the way we set `outlier` (#11634)Richard van der Hoff2022-01-051-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * `_auth_and_persist_outliers`: mark persisted events as outliers Mark any events that get persisted via `_auth_and_persist_outliers` as, well, outliers. Currently this will be a no-op as everything will already be flagged as an outlier, but I'm going to change that. * `process_remote_join`: stop flagging as outlier The events are now flagged as outliers later on, by `_auth_and_persist_outliers`. * `send_join`: remove `outlier=True` The events created here are returned in the result of `send_join` to `FederationHandler.do_invite_join`. From there they are passed into `FederationEventHandler.process_remote_join`, which passes them to `_auth_and_persist_outliers`... which sets the `outlier` flag. * `get_event_auth`: remove `outlier=True` stop flagging the events returned by `get_event_auth` as outliers. This method is only called by `_get_remote_auth_chain_for_event`, which passes the results into `_auth_and_persist_outliers`, which will flag them as outliers. * `_get_remote_auth_chain_for_event`: remove `outlier=True` we pass all the events into `_auth_and_persist_outliers`, which will now flag the events as outliers. * `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter This param is now never set to True, so we can remove it. * `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param This is no longer set anywhere, so we can remove it. * `get_pdu`: remove unused `outlier` parameter ... and chase it down into `get_pdu_from_destination_raw`. * `event_from_pdu_json`: remove redundant `outlier` param This is never set to `True`, so can be removed. * changelog * update docstring
* Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-101-0/+103
| | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi2021-12-091-5/+25
|
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-064-33/+188
| | | As a step towards allowing back-channel logout for OIDC.
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-29/+65
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Fix checking whether a room can be published on creation. (#11392)Patrick Cloke2021-11-191-33/+62
| | | | | | | | If `room_list_publication_rules` was configured with a rule with a non-wildcard alias and a room was created with an alias then an internal server error would have been thrown. This fixes the error and properly applies the publication rules during room creation.
* Keep fallback key marked as used if it's re-uploaded (#11382)Hubert Chathi2021-11-191-1/+31
|
* Rename `get_access_token_for_user_id` method to ↵reivilibre2021-11-171-5/+5
| | | | `create_access_token_for_user_id` (#11369)
* Avoid sharing room hierarchy responses between users (#11355)Sean Quah2021-11-161-0/+55
| | | | | Different users may be allowed to see different rooms within a space, so sharing responses between users is inadvisable.
* Misc typing fixes for tests, part 2 of N (#11330)David Robertson2021-11-161-3/+6
|
* Add support for the stable version of MSC2778 (#11335)Tulir Asokan2021-11-151-1/+4
| | | | | | | | | * Add support for the stable version of MSC2778 Signed-off-by: Tulir Asokan <tulir@maunium.net> * Expect m.login.application_service in login and password provider tests Signed-off-by: Tulir Asokan <tulir@maunium.net>
* Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-091-2/+6
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-2/+3
| | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Include cross-signing signatures when syncing remote devices for the first ↵Erik Johnston2021-11-091-0/+151
| | | | | | | time (#11234) When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results. c.f. #11159
* Add a linearizer on (appservice, stream) when handling ephemeral events. ↵Nick Barrett2021-11-031-0/+51
| | | | | (#11207) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-281-2/+33
| | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-271-0/+31
| | | Fixes: #9346
* Don't set new room alias before potential 403 (#10930)AndrewFerr2021-10-251-1/+101
| | | | | | Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Fix adding excluded users to the private room sharing tables when joining a ↵David Robertson2021-10-211-14/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | room (#11143) * We only need to fetch users in private rooms * Filter out `user_id` at the top * Discard excluded users in the top loop We weren't doing this in the "First, if they're our user" branch so this is a bugfix. * The caller must check that `user_id` is included This is in the docstring. There are two call sites: - one in `_handle_room_publicity_change`, which explicitly checks before calling; - and another in `_handle_room_membership_event`, which returns early if the user is excluded. So this change is safe. * Test joining a private room with an excluded user * Tweak an existing test * Changelog * test docstring * lint
* Don't remove local users from dir when the leave their last room (#11103)David Robertson2021-10-181-0/+50
|
* Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-151-35/+107
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix 500 error on `/messages` when we accumulate more than 5 backward ↵Eric Eastwood2021-10-141-0/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | extremities (#11027) Found while working on the Gitter backfill script and noticed it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390 When there are more than 5 backward extremities for a given depth, backfill will throw an error because we sliced the extremity list to 5 but then try to iterate over the full list. This causes us to look for state that we never fetched and we get a `KeyError`. Before when calling `/messages` when there are more than 5 backward extremities: ``` Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper callback_return = await self._async_render(request) File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render callback_return = await raw_callback_return File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET msgs = await self.pagination_handler.get_messages( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages await self.hs.get_federation_handler().maybe_backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill return await self._maybe_backfill_inner(room_id, current_depth, limit) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner likely_extremeties_domains = get_domains_from_state(states[e_id]) KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl' ```
* Merge remote-tracking branch 'origin/release-v1.45' into developDavid Robertson2021-10-131-7/+58
|\
| * Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-131-7/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-131-26/+197
|/ | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Fix overwriting profile when making room public (#11003)David Robertson2021-10-081-2/+69
| | | This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
* Don't alter directory entries for local users when setting a per-room ↵David Robertson2021-10-071-0/+34
| | | | | nickname (#11002) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-062-18/+42
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Consistently exclude from user_directory (#10960)David Robertson2021-10-041-12/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-041-2/+2
|
* Clean-up registration tests (#10945)Patrick Cloke2021-09-301-37/+52
| | | | Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent to properly configure auto-join rooms.
* Refactor user directory tests (#10935)David Robertson2021-09-301-194/+89
| | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-7/+7
|
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-243-8/+10
|
* Factor out common code for persisting fetched auth events (#10896)Richard van der Hoff2021-09-241-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Factor more stuff out of `_get_events_and_persist` It turns out that the event-sorting algorithm in `_get_events_and_persist` is also useful in other circumstances. Here we move the current `_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`, and then factor the sorting part out to `_auth_and_persist_fetched_events`. * `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment `get_event_auth` returns events with the outlier flag already set, so this is redundant (though we need to update a test where `get_event_auth` is mocked). * `_get_remote_auth_chain_for_event`: move existing-event tests earlier Move a couple of tests outside the loop. This is a bit inefficient for now, but a future commit will make it better. It should be functionally identical. * `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events` We can use the same codepath for persisting the events fetched as part of an auth chain as for those fetched individually by `_get_events_and_persist` for building the state at a backwards extremity. * `_get_remote_auth_chain_for_event`: use a dict for efficiency `_auth_and_persist_fetched_events` sorts the events itself, so we no longer need to care about maintaining the ordering from `get_event_auth` (and no longer need to sort by depth in `get_event_auth`). That means that we can use a map, making it easier to filter out events we already have, etc. * changelog * `_auth_and_persist_fetched_events`: improve docstring
* Improve typing in user_directory files (#10891)David Robertson2021-09-241-2/+3
| | | | | | | | | | | * Improve typing in user_directory files This makes the user_directory.py in storage pass most of mypy's checks (including `no-untyped-defs`). Unfortunately that file is in the tangled web of Store class inheritance so doesn't pass mypy at the moment. The handlers directory has already been mypyed. Co-authored-by: reivilibre <olivier@librepush.net>
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+1
|
* Fix reactivated users not being added to the user directory (#10782)David Robertson2021-09-231-1/+41
| | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add type hints for event streams. (#10856)Patrick Cloke2021-09-212-9/+39
|
* Always add local users to the user directory (#10796)David Robertson2021-09-211-2/+5
| | | | | | | | | | | | | | | | | | | | | | | It's a simplification, but one that'll help make the user directory logic easier to follow with the other changes upcoming. It's not strictly required for those changes, but this will help simplify the resulting logic that listens for `m.room.member` events and generally make the logic easier to follow. This means the config option `search_all_users` ends up controlling the search query only, and not the data we store. The cost of doing so is an extra row in the `user_directory` and `user_directory_search` tables for each local user which - belongs to no public rooms - belongs to no private rooms of size ≥ 2 I think the cost of this will be marginal (since they'll already have entries in `users` and `profiles` anyway). As a small upside, a homeserver whose directory was built with this change can toggle `search_all_users` without having to rebuild their directory. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Easy refactors of the user directory (#10789)David Robertson2021-09-101-3/+3
| | | No functional changes here. This came out as I was working to tackle #5677
* Pull out encrypted_by_default tests from user_directory tests (#10752)David Robertson2021-09-062-95/+109
|
* Ignore rooms with unknown room versions in the spaces summary. (#10727)Patrick Cloke2021-09-011-0/+25
| | | | This avoids breaking the entire endpoint if a room with an unsupported room version is encountered.
* Consider the `origin_server_ts` of the `m.space.child` event when ordering ↵Patrick Cloke2021-09-011-5/+13
| | | | | | | | | rooms. (#10730) This updates the ordering of the returned events from the spaces summary API to that defined in MSC2946 (which updates MSC1772). Previously a step was skipped causing ordering to be inconsistent with clients.
* Merge remote-tracking branch 'origin/release-v1.41' into developRichard van der Hoff2021-08-271-0/+112
|\
| * Fix incompatibility with Twisted < 21. (#10713)Richard van der Hoff2021-08-271-0/+112
| | | | | | | | | | | | | | Turns out that the functionality added in #10546 to skip TLS was incompatible with older Twisted versions, so we need to be a bit more inventive. Also, add a test to (hopefully) not break this in future. Sadly, testing TLS is really hard.
* | Split `FederationHandler` in half (#10692)Richard van der Hoff2021-08-262-6/+14
| | | | | | The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
* | Do not include rooms with an unknown room version in a sync response. (#10644)Patrick Cloke2021-08-191-6/+131
|/ | | | A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-179-13/+9
|
* Experimental support for MSC3266 Room Summary API. (#10394)Michael Telatynski2021-08-161-15/+93
|
* Support federation in the new spaces summary API (MSC2946). (#10569)Patrick Cloke2021-08-161-123/+169
|
* Handle string read receipt data (#10606)Šimon Brandner2021-08-161-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Test that we handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add changelog for #10606 Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add docs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Ignore malformed RRs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Only surround hidden = ... Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Remove unnecessary argument Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Update changelog.d/10606.bugfix Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Update the pagination parameter name based on MSC2946 review. (#10579)Patrick Cloke2021-08-111-7/+7
|
* Allow requesting the summary of a space which is joinable. (#10580)Patrick Cloke2021-08-111-2/+26
| | | | | | | As opposed to only allowing the summary of spaces which the user is already in or has world-readable visibility. This makes the logic consistent with whether a space/room is returned as part of a space and whether a space summary can start at a space.
* Fix type hints in space summary tests. (#10575)Patrick Cloke2021-08-111-6/+5
| | | And ensure that the file is checked via mypy.
* Add local support for the new spaces summary endpoint (MSC2946) (#10549)Patrick Cloke2021-08-101-107/+279
| | | | | This adds support for the /hierarchy endpoint, which is an update to MSC2946. Currently this only supports rooms known locally to the homeserver.
* Fix an edge-case with invited rooms over federation in the spaces summary. ↵Patrick Cloke2021-08-101-19/+87
| | | | | | | (#10560) If a room which the requesting user was invited to was queried over federation it will now properly appear in the spaces summary (instead of being stripped out by the requesting server).
* Do not remove `status_msg` when user going offline (#10550)Dirk Klimpel2021-08-091-2/+161
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Merge branch 'release-v1.40' into developBrendan Abolivier2021-08-091-6/+6
|\
| * Support MSC3289: Room version 8 (#10449)Patrick Cloke2021-08-091-6/+6
| | | | | | This adds support for MSC3289: room version 8. This is room version 7 + MSC3083.
* | Refactoring before implementing the updated spaces summary. (#10527)Patrick Cloke2021-08-051-78/+107
| | | | | | | | | | This should have no user-visible changes, but refactors some pieces of the SpaceSummaryHandler before adding support for the updated MSC2946.
* | Only return an appservice protocol if it has a service providing it. (#10532)Will Hunt2021-08-051-1/+121
|/ | | | | | If there are no services providing a protocol, omit it completely instead of returning an empty dictionary. This fixes a long-standing spec compliance bug.
* Fix backfilled events being rejected for no `state_groups` (#10439)Eric Eastwood2021-07-291-0/+131
| | | | | | | | | | | | | Reproducible on a federated homeserver when there is a membership auth event as a floating outlier. Then when we try to backfill one of that persons messages, it has missing membership auth to fetch which caused us to mistakenly replace the `context` for the message with that of the floating membership `outlier` event. Since `outliers` have no `state` or `state_group`, the error bubbles up when we continue down the persisting route: `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` Call stack: ``` backfill _auth_and_persist_event _check_event_auth _update_auth_events_and_context_for_auth ```
* Merge tag 'v1.39.0rc3' into developErik Johnston2021-07-281-5/+15
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
| * Always communicate device OTK counts to clients (#10485)Andrew Morgan2021-07-271-5/+15
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Support for MSC2285 (hidden read receipts) (#10413)Šimon Brandner2021-07-281-0/+294
|/ | | Implementation of matrix-org/matrix-doc#2285
* Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke2021-07-131-15/+176
| | | | | | | | | | Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
* [pyupgrade] `tests/` (#10347)Jonathan de Jong2021-07-131-1/+1
|
* Additional unit tests for spaces summary. (#10305)Patrick Cloke2021-07-121-1/+203
|
* Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-081-193/+10
| | | Fixes #9602
* Ignore EDUs for rooms we're not in (#10317)Dagfinn Ilmari Mannsåker2021-07-061-0/+37
|
* Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-011-2/+2
| | | Instead of mixing them with user authentication methods.
* Do not recurse into non-spaces in the spaces summary. (#10256)Patrick Cloke2021-06-291-23/+25
| | | | | Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.
* Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-241-1/+1
| | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
* MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-241-1/+1
| | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* Merge tag 'v1.37.0rc1' into developBrendan Abolivier2021-06-241-0/+76
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))
| * Fix wrapping of legacy check_registration_for_spam (#10238)Brendan Abolivier2021-06-231-0/+76
| | | | | | Fixes #10234
* | Improve the reliability of auto-joining remote rooms (#10237)Brendan Abolivier2021-06-231-1/+48
|/ | | | | | If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do. Fixes #10233 (at least to some extent)
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-221-1/+3
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Fix a missing await when in the spaces summary. (#10208)Patrick Cloke2021-06-181-1/+98
| | | | | | | This could cause a minor data leak if someone defined a non-restricted join rule with an allow key or used a restricted join rule in an older room version, but this is unlikely. Additionally this starts adding unit tests to the spaces summary handler.
* Standardise the module interface (#10062)Brendan Abolivier2021-06-182-48/+93
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* update black to 21.6b0 (#10197)Marcus2021-06-175-5/+5
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* Limit the number of in-flight /keys/query requests from a single device. ↵Patrick Cloke2021-06-091-3/+10
| | | | (#10144)
* Always update AS last_pos, even on no events (#10107)14mRh4X0r2021-06-071-4/+2
| | | | | | | | | | Fixes #1834. `get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether. Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list. Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-7/+1
|
* Add a test for update_presence (#10033)Andrew Morgan2021-05-211-1/+46
| | | | | | | https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale. The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug. The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-122-28/+27
|
* Sort child events according to MSC1772 for the spaces summary API. (#9954)Patrick Cloke2021-05-111-0/+81
| | | | | | | | | This should help ensure that equivalent results are achieved between homeservers querying for the summary of a space. This implements modified MSC1772 rules, according to MSC2946. The different is that the origin_server_ts of the m.room.create event is not used as a tie-breaker since this might not be known if the homeserver is not part of the room.
* Increase perf of handling presence when joining large rooms. (#9916)Erik Johnston2021-05-051-10/+4
|
* Fix tight loop handling presence replication. (#9900)Erik Johnston2021-04-281-0/+22
| | | | | Only affects workers. Introduced in #9819. Fixes #9899.
* Rename handler and config modules which end in handler/config. (#9816)Patrick Cloke2021-04-202-5/+5
|
* Add presence federation stream (#9819)Erik Johnston2021-04-201-8/+171
|
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-1/+1
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1418-18/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-0915-21/+15
|
* Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan2021-04-061-10/+11
| | | | | | | | | | | | At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
* Implement MSC3026: busy presence stateBrendan Abolivier2021-03-181-0/+20
|
* Add SSO attribute requirements for OIDC providers (#9609)Hubbe2021-03-161-0/+132
| | | | Allows limiting who can login using OIDC via the claims made from the IdP.
* Pass SSO IdP information to spam checker's registration function (#9626)Andrew Morgan2021-03-161-0/+31
| | | | | | | Fixes https://github.com/matrix-org/synapse/issues/9572 When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time. This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
* JWT OIDC secrets for Sign in with Apple (#9549)Richard van der Hoff2021-03-093-24/+166
| | | | | Apple had to be special. They want a client secret which is generated from an EC key. Fixes #9220. Also fixes #9212 while I'm here.
* Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff2021-03-044-50/+55
| | | This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
* Fix style checking due to updated black.Patrick Cloke2021-02-191-1/+4
|
* Be smarter about which hosts to send presence to when processing room joins ↵Andrew Morgan2021-02-191-4/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | (#9402) This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually. --- When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed. It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw. This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence: * If it was a local user join, send that user's latest presence to all servers in the room * If it was a remote user join, send the presence for all local users in the room to that homeserver We deduplicate by inserting all of those pending updates into a dictionary of the form: ``` { server_name1: {presence_update1, ...}, server_name2: {presence_update1, presence_update2, ...} } ``` Only after building this dict do we then start sending out presence updates.
* Add a config option to prioritise local users in user directory search ↵Andrew Morgan2021-02-191-0/+94
| | | | | | | | | results (#9383) This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default. Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical. This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
* Add configs to make profile data more private (#9203)AndrewFerr2021-02-191-1/+5
| | | | | | | Add off-by-default configuration settings to: - disable putting an invitee's profile info in invite events - disable profile lookup via federation Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Support for form_post in OIDC responses (#9376)Richard van der Hoff2021-02-171-13/+13
| | | Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1613-77/+101
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-161-26/+45
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Convert additional test-cases to homeserver test case. (#9396)Patrick Cloke2021-02-161-17/+16
| | | And convert some inlineDeferreds to async-friendly functions.
* Convert some test cases to use HomeserverTestCase. (#9377)Patrick Cloke2021-02-114-488/+301
| | | | This has the side-effect of being able to remove use of `inlineCallbacks` in the test-cases for cleaner tracebacks.
* Combine the CAS & SAML implementations for required attributes. (#9326)Patrick Cloke2021-02-112-3/+105
|
* Honour ratelimit flag for application services for invite ratelimiting (#9302)Erik Johnston2021-02-031-47/+0
|
* Put OIDC callback URI under /_synapse/client. (#9288)Richard van der Hoff2021-02-011-9/+6
|
* Merge branch 'social_login' into developRichard van der Hoff2021-02-013-18/+22
|\
| * Improve styling and wording of SSO redirect confirm template (#9272)Richard van der Hoff2021-02-013-18/+22
| |
* | Ratelimit invites by room and target user (#9258)Erik Johnston2021-01-291-1/+92
|/
* Land support for multiple OIDC providers (#9110)Richard van der Hoff2021-01-151-2/+2
| | | | | | | | | | | | | | | | | | | | | | | This is the final step for supporting multiple OIDC providers concurrently. First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before: oidc_config: enabled: true issuer: "https://oidc_provider" # etc After: oidc_providers: - idp_id: prov1 issuer: "https://oidc_provider" - idp_id: prov2 issuer: "https://another_oidc_provider" The old format is still grandfathered in. With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
* Improve UsernamePickerTestCase (#9112)Richard van der Hoff2021-01-151-118/+2
| | | | | * make the OIDC bits of the test work at a higher level - via the REST api instead of poking the OIDCHandler directly. * Move it to test_login.py, where I think it fits better.
* Store an IdP ID in the OIDC session (#9109)Richard van der Hoff2021-01-151-1/+2
| | | | | Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP. As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
* Split OidcProvider out of OidcHandler (#9107)Richard van der Hoff2021-01-141-45/+48
| | | | | | | The idea here is that we will have an instance of OidcProvider for each configured IdP, with OidcHandler just doing the marshalling of them. For now it's still hardcoded with a single provider.
* Preparatory refactors of OidcHandler (#9067)Richard van der Hoff2021-01-131-24/+37
| | | | | | | | Some light refactoring of OidcHandler, in preparation for bigger things: * remove inheritance from deprecated BaseHandler * add an object to hold the things that go into a session cookie * factor out a separate class for manipulating said cookies
* Remove user's avatar URL and displayname when deactivated. (#8932)Dirk Klimpel2021-01-121-0/+30
| | | This only applies if the user's data is to be erased.
* Remove SynapseRequest.get_user_agent (#9069)Richard van der Hoff2021-01-123-4/+3
| | | | | | | | | | | SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests", which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the LoggingContext and write the right entries to the request log). Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a SynapseRequest when there is nothing synapse-specific about the Request at all, and any old twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult. In short: move get_user_agent out to a utility function.
* Skip unit tests which require optional dependencies (#9031)Richard van der Hoff2021-01-071-1/+18
| | | If we are lacking an optional dependency, skip the tests that rely on it.
* Use the SSO handler helpers for CAS registration/login. (#8856)Patrick Cloke2021-01-031-0/+121
|
* Implement a username picker for synapse (#8942)Richard van der Hoff2020-12-181-1/+142
| | | | | | | | | | | | | | The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow `UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser back to the client, we redirect to a username-picker resource, which allows the user to enter a username. We *then* complete the SSO flow (including doing the client permission checks). The static resources for the username picker itself (in https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker) are essentially lifted wholesale from https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. As the comment says, we might want to think about making them customisable, but that can be a follow-up. Fixes #8876.
* Fix a bug that deactivated users appear in the directory (#8933)Dirk Klimpel2020-12-171-1/+39
| | | | | | | | | | Fixes a bug that deactivated users appear in the directory when their profile information was updated. To change profile information of deactivated users is neccesary for example you will remove displayname or avatar. But they should not appear in directory. They are deactivated. Co-authored-by: Erik Johnston <erikj@jki.re>
* Push login completion down into SsoHandler (#8941)Richard van der Hoff2020-12-161-4/+4
| | | This is another part of my work towards fixing #8876. It moves some of the logic currently in the SAML and OIDC handlers - in particular the call to `AuthHandler.complete_sso_login` down into the `SsoHandler`.
* Merge pull request #8951 from matrix-org/rav/username_picker_2Richard van der Hoff2020-12-161-81/+104
|\ | | | | More preparatory refactoring of the OidcHandler tests
| * Make `_make_callback_with_userinfo` asyncRichard van der Hoff2020-12-151-68/+83
| | | | | | | | | | | | ... so that we can test its behaviour when it raises. Also pull it out to the top level so that I can use it from other test classes.
| * Remove spurious mocking of complete_sso_loginRichard van der Hoff2020-12-151-2/+0
| | | | | | | | The tests that need this all do it already.
| * Test `get_extra_attributes` fallbackRichard van der Hoff2020-12-151-11/+21
| | | | | | | | | | despite the warnings saying "don't implement get_extra_attributes", we had implemented it, so the tests weren't doing what we thought they were.
* | Remove spurious "SynapseRequest" result from `make_request"Richard van der Hoff2020-12-155-13/+13
|/ | | | This was never used, so let's get rid of it.
* Preparatory refactoring of the SamlHandlerTestCase (#8938)Richard van der Hoff2020-12-152-54/+90
| | | | | | | | | | | | | * move simple_async_mock to test_utils ... so that it can be re-used * Remove references to `SamlHandler._map_saml_response_to_user` from tests This method is going away, so we can no longer use it as a test point. Instead, factor out a higher-level method which takes a SAML object, and verify correct behaviour by mocking out `AuthHandler.complete_sso_login`. * changelog
* Fix startup failure with localdb_enabled: False (#8937)Richard van der Hoff2020-12-141-0/+23
|
* Various clean-ups to the logging context code (#8935)Patrick Cloke2020-12-141-3/+3
|
* Preparatory refactoring of the OidcHandlerTestCase (#8911)Richard van der Hoff2020-12-141-141/+145
| | | | | | | | | | | | | | | | | | | | | | | * Remove references to handler._auth_handler (and replace them with hs.get_auth_handler) * Factor out a utility function for building Requests * Remove mocks of `OidcHandler._map_userinfo_to_user` This method is going away, so mocking it out is no longer a valid approach. Instead, we mock out lower-level methods (eg _remote_id_from_userinfo), or simply allow the regular implementation to proceed and update the expectations accordingly. * Remove references to `OidcHandler._map_userinfo_to_user` from tests This method is going away, so we can no longer use it as a test point. Instead we build mock "callback" requests which we pass into `handle_oidc_callback`, and verify correct behaviour by mocking out `AuthHandler.complete_sso_login`.
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-111-2/+2
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Skip the SAML tests if xmlsec1 isn't available. (#8905)Patrick Cloke2020-12-091-0/+23
|
* Merge tag 'v1.24.0rc2' into developPatrick Cloke2020-12-042-2/+29
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.24.0rc2 (2020-12-04) ============================== Bugfixes -------- - Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878)) Internal Changes ---------------- - Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
| * Fix a regression that mapping providers should be able to redirect users. ↵Patrick Cloke2020-12-042-2/+29
| | | | | | | | | | (#8878) This was broken in #8801.
* | Merge remote-tracking branch 'origin/develop' into rav/remove_unused_mocksRichard van der Hoff2020-12-022-31/+9
|\ \
| * \ Merge pull request #8858 from matrix-org/rav/sso_uiaRichard van der Hoff2020-12-022-19/+9
| |\ \ | | | | | | | | UIA: offer only available auth flows
| | * | Factor out FakeResponse from test_oidcRichard van der Hoff2020-12-021-16/+1
| | | |
| | * | fix up various test casesRichard van der Hoff2020-12-021-3/+8
| | |/ | | | | | | | | | | | | A few test cases were relying on being able to mount non-client servlets on the test resource. it's better to give them their own Resources.
* | | remove unused `resource_for_federation`Richard van der Hoff2020-12-021-1/+13
| | | | | | | | | | | | This is now only used in `test_typing`, so move it there.
* | | Remove redundant mockingRichard van der Hoff2020-12-022-4/+0
|/ /
* / Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-026-9/+9
|/ | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Fix a regression when grandfathering SAML users. (#8855)Patrick Cloke2020-12-022-1/+41
| | | | | | This was broken in #8801 when abstracting code shared with OIDC. After this change both SAML and OIDC have a concept of grandfathering users, but with different implementations.
* Add basic SAML tests for mapping users. (#8800)Patrick Cloke2020-12-022-17/+153
|
* Create a `PasswordProvider` wrapper object (#8849)Richard van der Hoff2020-12-021-2/+3
| | | | The idea here is to abstract out all the conditional code which tests which methods a given password provider has, to provide a consistent interface.
* Support "identifier" dicts in UIA (#8848)Richard van der Hoff2020-12-011-10/+1
| | | | | | | | | | The spec requires synapse to support `identifier` dicts for `m.login.password` user-interactive auth, which it did not (instead, it required an undocumented `user` parameter.) To fix this properly, we need to pull the code that interprets `identifier` into `AuthHandler.validate_login` so that it can be called from the UIA code. Fixes #5665.
* Don't offer password login when it is disabled (#8835)Richard van der Hoff2020-12-011-3/+105
| | | Fix a minor bug where we would offer "m.login.password" login if a custom auth provider supported it, even if password login was disabled.
* Add some tests for `password_auth_providers` (#8819)Richard van der Hoff2020-12-011-0/+486
| | | | These things seemed to be completely untested, so I added a load of tests for them.
* Simplify the way the `HomeServer` object caches its internal attributes. ↵Jonathan de Jong2020-11-301-3/+3
| | | | | (#8565) Changes `@cache_in_self` to use underscore-prefixed attributes.
* Support trying multiple localparts for OpenID Connect. (#8801)Patrick Cloke2020-11-251-1/+87
| | | | Abstracts the SAML and OpenID Connect code which attempts to regenerate the localpart of a matrix ID if it is already in use.
* Improve error checking for OIDC/SAML mapping providers (#8774)Patrick Cloke2020-11-191-20/+69
| | | | | | Checks that the localpart returned by mapping providers for SAML and OIDC are valid before registering new users. Extends the OIDC tests for existing users and invalid data.
* Consistently use room_id from federation request body (#8776)Richard van der Hoff2020-11-191-1/+0
| | | | | | | | | | | | | * Consistently use room_id from federation request body Some federation APIs have a redundant `room_id` path param (see https://github.com/matrix-org/matrix-doc/issues/2330). We should make sure we consistently use either the path param or the body param, and the body param is easier. * Kill off some references to "context" Once upon a time, "rooms" were known as "contexts". I think this kills of the last references to "contexts".
* Merge pull request #8761 from matrix-org/rav/test_request_renderingRichard van der Hoff2020-11-174-10/+0
|\ | | | | Make `make_request` actually render the request
| * Remove redundant `HomeserverTestCase.render`Richard van der Hoff2020-11-164-10/+0
| |
* | Abstract shared SSO code. (#8765)Patrick Cloke2020-11-171-7/+7
| | | | | | De-duplicates code between the SAML and OIDC implementations.
* | Add admin API for logging in as a user (#8617)Erik Johnston2020-11-171-4/+10
|/
* Block clients from sending server ACLs that lock the local server out. (#8708)Erik Johnston2020-11-031-0/+57
| | | Fixes #4042
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-292-2/+2
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Start fewer opentracing spans (#8640)Erik Johnston2020-10-261-12/+8
| | | | | | | #8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously. This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary. I don't really know how much this will help.
* Fix handling of User-Agent headers with bad utf-8. (#8632)Erik Johnston2020-10-231-6/+18
|
* Merge pull request #8537 from matrix-org/rav/simplify_locally_reject_inviteRichard van der Hoff2020-10-152-2/+1
|\ | | | | Simplify `_locally_reject_invite`
| * Simplify `_locally_reject_invite`Richard van der Hoff2020-10-131-1/+1
| | | | | | | | | | Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.
| * Remove redundant `token_id` parameter to create_eventRichard van der Hoff2020-10-131-1/+0
| | | | | | | | this is always the same as requester.access_token_id.
* | Fix not sending events over federation when using sharded event persisters ↵Erik Johnston2020-10-141-3/+10
|/ | | | | | | | | | | | | | | | | (#8536) * Fix outbound federaion with multiple event persisters. We incorrectly notified federation senders that the minimum persisted stream position had advanced when we got an `RDATA` from an event persister. Notifying of federation senders already correctly happens in the notifier, so we just delete the offending line. * Change some interfaces to use RoomStreamToken. By enforcing use of `RoomStreamTokens` we make it less likely that people pass in random ints that they got from somewhere random.
* Move additional tasks to the background worker, part 4 (#8513)Patrick Cloke2020-10-131-34/+14
|
* Fix message duplication if something goes wrong after persisting the event ↵Erik Johnston2020-10-131-0/+157
| | | | | (#8476) Should fix #3365.
* Remove the deprecated Handlers object (#8494)Patrick Cloke2020-10-099-40/+20
| | | All handlers now available via get_*_handler() methods on the HomeServer.
* Invalidate the cache when an olm fallback key is uploaded (#8501)Hubert Chathi2020-10-081-0/+20
|
* Add support for MSC2697: Dehydrated devices (#8380)Hubert Chathi2020-10-071-0/+82
| | | | This allows a user to store an offline device on the server and then restore it at a subsequent login.
* Merge pull request #8463 from matrix-org/rav/clean_up_event_handlingRichard van der Hoff2020-10-071-1/+1
|\ | | | | Reduce inconsistencies between codepaths for membership and non-membership events.
| * kill off `send_nonmember_event`Richard van der Hoff2020-10-051-1/+1
| | | | | | | | This is now redundant, and we can just call `handle_new_client_event` directly.
* | Add support for MSC2732: olm fallback keys (#8312)Hubert Chathi2020-10-061-0/+65
|/
* Add config option for always using "userinfo endpoint" for OIDC (#7658)BBBSnowball2020-10-011-2/+8
| | | This allows for connecting to certain IdPs, e.g. GitLab.
* Allow additional SSO properties to be passed to the client (#8413)Patrick Cloke2020-09-301-55/+105
|
* Allow existing users to login via OpenID Connect. (#8345)Tdxdxoz2020-09-251-0/+35
| | | | | | | Co-authored-by: Benjamin Koch <bbbsnowball@gmail.com> This adds configuration flags that will match a user to pre-existing users when logging in via OpenID Connect. This is useful when switching to an existing SSO system.
* Create function to check for long names in devices (#8364)Dionysis Grigoropoulos2020-09-221-0/+11
| | | | | | | | | * Create a new function to verify that the length of a device name is under a certain threshold. * Refactor old code and tests to use said function. * Verify device name length during registration of device * Add a test for the above Signed-off-by: Dionysis Grigoropoulos <dgrig@erethon.com>
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-2/+2
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Catch-up after Federation Outage (split, 4): catch-up loop (#8272)reivilibre2020-09-151-0/+5
|
* Fixup pusher pool notifications (#8287)Erik Johnston2020-09-091-0/+1
| | | | | `pusher_pool.on_new_notifications` expected a min and max stream ID, however that was not what we were passing in. Instead, let's just pass it the current max stream ID and have it track the last stream ID it got passed. I believe that it mostly worked as we called the function for every event. However, it would break for events that got persisted out of order, i.e, that were persisted but the max stream ID wasn't incremented as not all preceding events had finished persisting, and push for that event would be delayed until another event got pushed to the effected users.
* Revert "Fixup pusher pool notifications"Erik Johnston2020-09-091-1/+0
| | | | This reverts commit e7fd336a53a4ca489cdafc389b494d5477019dc0.
* Fixup pusher pool notificationsErik Johnston2020-09-091-0/+1
|
* Allow for make_awaitable's return value to be re-used. (#8261)Patrick Cloke2020-09-083-16/+16
|
* Rename 'populate_stats_process_rooms_2' background job back to ↵Andrew Morgan2020-09-081-9/+6
| | | | | | | 'populate_stats_process_rooms' again (#8243) Fixes https://github.com/matrix-org/synapse/issues/8238 Alongside the delta file, some changes were also necessary to the codebase to remove references to the now defunct `populate_stats_process_rooms_2` background job. Thankfully the latter doesn't seem to have made it into any documentation yet :)
* Stop sub-classing object (#8249)Patrick Cloke2020-09-042-2/+2
|
* Ensure that the OpenID Connect remote ID is a string. (#8190)Patrick Cloke2020-08-281-2/+39
|
* Do not yield on awaitables in tests. (#8193)Patrick Cloke2020-08-271-1/+3
|
* Convert stats and related calls to async/await (#8192)Patrick Cloke2020-08-272-17/+16
|
* Convert simple_update* and simple_select* to async (#8173)Patrick Cloke2020-08-271-2/+2
|
* Reduce run-times of tests by advancing the reactor less (#7757)Andrew Morgan2020-08-271-1/+1
|
* Do not propagate typing notifications from shadow-banned users. (#8176)Patrick Cloke2020-08-261-6/+20
|
* Convert simple_select_one and simple_select_one_onecol to async (#8162)Patrick Cloke2020-08-262-14/+46
|
* Allow denying or shadow banning registrations via the spam checker (#8034)Patrick Cloke2020-08-203-7/+69
|
* Convert some of the general database methods to async (#8100)Patrick Cloke2020-08-172-3/+3
|
* Convert stream database to async/await. (#8074)Patrick Cloke2020-08-171-1/+1
|
* Convert devices database to async/await. (#8069)Patrick Cloke2020-08-121-1/+1
|
* Convert account data, device inbox, and censor events databases to ↵Patrick Cloke2020-08-121-1/+2
| | | | async/await (#8063)
* Converts event_federation and registration databases to async/await (#8061)Patrick Cloke2020-08-111-5/+6
|
* Convert directory, e2e_room_keys, end_to_end_keys, monthly_active_users ↵Patrick Cloke2020-08-071-1/+1
| | | | database to async (#8042)
* Convert synapse.api to async/await (#8031)Patrick Cloke2020-08-061-2/+2
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-052-42/+42
|
* Prevent join->join membership transitions changing member count (#7977)Andrew Morgan2020-08-031-6/+40
| | | | | | | | | | | `StatsHandler` handles updates to the `current_state_delta_stream`, and updates room stats such as the amount of state events, joined users, etc. However, it counts every new join membership as a new user entering a room (and that user being in another room), whereas it's possible for a user's membership status to go from join -> join, for instance when they change their per-room profile information. This PR adds a check for join->join membership transitions, and bails out early, as none of the further checks are necessary at that point. Due to this bug, membership stats in many rooms have ended up being wildly larger than their true values. I am not sure if we also want to include a migration step which recalculates these statistics (possibly using the `_populate_stats_process_rooms` bg update). Bug introduced in the initial implementation https://github.com/matrix-org/synapse/pull/4338.
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-302-4/+4
|
* Convert appservice to async. (#7973)Patrick Cloke2020-07-301-2/+3
|
* Convert device handler to async/await (#7871)Patrick Cloke2020-07-172-12/+11
|
* Convert _base, profile, and _receipts handlers to async/await (#7860)Patrick Cloke2020-07-171-6/+11
|
* Fix testsErik Johnston2020-07-151-2/+2
|
* Convert E2E key and room key handlers to async/await. (#7851)Patrick Cloke2020-07-152-237/+422
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-30 03:14:32 +0000