summary refs log tree commit diff
path: root/synapse/handlers (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add a setting to disable TLS for sending email (#10546)Richard van der Hoff2021-08-061-17/+77
| | | This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
* Refactoring before implementing the updated spaces summary. (#10527)Patrick Cloke2021-08-051-49/+76
| | | | | This should have no user-visible changes, but refactors some pieces of the SpaceSummaryHandler before adding support for the updated MSC2946.
* Only return an appservice protocol if it has a service providing it. (#10532)Will Hunt2021-08-051-4/+3
| | | | | | If there are no services providing a protocol, omit it completely instead of returning an empty dictionary. This fixes a long-standing spec compliance bug.
* Send unstable-prefixed room_type in store-invite IS API requests (#10435)Michael Telatynski2021-08-042-1/+18
| | | | | | | | The room type is per MSC3288 to allow the identity-server to change invitation wording based on whether the invitation is to a room or a space. The prefixed key will be replaced once MSC3288 is accepted into the spec.
* Add support for MSC2716 marker events (#10498)Eric Eastwood2021-08-041-6/+113
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Add support for MSC2716 marker events * Process markers when we receive it over federation * WIP: make hs2 backfill historical messages after marker event * hs2 to better ask for insertion event extremity But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error * Add insertion_event_extremities table * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Messy: Fix undefined state_group for federated historical events ``` 2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill await self.backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill await self._auth_and_persist_event(dest, event, context, backfilled=True) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event await self._run_push_actions_and_persist_event(event, context, backfilled) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event await self.persist_events_and_notify( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify events, max_stream_token = await self.storage.persistence.persist_events( File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner return await func(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events ret_vals = await yieldable_gather_results(enqueue, partitioned.items()) File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop ret = await self._per_item_callback( File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch await self.persist_events_store._persist_events_and_state_updates( File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates await self.db_pool.runInteraction( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction result = await self.runWithConnection( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection return await make_deferred_yieldable( File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext result = inContext.theWork() # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda> inContext.theWork = lambda: context.call( # type: ignore[attr-defined] File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext return func(*args, **kw) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise raise exception.with_traceback(traceback) File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func return func(db_conn, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction r = func(cursor, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn self._store_event_state_mappings_txn(txn, events_and_contexts) File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn self.db_pool.simple_insert_many_txn( File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn txn.execute_batch(sql, vals) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch self.executemany(sql, args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany self._do_execute(self.txn.executemany, sql, *args) File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute return func(sql, *args) sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group ``` * Revert "Messy: Fix undefined state_group for federated historical events" This reverts commit 187ab28611546321e02770944c86f30ee2bc742a. * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Adapting to experimental room version * Some log cleanup * Add better comments around extremity fetching code and why * Rename to be more accurate to what the function returns * Add changelog * Ignore rejected events * Use simplified upsert * Add Erik's explanation of extra event checks See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332 * Clarify that the depth is not directly correlated to the backwards extremity that we return See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404 * lock only matters for sqlite See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061 * Move new SQL changes to its own delta file * Clean up upsert docstring * Bump database schema version (62)
* Make sync response cache time configurable. (#10513)Richard van der Hoff2021-08-031-3/+11
|
* Include room ID in ignored EDU log messages (#10507)Dagfinn Ilmari Mannsåker2021-08-032-2/+4
| | | | Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
* Make historical events discoverable from backfill for servers without any ↵Eric Eastwood2021-07-282-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | scrollback history (MSC2716) (#10245) * Make historical messages available to federated servers Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 Follow-up to https://github.com/matrix-org/synapse/pull/9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080 * Store insertion events in table * Make base insertion event float off on its own See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889 Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455 Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Fix federated events being rejected for no state_groups Add fix from https://github.com/matrix-org/synapse/pull/10439 until it merges. * Only connect base insertion event to prev_event_ids Per discussion with @erikjohnston, https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org * Make it possible to get the room_version with txn * Allow but ignore historical events in unsupported room version See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489 We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either. Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`. * Move to unique index syntax See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509 * High-level document how the insertion->chunk lookup works * Remove create_event fallback for room_versions See https://github.com/matrix-org/synapse/pull/10245/files#r677641879 * Use updated method name
* Merge tag 'v1.39.0rc3' into developErik Johnston2021-07-282-2/+4
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
| * Fix import of the default SAML mapping provider. (#10477)Jason Robinson2021-07-271-2/+0
| | | | | | | | Fix a circular import, which was causing exceptions on boot if SAML was configured.
| * Always communicate device OTK counts to clients (#10485)Andrew Morgan2021-07-271-0/+4
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Support for MSC2285 (hidden read receipts) (#10413)Šimon Brandner2021-07-282-6/+59
| | | | | | Implementation of matrix-org/matrix-doc#2285
* | Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke2021-07-263-22/+292
|/ | | | (#10254)
* Port the ThirdPartyEventRules module interface to the new generic interface ↵Brendan Abolivier2021-07-203-13/+9
| | | | | (#10386) Port the third-party event rules interface to the generic module interface introduced in v1.37.0
* Fix exception when failing to get remote room list (#10414)Erik Johnston2021-07-201-1/+5
|
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-197-33/+32
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-1629-176/+178
|
* Add a module type for account validity (#9884)Brendan Abolivier2021-07-162-2/+131
| | | | | This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity. To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
* Fix a number of logged errors caused by remote servers being down. (#10400)Erik Johnston2021-07-153-19/+37
|
* Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke2021-07-131-20/+48
| | | | | | | | | | Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
* Add base starting insertion event when no chunk ID is provided (MSC2716) ↵Eric Eastwood2021-07-081-0/+8
| | | | | | | | | (#10250) * Add base starting insertion point when no chunk ID is provided This is so we can have the marker event point to this initial insertion event and be able to traverse the events in the first chunk.
* Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-081-27/+0
| | | Fixes #9602
* Rebuild event auth when rebuilding an event after a call to a ↵Brendan Abolivier2021-07-081-4/+6
| | | | | `ThirdPartyEventRules` module (#10316) Because modules might send extra state events when processing an event (e.g. matrix-org/synapse-dinsic#100), and in some cases these extra events might get dropped if we don't recalculate the initial event's auth.
* Ignore EDUs for rooms we're not in (#10317)Dagfinn Ilmari Mannsåker2021-07-062-0/+29
|
* Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-015-21/+95
| | | Instead of mixing them with user authentication methods.
* Add SSO `external_ids` to Query User Account admin API (#10261)Dirk Klimpel2021-07-011-0/+7
| | | Related to #10251
* Correct type hints for synapse.event_auth. (#10253)Patrick Cloke2021-06-301-0/+7
|
* Do not recurse into non-spaces in the spaces summary. (#10256)Patrick Cloke2021-06-291-2/+9
| | | | | Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.
* Return errors from `send_join` etc if the event is rejected (#10243)Richard van der Hoff2021-06-241-7/+39
| | | Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
* Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-241-126/+51
| | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
* MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-242-11/+173
| | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* Improve the reliability of auto-joining remote rooms (#10237)Brendan Abolivier2021-06-231-16/+47
| | | | | | If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do. Fixes #10233 (at least to some extent)
* Send out invite rejections and knocks over federation (#10223)Richard van der Hoff2021-06-231-0/+14
| | | | | ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-222-5/+189
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Implement config option `sso.update_profile_information` (#10108)jkanefendt2021-06-211-1/+24
| | | | | | Implemented config option sso.update_profile_information to keep user's display name in sync with the SSO displayname. Signed-off-by: Johannes Kanefendt <johannes.kanefendt@krzn.de>
* Check third party rules before persisting knocks over federation (#10212)Andrew Morgan2021-06-211-2/+2
| | | | | An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules. This PR corrects that by performing the TPAR check *before* persisting the event.
* Fix a missing await when in the spaces summary. (#10208)Patrick Cloke2021-06-181-2/+1
| | | | | | | This could cause a minor data leak if someone defined a non-restricted join rule with an allow key or used a restricted join rule in an older room version, but this is unlikely. Additionally this starts adding unit tests to the spaces summary handler.
* Standardise the module interface (#10062)Brendan Abolivier2021-06-181-1/+1
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Remove support for ACME v1 (#10194)Brendan Abolivier2021-06-172-244/+0
| | | | | Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
* Update MSC3083 support per changes in the MSC. (#10189)Patrick Cloke2021-06-172-31/+40
| | | Adds a "type" field and generalize "space" to "room_id".
* Ensure that we do not cache empty sync responses after a timeout (#10158)Richard van der Hoff2021-06-171-10/+26
| | | Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
* update black to 21.6b0 (#10197)Marcus2021-06-171-1/+1
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* Add fields to better debug where events are being soft_failed (#10168)Eric Eastwood2021-06-171-3/+18
| | | Follow-up to https://github.com/matrix-org/synapse/pull/10156#discussion_r650292223
* Always require users to re-authenticate for dangerous operations. (#10184)Patrick Cloke2021-06-161-1/+6
| | | | | | | Dangerous actions means deactivating an account, modifying an account password, or adding a 3PID. Other actions (deleting devices, uploading keys) can re-use the same UI auth session if ui_auth.session_timeout is configured.
* Remove support for unstable MSC1772 prefixes. (#10161)Patrick Cloke2021-06-151-13/+3
| | | | The stable prefixes have been supported since v1.34.0. The unstable prefixes are not supported by any known clients.
* Remove the experimental flag for knocking and use stable prefixes / ↵Patrick Cloke2021-06-152-8/+3
| | | | | | | endpoints. (#10167) * Room version 7 for knocking. * Stable prefixes and endpoints (both client and federation) for knocking. * Removes the experimental configuration flag.
* Only send a presence state to a destination once (#10165)Erik Johnston2021-06-111-13/+12
| | | | It turns out that we were sending the same presence state to a remote potentially multiple times.
* Fix sending presence over federation when using workers (#10163)Erik Johnston2021-06-111-6/+19
| | | | | | When using a federation sender we'd send out all local presence updates over federation even when they shouldn't be. Fixes #10153.
* Add metrics to track how often events are `soft_failed` (#10156)Eric Eastwood2021-06-111-0/+7
| | | | | | | | | | | Spawned from missing messages we were seeing on `matrix.org` from a federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770. The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066 where the message and join event race and the message is `soft_failed` before the `join` event reaches the remote federated server. Less soft_failed events = better and usually this should only trigger for events where people are doing bad things and trying to fuzz and fake everything.
* Integrate knock rooms with the public rooms directory (#9359)Andrew Morgan2021-06-091-0/+1
| | | | | | | | | | This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403. Specifically, it: * Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory * Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72
* Implement knock feature (#6739)Sorunome2021-06-096-67/+495
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Limit the number of in-flight /keys/query requests from a single device. ↵Patrick Cloke2021-06-091-169/+181
| | | | (#10144)
* Add type hints to the federation server transport. (#10080)Patrick Cloke2021-06-081-3/+3
|
* Handle /backfill returning no events (#10133)Erik Johnston2021-06-081-13/+25
| | | Fixes #10123
* Always update AS last_pos, even on no events (#10107)14mRh4X0r2021-06-071-13/+12
| | | | | | | | | | Fixes #1834. `get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether. Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list. Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
* Delete completes to-device messages earlier in /sync (#10124)Richard van der Hoff2021-06-071-10/+11
| | | | I hope this will improve https://github.com/matrix-org/synapse/issues/9564.
* Don't try and backfill the same room in parallel. (#10116)Erik Johnston2021-06-041-0/+8
| | | | | If backfilling is slow then the client may time out and retry, causing Synapse to start a new `/backfill` before the existing backfill has finished, duplicating work.
* Limit number of events in a replication request (#10118)Erik Johnston2021-06-041-2/+3
| | | Fixes #9956.
* Do not show invite-only rooms in spaces summary (unless joined/invited). ↵Patrick Cloke2021-06-021-10/+9
| | | | (#10109)
* Make /sync do less state res (#10102)Erik Johnston2021-06-021-2/+2
|
* add a cache to have_seen_event (#9953)Richard van der Hoff2021-06-011-5/+7
| | | Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
* Limit the number of events sent over replication when persisting events. ↵Brendan Abolivier2021-05-271-7/+10
| | | | (#10082)
* Remove unused properties from the SpaceSummaryHandler. (#10038)Patrick Cloke2021-05-211-2/+0
|
* Allow a user who could join a restricted room to see it in spaces summary. ↵Patrick Cloke2021-05-202-53/+252
| | | | | | (#9922) This finishes up the experimental implementation of MSC3083 by showing the restricted rooms in the spaces summary (from MSC2946).
* Refactor checking restricted join rules (#10007)Patrick Cloke2021-05-183-51/+49
| | | | | To be more consistent with similar code. The check now automatically raises an AuthError instead of passing back a boolean. It also absorbs some shared logic between callers.
* Use a database table to hold the users that should have full presence sent ↵Andrew Morgan2021-05-181-26/+110
| | | | to them, instead of something in-memory (#9823)
* Fix the allowed range of valid ordering characters for spaces. (#10002)Patrick Cloke2021-05-171-2/+2
| | | | \x7F was meant to be \0x7E (~) this was originally incorrect in MSC1772.
* Clarify comments in the space summary handler. (#9974)Patrick Cloke2021-05-171-5/+46
|
* Split multiplart email sending into a dedicated handler (#9977)Brendan Abolivier2021-05-172-45/+108
| | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Correctly ratelimit invites when creating a room (#9968)Brendan Abolivier2021-05-122-7/+45
| | | | | * Correctly ratelimit invites when creating a room Also allow ratelimiting for more than one action at a time.
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-122-10/+22
|
* Run cache_joined_hosts_for_event in background (#9951)Erik Johnston2021-05-121-5/+40
|
* Sort child events according to MSC1772 for the spaces summary API. (#9954)Patrick Cloke2021-05-111-2/+69
| | | | | | | | | This should help ensure that equivalent results are achieved between homeservers querying for the summary of a space. This implements modified MSC1772 rules, according to MSC2946. The different is that the origin_server_ts of the m.room.create event is not used as a tie-breaker since this might not be known if the homeserver is not part of the room.
* Fix `m.room_key_request` to-device messages (#9961)Richard van der Hoff2021-05-111-6/+27
| | | fixes #9960
* Improve performance of backfilling in large rooms. (#9935)Erik Johnston2021-05-101-69/+54
| | | | | | We were pulling the full auth chain for the room out of the DB each time we backfilled, which can be *huge* for large rooms and is totally unnecessary.
* Always cache 'event_to_prev_state_group' (#9950)Erik Johnston2021-05-071-6/+7
| | | Fixes regression in send PDU times introduced in #9905.
* Include the time of the create event in Spaces Summary. (#9928)Patrick Cloke2021-05-051-0/+1
| | | | | This is an update based on changes to MSC2946. The origin_server_ts of the m.room.create event is copied into the creation_ts field for each room returned from the spaces summary.
* Increase perf of handling presence when joining large rooms. (#9916)Erik Johnston2021-05-051-72/+82
|
* Support stable MSC1772 spaces identifiers. (#9915)Patrick Cloke2021-05-051-2/+6
| | | | Support both the unstable and stable identifiers. A future release will disable the unstable identifiers.
* Don't set the external cache if its been done recently (#9905)Erik Johnston2021-05-052-5/+33
|
* Use get_current_users_in_room from store and not StateHandler (#9910)Erik Johnston2021-05-056-9/+9
|
* Add missing type hints to handlers and fix a Spam Checker type hint. (#9896)Patrick Cloke2021-04-295-53/+76
| | | | | The user_may_create_room_alias method on spam checkers declared the room_alias parameter as a str when in reality it is passed a RoomAlias object.
* Merge remote-tracking branch 'origin/release-v1.33.0' into developErik Johnston2021-04-281-1/+23
|\
| * Fix tight loop handling presence replication. (#9900)Erik Johnston2021-04-281-1/+23
| | | | | | | | | | Only affects workers. Introduced in #9819. Fixes #9899.
* | Add type hints to presence handler (#9885)Erik Johnston2021-04-281-70/+89
|/
* Use current state table for `presence.get_interested_remotes` (#9887)Erik Johnston2021-04-271-7/+2
| | | This should be a lot quicker than asking the state handler.
* Allow OIDC cookies to work on non-root public baseurls (#9726)Andrew Morgan2021-04-231-5/+17
| | | | | Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574. As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.
* Make DomainSpecificString an attrs class (#9875)Erik Johnston2021-04-231-0/+5
|
* Split presence out of master (#9820)Erik Johnston2021-04-231-20/+36
|
* Check for space membership during a remote join of a restricted room (#9814)Patrick Cloke2021-04-233-68/+124
| | | | | | When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Clear the resync bit after resyncing device lists (#9867)Richard van der Hoff2021-04-221-0/+7
| | | Fixes #9866.
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-225-8/+18
| | | This is no longer required, since we have dropped support for Python 3.5.
* Rename handler and config modules which end in handler/config. (#9816)Patrick Cloke2021-04-203-4/+1
|
* Add presence federation stream (#9819)Erik Johnston2021-04-201-20/+223
|
* Fix bug where we sent remote presence states to remote servers (#9850)Erik Johnston2021-04-201-3/+8
|
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-203-10/+7
|
* Port "Allow users to click account renewal links multiple times without ↵Andrew Morgan2021-04-192-25/+80
| | | | | hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
* Sanity check identity server passed to bind/unbind. (#9802)Denis Kasak2021-04-191-3/+26
| | | | Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* Don't send normal presence updates over federation replication stream (#9828)Erik Johnston2021-04-191-15/+63
|
* User directory: use calculated room membership state instead (#9821)Andrew Morgan2021-04-161-7/+8
| | | | | Fixes: #9797. Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
* Separate creating an event context from persisting it in the federation ↵Patrick Cloke2021-04-141-65/+113
| | | | | | handler (#9800) This refactoring allows adding logic that uses the event context before persisting it.
* Revert "Check for space membership during a remote join of a restricted ↵Patrick Cloke2021-04-143-227/+129
| | | | | | | | room. (#9763)" This reverts commit cc51aaaa7adb0ec2235e027b5184ebda9b660ec4. The PR was prematurely merged and not yet approved.
* Check for space membership during a remote join of a restricted room. (#9763)Patrick Cloke2021-04-143-129/+227
| | | | | | | When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Move some replication processing out of generic_worker (#9796)Erik Johnston2021-04-141-0/+246
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1446-46/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-131-1/+1
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-085-11/+18
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* MSC3083: Check for space membership during a local join of restricted rooms. ↵Patrick Cloke2021-04-081-1/+74
| | | | | | | | (#9735) When joining a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan2021-04-061-43/+235
| | | | | | | | | | | | At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
* Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-063-18/+8
|
* Add type hints to the federation handler and server. (#9743)Patrick Cloke2021-04-061-80/+81
|
* Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-052-3/+6
|
* Improve tracing for to device messages (#9686)Erik Johnston2021-04-013-17/+42
|
* Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff2021-03-301-1/+1
| | | | | | | `room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
* Make RateLimiter class check for ratelimit overrides (#9711)Erik Johnston2021-03-307-36/+50
| | | | | | | This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited. We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits. Fixes #9663
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-0/+3
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Spaces summary: call out to other servers (#9653)Richard van der Hoff2021-03-241-16/+119
| | | | | When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data. Fixes: #9447
* Add a type hints for service notices to the HomeServer object. (#9675)Patrick Cloke2021-03-241-2/+4
|
* Federation API for Space summary (#9652)Richard van der Hoff2021-03-231-45/+138
| | | | | Builds on the work done in #9643 to add a federation API for space summaries. There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-2326-26/+26
|
* Incorporate reviewBrendan Abolivier2021-03-191-1/+1
|
* Merge branch 'develop' into babolivier/msc3026Brendan Abolivier2021-03-192-1/+200
|\
| * Initial spaces summary API (#9643)Richard van der Hoff2021-03-181-0/+199
| | | | | | This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
| * Consistently check whether a password may be set for a user. (#9636)Dirk Klimpel2021-03-181-1/+1
| |
* | Fix lintBrendan Abolivier2021-03-191-7/+6
| |
* | Move support for MSC3026 behind an experimental flagBrendan Abolivier2021-03-181-2/+10
| |
* | Implement MSC3026: busy presence stateBrendan Abolivier2021-03-181-1/+2
|/
* only save remote cross-signing keys if they're different from the current ↵Hubert Chathi2021-03-171-4/+18
| | | | | ones (#9634) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix bad naming of storage function (#9637)Erik Johnston2021-03-172-3/+5
| | | | | | We had two functions named `get_forward_extremities_for_room` and `get_forward_extremeties_for_room` that took different paramters. We rename one of them to avoid confusion.
* Add type hints to the room member handler. (#9631)Patrick Cloke2021-03-173-4/+14
|
* Add SSO attribute requirements for OIDC providers (#9609)Hubbe2021-03-161-0/+13
| | | | Allows limiting who can login using OIDC via the claims made from the IdP.
* Return m.change_password.enabled=false if local database is disabled (#9588)Dirk Klimpel2021-03-161-0/+13
| | | | | Instead of if the user does not have a password hash. This allows a SSO user to add a password to their account, but only if the local password database is configured.
* Pass SSO IdP information to spam checker's registration function (#9626)Andrew Morgan2021-03-161-2/+2
| | | | | | | Fixes https://github.com/matrix-org/synapse/issues/9572 When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time. This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
* Handle an empty cookie as an invalid macaroon. (#9620)Patrick Cloke2021-03-161-1/+2
| | | | | * Handle an empty cookie as an invalid macaroon. * Newsfragment
* Add support for stable MSC2858 API (#9617)Richard van der Hoff2021-03-164-0/+10
| | | | | The stable format uses different brand identifiers, so we need to support two identifiers for each IdP.
* Optimise missing prev_event handling (#9601)Richard van der Hoff2021-03-151-21/+131
| | | | | | | | | | | | | | | | | | | | | | | | | | | Background: When we receive incoming federation traffic, and notice that we are missing prev_events from the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events, we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote server for the state at those prev_events, and then we may need to then ask the remote server for any events in that state which we don't already have, as well as the auth events for those missing state events, so that we can auth them. This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the observation that we are currently loading all of those auth events into memory at the start of the operation, but we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're actually going to need, but it doesn't.) The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about 60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache churn. (NB I've already tripled the size of the cache from its default of 10K). Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different (ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting that we clean it up.
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-4/+5
|
* Improve logging when processing incoming transactions (#9596)Richard van der Hoff2021-03-121-46/+16
| | | Put the room id in the logcontext, to make it easier to understand what's going on.
* Convert Requester to attrs (#9586)Richard van der Hoff2021-03-101-2/+3
| | | | | | ... because namedtuples suck Fix up a couple of other annotations to keep mypy happy.
* Fix the auth provider on the logins metric (#9573)Richard van der Hoff2021-03-101-16/+30
| | | | | We either need to pass the auth provider over the replication api, or make sure we report the auth provider on the worker that received the request. I've gone with the latter.
* Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-3/+3
| | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* JWT OIDC secrets for Sign in with Apple (#9549)Richard van der Hoff2021-03-091-5/+96
| | | | | Apple had to be special. They want a client secret which is generated from an EC key. Fixes #9220. Also fixes #9212 while I'm here.
* Fix additional type hints. (#9543)Patrick Cloke2021-03-091-1/+1
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-084-5/+5
|
* Create a SynapseReactor type which incorporates the necessary reactor ↵Patrick Cloke2021-03-081-1/+3
| | | | | interfaces. (#9528) This helps fix some type hints when running with Twisted 21.2.0.
* Prometheus metrics for logins and registrations (#9511)Richard van der Hoff2021-03-042-2/+34
| | | Add prom metrics for number of users successfully registering and logging in, by SSO provider.
* Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff2021-03-043-61/+74
| | | This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
* Prevent presence background jobs from running when presence is disabled (#9530)Aaron Raimist2021-03-031-14/+17
| | | | | Prevent presence background jobs from running when presence is disabled Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"Patrick Cloke2021-03-021-2/+1
| | | | | | | This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46. This is being backed out due to a regression (#9507) and additional review feedback being provided.
* Use the proper Request in type hints. (#9515)Patrick Cloke2021-03-012-3/+3
| | | | This also pins the Twisted version in the mypy job for CI until proper type hints are fixed throughout Synapse.
* Ensure pushers are deleted for deactivated accounts (#9285)Erik Johnston2021-02-251-0/+5
|
* Fix #8518 (sync requests being cached wrongly on timeout) (#9358)Jonathan de Jong2021-02-241-1/+2
| | | | | | | This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Ratelimit cross-user key sharing requests. (#8957)Patrick Cloke2021-02-193-6/+26
|
* Fix style checking due to updated black.Patrick Cloke2021-02-191-2/+1
|
* Be smarter about which hosts to send presence to when processing room joins ↵Andrew Morgan2021-02-191-14/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | (#9402) This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually. --- When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed. It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw. This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence: * If it was a local user join, send that user's latest presence to all servers in the room * If it was a remote user join, send the presence for all local users in the room to that homeserver We deduplicate by inserting all of those pending updates into a dictionary of the form: ``` { server_name1: {presence_update1, ...}, server_name2: {presence_update1, presence_update2, ...} } ``` Only after building this dict do we then start sending out presence updates.
* Add configs to make profile data more private (#9203)AndrewFerr2021-02-192-1/+17
| | | | | | | Add off-by-default configuration settings to: - disable putting an invitee's profile info in invite events - disable profile lookup via federation Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Add back the guard against the user directory stream position not existing. ↵Patrick Cloke2021-02-181-0/+4
| | | | | | (#9428) As the comment says, this guard was there for when the initial user directory update has yet to happen.
* Remove dead notify_for_states presence method (#9408)Andrew Morgan2021-02-171-11/+0
|
* Fix only handling the last presence state for each user (#9425)Andrew Morgan2021-02-171-2/+5
| | | | | | | | | | | | | | This is a small bug that I noticed while working on #8956. We have a for-loop which attempts to strip all presence changes for each user except for the final one, as we don't really care about older presence: https://github.com/matrix-org/synapse/blob/9e19c6aab4b5a99039f2ddc7d3120dd3b26c274b/synapse/handlers/presence.py#L368-L371 `new_states_dict` stores this stripped copy of latest presence state for each user, before it is... put into a new variable `new_state`, which is just overridden by the subsequent for loop. I believe this was instead meant to override `new_states`. Without doing so, it effectively meant: 1. The for loop had no effect. 2. We were still processing old presence state for users.
* Support for form_post in OIDC responses (#9376)Richard van der Hoff2021-02-171-22/+52
| | | Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
* Allow OIDC config to override discovered values (#9384)Richard van der Hoff2021-02-161-9/+18
| | | Fixes #9347
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1629-212/+331
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-161-36/+53
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Handle missing data in power levels events during room upgrade. (#9395)Patrick Cloke2021-02-161-5/+8
|
* Remove dead handled_events set in invite_join (#9394)Andrew Morgan2021-02-121-6/+0
| | | | | This PR removes a set that was created and [initially used](https://github.com/matrix-org/synapse/commit/1d2a0040cff8d04cdc7d7d09d8f04a5d628fa9dd#diff-0bc92da3d703202f5b9be2d3f845e375f5b1a6bc6ba61705a8af9be1121f5e42R435-R436), but is no longer today. May help cut down a bit on the time it takes to accept invites.
* Fix some typos.Patrick Cloke2021-02-121-1/+1
|
* Merge tag 'v1.27.0rc2' into developPatrick Cloke2021-02-111-0/+4
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.27.0rc2 (2021-02-11) ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
| * Backout changes for automatically calculating the public baseurl. (#9313)Patrick Cloke2021-02-111-0/+4
| | | | | | | | This breaks some people's configurations (if their Client-Server API is not accessed via port 443).
* | Combine the CAS & SAML implementations for required attributes. (#9326)Patrick Cloke2021-02-113-51/+86
| |
* | Merge pull request #9150 from Yoric/develop-contextDavid Teller2021-02-081-5/+16
|\ \ | | | | | | New API /_synapse/admin/rooms/{roomId}/context/{eventId}
| * | FIXUP: linterDavid Teller2021-01-281-1/+1
| | |
| * | FIXUP: Making get_event_context a bit more paranoidDavid Teller2021-01-281-2/+8
| | |
| * | FIXUP: Removing awaitableDavid Teller2021-01-281-3/+3
| | |
| * | FIXUP: Don't filter events at all for admin/v1/rooms/.../context/...David Teller2021-01-281-6/+4
| | |
| * | New API /_synapse/admin/rooms/{roomId}/context/{eventId}David Teller2021-01-281-2/+9
| | | | | | | | | | | | Signed-off-by: David Teller <davidt@element.io>
* | | Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixesRichard van der Hoff2021-02-032-4/+12
|\ \ \ | | |/ | |/|
| * | Honour ratelimit flag for application services for invite ratelimiting (#9302)Erik Johnston2021-02-032-4/+12
| | |
* | | Social login UI polish (#9301)Richard van der Hoff2021-02-031-2/+14
| | |
* | | Add debug for OIDC flow (#9307)Richard van der Hoff2021-02-031-15/+25
| | |
* | | Fix formatting for "bad session" error during sso registration flow (#9296)Richard van der Hoff2021-02-031-3/+16
|/ /
* | Add an admin API to get the current room state (#9168)Travis Ralston2021-02-021-1/+1
| | | | | | | | | | This could arguably replace the existing admin API for `/members`, however that is out of scope of this change. This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
* | Put SAML callback URI under /_synapse/client. (#9289)Richard van der Hoff2021-02-021-1/+1
| |
* | Put OIDC callback URI under /_synapse/client. (#9288)Richard van der Hoff2021-02-011-4/+4
| |
* | Merge branch 'social_login' into developRichard van der Hoff2021-02-013-11/+96
|\ \
| * | Collect terms consent from the user during SSO registration (#9276)Richard van der Hoff2021-02-012-0/+46
| | |
| * | Improve styling and wording of SSO UIA templates (#9286)Richard van der Hoff2021-02-011-1/+3
| | | | | | | | | fixes #9171
| * | Make importing display name and email optional (#9277)Richard van der Hoff2021-02-012-10/+47
| | |
* | | Merge branch 'social_login' into developRichard van der Hoff2021-02-012-18/+99
|\| |
| * | Replace username picker with a template (#9275)Richard van der Hoff2021-02-011-1/+1
| | | | | | | | | | | | | | | There's some prelimiary work here to pull out the construction of a jinja environment to a separate function. I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
| * | Improve styling and wording of SSO redirect confirm template (#9272)Richard van der Hoff2021-02-012-2/+32
| | |
| * | Split out a separate endpoint to complete SSO registration (#9262)Richard van der Hoff2021-02-011-15/+66
| | | | | | | | | There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
* | | Prevent email UIA failures from raising a LoginError (#9265)Andrew Morgan2021-02-011-10/+0
| | | | | | | | | | | | | | | | | | | | | Context, Fixes: https://github.com/matrix-org/synapse/issues/9263 In the past to fix an issue with old Riots re-requesting threepid validation tokens, we raised a `LoginError` during UIA instead of `InteractiveAuthIncompleteError`. This is now breaking the way Tchap logs in - which isn't standard, but also isn't disallowed by the spec. An easy fix is just to remove the 4 year old workaround.
* | | Ratelimit invites by room and target user (#9258)Erik Johnston2021-01-293-2/+34
| | |
* | | Merge branch 'social_login' into developRichard van der Hoff2021-01-284-26/+40
|\| |
| * | Add 'brand' field to MSC2858 response (#9242)Richard van der Hoff2021-01-274-2/+12
| | | | | | | | | | | | | | | | | | We've decided to add a 'brand' field to help clients decide how to style the buttons. Also, fix up the allowed characters for idp_id, while I'm in the area.
| * | Support for scraping email addresses from OIDC providers (#9245)Richard van der Hoff2021-01-271-24/+28
| | |
* | | Ratelimit 3PID /requestToken API (#9238)Erik Johnston2021-01-281-0/+28
| | |
* | | Add type hints to E2E handler. (#9232)Patrick Cloke2021-01-283-133/+193
| |/ |/| | | This finishes adding type hints to the `synapse.handlers` module.
* | Merge branch 'social_login' into developRichard van der Hoff2021-01-271-5/+18
|\|
| * Implement MSC2858 support (#9183)Richard van der Hoff2021-01-271-5/+18
| | | | | | Fixes #8928.
* | Add type hints to various handlers. (#9223)Patrick Cloke2021-01-269-127/+174
| | | | | | | | With this change all handlers except the e2e_* ones have type hints enabled.
* | Do not require the CAS service URL setting (use public_baseurl instead). (#9199)Patrick Cloke2021-01-261-5/+1
| | | | | | | | The current configuration is handled for backwards compatibility, but is considered deprecated.
* | Precompute joined hosts and store in Redis (#9198)Erik Johnston2021-01-262-0/+47
|/
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-205-1/+17
|
* Give `public_baseurl` a default value (#9159)Richard van der Hoff2021-01-201-2/+0
|
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-184-10/+173
|
* Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128)Richard van der Hoff2021-01-182-3/+3
| | | | | | | | | | | | | | | | * Factor out a common TestHtmlParser Looks like I'm doing this in a few different places. * Improve OIDC login test Complete the OIDC login flow, rather than giving up halfway through. * Ensure that OIDC login works with multiple OIDC providers * Fix bugs in handling clientRedirectUrl - don't drop duplicate query-params, or params with no value - allow utf-8 in query-params
* Ensure the user ID is serialized in the payload instead of used as an ↵Patrick Cloke2021-01-181-1/+1
| | | | instance name. (#9130)
* Land support for multiple OIDC providers (#9110)Richard van der Hoff2021-01-151-7/+20
| | | | | | | | | | | | | | | | | | | | | | | This is the final step for supporting multiple OIDC providers concurrently. First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before: oidc_config: enabled: true issuer: "https://oidc_provider" # etc After: oidc_providers: - idp_id: prov1 issuer: "https://oidc_provider" - idp_id: prov2 issuer: "https://another_oidc_provider" The old format is still grandfathered in. With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
* Store an IdP ID in the OIDC session (#9109)Richard van der Hoff2021-01-151-6/+16
| | | | | Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP. As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
* Merge pull request #9091 from matrix-org/rav/error_on_bad_ssoRichard van der Hoff2021-01-152-31/+39
|\ | | | | Give the user a better error when they present bad SSO creds
| * Move `complete_sso_ui_auth` into SSOHandlerRichard van der Hoff2021-01-132-28/+13
| | | | | | | | | | since we're hacking on this code anyway, may as well move it out of the cluttered AuthHandler.
| * Give the user a better error when they present bad SSO credsRichard van der Hoff2021-01-131-5/+28
| | | | | | | | | | | | | | | | | | If a user tries to do UI Auth via SSO, but uses the wrong account on the SSO IdP, try to give them a better error. Previously, the UIA would claim to be successful, but then the operation in question would simply fail with "auth fail". Instead, serve up an error page which explains the failure.
* | Split OidcProvider out of OidcHandler (#9107)Richard van der Hoff2021-01-141-98/+148
|/ | | | | | | The idea here is that we will have an instance of OidcProvider for each configured IdP, with OidcHandler just doing the marshalling of them. For now it's still hardcoded with a single provider.
* Extract OIDCProviderConfig objectRichard van der Hoff2021-01-131-17/+20
| | | | | Collect all the config options which related to an OIDC provider into a single object.
* Preparatory refactors of OidcHandler (#9067)Richard van der Hoff2021-01-131-141/+163
| | | | | | | | Some light refactoring of OidcHandler, in preparation for bigger things: * remove inheritance from deprecated BaseHandler * add an object to hold the things that go into a session cookie * factor out a separate class for manipulating said cookies
* Remove user's avatar URL and displayname when deactivated. (#8932)Dirk Klimpel2021-01-122-3/+23
| | | This only applies if the user's data is to be erased.
* UI Auth via SSO: redirect the user to an appropriate SSO. (#9081)Richard van der Hoff2021-01-123-18/+110
| | | | | | | If we have integrations with multiple identity providers, when the user does a UI Auth, we need to redirect them to the right one. There are a few steps to this. First of all we actually need to store the userid of the user we are trying to validate in the UIA session, since the /auth/sso/fallback/web request is unauthenticated. Then, once we get the /auth/sso/fallback/web request, we can fish the user id out of the session, and use it to look up the external id mappings, and hence pick an SSO provider for them.
* Kill off `HomeServer.get_ip_from_request()` (#9080)Richard van der Hoff2021-01-121-7/+2
| | | Homeserver.get_ip_from_request() used to be a bit more complicated, but now it is totally redundant. Let's get rid of it.
* Remove SynapseRequest.get_user_agent (#9069)Richard van der Hoff2021-01-122-5/+6
| | | | | | | | | | | SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests", which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the LoggingContext and write the right entries to the request log). Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a SynapseRequest when there is nothing synapse-specific about the Request at all, and any old twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult. In short: move get_user_agent out to a utility function.
* Allow running sendToDevice on workers (#9044)Erik Johnston2021-01-071-8/+23
|
* Ensure that remote users' device list resyncing always happens on master (#9043)Erik Johnston2021-01-071-4/+13
| | | Currently `DeviceMessageHandler` only ever exists on master, but that is about to change.
* Handle a display name / avatar URL not included in a federation request. (#9023)Patrick Cloke2021-01-061-2/+2
| | | | These may be omitted if not set, but Synapse assumed they would be in the response.
* Implement MSC2176: Updated redaction rules (#8984)Patrick Cloke2021-01-051-1/+1
| | | | An experimental room version ("org.matrix.msc2176") contains the new redaction rules for testing.
* Add initial support for a "pick your IdP" page (#9017)Richard van der Hoff2021-01-054-3/+24
| | | | | During login, if there are multiple IdPs enabled, offer the user a choice of IdPs.
* Combine the SSO Redirect Servlets (#9015)Richard van der Hoff2021-01-044-23/+138
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Implement CasHandler.handle_redirect_request ... to make it match OidcHandler and SamlHandler * Clean up interface for OidcHandler.handle_redirect_request Make it accept `client_redirect_url=None`. * Clean up interface for `SamlHandler.handle_redirect_request` ... bring it into line with CAS and OIDC by making it take a Request parameter, move the magic for `client_redirect_url` for UIA into the handler, and fix the return type to be a `str` rather than a `bytes`. * Define a common protocol for SSO auth provider impls * Give SsoIdentityProvider an ID and register them * Combine the SSO Redirect servlets Now that the SsoHandler knows about the identity providers, we can combine the various *RedirectServlets into a single implementation which delegates to the right IdP. * changelog
* Add type hints to the receipts and user directory handlers. (#8976)Patrick Cloke2021-01-042-32/+61
|
* Use the SSO handler helpers for CAS registration/login. (#8856)Patrick Cloke2021-01-032-39/+77
|
* Check if group IDs are valid before using them. (#8977)Patrick Cloke2020-12-301-1/+1
|
* Add additional type hints to the storage module. (#8980)Patrick Cloke2020-12-302-4/+2
|
* Add type hints to admin and room list handlers. (#8973)Patrick Cloke2020-12-292-68/+89
|
* Refactor the CAS handler in prep for using the abstracted SSO code. (#8958)Patrick Cloke2020-12-182-66/+158
| | | | | | This makes the CAS handler look more like the SAML/OIDC handlers: * Render errors to users instead of throwing JSON errors. * Internal reorganization.
* Send the location of the web client to the IS when inviting via 3PIDs. (#8930)Patrick Cloke2020-12-181-0/+5
| | | | Adds a new setting `email.invite_client_location` which, if defined, is passed to the identity server during invites.
* Implement a username picker for synapse (#8942)Richard van der Hoff2020-12-182-40/+273
| | | | | | | | | | | | | | The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow `UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser back to the client, we redirect to a username-picker resource, which allows the user to enter a username. We *then* complete the SSO flow (including doing the client permission checks). The static resources for the username picker itself (in https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker) are essentially lifted wholesale from https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. As the comment says, we might want to think about making them customisable, but that can be a follow-up. Fixes #8876.
* Allow re-using a UI auth validation for a period of time (#8970)Patrick Cloke2020-12-181-8/+24
|
* Merge remote-tracking branch 'origin/erikj/as_mau_block' into developErik Johnston2020-12-182-2/+13
|\
| * Correctly handle AS registerations and add testErik Johnston2020-12-172-2/+13
| |
* | Try and drop stale extremities. (#8929)Erik Johnston2020-12-181-1/+1
| | | | | | | | If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.
* | Fix a bug that deactivated users appear in the directory (#8933)Dirk Klimpel2020-12-171-2/+6
| | | | | | | | | | | | | | | | | | | | Fixes a bug that deactivated users appear in the directory when their profile information was updated. To change profile information of deactivated users is neccesary for example you will remove displayname or avatar. But they should not appear in directory. They are deactivated. Co-authored-by: Erik Johnston <erikj@jki.re>
* | Push login completion down into SsoHandler (#8941)Richard van der Hoff2020-12-163-82/+75
| | | | | | This is another part of my work towards fixing #8876. It moves some of the logic currently in the SAML and OIDC handlers - in particular the call to `AuthHandler.complete_sso_login` down into the `SsoHandler`.
* | Do not assume that the contents dictionary includes history_visibility. (#8945)Patrick Cloke2020-12-163-8/+10
| |
* | Preparatory refactoring of the SamlHandlerTestCase (#8938)Richard van der Hoff2020-12-151-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | * move simple_async_mock to test_utils ... so that it can be re-used * Remove references to `SamlHandler._map_saml_response_to_user` from tests This method is going away, so we can no longer use it as a test point. Instead, factor out a higher-level method which takes a SAML object, and verify correct behaviour by mocking out `AuthHandler.complete_sso_login`. * changelog
* | Fix startup failure with localdb_enabled: False (#8937)Richard van der Hoff2020-12-141-14/+12
| |
* | Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-119-22/+21
| | | | | | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* | Honour AS ratelimit settings for /login requests (#8920)Erik Johnston2020-12-111-3/+4
| | | | | | | | Fixes #8846.
* | Don't ratelimit autojoining of rooms (#8921)Erik Johnston2020-12-112-11/+17
| | | | | | Fixes #8866
* | Refactor `SsoHandler.get_mxid_from_sso` (#8900)Richard van der Hoff2020-12-102-28/+50
| | | | | | | | | | | | | | * Factor out _call_attribute_mapper and _register_mapped_user This is mostly an attempt to simplify `get_mxid_from_sso`. * Move mapping_lock down into SsoHandler.
* | Simplify the flow for SSO UIA (#8881)Richard van der Hoff2020-12-085-40/+142
| | | | | | | | | | | | | | | | | | * SsoHandler: remove inheritance from BaseHandler * Simplify the flow for SSO UIA We don't need to do all the magic for mapping users when we are doing UIA, so let's factor that out.
* | Merge tag 'v1.24.0rc2' into developPatrick Cloke2020-12-042-6/+23
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.24.0rc2 (2020-12-04) ============================== Bugfixes -------- - Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878)) Internal Changes ---------------- - Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
| * Fix a regression that mapping providers should be able to redirect users. ↵Patrick Cloke2020-12-042-6/+23
| | | | | | | | | | (#8878) This was broken in #8801.