diff options
| author | Andrew Morgan <andrew@amorgan.xyz> | 2020-05-11 16:46:33 +0100 |
|---|---|---|
| committer | Andrew Morgan <andrew@amorgan.xyz> | 2020-05-11 16:46:33 +0100 |
| commit | 5cf758cdd61acc2ae6c123ffb3c6f0b10197dc46 (patch) | |
| tree | 1b70e7664cfaf46050870a4a60e91b0ebf2a0aed | |
| parent | Extend spam checker to allow for multiple modules (#7435) (diff) | |
| parent | Don't UPGRADE database rows (diff) | |
| download | synapse-5cf758cdd61acc2ae6c123ffb3c6f0b10197dc46.tar.xz | |
Merge branch 'release-v1.13.0' into develop
* release-v1.13.0: Don't UPGRADE database rows RST indenting Put rollback instructions in upgrade notes Fix changelog typo Oh yeah, RST Absolute URL it is then Fix upgrade notes link Provide summary of upgrade issues in changelog. Fix ) Move next version notes from changelog to upgrade notes Changelog fixes 1.13.0rc1 Documentation on setting up redis (#7446) Rework UI Auth session validation for registration (#7455) Fix errors from malformed log line (#7454) Drop support for redis.dbid (#7450)
150 files changed, 582 insertions, 320 deletions
diff --git a/CHANGES.md b/CHANGES.md index adb2b3e163..f516dcf0d0 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,15 +1,157 @@ -Next version -============ - -* New templates (`sso_auth_confirm.html`, `sso_auth_success.html`, and - `sso_account_deactivated.html`) were added to Synapse. If your Synapse is - configured to use SSO and a custom `sso_redirect_confirm_template_dir` - configuration then these templates will need to be duplicated into that - directory. - -* Plugins using the `complete_sso_login` method of `synapse.module_api.ModuleApi` - should update to using the async/await version `complete_sso_login_async` which - includes additional checks. The non-async version is considered deprecated. +Synapse 1.13.0rc1 (2020-05-11) +============================== + +This release brings some potential changes necessary for certain +configurations of Synapse: + +* If your Synapse is configured to use SSO and have a custom + `sso_redirect_confirm_template_dir` configuration option set, you will need + to duplicate the new `sso_auth_confirm.html`, `sso_auth_success.html` and + `sso_account_deactivated.html` templates into that directory. +* Synapse plugins using the `complete_sso_login` method of + `synapse.module_api.ModuleApi` should instead switch to the async/await + version, `complete_sso_login_async`, which includes additional checks. The + former version is now deprecated. +* A bug was introduced in Synapse 1.4.0 which could cause the room directory + to be incomplete or empty if Synapse was upgraded directly from v1.2.1 or + earlier, to versions between v1.4.0 and v1.12.x. + +Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes +and for general upgrade guidance. + +Features +-------- + +- Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. ([\#7006](https://github.com/matrix-org/synapse/issues/7006)) +- Set `Referrer-Policy` header to `no-referrer` on media downloads. ([\#7009](https://github.com/matrix-org/synapse/issues/7009)) +- Add support for running replication over Redis when using workers. ([\#7040](https://github.com/matrix-org/synapse/issues/7040), [\#7325](https://github.com/matrix-org/synapse/issues/7325), [\#7352](https://github.com/matrix-org/synapse/issues/7352), [\#7401](https://github.com/matrix-org/synapse/issues/7401), [\#7427](https://github.com/matrix-org/synapse/issues/7427), [\#7439](https://github.com/matrix-org/synapse/issues/7439), [\#7446](https://github.com/matrix-org/synapse/issues/7446), [\#7450](https://github.com/matrix-org/synapse/issues/7450), [\#7454](https://github.com/matrix-org/synapse/issues/7454)) +- Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users. ([\#7051](https://github.com/matrix-org/synapse/issues/7051)) +- Add options to prevent users from changing their profile or associated 3PIDs. ([\#7096](https://github.com/matrix-org/synapse/issues/7096)) +- Support SSO in the user interactive authentication workflow. ([\#7102](https://github.com/matrix-org/synapse/issues/7102), [\#7186](https://github.com/matrix-org/synapse/issues/7186), [\#7279](https://github.com/matrix-org/synapse/issues/7279), [\#7343](https://github.com/matrix-org/synapse/issues/7343)) +- Allow server admins to define and enforce a password policy ([MSC2000](https://github.com/matrix-org/matrix-doc/issues/2000)). ([\#7118](https://github.com/matrix-org/synapse/issues/7118)) +- Improve the support for SSO authentication on the login fallback page. ([\#7152](https://github.com/matrix-org/synapse/issues/7152), [\#7235](https://github.com/matrix-org/synapse/issues/7235)) +- Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. ([\#7153](https://github.com/matrix-org/synapse/issues/7153)) +- Admin users are no longer required to be in a room to create an alias for it. ([\#7191](https://github.com/matrix-org/synapse/issues/7191)) +- Require admin privileges to enable room encryption by default. This does not affect existing rooms. ([\#7230](https://github.com/matrix-org/synapse/issues/7230)) +- Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. ([\#7265](https://github.com/matrix-org/synapse/issues/7265)) +- Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. ([\#7315](https://github.com/matrix-org/synapse/issues/7315)) +- Add a configuration setting to tweak the threshold for dummy events. ([\#7422](https://github.com/matrix-org/synapse/issues/7422)) + + +Bugfixes +-------- + +- Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. ([\#6573](https://github.com/matrix-org/synapse/issues/6573)) +- Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. ([\#6634](https://github.com/matrix-org/synapse/issues/6634)) +- Fix missing field `default` when fetching user-defined push rules. ([\#6639](https://github.com/matrix-org/synapse/issues/6639)) +- Improve error responses when accessing remote public room lists. ([\#6899](https://github.com/matrix-org/synapse/issues/6899), [\#7368](https://github.com/matrix-org/synapse/issues/7368)) +- Transfer alias mappings on room upgrade. ([\#6946](https://github.com/matrix-org/synapse/issues/6946)) +- Ensure that a user interactive authentication session is tied to a single request. ([\#7068](https://github.com/matrix-org/synapse/issues/7068), [\#7455](https://github.com/matrix-org/synapse/issues/7455)) +- Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. ([\#7089](https://github.com/matrix-org/synapse/issues/7089)) +- Return the proper error (`M_BAD_ALIAS`) when a non-existant canonical alias is provided. ([\#7109](https://github.com/matrix-org/synapse/issues/7109)) +- Fix a bug which meant that groups updates were not correctly replicated between workers. ([\#7117](https://github.com/matrix-org/synapse/issues/7117)) +- Fix starting workers when federation sending not split out. ([\#7133](https://github.com/matrix-org/synapse/issues/7133)) +- Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. ([\#7150](https://github.com/matrix-org/synapse/issues/7150)) +- Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. ([\#7151](https://github.com/matrix-org/synapse/issues/7151)) +- Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. ([\#7155](https://github.com/matrix-org/synapse/issues/7155)) +- Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. ([\#7159](https://github.com/matrix-org/synapse/issues/7159)) +- Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. ([\#7177](https://github.com/matrix-org/synapse/issues/7177)) +- Fix a bug which could cause incorrect 'cyclic dependency' error. ([\#7178](https://github.com/matrix-org/synapse/issues/7178)) +- Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. ([\#7199](https://github.com/matrix-org/synapse/issues/7199)) +- Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. ([\#7203](https://github.com/matrix-org/synapse/issues/7203)) +- Do not allow a deactivated user to login via SSO. ([\#7240](https://github.com/matrix-org/synapse/issues/7240), [\#7259](https://github.com/matrix-org/synapse/issues/7259)) +- Fix --help command-line argument. ([\#7249](https://github.com/matrix-org/synapse/issues/7249)) +- Fix room publish permissions not being checked on room creation. ([\#7260](https://github.com/matrix-org/synapse/issues/7260)) +- Reject unknown session IDs during user interactive authentication instead of silently creating a new session. ([\#7268](https://github.com/matrix-org/synapse/issues/7268)) +- Fix a SQL query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. ([\#7274](https://github.com/matrix-org/synapse/issues/7274)) +- Persist user interactive authentication sessions across workers and Synapse restarts. ([\#7302](https://github.com/matrix-org/synapse/issues/7302)) +- Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. ([\#7316](https://github.com/matrix-org/synapse/issues/7316)) +- Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. ([\#7337](https://github.com/matrix-org/synapse/issues/7337), [\#7358](https://github.com/matrix-org/synapse/issues/7358)) +- Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. ([\#7341](https://github.com/matrix-org/synapse/issues/7341)) +- Fix incorrect metrics reporting for `renew_attestations` background task. ([\#7344](https://github.com/matrix-org/synapse/issues/7344)) +- Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. ([\#7367](https://github.com/matrix-org/synapse/issues/7367)) +- Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. ([\#7387](https://github.com/matrix-org/synapse/issues/7387)) +- Fix bug in `EventContext.deserialize`. ([\#7393](https://github.com/matrix-org/synapse/issues/7393)) + + +Improved Documentation +---------------------- + +- Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. ([\#6892](https://github.com/matrix-org/synapse/issues/6892)) +- Improve the documentation for database configuration. ([\#6988](https://github.com/matrix-org/synapse/issues/6988)) +- Improve the documentation of application service configuration files. ([\#7091](https://github.com/matrix-org/synapse/issues/7091)) +- Update pre-built package name for FreeBSD. ([\#7107](https://github.com/matrix-org/synapse/issues/7107)) +- Update postgres docs with login troubleshooting information. ([\#7119](https://github.com/matrix-org/synapse/issues/7119)) +- Clean up INSTALL.md a bit. ([\#7141](https://github.com/matrix-org/synapse/issues/7141)) +- Add documentation for running a local CAS server for testing. ([\#7147](https://github.com/matrix-org/synapse/issues/7147)) +- Improve README.md by being explicit about public IP recommendation for TURN relaying. ([\#7167](https://github.com/matrix-org/synapse/issues/7167)) +- Fix a small typo in the `metrics_flags` config option. ([\#7171](https://github.com/matrix-org/synapse/issues/7171)) +- Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. ([\#7234](https://github.com/matrix-org/synapse/issues/7234)) +- Add documentation to the `password_providers` config option. Add known password provider implementations to docs. ([\#7238](https://github.com/matrix-org/synapse/issues/7238), [\#7248](https://github.com/matrix-org/synapse/issues/7248)) +- Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. ([\#7251](https://github.com/matrix-org/synapse/issues/7251)) +- Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. ([\#7272](https://github.com/matrix-org/synapse/issues/7272)) +- Add documentation on monitoring workers with Prometheus. ([\#7357](https://github.com/matrix-org/synapse/issues/7357)) +- Clarify endpoint usage in the users admin api documentation. ([\#7361](https://github.com/matrix-org/synapse/issues/7361)) + + +Deprecations and Removals +------------------------- + +- Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. ([\#7137](https://github.com/matrix-org/synapse/issues/7137)) + + +Internal Changes +---------------- + +- Add benchmarks for LruCache. ([\#6446](https://github.com/matrix-org/synapse/issues/6446)) +- Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#6881](https://github.com/matrix-org/synapse/issues/6881)) +- Change device list streams to have one row per ID. ([\#7010](https://github.com/matrix-org/synapse/issues/7010)) +- Remove concept of a non-limited stream. ([\#7011](https://github.com/matrix-org/synapse/issues/7011)) +- Move catchup of replication streams logic to worker. ([\#7024](https://github.com/matrix-org/synapse/issues/7024), [\#7195](https://github.com/matrix-org/synapse/issues/7195), [\#7226](https://github.com/matrix-org/synapse/issues/7226), [\#7239](https://github.com/matrix-org/synapse/issues/7239), [\#7286](https://github.com/matrix-org/synapse/issues/7286), [\#7290](https://github.com/matrix-org/synapse/issues/7290), [\#7318](https://github.com/matrix-org/synapse/issues/7318), [\#7326](https://github.com/matrix-org/synapse/issues/7326), [\#7378](https://github.com/matrix-org/synapse/issues/7378), [\#7421](https://github.com/matrix-org/synapse/issues/7421)) +- Convert some of synapse.rest.media to async/await. ([\#7110](https://github.com/matrix-org/synapse/issues/7110), [\#7184](https://github.com/matrix-org/synapse/issues/7184), [\#7241](https://github.com/matrix-org/synapse/issues/7241)) +- De-duplicate / remove unused REST code for login and auth. ([\#7115](https://github.com/matrix-org/synapse/issues/7115)) +- Convert `*StreamRow` classes to inner classes. ([\#7116](https://github.com/matrix-org/synapse/issues/7116)) +- Clean up some LoggingContext code. ([\#7120](https://github.com/matrix-org/synapse/issues/7120), [\#7181](https://github.com/matrix-org/synapse/issues/7181), [\#7183](https://github.com/matrix-org/synapse/issues/7183), [\#7408](https://github.com/matrix-org/synapse/issues/7408), [\#7426](https://github.com/matrix-org/synapse/issues/7426)) +- Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. ([\#7128](https://github.com/matrix-org/synapse/issues/7128)) +- Refactored the CAS authentication logic to a separate class. ([\#7136](https://github.com/matrix-org/synapse/issues/7136)) +- Run replication streamers on workers. ([\#7146](https://github.com/matrix-org/synapse/issues/7146)) +- Add tests for outbound device pokes. ([\#7157](https://github.com/matrix-org/synapse/issues/7157)) +- Fix device list update stream ids going backward. ([\#7158](https://github.com/matrix-org/synapse/issues/7158)) +- Use `stream.current_token()` and remove `stream_positions()`. ([\#7172](https://github.com/matrix-org/synapse/issues/7172)) +- Move client command handling out of TCP protocol. ([\#7185](https://github.com/matrix-org/synapse/issues/7185)) +- Move server command handling out of TCP protocol. ([\#7187](https://github.com/matrix-org/synapse/issues/7187)) +- Fix consistency of HTTP status codes reported in log lines. ([\#7188](https://github.com/matrix-org/synapse/issues/7188)) +- Only run one background database update at a time. ([\#7190](https://github.com/matrix-org/synapse/issues/7190)) +- Remove sent outbound device list pokes from the database. ([\#7192](https://github.com/matrix-org/synapse/issues/7192)) +- Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. ([\#7193](https://github.com/matrix-org/synapse/issues/7193)) +- Remove some extraneous debugging log lines. ([\#7207](https://github.com/matrix-org/synapse/issues/7207)) +- Add explicit Python build tooling as dependencies for the snapcraft build. ([\#7213](https://github.com/matrix-org/synapse/issues/7213)) +- Add typing information to federation server code. ([\#7219](https://github.com/matrix-org/synapse/issues/7219)) +- Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. ([\#7225](https://github.com/matrix-org/synapse/issues/7225)) +- Unblacklist '/upgrade creates a new room' sytest for workers. ([\#7228](https://github.com/matrix-org/synapse/issues/7228)) +- Remove redundant checks on `daemonize` from synctl. ([\#7233](https://github.com/matrix-org/synapse/issues/7233)) +- Upgrade jQuery to v3.4.1 on fallback login/registration pages. ([\#7236](https://github.com/matrix-org/synapse/issues/7236)) +- Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. ([\#7237](https://github.com/matrix-org/synapse/issues/7237)) +- Correct the parameters of a test fixture. Contributed by Isaiah Singletary. ([\#7243](https://github.com/matrix-org/synapse/issues/7243)) +- Convert auth handler to async/await. ([\#7261](https://github.com/matrix-org/synapse/issues/7261)) +- Add some unit tests for replication. ([\#7278](https://github.com/matrix-org/synapse/issues/7278)) +- Improve typing annotations in `synapse.replication.tcp.streams.Stream`. ([\#7291](https://github.com/matrix-org/synapse/issues/7291)) +- Reduce log verbosity of url cache cleanup tasks. ([\#7295](https://github.com/matrix-org/synapse/issues/7295)) +- Fix sample SAML Service Provider configuration. Contributed by @frcl. ([\#7300](https://github.com/matrix-org/synapse/issues/7300)) +- Fix StreamChangeCache to work with multiple entities changing on the same stream id. ([\#7303](https://github.com/matrix-org/synapse/issues/7303)) +- Fix an incorrect import in IdentityHandler. ([\#7319](https://github.com/matrix-org/synapse/issues/7319)) +- Reduce logging verbosity for successful federation requests. ([\#7321](https://github.com/matrix-org/synapse/issues/7321)) +- Convert some federation handler code to async/await. ([\#7338](https://github.com/matrix-org/synapse/issues/7338)) +- Fix collation for postgres for unit tests. ([\#7359](https://github.com/matrix-org/synapse/issues/7359)) +- Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. ([\#7363](https://github.com/matrix-org/synapse/issues/7363)) +- Add an `instance_name` to `RDATA` and `POSITION` replication commands. ([\#7364](https://github.com/matrix-org/synapse/issues/7364)) +- Thread through instance name to replication client. ([\#7369](https://github.com/matrix-org/synapse/issues/7369)) +- Convert synapse.server_notices to async/await. ([\#7394](https://github.com/matrix-org/synapse/issues/7394)) +- Convert synapse.notifier to async/await. ([\#7395](https://github.com/matrix-org/synapse/issues/7395)) +- Fix issues with the Python package manifest. ([\#7404](https://github.com/matrix-org/synapse/issues/7404)) +- Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. ([\#7420](https://github.com/matrix-org/synapse/issues/7420)) +- Speed up fetching device lists changes when handling `/sync` requests. ([\#7423](https://github.com/matrix-org/synapse/issues/7423)) +- Run group attestation renewal in series rather than parallel for performance. ([\#7442](https://github.com/matrix-org/synapse/issues/7442)) Synapse 1.12.4 (2020-04-23) diff --git a/UPGRADE.rst b/UPGRADE.rst index d1408be2af..41c47e964d 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -78,12 +78,13 @@ for example: Upgrading to v1.13.0 ==================== + Incorrect database migration in old synapse versions ---------------------------------------------------- A bug was introduced in Synapse 1.4.0 which could cause the room directory to -be incomplete or empty if Synapse was upgraded directly from v1.2.1 or earlier, -to versions between v1.4.0 and v1.12.x. +be incomplete or empty if Synapse was upgraded directly from v1.2.1 or +earlier, to versions between v1.4.0 and v1.12.x. This will *not* be a problem for Synapse installations which were: * created at v1.4.0 or later, @@ -105,6 +106,42 @@ affected can be repaired as follows: 2. Restart synapse. +New Single Sign-on HTML Templates +--------------------------------- + +New templates (``sso_auth_confirm.html``, ``sso_auth_success.html``, and +``sso_account_deactivated.html``) were added to Synapse. If your Synapse is +configured to use SSO and a custom ``sso_redirect_confirm_template_dir`` +configuration then these templates will need to be copied from +`synapse/res/templates <synapse/res/templates>`_ into that directory. + +Synapse SSO Plugins Method Deprecation +-------------------------------------- + +Plugins using the ``complete_sso_login`` method of +``synapse.module_api.ModuleApi`` should update to using the async/await +version ``complete_sso_login_async`` which includes additional checks. The +non-async version is considered deprecated. + +Rolling back to v1.12.4 after a failed upgrade +---------------------------------------------- + +v1.13.0 includes a lot of large changes. If something problematic occurs, you +may want to roll-back to a previous version of Synapse. Because v1.13.0 also +includes a new database schema version, reverting that version is also required +alongside the generic rollback instructions mentioned above. In short, to roll +back to v1.12.4 you need to: + +1. Stop the server +2. Decrease the schema version in the database: + + .. code:: sql + + UPDATE schema_version SET version = 57; + +3. Downgrade Synapse by following the instructions for your installation method + in the "Rolling back to older versions" section above. + Upgrading to v1.12.0 ==================== diff --git a/changelog.d/6446.misc b/changelog.d/6446.misc deleted file mode 100644 index c42df16f1a..0000000000 --- a/changelog.d/6446.misc +++ /dev/null @@ -1 +0,0 @@ -Add benchmarks for LruCache. diff --git a/changelog.d/6573.bugfix b/changelog.d/6573.bugfix deleted file mode 100644 index 1bb8014db7..0000000000 --- a/changelog.d/6573.bugfix +++ /dev/null @@ -1 +0,0 @@ -Don't attempt to use an invalid sqlite config if no database configuration is provided. Contributed by @nekatak. diff --git a/changelog.d/6634.bugfix b/changelog.d/6634.bugfix deleted file mode 100644 index ec48fdc0a0..0000000000 --- a/changelog.d/6634.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix single-sign on with CAS systems: pass the same service URL when requesting the CAS ticket and when calling the `proxyValidate` URL. Contributed by @Naugrimm. diff --git a/changelog.d/6639.bugfix b/changelog.d/6639.bugfix deleted file mode 100644 index c7593a6e84..0000000000 --- a/changelog.d/6639.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix missing field `default` when fetching user-defined push rules. diff --git a/changelog.d/6881.misc b/changelog.d/6881.misc deleted file mode 100644 index 03b89ccd3d..0000000000 --- a/changelog.d/6881.misc +++ /dev/null @@ -1 +0,0 @@ -Return total number of users and profile attributes in admin users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. diff --git a/changelog.d/6892.doc b/changelog.d/6892.doc deleted file mode 100644 index 0d04cf0bdb..0000000000 --- a/changelog.d/6892.doc +++ /dev/null @@ -1 +0,0 @@ -Update Debian installation instructions to recommend installing the `virtualenv` package instead of `python3-virtualenv`. \ No newline at end of file diff --git a/changelog.d/6899.bugfix b/changelog.d/6899.bugfix deleted file mode 100644 index efa8a40b1f..0000000000 --- a/changelog.d/6899.bugfix +++ /dev/null @@ -1 +0,0 @@ -Improve error responses when accessing remote public room lists. \ No newline at end of file diff --git a/changelog.d/6946.bugfix b/changelog.d/6946.bugfix deleted file mode 100644 index a238c83a18..0000000000 --- a/changelog.d/6946.bugfix +++ /dev/null @@ -1 +0,0 @@ -Transfer alias mappings on room upgrade. \ No newline at end of file diff --git a/changelog.d/6988.doc b/changelog.d/6988.doc deleted file mode 100644 index b6f71bb966..0000000000 --- a/changelog.d/6988.doc +++ /dev/null @@ -1 +0,0 @@ -Improve the documentation for database configuration. diff --git a/changelog.d/7006.feature b/changelog.d/7006.feature deleted file mode 100644 index d2ce9dbaca..0000000000 --- a/changelog.d/7006.feature +++ /dev/null @@ -1 +0,0 @@ -Extend the `web_client_location` option to accept an absolute URL to use as a redirect. Adds a warning when running the web client on the same hostname as homeserver. Contributed by Martin Milata. diff --git a/changelog.d/7009.feature b/changelog.d/7009.feature deleted file mode 100644 index cd2705d5ba..0000000000 --- a/changelog.d/7009.feature +++ /dev/null @@ -1 +0,0 @@ -Set `Referrer-Policy` header to `no-referrer` on media downloads. diff --git a/changelog.d/7010.misc b/changelog.d/7010.misc deleted file mode 100644 index 4ba1f6cdf8..0000000000 --- a/changelog.d/7010.misc +++ /dev/null @@ -1 +0,0 @@ -Change device list streams to have one row per ID. diff --git a/changelog.d/7011.misc b/changelog.d/7011.misc deleted file mode 100644 index 41c3b37574..0000000000 --- a/changelog.d/7011.misc +++ /dev/null @@ -1 +0,0 @@ -Remove concept of a non-limited stream. diff --git a/changelog.d/7024.misc b/changelog.d/7024.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7024.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7040.feature b/changelog.d/7040.feature deleted file mode 100644 index ce6140fdd1..0000000000 --- a/changelog.d/7040.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7051.feature b/changelog.d/7051.feature deleted file mode 100644 index 3e36a3f65e..0000000000 --- a/changelog.d/7051.feature +++ /dev/null @@ -1 +0,0 @@ -Admin API `POST /_synapse/admin/v1/join/<roomIdOrAlias>` to join users to a room like `auto_join_rooms` for creation of users. \ No newline at end of file diff --git a/changelog.d/7068.bugfix b/changelog.d/7068.bugfix deleted file mode 100644 index d1693a7f22..0000000000 --- a/changelog.d/7068.bugfix +++ /dev/null @@ -1 +0,0 @@ -Ensure that a user inteactive authentication session is tied to a single request. diff --git a/changelog.d/7089.bugfix b/changelog.d/7089.bugfix deleted file mode 100644 index f1f440f23a..0000000000 --- a/changelog.d/7089.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug in the federation API which could cause occasional "Failed to get PDU" errors. diff --git a/changelog.d/7091.doc b/changelog.d/7091.doc deleted file mode 100644 index 463536c812..0000000000 --- a/changelog.d/7091.doc +++ /dev/null @@ -1 +0,0 @@ -Improve the documentation of application service configuration files. diff --git a/changelog.d/7096.feature b/changelog.d/7096.feature deleted file mode 100644 index 00f47b2a14..0000000000 --- a/changelog.d/7096.feature +++ /dev/null @@ -1 +0,0 @@ -Add options to prevent users from changing their profile or associated 3PIDs. \ No newline at end of file diff --git a/changelog.d/7102.feature b/changelog.d/7102.feature deleted file mode 100644 index 01057aa396..0000000000 --- a/changelog.d/7102.feature +++ /dev/null @@ -1 +0,0 @@ -Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7107.doc b/changelog.d/7107.doc deleted file mode 100644 index f6da32d406..0000000000 --- a/changelog.d/7107.doc +++ /dev/null @@ -1 +0,0 @@ -Update pre-built package name for FreeBSD. diff --git a/changelog.d/7109.bugfix b/changelog.d/7109.bugfix deleted file mode 100644 index 268de9978e..0000000000 --- a/changelog.d/7109.bugfix +++ /dev/null @@ -1 +0,0 @@ -Return the proper error (M_BAD_ALIAS) when a non-existant canonical alias is provided. diff --git a/changelog.d/7110.misc b/changelog.d/7110.misc deleted file mode 100644 index fac5bc0403..0000000000 --- a/changelog.d/7110.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some of synapse.rest.media to async/await. diff --git a/changelog.d/7115.misc b/changelog.d/7115.misc deleted file mode 100644 index 7d4a011e3e..0000000000 --- a/changelog.d/7115.misc +++ /dev/null @@ -1 +0,0 @@ -De-duplicate / remove unused REST code for login and auth. diff --git a/changelog.d/7116.misc b/changelog.d/7116.misc deleted file mode 100644 index 89d90bd49e..0000000000 --- a/changelog.d/7116.misc +++ /dev/null @@ -1 +0,0 @@ -Convert `*StreamRow` classes to inner classes. diff --git a/changelog.d/7117.bugfix b/changelog.d/7117.bugfix deleted file mode 100644 index 1896d7ad49..0000000000 --- a/changelog.d/7117.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which meant that groups updates were not correctly replicated between workers. diff --git a/changelog.d/7118.feature b/changelog.d/7118.feature deleted file mode 100644 index 5cbfd98160..0000000000 --- a/changelog.d/7118.feature +++ /dev/null @@ -1 +0,0 @@ -Allow server admins to define and enforce a password policy (MSC2000). \ No newline at end of file diff --git a/changelog.d/7119.doc b/changelog.d/7119.doc deleted file mode 100644 index 05192966c3..0000000000 --- a/changelog.d/7119.doc +++ /dev/null @@ -1 +0,0 @@ -Update postgres docs with login troubleshooting information. \ No newline at end of file diff --git a/changelog.d/7120.misc b/changelog.d/7120.misc deleted file mode 100644 index 731f4dcb52..0000000000 --- a/changelog.d/7120.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7128.misc b/changelog.d/7128.misc deleted file mode 100644 index 5703f6d2ec..0000000000 --- a/changelog.d/7128.misc +++ /dev/null @@ -1 +0,0 @@ -Add explicit `instance_id` for USER_SYNC commands and remove implicit `conn_id` usage. diff --git a/changelog.d/7133.bugfix b/changelog.d/7133.bugfix deleted file mode 100644 index 61a86fd34e..0000000000 --- a/changelog.d/7133.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix starting workers when federation sending not split out. diff --git a/changelog.d/7136.misc b/changelog.d/7136.misc deleted file mode 100644 index 3f666d25fd..0000000000 --- a/changelog.d/7136.misc +++ /dev/null @@ -1 +0,0 @@ -Refactored the CAS authentication logic to a separate class. diff --git a/changelog.d/7137.removal b/changelog.d/7137.removal deleted file mode 100644 index 75266a06bb..0000000000 --- a/changelog.d/7137.removal +++ /dev/null @@ -1 +0,0 @@ -Remove nonfunctional `captcha_bypass_secret` option from `homeserver.yaml`. \ No newline at end of file diff --git a/changelog.d/7141.doc b/changelog.d/7141.doc deleted file mode 100644 index 2fcbd666c2..0000000000 --- a/changelog.d/7141.doc +++ /dev/null @@ -1 +0,0 @@ -Clean up INSTALL.md a bit. \ No newline at end of file diff --git a/changelog.d/7146.misc b/changelog.d/7146.misc deleted file mode 100644 index facde06959..0000000000 --- a/changelog.d/7146.misc +++ /dev/null @@ -1 +0,0 @@ -Run replication streamers on workers. diff --git a/changelog.d/7147.doc b/changelog.d/7147.doc deleted file mode 100644 index 2c855ff5f7..0000000000 --- a/changelog.d/7147.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation for running a local CAS server for testing. diff --git a/changelog.d/7150.bugfix b/changelog.d/7150.bugfix deleted file mode 100644 index 1feb294799..0000000000 --- a/changelog.d/7150.bugfix +++ /dev/null @@ -1 +0,0 @@ -Ensure `is_verified` is a boolean in responses to `GET /_matrix/client/r0/room_keys/keys`. Also warn the user if they forgot the `version` query param. \ No newline at end of file diff --git a/changelog.d/7151.bugfix b/changelog.d/7151.bugfix deleted file mode 100644 index 8aaa2dc659..0000000000 --- a/changelog.d/7151.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix error page being shown when a custom SAML handler attempted to redirect when processing an auth response. diff --git a/changelog.d/7152.feature b/changelog.d/7152.feature deleted file mode 100644 index fafa79c7e7..0000000000 --- a/changelog.d/7152.feature +++ /dev/null @@ -1 +0,0 @@ -Improve the support for SSO authentication on the login fallback page. diff --git a/changelog.d/7153.feature b/changelog.d/7153.feature deleted file mode 100644 index 414ebe1f69..0000000000 --- a/changelog.d/7153.feature +++ /dev/null @@ -1 +0,0 @@ -Always whitelist the login fallback in the SSO configuration if `public_baseurl` is set. diff --git a/changelog.d/7155.bugfix b/changelog.d/7155.bugfix deleted file mode 100644 index 0bf51e7aba..0000000000 --- a/changelog.d/7155.bugfix +++ /dev/null @@ -1 +0,0 @@ -Avoid importing `sqlite3` when using the postgres backend. Contributed by David Vo. diff --git a/changelog.d/7157.misc b/changelog.d/7157.misc deleted file mode 100644 index 0eb1128c7a..0000000000 --- a/changelog.d/7157.misc +++ /dev/null @@ -1 +0,0 @@ -Add tests for outbound device pokes. diff --git a/changelog.d/7158.misc b/changelog.d/7158.misc deleted file mode 100644 index 269b8daeb0..0000000000 --- a/changelog.d/7158.misc +++ /dev/null @@ -1 +0,0 @@ -Fix device list update stream ids going backward. diff --git a/changelog.d/7159.bugfix b/changelog.d/7159.bugfix deleted file mode 100644 index 1b341b127b..0000000000 --- a/changelog.d/7159.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix excessive CPU usage by `prune_old_outbound_device_pokes` job. diff --git a/changelog.d/7167.doc b/changelog.d/7167.doc deleted file mode 100644 index a7e7ba9b51..0000000000 --- a/changelog.d/7167.doc +++ /dev/null @@ -1 +0,0 @@ -Improve README.md by being explicit about public IP recommendation for TURN relaying. diff --git a/changelog.d/7171.doc b/changelog.d/7171.doc deleted file mode 100644 index 25a3bd8ac6..0000000000 --- a/changelog.d/7171.doc +++ /dev/null @@ -1 +0,0 @@ -Fix a small typo in the `metrics_flags` config option. \ No newline at end of file diff --git a/changelog.d/7172.misc b/changelog.d/7172.misc deleted file mode 100644 index ffecdf97fe..0000000000 --- a/changelog.d/7172.misc +++ /dev/null @@ -1 +0,0 @@ -Use `stream.current_token()` and remove `stream_positions()`. diff --git a/changelog.d/7177.bugfix b/changelog.d/7177.bugfix deleted file mode 100644 index 329a96cb0b..0000000000 --- a/changelog.d/7177.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which could cause outbound federation traffic to stop working if a client uploaded an incorrect e2e device signature. \ No newline at end of file diff --git a/changelog.d/7178.bugfix b/changelog.d/7178.bugfix deleted file mode 100644 index 35ea645d75..0000000000 --- a/changelog.d/7178.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which could cause incorrect 'cyclic dependency' error. diff --git a/changelog.d/7181.misc b/changelog.d/7181.misc deleted file mode 100644 index 731f4dcb52..0000000000 --- a/changelog.d/7181.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7183.misc b/changelog.d/7183.misc deleted file mode 100644 index 731f4dcb52..0000000000 --- a/changelog.d/7183.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7184.misc b/changelog.d/7184.misc deleted file mode 100644 index fac5bc0403..0000000000 --- a/changelog.d/7184.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some of synapse.rest.media to async/await. diff --git a/changelog.d/7185.misc b/changelog.d/7185.misc deleted file mode 100644 index deb9ca7021..0000000000 --- a/changelog.d/7185.misc +++ /dev/null @@ -1 +0,0 @@ -Move client command handling out of TCP protocol. diff --git a/changelog.d/7186.feature b/changelog.d/7186.feature deleted file mode 100644 index 01057aa396..0000000000 --- a/changelog.d/7186.feature +++ /dev/null @@ -1 +0,0 @@ -Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7187.misc b/changelog.d/7187.misc deleted file mode 100644 index 60d68ae877..0000000000 --- a/changelog.d/7187.misc +++ /dev/null @@ -1 +0,0 @@ -Move server command handling out of TCP protocol. diff --git a/changelog.d/7188.misc b/changelog.d/7188.misc deleted file mode 100644 index f72955b95b..0000000000 --- a/changelog.d/7188.misc +++ /dev/null @@ -1 +0,0 @@ -Fix consistency of HTTP status codes reported in log lines. diff --git a/changelog.d/7190.misc b/changelog.d/7190.misc deleted file mode 100644 index 34348873f1..0000000000 --- a/changelog.d/7190.misc +++ /dev/null @@ -1 +0,0 @@ -Only run one background database update at a time. diff --git a/changelog.d/7191.feature b/changelog.d/7191.feature deleted file mode 100644 index 83d5685bb2..0000000000 --- a/changelog.d/7191.feature +++ /dev/null @@ -1 +0,0 @@ -Admin users are no longer required to be in a room to create an alias for it. diff --git a/changelog.d/7192.misc b/changelog.d/7192.misc deleted file mode 100644 index e401e36399..0000000000 --- a/changelog.d/7192.misc +++ /dev/null @@ -1 +0,0 @@ -Remove sent outbound device list pokes from the database. diff --git a/changelog.d/7193.misc b/changelog.d/7193.misc deleted file mode 100644 index 383a738e64..0000000000 --- a/changelog.d/7193.misc +++ /dev/null @@ -1 +0,0 @@ -Add a background database update job to clear out duplicate `device_lists_outbound_pokes`. diff --git a/changelog.d/7195.misc b/changelog.d/7195.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7195.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7199.bugfix b/changelog.d/7199.bugfix deleted file mode 100644 index b234163ea8..0000000000 --- a/changelog.d/7199.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent. diff --git a/changelog.d/7203.bugfix b/changelog.d/7203.bugfix deleted file mode 100644 index 8b383952e5..0000000000 --- a/changelog.d/7203.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix some worker-mode replication handling not being correctly recorded in CPU usage stats. diff --git a/changelog.d/7207.misc b/changelog.d/7207.misc deleted file mode 100644 index 4f9b6a1089..0000000000 --- a/changelog.d/7207.misc +++ /dev/null @@ -1 +0,0 @@ -Remove some extraneous debugging log lines. \ No newline at end of file diff --git a/changelog.d/7213.misc b/changelog.d/7213.misc deleted file mode 100644 index 03cbfb5f62..0000000000 --- a/changelog.d/7213.misc +++ /dev/null @@ -1 +0,0 @@ -Add explicit Python build tooling as dependencies for the snapcraft build. diff --git a/changelog.d/7219.misc b/changelog.d/7219.misc deleted file mode 100644 index dbf7a530be..0000000000 --- a/changelog.d/7219.misc +++ /dev/null @@ -1 +0,0 @@ -Add typing annotations in `synapse.federation`. diff --git a/changelog.d/7225.misc b/changelog.d/7225.misc deleted file mode 100644 index 375e2a475f..0000000000 --- a/changelog.d/7225.misc +++ /dev/null @@ -1 +0,0 @@ -Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes. \ No newline at end of file diff --git a/changelog.d/7226.misc b/changelog.d/7226.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7226.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7228.misc b/changelog.d/7228.misc deleted file mode 100644 index 50e206377f..0000000000 --- a/changelog.d/7228.misc +++ /dev/null @@ -1 +0,0 @@ -Unblacklist '/upgrade creates a new room' sytest for workers. \ No newline at end of file diff --git a/changelog.d/7230.feature b/changelog.d/7230.feature deleted file mode 100644 index aab777648f..0000000000 --- a/changelog.d/7230.feature +++ /dev/null @@ -1 +0,0 @@ -Require admin privileges to enable room encryption by default. This does not affect existing rooms. diff --git a/changelog.d/7233.misc b/changelog.d/7233.misc deleted file mode 100644 index d9ad582726..0000000000 --- a/changelog.d/7233.misc +++ /dev/null @@ -1 +0,0 @@ -Remove redundant checks on `daemonize` from synctl. diff --git a/changelog.d/7234.doc b/changelog.d/7234.doc deleted file mode 100644 index d284f1422b..0000000000 --- a/changelog.d/7234.doc +++ /dev/null @@ -1 +0,0 @@ -Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution. diff --git a/changelog.d/7235.feature b/changelog.d/7235.feature deleted file mode 100644 index fafa79c7e7..0000000000 --- a/changelog.d/7235.feature +++ /dev/null @@ -1 +0,0 @@ -Improve the support for SSO authentication on the login fallback page. diff --git a/changelog.d/7236.misc b/changelog.d/7236.misc deleted file mode 100644 index e4a2702b54..0000000000 --- a/changelog.d/7236.misc +++ /dev/null @@ -1 +0,0 @@ -Upgrade jQuery to v3.4.1 on fallback login/registration pages. \ No newline at end of file diff --git a/changelog.d/7237.misc b/changelog.d/7237.misc deleted file mode 100644 index 92e67ea31f..0000000000 --- a/changelog.d/7237.misc +++ /dev/null @@ -1 +0,0 @@ -Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible. \ No newline at end of file diff --git a/changelog.d/7238.doc b/changelog.d/7238.doc deleted file mode 100644 index 0e3b4be428..0000000000 --- a/changelog.d/7238.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation to the `password_providers` config option. Add known password provider implementations to docs. \ No newline at end of file diff --git a/changelog.d/7239.misc b/changelog.d/7239.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7239.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7240.bugfix b/changelog.d/7240.bugfix deleted file mode 100644 index 83b18d3e11..0000000000 --- a/changelog.d/7240.bugfix +++ /dev/null @@ -1 +0,0 @@ -Do not allow a deactivated user to login via SSO. diff --git a/changelog.d/7241.misc b/changelog.d/7241.misc deleted file mode 100644 index fac5bc0403..0000000000 --- a/changelog.d/7241.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some of synapse.rest.media to async/await. diff --git a/changelog.d/7243.misc b/changelog.d/7243.misc deleted file mode 100644 index a39c257a54..0000000000 --- a/changelog.d/7243.misc +++ /dev/null @@ -1 +0,0 @@ -Correct the parameters of a test fixture. Contributed by Isaiah Singletary. diff --git a/changelog.d/7248.doc b/changelog.d/7248.doc deleted file mode 100644 index 0e3b4be428..0000000000 --- a/changelog.d/7248.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation to the `password_providers` config option. Add known password provider implementations to docs. \ No newline at end of file diff --git a/changelog.d/7249.bugfix b/changelog.d/7249.bugfix deleted file mode 100644 index 6ae700d365..0000000000 --- a/changelog.d/7249.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix --help command-line argument. diff --git a/changelog.d/7251.doc b/changelog.d/7251.doc deleted file mode 100644 index 596a28e65d..0000000000 --- a/changelog.d/7251.doc +++ /dev/null @@ -1 +0,0 @@ -Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev. diff --git a/changelog.d/7259.bugfix b/changelog.d/7259.bugfix deleted file mode 100644 index 55bb06be8c..0000000000 --- a/changelog.d/7259.bugfix +++ /dev/null @@ -1 +0,0 @@ - Do not allow a deactivated user to login via SSO. diff --git a/changelog.d/7260.bugfix b/changelog.d/7260.bugfix deleted file mode 100644 index 9e50b56f23..0000000000 --- a/changelog.d/7260.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix room publish permissions not being checked on room creation. diff --git a/changelog.d/7261.misc b/changelog.d/7261.misc deleted file mode 100644 index 88165f0105..0000000000 --- a/changelog.d/7261.misc +++ /dev/null @@ -1 +0,0 @@ -Convert auth handler to async/await. diff --git a/changelog.d/7265.feature b/changelog.d/7265.feature deleted file mode 100644 index 345b63e0b7..0000000000 --- a/changelog.d/7265.feature +++ /dev/null @@ -1 +0,0 @@ -Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews. \ No newline at end of file diff --git a/changelog.d/7268.bugfix b/changelog.d/7268.bugfix deleted file mode 100644 index ab280da18e..0000000000 --- a/changelog.d/7268.bugfix +++ /dev/null @@ -1 +0,0 @@ -Reject unknown session IDs during user interactive authentication instead of silently creating a new session. diff --git a/changelog.d/7272.doc b/changelog.d/7272.doc deleted file mode 100644 index 13a1ee340d..0000000000 --- a/changelog.d/7272.doc +++ /dev/null @@ -1 +0,0 @@ -Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins. \ No newline at end of file diff --git a/changelog.d/7274.bugfix b/changelog.d/7274.bugfix deleted file mode 100644 index 211a38befc..0000000000 --- a/changelog.d/7274.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a sql query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log. diff --git a/changelog.d/7278.misc b/changelog.d/7278.misc deleted file mode 100644 index 8a4c4328f4..0000000000 --- a/changelog.d/7278.misc +++ /dev/null @@ -1 +0,0 @@ -Add some unit tests for replication. diff --git a/changelog.d/7279.feature b/changelog.d/7279.feature deleted file mode 100644 index 9aed075474..0000000000 --- a/changelog.d/7279.feature +++ /dev/null @@ -1 +0,0 @@ - Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7286.misc b/changelog.d/7286.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7286.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7290.misc b/changelog.d/7290.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7290.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7291.misc b/changelog.d/7291.misc deleted file mode 100644 index 02e7ae3fa2..0000000000 --- a/changelog.d/7291.misc +++ /dev/null @@ -1 +0,0 @@ -Improve typing annotations in `synapse.replication.tcp.streams.Stream`. diff --git a/changelog.d/7295.misc b/changelog.d/7295.misc deleted file mode 100644 index 239195e75c..0000000000 --- a/changelog.d/7295.misc +++ /dev/null @@ -1 +0,0 @@ -Reduce log verbosity of url cache cleanup tasks. diff --git a/changelog.d/7300.misc b/changelog.d/7300.misc deleted file mode 100644 index 7b3bc362b8..0000000000 --- a/changelog.d/7300.misc +++ /dev/null @@ -1 +0,0 @@ -Fix sample SAML Service Provider configuration. Contributed by @frcl. diff --git a/changelog.d/7302.bugfix b/changelog.d/7302.bugfix deleted file mode 100644 index 820646d1f9..0000000000 --- a/changelog.d/7302.bugfix +++ /dev/null @@ -1 +0,0 @@ -Persist user interactive authentication sessions across workers and Synapse restarts. diff --git a/changelog.d/7303.misc b/changelog.d/7303.misc deleted file mode 100644 index aa89c2b254..0000000000 --- a/changelog.d/7303.misc +++ /dev/null @@ -1 +0,0 @@ -Fix StreamChangeCache to work with multiple entities changing on the same stream id. diff --git a/changelog.d/7315.feature b/changelog.d/7315.feature deleted file mode 100644 index ebcb4741b7..0000000000 --- a/changelog.d/7315.feature +++ /dev/null @@ -1 +0,0 @@ -Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver. diff --git a/changelog.d/7316.bugfix b/changelog.d/7316.bugfix deleted file mode 100644 index 0692696c7b..0000000000 --- a/changelog.d/7316.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fixed backwards compatibility logic of the first value of `trusted_third_party_id_servers` being used for `account_threepid_delegates.email`, which occurs when the former, deprecated option is set and the latter is not. \ No newline at end of file diff --git a/changelog.d/7318.misc b/changelog.d/7318.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7318.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7319.misc b/changelog.d/7319.misc deleted file mode 100644 index 62ea6b6df9..0000000000 --- a/changelog.d/7319.misc +++ /dev/null @@ -1 +0,0 @@ -Fix an incorrect import in IdentityHandler. \ No newline at end of file diff --git a/changelog.d/7321.misc b/changelog.d/7321.misc deleted file mode 100644 index a4b3e67af9..0000000000 --- a/changelog.d/7321.misc +++ /dev/null @@ -1 +0,0 @@ -Reduce logging verbosity for successful federation requests. diff --git a/changelog.d/7325.feature b/changelog.d/7325.feature deleted file mode 100644 index ce6140fdd1..0000000000 --- a/changelog.d/7325.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7326.misc b/changelog.d/7326.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7326.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7337.bugfix b/changelog.d/7337.bugfix deleted file mode 100644 index f49c600173..0000000000 --- a/changelog.d/7337.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. diff --git a/changelog.d/7338.misc b/changelog.d/7338.misc deleted file mode 100644 index 7cafd074ca..0000000000 --- a/changelog.d/7338.misc +++ /dev/null @@ -1 +0,0 @@ -Convert some federation handler code to async/await. diff --git a/changelog.d/7341.bugfix b/changelog.d/7341.bugfix deleted file mode 100644 index 8f0958bcb4..0000000000 --- a/changelog.d/7341.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bad error handling that would cause Synapse to crash if it's provided with a YAML configuration file that's either empty or doesn't parse into a key-value map. diff --git a/changelog.d/7343.feature b/changelog.d/7343.feature deleted file mode 100644 index 01057aa396..0000000000 --- a/changelog.d/7343.feature +++ /dev/null @@ -1 +0,0 @@ -Support SSO in the user interactive authentication workflow. diff --git a/changelog.d/7344.bugfix b/changelog.d/7344.bugfix deleted file mode 100644 index 8c38f9ef80..0000000000 --- a/changelog.d/7344.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix incorrect metrics reporting for `renew_attestations` background task. diff --git a/changelog.d/7352.feature b/changelog.d/7352.feature deleted file mode 100644 index ce6140fdd1..0000000000 --- a/changelog.d/7352.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7357.doc b/changelog.d/7357.doc deleted file mode 100644 index a3d5616ad2..0000000000 --- a/changelog.d/7357.doc +++ /dev/null @@ -1 +0,0 @@ -Add documentation on monitoring workers with Prometheus. diff --git a/changelog.d/7358.bugfix b/changelog.d/7358.bugfix deleted file mode 100644 index f49c600173..0000000000 --- a/changelog.d/7358.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug where event updates might not be sent over replication to worker processes after the stream falls behind. diff --git a/changelog.d/7359.misc b/changelog.d/7359.misc deleted file mode 100644 index b99f257d9a..0000000000 --- a/changelog.d/7359.misc +++ /dev/null @@ -1 +0,0 @@ -Fix collation for postgres for unit tests. diff --git a/changelog.d/7361.doc b/changelog.d/7361.doc deleted file mode 100644 index b35dbc36ee..0000000000 --- a/changelog.d/7361.doc +++ /dev/null @@ -1 +0,0 @@ -Clarify endpoint usage in the users admin api documentation. \ No newline at end of file diff --git a/changelog.d/7363.misc b/changelog.d/7363.misc deleted file mode 100644 index 1e3cddde79..0000000000 --- a/changelog.d/7363.misc +++ /dev/null @@ -1 +0,0 @@ -Convert RegistrationWorkerStore.is_server_admin and dependent code to async/await. \ No newline at end of file diff --git a/changelog.d/7364.misc b/changelog.d/7364.misc deleted file mode 100644 index bb5d727cf4..0000000000 --- a/changelog.d/7364.misc +++ /dev/null @@ -1 +0,0 @@ -Add an `instance_name` to `RDATA` and `POSITION` replication commands. diff --git a/changelog.d/7367.bugfix b/changelog.d/7367.bugfix deleted file mode 100644 index 12171d4e1c..0000000000 --- a/changelog.d/7367.bugfix +++ /dev/null @@ -1 +0,0 @@ -Prevent non-federating rooms from appearing in responses to federated `POST /publicRoom` requests when a filter was included. diff --git a/changelog.d/7368.bugfix b/changelog.d/7368.bugfix deleted file mode 100644 index efa8a40b1f..0000000000 --- a/changelog.d/7368.bugfix +++ /dev/null @@ -1 +0,0 @@ -Improve error responses when accessing remote public room lists. \ No newline at end of file diff --git a/changelog.d/7369.misc b/changelog.d/7369.misc deleted file mode 100644 index 060b09c888..0000000000 --- a/changelog.d/7369.misc +++ /dev/null @@ -1 +0,0 @@ -Thread through instance name to replication client. diff --git a/changelog.d/7378.misc b/changelog.d/7378.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7378.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7387.bugfix b/changelog.d/7387.bugfix deleted file mode 100644 index a250517b49..0000000000 --- a/changelog.d/7387.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug which would cause the room durectory to be incorrectly populated if Synapse was upgraded directly from v1.2.1 or earlier to v1.4.0 or later. Note that this fix does not apply retrospectively; see the [upgrade notes](UPGRADE.rst#upgrading-to-v1130) for more information. diff --git a/changelog.d/7393.bugfix b/changelog.d/7393.bugfix deleted file mode 100644 index 74419af858..0000000000 --- a/changelog.d/7393.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bug in `EventContext.deserialize`. diff --git a/changelog.d/7394.misc b/changelog.d/7394.misc deleted file mode 100644 index f1390308b3..0000000000 --- a/changelog.d/7394.misc +++ /dev/null @@ -1 +0,0 @@ -Convert synapse.server_notices to async/await. diff --git a/changelog.d/7395.misc b/changelog.d/7395.misc deleted file mode 100644 index bc0ad59e2c..0000000000 --- a/changelog.d/7395.misc +++ /dev/null @@ -1 +0,0 @@ -Convert synapse.notifier to async/await. diff --git a/changelog.d/7401.feature b/changelog.d/7401.feature deleted file mode 100644 index ce6140fdd1..0000000000 --- a/changelog.d/7401.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7404.misc b/changelog.d/7404.misc deleted file mode 100644 index 9ac17958cc..0000000000 --- a/changelog.d/7404.misc +++ /dev/null @@ -1 +0,0 @@ -Fix issues with the Python package manifest. diff --git a/changelog.d/7408.misc b/changelog.d/7408.misc deleted file mode 100644 index 731f4dcb52..0000000000 --- a/changelog.d/7408.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7420.misc b/changelog.d/7420.misc deleted file mode 100644 index e834a9163e..0000000000 --- a/changelog.d/7420.misc +++ /dev/null @@ -1 +0,0 @@ -Prevent methods in `synapse.handlers.auth` from polling the homeserver config every request. \ No newline at end of file diff --git a/changelog.d/7421.misc b/changelog.d/7421.misc deleted file mode 100644 index 676f285377..0000000000 --- a/changelog.d/7421.misc +++ /dev/null @@ -1 +0,0 @@ -Move catchup of replication streams logic to worker. diff --git a/changelog.d/7422.feature b/changelog.d/7422.feature deleted file mode 100644 index d6d5bb2169..0000000000 --- a/changelog.d/7422.feature +++ /dev/null @@ -1 +0,0 @@ -Add a configuration setting to tweak the threshold for dummy events. diff --git a/changelog.d/7423.misc b/changelog.d/7423.misc deleted file mode 100644 index eb1767ac13..0000000000 --- a/changelog.d/7423.misc +++ /dev/null @@ -1 +0,0 @@ -Speed up fetching device lists changes when handling `/sync` requests. diff --git a/changelog.d/7426.misc b/changelog.d/7426.misc deleted file mode 100644 index 731f4dcb52..0000000000 --- a/changelog.d/7426.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up some LoggingContext code. diff --git a/changelog.d/7427.feature b/changelog.d/7427.feature deleted file mode 100644 index ce6140fdd1..0000000000 --- a/changelog.d/7427.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7439.feature b/changelog.d/7439.feature deleted file mode 100644 index ce6140fdd1..0000000000 --- a/changelog.d/7439.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for running replication over Redis when using workers. diff --git a/changelog.d/7442.misc b/changelog.d/7442.misc deleted file mode 100644 index a8fd5ad803..0000000000 --- a/changelog.d/7442.misc +++ /dev/null @@ -1 +0,0 @@ -Run group attestation renewal in series rather than parallel for performance. \ No newline at end of file diff --git a/docs/workers.md b/docs/workers.md index cc0b23197f..7512eff43a 100644 --- a/docs/workers.md +++ b/docs/workers.md @@ -1,23 +1,31 @@ # Scaling synapse via workers -Synapse has experimental support for splitting out functionality into -multiple separate python processes, helping greatly with scalability. These +For small instances it recommended to run Synapse in monolith mode (the +default). For larger instances where performance is a concern it can be helpful +to split out functionality into multiple separate python processes. These processes are called 'workers', and are (eventually) intended to scale horizontally independently. -All of the below is highly experimental and subject to change as Synapse evolves, -but documenting it here to help folks needing highly scalable Synapses similar -to the one running matrix.org! +Synapse's worker support is under active development and subject to change as +we attempt to rapidly scale ever larger Synapse instances. However we are +documenting it here to help admins needing a highly scalable Synapse instance +similar to the one running `matrix.org`. -All processes continue to share the same database instance, and as such, workers -only work with postgres based synapse deployments (sharing a single sqlite -across multiple processes is a recipe for disaster, plus you should be using -postgres anyway if you care about scalability). +All processes continue to share the same database instance, and as such, +workers only work with PostgreSQL-based Synapse deployments. SQLite should only +be used for demo purposes and any admin considering workers should already be +running PostgreSQL. -The workers communicate with the master synapse process via a synapse-specific -TCP protocol called 'replication' - analogous to MySQL or Postgres style -database replication; feeding a stream of relevant data to the workers so they -can be kept in sync with the main synapse process and database state. +## Master/worker communication + +The workers communicate with the master process via a Synapse-specific protocol +called 'replication' (analogous to MySQL- or Postgres-style database +replication) which feeds a stream of relevant data from the master to the +workers so they can be kept in sync with the master process and database state. + +Additionally, workers may make HTTP requests to the master, to send information +in the other direction. Typically this is used for operations which need to +wait for a reply - such as sending an event. ## Configuration @@ -27,66 +35,61 @@ the correct worker, or to the main synapse instance. Note that this includes requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse proxy. -To enable workers, you need to add two replication listeners to the master -synapse, e.g.: - - listeners: - # The TCP replication port - - port: 9092 - bind_address: '127.0.0.1' - type: replication - # The HTTP replication port - - port: 9093 - bind_address: '127.0.0.1' - type: http - resources: - - names: [replication] +To enable workers, you need to add *two* replication listeners to the +main Synapse configuration file (`homeserver.yaml`). For example: -Under **no circumstances** should these replication API listeners be exposed to -the public internet; it currently implements no authentication whatsoever and is -unencrypted. - -(Roughly, the TCP port is used for streaming data from the master to the -workers, and the HTTP port for the workers to send data to the main -synapse process.) +```yaml +listeners: + # The TCP replication port + - port: 9092 + bind_address: '127.0.0.1' + type: replication + + # The HTTP replication port + - port: 9093 + bind_address: '127.0.0.1' + type: http + resources: + - names: [replication] +``` -You then create a set of configs for the various worker processes. These -should be worker configuration files, and should be stored in a dedicated -subdirectory, to allow synctl to manipulate them. +Under **no circumstances** should these replication API listeners be exposed to +the public internet; they have no authentication and are unencrypted. -Each worker configuration file inherits the configuration of the main homeserver -configuration file. You can then override configuration specific to that worker, -e.g. the HTTP listener that it provides (if any); logging configuration; etc. -You should minimise the number of overrides though to maintain a usable config. +You should then create a set of configs for the various worker processes. Each +worker configuration file inherits the configuration of the main homeserver +configuration file. You can then override configuration specific to that +worker, e.g. the HTTP listener that it provides (if any); logging +configuration; etc. You should minimise the number of overrides though to +maintain a usable config. In the config file for each worker, you must specify the type of worker application (`worker_app`). The currently available worker applications are -listed below. You must also specify the replication endpoints that it's talking -to on the main synapse process. `worker_replication_host` should specify the -host of the main synapse, `worker_replication_port` should point to the TCP +listed below. You must also specify the replication endpoints that it should +talk to on the main synapse process. `worker_replication_host` should specify +the host of the main synapse, `worker_replication_port` should point to the TCP replication listener port and `worker_replication_http_port` should point to the HTTP replication port. -Currently, the `event_creator` and `federation_reader` workers require specifying -`worker_replication_http_port`. - -For instance: +For example: - worker_app: synapse.app.synchrotron +```yaml +worker_app: synapse.app.synchrotron - # The replication listener on the synapse to talk to. - worker_replication_host: 127.0.0.1 - worker_replication_port: 9092 - worker_replication_http_port: 9093 +# The replication listener on the synapse to talk to. +worker_replication_host: 127.0.0.1 +worker_replication_port: 9092 +worker_replication_http_port: 9093 - worker_listeners: - - type: http - port: 8083 - resources: - - names: - - client +worker_listeners: + - type: http + port: 8083 + resources: + - names: + - client - worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml +worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml +``` ...is a full configuration for a synchrotron worker instance, which will expose a plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided @@ -101,6 +104,50 @@ recommend the use of `systemd` where available: for information on setting up `systemd` to start synapse workers, see [systemd-with-workers](systemd-with-workers). To use `synctl`, see below. +### **Experimental** support for replication over redis + +As of Synapse v1.13.0, it is possible to configure Synapse to send replication +via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an +alternative to direct TCP connections to the master: rather than all the +workers connecting to the master, all the workers and the master connect to +Redis, which relays replication commands between processes. This can give a +significant cpu saving on the master and will be a prerequisite for upcoming +performance improvements. + +Note that this support is currently experimental; you may experience lost +messages and similar problems! It is strongly recommended that admins setting +up workers for the first time use direct TCP replication as above. + +To configure Synapse to use Redis: + +1. Install Redis following the normal procedure for your distribution - for + example, on Debian, `apt install redis-server`. (It is safe to use an + existing Redis deployment if you have one: we use a pub/sub stream named + according to the `server_name` of your synapse server.) +2. Check Redis is running and accessible: you should be able to `echo PING | nc -q1 + localhost 6379` and get a response of `+PONG`. +3. Install the python prerequisites. If you installed synapse into a + virtualenv, this can be done with: + ```sh + pip install matrix-synapse[redis] + ``` + The debian packages from matrix.org already include the required + dependencies. +4. Add config to the shared configuration (`homeserver.yaml`): + ```yaml + redis: + enabled: true + ``` + Optional parameters which can go alongside `enabled` are `host`, `port`, + `password`. Normally none of these are required. +5. Restart master and all workers. + +Once redis replication is in use, `worker_replication_port` is redundant and +can be removed from the worker configuration files. Similarly, the +configuration for the `listener` for the TCP replication port can be removed +from the main configuration file. Note that the HTTP replication port is +still required. + ### Using synctl If you want to use `synctl` to manage your synapse processes, you will need to diff --git a/synapse/__init__.py b/synapse/__init__.py index d8d340f426..6cd16a820b 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -36,7 +36,7 @@ try: except ImportError: pass -__version__ = "1.12.4" +__version__ = "1.13.0rc1" if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): # We import here so that we don't have to install a bunch of deps when diff --git a/synapse/config/redis.py b/synapse/config/redis.py index 81a27619ec..d5d3ca1c9e 100644 --- a/synapse/config/redis.py +++ b/synapse/config/redis.py @@ -31,5 +31,4 @@ class RedisConfig(Config): self.redis_host = redis_config.get("host", "localhost") self.redis_port = redis_config.get("port", 6379) - self.redis_dbid = redis_config.get("dbid") self.redis_password = redis_config.get("password") diff --git a/synapse/handlers/auth.py b/synapse/handlers/auth.py index f8d2331bf1..65c66a00b1 100644 --- a/synapse/handlers/auth.py +++ b/synapse/handlers/auth.py @@ -252,6 +252,7 @@ class AuthHandler(BaseHandler): clientdict: Dict[str, Any], clientip: str, description: str, + validate_clientdict: bool = True, ) -> Tuple[dict, dict, str]: """ Takes a dictionary sent by the client in the login / registration @@ -277,6 +278,10 @@ class AuthHandler(BaseHandler): description: A human readable string to be displayed to the user that describes the operation happening on their account. + validate_clientdict: Whether to validate that the operation happening + on the account has not changed. If this is false, + the client dict is persisted instead of validated. + Returns: A tuple of (creds, params, session_id). @@ -317,30 +322,51 @@ class AuthHandler(BaseHandler): except StoreError: raise SynapseError(400, "Unknown session ID: %s" % (sid,)) + # If the client provides parameters, update what is persisted, + # otherwise use whatever was last provided. + # + # This was designed to allow the client to omit the parameters + # and just supply the session in subsequent calls so it split + # auth between devices by just sharing the session, (eg. so you + # could continue registration from your phone having clicked the + # email auth link on there). It's probably too open to abuse + # because it lets unauthenticated clients store arbitrary objects + # on a homeserver. + # + # Revisit: Assuming the REST APIs do sensible validation, the data + # isn't arbitrary. + # + # Note that the registration endpoint explicitly removes the + # "initial_device_display_name" parameter if it is provided + # without a "password" parameter. See the changes to + # synapse.rest.client.v2_alpha.register.RegisterRestServlet.on_POST + # in commit 544722bad23fc31056b9240189c3cbbbf0ffd3f9. if not clientdict: - # This was designed to allow the client to omit the parameters - # and just supply the session in subsequent calls so it split - # auth between devices by just sharing the session, (eg. so you - # could continue registration from your phone having clicked the - # email auth link on there). It's probably too open to abuse - # because it lets unauthenticated clients store arbitrary objects - # on a homeserver. - # Revisit: Assuming the REST APIs do sensible validation, the data - # isn't arbitrary. clientdict = session.clientdict # Ensure that the queried operation does not vary between stages of # the UI authentication session. This is done by generating a stable - # comparator based on the URI, method, and body (minus the auth dict) - # and storing it during the initial query. Subsequent queries ensure - # that this comparator has not changed. - comparator = (uri, method, clientdict) - if (session.uri, session.method, session.clientdict) != comparator: + # comparator based on the URI, method, and client dict (minus the + # auth dict) and storing it during the initial query. Subsequent + # queries ensure that this comparator has not changed. + if validate_clientdict: + session_comparator = (session.uri, session.method, session.clientdict) + comparator = (uri, method, clientdict) + else: + session_comparator = (session.uri, session.method) # type: ignore + comparator = (uri, method) # type: ignore + + if session_comparator != comparator: raise SynapseError( 403, "Requested operation has changed during the UI authentication session.", ) + # For backwards compatibility the registration endpoint persists + # changes to the client dict instead of validating them. + if not validate_clientdict: + await self.store.set_ui_auth_clientdict(sid, clientdict) + if not authdict: raise InteractiveAuthIncompleteError( self._auth_dict_for_flows(flows, session.session_id) diff --git a/synapse/replication/tcp/handler.py b/synapse/replication/tcp/handler.py index 7c5d6c76e7..1b05468483 100644 --- a/synapse/replication/tcp/handler.py +++ b/synapse/replication/tcp/handler.py @@ -119,10 +119,9 @@ class ReplicationCommandHandler: import txredisapi logger.info( - "Connecting to redis (host=%r port=%r DBID=%r)", + "Connecting to redis (host=%r port=%r)", hs.config.redis_host, hs.config.redis_port, - hs.config.redis_dbid, ) # We need two connections to redis, one for the subscription stream and @@ -133,7 +132,6 @@ class ReplicationCommandHandler: outbound_redis_connection = txredisapi.lazyConnection( host=hs.config.redis_host, port=hs.config.redis_port, - dbid=hs.config.redis_dbid, password=hs.config.redis.redis_password, reconnect=True, ) diff --git a/synapse/replication/tcp/redis.py b/synapse/replication/tcp/redis.py index db69f92557..55bfa71dfd 100644 --- a/synapse/replication/tcp/redis.py +++ b/synapse/replication/tcp/redis.py @@ -96,7 +96,7 @@ class RedisSubscriber(txredisapi.SubscriberProtocol, AbstractConnection): cmd = parse_command_from_line(message) except Exception: logger.exception( - "[%s] failed to parse line: %r", message, + "Failed to parse replication line: %r", message, ) return diff --git a/synapse/rest/client/v2_alpha/register.py b/synapse/rest/client/v2_alpha/register.py index af08cc6cce..e77dd6bf92 100644 --- a/synapse/rest/client/v2_alpha/register.py +++ b/synapse/rest/client/v2_alpha/register.py @@ -516,6 +516,7 @@ class RegisterRestServlet(RestServlet): body, self.hs.get_ip_from_request(request), "register a new account", + validate_clientdict=False, ) # Check that we're not trying to register a denied 3pid. diff --git a/synapse/storage/data_stores/main/ui_auth.py b/synapse/storage/data_stores/main/ui_auth.py index c8eebc9378..1d8ee22fb1 100644 --- a/synapse/storage/data_stores/main/ui_auth.py +++ b/synapse/storage/data_stores/main/ui_auth.py @@ -172,6 +172,27 @@ class UIAuthWorkerStore(SQLBaseStore): return results + async def set_ui_auth_clientdict( + self, session_id: str, clientdict: JsonDict + ) -> None: + """ + Store an updated clientdict for a given session ID. + + Args: + session_id: The ID of this session as returned from check_auth + clientdict: + The dictionary from the client root level, not the 'auth' key. + """ + # The clientdict gets stored as JSON. + clientdict_json = json.dumps(clientdict) + + self.db.simple_update_one( + table="ui_auth_sessions", + keyvalues={"session_id": session_id}, + updatevalues={"clientdict": clientdict_json}, + desc="set_ui_auth_client_dict", + ) + async def set_ui_auth_session_data(self, session_id: str, key: str, value: Any): """ Store a key-value pair into the sessions data associated with this diff --git a/tests/rest/client/v2_alpha/test_auth.py b/tests/rest/client/v2_alpha/test_auth.py index 587be7b2e7..a56c50a5b7 100644 --- a/tests/rest/client/v2_alpha/test_auth.py +++ b/tests/rest/client/v2_alpha/test_auth.py @@ -12,16 +12,20 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +from typing import List, Union from twisted.internet.defer import succeed import synapse.rest.admin from synapse.api.constants import LoginType from synapse.handlers.ui_auth.checkers import UserInteractiveAuthChecker -from synapse.rest.client.v2_alpha import auth, register +from synapse.http.site import SynapseRequest +from synapse.rest.client.v1 import login +from synapse.rest.client.v2_alpha import auth, devices, register +from synapse.types import JsonDict from tests import unittest +from tests.server import FakeChannel class DummyRecaptchaChecker(UserInteractiveAuthChecker): @@ -34,11 +38,15 @@ class DummyRecaptchaChecker(UserInteractiveAuthChecker): return succeed(True) +class DummyPasswordChecker(UserInteractiveAuthChecker): + def check_auth(self, authdict, clientip): + return succeed(authdict["identifier"]["user"]) + + class FallbackAuthTests(unittest.HomeserverTestCase): servlets = [ auth.register_servlets, - synapse.rest.admin.register_servlets_for_client_rest_resource, register.register_servlets, ] hijack_auth = False @@ -59,79 +67,84 @@ class FallbackAuthTests(unittest.HomeserverTestCase): auth_handler = hs.get_auth_handler() auth_handler.checkers[LoginType.RECAPTCHA] = self.recaptcha_checker - @unittest.INFO - def test_fallback_captcha(self): - + def register(self, expected_response: int, body: JsonDict) -> FakeChannel: + """Make a register request.""" request, channel = self.make_request( - "POST", - "register", - {"username": "user", "type": "m.login.password", "password": "bar"}, - ) + "POST", "register", body + ) # type: SynapseRequest, FakeChannel self.render(request) - # Returns a 401 as per the spec - self.assertEqual(request.code, 401) - # Grab the session - session = channel.json_body["session"] - # Assert our configured public key is being given - self.assertEqual( - channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" - ) + self.assertEqual(request.code, expected_response) + return channel + + def recaptcha( + self, session: str, expected_post_response: int, post_session: str = None + ) -> None: + """Get and respond to a fallback recaptcha. Returns the second request.""" + if post_session is None: + post_session = session request, channel = self.make_request( "GET", "auth/m.login.recaptcha/fallback/web?session=" + session - ) + ) # type: SynapseRequest, FakeChannel self.render(request) self.assertEqual(request.code, 200) request, channel = self.make_request( "POST", "auth/m.login.recaptcha/fallback/web?session=" - + session + + post_session + "&g-recaptcha-response=a", ) self.render(request) - self.assertEqual(request.code, 200) + self.assertEqual(request.code, expected_post_response) # The recaptcha handler is called with the response given attempts = self.recaptcha_checker.recaptcha_attempts self.assertEqual(len(attempts), 1) self.assertEqual(attempts[0][0]["response"], "a") - # also complete the dummy auth - request, channel = self.make_request( - "POST", "register", {"auth": {"session": session, "type": "m.login.dummy"}} + @unittest.INFO + def test_fallback_captcha(self): + """Ensure that fallback auth via a captcha works.""" + # Returns a 401 as per the spec + channel = self.register( + 401, {"username": "user", "type": "m.login.password", "password": "bar"}, ) - self.render(request) + + # Grab the session + session = channel.json_body["session"] + # Assert our configured public key is being given + self.assertEqual( + channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" + ) + + # Complete the recaptcha step. + self.recaptcha(session, 200) + + # also complete the dummy auth + self.register(200, {"auth": {"session": session, "type": "m.login.dummy"}}) # Now we should have fulfilled a complete auth flow, including # the recaptcha fallback step, we can then send a # request to the register API with the session in the authdict. - request, channel = self.make_request( - "POST", "register", {"auth": {"session": session}} - ) - self.render(request) - self.assertEqual(channel.code, 200) + channel = self.register(200, {"auth": {"session": session}}) # We're given a registered user. self.assertEqual(channel.json_body["user_id"], "@user:test") - def test_cannot_change_operation(self): + def test_legacy_registration(self): """ - The initial requested operation cannot be modified during the user interactive authentication session. + Registration allows the parameters to vary through the process. """ # Make the initial request to register. (Later on a different password # will be used.) - request, channel = self.make_request( - "POST", - "register", - {"username": "user", "type": "m.login.password", "password": "bar"}, + # Returns a 401 as per the spec + channel = self.register( + 401, {"username": "user", "type": "m.login.password", "password": "bar"}, ) - self.render(request) - # Returns a 401 as per the spec - self.assertEqual(request.code, 401) # Grab the session session = channel.json_body["session"] # Assert our configured public key is being given @@ -139,65 +152,39 @@ class FallbackAuthTests(unittest.HomeserverTestCase): channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" ) - request, channel = self.make_request( - "GET", "auth/m.login.recaptcha/fallback/web?session=" + session - ) - self.render(request) - self.assertEqual(request.code, 200) - - request, channel = self.make_request( - "POST", - "auth/m.login.recaptcha/fallback/web?session=" - + session - + "&g-recaptcha-response=a", - ) - self.render(request) - self.assertEqual(request.code, 200) - - # The recaptcha handler is called with the response given - attempts = self.recaptcha_checker.recaptcha_attempts - self.assertEqual(len(attempts), 1) - self.assertEqual(attempts[0][0]["response"], "a") + # Complete the recaptcha step. + self.recaptcha(session, 200) # also complete the dummy auth - request, channel = self.make_request( - "POST", "register", {"auth": {"session": session, "type": "m.login.dummy"}} - ) - self.render(request) + self.register(200, {"auth": {"session": session, "type": "m.login.dummy"}}) # Now we should have fulfilled a complete auth flow, including # the recaptcha fallback step. Make the initial request again, but - # with a different password. This causes the request to fail since the - # operaiton was modified during the ui auth session. - request, channel = self.make_request( - "POST", - "register", + # with a changed password. This still completes. + channel = self.register( + 200, { "username": "user", "type": "m.login.password", - "password": "foo", # Note this doesn't match the original request. + "password": "foo", # Note that this is different. "auth": {"session": session}, }, ) - self.render(request) - self.assertEqual(channel.code, 403) + + # We're given a registered user. + self.assertEqual(channel.json_body["user_id"], "@user:test") def test_complete_operation_unknown_session(self): """ Attempting to mark an invalid session as complete should error. """ - # Make the initial request to register. (Later on a different password # will be used.) - request, channel = self.make_request( - "POST", - "register", - {"username": "user", "type": "m.login.password", "password": "bar"}, + # Returns a 401 as per the spec + channel = self.register( + 401, {"username": "user", "type": "m.login.password", "password": "bar"} ) - self.render(request) - # Returns a 401 as per the spec - self.assertEqual(request.code, 401) # Grab the session session = channel.json_body["session"] # Assert our configured public key is being given @@ -205,19 +192,160 @@ class FallbackAuthTests(unittest.HomeserverTestCase): channel.json_body["params"]["m.login.recaptcha"]["public_key"], "brokencake" ) + # Attempt to complete the recaptcha step with an unknown session. + # This results in an error. + self.recaptcha(session, 400, session + "unknown") + + +class UIAuthTests(unittest.HomeserverTestCase): + servlets = [ + auth.register_servlets, + devices.register_servlets, + login.register_servlets, + synapse.rest.admin.register_servlets_for_client_rest_resource, + register.register_servlets, + ] + + def prepare(self, reactor, clock, hs): + auth_handler = hs.get_auth_handler() + auth_handler.checkers[LoginType.PASSWORD] = DummyPasswordChecker(hs) + + self.user_pass = "pass" + self.user = self.register_user("test", self.user_pass) + self.user_tok = self.login("test", self.user_pass) + + def get_device_ids(self) -> List[str]: + # Get the list of devices so one can be deleted. request, channel = self.make_request( - "GET", "auth/m.login.recaptcha/fallback/web?session=" + session - ) + "GET", "devices", access_token=self.user_tok, + ) # type: SynapseRequest, FakeChannel self.render(request) + + # Get the ID of the device. self.assertEqual(request.code, 200) + return [d["device_id"] for d in channel.json_body["devices"]] - # Attempt to complete an unknown session, which should return an error. - unknown_session = session + "unknown" + def delete_device( + self, device: str, expected_response: int, body: Union[bytes, JsonDict] = b"" + ) -> FakeChannel: + """Delete an individual device.""" request, channel = self.make_request( - "POST", - "auth/m.login.recaptcha/fallback/web?session=" - + unknown_session - + "&g-recaptcha-response=a", - ) + "DELETE", "devices/" + device, body, access_token=self.user_tok + ) # type: SynapseRequest, FakeChannel self.render(request) - self.assertEqual(request.code, 400) + + # Ensure the response is sane. + self.assertEqual(request.code, expected_response) + + return channel + + def delete_devices(self, expected_response: int, body: JsonDict) -> FakeChannel: + """Delete 1 or more devices.""" + # Note that this uses the delete_devices endpoint so that we can modify + # the payload half-way through some tests. + request, channel = self.make_request( + "POST", "delete_devices", body, access_token=self.user_tok, + ) # type: SynapseRequest, FakeChannel + self.render(request) + + # Ensure the response is sane. + self.assertEqual(request.code, expected_response) + + return channel + + def test_ui_auth(self): + """ + Test user interactive authentication outside of registration. + """ + device_id = self.get_device_ids()[0] + + # Attempt to delete this device. + # Returns a 401 as per the spec + channel = self.delete_device(device_id, 401) + + # Grab the session + session = channel.json_body["session"] + # Ensure that flows are what is expected. + self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"]) + + # Make another request providing the UI auth flow. + self.delete_device( + device_id, + 200, + { + "auth": { + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": self.user}, + "password": self.user_pass, + "session": session, + }, + }, + ) + + def test_cannot_change_body(self): + """ + The initial requested client dict cannot be modified during the user interactive authentication session. + """ + # Create a second login. + self.login("test", self.user_pass) + + device_ids = self.get_device_ids() + self.assertEqual(len(device_ids), 2) + + # Attempt to delete the first device. + # Returns a 401 as per the spec + channel = self.delete_devices(401, {"devices": [device_ids[0]]}) + + # Grab the session + session = channel.json_body["session"] + # Ensure that flows are what is expected. + self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"]) + + # Make another request providing the UI auth flow, but try to delete the + # second device. This results in an error. + self.delete_devices( + 403, + { + "devices": [device_ids[1]], + "auth": { + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": self.user}, + "password": self.user_pass, + "session": session, + }, + }, + ) + + def test_cannot_change_uri(self): + """ + The initial requested URI cannot be modified during the user interactive authentication session. + """ + # Create a second login. + self.login("test", self.user_pass) + + device_ids = self.get_device_ids() + self.assertEqual(len(device_ids), 2) + + # Attempt to delete the first device. + # Returns a 401 as per the spec + channel = self.delete_device(device_ids[0], 401) + + # Grab the session + session = channel.json_body["session"] + # Ensure that flows are what is expected. + self.assertIn({"stages": ["m.login.password"]}, channel.json_body["flows"]) + + # Make another request providing the UI auth flow, but try to delete the + # second device. This results in an error. + self.delete_device( + device_ids[1], + 403, + { + "auth": { + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": self.user}, + "password": self.user_pass, + "session": session, + }, + }, + ) diff --git a/tox.ini b/tox.ini index ad1902d47d..203c648008 100644 --- a/tox.ini +++ b/tox.ini @@ -206,6 +206,7 @@ commands = mypy \ synapse/util/caches/stream_change_cache.py \ tests/replication/tcp/streams \ tests/test_utils \ + tests/rest/client/v2_alpha/test_auth.py \ tests/util/test_stream_change_cache.py # To find all folders that pass mypy you run: |