diff options
67 files changed, 389 insertions, 101 deletions
diff --git a/CHANGES.md b/CHANGES.md index 9837710d06..c216d28818 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,26 +1,98 @@ -Synapse 1.xx.0 (2021-xx-xx) -=========================== +Synapse 1.28.0rc1 (2021-02-19) +============================== Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails). +This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. + Removal warning --------------- The v1 list accounts API is deprecated and will be removed in a future release. This API was undocumented and misleading. It can be replaced by the -[v2 list accounts API](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/user_admin_api.rst#list-accounts), +[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts), which has been available since Synapse 1.7.0 (2019-12-13). Please check if you're using any scripts which use the admin API and replace `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`. +Features +-------- + +- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150)) +- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301)) +- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311)) +- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376)) +- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400)) + + +Bugfixes +-------- + +- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257)) +- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296)) +- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393)) +- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333)) +- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361)) +- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391)) +- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395)) +- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421)) +- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425)) +- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426)) +- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428)) + + +Updates to the Docker image +--------------------------- + +- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433)) + + +Improved Documentation +---------------------- + +- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281)) +- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291)) +- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308)) +- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317)) +- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322)) +- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390)) +- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404)) +- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407)) +- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434)) + + +Deprecations and Removals +------------------------- + +- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429)) + + +Internal Changes +---------------- + +- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003)) +- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123)) +- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240)) +- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299)) +- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305)) +- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307)) +- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326)) +- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362)) +- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396)) +- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381)) +- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384)) +- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394)) +- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408)) + + Synapse 1.27.0 (2021-02-16) =========================== Note that this release includes a change in Synapse to use Redis as a cache ─ as well as a pub/sub mechanism ─ if Redis support is enabled for workers. No action is needed by server administrators, and we do not expect resource usage of the Redis instance to change dramatically. -This release also changes the callback URI for OpenID Connect (OIDC) identity providers. If your server is configured to use single sign-on via an OIDC/OAuth2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. +This release also changes the callback URI for OpenID Connect (OIDC) and SAML2 identity providers. If your server is configured to use single sign-on via an OIDC/OAuth2 or SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. This release also changes escaping of variables in the HTML templates for SSO or email notifications. If you have customised these templates, please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. diff --git a/UPGRADE.rst b/UPGRADE.rst index 22edfe0d60..6f628a6947 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -88,20 +88,21 @@ for example: Upgrading to v1.27.0 ==================== -Changes to callback URI for OAuth2 / OpenID Connect ---------------------------------------------------- +Changes to callback URI for OAuth2 / OpenID Connect and SAML2 +------------------------------------------------------------- -This version changes the URI used for callbacks from OAuth2 identity providers. If -your server is configured for single sign-on via an OpenID Connect or OAuth2 identity -provider, you will need to add ``[synapse public baseurl]/_synapse/client/oidc/callback`` -to the list of permitted "redirect URIs" at the identity provider. +This version changes the URI used for callbacks from OAuth2 and SAML2 identity providers: -See `docs/openid.md <docs/openid.md>`_ for more information on setting up OpenID -Connect. +* If your server is configured for single sign-on via an OpenID Connect or OAuth2 identity + provider, you will need to add ``[synapse public baseurl]/_synapse/client/oidc/callback`` + to the list of permitted "redirect URIs" at the identity provider. -(Note: a similar change is being made for SAML2; in this case the old URI -``[synapse public baseurl]/_matrix/saml2`` is being deprecated, but will continue to -work, so no immediate changes are required for existing installations.) + See `docs/openid.md <docs/openid.md>`_ for more information on setting up OpenID + Connect. + +* If your server is configured for single sign-on via a SAML2 identity provider, you will + need to add ``[synapse public baseurl]/_synapse/client/saml2/authn_response`` as a permitted + "ACS location" (also known as "allowed callback URLs") at the identity provider. Changes to HTML templates ------------------------- diff --git a/changelog.d/9003.misc b/changelog.d/9003.misc deleted file mode 100644 index 557c8b2353..0000000000 --- a/changelog.d/9003.misc +++ /dev/null @@ -1 +0,0 @@ -Fix 'object name reserved for internal use' errors with recent versions of SQLite. diff --git a/changelog.d/9123.misc b/changelog.d/9123.misc deleted file mode 100644 index 329600c40c..0000000000 --- a/changelog.d/9123.misc +++ /dev/null @@ -1 +0,0 @@ -Add experimental support for running Synapse with PyPy. diff --git a/changelog.d/9150.feature b/changelog.d/9150.feature deleted file mode 100644 index 48a8148dee..0000000000 --- a/changelog.d/9150.feature +++ /dev/null @@ -1 +0,0 @@ -New API /_synapse/admin/rooms/{roomId}/context/{eventId}. diff --git a/changelog.d/9203.feature b/changelog.d/9203.feature new file mode 100644 index 0000000000..36b66a47a8 --- /dev/null +++ b/changelog.d/9203.feature @@ -0,0 +1 @@ +Add some configuration settings to make users' profile data more private. diff --git a/changelog.d/9240.misc b/changelog.d/9240.misc deleted file mode 100644 index 850201f6cd..0000000000 --- a/changelog.d/9240.misc +++ /dev/null @@ -1 +0,0 @@ -Deny access to additional IP addresses by default. diff --git a/changelog.d/9257.bugfix b/changelog.d/9257.bugfix deleted file mode 100644 index 5d0bd88dce..0000000000 --- a/changelog.d/9257.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix long-standing bug where sending email push would fail for rooms that the server had since left. diff --git a/changelog.d/9281.doc b/changelog.d/9281.doc deleted file mode 100644 index 4dea375f80..0000000000 --- a/changelog.d/9281.doc +++ /dev/null @@ -1 +0,0 @@ -Reorganizing CHANGELOG.md. \ No newline at end of file diff --git a/changelog.d/9291.doc b/changelog.d/9291.doc deleted file mode 100644 index 422acd3891..0000000000 --- a/changelog.d/9291.doc +++ /dev/null @@ -1 +0,0 @@ -Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. diff --git a/changelog.d/9296.bugfix b/changelog.d/9296.bugfix deleted file mode 100644 index d723f8c5bd..0000000000 --- a/changelog.d/9296.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. diff --git a/changelog.d/9299.misc b/changelog.d/9299.misc deleted file mode 100644 index c883a677ed..0000000000 --- a/changelog.d/9299.misc +++ /dev/null @@ -1 +0,0 @@ -Update the `Cursor` type hints to better match PEP 249. diff --git a/changelog.d/9300.feature b/changelog.d/9300.feature deleted file mode 100644 index a2d0b27da4..0000000000 --- a/changelog.d/9300.feature +++ /dev/null @@ -1 +0,0 @@ -Further improvements to the user experience of registration via single sign-on. diff --git a/changelog.d/9301.feature b/changelog.d/9301.feature deleted file mode 100644 index a2d0b27da4..0000000000 --- a/changelog.d/9301.feature +++ /dev/null @@ -1 +0,0 @@ -Further improvements to the user experience of registration via single sign-on. diff --git a/changelog.d/9305.misc b/changelog.d/9305.misc deleted file mode 100644 index 456bfbfdd7..0000000000 --- a/changelog.d/9305.misc +++ /dev/null @@ -1 +0,0 @@ -Add debug logging for SRV lookups. Contributed by @Bubu. diff --git a/changelog.d/9307.misc b/changelog.d/9307.misc deleted file mode 100644 index 2f54d1ad07..0000000000 --- a/changelog.d/9307.misc +++ /dev/null @@ -1 +0,0 @@ -Improve logging for OIDC login flow. diff --git a/changelog.d/9308.doc b/changelog.d/9308.doc deleted file mode 100644 index 847f2908af..0000000000 --- a/changelog.d/9308.doc +++ /dev/null @@ -1 +0,0 @@ -Correct name of Synapse's service file in TURN howto. diff --git a/changelog.d/9311.feature b/changelog.d/9311.feature deleted file mode 100644 index 293f2118e5..0000000000 --- a/changelog.d/9311.feature +++ /dev/null @@ -1 +0,0 @@ -Add hook to spam checker modules that allow checking file uploads and remote downloads. diff --git a/changelog.d/9317.doc b/changelog.d/9317.doc deleted file mode 100644 index f4d508e090..0000000000 --- a/changelog.d/9317.doc +++ /dev/null @@ -1 +0,0 @@ -Fix the braces in the `oidc_providers` section of the sample config. diff --git a/changelog.d/9321.bugfix b/changelog.d/9321.bugfix deleted file mode 100644 index f1c882b13c..0000000000 --- a/changelog.d/9321.bugfix +++ /dev/null @@ -1 +0,0 @@ -Assert a maximum length for some parameters for spec compliance. diff --git a/changelog.d/9322.doc b/changelog.d/9322.doc deleted file mode 100644 index c393a3a299..0000000000 --- a/changelog.d/9322.doc +++ /dev/null @@ -1 +0,0 @@ -Update installation instructions on Fedora. diff --git a/changelog.d/9326.misc b/changelog.d/9326.misc deleted file mode 100644 index 768c18d27e..0000000000 --- a/changelog.d/9326.misc +++ /dev/null @@ -1 +0,0 @@ -Share the code for handling required attributes between the CAS and SAML handlers. diff --git a/changelog.d/9333.bugfix b/changelog.d/9333.bugfix deleted file mode 100644 index c34ba378c5..0000000000 --- a/changelog.d/9333.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". diff --git a/changelog.d/9361.bugfix b/changelog.d/9361.bugfix deleted file mode 100644 index 4d0477f033..0000000000 --- a/changelog.d/9361.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. diff --git a/changelog.d/9362.misc b/changelog.d/9362.misc deleted file mode 100644 index c75cfeb2a4..0000000000 --- a/changelog.d/9362.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up the code to load the metadata for OpenID Connect identity providers. diff --git a/changelog.d/9376.feature b/changelog.d/9376.feature deleted file mode 100644 index 68ea21dbdd..0000000000 --- a/changelog.d/9376.feature +++ /dev/null @@ -1 +0,0 @@ -Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. diff --git a/changelog.d/9377.misc b/changelog.d/9377.misc deleted file mode 100644 index df1348ec42..0000000000 --- a/changelog.d/9377.misc +++ /dev/null @@ -1 +0,0 @@ -Convert tests to use `HomeserverTestCase`. diff --git a/changelog.d/9381.misc b/changelog.d/9381.misc deleted file mode 100644 index 5688166120..0000000000 --- a/changelog.d/9381.misc +++ /dev/null @@ -1 +0,0 @@ -Update the version of black used to 20.8b1. diff --git a/changelog.d/9383.feature b/changelog.d/9383.feature new file mode 100644 index 0000000000..8957c9cc5e --- /dev/null +++ b/changelog.d/9383.feature @@ -0,0 +1 @@ +Add a configuration option, `user_directory.prefer_local_users`, which when enabled will make it more likely for users on the same server as you to appear above other users. \ No newline at end of file diff --git a/changelog.d/9384.misc b/changelog.d/9384.misc deleted file mode 100644 index 9db61f44db..0000000000 --- a/changelog.d/9384.misc +++ /dev/null @@ -1 +0,0 @@ -Allow OIDC config to override discovered values. diff --git a/changelog.d/9390.doc b/changelog.d/9390.doc deleted file mode 100644 index 8ed2aa8b6a..0000000000 --- a/changelog.d/9390.doc +++ /dev/null @@ -1 +0,0 @@ -Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. diff --git a/changelog.d/9391.bugfix b/changelog.d/9391.bugfix deleted file mode 100644 index b5e68e2ac7..0000000000 --- a/changelog.d/9391.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix bug where Synapse would occaisonally stop reconnecting after the connection was lost. diff --git a/changelog.d/9393.bugfix b/changelog.d/9393.bugfix deleted file mode 100644 index f1c882b13c..0000000000 --- a/changelog.d/9393.bugfix +++ /dev/null @@ -1 +0,0 @@ -Assert a maximum length for some parameters for spec compliance. diff --git a/changelog.d/9394.misc b/changelog.d/9394.misc deleted file mode 100644 index b3e90143cc..0000000000 --- a/changelog.d/9394.misc +++ /dev/null @@ -1 +0,0 @@ -Remove some dead code from the acceptance of room invites path. \ No newline at end of file diff --git a/changelog.d/9395.bugfix b/changelog.d/9395.bugfix deleted file mode 100644 index d45cc4ffb9..0000000000 --- a/changelog.d/9395.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". diff --git a/changelog.d/9396.misc b/changelog.d/9396.misc deleted file mode 100644 index df1348ec42..0000000000 --- a/changelog.d/9396.misc +++ /dev/null @@ -1 +0,0 @@ -Convert tests to use `HomeserverTestCase`. diff --git a/changelog.d/9400.feature b/changelog.d/9400.feature deleted file mode 100644 index 3067c3907b..0000000000 --- a/changelog.d/9400.feature +++ /dev/null @@ -1 +0,0 @@ -Add the shadow-banning status to the display user admin API. \ No newline at end of file diff --git a/changelog.d/9402.bugfix b/changelog.d/9402.bugfix new file mode 100644 index 0000000000..7729225ba2 --- /dev/null +++ b/changelog.d/9402.bugfix @@ -0,0 +1 @@ +Fix a bug where a lot of unnecessary presence updates were sent when joining a room. diff --git a/changelog.d/9404.doc b/changelog.d/9404.doc deleted file mode 100644 index aa2e63f2f6..0000000000 --- a/changelog.d/9404.doc +++ /dev/null @@ -1 +0,0 @@ -Update docs for using Gitea as OpenID provider. diff --git a/changelog.d/9407.doc b/changelog.d/9407.doc deleted file mode 100644 index 36979bc0d8..0000000000 --- a/changelog.d/9407.doc +++ /dev/null @@ -1 +0,0 @@ -Document that pusher instances are shardable. diff --git a/changelog.d/9408.misc b/changelog.d/9408.misc deleted file mode 100644 index 600bacbfe7..0000000000 --- a/changelog.d/9408.misc +++ /dev/null @@ -1 +0,0 @@ -Clean up an unused method in the presence handler code. \ No newline at end of file diff --git a/changelog.d/9421.bugfix b/changelog.d/9421.bugfix deleted file mode 100644 index b73ed5664c..0000000000 --- a/changelog.d/9421.bugfix +++ /dev/null @@ -1 +0,0 @@ -Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. diff --git a/changelog.d/9425.bugfix b/changelog.d/9425.bugfix deleted file mode 100644 index f5b8857cdb..0000000000 --- a/changelog.d/9425.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. \ No newline at end of file diff --git a/changelog.d/9426.bugfix b/changelog.d/9426.bugfix deleted file mode 100644 index a852a315ad..0000000000 --- a/changelog.d/9426.bugfix +++ /dev/null @@ -1 +0,0 @@ -The `ui_auth.session_timeout` configuration setting can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. diff --git a/changelog.d/9428.bugfix b/changelog.d/9428.bugfix deleted file mode 100644 index 132e35440a..0000000000 --- a/changelog.d/9428.bugfix +++ /dev/null @@ -1 +0,0 @@ -Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. diff --git a/changelog.d/9429.removal b/changelog.d/9429.removal deleted file mode 100644 index 1ff9089a20..0000000000 --- a/changelog.d/9429.removal +++ /dev/null @@ -1 +0,0 @@ -Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. \ No newline at end of file diff --git a/changelog.d/9432.misc b/changelog.d/9432.misc new file mode 100644 index 0000000000..1e07da2033 --- /dev/null +++ b/changelog.d/9432.misc @@ -0,0 +1 @@ +Add documentation and type hints to `parse_duration`. diff --git a/changelog.d/9433.docker b/changelog.d/9433.docker deleted file mode 100644 index cbe3a674d3..0000000000 --- a/changelog.d/9433.docker +++ /dev/null @@ -1 +0,0 @@ -Drop support for ARMv7 in Docker images. diff --git a/changelog.d/9440.bugfix b/changelog.d/9440.bugfix new file mode 100644 index 0000000000..47b9842b37 --- /dev/null +++ b/changelog.d/9440.bugfix @@ -0,0 +1 @@ +Fix bug introduced in v1.27.0 where allowing a user to choose their own username when logging in via single sign-on did not work unless an `idp_icon` was defined. diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml index 52380dfb04..c660c62620 100644 --- a/docs/sample_config.yaml +++ b/docs/sample_config.yaml @@ -101,6 +101,14 @@ pid_file: DATADIR/homeserver.pid # #limit_profile_requests_to_users_who_share_rooms: true +# Uncomment to prevent a user's profile data from being retrieved and +# displayed in a room until they have joined it. By default, a user's +# profile data is included in an invite event, regardless of the values +# of the above two settings, and whether or not the users share a server. +# Defaults to 'true'. +# +#include_profile_data_on_invite: false + # If set to 'true', removes the need for authentication to access the server's # public rooms directory through the client API, meaning that anyone can # query the room directory. Defaults to 'false'. @@ -699,6 +707,12 @@ acme: # - matrix.org # - example.com +# Uncomment to disable profile lookup over federation. By default, the +# Federation API allows other homeservers to obtain profile data of any user +# on this homeserver. Defaults to 'true'. +# +#allow_profile_lookup_over_federation: false + ## Caching ## @@ -2540,9 +2554,14 @@ spam_checker: # rebuild the user_directory search indexes, see # https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md # +# 'prefer_local_users' defines whether to prioritise local users in +# search query results. If True, local users are more likely to appear above +# remote users when searching the user directory. Defaults to false. +# #user_directory: # enabled: true # search_all_users: false +# prefer_local_users: false # User Consent configuration diff --git a/synapse/__init__.py b/synapse/__init__.py index 359276427f..2e70f46186 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -48,7 +48,7 @@ try: except ImportError: pass -__version__ = "1.27.0" +__version__ = "1.28.0rc1" if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): # We import here so that we don't have to install a bunch of deps when diff --git a/synapse/config/_base.py b/synapse/config/_base.py index 97399eb9ba..e89decda34 100644 --- a/synapse/config/_base.py +++ b/synapse/config/_base.py @@ -21,7 +21,7 @@ import os from collections import OrderedDict from hashlib import sha256 from textwrap import dedent -from typing import Any, Iterable, List, MutableMapping, Optional +from typing import Any, Iterable, List, MutableMapping, Optional, Union import attr import jinja2 @@ -147,7 +147,20 @@ class Config: return int(value) * size @staticmethod - def parse_duration(value): + def parse_duration(value: Union[str, int]) -> int: + """Convert a duration as a string or integer to a number of milliseconds. + + If an integer is provided it is treated as milliseconds and is unchanged. + + String durations can have a suffix of 's', 'm', 'h', 'd', 'w', or 'y'. + No suffix is treated as milliseconds. + + Args: + value: The duration to parse. + + Returns: + The number of milliseconds in the duration. + """ if isinstance(value, int): return value second = 1000 diff --git a/synapse/config/federation.py b/synapse/config/federation.py index 9f3c57e6a1..55e4db5442 100644 --- a/synapse/config/federation.py +++ b/synapse/config/federation.py @@ -41,6 +41,10 @@ class FederationConfig(Config): ) self.federation_metrics_domains = set(federation_metrics_domains) + self.allow_profile_lookup_over_federation = config.get( + "allow_profile_lookup_over_federation", True + ) + def generate_config_section(self, config_dir_path, server_name, **kwargs): return """\ ## Federation ## @@ -66,6 +70,12 @@ class FederationConfig(Config): #federation_metrics_domains: # - matrix.org # - example.com + + # Uncomment to disable profile lookup over federation. By default, the + # Federation API allows other homeservers to obtain profile data of any user + # on this homeserver. Defaults to 'true'. + # + #allow_profile_lookup_over_federation: false """ diff --git a/synapse/config/server.py b/synapse/config/server.py index 6f3325ff81..0bfd4398e2 100644 --- a/synapse/config/server.py +++ b/synapse/config/server.py @@ -263,6 +263,12 @@ class ServerConfig(Config): False, ) + # Whether to retrieve and display profile data for a user when they + # are invited to a room + self.include_profile_data_on_invite = config.get( + "include_profile_data_on_invite", True + ) + if "restrict_public_rooms_to_local_users" in config and ( "allow_public_rooms_without_auth" in config or "allow_public_rooms_over_federation" in config @@ -848,6 +854,14 @@ class ServerConfig(Config): # #limit_profile_requests_to_users_who_share_rooms: true + # Uncomment to prevent a user's profile data from being retrieved and + # displayed in a room until they have joined it. By default, a user's + # profile data is included in an invite event, regardless of the values + # of the above two settings, and whether or not the users share a server. + # Defaults to 'true'. + # + #include_profile_data_on_invite: false + # If set to 'true', removes the need for authentication to access the server's # public rooms directory through the client API, meaning that anyone can # query the room directory. Defaults to 'false'. diff --git a/synapse/config/user_directory.py b/synapse/config/user_directory.py index c8d19c5d6b..89dbebd148 100644 --- a/synapse/config/user_directory.py +++ b/synapse/config/user_directory.py @@ -26,6 +26,7 @@ class UserDirectoryConfig(Config): def read_config(self, config, **kwargs): self.user_directory_search_enabled = True self.user_directory_search_all_users = False + self.user_directory_search_prefer_local_users = False user_directory_config = config.get("user_directory", None) if user_directory_config: self.user_directory_search_enabled = user_directory_config.get( @@ -34,6 +35,9 @@ class UserDirectoryConfig(Config): self.user_directory_search_all_users = user_directory_config.get( "search_all_users", False ) + self.user_directory_search_prefer_local_users = user_directory_config.get( + "prefer_local_users", False + ) def generate_config_section(self, config_dir_path, server_name, **kwargs): return """ @@ -49,7 +53,12 @@ class UserDirectoryConfig(Config): # rebuild the user_directory search indexes, see # https://github.com/matrix-org/synapse/blob/master/docs/user_directory.md # + # 'prefer_local_users' defines whether to prioritise local users in + # search query results. If True, local users are more likely to appear above + # remote users when searching the user directory. Defaults to false. + # #user_directory: # enabled: true # search_all_users: false + # prefer_local_users: false """ diff --git a/synapse/federation/sender/__init__.py b/synapse/federation/sender/__init__.py index 97fc4d0a82..24ebc4b803 100644 --- a/synapse/federation/sender/__init__.py +++ b/synapse/federation/sender/__init__.py @@ -474,7 +474,7 @@ class FederationSender: self._processing_pending_presence = False def send_presence_to_destinations( - self, states: List[UserPresenceState], destinations: List[str] + self, states: Iterable[UserPresenceState], destinations: Iterable[str] ) -> None: """Send the given presence states to the given destinations. destinations (list[str]) diff --git a/synapse/federation/transport/server.py b/synapse/federation/transport/server.py index cce83704d4..2cf935f38d 100644 --- a/synapse/federation/transport/server.py +++ b/synapse/federation/transport/server.py @@ -484,10 +484,9 @@ class FederationQueryServlet(BaseFederationServlet): # This is when we receive a server-server Query async def on_GET(self, origin, content, query, query_type): - return await self.handler.on_query_request( - query_type, - {k.decode("utf8"): v[0].decode("utf-8") for k, v in query.items()}, - ) + args = {k.decode("utf8"): v[0].decode("utf-8") for k, v in query.items()} + args["origin"] = origin + return await self.handler.on_query_request(query_type, args) class FederationMakeJoinServlet(BaseFederationServlet): diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py index c03f6c997b..1b7c065b34 100644 --- a/synapse/handlers/message.py +++ b/synapse/handlers/message.py @@ -387,6 +387,12 @@ class EventCreationHandler: self.room_invite_state_types = self.hs.config.room_invite_state_types + self.membership_types_to_include_profile_data_in = ( + {Membership.JOIN, Membership.INVITE} + if self.hs.config.include_profile_data_on_invite + else {Membership.JOIN} + ) + self.send_event = ReplicationSendEventRestServlet.make_client(hs) # This is only used to get at ratelimit function, and maybe_kick_guest_users @@ -500,7 +506,7 @@ class EventCreationHandler: membership = builder.content.get("membership", None) target = UserID.from_string(builder.state_key) - if membership in {Membership.JOIN, Membership.INVITE}: + if membership in self.membership_types_to_include_profile_data_in: # If event doesn't include a display name, add one. profile = self.profile_handler content = builder.content diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py index fb85b19770..b6a9ce4f38 100644 --- a/synapse/handlers/presence.py +++ b/synapse/handlers/presence.py @@ -849,6 +849,9 @@ class PresenceHandler(BasePresenceHandler): """Process current state deltas to find new joins that need to be handled. """ + # A map of destination to a set of user state that they should receive + presence_destinations = {} # type: Dict[str, Set[UserPresenceState]] + for delta in deltas: typ = delta["type"] state_key = delta["state_key"] @@ -858,6 +861,7 @@ class PresenceHandler(BasePresenceHandler): logger.debug("Handling: %r %r, %s", typ, state_key, event_id) + # Drop any event that isn't a membership join if typ != EventTypes.Member: continue @@ -880,13 +884,38 @@ class PresenceHandler(BasePresenceHandler): # Ignore changes to join events. continue - await self._on_user_joined_room(room_id, state_key) + # Retrieve any user presence state updates that need to be sent as a result, + # and the destinations that need to receive it + destinations, user_presence_states = await self._on_user_joined_room( + room_id, state_key + ) + + # Insert the destinations and respective updates into our destinations dict + for destination in destinations: + presence_destinations.setdefault(destination, set()).update( + user_presence_states + ) + + # Send out user presence updates for each destination + for destination, user_state_set in presence_destinations.items(): + self.federation.send_presence_to_destinations( + destinations=[destination], states=user_state_set + ) - async def _on_user_joined_room(self, room_id: str, user_id: str) -> None: + async def _on_user_joined_room( + self, room_id: str, user_id: str + ) -> Tuple[List[str], List[UserPresenceState]]: """Called when we detect a user joining the room via the current state - delta stream. - """ + delta stream. Returns the destinations that need to be updated and the + presence updates to send to them. + + Args: + room_id: The ID of the room that the user has joined. + user_id: The ID of the user that has joined the room. + Returns: + A tuple of destinations and presence updates to send to them. + """ if self.is_mine_id(user_id): # If this is a local user then we need to send their presence # out to hosts in the room (who don't already have it) @@ -894,15 +923,15 @@ class PresenceHandler(BasePresenceHandler): # TODO: We should be able to filter the hosts down to those that # haven't previously seen the user - state = await self.current_state_for_user(user_id) - hosts = await self.state.get_current_hosts_in_room(room_id) + remote_hosts = await self.state.get_current_hosts_in_room(room_id) # Filter out ourselves. - hosts = {host for host in hosts if host != self.server_name} + filtered_remote_hosts = [ + host for host in remote_hosts if host != self.server_name + ] - self.federation.send_presence_to_destinations( - states=[state], destinations=hosts - ) + state = await self.current_state_for_user(user_id) + return filtered_remote_hosts, [state] else: # A remote user has joined the room, so we need to: # 1. Check if this is a new server in the room @@ -915,6 +944,8 @@ class PresenceHandler(BasePresenceHandler): # TODO: Check that this is actually a new server joining the # room. + remote_host = get_domain_from_id(user_id) + users = await self.state.get_current_users_in_room(room_id) user_ids = list(filter(self.is_mine_id, users)) @@ -934,10 +965,7 @@ class PresenceHandler(BasePresenceHandler): or state.status_msg is not None ] - if states: - self.federation.send_presence_to_destinations( - states=states, destinations=[get_domain_from_id(user_id)] - ) + return [remote_host], states def should_notify(old_state, new_state): diff --git a/synapse/handlers/profile.py b/synapse/handlers/profile.py index 2f62d84fb5..dd59392bda 100644 --- a/synapse/handlers/profile.py +++ b/synapse/handlers/profile.py @@ -310,6 +310,15 @@ class ProfileHandler(BaseHandler): await self._update_join_states(requester, target_user) async def on_profile_query(self, args: JsonDict) -> JsonDict: + """Handles federation profile query requests.""" + + if not self.hs.config.allow_profile_lookup_over_federation: + raise SynapseError( + 403, + "Profile lookup over federation is disabled on this homeserver", + Codes.FORBIDDEN, + ) + user = UserID.from_string(args["user_id"]) if not self.hs.is_mine(user): raise SynapseError(400, "User is not hosted on this homeserver") diff --git a/synapse/replication/http/federation.py b/synapse/replication/http/federation.py index 7a0dbb5b1a..8af53b4f28 100644 --- a/synapse/replication/http/federation.py +++ b/synapse/replication/http/federation.py @@ -213,8 +213,9 @@ class ReplicationGetQueryRestServlet(ReplicationEndpoint): content = parse_json_object_from_request(request) args = content["args"] + args["origin"] = content["origin"] - logger.info("Got %r query", query_type) + logger.info("Got %r query from %s", query_type, args["origin"]) result = await self.registry.on_query(query_type, args) diff --git a/synapse/res/templates/sso_auth_account_details.html b/synapse/res/templates/sso_auth_account_details.html index f4fdc40b22..00e1dcdbb8 100644 --- a/synapse/res/templates/sso_auth_account_details.html +++ b/synapse/res/templates/sso_auth_account_details.html @@ -145,7 +145,7 @@ <input type="submit" value="Continue" class="primary-button"> {% if user_attributes.avatar_url or user_attributes.display_name or user_attributes.emails %} <section class="idp-pick-details"> - <h2><img src="{{ idp.idp_icon | mxc_to_http(24, 24) }}"/>Information from {{ idp.idp_name }}</h2> + <h2>{% if idp.idp_icon %}<img src="{{ idp.idp_icon | mxc_to_http(24, 24) }}"/>{% endif %}Information from {{ idp.idp_name }}</h2> {% if user_attributes.avatar_url %} <label class="idp-detail idp-avatar" for="idp-avatar"> <div class="check-row"> diff --git a/synapse/rest/synapse/client/__init__.py b/synapse/rest/synapse/client/__init__.py index e5ef515090..8588b6d271 100644 --- a/synapse/rest/synapse/client/__init__.py +++ b/synapse/rest/synapse/client/__init__.py @@ -54,11 +54,7 @@ def build_synapse_client_resource_tree(hs: "HomeServer") -> Mapping[str, Resourc if hs.config.saml2_enabled: from synapse.rest.synapse.client.saml2 import SAML2Resource - res = SAML2Resource(hs) - resources["/_synapse/client/saml2"] = res - - # This is also mounted under '/_matrix' for backwards-compatibility. - resources["/_matrix/saml2"] = res + resources["/_synapse/client/saml2"] = SAML2Resource(hs) return resources diff --git a/synapse/storage/databases/main/user_directory.py b/synapse/storage/databases/main/user_directory.py index 63f88eac51..02ee15676c 100644 --- a/synapse/storage/databases/main/user_directory.py +++ b/synapse/storage/databases/main/user_directory.py @@ -556,6 +556,11 @@ class UserDirectoryStore(UserDirectoryBackgroundUpdateStore): def __init__(self, database: DatabasePool, db_conn, hs): super().__init__(database, db_conn, hs) + self._prefer_local_users_in_search = ( + hs.config.user_directory_search_prefer_local_users + ) + self._server_name = hs.config.server_name + async def remove_from_user_dir(self, user_id: str) -> None: def _remove_from_user_dir_txn(txn): self.db_pool.simple_delete_txn( @@ -754,9 +759,24 @@ class UserDirectoryStore(UserDirectoryBackgroundUpdateStore): ) """ + # We allow manipulating the ranking algorithm by injecting statements + # based on config options. + additional_ordering_statements = [] + ordering_arguments = () + if isinstance(self.database_engine, PostgresEngine): full_query, exact_query, prefix_query = _parse_query_postgres(search_term) + # If enabled, this config option will rank local users higher than those on + # remote instances. + if self._prefer_local_users_in_search: + # This statement checks whether a given user's user ID contains a server name + # that matches the local server + statement = "* (CASE WHEN user_id LIKE ? THEN 2.0 ELSE 1.0 END)" + additional_ordering_statements.append(statement) + + ordering_arguments += ("%:" + self._server_name,) + # We order by rank and then if they have profile info # The ranking algorithm is hand tweaked for "best" results. Broadly # the idea is we give a higher weight to exact matches. @@ -767,7 +787,7 @@ class UserDirectoryStore(UserDirectoryBackgroundUpdateStore): FROM user_directory_search as t INNER JOIN user_directory AS d USING (user_id) WHERE - %s + %(where_clause)s AND vector @@ to_tsquery('simple', ?) ORDER BY (CASE WHEN d.user_id IS NOT NULL THEN 4.0 ELSE 1.0 END) @@ -787,33 +807,54 @@ class UserDirectoryStore(UserDirectoryBackgroundUpdateStore): 8 ) ) + %(order_case_statements)s DESC, display_name IS NULL, avatar_url IS NULL LIMIT ? - """ % ( - where_clause, + """ % { + "where_clause": where_clause, + "order_case_statements": " ".join(additional_ordering_statements), + } + args = ( + join_args + + (full_query, exact_query, prefix_query) + + ordering_arguments + + (limit + 1,) ) - args = join_args + (full_query, exact_query, prefix_query, limit + 1) elif isinstance(self.database_engine, Sqlite3Engine): search_query = _parse_query_sqlite(search_term) + # If enabled, this config option will rank local users higher than those on + # remote instances. + if self._prefer_local_users_in_search: + # This statement checks whether a given user's user ID contains a server name + # that matches the local server + # + # Note that we need to include a comma at the end for valid SQL + statement = "user_id LIKE ? DESC," + additional_ordering_statements.append(statement) + + ordering_arguments += ("%:" + self._server_name,) + sql = """ SELECT d.user_id AS user_id, display_name, avatar_url FROM user_directory_search as t INNER JOIN user_directory AS d USING (user_id) WHERE - %s + %(where_clause)s AND value MATCH ? ORDER BY rank(matchinfo(user_directory_search)) DESC, + %(order_statements)s display_name IS NULL, avatar_url IS NULL LIMIT ? - """ % ( - where_clause, - ) - args = join_args + (search_query, limit + 1) + """ % { + "where_clause": where_clause, + "order_statements": " ".join(additional_ordering_statements), + } + args = join_args + (search_query,) + ordering_arguments + (limit + 1,) else: # This should be unreachable. raise Exception("Unrecognized database engine") diff --git a/tests/handlers/test_presence.py b/tests/handlers/test_presence.py index be2ee26f07..996c614198 100644 --- a/tests/handlers/test_presence.py +++ b/tests/handlers/test_presence.py @@ -521,7 +521,7 @@ class PresenceJoinTestCase(unittest.HomeserverTestCase): ) self.assertEqual(expected_state.state, PresenceState.ONLINE) self.federation_sender.send_presence_to_destinations.assert_called_once_with( - destinations=["server2"], states=[expected_state] + destinations=["server2"], states={expected_state} ) # @@ -533,7 +533,7 @@ class PresenceJoinTestCase(unittest.HomeserverTestCase): self.federation_sender.send_presence.assert_not_called() self.federation_sender.send_presence_to_destinations.assert_called_once_with( - destinations=["server3"], states=[expected_state] + destinations=["server3"], states={expected_state} ) def test_remote_gets_presence_when_local_user_joins(self): @@ -584,8 +584,14 @@ class PresenceJoinTestCase(unittest.HomeserverTestCase): self.presence_handler.current_state_for_user("@test2:server") ) self.assertEqual(expected_state.state, PresenceState.ONLINE) - self.federation_sender.send_presence_to_destinations.assert_called_once_with( - destinations={"server2", "server3"}, states=[expected_state] + self.assertEqual( + self.federation_sender.send_presence_to_destinations.call_count, 2 + ) + self.federation_sender.send_presence_to_destinations.assert_any_call( + destinations=["server3"], states={expected_state} + ) + self.federation_sender.send_presence_to_destinations.assert_any_call( + destinations=["server2"], states={expected_state} ) def _add_new_user(self, room_id, user_id): diff --git a/tests/handlers/test_profile.py b/tests/handlers/test_profile.py index 18ca8b84f5..75c6a4e21c 100644 --- a/tests/handlers/test_profile.py +++ b/tests/handlers/test_profile.py @@ -161,7 +161,11 @@ class ProfileTestCase(unittest.HomeserverTestCase): response = self.get_success( self.query_handlers["profile"]( - {"user_id": "@caroline:test", "field": "displayname"} + { + "user_id": "@caroline:test", + "field": "displayname", + "origin": "servername.tld", + } ) ) diff --git a/tests/handlers/test_user_directory.py b/tests/handlers/test_user_directory.py index 3572e54c5d..98b2f5b383 100644 --- a/tests/handlers/test_user_directory.py +++ b/tests/handlers/test_user_directory.py @@ -18,6 +18,7 @@ from twisted.internet import defer import synapse.rest.admin from synapse.api.constants import EventTypes, RoomEncryptionAlgorithms, UserTypes +from synapse.api.room_versions import RoomVersion, RoomVersions from synapse.rest.client.v1 import login, room from synapse.rest.client.v2_alpha import user_directory from synapse.storage.roommember import ProfileInfo @@ -46,6 +47,8 @@ class UserDirectoryTestCase(unittest.HomeserverTestCase): def prepare(self, reactor, clock, hs): self.store = hs.get_datastore() self.handler = hs.get_user_directory_handler() + self.event_builder_factory = self.hs.get_event_builder_factory() + self.event_creation_handler = self.hs.get_event_creation_handler() def test_handle_local_profile_change_with_support_user(self): support_user_id = "@support:test" @@ -547,6 +550,100 @@ class UserDirectoryTestCase(unittest.HomeserverTestCase): s = self.get_success(self.handler.search_users(u1, u4, 10)) self.assertEqual(len(s["results"]), 1) + @override_config( + { + "user_directory": { + "enabled": True, + "search_all_users": True, + "prefer_local_users": True, + } + } + ) + def test_prefer_local_users(self): + """Tests that local users are shown higher in search results when + user_directory.prefer_local_users is True. + """ + # Create a room and few users to test the directory with + searching_user = self.register_user("searcher", "password") + searching_user_tok = self.login("searcher", "password") + + room_id = self.helper.create_room_as( + searching_user, + room_version=RoomVersions.V1.identifier, + tok=searching_user_tok, + ) + + # Create a few local users and join them to the room + local_user_1 = self.register_user("user_xxxxx", "password") + local_user_2 = self.register_user("user_bbbbb", "password") + local_user_3 = self.register_user("user_zzzzz", "password") + + self._add_user_to_room(room_id, RoomVersions.V1, local_user_1) + self._add_user_to_room(room_id, RoomVersions.V1, local_user_2) + self._add_user_to_room(room_id, RoomVersions.V1, local_user_3) + + # Create a few "remote" users and join them to the room + remote_user_1 = "@user_aaaaa:remote_server" + remote_user_2 = "@user_yyyyy:remote_server" + remote_user_3 = "@user_ccccc:remote_server" + self._add_user_to_room(room_id, RoomVersions.V1, remote_user_1) + self._add_user_to_room(room_id, RoomVersions.V1, remote_user_2) + self._add_user_to_room(room_id, RoomVersions.V1, remote_user_3) + + local_users = [local_user_1, local_user_2, local_user_3] + remote_users = [remote_user_1, remote_user_2, remote_user_3] + + # Populate the user directory via background update + self._add_background_updates() + while not self.get_success( + self.store.db_pool.updates.has_completed_background_updates() + ): + self.get_success( + self.store.db_pool.updates.do_next_background_update(100), by=0.1 + ) + + # The local searching user searches for the term "user", which other users have + # in their user id + results = self.get_success( + self.handler.search_users(searching_user, "user", 20) + )["results"] + received_user_id_ordering = [result["user_id"] for result in results] + + # Typically we'd expect Synapse to return users in lexicographical order, + # assuming they have similar User IDs/display names, and profile information. + + # Check that the order of returned results using our module is as we expect, + # i.e our local users show up first, despite all users having lexographically mixed + # user IDs. + [self.assertIn(user, local_users) for user in received_user_id_ordering[:3]] + [self.assertIn(user, remote_users) for user in received_user_id_ordering[3:]] + + def _add_user_to_room( + self, + room_id: str, + room_version: RoomVersion, + user_id: str, + ): + # Add a user to the room. + builder = self.event_builder_factory.for_room_version( + room_version, + { + "type": "m.room.member", + "sender": user_id, + "state_key": user_id, + "room_id": room_id, + "content": {"membership": "join"}, + }, + ) + + event, context = self.get_success( + self.event_creation_handler.create_new_client_event(builder) + ) + + self.get_success( + self.hs.get_storage().persistence.persist_event(event, context) + ) + class TestUserDirSearchDisabled(unittest.HomeserverTestCase): user_id = "@test:test" |