summary refs log tree commit diff
path: root/tests (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre2022-01-051-59/+0
| | | | | #11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
* Include the topic event in the prejoin state, per MSC3173. (#11666)Patrick Cloke2022-01-041-0/+9
| | | | Invites and knocks will now include the topic in the stripped state send to clients before joining the room.
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-1/+5
| | | To improve type hints throughout the code.
* Fix a type annotation in `test_account_data.py` and remove it from the Mypy ↵reivilibre2021-12-291-2/+2
| | | | | exclusion list. (#11657) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Refactor `tests.util.setup_test_homeserver` and ↵reivilibre2021-12-214-185/+194
| | | | `tests.server.setup_test_homeserver`. (#11503)
* Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-212-8/+11
|
* Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff2021-12-201-11/+34
| | | This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
* Merge remote-tracking branch 'origin/release-v1.49' into developRichard van der Hoff2021-12-201-5/+5
|\
| * Disable aggregation bundling on `/sync` responses (#11583)Richard van der Hoff2021-12-201-5/+5
| | | | | | | | | | | | | | | | | | | | | | * Disable aggregation bundling on `/sync` responses A partial revert of #11478. This turns out to have had a significant CPU impact on initial-sync handling. For now, let's disable it, until we find a more efficient way of achieving this. * Fix tests. Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* | Use mock from standard library (#11588)V024602021-12-201-9/+8
| | | | | | Instead of the backported version.
* | Add type hints to `synapse/tests/rest/admin` (#11590)Dirk Klimpel2021-12-165-57/+70
| |
* | Add experimental support for MSC3202: allowing application services to ↵reivilibre2021-12-151-0/+64
| | | | | | | | masquerade as specific devices. (#11538)
* | Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah2021-12-141-1/+3
| |
* | Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-141-35/+0
| |
* | Add missing type hints to `synapse.appservice` (#11360)Patrick Cloke2021-12-141-8/+3
| |
* | Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah2021-12-141-1/+1
| | | | | | | | | | `COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
* | Move HTML parsing to a separate file for URL previews. (#11566)Patrick Cloke2021-12-132-23/+24
| | | | | | | | | | | | | | * Splits the logic for parsing HTML from the resource handling code. * Fix a circular import in the oEmbed code (which uses the HTML parsing code). * Renames some of the HTML parsing methods to: * Make it clear which methods are "internal" to the module. * Clarify what the methods do.
* | Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-101-0/+103
| | | | | | | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* | Test to ensure we share the same `state_group` across the whole historical ↵Eric Eastwood2021-12-101-0/+180
| | | | | | | | | | | | | | | | batch (MSC2716) (#11487) Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 We did some work on making sure the `state_groups` were shared in https://github.com/matrix-org/synapse/pull/10975
* | Ensure emails are canonicalized before fetching associated user. (#11547)Patrick Cloke2021-12-101-1/+2
| | | | | | | | This should fix pushers with an email in non-canonical form is used as the pushkey.
* | Do not allow cross-room relations, per MSC2674. (#11516)Patrick Cloke2021-12-091-0/+115
| |
* | Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi2021-12-091-5/+25
| |
* | Add missing `errcode` to `parse_string` and `parse_boolean` (#11542)Dirk Klimpel2021-12-094-10/+10
| |
* | Clean up `synapse.rest.admin` (#11535)Dirk Klimpel2021-12-081-1/+1
| |
* | Use HTTPStatus constants in place of literals in ↵reivilibre2021-12-081-47/+87
| | | | | | | | `tests.rest.client.test_auth`. (#11520)
* | Send and handle cross-signing messages using the stable prefix. (#10520)Hubert Chathi2021-12-081-2/+3
| |
* | Fix case in `wait_for_background_updates` where `self.store` does not exist ↵Andrew Morgan2021-12-071-7/+4
|/ | | | | | (#11331) Pull the DataStore from the HomeServer instance, which always exists.
* Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah2021-12-071-0/+59
| | | | | | `matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
* Fix 'delete room' admin api to work on incomplete rooms (#11523)Richard van der Hoff2021-12-071-17/+25
| | | | | If, for some reason, we don't have the create event, we should still be able to purge a room.
* Fix the test breakage introduced by #11435 as a result of concurrent PRs ↵reivilibre2021-12-071-1/+1
| | | | (#11522)
* Stabilise support for MSC2918 refresh tokens as they have now been merged ↵reivilibre2021-12-061-15/+15
| | | | into the Matrix specification. (#11435)
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-064-33/+188
| | | As a step towards allowing back-channel logout for OIDC.
* Add admin API to get some information about federation status (#11407)Dirk Klimpel2021-12-061-0/+456
|
* Include bundled aggregations in /sync and related fixes (#11478)Patrick Cloke2021-12-061-39/+96
| | | | | | | | Due to updates to MSC2675 this includes a few fixes: * Include bundled aggregations for /sync. * Do not include bundled aggregations for /initialSync and /events. * Do not bundle aggregations for state events. * Clarifies comments and variable names.
* Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)Sean Quah2021-12-061-59/+0
|
* Support configuring the lifetime of non-refreshable access tokens separately ↵reivilibre2021-12-032-0/+154
| | | | to refreshable access tokens. (#11445)
* Add type hints to `synapse/tests/rest/admin` (#11501)Dirk Klimpel2021-12-039-225/+256
|
* Revert accidental commits to develop.Olivier Wilkinson (reivilibre)2021-12-034-194/+185
|
* Give `tests.server.setup_test_homeserver` (nominally!) the same behaviourOlivier Wilkinson (reivilibre)2021-12-031-0/+3
| | | | | | | | | | by calling into `make_test_homeserver_synchronous`. The function *could* have been inlined at this point but the function is big enough and it felt fine to leave it as is. At least there isn't a confusing name clash anymore!
* Move `tests.utils.setup_test_homeserver` to `tests.server`Olivier Wilkinson (reivilibre)2021-12-034-177/+188
| | | | | | | It had no users. We have just taken the identity of a previous function but don't provide the same behaviour, so we need to fix this in the next commit...
* Convert one of the `setup_test_homeserver`s to ↵Olivier Wilkinson (reivilibre)2021-12-031-8/+3
| | | | | | | | `make_test_homeserver_synchronous` and pass in the homeserver rather than calling a same-named function to ask for one. Later commits will jiggle things around to make this sensible.
* Clean up tests.storage.test_appservice (#11492)reivilibre2021-12-021-177/+168
|
* Clean up `tests.storage.test_main` to remove use of legacy code. (#11493)reivilibre2021-12-021-17/+10
|
* Clean up `tests.test_visibility` to remove legacy code. (#11495)reivilibre2021-12-021-202/+39
|
* Fix media repository failing when media store path contains symlinks (#11446)Sean Quah2021-12-021-1/+108
|
* Add type annotations to `tests.storage.test_appservice`. (#11488)reivilibre2021-12-021-48/+92
|
* Add a license header and comment. (#11479)Patrick Cloke2021-12-011-0/+15
|
* Don't start Synapse master process if `worker_app` is set (#11416)Shay2021-11-301-0/+31
| | | | | | | | | | | | | | | | | | | | | * Add check to catch syanpse master process starting when workers are configured * add test to verify that starting master process with worker config raises error * newsfragment * specify config.worker.worker_app in check * update test * report specific config option that triggered the error Co-authored-by: reivilibre <oliverw@matrix.org> * clarify error message Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* Bundle relations of relations into the `/relations` result. (#11284)Patrick Cloke2021-11-301-0/+118
| | | | | Per updates to MSC2675 which now states that bundled aggregations should be included from the `/relations` endpoint.
* Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)Sean Quah2021-11-301-0/+12
| | | | | | | | | | | When all entries in an `LruCache` have a size of 0 according to the provided `size_callback`, and `drop_from_cache` is called on a cache node, the node would be unlinked from the LRU linked list but remain in the cache dictionary. An assertion would be later be tripped due to the inconsistency. Avoid unintentionally calling `__len__` and use a strict `is None` check instead when unwrapping the weak reference.
* Remove unnecessary `json.dumps` from `tests.rest.admin` (#11461)Dirk Klimpel2021-11-301-41/+20
| | | | | The tests helpers automatically convert dictionaries to JSON payloads, no need to do it manually for each test.
* Add missing copyright header. (#11460)Patrick Cloke2021-11-301-0/+14
|
* Convert status codes to `HTTPStatus` in `tests.rest.admin` (#11455)Dirk Klimpel2021-11-3011-557/+886
|
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-29/+65
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-296-30/+104
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Fix perspectives requests for multiple keys for the same server (#11440)Richard van der Hoff2021-11-291-0/+71
| | | | | If we tried to request multiple keys for the same server, we would end up dropping some of those requests.
* Add a test case for the SendJoinParser (#11441)David Robertson2021-11-291-0/+50
| | | This would have caught the bug #11438 introduced in #11217 and fixed in #11439.
* Update MSC2918 refresh token support to confirm with the latest revision: ↵reivilibre2021-11-261-14/+44
| | | | accept the `refresh_tokens` parameter in the request body rather than in the URL parameters. (#11430)
* Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)Sean Quah2021-11-261-1/+5
| | | | Also refactor the stream ID trackers/generators a bit and try to document them better.
* Support expiry of refresh tokens and expiry of the overall session when ↵reivilibre2021-11-261-1/+124
| | | | refresh tokens are in use. (#11425)
* Track ongoing event fetches correctly (again) (#11376)Sean Quah2021-11-261-1/+138
| | | | | | | | | | | | | The previous fix for the ongoing event fetches counter (8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and incorrect. When the database is unreachable, `_do_fetch` never gets run and so `_event_fetch_ongoing` is never decremented. The previous fix also moved the `_event_fetch_ongoing` decrement outside of the `_event_fetch_lock` which allowed race conditions to corrupt the counter.
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-251-2/+2
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-242-12/+21
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Rename unstable `access_token_lifetime` configuration option to ↵reivilibre2021-11-231-1/+1
| | | | `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. (#11388)
* Add missing type hints to config base classes (#11377)Patrick Cloke2021-11-231-8/+14
|
* Remove code invalidated by deprecated config flag ↵Shay2021-11-231-1/+0
| | | | | | | | | | | | | | | | | | | | | 'trust_identity_servers_for_password_resets' (#11395) * remove background update code related to deprecated config flag * changelog entry * update changelog * Delete 11394.removal Duplicate, wrong number * add no-op background update and change newfragment so it will be consolidated with associated work * remove unused code * Remove code associated with deprecated flag from legacy docker dynamic config file Co-authored-by: reivilibre <oliverw@matrix.org>
* Merge branch 'master' into developSean Quah2021-11-232-0/+253
|\
| * Prevent the media store from writing outside of the configured directorySean Quah2021-11-192-0/+253
| | | | | | | | | | Also tighten validation of server names by forbidding invalid characters in IPv6 addresses and empty domain labels.
* | Add config for customizing the claim used for JWT logins. (#11361)Kostas2021-11-221-32/+36
| | | | | | | | | | Allows specifying a different claim (from the default "sub") to use when calculating the localpart of the Matrix ID used during the JWT login.
* | Store arbitrary relations from events. (#11391)Patrick Cloke2021-11-222-1/+117
| | | | | | | | | | Instead of only known relation types. This also reworks the background update for thread relations to crawl events and search for any relation type, not just threaded relations.
* | Add an admin API to run background jobs. (#11352)Dirk Klimpel2021-11-191-8/+146
| | | | | | | | | | | | Instead of having admins poke into the database directly. Can currently run jobs to populate stats and to populate the user directory.
* | Fix checking whether a room can be published on creation. (#11392)Patrick Cloke2021-11-191-33/+62
| | | | | | | | | | | | | | | | If `room_list_publication_rules` was configured with a rule with a non-wildcard alias and a room was created with an alias then an internal server error would have been thrown. This fixes the error and properly applies the publication rules during room creation.
* | Keep fallback key marked as used if it's re-uploaded (#11382)Hubert Chathi2021-11-191-1/+31
| |
* | Fix verification of objects signed with old local keys (#11379)Richard van der Hoff2021-11-191-3/+53
| | | | | | | | | | | | | | Fixes a bug introduced in #11129: objects signed by the local server, but with keys other than the current one, could not be successfully verified. We need to check the key id in the signature, and track down the right key.
* | Remove legacy code related to deprecated ↵Shay2021-11-181-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `trust_identity_server_for_password_resets` config flag (#11333) * remove code legacy code related to deprecated config flag "trust_identity_server_for_password_resets" from synapse/config/emailconfig.py * remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/config/registration.py * remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/handlers/identity.py * add tests to ensure config error is thrown and synapse refuses to start when depreciated config flag is found * add changelog * slightly change behavior to only check for deprecated flag if set to 'true' * Update changelog.d/11333.misc Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* | Add dedicated admin API for blocking a room (#11324)Dirk Klimpel2021-11-181-0/+228
| |
* | Do not allow MSC3440 threads to fork threads (#11161)Patrick Cloke2021-11-181-0/+62
| | | | | | | | | | | | | | | | | | | | | | Adds validation to the Client-Server API to ensure that the potential thread head does not relate to another event already. This results in not allowing a thread to "fork" into other threads. If the target event is unknown for some reason (maybe it isn't visible to your homeserver), but is the target of other events it is assumed that the thread can be created from it. Otherwise, it is rejected as an unknown event.
* | Rename `get_access_token_for_user_id` method to ↵reivilibre2021-11-173-11/+11
| | | | | | | | `create_access_token_for_user_id` (#11369)
* | Fix incorrect return value in tests. (#11359)Patrick Cloke2021-11-161-1/+1
| |
* | Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-161-1/+51
|\| | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-1/+51
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Avoid sharing room hierarchy responses between users (#11355)Sean Quah2021-11-161-0/+55
| | | | | | | | | | Different users may be allowed to see different rooms within a space, so sharing responses between users is inadvisable.
* | Add ability to un-shadow-ban via the admin API. (#11347)Patrick Cloke2021-11-161-6/+20
| |
* | Misc typing fixes for tests, part 2 of N (#11330)David Robertson2021-11-164-29/+65
| |
* | Database storage profile passes mypy (#11342)David Robertson2021-11-151-3/+6
| | | | | | | | | | | | It already seems to pass mypy. I wonder what changed, given that it was on the exclusion list. So this commit consists of me ensuring `--disallow-untyped-defs` passes and a minor fixup to a function that returned either `True` or `None`.
* | Add support for the stable version of MSC2778 (#11335)Tulir Asokan2021-11-152-2/+8
| | | | | | | | | | | | | | | | | | * Add support for the stable version of MSC2778 Signed-off-by: Tulir Asokan <tulir@maunium.net> * Expect m.login.application_service in login and password provider tests Signed-off-by: Tulir Asokan <tulir@maunium.net>
* | Test room alias deletion (#11327)David Robertson2021-11-121-28/+77
| | | | | | | | | | | | | | | | | | | | * Prefer `HTTPStatus` over plain `int` This is an Opinion that no-one has seemed to object to yet. * `--disallow-untyped-defs` for `tests.rest.client.test_directory` * Improve synapse's annotations for deleting aliases * Test case for deleting a room alias * Changelog
* | Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-121-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* | Misc typing fixes for `tests`, part 1 of N (#11323)David Robertson2021-11-124-27/+47
| | | | | | | | | | | | | | | | | | | | * Annotate HomeserverTestCase.servlets * Correct annotation of federation_auth_origin * Use AnyStr custom_headers instead of a Union This allows (str, str) and (bytes, bytes). This disallows (str, bytes) and (bytes, str) * DomainSpecificString.SIGIL is a ClassVar
* | Convert delete room admin API to async endpoint (#11223)Dirk Klimpel2021-11-122-65/+709
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Remove redundant parameters on `_check_event_auth` (#11292)Richard van der Hoff2021-11-101-2/+0
| | | | | | as of #11012, these parameters are unused.
* | Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-091-2/+6
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Allow admins to proactively block rooms (#11228)David Robertson2021-11-091-0/+28
| | | | | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-094-35/+438
|/ | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Include cross-signing signatures when syncing remote devices for the first ↵Erik Johnston2021-11-091-0/+151
| | | | | | | time (#11234) When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results. c.f. #11159
* Require body for read receipts with user-agent exceptions (#11157)rogersheu2021-11-091-2/+28
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Add some background update admin APIs (#11263)Erik Johnston2021-11-081-0/+218
| | | Fixes #11259
* Default value for `public_baseurl` (#11210)Richard van der Hoff2021-11-083-3/+1
| | | | | We might as well use a default value for `public_baseurl` based on `server_name` - in many cases, it will be correct.
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-051-0/+69
| | | Fixes #11252
* Additional test for `cachedList` (#11246)Richard van der Hoff2021-11-041-0/+43
| | | | | I was trying to understand how `cachedList` works, and ended up writing this extra test. I figure we may as well keep it.
* Add a linearizer on (appservice, stream) when handling ephemeral events. ↵Nick Barrett2021-11-031-0/+51
| | | | | (#11207) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Remove a debug statement from tests. (#11239)Patrick Cloke2021-11-031-1/+0
|
* Merge remote-tracking branch 'origin/master' into developErik Johnston2021-11-021-0/+15
|\
| * Fix URL preview errors when previewing XML documents. (#11196)Patrick Cloke2021-10-271-0/+15
| |
* | Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-021-0/+74
| |
* | Add search by room ID and room alias to List Room admin API (#11099)Dirk Klimpel2021-11-021-39/+49
| | | | | | | | Fixes: #10874 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | ObservableDeferred: run observers in order (#11229)Richard van der Hoff2021-11-022-4/+69
| |
* | Handle missing Content-Type header when accessing remote media (#11200)Shay2021-11-011-2/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add code to handle missing content-type header and a test to verify that it works * add handling for missing content-type in the /upload endpoint as well * slightly refactor test code to put private method in approriate place * handle possible null value for content-type when pulling from the local db * add changelog * refactor test and add code to handle missing content-type in cached remote media * requested changes * Update changelog.d/11200.bugfix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* | Fix a bug in unit test `test_block_room_and_not_purge` (#11226)Dirk Klimpel2021-11-011-2/+2
| |
* | Make `check_event_allowed` module API callback not fail open (accept events) ↵reivilibre2021-11-011-13/+3
| | | | | | | | when an exception is raised (#11033)
* | Remove deprecated delete room admin API (#11213)Dirk Klimpel2021-11-011-24/+15
| | | | | | | | | | Remove deprecated delete room admin API, `POST /_synapse/admin/v1/rooms/<room_id>/delete`
* | Support for serving server well-known files (#11211)Richard van der Hoff2021-11-011-6/+26
| | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/8308
* | Test that `ClientIpStore` combines database and in-memory data correctly ↵Sean Quah2021-11-011-0/+206
| | | | | | | | (#11179)
* | Add a module API method to retrieve state from a room (#11204)Brendan Abolivier2021-10-291-1/+24
| |
* | Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-282-2/+62
| | | | | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* | Fetch verify key locally rather than trying to do so over federation if ↵Shay2021-10-281-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | origin and host are the same. (#11129) * add tests for fetching key locally * add logic to check if origin server is same as host and fetch verify key locally rather than over federation * add changelog * slight refactor, add docstring, change changelog entry * Make changelog entry one line * remove verify_json_locally and push locality check to process_request, add function process_request_locally * remove leftover code reference * refactor to add common call to 'verify_json and associated handling code * add type hint to process_json * add some docstrings + very slight refactor
* | Add a ModuleApi method to update a user's membership in a room (#11147)Brendan Abolivier2021-10-281-1/+125
| | | | | | Co-authored-by: reivilibre <oliverw@matrix.org>
* | Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-272-0/+121
| | | | | | Fixes: #9346
* | Implement an `on_new_event` callback (#11126)Brendan Abolivier2021-10-261-3/+90
|/ | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Enable changing user type via users admin API (#11174)Jason Robinson2021-10-261-0/+51
| | | | | | | | Users admin API can now also modify user type in addition to allowing it to be set on user creation. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Don't set new room alias before potential 403 (#10930)AndrewFerr2021-10-251-1/+101
| | | | | | Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Fix synapse.config module "read" command (#11145)Jason Robinson2021-10-223-55/+104
| | | | | | | | | | `synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse. The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example. Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Add a thread relation type per MSC3440. (#11088)Patrick Cloke2021-10-211-6/+34
| | | | Adds experimental support for MSC3440's `io.element.thread` relation type (and the aggregation for it).
* Fix adding excluded users to the private room sharing tables when joining a ↵David Robertson2021-10-211-14/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | room (#11143) * We only need to fetch users in private rooms * Filter out `user_id` at the top * Discard excluded users in the top loop We weren't doing this in the "First, if they're our user" branch so this is a bugfix. * The caller must check that `user_id` is included This is in the docstring. There are two call sites: - one in `_handle_room_publicity_change`, which explicitly checks before calling; - and another in `_handle_room_membership_event`, which returns early if the user is excluded. So this change is safe. * Test joining a private room with an excluded user * Tweak an existing test * Changelog * test docstring * lint
* Fix setting a user's external_id via the admin API returns 500 and deletes ↵Dirk Klimpel2021-10-211-3/+212
| | | | | users existing external mappings if that external ID is already mapped (#11051) Fixes #10846
* Show error when timestamp in seconds is provided to the /purge_media_cache ↵Aaron R2021-10-201-4/+102
| | | | API (#11101)
* Merge branch 'master' into developSean Quah2021-10-201-34/+3
|\
| * Revert change to counting of deactivated users towards the monthly active ↵Sean Quah2021-10-201-34/+3
| | | | | | | | | | | | | | | | users limit (#11127) Temporarily revert "Add functionality to remove deactivated users from the monthly_active_users table (#10947)". This reverts commit eda8c88b84ee7506379a71ac2a7a88c08b759d43.
* | Check *all* auth events for room id and rejection (#11009)Richard van der Hoff2021-10-181-48/+90
| | | | | | | | | | | | | | | | | | | | | | This fixes a bug where we would accept an event whose `auth_events` include rejected events, if the rejected event was shadowed by another `auth_event` with same `(type, state_key)`. The approach is to pass a list of auth events into `check_auth_rules_for_event` instead of a dict, which of course means updating the call sites. This is an extension of #10956.
* | Don't remove local users from dir when the leave their last room (#11103)David Robertson2021-10-181-0/+50
| |
* | Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-152-79/+140
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Fix 500 error on `/messages` when we accumulate more than 5 backward ↵Eric Eastwood2021-10-141-0/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | extremities (#11027) Found while working on the Gitter backfill script and noticed it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390 When there are more than 5 backward extremities for a given depth, backfill will throw an error because we sliced the extremity list to 5 but then try to iterate over the full list. This causes us to look for state that we never fetched and we get a `KeyError`. Before when calling `/messages` when there are more than 5 backward extremities: ``` Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper callback_return = await self._async_render(request) File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render callback_return = await raw_callback_return File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET msgs = await self.pagination_handler.get_messages( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages await self.hs.get_federation_handler().maybe_backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill return await self._maybe_backfill_inner(room_id, current_depth, limit) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner likely_extremeties_domains = get_domains_from_state(states[e_id]) KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl' ```
* | Ensure each charset is attempted only once during media preview. (#11089)Patrick Cloke2021-10-141-8/+35
| | | | | | | | There's no point in trying more than once since it is guaranteed to continually fail.
* | Attempt different character encodings when previewing a URL. (#11077)Patrick Cloke2021-10-141-26/+40
| | | | | | | | This follows similar logic to BeautifulSoup where we attempt different character encodings until we find one which works.
* | Fix-up some type hints in the relations tests. (#11076)Patrick Cloke2021-10-143-51/+62
| |
* | Add a test for a workaround concerning the behaviour of third-party rule ↵reivilibre2021-10-141-6/+50
| | | | | | | | modules and `SynapseError`s. (#11071)
* | Merge remote-tracking branch 'origin/release-v1.45' into developDavid Robertson2021-10-132-8/+74
|\|
| * Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-134-10/+628
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-131-26/+197
| | | | | | | | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Be more lenient when parsing the version for oEmbed responses. (#11065)Patrick Cloke2021-10-131-0/+51
| |
* | Simplify the user admin API tests (#11048)Dirk Klimpel2021-10-121-255/+146
| |
* | Add tests for `MediaFilePaths` (#11057)Sean Quah2021-10-121-0/+238
| |
* | Merge remote-tracking branch 'origin/release-v1.45' into developErik Johnston2021-10-121-13/+11
|\|
| * Reset global cache state before cache tests. (#11036)Patrick Cloke2021-10-121-13/+11
| | | | | | | | | | | | This reverts #11019 and structures the code a bit more like it was before #10985. The global cache state must be reset before running the tests since other test cases might have configured caching (and thus touched the global state).
* | Fix inconsistent behavior of `get_last_client_by_ip` (#10970)Sean Quah2021-10-121-0/+43
| | | | | | | | | | | | | | Make `get_last_client_by_ip` return the same dictionary structure regardless of whether the data has been persisted to the database. This change will allow slightly cleaner type hints to be applied later on.
* | Add an approximate difference method to StateFilters (#10825)reivilibre2021-10-121-2/+511
|/
* Autodiscover oEmbed endpoint from returned HTML (#10822)Patrick Cloke2021-10-082-17/+123
| | | | | Searches the returned HTML for an oEmbed endpoint using the autodiscovery mechanism (`<link rel=...>`), and will request it to generate the preview.
* Fix overwriting profile when making room public (#11003)David Robertson2021-10-081-2/+69
| | | This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
* Include exception in json logging (#11028)Nick Barrett2021-10-081-0/+28
|
* Ensure each cache config test uses separate state. (#11019)Patrick Cloke2021-10-071-8/+12
| | | Hopefully this fixes these tests sometimes failing in CI.
* Don't alter directory entries for local users when setting a per-room ↵David Robertson2021-10-071-0/+34
| | | | | nickname (#11002) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add a spamchecker method to allow or deny 3pid invites (#10894)Brendan Abolivier2021-10-061-0/+70
| | | | | This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline. Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-066-65/+60
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Add a spamchecker callback to allow or deny room joins (#10910)Brendan Abolivier2021-10-061-0/+101
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-069-78/+61
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)David Robertson2021-10-051-16/+78
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two steps to rebuilding the user directory: 1. a scan over rooms, followed by 2. a scan over local users. The former reads avatars and display names from the `room_memberships` table and therefore contains potentially private avatars and display names. The latter reads from the the `profiles` table which only contains public data; moreover it will overwrite any private profiles that the rooms scan may have written to the user directory. This means that the rebuild could leak private user while the rebuild was in progress, only to later cover up the leaks once the rebuild had completed. This change skips over local users when writing user_directory rows when scanning rooms. Doing so means that it'll take longer for a rebuild to make local users searchable, which is unfortunate. I think a future PR can improve this by swapping the order of the two steps above. (And indeed there's more to do here, e.g. copying from `profiles` without going via Python.) Small tidy-ups while I'm here: * Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`. * Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`. * Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add functionality to remove deactivated users from the monthly_active_users ↵Hillery Shay2021-10-041-3/+34
| | | | | | | | | | | | | | | | | | | | | | | | | table (#10947) * add test * add function to remove user from monthly active table in deactivate code * add function to remove user from monthly active table * add changelog entry * update changelog number * requested changes * update docstring on new function * fix lint error * Update synapse/storage/databases/main/monthly_active_users.py Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Make is_public Optional[bool] for create_room_as test util (#10951) (#10963)AndrewFerr2021-10-041-6/+7
| | | Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Merge tag 'v1.44.0rc3' into developBrendan Abolivier2021-10-041-0/+34
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc3 (2021-10-04) ============================== Bugfixes -------- - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933)) - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
| * Fix error in `get_user_ip_and_agents` when fetching from the database (#10968)Sean Quah2021-10-011-0/+34
| |
| * Strip "join_authorised_via_users_server" from join events which do not need ↵Patrick Cloke2021-10-012-7/+9
| | | | | | | | | | | | | | | | it. (#10933) This fixes a "Event not signed by authorising server" error when transition room member from join -> join, e.g. when updating a display name or avatar URL for restricted rooms.
* | Consistently exclude from user_directory (#10960)David Robertson2021-10-044-26/+366
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-047-13/+13
| |
* | type-hint `HomeserverTestcase.setup_test_homeserver` (#10961)David Robertson2021-10-013-9/+20
| | | | | | | | | | * type-hint `HomeserverTestcase.setup_test_homeserver` For better IDE completion. A small drive-by.
* | Clean-up registration tests (#10945)Patrick Cloke2021-09-301-37/+52
| | | | | | | | Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent to properly configure auto-join rooms.
* | Strip "join_authorised_via_users_server" from join events which do not need ↵Patrick Cloke2021-09-302-7/+9
| | | | | | | | | | | | | | it. (#10933) This fixes a "Event not signed by authorising server" error when transition room member from join -> join, e.g. when updating a display name or avatar URL for restricted rooms.
* | Fix errors in Synapse logs from unit tests. (#10939)Patrick Cloke2021-09-304-26/+34
| | | | | | | | | | Fix some harmless errors from background processes (mostly due to awaiting Mock objects) that occurred in the Synapse logs during unit tests.
* | Pass str to twisted's IReactorTCP (#10895)David Robertson2021-09-302-6/+6
| | | | | | | | | | | | | | This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job. Until that change is in a twisted release, we'll have to ignore the type of the `host` argument. I've raised #10899 to remind us to review the issue in a few months' time.
* | Refactor user directory tests (#10935)David Robertson2021-09-303-199/+280
| | | | | | | | | | | | | | | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Split `event_auth.check` into two parts (#10940)Richard van der Hoff2021-09-291-72/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | Broadly, the existing `event_auth.check` function has two parts: * a validation section: checks that the event isn't too big, that it has the rught signatures, etc. This bit is independent of the rest of the state in the room, and so need only be done once for each event. * an auth section: ensures that the event is allowed, given the rest of the state in the room. This gets done multiple times, against various sets of room state, because it forms part of the state res algorithm. Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think that makes everything hard to follow. Instead, we split the function in two and call each part separately where it is needed.
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-2913-34/+34
| |
* | Implement MSC3069: Guest support on whoami (#9655)Travis Ralston2021-09-291-5/+44
| |
* | Fix empty `url_cache_thumbnails/yyyy-mm-dd/` directories being left behind ↵Sean Quah2021-09-291-0/+31
| | | | | | | | (#10924)
* | Inline `_check_event_auth` for outliers (#10926)Richard van der Hoff2021-09-281-1/+0
|/ | | | | | | | | | | | | | | | | | | | | | | | | | * Inline `_check_event_auth` for outliers When we are persisting an outlier, most of `_check_event_auth` is redundant: * `_update_auth_events_and_context_for_auth` does nothing, because the `input_auth_events` are (now) exactly the event's auth_events, which means that `missing_auth` is empty. * we don't care about soft-fail, kicking guest users or `send_on_behalf_of` for outliers ... so the only thing that matters is the auth itself, so let's just do that. * `_auth_and_persist_fetched_events_inner`: de-async `prep` `prep` no longer calls any `async` methods, so let's make it synchronous. * Simplify `_check_event_auth` We no longer need to support outliers here, which makes things rather simpler. * changelog * lint
* Avoid storing URL cache files in storage providers (#10911)Sean Quah2021-09-271-0/+130
| | | | | URL cache files are short-lived and it does not make sense to offload them (eg. to the cloud) or back them up.
* Add a spamchecker callback to allow or deny room creation based on invites ↵Brendan Abolivier2021-09-241-2/+117
| | | | | | | (#10898) This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline. This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-247-13/+15
|
* Factor out common code for persisting fetched auth events (#10896)Richard van der Hoff2021-09-241-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Factor more stuff out of `_get_events_and_persist` It turns out that the event-sorting algorithm in `_get_events_and_persist` is also useful in other circumstances. Here we move the current `_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`, and then factor the sorting part out to `_auth_and_persist_fetched_events`. * `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment `get_event_auth` returns events with the outlier flag already set, so this is redundant (though we need to update a test where `get_event_auth` is mocked). * `_get_remote_auth_chain_for_event`: move existing-event tests earlier Move a couple of tests outside the loop. This is a bit inefficient for now, but a future commit will make it better. It should be functionally identical. * `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events` We can use the same codepath for persisting the events fetched as part of an auth chain as for those fetched individually by `_get_events_and_persist` for building the state at a backwards extremity. * `_get_remote_auth_chain_for_event`: use a dict for efficiency `_auth_and_persist_fetched_events` sorts the events itself, so we no longer need to care about maintaining the ordering from `get_event_auth` (and no longer need to sort by depth in `get_event_auth`). That means that we can use a map, making it easier to filter out events we already have, etc. * changelog * `_auth_and_persist_fetched_events`: improve docstring
* Add reactor to `SynapseRequest` and fix up types. (#10868)Erik Johnston2021-09-2413-37/+69
|
* Fix AuthBlocking check when requester is appservice (#10881)Jason Robinson2021-09-241-0/+62
| | | | | If the MAU count had been reached, Synapse incorrectly blocked appservice users even though they've been explicitly configured not to be tracked (the default). This was due to bypassing the relevant if as it was chained behind another earlier hit if as an elif. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* Improve typing in user_directory files (#10891)David Robertson2021-09-241-2/+3
| | | | | | | | | | | * Improve typing in user_directory files This makes the user_directory.py in storage pass most of mypy's checks (including `no-untyped-defs`). Unfortunately that file is in the tangled web of Store class inheritance so doesn't pass mypy at the moment. The handlers directory has already been mypyed. Co-authored-by: reivilibre <olivier@librepush.net>
* Allow `.` and `~` chars in registration tokens (#10887)Callum Brown2021-09-231-3/+5
| | | | Per updates to MSC3231 in order to use the same grammar as other identifiers.
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-2310-20/+19
|
* Fix reactivated users not being added to the user directory (#10782)David Robertson2021-09-231-1/+41
| | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-234-12/+8
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-232-2/+2
|
* Treat "\u0000" as "\u0020" for the purposes of message search (message ↵Hillery Shay2021-09-221-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | indexing) (#10820) * add test to check if null code points are being inserted * add logic to detect and replace null code points before insertion into db * lints * add license to test * change approach to null substitution * add type hint for SearchEntry * Add changelog entry Signed-off-by: H.Shay <shaysquared@gmail.com> * updated changelog * update chanelog message * remove duplicate changelog * Update synapse/storage/databases/main/events.py remove extra space Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * rename and move test file, update tests, delete old test file * fix typo in comments * update _find_highlights_in_postgres to replace null byte with space * replace null byte in sqlite search insertion * beef up and reorganize test for this pr * update changelog * add type hints and update docstring * check db engine directly vs using env variable * refactor tests to be less repetetive * move rplace logic into seperate function * requested changes * Fix typo. * Update synapse/storage/databases/main/search.py Co-authored-by: reivilibre <olivier@librepush.net> * Update changelog.d/10820.misc Co-authored-by: Aaron Raimist <aaron@raim.ist> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Aaron Raimist <aaron@raim.ist>
* Include more information in oEmbed previews. (#10819)Patrick Cloke2021-09-221-9/+21
| | | | | | | * Improved titles (fall back to the author name if there's not title) and include the site name. * Handle photo/video payloads. * Include the original URL in the Open Graph response. * Fix the expiration time (by properly converting from seconds to milliseconds).
* Extend ModuleApi with the methods we'll need to reject spam based on …IP - ↵David Teller2021-09-221-0/+72
| | | | | | | | resolves #10832 (#10833) Extend ModuleApi with the methods we'll need to reject spam based on IP - resolves #10832 Signed-off-by: David Teller <davidt@element.io>
* Add type hints for event streams. (#10856)Patrick Cloke2021-09-214-13/+55
|
* Refactor oEmbed previews (#10814)Patrick Cloke2021-09-211-13/+13
| | | | | | | | | | | | | The major change is moving the decision of whether to use oEmbed further up the call-stack. This reverts the _download_url method to being a "dumb" functionwhich takes a single URL and downloads it (as it was before #7920). This also makes more minor refactorings: * Renames internal variables for clarity. * Factors out shared code between the HTML and rich oEmbed previews. * Fixes tests to preview an oEmbed image.
* Test that state events sent by modules correctly end up in the room's state ↵Brendan Abolivier2021-09-211-0/+84
| | | | | | | (#10835) Test for #10830 Ideally the test would also make sure the new state event comes down sync, but this is probably good enough.
* Allow Synapse Admin API's Room Search to accept non-ASCII characters (#10859)Hillery Shay2021-09-211-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add tests for checking if room search works with non-ascii char * change encoding on parse_string to UTF-8 * lints * properly encode search term * lints * add changelog file * update changelog number * set changelog entry filetype to .bugfix * Revert "set changelog entry filetype to .bugfix" This reverts commit be8e5a314251438ec4ec7dbc59ba32162c93e550. * update changelog message and file type * change parse_string default encoding back to ascii and update room search admin api calll to parse string * refactor tests * Update tests/rest/admin/test_room.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Always add local users to the user directory (#10796)David Robertson2021-09-212-8/+11
| | | | | | | | | | | | | | | | | | | | | | | It's a simplification, but one that'll help make the user directory logic easier to follow with the other changes upcoming. It's not strictly required for those changes, but this will help simplify the resulting logic that listens for `m.room.member` events and generally make the logic easier to follow. This means the config option `search_all_users` ends up controlling the search query only, and not the data we store. The cost of doing so is an extra row in the `user_directory` and `user_directory_search` tables for each local user which - belongs to no public rooms - belongs to no private rooms of size ≥ 2 I think the cost of this will be marginal (since they'll already have entries in `users` and `profiles` anyway). As a small upside, a homeserver whose directory was built with this change can toggle `search_all_users` without having to rebuild their directory. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Allow sending a membership event to unban a user (#10807)Aaron Raimist2021-09-212-1/+97
| | | | | * Allow membership event to unban user Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Create a constant for a small png image in tests. (#10834)Patrick Cloke2021-09-167-94/+44
| | | To avoid duplicating it between a few tests.
* Make StateFilter frozen so we can hash it (#10816)reivilibre2021-09-141-17/+29
| | | Also enables Mypy for related tests.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-6/+12
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Add types to synapse.util. (#10601)reivilibre2021-09-101-2/+2
|
* Handle room upgrades for spaces (#10774)Patrick Cloke2021-09-101-4/+63
| | | | | By copying the `room_type` field of the create event and migrating any non-empty `m.space.child` events to the new room that is created.
* Easy refactors of the user directory (#10789)David Robertson2021-09-101-3/+3
| | | No functional changes here. This came out as I was working to tackle #5677
* Remove unstable MSC2858 API, including `experimental.msc2858_enabled` config ↵Sean2021-09-091-58/+7
| | | | | option (#10693) Signed-off-by: Sean Quah <seanq@element.io>
* Get rid of `_auth_and_persist_event` (#10781)Richard van der Hoff2021-09-081-3/+12
| | | This is only called in two places, and the code seems much clearer without it.
* Skip handling of push actions for outlier events (#10780)Erik Johnston2021-09-081-0/+1
| | | | Outlier events don't ever have push actions associated with them, so we can skip some expensive queries during event persistence.
* Request JSON for oEmbed requests (and ignore XML only providers). (#10759)Patrick Cloke2021-09-081-1/+54
| | | | | | | | This adds the format to the request arguments / URL to ensure that JSON data is returned (which is all that Synapse supports). This also adds additional error checking / filtering to the configuration file to ignore XML-only providers.
* Merge tag 'v1.42.0rc2' into developBrendan Abolivier2021-09-061-1/+45
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.42.0rc2 (2021-09-06) ============================== This version of Synapse removes deprecated room-management admin APIs, removes out-of-date email pushers, and improves error handling for fallback templates for user-interactive authentication. For more information on these points, server administrators are encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375). ([\#10747](https://github.com/matrix-org/synapse/issues/10747)) Internal Changes ---------------- - Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768))
| * Support MSC3375: room version 9. (#10747)Patrick Cloke2021-09-031-1/+45
| |
* | Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-061-4/+4
| |
* | Pull out encrypted_by_default tests from user_directory tests (#10752)David Robertson2021-09-062-95/+109
| |
* | Ignore rooms with unknown room versions in the spaces summary. (#10727)Patrick Cloke2021-09-011-0/+25
| | | | | | | | This avoids breaking the entire endpoint if a room with an unsupported room version is encountered.
* | Consider the `origin_server_ts` of the `m.space.child` event when ordering ↵Patrick Cloke2021-09-011-5/+13
| | | | | | | | | | | | | | | | | | rooms. (#10730) This updates the ordering of the returned events from the spaces summary API to that defined in MSC2946 (which updates MSC1772). Previously a step was skipped causing ordering to be inconsistent with clients.
* | Populate `rooms.creator` field for easy lookup (#10697)Eric Eastwood2021-09-011-0/+98
| | | | | | | | | | | | Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values
* | Merge tag 'v1.42.0rc1' into developOlivier Wilkinson (reivilibre)2021-09-011-0/+44
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.42.0rc1 (2021-09-01) ============================== Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142)) - Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452)) - Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524)) - Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725)) Bugfixes -------- - Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232)) - Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561)) - Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734)) - Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593)) - Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644)) - Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654)) - Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677)) - Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679)) - Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684)) - Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703)) - Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713)) Improved Documentation ---------------------- - Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192)) - Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595)) - Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639)) - Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708)) - Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711)) Deprecations and Removals ------------------------- - Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830)) Internal Changes ---------------- - Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608)) - Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640)) - Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621)) - Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627)) - Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642)) - Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630)) - Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645)) - Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647)) - Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672)) - Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662)) - Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664)) - Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674)) - Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667)) - Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686)) - Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692)) - Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))
| * Fix iteration in _remove_deleted_email_pushers background job. (#10734)Andrew Morgan2021-09-011-0/+44
| |
* | Correctly include room avatars in email notifications (#10658)David Robertson2021-09-011-5/+47
| | | | | | | | | | | | | | | | Judging by the template, this was intended ages ago, but we never actually passed an avatar URL to the template. So let's provide one. Closes #1546. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Allow configuration of the oEmbed URLs. (#10714)Patrick Cloke2021-08-311-110/+102
|/ | | | | This adds configuration options (under an `oembed` section) to configure which URLs are matched to use oEmbed for URL previews.
* Merge branch 'master' into developRichard van der Hoff2021-08-312-1/+139
|\
| * Fix up unit tests (#10723)Richard van der Hoff2021-08-311-2/+15
| | | | | | These were broken in an incorrect merge of GHSA-jj53-8fmw-f2w2 (cb35df9)
| * Merge pull request from GHSA-jj53-8fmw-f2w2reivilibre2021-08-311-0/+43
| |
| * Merge pull request from GHSA-3x4c-pq33-4w3qreivilibre2021-08-311-1/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add some tests to characterise the problem Some failing. Current states: RoomsMemberListTestCase test_get_member_list ... [OK] test_get_member_list_mixed_memberships ... [OK] test_get_member_list_no_permission ... [OK] test_get_member_list_no_permission_former_member ... [OK] test_get_member_list_no_permission_former_member_with_at_token ... [FAIL] test_get_member_list_no_room ... [OK] test_get_member_list_no_permission_with_at_token ... [FAIL] * Correct the tests * Check user is/was member before divulging room membership * Pull out only the 1 membership event we want. * Update tests/rest/client/v1/test_rooms.py Co-authored-by: Erik Johnston <erik@matrix.org> * Fixup tests (following apply review suggestion) Co-authored-by: Erik Johnston <erik@matrix.org>
* | Merge remote-tracking branch 'origin/release-v1.41' into developRichard van der Hoff2021-08-272-3/+124
|\|
| * Fix incompatibility with Twisted < 21. (#10713)Richard van der Hoff2021-08-272-3/+124
| | | | | | | | | | | | | | Turns out that the functionality added in #10546 to skip TLS was incompatible with older Twisted versions, so we need to be a bit more inventive. Also, add a test to (hopefully) not break this in future. Sadly, testing TLS is really hard.
* | Improve ServerNoticeServlet to avoid duplicate requests (#10679)Dirk Klimpel2021-08-271-0/+450
| | | | | | Fixes: #9544
* | Split `FederationHandler` in half (#10692)Richard van der Hoff2021-08-265-11/+23
| | | | | | The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
* | Validate new m.room.power_levels events (#10232)Aaron Raimist2021-08-261-0/+78
| | | | | | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* | Remove pushers when deleting 3pid from account (#10581)Azrenbeth2021-08-261-0/+39
| | | | | | | | | | When a user deletes an email from their account it will now also remove all pushers for that email and that user (even if these pushers were created by a different client)
* | Fix error when selecting between thumbnails with the same quality (#10684)Sean2021-08-251-1/+38
| | | | | | Fixes #10318
* | Merge branch 'master' into developErik Johnston2021-08-241-2/+2
|\|
| * Set room version 8 as preferred for restricted rooms. (#10571)Patrick Cloke2021-08-201-2/+2
| |
* | Implement MSC3231: Token authenticated registration (#10142)Callum Brown2021-08-212-0/+1144
| | | | | | | | | | Signed-off-by: Callum Brown <callum@calcuode.com> This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
* | Flatten tests/rest/client/{v1,v2_alpha} too (#10667)David Robertson2021-08-2025-16/+3
| |
* | Simplify tests for the device admin rest API. (#10664)Dirk Klimpel2021-08-201-81/+18
| | | | | | | | By replacing duplicated code with parameterized tests and avoiding unnecessary dumping of JSON data.
* | Follow-up: format changelog, add licence (#10593)David Robertson2021-08-201-0/+14
| | | | | | | | Merged before approval; these comments from @clokep on that PR.
* | Validate device_keys for C-S /keys/query requests (#10593)David Robertson2021-08-201-0/+77
| | | | | | | | | | | | | | | | * Validate device_keys for C-S /keys/query requests Closes #10354 A small, not particularly critical fix. I'm interested in seeing if we can find a more systematic approach though. #8445 is the place for any discussion.
* | Split `on_receive_pdu` in half (#10640)Richard van der Hoff2021-08-191-8/+2
| | | | | | Here we split on_receive_pdu into two functions (on_receive_pdu and process_pulled_event), rather than having both cases in the same method. There's a tiny bit of overlap, but not that much.
* | Do not include rooms with an unknown room version in a sync response. (#10644)Patrick Cloke2021-08-192-6/+132
| | | | | | | | A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.
* | Support MSC3283: Expose `enable_set_displayname` in capabilities (#10452)Dirk Klimpel2021-08-191-23/+86
| |
* | Remove not needed database updates in modify user admin API (#10627)Dirk Klimpel2021-08-191-4/+58
| |
* | Convert room member storage tuples to attrs. (#10629)Patrick Cloke2021-08-181-3/+6
| | | | | | | | Instead of using namedtuples. This helps with asserting type hints and code completion.
* | Remove deprecated Shutdown Room and Purge Room Admin API (#8830)Dirk Klimpel2021-08-171-162/+0
| |
* | Port the PresenceRouter module interface to the new generic interface (#10524)Azrenbeth2021-08-171-5/+104
|/ | | Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-1775-103/+82
|
* Allow to edit `external_ids` by Edit User admin API (#10598)Dirk Klimpel2021-08-171-30/+197
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Allow using several custom template directories (#10587)Brendan Abolivier2021-08-171-3/+61
| | | Allow using several directories in read_templates.
* Add an admin API to check if a username is available (#10578)Will Hunt2021-08-171-0/+62
| | | This adds a new API GET /_synapse/admin/v1/username_available?username=foo to check if a username is available. It is the counterpart to https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available, except that it works even if registration is disabled.
* Experimental support for MSC3266 Room Summary API. (#10394)Michael Telatynski2021-08-161-15/+93
|
* Support federation in the new spaces summary API (MSC2946). (#10569)Patrick Cloke2021-08-161-123/+169
|
* Handle string read receipt data (#10606)Šimon Brandner2021-08-161-0/+23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Test that we handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add changelog for #10606 Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add docs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Ignore malformed RRs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Only surround hidden = ... Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Remove unnecessary argument Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Update changelog.d/10606.bugfix Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Update the pagination parameter name based on MSC2946 review. (#10579)Patrick Cloke2021-08-111-7/+7
|
* Admin API to delete media for a specific user (#10558)Dirk Klimpel2021-08-111-113/+208
|