summary refs log tree commit diff
path: root/tests (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-013-68/+348
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add type hints to `tests/rest/admin` (#11851)Dirk Klimpel2022-01-313-229/+183
|
* Configurable limits on avatars (#11846)Brendan Abolivier2022-01-282-2/+248
| | | | | | Only allow files which file size and content types match configured limits to be set as avatar. Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
* Add a module callback to set username at registration (#11790)Brendan Abolivier2022-01-261-2/+77
| | | | | | This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step). This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not. Co-authored-by: David Robertson <davidr@element.io>
* Improvements to bundling aggregations. (#11815)Patrick Cloke2022-01-261-1/+1
| | | | | | | | | | | This is some odds and ends found during the review of #11791 and while continuing to work in this code: * Return attrs classes instead of dictionaries from some methods to improve type safety. * Call `get_bundled_aggregations` fewer times. * Adds a missing assertion in the tests. * Do not return empty bundled aggregations for an event (preferring to not include the bundle at all, as the docstring states).
* Add a config flag to inhibit `M_USER_IN_USE` during registration (#11743)Brendan Abolivier2022-01-261-0/+41
| | | | | | | This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work). This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique. More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
* Add admin API to get a list of federated rooms (#11658)Dirk Klimpel2022-01-251-25/+277
|
* Add admin API to reset connection timeouts for remote server (#11639)Dirk Klimpel2022-01-251-4/+51
| | | * Fix get federation status of destination if no error occured
* Support rendering previews with data: URLs in them (#11767)Patrick Cloke2022-01-243-10/+107
| | | | | Images which are data URLs will no longer break URL previews and will properly be "downloaded" and thumbnailed.
* Merge tag 'v1.51.0rc2' into developAndrew Morgan2022-01-241-6/+24
|\ | | | | | | | | | | | | | | | | | | Synapse 1.51.0rc2 (2022-01-24) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
| * Fix logic for dropping old events in fed queue (#11806)Andrew Morgan2022-01-241-6/+24
| | | | | | | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Richard van der Hoff <richard@matrix.org>
| * Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke2022-01-211-35/+73
| |
* | Remove account data (including client config, push rules and ignored users) ↵reivilibre2022-01-241-0/+219
| | | | | | | | | | upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff2022-01-211-1/+4
| | | | | | | | ... and start populating them for new events
* | Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke2022-01-211-35/+73
| |
* | Drop unused table `public_room_list_stream`. (#11795)Richard van der Hoff2022-01-211-1/+0
| | | | | | This is a follow-up to #10565.
* | Make the `get_global_account_data_by_type_for_user` cache be a tree-cache ↵reivilibre2022-01-211-2/+2
|/ | | | whose key is prefixed with the user ID (#11788)
* Correctly await on_logged_out callbacks (#11786)Brendan Abolivier2022-01-201-1/+27
|
* Add a regression test for using both webclient and client resources ↵Andrew Morgan2022-01-201-0/+108
| | | | simultaneously (#11765)
* Fix a bug that corrupted the cache of federated space hierarchies (#11775)Sean Quah2022-01-201-2/+90
| | | | `FederationClient.get_room_hierarchy()` caches its return values, so refactor the code to avoid modifying the returned room summary.
* Include whether the requesting user has participated in a thread. (#11577)Patrick Cloke2022-01-181-0/+3
| | | | | | Per updates to MSC3440. This is implement as a separate method since it needs to be cached on a per-user basis, instead of a per-thread basis.
* Make pagination of rooms in admin api stable (#11737)Daniel Sonck2022-01-171-19/+28
| | | | | | | | | | | | | | Always add state.room_id after the configurable ORDER BY. Otherwise, for any sort, certain pages can contain results from other pages. (Especially when sorting by creator, since there may be many rooms by the same creator) * Document different order direction of numerical fields "joined_members", "joined_local_members", "version" and "state_events" are ordered in descending direction by default (dir=f). Added a note in tests to explain the differences in ordering. Signed-off-by: Daniël Sonck <daniel@sonck.nl>
* Merge branch 'release-v1.50' into developOlivier Wilkinson (reivilibre)2022-01-141-1/+159
|\
| * Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could ↵reivilibre2022-01-131-1/+111
| | | | | | | | | | fail because too many EDUs were produced for device updates. (#11730) Co-authored-by: David Robertson <davidr@element.io>
| * Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not ↵reivilibre2022-01-121-1/+49
| | | | | | | | | | be sent to remote homeservers if there were too many to send at once. (#11729) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
| * Deal with mypy errors w/ type-hinted pynacl 1.5.0 (#11714)David Robertson2022-01-121-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | * Deal with mypy errors w/ type-hinted pynacl 1.5.0 Fixes #11644. I really don't like that we're monkey patching pynacl SignedKey instances with alg and version objects. But I'm too scared to make the changes necessary right now. (Ideally I would replace `signedjson.types.SingingKey` with a runtime class which wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
* | Remove the 'password_hash' from the Users Admin API endpoint response ↵Andrew Morgan2022-01-141-17/+33
| | | | | | | | dictionary (#11576)
* | Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke2022-01-132-21/+20
| | | | | | | | This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
* | Include bundled aggregations in the sync response cache. (#11659)Patrick Cloke2022-01-131-5/+5
| |
* | Allow tracking puppeted users for MAU (#11561)Jason Robinson2022-01-121-0/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently when puppeting another user, the user doing the puppeting is tracked for client IPs and MAU (if configured). When tracking MAU is important, it becomes necessary to be possible to also track the client IPs and MAU of puppeted users. As an example a client that manages user creation and creation of tokens via the Synapse admin API, passing those tokens for the client to use. This PR adds optional configuration to enable tracking of puppeted users into monthly active users. The default behaviour stays the same. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Deal with mypy errors w/ type-hinted pynacl 1.5.0 (#11714)David Robertson2022-01-101-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | * Deal with mypy errors w/ type-hinted pynacl 1.5.0 Fixes #11644. I really don't like that we're monkey patching pynacl SignedKey instances with alg and version objects. But I'm too scared to make the changes necessary right now. (Ideally I would replace `signedjson.types.SingingKey` with a runtime class which wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
* | Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)Patrick Cloke2022-01-071-0/+32
| | | | | | | | | | | | | | By returning all of the m.space.child state of the space, not just the first 50. The number of rooms returned is still capped at 50. For the federation API this implies that the requesting server will need to individually query for any other rooms it is not joined to.
* | Optionally use an on-disk sqlite db in tests (#11702)David Robertson2022-01-072-1/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Optionally use an on-disk sqlite db in tests When debugging a test it is sometimes useful to inspect the state of the DB. This is not easy when the db is in-memory: one cannot attach the sqlite CLI to another process's DB. With this change, if SYNAPSE_TEST_PERSIST_SQLITE_DB is set, we use `_trial_temp/test.db` as our sqlite database. One can then use `sqlite3 _trial_temp/test.db` and query to your heart's content. The DB is destroyed and recreated between different test cases. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Test that bans win a join against a race when computing `/sync` response ↵David Robertson2022-01-072-3/+104
| | | | | | | | (#11701)
* | Bundle aggregations outside of the serialization method. (#11612)Patrick Cloke2022-01-071-1/+1
| | | | | | | | | | | | | | | | This makes the serialization of events synchronous (and it no longer access the database), but we must manually calculate and provide the bundled aggregations. Overall this should cause no change in behavior, but is prep work for other improvements.
* | Remove the /send_relation endpoint. (#11682)Patrick Cloke2022-01-061-13/+13
| | | | | | | | This was removed from MSC2674 before that was approved and is not used by any known clients.
* | Strip unauthorized fields from `unsigned` object in events received over ↵Shay2022-01-061-0/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | federation (#11530) * add some tests to verify we are stripping unauthorized fields out of unsigned * add function to strip unauthorized fields from the unsigned object of event * newsfragment * update newsfragment number * add check to on_send_membership_event * refactor tests * fix lint error * slightly refactor tests and add some comments * slight refactor * refactor tests * fix import error * slight refactor * remove unsigned filtration code from synapse/handlers/federation_event.py * lint * move unsigned filtering code to event base * refactor tests * update newsfragment * requested changes * remove unused retun values
* | Fix get federation status of destination if no error occured (#11593)Dirk Klimpel2022-01-051-18/+57
| |
* | Cache empty responses from `/user/devices` (#11587)David Robertson2022-01-052-2/+98
|/ | | If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
* Refactor the way we set `outlier` (#11634)Richard van der Hoff2022-01-051-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * `_auth_and_persist_outliers`: mark persisted events as outliers Mark any events that get persisted via `_auth_and_persist_outliers` as, well, outliers. Currently this will be a no-op as everything will already be flagged as an outlier, but I'm going to change that. * `process_remote_join`: stop flagging as outlier The events are now flagged as outliers later on, by `_auth_and_persist_outliers`. * `send_join`: remove `outlier=True` The events created here are returned in the result of `send_join` to `FederationHandler.do_invite_join`. From there they are passed into `FederationEventHandler.process_remote_join`, which passes them to `_auth_and_persist_outliers`... which sets the `outlier` flag. * `get_event_auth`: remove `outlier=True` stop flagging the events returned by `get_event_auth` as outliers. This method is only called by `_get_remote_auth_chain_for_event`, which passes the results into `_auth_and_persist_outliers`, which will flag them as outliers. * `_get_remote_auth_chain_for_event`: remove `outlier=True` we pass all the events into `_auth_and_persist_outliers`, which will now flag the events as outliers. * `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter This param is now never set to True, so we can remove it. * `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param This is no longer set anywhere, so we can remove it. * `get_pdu`: remove unused `outlier` parameter ... and chase it down into `get_pdu_from_destination_raw`. * `event_from_pdu_json`: remove redundant `outlier` param This is never set to `True`, so can be removed. * changelog * update docstring
* Add admin API to get users' account data (#11664)Dirk Klimpel2022-01-051-0/+90
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre2022-01-051-59/+0
| | | | | #11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
* Include the topic event in the prejoin state, per MSC3173. (#11666)Patrick Cloke2022-01-041-0/+9
| | | | Invites and knocks will now include the topic in the stripped state send to clients before joining the room.
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-1/+5
| | | To improve type hints throughout the code.
* Fix a type annotation in `test_account_data.py` and remove it from the Mypy ↵reivilibre2021-12-291-2/+2
| | | | | exclusion list. (#11657) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Refactor `tests.util.setup_test_homeserver` and ↵reivilibre2021-12-214-185/+194
| | | | `tests.server.setup_test_homeserver`. (#11503)
* Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-212-8/+11
|
* Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff2021-12-201-11/+34
| | | This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
* Merge remote-tracking branch 'origin/release-v1.49' into developRichard van der Hoff2021-12-201-5/+5
|\
| * Disable aggregation bundling on `/sync` responses (#11583)Richard van der Hoff2021-12-201-5/+5
| | | | | | | | | | | | | | | | | | | | | | * Disable aggregation bundling on `/sync` responses A partial revert of #11478. This turns out to have had a significant CPU impact on initial-sync handling. For now, let's disable it, until we find a more efficient way of achieving this. * Fix tests. Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* | Use mock from standard library (#11588)V024602021-12-201-9/+8
| | | | | | Instead of the backported version.
* | Add type hints to `synapse/tests/rest/admin` (#11590)Dirk Klimpel2021-12-165-57/+70
| |
* | Add experimental support for MSC3202: allowing application services to ↵reivilibre2021-12-151-0/+64
| | | | | | | | masquerade as specific devices. (#11538)
* | Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah2021-12-141-1/+3
| |
* | Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-141-35/+0
| |
* | Add missing type hints to `synapse.appservice` (#11360)Patrick Cloke2021-12-141-8/+3
| |
* | Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah2021-12-141-1/+1
| | | | | | | | | | `COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
* | Move HTML parsing to a separate file for URL previews. (#11566)Patrick Cloke2021-12-132-23/+24
| | | | | | | | | | | | | | * Splits the logic for parsing HTML from the resource handling code. * Fix a circular import in the oEmbed code (which uses the HTML parsing code). * Renames some of the HTML parsing methods to: * Make it clear which methods are "internal" to the module. * Clarify what the methods do.
* | Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-101-0/+103
| | | | | | | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* | Test to ensure we share the same `state_group` across the whole historical ↵Eric Eastwood2021-12-101-0/+180
| | | | | | | | | | | | | | | | batch (MSC2716) (#11487) Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 We did some work on making sure the `state_groups` were shared in https://github.com/matrix-org/synapse/pull/10975
* | Ensure emails are canonicalized before fetching associated user. (#11547)Patrick Cloke2021-12-101-1/+2
| | | | | | | | This should fix pushers with an email in non-canonical form is used as the pushkey.
* | Do not allow cross-room relations, per MSC2674. (#11516)Patrick Cloke2021-12-091-0/+115
| |
* | Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi2021-12-091-5/+25
| |
* | Add missing `errcode` to `parse_string` and `parse_boolean` (#11542)Dirk Klimpel2021-12-094-10/+10
| |
* | Clean up `synapse.rest.admin` (#11535)Dirk Klimpel2021-12-081-1/+1
| |
* | Use HTTPStatus constants in place of literals in ↵reivilibre2021-12-081-47/+87
| | | | | | | | `tests.rest.client.test_auth`. (#11520)
* | Send and handle cross-signing messages using the stable prefix. (#10520)Hubert Chathi2021-12-081-2/+3
| |
* | Fix case in `wait_for_background_updates` where `self.store` does not exist ↵Andrew Morgan2021-12-071-7/+4
|/ | | | | | (#11331) Pull the DataStore from the HomeServer instance, which always exists.
* Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah2021-12-071-0/+59
| | | | | | `matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
* Fix 'delete room' admin api to work on incomplete rooms (#11523)Richard van der Hoff2021-12-071-17/+25
| | | | | If, for some reason, we don't have the create event, we should still be able to purge a room.
* Fix the test breakage introduced by #11435 as a result of concurrent PRs ↵reivilibre2021-12-071-1/+1
| | | | (#11522)
* Stabilise support for MSC2918 refresh tokens as they have now been merged ↵reivilibre2021-12-061-15/+15
| | | | into the Matrix specification. (#11435)
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-064-33/+188
| | | As a step towards allowing back-channel logout for OIDC.
* Add admin API to get some information about federation status (#11407)Dirk Klimpel2021-12-061-0/+456
|
* Include bundled aggregations in /sync and related fixes (#11478)Patrick Cloke2021-12-061-39/+96
| | | | | | | | Due to updates to MSC2675 this includes a few fixes: * Include bundled aggregations for /sync. * Do not include bundled aggregations for /initialSync and /events. * Do not bundle aggregations for state events. * Clarifies comments and variable names.
* Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)Sean Quah2021-12-061-59/+0
|
* Support configuring the lifetime of non-refreshable access tokens separately ↵reivilibre2021-12-032-0/+154
| | | | to refreshable access tokens. (#11445)
* Add type hints to `synapse/tests/rest/admin` (#11501)Dirk Klimpel2021-12-039-225/+256
|
* Revert accidental commits to develop.Olivier Wilkinson (reivilibre)2021-12-034-194/+185
|
* Give `tests.server.setup_test_homeserver` (nominally!) the same behaviourOlivier Wilkinson (reivilibre)2021-12-031-0/+3
| | | | | | | | | | by calling into `make_test_homeserver_synchronous`. The function *could* have been inlined at this point but the function is big enough and it felt fine to leave it as is. At least there isn't a confusing name clash anymore!
* Move `tests.utils.setup_test_homeserver` to `tests.server`Olivier Wilkinson (reivilibre)2021-12-034-177/+188
| | | | | | | It had no users. We have just taken the identity of a previous function but don't provide the same behaviour, so we need to fix this in the next commit...
* Convert one of the `setup_test_homeserver`s to ↵Olivier Wilkinson (reivilibre)2021-12-031-8/+3
| | | | | | | | `make_test_homeserver_synchronous` and pass in the homeserver rather than calling a same-named function to ask for one. Later commits will jiggle things around to make this sensible.
* Clean up tests.storage.test_appservice (#11492)reivilibre2021-12-021-177/+168
|
* Clean up `tests.storage.test_main` to remove use of legacy code. (#11493)reivilibre2021-12-021-17/+10
|
* Clean up `tests.test_visibility` to remove legacy code. (#11495)reivilibre2021-12-021-202/+39
|
* Fix media repository failing when media store path contains symlinks (#11446)Sean Quah2021-12-021-1/+108
|
* Add type annotations to `tests.storage.test_appservice`. (#11488)reivilibre2021-12-021-48/+92
|
* Add a license header and comment. (#11479)Patrick Cloke2021-12-011-0/+15
|
* Don't start Synapse master process if `worker_app` is set (#11416)Shay2021-11-301-0/+31
| | | | | | | | | | | | | | | | | | | | | * Add check to catch syanpse master process starting when workers are configured * add test to verify that starting master process with worker config raises error * newsfragment * specify config.worker.worker_app in check * update test * report specific config option that triggered the error Co-authored-by: reivilibre <oliverw@matrix.org> * clarify error message Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* Bundle relations of relations into the `/relations` result. (#11284)Patrick Cloke2021-11-301-0/+118
| | | | | Per updates to MSC2675 which now states that bundled aggregations should be included from the `/relations` endpoint.
* Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)Sean Quah2021-11-301-0/+12
| | | | | | | | | | | When all entries in an `LruCache` have a size of 0 according to the provided `size_callback`, and `drop_from_cache` is called on a cache node, the node would be unlinked from the LRU linked list but remain in the cache dictionary. An assertion would be later be tripped due to the inconsistency. Avoid unintentionally calling `__len__` and use a strict `is None` check instead when unwrapping the weak reference.
* Remove unnecessary `json.dumps` from `tests.rest.admin` (#11461)Dirk Klimpel2021-11-301-41/+20
| | | | | The tests helpers automatically convert dictionaries to JSON payloads, no need to do it manually for each test.
* Add missing copyright header. (#11460)Patrick Cloke2021-11-301-0/+14
|
* Convert status codes to `HTTPStatus` in `tests.rest.admin` (#11455)Dirk Klimpel2021-11-3011-557/+886
|
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-29/+65
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-296-30/+104
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Fix perspectives requests for multiple keys for the same server (#11440)Richard van der Hoff2021-11-291-0/+71
| | | | | If we tried to request multiple keys for the same server, we would end up dropping some of those requests.
* Add a test case for the SendJoinParser (#11441)David Robertson2021-11-291-0/+50
| | | This would have caught the bug #11438 introduced in #11217 and fixed in #11439.
* Update MSC2918 refresh token support to confirm with the latest revision: ↵reivilibre2021-11-261-14/+44
| | | | accept the `refresh_tokens` parameter in the request body rather than in the URL parameters. (#11430)
* Add type hints to `synapse/storage/databases/main/events_worker.py` (#11411)Sean Quah2021-11-261-1/+5
| | | | Also refactor the stream ID trackers/generators a bit and try to document them better.
* Support expiry of refresh tokens and expiry of the overall session when ↵reivilibre2021-11-261-1/+124
| | | | refresh tokens are in use. (#11425)
* Track ongoing event fetches correctly (again) (#11376)Sean Quah2021-11-261-1/+138
| | | | | | | | | | | | | The previous fix for the ongoing event fetches counter (8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and incorrect. When the database is unreachable, `_do_fetch` never gets run and so `_event_fetch_ongoing` is never decremented. The previous fix also moved the `_event_fetch_ongoing` decrement outside of the `_event_fetch_lock` which allowed race conditions to corrupt the counter.
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-251-2/+2
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-242-12/+21
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Rename unstable `access_token_lifetime` configuration option to ↵reivilibre2021-11-231-1/+1
| | | | `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. (#11388)
* Add missing type hints to config base classes (#11377)Patrick Cloke2021-11-231-8/+14
|
* Remove code invalidated by deprecated config flag ↵Shay2021-11-231-1/+0
| | | | | | | | | | | | | | | | | | | | | 'trust_identity_servers_for_password_resets' (#11395) * remove background update code related to deprecated config flag * changelog entry * update changelog * Delete 11394.removal Duplicate, wrong number * add no-op background update and change newfragment so it will be consolidated with associated work * remove unused code * Remove code associated with deprecated flag from legacy docker dynamic config file Co-authored-by: reivilibre <oliverw@matrix.org>
* Merge branch 'master' into developSean Quah2021-11-232-0/+253
|\
| * Prevent the media store from writing outside of the configured directorySean Quah2021-11-192-0/+253
| | | | | | | | | | Also tighten validation of server names by forbidding invalid characters in IPv6 addresses and empty domain labels.
* | Add config for customizing the claim used for JWT logins. (#11361)Kostas2021-11-221-32/+36
| | | | | | | | | | Allows specifying a different claim (from the default "sub") to use when calculating the localpart of the Matrix ID used during the JWT login.
* | Store arbitrary relations from events. (#11391)Patrick Cloke2021-11-222-1/+117
| | | | | | | | | | Instead of only known relation types. This also reworks the background update for thread relations to crawl events and search for any relation type, not just threaded relations.
* | Add an admin API to run background jobs. (#11352)Dirk Klimpel2021-11-191-8/+146
| | | | | | | | | | | | Instead of having admins poke into the database directly. Can currently run jobs to populate stats and to populate the user directory.
* | Fix checking whether a room can be published on creation. (#11392)Patrick Cloke2021-11-191-33/+62
| | | | | | | | | | | | | | | | If `room_list_publication_rules` was configured with a rule with a non-wildcard alias and a room was created with an alias then an internal server error would have been thrown. This fixes the error and properly applies the publication rules during room creation.
* | Keep fallback key marked as used if it's re-uploaded (#11382)Hubert Chathi2021-11-191-1/+31
| |
* | Fix verification of objects signed with old local keys (#11379)Richard van der Hoff2021-11-191-3/+53
| | | | | | | | | | | | | | Fixes a bug introduced in #11129: objects signed by the local server, but with keys other than the current one, could not be successfully verified. We need to check the key id in the signature, and track down the right key.
* | Remove legacy code related to deprecated ↵Shay2021-11-181-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `trust_identity_server_for_password_resets` config flag (#11333) * remove code legacy code related to deprecated config flag "trust_identity_server_for_password_resets" from synapse/config/emailconfig.py * remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/config/registration.py * remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/handlers/identity.py * add tests to ensure config error is thrown and synapse refuses to start when depreciated config flag is found * add changelog * slightly change behavior to only check for deprecated flag if set to 'true' * Update changelog.d/11333.misc Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* | Add dedicated admin API for blocking a room (#11324)Dirk Klimpel2021-11-181-0/+228
| |
* | Do not allow MSC3440 threads to fork threads (#11161)Patrick Cloke2021-11-181-0/+62
| | | | | | | | | | | | | | | | | | | | | | Adds validation to the Client-Server API to ensure that the potential thread head does not relate to another event already. This results in not allowing a thread to "fork" into other threads. If the target event is unknown for some reason (maybe it isn't visible to your homeserver), but is the target of other events it is assumed that the thread can be created from it. Otherwise, it is rejected as an unknown event.
* | Rename `get_access_token_for_user_id` method to ↵reivilibre2021-11-173-11/+11
| | | | | | | | `create_access_token_for_user_id` (#11369)
* | Fix incorrect return value in tests. (#11359)Patrick Cloke2021-11-161-1/+1
| |
* | Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-161-1/+51
|\| | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-1/+51
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Avoid sharing room hierarchy responses between users (#11355)Sean Quah2021-11-161-0/+55
| | | | | | | | | | Different users may be allowed to see different rooms within a space, so sharing responses between users is inadvisable.
* | Add ability to un-shadow-ban via the admin API. (#11347)Patrick Cloke2021-11-161-6/+20
| |
* | Misc typing fixes for tests, part 2 of N (#11330)David Robertson2021-11-164-29/+65
| |
* | Database storage profile passes mypy (#11342)David Robertson2021-11-151-3/+6
| | | | | | | | | | | | It already seems to pass mypy. I wonder what changed, given that it was on the exclusion list. So this commit consists of me ensuring `--disallow-untyped-defs` passes and a minor fixup to a function that returned either `True` or `None`.
* | Add support for the stable version of MSC2778 (#11335)Tulir Asokan2021-11-152-2/+8
| | | | | | | | | | | | | | | | | | * Add support for the stable version of MSC2778 Signed-off-by: Tulir Asokan <tulir@maunium.net> * Expect m.login.application_service in login and password provider tests Signed-off-by: Tulir Asokan <tulir@maunium.net>
* | Test room alias deletion (#11327)David Robertson2021-11-121-28/+77
| | | | | | | | | | | | | | | | | | | | * Prefer `HTTPStatus` over plain `int` This is an Opinion that no-one has seemed to object to yet. * `--disallow-untyped-defs` for `tests.rest.client.test_directory` * Improve synapse's annotations for deleting aliases * Test case for deleting a room alias * Changelog
* | Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-121-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* | Misc typing fixes for `tests`, part 1 of N (#11323)David Robertson2021-11-124-27/+47
| | | | | | | | | | | | | | | | | | | | * Annotate HomeserverTestCase.servlets * Correct annotation of federation_auth_origin * Use AnyStr custom_headers instead of a Union This allows (str, str) and (bytes, bytes). This disallows (str, bytes) and (bytes, str) * DomainSpecificString.SIGIL is a ClassVar
* | Convert delete room admin API to async endpoint (#11223)Dirk Klimpel2021-11-122-65/+709
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Remove redundant parameters on `_check_event_auth` (#11292)Richard van der Hoff2021-11-101-2/+0
| | | | | | as of #11012, these parameters are unused.
* | Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-091-2/+6
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Allow admins to proactively block rooms (#11228)David Robertson2021-11-091-0/+28
| | | | | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-094-35/+438
|/ | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Include cross-signing signatures when syncing remote devices for the first ↵Erik Johnston2021-11-091-0/+151
| | | | | | | time (#11234) When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results. c.f. #11159
* Require body for read receipts with user-agent exceptions (#11157)rogersheu2021-11-091-2/+28
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Add some background update admin APIs (#11263)Erik Johnston2021-11-081-0/+218
| | | Fixes #11259
* Default value for `public_baseurl` (#11210)Richard van der Hoff2021-11-083-3/+1
| | | | | We might as well use a default value for `public_baseurl` based on `server_name` - in many cases, it will be correct.
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-051-0/+69
| | | Fixes #11252
* Additional test for `cachedList` (#11246)Richard van der Hoff2021-11-041-0/+43
| | | | | I was trying to understand how `cachedList` works, and ended up writing this extra test. I figure we may as well keep it.
* Add a linearizer on (appservice, stream) when handling ephemeral events. ↵Nick Barrett2021-11-031-0/+51
| | | | | (#11207) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Remove a debug statement from tests. (#11239)Patrick Cloke2021-11-031-1/+0
|
* Merge remote-tracking branch 'origin/master' into developErik Johnston2021-11-021-0/+15
|\
| * Fix URL preview errors when previewing XML documents. (#11196)Patrick Cloke2021-10-271-0/+15
| |
* | Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-021-0/+74
| |
* | Add search by room ID and room alias to List Room admin API (#11099)Dirk Klimpel2021-11-021-39/+49
| | | | | | | | Fixes: #10874 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | ObservableDeferred: run observers in order (#11229)Richard van der Hoff2021-11-022-4/+69
| |
* | Handle missing Content-Type header when accessing remote media (#11200)Shay2021-11-011-2/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add code to handle missing content-type header and a test to verify that it works * add handling for missing content-type in the /upload endpoint as well * slightly refactor test code to put private method in approriate place * handle possible null value for content-type when pulling from the local db * add changelog * refactor test and add code to handle missing content-type in cached remote media * requested changes * Update changelog.d/11200.bugfix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* | Fix a bug in unit test `test_block_room_and_not_purge` (#11226)Dirk Klimpel2021-11-011-2/+2
| |
* | Make `check_event_allowed` module API callback not fail open (accept events) ↵reivilibre2021-11-011-13/+3
| | | | | | | | when an exception is raised (#11033)
* | Remove deprecated delete room admin API (#11213)Dirk Klimpel2021-11-011-24/+15
| | | | | | | | | | Remove deprecated delete room admin API, `POST /_synapse/admin/v1/rooms/<room_id>/delete`
* | Support for serving server well-known files (#11211)Richard van der Hoff2021-11-011-6/+26
| | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/8308
* | Test that `ClientIpStore` combines database and in-memory data correctly ↵Sean Quah2021-11-011-0/+206
| | | | | | | | (#11179)
* | Add a module API method to retrieve state from a room (#11204)Brendan Abolivier2021-10-291-1/+24
| |
* | Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-282-2/+62
| | | | | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* | Fetch verify key locally rather than trying to do so over federation if ↵Shay2021-10-281-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | origin and host are the same. (#11129) * add tests for fetching key locally * add logic to check if origin server is same as host and fetch verify key locally rather than over federation * add changelog * slight refactor, add docstring, change changelog entry * Make changelog entry one line * remove verify_json_locally and push locality check to process_request, add function process_request_locally * remove leftover code reference * refactor to add common call to 'verify_json and associated handling code * add type hint to process_json * add some docstrings + very slight refactor
* | Add a ModuleApi method to update a user's membership in a room (#11147)Brendan Abolivier2021-10-281-1/+125
| | | | | | Co-authored-by: reivilibre <oliverw@matrix.org>
* | Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-272-0/+121
| | | | | | Fixes: #9346
* | Implement an `on_new_event` callback (#11126)Brendan Abolivier2021-10-261-3/+90
|/ | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Enable changing user type via users admin API (#11174)Jason Robinson2021-10-261-0/+51
| | | | | | | | Users admin API can now also modify user type in addition to allowing it to be set on user creation. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Don't set new room alias before potential 403 (#10930)AndrewFerr2021-10-251-1/+101
| | | | | | Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Fix synapse.config module "read" command (#11145)Jason Robinson2021-10-223-55/+104
| | | | | | | | | | `synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse. The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example. Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Add a thread relation type per MSC3440. (#11088)Patrick Cloke2021-10-211-6/+34
| | | | Adds experimental support for MSC3440's `io.element.thread` relation type (and the aggregation for it).
* Fix adding excluded users to the private room sharing tables when joining a ↵David Robertson2021-10-211-14/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | room (#11143) * We only need to fetch users in private rooms * Filter out `user_id` at the top * Discard excluded users in the top loop We weren't doing this in the "First, if they're our user" branch so this is a bugfix. * The caller must check that `user_id` is included This is in the docstring. There are two call sites: - one in `_handle_room_publicity_change`, which explicitly checks before calling; - and another in `_handle_room_membership_event`, which returns early if the user is excluded. So this change is safe. * Test joining a private room with an excluded user * Tweak an existing test * Changelog * test docstring * lint
* Fix setting a user's external_id via the admin API returns 500 and deletes ↵Dirk Klimpel2021-10-211-3/+212
| | | | | users existing external mappings if that external ID is already mapped (#11051) Fixes #10846
* Show error when timestamp in seconds is provided to the /purge_media_cache ↵Aaron R2021-10-201-4/+102
| | | | API (#11101)
* Merge branch 'master' into developSean Quah2021-10-201-34/+3
|\
| * Revert change to counting of deactivated users towards the monthly active ↵Sean Quah2021-10-201-34/+3
| | | | | | | | | | | | | | | | users limit (#11127) Temporarily revert "Add functionality to remove deactivated users from the monthly_active_users table (#10947)". This reverts commit eda8c88b84ee7506379a71ac2a7a88c08b759d43.
* | Check *all* auth events for room id and rejection (#11009)Richard van der Hoff2021-10-181-48/+90
| | | | | | | | | | | | | | | | | | | | | | This fixes a bug where we would accept an event whose `auth_events` include rejected events, if the rejected event was shadowed by another `auth_event` with same `(type, state_key)`. The approach is to pass a list of auth events into `check_auth_rules_for_event` instead of a dict, which of course means updating the call sites. This is an extension of #10956.
* | Don't remove local users from dir when the leave their last room (#11103)David Robertson2021-10-181-0/+50
| |
* | Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-152-79/+140
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Fix 500 error on `/messages` when we accumulate more than 5 backward ↵Eric Eastwood2021-10-141-0/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | extremities (#11027) Found while working on the Gitter backfill script and noticed it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390 When there are more than 5 backward extremities for a given depth, backfill will throw an error because we sliced the extremity list to 5 but then try to iterate over the full list. This causes us to look for state that we never fetched and we get a `KeyError`. Before when calling `/messages` when there are more than 5 backward extremities: ``` Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper callback_return = await self._async_render(request) File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render callback_return = await raw_callback_return File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET msgs = await self.pagination_handler.get_messages( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages await self.hs.get_federation_handler().maybe_backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill return await self._maybe_backfill_inner(room_id, current_depth, limit) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner likely_extremeties_domains = get_domains_from_state(states[e_id]) KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl' ```
* | Ensure each charset is attempted only once during media preview. (#11089)Patrick Cloke2021-10-141-8/+35
| | | | | | | | There's no point in trying more than once since it is guaranteed to continually fail.
* | Attempt different character encodings when previewing a URL. (#11077)Patrick Cloke2021-10-141-26/+40
| | | | | | | | This follows similar logic to BeautifulSoup where we attempt different character encodings until we find one which works.
* | Fix-up some type hints in the relations tests. (#11076)Patrick Cloke2021-10-143-51/+62
| |
* | Add a test for a workaround concerning the behaviour of third-party rule ↵reivilibre2021-10-141-6/+50
| | | | | | | | modules and `SynapseError`s. (#11071)
* | Merge remote-tracking branch 'origin/release-v1.45' into developDavid Robertson2021-10-132-8/+74
|\|
| * Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-134-10/+628
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-131-26/+197
| | | | | | | | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Be more lenient when parsing the version for oEmbed responses. (#11065)Patrick Cloke2021-10-131-0/+51
| |
* | Simplify the user admin API tests (#11048)Dirk Klimpel2021-10-121-255/+146
| |
* | Add tests for `MediaFilePaths` (#11057)Sean Quah2021-10-121-0/+238
| |
* | Merge remote-tracking branch 'origin/release-v1.45' into developErik Johnston2021-10-121-13/+11
|\|
| * Reset global cache state before cache tests. (#11036)Patrick Cloke2021-10-121-13/+11
| | | | | | | | | | | | This reverts #11019 and structures the code a bit more like it was before #10985. The global cache state must be reset before running the tests since other test cases might have configured caching (and thus touched the global state).
* | Fix inconsistent behavior of `get_last_client_by_ip` (#10970)Sean Quah2021-10-121-0/+43
| | | | | | | | | | | | | | Make `get_last_client_by_ip` return the same dictionary structure regardless of whether the data has been persisted to the database. This change will allow slightly cleaner type hints to be applied later on.
* | Add an approximate difference method to StateFilters (#10825)reivilibre2021-10-121-2/+511
|/
* Autodiscover oEmbed endpoint from returned HTML (#10822)Patrick Cloke2021-10-082-17/+123
| | | | | Searches the returned HTML for an oEmbed endpoint using the autodiscovery mechanism (`<link rel=...>`), and will request it to generate the preview.
* Fix overwriting profile when making room public (#11003)David Robertson2021-10-081-2/+69
| | | This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
* Include exception in json logging (#11028)Nick Barrett2021-10-081-0/+28
|
* Ensure each cache config test uses separate state. (#11019)Patrick Cloke2021-10-071-8/+12
| | | Hopefully this fixes these tests sometimes failing in CI.
* Don't alter directory entries for local users when setting a per-room ↵David Robertson2021-10-071-0/+34
| | | | | nickname (#11002) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add a spamchecker method to allow or deny 3pid invites (#10894)Brendan Abolivier2021-10-061-0/+70
| | | | | This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline. Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-066-65/+60
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Add a spamchecker callback to allow or deny room joins (#10910)Brendan Abolivier2021-10-061-0/+101
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-069-78/+61
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)David Robertson2021-10-051-16/+78
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two steps to rebuilding the user directory: 1. a scan over rooms, followed by 2. a scan over local users. The former reads avatars and display names from the `room_memberships` table and therefore contains potentially private avatars and display names. The latter reads from the the `profiles` table which only contains public data; moreover it will overwrite any private profiles that the rooms scan may have written to the user directory. This means that the rebuild could leak private user while the rebuild was in progress, only to later cover up the leaks once the rebuild had completed. This change skips over local users when writing user_directory rows when scanning rooms. Doing so means that it'll take longer for a rebuild to make local users searchable, which is unfortunate. I think a future PR can improve this by swapping the order of the two steps above. (And indeed there's more to do here, e.g. copying from `profiles` without going via Python.) Small tidy-ups while I'm here: * Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`. * Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`. * Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add functionality to remove deactivated users from the monthly_active_users ↵Hillery Shay2021-10-041-3/+34
| | | | | | | | | | | | | | | | | | | | | | | | | table (#10947) * add test * add function to remove user from monthly active table in deactivate code * add function to remove user from monthly active table * add changelog entry * update changelog number * requested changes * update docstring on new function * fix lint error * Update synapse/storage/databases/main/monthly_active_users.py Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Make is_public Optional[bool] for create_room_as test util (#10951) (#10963)AndrewFerr2021-10-041-6/+7
| | | Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Merge tag 'v1.44.0rc3' into developBrendan Abolivier2021-10-041-0/+34
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc3 (2021-10-04) ============================== Bugfixes -------- - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933)) - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
| * Fix error in `get_user_ip_and_agents` when fetching from the database (#10968)Sean Quah2021-10-011-0/+34
| |
| * Strip "join_authorised_via_users_server" from join events which do not need ↵Patrick Cloke2021-10-012-7/+9
| | | | | | | | | | | | | | | | it. (#10933) This fixes a "Event not signed by authorising server" error when transition room member from join -> join, e.g. when updating a display name or avatar URL for restricted rooms.
* | Consistently exclude from user_directory (#10960)David Robertson2021-10-044-26/+366
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-047-13/+13
| |
* | type-hint `HomeserverTestcase.setup_test_homeserver` (#10961)David Robertson2021-10-013-9/+20
| | | | | | | | | | * type-hint `HomeserverTestcase.setup_test_homeserver` For better IDE completion. A small drive-by.
* | Clean-up registration tests (#10945)Patrick Cloke2021-09-301-37/+52
| | | | | | | | Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent to properly configure auto-join rooms.
* | Strip "join_authorised_via_users_server" from join events which do not need ↵Patrick Cloke2021-09-302-7/+9
| | | | | | | | | | | | | | it. (#10933) This fixes a "Event not signed by authorising server" error when transition room member from join -> join, e.g. when updating a display name or avatar URL for restricted rooms.
* | Fix errors in Synapse logs from unit tests. (#10939)Patrick Cloke2021-09-304-26/+34
| | | | | | | | | | Fix some harmless errors from background processes (mostly due to awaiting Mock objects) that occurred in the Synapse logs during unit tests.
* | Pass str to twisted's IReactorTCP (#10895)David Robertson2021-09-302-6/+6
| | | | | | | | | | | | | | This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job. Until that change is in a twisted release, we'll have to ignore the type of the `host` argument. I've raised #10899 to remind us to review the issue in a few months' time.
* | Refactor user directory tests (#10935)David Robertson2021-09-303-199/+280
| | | | | | | | | | | | | | | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Split `event_auth.check` into two parts (#10940)Richard van der Hoff2021-09-291-72/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | Broadly, the existing `event_auth.check` function has two parts: * a validation section: checks that the event isn't too big, that it has the rught signatures, etc. This bit is independent of the rest of the state in the room, and so need only be done once for each event. * an auth section: ensures that the event is allowed, given the rest of the state in the room. This gets done multiple times, against various sets of room state, because it forms part of the state res algorithm. Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think that makes everything hard to follow. Instead, we split the function in two and call each part separately where it is needed.
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-2913-34/+34
| |
* | Implement MSC3069: Guest support on whoami (#9655)Travis Ralston2021-09-291-5/+44
| |
* | Fix empty `url_cache_thumbnails/yyyy-mm-dd/` directories being left behind ↵Sean Quah2021-09-291-0/+31
| | | | | | | | (#10924)
* | Inline `_check_event_auth` for outliers (#10926)Richard van der Hoff2021-09-281-1/+0
|/ | | | | | | | | | | | | | | | | | | | | | | | | | * Inline `_check_event_auth` for outliers When we are persisting an outlier, most of `_check_event_auth` is redundant: * `_update_auth_events_and_context_for_auth` does nothing, because the `input_auth_events` are (now) exactly the event's auth_events, which means that `missing_auth` is empty. * we don't care about soft-fail, kicking guest users or `send_on_behalf_of` for outliers ... so the only thing that matters is the auth itself, so let's just do that. * `_auth_and_persist_fetched_events_inner`: de-async `prep` `prep` no longer calls any `async` methods, so let's make it synchronous. * Simplify `_check_event_auth` We no longer need to support outliers here, which makes things rather simpler. * changelog * lint
* Avoid storing URL cache files in storage providers (#10911)Sean Quah2021-09-271-0/+130
| | | | | URL cache files are short-lived and it does not make sense to offload them (eg. to the cloud) or back them up.
* Add a spamchecker callback to allow or deny room creation based on invites ↵Brendan Abolivier2021-09-241-2/+117
| | | | | | | (#10898) This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline. This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-247-13/+15
|
* Factor out common code for persisting fetched auth events (#10896)Richard van der Hoff2021-09-241-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Factor more stuff out of `_get_events_and_persist` It turns out that the event-sorting algorithm in `_get_events_and_persist` is also useful in other circumstances. Here we move the current `_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`, and then factor the sorting part out to `_auth_and_persist_fetched_events`. * `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment `get_event_auth` returns events with the outlier flag already set, so this is redundant (though we need to update a test where `get_event_auth` is mocked). * `_get_remote_auth_chain_for_event`: move existing-event tests earlier Move a couple of tests outside the loop. This is a bit inefficient for now, but a future commit will make it better. It should be functionally identical. * `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events` We can use the same codepath for persisting the events fetched as part of an auth chain as for those fetched individually by `_get_events_and_persist` for building the state at a backwards extremity. * `_get_remote_auth_chain_for_event`: use a dict for efficiency `_auth_and_persist_fetched_events` sorts the events itself, so we no longer need to care about maintaining the ordering from `get_event_auth` (and no longer need to sort by depth in `get_event_auth`). That means that we can use a map, making it easier to filter out events we already have, etc. * changelog * `_auth_and_persist_fetched_events`: improve docstring
* Add reactor to `SynapseRequest` and fix up types. (#10868)Erik Johnston2021-09-2413-37/+69
|
* Fix AuthBlocking check when requester is appservice (#10881)Jason Robinson2021-09-241-0/+62
| | | | | If the MAU count had been reached, Synapse incorrectly blocked appservice users even though they've been explicitly configured not to be tracked (the default). This was due to bypassing the relevant if as it was chained behind another earlier hit if as an elif. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* Improve typing in user_directory files (#10891)David Robertson2021-09-241-2/+3
| | | | | | | | | | | * Improve typing in user_directory files This makes the user_directory.py in storage pass most of mypy's checks (including `no-untyped-defs`). Unfortunately that file is in the tangled web of Store class inheritance so doesn't pass mypy at the moment. The handlers directory has already been mypyed. Co-authored-by: reivilibre <olivier@librepush.net>
* Allow `.` and `~` chars in registration tokens (#10887)Callum Brown2021-09-231-3/+5
| | | | Per updates to MSC3231 in order to use the same grammar as other identifiers.
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-2310-20/+19
|
* Fix reactivated users not being added to the user directory (#10782)David Robertson2021-09-231-1/+41
| | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-234-12/+8
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-232-2/+2
|
* Treat "\u0000" as "\u0020" for the purposes of message search (message ↵Hillery Shay2021-09-221-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | indexing) (#10820) * add test to check if null code points are being inserted * add logic to detect and replace null code points before insertion into db * lints * add license to test * change approach to null substitution * add type hint for SearchEntry * Add changelog entry Signed-off-by: H.Shay <shaysquared@gmail.com> * updated changelog * update chanelog message * remove duplicate changelog * Update synapse/storage/databases/main/events.py remove extra space Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * rename and move test file, update tests, delete old test file * fix typo in comments * update _find_highlights_in_postgres to replace null byte with space * replace null byte in sqlite search insertion * beef up and reorganize test for this pr * update changelog * add type hints and update docstring * check db engine directly vs using env variable * refactor tests to be less repetetive * move rplace logic into seperate function * requested changes * Fix typo. * Update synapse/storage/databases/main/search.py Co-authored-by: reivilibre <olivier@librepush.net> * Update changelog.d/10820.misc Co-authored-by: Aaron Raimist <aaron@raim.ist> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Aaron Raimist <aaron@raim.ist>
* Include more information in oEmbed previews. (#10819)Patrick Cloke2021-09-221-9/+21
| | | | | | | * Improved titles (fall back to the author name if there's not title) and include the site name. * Handle photo/video payloads. * Include the original URL in the Open Graph response. * Fix the expiration time (by properly converting from seconds to milliseconds).
* Extend ModuleApi with the methods we'll need to reject spam based on …IP - ↵David Teller2021-09-221-0/+72
| | | | | | | | resolves #10832 (#10833) Extend ModuleApi with the methods we'll need to reject spam based on IP - resolves #10832 Signed-off-by: David Teller <davidt@element.io>
* Add type hints for event streams. (#10856)Patrick Cloke2021-09-214-13/+55
|
* Refactor oEmbed previews (#10814)Patrick Cloke2021-09-211-13/+13
| | | | | | | | | | | | | The major change is moving the decision of whether to use oEmbed further up the call-stack. This reverts the _download_url method to being a "dumb" functionwhich takes a single URL and downloads it (as it was before #7920). This also makes more minor refactorings: * Renames internal variables for clarity. * Factors out shared code between the HTML and rich oEmbed previews. * Fixes tests to preview an oEmbed image.
* Test that state events sent by modules correctly end up in the room's state ↵Brendan Abolivier2021-09-211-0/+84
| | | | | | | (#10835) Test for #10830 Ideally the test would also make sure the new state event comes down sync, but this is probably good enough.
* Allow Synapse Admin API's Room Search to accept non-ASCII characters (#10859)Hillery Shay2021-09-211-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add tests for checking if room search works with non-ascii char * change encoding on parse_string to UTF-8 * lints * properly encode search term * lints * add changelog file * update changelog number * set changelog entry filetype to .bugfix * Revert "set changelog entry filetype to .bugfix" This reverts commit be8e5a314251438ec4ec7dbc59ba32162c93e550. * update changelog message and file type * change parse_string default encoding back to ascii and update room search admin api calll to parse string * refactor tests * Update tests/rest/admin/test_room.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Always add local users to the user directory (#10796)David Robertson2021-09-212-8/+11
| | | | | | | | | | | | | | | | | | | | | | | It's a simplification, but one that'll help make the user directory logic easier to follow with the other changes upcoming. It's not strictly required for those changes, but this will help simplify the resulting logic that listens for `m.room.member` events and generally make the logic easier to follow. This means the config option `search_all_users` ends up controlling the search query only, and not the data we store. The cost of doing so is an extra row in the `user_directory` and `user_directory_search` tables for each local user which - belongs to no public rooms - belongs to no private rooms of size ≥ 2 I think the cost of this will be marginal (since they'll already have entries in `users` and `profiles` anyway). As a small upside, a homeserver whose directory was built with this change can toggle `search_all_users` without having to rebuild their directory. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Allow sending a membership event to unban a user (#10807)Aaron Raimist2021-09-212-1/+97
| | | | | * Allow membership event to unban user Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Create a constant for a small png image in tests. (#10834)Patrick Cloke2021-09-167-94/+44
| | | To avoid duplicating it between a few tests.
* Make StateFilter frozen so we can hash it (#10816)reivilibre2021-09-141-17/+29
| | | Also enables Mypy for related tests.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-6/+12
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Add types to synapse.util. (#10601)reivilibre2021-09-101-2/+2
|
* Handle room upgrades for spaces (#10774)Patrick Cloke2021-09-101-4/+63
| | | | | By copying the `room_type` field of the create event and migrating any non-empty `m.space.child` events to the new room that is created.
* Easy refactors of the user directory (#10789)David Robertson2021-09-101-3/+3
| | | No functional changes here. This came out as I was working to tackle #5677
* Remove unstable MSC2858 API, including `experimental.msc2858_enabled` config ↵Sean2021-09-091-58/+7
| | | | | option (#10693) Signed-off-by: Sean Quah <seanq@element.io>
* Get rid of `_auth_and_persist_event` (#10781)Richard van der Hoff2021-09-081-3/+12
| | | This is only called in two places, and the code seems much clearer without it.
* Skip handling of push actions for outlier events (#10780)Erik Johnston2021-09-081-0/+1
| | | | Outlier events don't ever have push actions associated with them, so we can skip some expensive queries during event persistence.
* Request JSON for oEmbed requests (and ignore XML only providers). (#10759)Patrick Cloke2021-09-081-1/+54
| | | | | | | | This adds the format to the request arguments / URL to ensure that JSON data is returned (which is all that Synapse supports). This also adds additional error checking / filtering to the configuration file to ignore XML-only providers.
* Merge tag 'v1.42.0rc2' into developBrendan Abolivier2021-09-061-1/+45
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.42.0rc2 (2021-09-06) ============================== This version of Synapse removes deprecated room-management admin APIs, removes out-of-date email pushers, and improves error handling for fallback templates for user-interactive authentication. For more information on these points, server administrators are encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375). ([\#10747](https://github.com/matrix-org/synapse/issues/10747)) Internal Changes ---------------- - Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768))
| * Support MSC3375: room version 9. (#10747)Patrick Cloke2021-09-031-1/+45
| |
* | Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-061-4/+4
| |
* | Pull out encrypted_by_default tests from user_directory tests (#10752)David Robertson2021-09-062-95/+109
| |