summary refs log tree commit diff
path: root/synapse (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Support Client-Server API r0.6.1 (#11097)Aaron R2021-11-011-0/+1
| | | | | Fixes #11064 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add metrics to the threadpools (#11178)Erik Johnston2021-11-013-1/+48
|
* Stop synapse from saving messages in device_inbox for hidden devices. (#10097)JohannesKleine2021-11-011-2/+6
| | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fix comments referencing v1.46.0 from PR #10969. (#11212)Dirk Klimpel2021-10-292-2/+2
| | | | #10969 was merged after 1.46.0rc1 was cut and will be included in v1.47.0rc1 instead.
* Add a module API method to retrieve state from a room (#11204)Brendan Abolivier2021-10-291-0/+49
|
* Additional type hints for relations database class. (#11205)Patrick Cloke2021-10-281-15/+23
|
* Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-282-0/+36
| | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* Fetch verify key locally rather than trying to do so over federation if ↵Shay2021-10-281-29/+45
| | | | | | | | | | | | | | | | | | | | | | | origin and host are the same. (#11129) * add tests for fetching key locally * add logic to check if origin server is same as host and fetch verify key locally rather than over federation * add changelog * slight refactor, add docstring, change changelog entry * Make changelog entry one line * remove verify_json_locally and push locality check to process_request, add function process_request_locally * remove leftover code reference * refactor to add common call to 'verify_json and associated handling code * add type hint to process_json * add some docstrings + very slight refactor
* Add a ModuleApi method to update a user's membership in a room (#11147)Brendan Abolivier2021-10-281-1/+99
| | | Co-authored-by: reivilibre <oliverw@matrix.org>
* Type hints for the remaining two files in `synapse.http`. (#11164)David Robertson2021-10-284-27/+53
| | | | | | | | | | | | | | | | | | | * Teach MyPy that the sentinel context is False This means that if `ctx: LoggingContextOrSentinel` then `bool(ctx)` narrows us to `ctx:LoggingContext`, which is a really neat find! * Annotate RequestMetrics - Raise errors for sentry if we use the sentinel context - Ensure we don't raise an error and carry on, but not recording stats - Include stack trace in the error case to lower Sean's blood pressure * Make mypy pass for synapse.http.request_metrics * Make synapse.http.connectproxyclient pass mypy Co-authored-by: reivilibre <oliverw@matrix.org>
* Annotate `log_function` decorator (#10943)reivilibre2021-10-2711-18/+57
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fixed config parse bug in review_recent_signups (#11191)Samuel Philipp2021-10-271-2/+7
|
* Refactor `Filter` to handle fields according to data being filtered. (#11194)Patrick Cloke2021-10-274-69/+86
| | | | This avoids filtering against fields which cannot exist on an event source. E.g. presence updates don't have a room.
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-273-15/+134
| | | Fixes: #9346
* Include the stable identifier for MSC3288. (#11187)Patrick Cloke2021-10-271-0/+2
| | | | | Includes both the stable and unstable identifier to store-invite calls to the identity server. In the future we should remove the unstable identifier.
* Merge remote-tracking branch 'origin/release-v1.46' into developErik Johnston2021-10-273-2/+3
|\
| * Shut down the DNS threadpool (#11190)Sean Quah2021-10-271-0/+1
| | | | | | | | | | The DNS threadpool must be explicitly stopped, otherwise Synapse will hang indefinitely when asked to shut down.
| * Fix thread BG update to not seq scan event_json (#11192)Erik Johnston2021-10-271-1/+1
| | | | | | | | For some reason the query optimiser decided to seq scan both tables, rather than index scanning `event_json`.
| * 1.46.0rc1Sean Quah2021-10-261-1/+1
| |
* | Implement an `on_new_event` callback (#11126)Brendan Abolivier2021-10-265-9/+53
|/ | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Move DNS lookups into separate thread pool (#11177)Erik Johnston2021-10-262-1/+148
| | | | | | This is to stop large bursts of lookups starving out other users of the thread pools. Fixes #11049.
* Add a background update for updating MSC3440 relation threads. (#11181)Patrick Cloke2021-10-262-2/+101
|
* Document the version each module API method was added to Synapse (#11183)Brendan Abolivier2021-10-261-11/+88
|
* Enable changing user type via users admin API (#11174)Jason Robinson2021-10-262-0/+21
| | | | | | | | Users admin API can now also modify user type in addition to allowing it to be set on user creation. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Fix cyclic import in the module API (#11180)Brendan Abolivier2021-10-251-2/+4
| | | | | Introduced in #10548 See https://github.com/matrix-org/synapse-email-account-validity/runs/3979337154?check_suite_focus=true for an example of a module's CI choking over this issue.
* Don't set new room alias before potential 403 (#10930)AndrewFerr2021-10-252-11/+11
| | | | | | Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Ensure that we correctly auth events returned by `send_join` (#11012)Richard van der Hoff2021-10-251-86/+60
| | | | | | | This is the final piece of the jigsaw for #9595. As with other changes before this one (eg #10771), we need to make sure that we auth the auth events in the right order, and actually check that their predecessors haven't been rejected. To do this I've reused the existing code we use when persisting outliers elsewhere. I've removed the code for attempting to fetch missing auth_events - the events should have been present in the send_join response, so the likely reason they are missing is that we couldn't verify them, so requesting them again is unlikely to help. Instead, we simply drop any state which relies on those auth events, as we do at a backwards-extremity. See also matrix-org/complement#216 for a test for this.
* Fix module API's `get_user_ip_and_agents` function when run on workers (#11112)Sean Quah2021-10-252-40/+90
|
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-2257-143/+341
|
* Fix synapse.config module "read" command (#11145)Jason Robinson2021-10-221-13/+33
| | | | | | | | | | `synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse. The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example. Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Add a thread relation type per MSC3440. (#11088)Patrick Cloke2021-10-216-2/+84
| | | | Adds experimental support for MSC3440's `io.element.thread` relation type (and the aggregation for it).
* Fix adding excluded users to the private room sharing tables when joining a ↵David Robertson2021-10-211-15/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | room (#11143) * We only need to fetch users in private rooms * Filter out `user_id` at the top * Discard excluded users in the top loop We weren't doing this in the "First, if they're our user" branch so this is a bugfix. * The caller must check that `user_id` is included This is in the docstring. There are two call sites: - one in `_handle_room_publicity_change`, which explicitly checks before calling; - and another in `_handle_room_membership_event`, which returns early if the user is excluded. So this change is safe. * Test joining a private room with an excluded user * Tweak an existing test * Changelog * test docstring * lint
* Improve docstrings for methods related to sending EDUs to application ↵Andrew Morgan2021-10-216-23/+147
| | | | services (#11138)
* Add missing type hints to synapse.crypto. (#11146)Patrick Cloke2021-10-213-18/+32
| | | And require type hints for this module.
* Fix setting a user's external_id via the admin API returns 500 and deletes ↵Dirk Klimpel2021-10-212-34/+108
| | | | | users existing external mappings if that external ID is already mapped (#11051) Fixes #10846
* Consider IP whitelist for identity server resolution (#11120)Robert Edström2021-10-201-1/+3
| | | Signed-off-by: Robert Edström <github@legogris.se>
* Clean up `_update_auth_events_and_context_for_auth` (#11122)Richard van der Hoff2021-10-201-114/+37
| | | Remove some redundant code, and generally simplify.
* Show error when timestamp in seconds is provided to the /purge_media_cache ↵Aaron R2021-10-201-6/+27
| | | | API (#11101)
* Merge branch 'master' into developSean Quah2021-10-203-29/+1
|\
| * 1.45.1Sean Quah2021-10-201-1/+1
| |
| * Revert change to counting of deactivated users towards the monthly active ↵Sean Quah2021-10-202-28/+0
| | | | | | | | | | | | | | | | users limit (#11127) Temporarily revert "Add functionality to remove deactivated users from the monthly_active_users table (#10947)". This reverts commit eda8c88b84ee7506379a71ac2a7a88c08b759d43.
* | Add missing type hints to event fetching. (#11121)Patrick Cloke2021-10-191-61/+81
| | | | | | | | Updates the event rows returned from the database to be attrs classes instead of dictionaries.
* | Merge branch 'master' into developDavid Robertson2021-10-191-1/+1
|\|
| * 1.45.0David Robertson2021-10-191-1/+1
| |
* | Move _persist_auth_tree into FederationEventHandler (#11115)Richard van der Hoff2021-10-192-125/+119
| | | | | | | | | | This is just a lift-and-shift, because it fits more naturally here. We do rename it to `process_remote_join` at the same time though.
* | Rename `_auth_and_persist_fetched_events` (#11116)Richard van der Hoff2021-10-191-14/+9
| | | | | | | | ... to `_auth_and_persist_outliers`, since that reflects its purpose better.
* | Include rejected status when we log events. (#11008)Richard van der Hoff2021-10-191-6/+10
| | | | | | | | | | | | If we find ourselves dealing with rejected events, we proably want to know about it. Let's include it in the stringification of the event so that it gets logged.
* | Add missing type hints to synapse.api. (#11109)Patrick Cloke2021-10-188-99/+80
| | | | | | | | * Convert UserPresenceState to attrs. * Remove args/kwargs from error classes and explicitly pass msg/errorcode.
* | Check auth on received events' auth_events (#11001)Richard van der Hoff2021-10-181-2/+97
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, when we receive an event whose auth_events differ from those we expect, we state-resolve between the two state sets, and check that the event passes auth based on the resolved state. This means that it's possible for us to accept events which don't pass auth at their declared auth_events (or where the auth events themselves were rejected), leading to problems down the line like #10083. This change means we will: * ignore any events where we cannot find the auth events * reject any events whose auth events were rejected * reject any events which do not pass auth at their declared auth_events. Together with a whole raft of previous work, this is a partial fix to #9595. Fixes #6643. Based on #11009.
* | Check *all* auth events for room id and rejection (#11009)Richard van der Hoff2021-10-186-37/+31
| | | | | | | | | | | | | | | | | | | | | | This fixes a bug where we would accept an event whose `auth_events` include rejected events, if the rejected event was shadowed by another `auth_event` with same `(type, state_key)`. The approach is to pass a list of auth events into `check_auth_rules_for_event` instead of a dict, which of course means updating the call sites. This is an extension of #10956.
* | `_run_push_actions_and_persist_event`: handle no min_depth (#11014)Richard van der Hoff2021-10-182-11/+19
| | | | | | | | | | Make sure that we correctly handle rooms where we do not yet have a `min_depth`, and also add some comments and logging.
* | Fix broken export-data admin command and add a test for it to CI (#11078)Hillery Shay2021-10-181-8/+6
| | | | | | Fix broken export-data admin command and add a test for it to CI
* | Don't remove local users from dir when the leave their last room (#11103)David Robertson2021-10-181-5/+8
| |
* | Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-151-4/+7
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Move experimental & retention config out of the server module. (#11070)Patrick Cloke2021-10-158-214/+247
| |
* | Fix logging context warnings when losing replication connection (#10984)Sean Quah2021-10-152-10/+26
| | | | | | | | | | | | Instead of triggering `__exit__` manually on the replication handler's logging context, use it as a context manager so that there is an `__enter__` call to balance the `__exit__`.
* | Fix 500 error on `/messages` when we accumulate more than 5 backward ↵Eric Eastwood2021-10-142-12/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | extremities (#11027) Found while working on the Gitter backfill script and noticed it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390 When there are more than 5 backward extremities for a given depth, backfill will throw an error because we sliced the extremity list to 5 but then try to iterate over the full list. This causes us to look for state that we never fetched and we get a `KeyError`. Before when calling `/messages` when there are more than 5 backward extremities: ``` Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper callback_return = await self._async_render(request) File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render callback_return = await raw_callback_return File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET msgs = await self.pagination_handler.get_messages( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages await self.hs.get_federation_handler().maybe_backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill return await self._maybe_backfill_inner(room_id, current_depth, limit) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner likely_extremeties_domains = get_domains_from_state(states[e_id]) KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl' ```
* | Ensure each charset is attempted only once during media preview. (#11089)Patrick Cloke2021-10-141-6/+28
| | | | | | | | There's no point in trying more than once since it is guaranteed to continually fail.
* | Attempt different character encodings when previewing a URL. (#11077)Patrick Cloke2021-10-141-41/+39
| | | | | | | | This follows similar logic to BeautifulSoup where we attempt different character encodings until we find one which works.
* | Merge tag 'v1.45.0rc2' into developDavid Robertson2021-10-141-1/+1
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.45.0rc2 (2021-10-14) ============================== **Note:** This release candidate [fixes](https://github.com/matrix-org/synapse/issues/11053) the user directory [bug](https://github.com/matrix-org/synapse/issues/11025) present in 1.45.0rc1. However, the [performance issue](https://github.com/matrix-org/synapse/issues/11049) which appeared in v1.44.0 is yet to be resolved. Bugfixes -------- - Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](https://github.com/matrix-org/synapse/issues/11045)) - Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a user not in the `users` table. ([\#11053](https://github.com/matrix-org/synapse/issues/11053)) - Fix a bug introduced in Synapse v1.44.0 when logging errors during oEmbed processing. ([\#11061](https://github.com/matrix-org/synapse/issues/11061)) Internal Changes ---------------- - Add an 'approximate difference' method to `StateFilter`. ([\#10825](https://github.com/matrix-org/synapse/issues/10825)) - Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](https://github.com/matrix-org/synapse/issues/10970)) - Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly. ([\#10996](https://github.com/matrix-org/synapse/issues/10996)) - Ensure that cache config tests do not share state. ([\#11036](https://github.com/matrix-org/synapse/issues/11036))
| * 1.45.0rc2David Robertson2021-10-141-1/+1
| |
* | Resolve and share `state_groups` for all historical events in batch ↵Eric Eastwood2021-10-137-47/+113
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#10975) Resolve and share `state_groups` for all historical events in batch. This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch. This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room. For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch. --- How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example: **Before** (new `state_group` for every event 😬, very inefficient): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10 create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11 create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12 create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13 create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14 ``` **After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10 create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10 create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10 create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10 create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10 ```
* | Rearrange the user_directory's `_handle_deltas` function (#11035)David Robertson2021-10-131-57/+78
| | | | | | | | | | * Pull out `_handle_room_membership_event` * Discard excluded users early * Rearrange logic so the change is membership is effectively switched over. See PR for rationale.
* | Remove dead code from `MediaFilePaths` (#11056)Sean Quah2021-10-131-17/+0
| |
* | Merge remote-tracking branch 'origin/release-v1.45' into developDavid Robertson2021-10-131-2/+22
|\|
| * Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-135-83/+288
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Add type hints to synapse.events.*. (#11066)Patrick Cloke2021-10-139-145/+201
| | | | | | Except `synapse/events/__init__.py`, which will be done in a follow-up.
* | Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-136-171/+429
| | | | | | | | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Be more lenient when parsing the version for oEmbed responses. (#11065)Patrick Cloke2021-10-132-8/+7
| |
* | Mark Module API error imports as re-exported and mark Synapse as containing ↵reivilibre2021-10-132-2/+9
| | | | | | | | type annotations (#11054)
* | Merge remote-tracking branch 'origin/release-v1.45' into developPatrick Cloke2021-10-121-1/+1
|\|
| * Fix formatting string when oEmbed errors occur. (#11061)Patrick Cloke2021-10-121-1/+1
| |
* | Update `_wrap_in_base_path` type hints to preserve function arguments (#11055)Sean Quah2021-10-121-3/+6
| |
* | Merge remote-tracking branch 'origin/release-v1.45' into developErik Johnston2021-10-121-15/+67
|\|
| * Fix race in `MultiWriterIdGenerator` (#11045)Erik Johnston2021-10-121-15/+67
| | | | | | | | | | | | | | | | | | | | The race allowed the current position to advance too far when stream IDs are still being persisted. This happened when it received a new stream ID from a remote write between a new stream ID being allocated and it being added to the set of unpersisted stream IDs. Fixes #9424.
* | Add type hints to `synapse.storage.databases.main.client_ips` (#10972)Sean Quah2021-10-123-45/+116
| |
* | Merge tag 'v1.45.0rc1' into developBrendan Abolivier2021-10-121-1/+1
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.45.0rc1 (2021-10-12) ============================== **Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them. Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information. Features -------- - Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655)) - Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822)) - Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894)) - Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910)) - Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954)) - Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028)) Bugfixes -------- - Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922)) - Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924)) - Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927)) - Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947)) - Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956)) - Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960)) - Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962)) - Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981)) - Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982)) - Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995)) - Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002)) - Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003)) - Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042)) - Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877)) Improved Documentation ---------------------- - Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971)) - Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973)) - Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990)) - Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991)) Internal Changes ---------------- - Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888)) - Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892)) - Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895)) - Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902)) - Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903)) - Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915)) - Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916), [\#10959](https://github.com/matrix-org/synapse/issues/10959), [\#10985](https://github.com/matrix-org/synapse/issues/10985)) - Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926), [\#10940](https://github.com/matrix-org/synapse/issues/10940), [\#10986](https://github.com/matrix-org/synapse/issues/10986), [\#10987](https://github.com/matrix-org/synapse/issues/10987), [\#10988](https://github.com/matrix-org/synapse/issues/10988), [\#11010](https://github.com/matrix-org/synapse/issues/11010), [\#11011](https://github.com/matrix-org/synapse/issues/11011)) - Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934)) - Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935)) - Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936)) - Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939)) - Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945)) - Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958)) - Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961)) - Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963)) - Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966)) - Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974)) - Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983)) - Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992)) - Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993)) - Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994)) - Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004)) - Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005)) - Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006)) - Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017)) - Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019)) - Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021)) - Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023)) - When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034)) - Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043))
| * 1.45.0rc1Brendan Abolivier2021-10-121-1/+1
| |
* | Fix inconsistent behavior of `get_last_client_by_ip` (#10970)Sean Quah2021-10-121-4/+9
| | | | | | | | | | | | | | Make `get_last_client_by_ip` return the same dictionary structure regardless of whether the data has been persisted to the database. This change will allow slightly cleaner type hints to be applied later on.
* | Fix opentracing and Prometheus metrics for replication requests (#10996)Sean Quah2021-10-122-76/+86
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes two bugs to do with decorators not instrumenting `ReplicationEndpoint`'s `send_request` correctly. There are two decorators on `send_request`: Prometheus' `Gauge.track_inprogress()` and Synapse's `opentracing.trace`. `Gauge.track_inprogress()` does not have any support for async functions when used as a decorator. Since async functions behave like regular functions that return coroutines, only the creation of the coroutine was covered by the metric and none of the actual body of `send_request`. `Gauge.track_inprogress()` returns a regular, non-async function wrapping `send_request`, which is the source of the next bug. The `opentracing.trace` decorator would normally handle async functions correctly, but since the wrapped `send_request` is a non-async function, the decorator ends up suffering from the same issue as `Gauge.track_inprogress()`: the opentracing span only measures the creation of the coroutine and none of the actual function body. Using `Gauge.track_inprogress()` as a context manager instead of a decorator resolves both bugs.
* | Add an approximate difference method to StateFilters (#10825)reivilibre2021-10-121-1/+171
|/
* disallow-untyped-defs for synapse.push (#11023)David Robertson2021-10-115-10/+24
|
* Pass through `SynapseError`s that are raised from experimental ↵reivilibre2021-10-111-0/+9
| | | | | `check_event_allowed` callback of the module API (#11042) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Refactor MSC2716 `/batch_send` endpoint into separate handler functions (#10974)Eric Eastwood2021-10-083-283/+484
|
* Autodiscover oEmbed endpoint from returned HTML (#10822)Patrick Cloke2021-10-082-38/+100
| | | | | Searches the returned HTML for an oEmbed endpoint using the autodiscovery mechanism (`<link rel=...>`), and will request it to generate the preview.
* Annotate synapse.storage.util (#10892)David Robertson2021-10-086-65/+117
| | | | | Also mark `synapse.streams` as having has no untyped defs Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-089-44/+55
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix overwriting profile when making room public (#11003)David Robertson2021-10-081-29/+34
| | | This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
* Remove the deprecated BaseHandler. (#11005)Patrick Cloke2021-10-0823-215/+165
| | | | | | | | The shared ratelimit function was replaced with a dedicated RequestRatelimiter class (accessible from the HomeServer object). Other properties were copied to each sub-class that inherited from BaseHandler.
* Fix long-standing bug where `ReadWriteLock` could drop logging contexts (#10993)Sean Quah2021-10-081-2/+4
| | | | | | | | | | | Use `PreserveLoggingContext()` to ensure that logging contexts are not lost when exiting a read/write lock. When exiting a read/write lock, callbacks on a `Deferred` are triggered as a signal to any waiting coroutines. Any waiting coroutine that becomes runnable is likely to follow the Synapse logging context rules and will restore its own logging context, then either run to completion or await another `Deferred`, resetting the logging context in the process.
* Include exception in json logging (#11028)Nick Barrett2021-10-081-0/+6
|
* disallow-untyped-defs for synapse.server_notices (#11021)David Robertson2021-10-071-6/+2
|
* Don't alter directory entries for local users when setting a per-room ↵David Robertson2021-10-071-7/+13
| | | | | nickname (#11002) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Improve the logging in _auth_and_persist_outliers (#11010)Richard van der Hoff2021-10-071-1/+4
| | | Include the event ids being peristed
* Add a comment in _process_received_pdu (#11011)Richard van der Hoff2021-10-071-0/+3
|
* Catch AttributeErrors when calling registerProducer (#10995)Brendan Abolivier2021-10-071-1/+4
| | | Looks like the wrong exception type was caught in #10932.
* `disallow-untyped-defs` for `synapse.state` (#11004)David Robertson2021-10-063-4/+4
| | | | | * `disallow-untyped-defs` for `synapse.state` Much smaller than I was expecting!
* Add a spamchecker method to allow or deny 3pid invites (#10894)Brendan Abolivier2021-10-062-0/+47
| | | | | This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline. Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-0622-92/+60
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Add a spamchecker callback to allow or deny room joins (#10910)Brendan Abolivier2021-10-063-0/+57
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-061-1/+1
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* More types for synapse.util, part 1 (#10888)David Robertson2021-10-0610-73/+109
| | | | | | | | | | | | | | The following modules now pass `disallow_untyped_defs`: * synapse.util.caches.cached_call * synapse.util.caches.lrucache * synapse.util.caches.response_cache * synapse.util.caches.stream_change_cache * synapse.util.caches.ttlcache pass * synapse.util.daemonize * synapse.util.patch_inline_callbacks pass `no-untyped-defs` * synapse.util.versionstring Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
* Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)David Robertson2021-10-051-13/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two steps to rebuilding the user directory: 1. a scan over rooms, followed by 2. a scan over local users. The former reads avatars and display names from the `room_memberships` table and therefore contains potentially private avatars and display names. The latter reads from the the `profiles` table which only contains public data; moreover it will overwrite any private profiles that the rooms scan may have written to the user directory. This means that the rebuild could leak private user while the rebuild was in progress, only to later cover up the leaks once the rebuild had completed. This change skips over local users when writing user_directory rows when scanning rooms. Doing so means that it'll take longer for a rebuild to make local users searchable, which is unfortunate. I think a future PR can improve this by swapping the order of the two steps above. (And indeed there's more to do here, e.g. copying from `profiles` without going via Python.) Small tidy-ups while I'm here: * Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`. * Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`. * Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix logic flaw preventing tracking of MSC2716 events in existing room ↵Eric Eastwood2021-10-052-9/+6
| | | | | | | | | | | | | | | versions (#10962) We correctly allowed using the MSC2716 batch endpoint for the room creator in existing room versions but accidentally didn't track the events because of a logic flaw. This prevented you from connecting subsequent chunks together because it would throw the unknown batch ID error. We only want to process MSC2716 events when: - The room version supports MSC2716 - Any room where the homeserver has the `msc2716_enabled` experimental feature enabled and the event is from the room creator
* Merge branch 'master' into developBrendan Abolivier2021-10-051-1/+1
|\
| * 1.44.0 v1.44.0Brendan Abolivier2021-10-051-1/+1
| |
* | Ensure that we reject events which use rejected events for auth (#10956)Richard van der Hoff2021-10-051-0/+6
| | | | | | | | | | | | | | | | | | | | When we consider whether to accept events, we should not accept those which depend on rejected events for their auth events. This (together with earlier changes such as https://github.com/matrix-org/synapse/pull/10771 and https://github.com/matrix-org/synapse/pull/10896) forms a partial fix to https://github.com/matrix-org/synapse/issues/9595. There still remain code paths where we do not check the `auth_events` at all.
* | Host `cache_joined_hosts_for_event` to caller (#10986)Richard van der Hoff2021-10-051-10/+8
| | | | | | | | | | `_check_event_auth` is only called in two places, and only one of those sets `send_on_behalf_of`. Warming the cache isn't really part of auth anyway, so moving it out makes a lot more sense.
* | `_update_auth_events_and_context_for_auth`: add some comments (#10987)Richard van der Hoff2021-10-051-0/+26
| | | | | | Add some more comments about wtf is going on here.
* | `_check_event_auth`: move event validation earlier (#10988)Richard van der Hoff2021-10-051-4/+9
| | | | | | | | | | | | There's little point in doing a fancy state reconciliation dance if the event itself is invalid. Likewise, there's no point checking it again in `_check_for_soft_fail`.
* | Log stack traces when a missing opentracing span is detected (#10983)Sean Quah2021-10-051-0/+1
| | | | | | | | Make it easier to track down where opentracing spans are going missing by including stack traces in the logs.
* | Add functionality to remove deactivated users from the monthly_active_users ↵Hillery Shay2021-10-042-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table (#10947) * add test * add function to remove user from monthly active table in deactivate code * add function to remove user from monthly active table * add changelog entry * update changelog number * requested changes * update docstring on new function * fix lint error * Update synapse/storage/databases/main/monthly_active_users.py Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Merge tag 'v1.44.0rc3' into developBrendan Abolivier2021-10-042-3/+3
|\| | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc3 (2021-10-04) ============================== Bugfixes -------- - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933)) - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
| * 1.44.0rc3 v1.44.0rc3Brendan Abolivier2021-10-041-1/+1
| |
| * Fix error in `get_user_ip_and_agents` when fetching from the database (#10968)Sean Quah2021-10-011-2/+2
| |
| * Strip "join_authorised_via_users_server" from join events which do not need ↵Patrick Cloke2021-10-018-18/+36
| | | | | | | | | | | | | | | | it. (#10933) This fixes a "Event not signed by authorising server" error when transition room member from join -> join, e.g. when updating a display name or avatar URL for restricted rooms.
* | Consistently exclude from user_directory (#10960)David Robertson2021-10-042-31/+42
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-0415-55/+69
| |
* | Add type hints to filtering classes. (#10958)Patrick Cloke2021-10-012-45/+80
| |
* | Clean-up registration tests (#10945)Patrick Cloke2021-09-301-1/+3
| | | | | | | | Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent to properly configure auto-join rooms.
* | Strip "join_authorised_via_users_server" from join events which do not need ↵Patrick Cloke2021-09-308-18/+36
| | | | | | | | | | | | | | it. (#10933) This fixes a "Event not signed by authorising server" error when transition room member from join -> join, e.g. when updating a display name or avatar URL for restricted rooms.
* | Merge tag 'v1.44.0rc2' into developOlivier Wilkinson (reivilibre)2021-09-303-11/+8
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc2 (2021-09-30) ============================== Bugfixes -------- - Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error. ([\#10938](https://github.com/matrix-org/synapse/issues/10938)) - Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944)) Improved Documentation ---------------------- - Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))
| * 1.44.0rc2 v1.44.0rc2Olivier Wilkinson (reivilibre)2021-09-301-1/+1
| |
| * Fix event context for outliers in important MSC2716 spot (#10938)Eric Eastwood2021-09-301-9/+4
| | | | | | | | | | | | | | | | | | Fix event context for outlier causing failures in all of the MSC2716 Complement tests. The `EventContext.for_outlier` refactor happened in https://github.com/matrix-org/synapse/pull/10883 and this spot was left out.
| * Fix issue causing sending presence to ASes to fail (due to incomplete type ↵reivilibre2021-09-301-1/+3
| | | | | | | | annotations) (#10944)
* | Fix getTurnServer response: return an integer ttl (#10922)Lukas Lihotzki2021-09-301-1/+1
| | | | | | | | | | | | | | | | | | `ttl` must be an integer according to the OpenAPI spec: https://github.com/matrix-org/matrix-doc/blob/old_master/data/api/client-server/voip.yaml#L70 True division (`/`) returns a float instead (`"ttl": 7200.0`). Floor division (`//`) returns an integer, so the response is spec compliant. Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
* | Pass str to twisted's IReactorTCP (#10895)David Robertson2021-09-303-5/+20
| | | | | | | | | | | | | | This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job. Until that change is in a twisted release, we'll have to ignore the type of the `host` argument. I've raised #10899 to remind us to review the issue in a few months' time.
* | Refactor user directory tests (#10935)David Robertson2021-09-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Split `event_auth.check` into two parts (#10940)Richard van der Hoff2021-09-298-100/+140
| | | | | | | | | | | | | | | | | | | | | | | | | | Broadly, the existing `event_auth.check` function has two parts: * a validation section: checks that the event isn't too big, that it has the rught signatures, etc. This bit is independent of the rest of the state in the room, and so need only be done once for each event. * an auth section: ensures that the event is allowed, given the rest of the state in the room. This gets done multiple times, against various sets of room state, because it forms part of the state res algorithm. Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think that makes everything hard to follow. Instead, we split the function in two and call each part separately where it is needed.
* | Merge tag 'v1.44.0rc1' into developOlivier Wilkinson (reivilibre)2021-09-292-4/+12
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc1 (2021-09-29) ============================== Features -------- - Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\#10776](https://github.com/matrix-org/synapse/issues/10776)) - Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814), [\#10819](https://github.com/matrix-org/synapse/issues/10819)) - Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868), [\#10905](https://github.com/matrix-org/synapse/issues/10905)) - Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\#10898](https://github.com/matrix-org/synapse/issues/10898)) Bugfixes -------- - Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690)) - Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782)) - Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807)) - Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810)) - Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\#10827](https://github.com/matrix-org/synapse/issues/10827)) - Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843)) - Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859)) - Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\#10873](https://github.com/matrix-org/synapse/issues/10873)) - Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\#10875](https://github.com/matrix-org/synapse/issues/10875)) - Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\#10881](https://github.com/matrix-org/synapse/issues/10881)) - Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907)) - Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911)) - Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913)) Improved Documentation ---------------------- - Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845)) - Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865)) - Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869)) Internal Changes ---------------- - Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659)) - Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777)) - Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785), [\#10817](https://github.com/matrix-org/synapse/issues/10817)) - Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796)) - Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812), [\#10885](https://github.com/matrix-org/synapse/issues/10885), [\#10893](https://github.com/matrix-org/synapse/issues/10893), [\#10897](https://github.com/matrix-org/synapse/issues/10897)) - Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815)) - Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816)) - Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820)) - Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823)) - Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826)) - Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829)) - Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831), [\#10856](https://github.com/matrix-org/synapse/issues/10856)) - Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833)) - Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834)) - Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835)) - Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\#10838](https://github.com/matrix-org/synapse/issues/10838)) - Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\#10839](https://github.com/matrix-org/synapse/issues/10839)) - Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867)) - Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879)) - Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829). ([\#10880](https://github.com/matrix-org/synapse/issues/10880)) - Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883), [\#10884](https://github.com/matrix-org/synapse/issues/10884), [\#10896](https://github.com/matrix-org/synapse/issues/10896), [\#10901](https://github.com/matrix-org/synapse/issues/10901)) - Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231). ([\#10887](https://github.com/matrix-org/synapse/issues/10887)) - Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889)) - Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891)) - Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906)) - Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917)) - Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925)) - Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931)) F124520CEEE062448FE1C8442D2EFA2F32FBE047 <olivier@librepush.net>" [ultimate]
| * Fix exception responding to request that has been closed (#10932)Erik Johnston2021-09-281-3/+11
| | | | | | Introduced in #10905
| * 1.44.0rc1Erik Johnston2021-09-281-1/+1
| |
* | add event id to logcontext when handling incoming PDUs (#10936)Richard van der Hoff2021-09-291-1/+4
| |
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-2940-98/+106
| |
* | Implement MSC3069: Guest support on whoami (#9655)Travis Ralston2021-09-291-2/+6
| |
* | Use `RoomVersion` objects (#10934)Richard van der Hoff2021-09-294-48/+49
| | | | | | Various refactors to use `RoomVersion` objects instead of room version identifiers.
* | Fix empty `url_cache_thumbnails/yyyy-mm-dd/` directories being left behind ↵Sean Quah2021-09-291-31/+43
| | | | | | | | (#10924)
* | Ensure `(room_id, next_batch_id)` is unique to avoid cross-talk/conflicts ↵Eric Eastwood2021-09-283-4/+42
| | | | | | | | | | | | | | | | between batches (MSC2716) (#10877) Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) Part of https://github.com/matrix-org/synapse/issues/10737
* | Update utility code to handle C implementations of frozendict (#10902)Hillery Shay2021-09-281-1/+7
| | | | | | | | | | | | | | * update _handle_frozendict to work with c implementations of frozen dict * add changelog * add clarifying comment to _handle_frozendict
* | Drop backwards-compatibility support for "outlier" (#10903)Richard van der Hoff2021-09-282-25/+3
| | | | | | | | | | Before Synapse 1.31 (#9411), we relied on `outlier` being stored in the `internal_metadata` column. We can now assume nobody will roll back their deployment that far and drop the legacy support.
* | Inline `_check_event_auth` for outliers (#10926)Richard van der Hoff2021-09-281-57/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Inline `_check_event_auth` for outliers When we are persisting an outlier, most of `_check_event_auth` is redundant: * `_update_auth_events_and_context_for_auth` does nothing, because the `input_auth_events` are (now) exactly the event's auth_events, which means that `missing_auth` is empty. * we don't care about soft-fail, kicking guest users or `send_on_behalf_of` for outliers ... so the only thing that matters is the auth itself, so let's just do that. * `_auth_and_persist_fetched_events_inner`: de-async `prep` `prep` no longer calls any `async` methods, so let's make it synchronous. * Simplify `_check_event_auth` We no longer need to support outliers here, which makes things rather simpler. * changelog * lint
* | Clean-up type hints in server config (#10915)Patrick Cloke2021-09-282-55/+53
| | | | | | | | | | | | By using attrs instead of dicts to store configuration. Also updates some of the attrs classes to use proper type hints and auto_attribs.
* | Only do restricted join rules signature checks for room versions 8/9. (#10927)Patrick Cloke2021-09-281-1/+2
|/ | | | Otherwise the presence of a (bogus, unused) field could cause auth checks to fail.
* Encode JSON responses on a thread in C, mk2 (#10905)Erik Johnston2021-09-283-18/+75
| | | | | | | | | | Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library. Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this: 1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or 2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types. I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
* Avoid storing URL cache files in storage providers (#10911)Sean Quah2021-09-273-6/+16
| | | | | URL cache files are short-lived and it does not make sense to offload them (eg. to the cloud) or back them up.
* Fix race conditions when creating media store and config directories (#10913)Sean Quah2021-09-273-13/+5
|
* Document changes to schema version 61 - 64 (#10917)Eric Eastwood2021-09-241-0/+11
| | | | | As pointed out by @richvdh, https://github.com/matrix-org/synapse/pull/10838#discussion_r715424244 Retroactively summarize `61` - `64`
* Add a spamchecker callback to allow or deny room creation based on invites ↵Brendan Abolivier2021-09-242-4/+52
| | | | | | | (#10898) This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline. This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
* Stop trying to auth/persist events whose auth events we do not have. (#10907)Richard van der Hoff2021-09-241-8/+16
|
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-2440-100/+112
|
* Factor out common code for persisting fetched auth events (#10896)Richard van der Hoff2021-09-242-57/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Factor more stuff out of `_get_events_and_persist` It turns out that the event-sorting algorithm in `_get_events_and_persist` is also useful in other circumstances. Here we move the current `_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`, and then factor the sorting part out to `_auth_and_persist_fetched_events`. * `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment `get_event_auth` returns events with the outlier flag already set, so this is redundant (though we need to update a test where `get_event_auth` is mocked). * `_get_remote_auth_chain_for_event`: move existing-event tests earlier Move a couple of tests outside the loop. This is a bit inefficient for now, but a future commit will make it better. It should be functionally identical. * `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events` We can use the same codepath for persisting the events fetched as part of an auth chain as for those fetched individually by `_get_events_and_persist` for building the state at a backwards extremity. * `_get_remote_auth_chain_for_event`: use a dict for efficiency `_auth_and_persist_fetched_events` sorts the events itself, so we no longer need to care about maintaining the ordering from `get_event_auth` (and no longer need to sort by depth in `get_event_auth`). That means that we can use a map, making it easier to filter out events we already have, etc. * changelog * `_auth_and_persist_fetched_events`: improve docstring
* Simplify `_auth_and_persist_fetched_events` (#10901)Richard van der Hoff2021-09-241-69/+22
| | | | | Combine the two loops over the list of events, and hence get rid of `_NewEventInfo`. Also pass the event back alongside the context, so that it's easier to process the result.
* Add reactor to `SynapseRequest` and fix up types. (#10868)Erik Johnston2021-09-2410-45/+53
|
* Fix AuthBlocking check when requester is appservice (#10881)Jason Robinson2021-09-241-1/+1
| | | | | If the MAU count had been reached, Synapse incorrectly blocked appservice users even though they've been explicitly configured not to be tracked (the default). This was due to bypassing the relevant if as it was chained behind another earlier hit if as an elif. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* Improve typing in user_directory files (#10891)David Robertson2021-09-241-35/+89
| | | | | | | | | | | * Improve typing in user_directory files This makes the user_directory.py in storage pass most of mypy's checks (including `no-untyped-defs`). Unfortunately that file is in the tangled web of Store class inheritance so doesn't pass mypy at the moment. The handlers directory has already been mypyed. Co-authored-by: reivilibre <olivier@librepush.net>
* In `_purge_history_txn`, ensure that txn.fetchall has elements before ↵Kokokokoka2021-09-241-9/+13
| | | | | | accessing rows (#10690) This change adds a check for row existence before accessing row element, this should fix issue #10669 Signed-off-by: Vasya Boytsov vasiliy.boytsov@phystech.edu
* Allow `.` and `~` chars in registration tokens (#10887)Callum Brown2021-09-231-1/+1
| | | | Per updates to MSC3231 in order to use the same grammar as other identifiers.
* Factor out `_get_remote_auth_chain_for_event` from ↵Richard van der Hoff2021-09-231-52/+72
| | | | | | | | | | | | | | | | | `_update_auth_events_and_context_for_auth` (#10884) * Reload auth events from db after fetching and persisting In `_update_auth_events_and_context_for_auth`, when we fetch the remote auth tree and persist the returned events: load the missing events from the database rather than using the copies we got from the remote server. This is mostly in preparation for additional refactors, but does have an advantage in that if we later get around to checking the rejected status, we'll be able to make use of it. * Factor out `_get_remote_auth_chain_for_event` from `_update_auth_events_and_context_for_auth` * changelog
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-2321-44/+57
|
* Fix reactivated users not being added to the user directory (#10782)David Robertson2021-09-231-3/+6
| | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-2327-107/+128
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-2319-30/+30
|
* Factor out a separate `EventContext.for_outlier` (#10883)Richard van der Hoff2021-09-224-44/+20
| | | | | | Constructing an EventContext for an outlier is actually really simple, and there's no sense in going via an `async` method in the `StateHandler`. This also means that we can resolve a bunch of FIXMEs.
* Treat "\u0000" as "\u0020" for the purposes of message search (message ↵Hillery Shay2021-09-221-9/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | indexing) (#10820) * add test to check if null code points are being inserted * add logic to detect and replace null code points before insertion into db * lints * add license to test * change approach to null substitution * add type hint for SearchEntry * Add changelog entry Signed-off-by: H.Shay <shaysquared@gmail.com> * updated changelog * update chanelog message * remove duplicate changelog * Update synapse/storage/databases/main/events.py remove extra space Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * rename and move test file, update tests, delete old test file * fix typo in comments * update _find_highlights_in_postgres to replace null byte with space * replace null byte in sqlite search insertion * beef up and reorganize test for this pr * update changelog * add type hints and update docstring * check db engine directly vs using env variable * refactor tests to be less repetetive * move rplace logic into seperate function * requested changes * Fix typo. * Update synapse/storage/databases/main/search.py Co-authored-by: reivilibre <olivier@librepush.net> * Update changelog.d/10820.misc Co-authored-by: Aaron Raimist <aaron@raim.ist> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Aaron Raimist <aaron@raim.ist>
* Fix invalidating OTK count cache after claim (#10875)Tulir Asokan2021-09-221-0/+4
| | | | | | | The invalidation was missing in `_claim_e2e_one_time_key_returning`, which is used on SQLite 3.24+ and Postgres. This could break e2ee if nothing else happened to invalidate the caches before the keys ran out. Signed-off-by: Tulir Asokan <tulir@beeper.com>
* Ensure we mark sent knocks as outliers (#10873)Richard van der Hoff2021-09-221-0/+7
|
* Include more information in oEmbed previews. (#10819)Patrick Cloke2021-09-222-5/+46
| | | | | | | * Improved titles (fall back to the author name if there's not title) and include the site name. * Handle photo/video payloads. * Include the original URL in the Open Graph response. * Fix the expiration time (by properly converting from seconds to milliseconds).
* Fix /initialSync error due to unhashable `RoomStreamToken` (#10827)Sean Quah2021-09-222-6/+18
| | | | | | | | The deprecated /initialSync endpoint maintains a cache of responses, using parameter values as part of the cache key. When a `from` or `to` parameter is specified, it gets converted into a `StreamToken`, which contains a `RoomStreamToken` and forms part of the cache key. `RoomStreamToken`s need to be made hashable for this to work.
* Opt out of cache expiry for `get_users_who_share_room_with_user` (#10826)David Robertson2021-09-224-6/+28
| | | | * Allow LruCaches to opt out of time-based expiry * Don't expire `get_users_who_share_room` & friends
* Extend ModuleApi with the methods we'll need to reject spam based on …IP - ↵David Teller2021-09-222-8/+101
| | | | | | | | resolves #10832 (#10833) Extend ModuleApi with the methods we'll need to reject spam based on IP - resolves #10832 Signed-off-by: David Teller <davidt@element.io>
* Include outlier status in `str(event)` for V2/V3 events (#10879)Richard van der Hoff2021-09-221-22/+12
| | | | | | | | | | | | | | | | | | | | | I meant to do this before, in #10591, but because I'm stupid I forgot to do it for V2 and V3 events. I've factored the common code out to `EventBase` to save us having two copies of it. This means that for `FrozenEvent` we replace `self.get("event_id", None)` with `self.event_id`, which I think is safe. `get()` is an alias for `self._dict.get()`, whereas `event_id()` is an `@property` method which looks up `self._event_id`, which is populated during construction from the same dict. We don't seem to rely on the fallback, because if the `event_id` key is absent from the dict then construction of the `EventBase` object will fail. Long story short, the only way this could change behaviour is if `event_dict["event_id"]` is changed *after* the `EventBase` object is constructed without updating the `_event_id` field, or vice versa - either of which would be very problematic anyway and the behavior of `str(event)` is the least of our worries.
* Track why we're evicting from caches (#10829)David Robertson2021-09-223-15/+30
| | | So we can see distinguish between "evicting because the cache is too big" and "evicting because the cache entries haven't been recently used".
* Rename MSC2716 things from `chunk` to `batch` to match `/batch_send` ↵Eric Eastwood2021-09-2112-117/+161
| | | | | | | | endpoint (#10838) See https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684574497 Dropping support for older MSC2716 room versions so we don't have to worry about supporting both chunk and batch events.
* Add type hints for event streams. (#10856)Patrick Cloke2021-09-2113-47/+113
|
* Add types to http.site (#10867)Erik Johnston2021-09-211-18/+22
|
* Refactor oEmbed previews (#10814)Patrick Cloke2021-09-212-199/+272
| | | | | | | | | | | | | The major change is moving the decision of whether to use oEmbed further up the call-stack. This reverts the _download_url method to being a "dumb" functionwhich takes a single URL and downloads it (as it was before #7920). This also makes more minor refactorings: * Renames internal variables for clarity. * Factors out shared code between the HTML and rich oEmbed previews. * Fixes tests to preview an oEmbed image.
* Allow Synapse Admin API's Room Search to accept non-ASCII characters (#10859)Hillery Shay2021-09-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add tests for checking if room search works with non-ascii char * change encoding on parse_string to UTF-8 * lints * properly encode search term * lints * add changelog file * update changelog number * set changelog entry filetype to .bugfix * Revert "set changelog entry filetype to .bugfix" This reverts commit be8e5a314251438ec4ec7dbc59ba32162c93e550. * update changelog message and file type * change parse_string default encoding back to ascii and update room search admin api calll to parse string * refactor tests * Update tests/rest/admin/test_room.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Rename `/batch_send` query parameter from `?prev_event` to more obvious ↵Eric Eastwood2021-09-211-7/+9
| | | | | | | usage with `?prev_event_id` (MSC2716) (#10839) As mentioned in https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r705872887 and https://github.com/matrix-org/synapse/issues/10737
* Merge tag 'v1.43.0' into developDavid Robertson2021-09-211-1/+1
|\
| * 1.43.0David Robertson2021-09-211-1/+1
| |
* | Always add local users to the user directory (#10796)David Robertson2021-09-215-42/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It's a simplification, but one that'll help make the user directory logic easier to follow with the other changes upcoming. It's not strictly required for those changes, but this will help simplify the resulting logic that listens for `m.room.member` events and generally make the logic easier to follow. This means the config option `search_all_users` ends up controlling the search query only, and not the data we store. The cost of doing so is an extra row in the `user_directory` and `user_directory_search` tables for each local user which - belongs to no public rooms - belongs to no private rooms of size ≥ 2 I think the cost of this will be marginal (since they'll already have entries in `users` and `profiles` anyway). As a small upside, a homeserver whose directory was built with this change can toggle `search_all_users` without having to rebuild their directory. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Allow sending a membership event to unban a user (#10807)Aaron Raimist2021-09-211-1/+1
| | | | | | | | | | * Allow membership event to unban user Signed-off-by: Aaron Raimist <aaron@raim.ist>
* | Require type hints in the handlers module. (#10831)Patrick Cloke2021-09-2033-295/+190
| | | | | | | | | | | | | | Adds missing type hints to methods in the synapse.handlers module and requires all methods to have type hints there. This also removes the unused construct_auth_difference method from the FederationHandler.
* | Fix remove_stale_pushers job on SQLite. (#10843)reivilibre2021-09-208-23/+26
| |
* | Merge branch 'release-v1.43' into developDavid Robertson2021-09-172-127/+142
|\|
| * 1.43.0rc2David Robertson2021-09-171-1/+1
| |
| * Cleanup opentracing logging for syncs (#10828)Erik Johnston2021-09-151-126/+141
| | | | | | | | | | We added a bunch of spans in #10704, but this ended up adding a lot of redundant spans for rooms where nothing changed, so instead we only start the span if there might be something interesting going on.
* | Add type hints to state database module. (#10823)Patrick Cloke2021-09-154-72/+131
| |
* | Add missing type hints to non-client REST servlets. (#10817)Patrick Cloke2021-09-1524-94/+166
| | | | | | | | Including admin, consent, key, synapse, and media. All REST servlets (the synapse.rest module) now require typed method definitions.
* | Use direct references for some configuration variables (part 2) (#10812)Patrick Cloke2021-09-1515-45/+50
| |
* | Verify `?chunk_id` actually corresponds to an insertion event that exists ↵Eric Eastwood2021-09-153-1/+50
| | | | | | | | (MSC2716) (#10776)
* | Split out `/batch_send` meta events to their own fields (MSC2716) (#10777)Eric Eastwood2021-09-151-11/+18
| |
* | Make StateFilter frozen so we can hash it (#10816)reivilibre2021-09-141-13/+32
| | | | | | Also enables Mypy for related tests.
* | Prevent logging context going missing on federation request timeout (#10810)Sean Quah2021-09-141-4/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In `MatrixFederationHttpClient._send_request()`, we make a HTTP request using an `Agent`, wrap that request in a timeout and await the resulting `Deferred`. On its own, the `Agent` performing the HTTP request correctly stashes and restores the logging context while waiting. The addition of the timeout introduces a path where the logging context is not restored when execution resumes. To address this, we wrap the timeout `Deferred` in a `make_deferred_yieldable()` to stash the logging context and restore it on completion of the `await`. However this is not sufficient, since by the time we construct the timeout `Deferred`, the `Agent` has already stashed and cleared the logging context when using `make_deferred_yieldable()` to produce its `Deferred` for the request. Hence, we wrap the `Agent` request in a `run_in_background()` to "fork" and preserve the logging context so that we can stash and restore it when `await`ing the timeout `Deferred`. This approach is similar to the one used with `defer.gatherResults`. Note that the code is still not fully correct. When a timeout occurs, the request remains running in the background (existing behavior which is nothing to do with the new call to `run_in_background`) and may re-start the logging context after it has finished.
* | Merge tag 'v1.43.0rc1' into developAndrew Morgan2021-09-141-1/+1
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.43.0rc1 (2021-09-14) This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30. Features -------- - Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566)) - Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643)) - Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712)) - Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714), [\#10759](https://github.com/matrix-org/synapse/issues/10759)) - Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772)) Bugfixes -------- - Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658)) - Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730)) - Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743)) - Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757)) - Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733)) - Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10760](https://github.com/matrix-org/synapse/issues/10760)) - Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774)) - Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783)) Improved Documentation ---------------------- - Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556)) - Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648)) - Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735)) - Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758)) - Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html). ([\#10795](https://github.com/matrix-org/synapse/issues/10795)) - Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804)) Deprecations and Removals ------------------------- - Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693)) Internal Changes ---------------- - Add OpenTracing logging to help debug stuck messages (as described by issue [#9424](https://github.com/matrix-org/synapse/issues/9424)). ([\#10704](https://github.com/matrix-org/synapse/issues/10704)) - Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601)) - Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697)) - Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707), [\#10728](https://github.com/matrix-org/synapse/issues/10728), [\#10736](https://github.com/matrix-org/synapse/issues/10736)) - Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727)) - Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738)) - Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744), [\#10745](https://github.com/matrix-org/synapse/issues/10745), [\#10746](https://github.com/matrix-org/synapse/issues/10746), [\#10771](https://github.com/matrix-org/synapse/issues/10771), [\#10773](https://github.com/matrix-org/synapse/issues/10773), [\#10781](https://github.com/matrix-org/synapse/issues/10781)) - Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748)) - Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750)) - Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752)) - Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753)) - Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754), [\#10755](https://github.com/matrix-org/synapse/issues/10755), [\#10756](https://github.com/matrix-org/synapse/issues/10756), [\#10780](https://github.com/matrix-org/synapse/issues/10780), [\#10784](https://github.com/matrix-org/synapse/issues/10784)) - Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775)) - Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778)) - Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779)) - Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788)) - Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789)) - Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798)) - Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799))
| * 1.43.0rc1Andrew Morgan2021-09-141-1/+1
| |
* | Convert media repo's FileInfo to attrs. (#10785)Patrick Cloke2021-09-144-108/+139
| | | | | | | | This is mostly an internal change, but improves type hints in the media code.
* | Name the type of token in "Invalid token" messages (#10815)David Robertson2021-09-143-6/+6
|/ | | | | | I had one of these error messages yesterday and assumed it was an invalid auth token (because that was an HTTP query parameter in the test) I was working on. In fact, it was an invalid next batch token for syncing.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-1364-127/+139
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Fix copy-paste error in the password section of the sample-config. (#10804)BramvdnHeuvel2021-09-131-1/+1
|
* Use the BaseReporter super-class for _WrappedRustReporter. (#10799)Patrick Cloke2021-09-131-1/+11
| | | | This fixes mypy errors with jaeger-client >= 4.7.0 and should be a no-op for versions before that.
* Add types to synapse.util. (#10601)reivilibre2021-09-1037-240/+333
|
* Handle room upgrades for spaces (#10774)Patrick Cloke2021-09-101-2/+17
| | | | | By copying the `room_type` field of the create event and migrating any non-empty `m.space.child` events to the new room that is created.
* Easy refactors of the user directory (#10789)David Robertson2021-09-105-34/+49
| | | No functional changes here. This came out as I was working to tackle #5677
* Ask consent on SSO registration with default mxid (#10733)AndrewFerr2021-09-102-23/+62
| | | | | | Fixes #10732: consent flow skipped during SSO user registration if username is left at default Signed-off-by: Andrew Ferrazzutti fair@miscworks.net
* Don't needlessly batch in `add_event_to_cache` (#10784)Erik Johnston2021-09-101-27/+24
| | | | | We've already batched up the events previously, and assume in other places in the events.py file that we have. Removing this makes it easier to adjust the batch sizes in one place.
* Remove unstable MSC2858 API, including `experimental.msc2858_enabled` config ↵Sean2021-09-097-69/+11
| | | | | option (#10693) Signed-off-by: Sean Quah <seanq@element.io>
* Prefer room v9 for restricted rooms. (#10772)Patrick Cloke2021-09-091-1/+1
| | | | | Hint to clients via the room capabilities API (MSC3244) that room version 9 should be preferred for creating a room with restricted join rules (instead of room version 8).
* Get rid of `_auth_and_persist_event` (#10781)Richard van der Hoff2021-09-081-60/+30
| | | This is only called in two places, and the code seems much clearer without it.
* Document that /spaces & /hierarchy can be routed to workers. (#10648)Patrick Cloke2021-09-081-24/+18
| | | Also refactors some of the registration of endpoints on workers.
* Skip handling of push actions for outlier events (#10780)Erik Johnston2021-09-081-4/+17
| | | | Outlier events don't ever have push actions associated with them, so we can skip some expensive queries during event persistence.
* Add a constant for m.federate. (#10775)Patrick Cloke2021-09-087-10/+27
|
* Change logging of puppeted requests to better differentiate users (#10779)Erik Johnston2021-09-081-1/+1
| | | This used to be a comma and got accidentally changed to a period in #9654, but a pipe character is more easier to parse visually.
* Request JSON for oEmbed requests (and ignore XML only providers). (#10759)Patrick Cloke2021-09-082-7/+43
| | | | | | | | This adds the format to the request arguments / URL to ensure that JSON data is returned (which is all that Synapse supports). This also adds additional error checking / filtering to the configuration file to ignore XML-only providers.
* Add some assertions about outliers (#10773)Richard van der Hoff2021-09-081-71/+77
| | | | I think I have finally teased apart the codepaths which handle outliers, and those that handle non-outliers. Let's add some assertions to demonstrate my newfound knowledge.
* Persist auth events before the events that rely on them (#10771)Richard van der Hoff2021-09-081-36/+65
| | | | | | | If we're persisting an event E which has auth_events A1, A2, then we ought to make sure that we correctly auth and persist A1 and A2, before we blindly accept E. This PR does part of that - it persists the auth events first - but it does not fully solve the problem, because we still don't check that the auth events weren't rejected.
* Merge branch 'master' into developAzrenbeth2021-09-071-1/+1
|\
| * 1.42.0Azrenbeth2021-09-071-1/+1
| |
* | Use attrs internally for the URL preview code & add documentation. (#10753)Patrick Cloke2021-09-071-42/+79
| |
* | Return stripped m.space.child events via the space summary. (#10760)Patrick Cloke2021-09-071-14/+12
| | | | | | | | | | | | | | | | | | The full event content cannot be trusted from this API (as no auth chain, etc.) is processed over federation. Returning the full event content was a bug as MSC2946 specifies that only the stripped state should be returned. This also avoids calculating aggregations / annotations which go unused.
* | Underscore-prefix private fields in `FederationEventHandler` (#10746)Richard van der Hoff2021-09-071-71/+73
| |
* | Merge tag 'v1.42.0rc2' into developBrendan Abolivier2021-09-066-1/+75
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.42.0rc2 (2021-09-06) ============================== This version of Synapse removes deprecated room-management admin APIs, removes out-of-date email pushers, and improves error handling for fallback templates for user-interactive authentication. For more information on these points, server administrators are encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375). ([\#10747](https://github.com/matrix-org/synapse/issues/10747)) Internal Changes ---------------- - Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768))
| * 1.42.0rc2Brendan Abolivier2021-09-061-1/+1
| |
| * Add a warning when using deprecated template_dir settings (#10768)Brendan Abolivier2021-09-063-0/+41
| | | | | | The deprecation itself happened in #10596 which shipped with Synapse v1.41.0. However, it doesn't seem fair to suddenly drop support for these settings in ~4-6w without being more vocal about said deprecation.
| * Support MSC3375: room version 9. (#10747)Patrick Cloke2021-09-032-0/+33
| |
* | Add logging to help debug #9424 (#10704)Azrenbeth2021-09-061-13/+54
| |
* | Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-065-9/+113
| |
* | Stop using BaseHandler in `FederationEventHandler` (#10745)Richard van der Hoff2021-09-061-9/+10
| | | | | | | | It's now only used in a couple of places, so we can drop it altogether.
* | Allow `room_alias_name` parameter to be handled by /createRoom calls on ↵Andrew Morgan2021-09-061-2/+2
| | | | | | | | workers (#10757)
* | Ease searching for M_TOO_LARGE-related error codes (#10750)Andrew Morgan2021-09-061-9/+6
| |
* | Move `maybe_kick_guest_users` out of `BaseHandler` (#10744)Richard van der Hoff2021-09-068-85/+124
| | | | | | This is part of my ongoing war against BaseHandler. I've moved kick_guest_users into RoomMemberHandler (since it calls out to that handler anyway), and split maybe_kick_guest_users into the two places it is called.
* | Speed up persisting redacted events (#10756)Erik Johnston2021-09-061-11/+11
| |
* | Allow room creator to send MSC2716 related events in existing room versions ↵Eric Eastwood2021-09-043-10/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#10566) * Allow room creator to send MSC2716 related events in existing room versions Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869 Restoring `get_create_event_for_room_txn` from, https://github.com/matrix-org/synapse/pull/10245/commits/44bb3f0cf5cb365ef9281554daceeecfb17cc94d * Add changelog * Stop people from trying to redact MSC2716 events in unsupported room versions * Populate rooms.creator column for easy lookup > From some [out of band discussion](https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`). > > https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642 * Remove and switch away from get_create_event_for_room_txn * Fix no create event being found because no state events persisted yet * Fix and add tests for rooms creator bg update * Populate rooms.creator field for easy lookup Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values * Add changelog * Fix usage * Remove extra delta already included in #10697 * Don't worry about setting creator for invite * Only iterate over rows missing the creator See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898 * Use constant to fetch room creator field See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029 * More protection from other random types See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853 * Move new background update to end of list See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181 * Fix query casing * Fix ambiguity iterating over cursor instead of list Fix `psycopg2.ProgrammingError: no results to fetch` error when tests run with Postgres. ``` SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room ``` --- We use `txn.fetchall` because it will return the results as a list or an empty list when there are no results. Docs: > `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used: > > https://www.psycopg.org/docs/cursor.html#cursor-iterable And I'm guessing iterating over a raw cursor does something weird when there are no results. --- Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530 ``` tests.test_visibility.FilterEventsForServerTestCase.test_large_room =============================================================================== [FAIL] Traceback (most recent call last): File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column self.get_success( File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success return self.successResultOf(d) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf self.fail( twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead: Traceback (most recent call last): File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback self._startRunCallbacks(fail) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks self._runCallbacks() File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks current.result = callback( # type: ignore[misc] File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult current_context.run(_inlineCallbacks, r, gen, status) --- <exception caught here> --- File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks result = current_context.run( File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update await self._do_background_update(desired_duration_ms) File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update items_updated = await update_handler(progress, batch_size) File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column end = await self.db_pool.runInteraction( File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction result = await self.runWithConnection( File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection return await make_deferred_yieldable( File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks current.result = callback( # type: ignore[misc] File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda> d.addCallback(lambda x: function(*args, **kwargs)) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection compat.reraise(excValue, excTraceback) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction return function(*args, **kwargs) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise raise exception.with_traceback(traceback) File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection result = func(conn, *args, **kw) File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func return func(db_conn, *args, **kwargs) File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction r = func(cursor, *args, **kwargs) File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn for room_id, event_json in txn: psycopg2.ProgrammingError: no results to fetch ``` * Move code not under the MSC2716 room version underneath an experimental config option See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909 * Add ordering to rooms creator background update See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277 * Add comment to better document constant See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458 * Use constant field
* | Speed up MultiWriterIdGenerator when lots of IDs are in flight. (#10755)Erik Johnston2021-09-031-2/+3
| |
* | Add a partial index to `presence_stream` to speed up startups (#10748)Sean2021-09-032-1/+40
| | | | | | Signed-off-by: Sean Quah <seanq@element.io>
* | Use `execute_values` more in PostgreSQL (#10754)Erik Johnston2021-09-031-19/+42
| | | | | | `execute_values` is a faster version of `execute_batch`.
* | Raise an error if an unknown preset is used to create a room. (#10738)Patrick Cloke2021-09-031-1/+6
| | | | | | | | Raises a 400 error instead of a 500 if an unknown preset is passed from a client to create a room.
* | Additional type hints for client REST servlets (part 5) (#10736)Patrick Cloke2021-09-036-68/+142
| | | | | | | | Additionally this enforce type hints on all function signatures inside of the synapse.rest.client package.
* | Fix bug with reusing 'txn' when persisting event. (#10743)Erik Johnston2021-09-031-1/+7
| | | | | | | | This will only happen when a server has multiple out of band membership events in a single room.
* | Ignore rooms with unknown room versions in the spaces summary. (#10727)Patrick Cloke2021-09-011-2/+14
| | | | | | | | This avoids breaking the entire endpoint if a room with an unsupported room version is encountered.
* | Consider the `origin_server_ts` of the `m.space.child` event when ordering ↵Patrick Cloke2021-09-011-7/+8
| | | | | | | | | | | | | | | | | | rooms. (#10730) This updates the ordering of the returned events from the spaces summary API to that defined in MSC2946 (which updates MSC1772). Previously a step was skipped causing ordering to be inconsistent with clients.
* | Additional type hints for client REST servlets (part 4) (#10728)Patrick Cloke2021-09-018-100/+144
| |
* | Populate `rooms.creator` field for easy lookup (#10697)Eric Eastwood2021-09-014-4/+114
| | | | | | | | | | | | Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values
* | Merge tag 'v1.42.0rc1' into developOlivier Wilkinson (reivilibre)2021-09-012-2/+3
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.42.0rc1 (2021-09-01) ============================== Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142)) - Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452)) - Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524)) - Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725)) Bugfixes -------- - Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232)) - Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561)) - Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734)) - Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593)) - Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644)) - Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654)) - Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677)) - Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679)) - Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684)) - Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703)) - Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713)) Improved Documentation ---------------------- - Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192)) - Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595)) - Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639)) - Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708)) - Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711)) Deprecations and Removals ------------------------- - Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830)) Internal Changes ---------------- - Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608)) - Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640)) - Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621)) - Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627)) - Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642)) - Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630)) - Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645)) - Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647)) - Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672)) - Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662)) - Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664)) - Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674)) - Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667)) - Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686)) - Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692)) - Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))
| * 1.42.0rc1Olivier Wilkinson (reivilibre)2021-09-011-1/+1
| |
| * Fix iteration in _remove_deleted_email_pushers background job. (#10734)Andrew Morgan2021-09-011-1/+2
| |
* | Correctly include room avatars in email notifications (#10658)David Robertson2021-09-011-1/+23
| | | | | | | | | | | | | | | | Judging by the template, this was intended ages ago, but we never actually passed an avatar URL to the template. So let's provide one. Closes #1546. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Skip the final GC on shutdown to improve restart times (#10712)Sean2021-09-011-0/+7
| | | | | | | | | | | | | | | | | | | | Use `gc.freeze()` on exit to exclude all existing objects from the final GC. In testing, this sped up shutdown by up to a few seconds. `gc.freeze()` runs in constant time, so there is little chance of performance regression. Signed-off-by: Sean Quah <seanq@element.io>
* | Allow configuration of the oEmbed URLs. (#10714)Patrick Cloke2021-08-315-142/+339
| | | | | | | | | | This adds configuration options (under an `oembed` section) to configure which URLs are matched to use oEmbed for URL previews.
* | Additional type hints for the client REST servlets (part 3). (#10707)Patrick Cloke2021-08-316-150/+305
|/