summary refs log tree commit diff
Commit message (Collapse)AuthorAgeFilesLines
* Add support for the stable version of MSC2778 (#11335)Tulir Asokan2021-11-154-4/+16
| | | | | | | | | * Add support for the stable version of MSC2778 Signed-off-by: Tulir Asokan <tulir@maunium.net> * Expect m.login.application_service in login and password provider tests Signed-off-by: Tulir Asokan <tulir@maunium.net>
* Remove unused tables `room_stats_historical` and `user_stats_historical` ↵Shay2021-11-123-1/+24
| | | | | | | | | | | | | | | | | (#11280) * remove unused tables room_stats_historical and user_stats_historical * update changelog number * Bump schema compat version comment * make linter happy * Update comment to give more info Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* Annotations for state_deltas.py (#11316)David Robertson2021-11-123-4/+17
| | | | | I was sad that I couldn't do better for `_curr_state_delta_stream_cache`. At least it's explicitly called out in a comment with #TODO.
* Rollback #11322 due to wrong syntax in mypy.ini. (#11332)Patrick Cloke2021-11-122-2/+15
| | | | This was only checking the __init__ files in modules instead of all files in a module, which don't pass yet.
* Test room alias deletion (#11327)David Robertson2021-11-125-31/+91
| | | | | | | | | | * Prefer `HTTPStatus` over plain `int` This is an Opinion that no-one has seemed to object to yet. * `--disallow-untyped-defs` for `tests.rest.client.test_directory` * Improve synapse's annotations for deleting aliases * Test case for deleting a room alias * Changelog
* Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-123-4/+56
| | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* Add type hints to media repository storage module (#11311)Patrick Cloke2021-11-124-62/+89
|
* Attempt to annotate events_forward_extremities (#11314)David Robertson2021-11-124-8/+17
| | | | | | * Make DataStore inherit from EventForwardExtremitiesStore before CacheInvalidationWorkerStore the former implicitly inherits from the latter, so they should be ordered like this when used.
* Misc typing fixes for `tests`, part 1 of N (#11323)David Robertson2021-11-127-29/+53
| | | | | | | | | | * Annotate HomeserverTestCase.servlets * Correct annotation of federation_auth_origin * Use AnyStr custom_headers instead of a Union This allows (str, str) and (bytes, bytes). This disallows (str, bytes) and (bytes, str) * DomainSpecificString.SIGIL is a ClassVar
* Generalize the disallowed_untyped_defs in mypy.ini (#11322)Patrick Cloke2021-11-122-14/+3
|
* Add more type hints to synapse.util. (#11321)Patrick Cloke2021-11-123-15/+24
|
* Suggest using `/etc/matrix-synapse/conf.d/` for configuration with Debian ↵reivilibre2021-11-122-0/+7
| | | | packages (#11281)
* Convert delete room admin API to async endpoint (#11223)Dirk Klimpel2021-11-129-91/+1317
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Annotations for user_erasure_store (#11313)David Robertson2021-11-113-5/+9
| | | | I'm not sure why this was excluded---it seemed to be passing for me. But it's easy enough to fixup.
* Get db signatures file to pass mypy (#11312)David Robertson2021-11-114-10/+8
|
* Correct type hint for room_batch.py (#11310)David Robertson2021-11-113-11/+12
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Move sql file for `remove_deleted_devices_from_device_inbox` into v65 (#11303)Dirk Klimpel2021-11-112-1/+2
|
* Add type hints to some storage classes (#11307)Patrick Cloke2021-11-119-54/+116
|
* Fix error in thumbnail generation (#11288)Neeeflix2021-11-102-4/+5
| | | Signed-off-by: Jonas Zeunert <jonas@zeunert.org>
* Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-1017-190/+223
|
* Add type hints to synapse._scripts (#11297)Patrick Cloke2021-11-105-20/+30
|
* Add documentation page stubs for Single Sign-On, SAML and CAS pages (#11298)Andrew Morgan2021-11-105-3/+25
|
* Remove redundant parameters on `_check_event_auth` (#11292)Richard van der Hoff2021-11-103-12/+1
| | | as of #11012, these parameters are unused.
* Merge tag 'v1.47.0rc2' into developOlivier Wilkinson (reivilibre)2021-11-1060-60/+101
|\ | | | | | | | | | | | | | | Synapse 1.47.0rc2 (2021-11-10) ============================== This fixes an issue with publishing the Debian packages for 1.47.0rc1. It is otherwise identical to 1.47.0rc1.
| * Changelog tweak from feedback v1.47.0rc2Olivier Wilkinson (reivilibre)2021-11-101-1/+1
| |
| * Move Debian changelog entries to rc2 since rc1 was not publishedOlivier Wilkinson (reivilibre)2021-11-101-7/+4
| |
| * 1.47.0rc2Olivier Wilkinson (reivilibre)2021-11-103-1/+14
| |
| * Correct the Debian changelogOlivier Wilkinson (reivilibre)2021-11-101-1/+1
| |
| * Update __init__.py v1.47.0rc1Olivier Wilkinson (reivilibre)2021-11-091-1/+1
| |
| * Changelog tweaks from reviewOlivier Wilkinson (reivilibre)2021-11-091-5/+5
| |
| * Make Deprecations and Removals more prominentOlivier Wilkinson (reivilibre)2021-11-091-7/+7
| |
| * Changelog tweaksOlivier Wilkinson (reivilibre)2021-11-091-6/+6
| |
| * 1.47.0rc1Olivier Wilkinson (reivilibre)2021-11-0959-59/+90
| |
* | Fix typos in the `username_available` admin API documentation. (#11286)Stanislav Motylkov2021-11-092-1/+2
| |
* | Split out federated PDU retrieval into a non-cached version (#11242)Eric Eastwood2021-11-092-22/+59
| | | | | | | | Context: https://github.com/matrix-org/synapse/pull/11114/files#r741643968
* | Ignore missing imports for parameterized. (#11285)Patrick Cloke2021-11-092-0/+4
| | | | | | | | This was due to a conflict between #11282, which changed mypy configuration, and #11228, a normal change.
* | Require mypy for synapse/ & tests/ unless excluded (#11282)Dan Callahan2021-11-093-77/+168
| | | | | | Signed-off-by: Dan Callahan <danc@element.io>
* | Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-096-17/+78
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Allow admins to proactively block rooms (#11228)David Robertson2021-11-096-21/+103
| | | | | | | | Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-0915-110/+680
| | | | | | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* | Fix typo in `RelationAggregationPaginationServlet` error response (#11278)Andrew Morgan2021-11-092-1/+4
|/
* Include cross-signing signatures when syncing remote devices for the first ↵Erik Johnston2021-11-093-86/+277
| | | | | | | time (#11234) When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results. c.f. #11159
* Require body for read receipts with user-agent exceptions (#11157)rogersheu2021-11-093-3/+40
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Rename to more clear `get_insertion_event_id_by_batch_id` (MSC2716) (#11244)Eric Eastwood2021-11-084-3/+4
| | | | | `get_insertion_event_by_batch_id` -> `get_insertion_event_id_by_batch_id` Split out from https://github.com/matrix-org/synapse/pull/11114
* Add some background update admin APIs (#11263)Erik Johnston2021-11-088-18/+468
| | | Fixes #11259
* Fix typo in comment from #11255. (#11276)Patrick Cloke2021-11-082-1/+2
|
* Default value for `public_baseurl` (#11210)Richard van der Hoff2021-11-0816-73/+62
| | | | | We might as well use a default value for `public_baseurl` based on `server_name` - in many cases, it will be correct.
* Address review feedback from #11269 (#11273)Dan Callahan2021-11-082-2/+3
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Blacklist new sytest validation test (#11270)Erik Johnston2021-11-082-0/+4
|
* Handle federation inbound instances being killed more gracefully (#11262)Erik Johnston2021-11-083-10/+27
| | | | | | | | | | | | | | | | | * Make lock better handle process being killed If the process gets killed and restarted (so that it didn't have a chance to drop its locks gracefully) then there may still be locks in the DB that are for the same instance that haven't yet timed out but are safe to delete. We handle this case by a) checking if the current instance already has taken out the lock, and b) if not then ignoring locks that are for the same instance. * Periodically check for old staged events This is to protect against other instances dying and their locks timing out.
* Minor cleanup to Debian packaging (#11269)Dan Callahan2021-11-0711-53/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove unused Vagrant scripts * Change package Architecture to any * Preinstall the wheel package when building venvs. Addresses the following warnings during Debian builds: Using legacy 'setup.py install' for jaeger-client, since package 'wheel' is not installed. Using legacy 'setup.py install' for matrix-synapse-ldap3, since package 'wheel' is not installed. Using legacy 'setup.py install' for opentracing, since package 'wheel' is not installed. Using legacy 'setup.py install' for psycopg2, since package 'wheel' is not installed. Using legacy 'setup.py install' for systemd-python, since package 'wheel' is not installed. Using legacy 'setup.py install' for pympler, since package 'wheel' is not installed. Using legacy 'setup.py install' for threadloop, since package 'wheel' is not installed. Using legacy 'setup.py install' for thrift, since package 'wheel' is not installed. * Allow /etc/default/matrix-synapse to be missing Per the systemd.exec manpage, prefixing an EnvironmentFile with "-": > indicates that if the file does not exist, it will not be read and no > error or warning message is logged. Signed-off-by: Dan Callahan <danc@element.io>
* Add doc to integrate synapse with LemonLDAP OIDC (#11257)Julian2021-11-052-0/+39
| | | | Co-authored-by: David Robertson <david.m.robertson1@gmail.com> Co-authored-by: Julian Vanden Broeck <julian.vandenbroeck@dalibo.com>
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-053-11/+82
| | | Fixes #11252
* Make minor correction to type of auth_checkers callbacks (#11253)reivilibre2021-11-043-2/+5
|
* Additional test for `cachedList` (#11246)Richard van der Hoff2021-11-042-0/+44
| | | | | I was trying to understand how `cachedList` works, and ended up writing this extra test. I figure we may as well keep it.
* Track ongoing event fetches correctly in the presence of failure (#11240)Sean Quah2021-11-042-22/+35
| | | | | | When an event fetcher aborts due to an exception, `_event_fetch_ongoing` must be decremented, otherwise the event fetcher would never be replaced. If enough event fetchers were to fail, no more events would be fetched and requests would get stuck waiting for events.
* Add a linearizer on (appservice, stream) when handling ephemeral events. ↵Nick Barrett2021-11-033-18/+103
| | | | | (#11207) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Enable passing typing stream writers as a list. (#11237)Nick Barrett2021-11-038-16/+24
| | | | This makes the typing stream writer config match the other stream writers that only currently support a single worker.
* Remove a debug statement from tests. (#11239)Patrick Cloke2021-11-032-1/+1
|
* Add twine and towncrier as dev dependencies (#11233)Erik Johnston2021-11-032-0/+4
| | | | We don't pin them as we execute them as commands, rather than use them as libs.
* fix a small typo in the delete room api docsAndrew Morgan2021-11-031-1/+1
|
* Support sending no `state_events_at_start` in the MSC2716 `/batch_send` ↵Eric Eastwood2021-11-032-12/+18
| | | | | | | | endpoint (#11188) As brought up by @tulir, https://matrix.to/#/!SBYNQlpqkwJzFIdzxI:nevarro.space/$Gwnb2ZvXHc3poYXuBhho0cmoYq4KJ11Jh3m5s8kjNOM?via=nevarro.space&via=beeper.com&via=matrix.org This use case only works if the user is already joined in the current room state at the given `?prev_event_id`
* Add index to `local_group_updates.stream_id` (#11231)Erik Johnston2021-11-024-1/+37
| | | | This should speed up startup times and generally increase performance of groups.
* Merge remote-tracking branch 'origin/master' into developErik Johnston2021-11-026-6/+49
|\
| * Update changelog v1.46.0Erik Johnston2021-11-021-2/+2
| |
| * 1.46.0Erik Johnston2021-11-024-4/+16
| |
| * Force deb compression with `xz`. (#11197)Richard van der Hoff2021-10-272-0/+12
| | | | | | Fixes a problem where `impish` packages could not be processed by `reprepro`.
| * Fix URL preview errors when previewing XML documents. (#11196)Patrick Cloke2021-10-273-3/+22
| |
* | Add remaining type hints to `synapse.events`. (#11098)Patrick Cloke2021-11-0215-110/+185
| |
* | Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-024-0/+186
| |
* | Fix providing a `RoomStreamToken` instance to ↵Andrew Morgan2021-11-025-37/+30
| | | | | | | | | | `_notify_app_services_ephemeral` (#11137) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Add search by room ID and room alias to List Room admin API (#11099)Dirk Klimpel2021-11-024-53/+76
| | | | | | | | Fixes: #10874 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | ObservableDeferred: run observers in order (#11229)Richard van der Hoff2021-11-024-20/+88
| |
* | Update outdated links in `PULL_REQUEST_TEMPLATE.md` (#11225)Dirk Klimpel2021-11-012-4/+6
| |
* | Handle missing Content-Type header when accessing remote media (#11200)Shay2021-11-014-4/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * add code to handle missing content-type header and a test to verify that it works * add handling for missing content-type in the /upload endpoint as well * slightly refactor test code to put private method in approriate place * handle possible null value for content-type when pulling from the local db * add changelog * refactor test and add code to handle missing content-type in cached remote media * requested changes * Update changelog.d/11200.bugfix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* | Add `use_float=true` to ijson calls in Synapse (#11217)Shay2021-11-012-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | * add use_float=true to ijson calls * lints * add changelog * Update changelog.d/11217.bugfix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* | Fix a bug in unit test `test_block_room_and_not_purge` (#11226)Dirk Klimpel2021-11-012-2/+3
| |
* | Make `check_event_allowed` module API callback not fail open (accept events) ↵reivilibre2021-11-015-17/+24
| | | | | | | | when an exception is raised (#11033)
* | Remove deprecated delete room admin API (#11213)Dirk Klimpel2021-11-016-124/+79
| | | | | | | | | | Remove deprecated delete room admin API, `POST /_synapse/admin/v1/rooms/<room_id>/delete`
* | Support for serving server well-known files (#11211)Richard van der Hoff2021-11-018-47/+159
| | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/8308
* | Add domain specific matching for haproxy config (#11128)Brett Bethke2021-11-012-1/+2
| |
* | Docker: avoid changing userid unnecessarily (#11209)Richard van der Hoff2021-11-013-23/+28
| | | | | | | | | | | | | | | | | | | | | | | | * Docker image: avoid changing user during `generate` The intention was always that the config files get written as the initial user (normally root) - only the data directory needs to be writable by Synapse. This got changed in https://github.com/matrix-org/synapse/pull/5970, but that seems to have been a mistake. * Avoid changing user if no explicit UID is given * changelog
* | Support Client-Server API r0.6.1 (#11097)Aaron R2021-11-012-0/+2
| | | | | | | | | | Fixes #11064 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* | Improve code formatting and fix a few typos in docs (#11221)Sumner Evans2021-11-0120-168/+233
| | | | | | | | | | | | * Labeled a lot more code blocks with the appropriate type * Fixed a couple of minor typos (missing/extraneous commas) Signed-off-by: Sumner Evans <me@sumnerevans.com>
* | Add metrics to the threadpools (#11178)Erik Johnston2021-11-014-1/+49
| |
* | Test that `ClientIpStore` combines database and in-memory data correctly ↵Sean Quah2021-11-012-0/+207
| | | | | | | | (#11179)
* | Stop synapse from saving messages in device_inbox for hidden devices. (#10097)JohannesKleine2021-11-012-2/+7
| | | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Deprecate user_may_create_room_with_invites (#11206)Brendan Abolivier2021-11-013-36/+14
| |
* | Fix comments referencing v1.46.0 from PR #10969. (#11212)Dirk Klimpel2021-10-293-2/+3
| | | | | | | | #10969 was merged after 1.46.0rc1 was cut and will be included in v1.47.0rc1 instead.
* | Add a module API method to retrieve state from a room (#11204)Brendan Abolivier2021-10-293-1/+74
| |
* | Clarify lack of Windows support in documentation (#11198)Sean Quah2021-10-293-12/+14
| |
* | Additional type hints for relations database class. (#11205)Patrick Cloke2021-10-283-15/+25
| |
* | Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-285-2/+99
| | | | | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* | Fetch verify key locally rather than trying to do so over federation if ↵Shay2021-10-283-29/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | origin and host are the same. (#11129) * add tests for fetching key locally * add logic to check if origin server is same as host and fetch verify key locally rather than over federation * add changelog * slight refactor, add docstring, change changelog entry * Make changelog entry one line * remove verify_json_locally and push locality check to process_request, add function process_request_locally * remove leftover code reference * refactor to add common call to 'verify_json and associated handling code * add type hint to process_json * add some docstrings + very slight refactor
* | Add a ModuleApi method to update a user's membership in a room (#11147)Brendan Abolivier2021-10-283-2/+225
| | | | | | Co-authored-by: reivilibre <oliverw@matrix.org>
* | Type hints for the remaining two files in `synapse.http`. (#11164)David Robertson2021-10-286-37/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Teach MyPy that the sentinel context is False This means that if `ctx: LoggingContextOrSentinel` then `bool(ctx)` narrows us to `ctx:LoggingContext`, which is a really neat find! * Annotate RequestMetrics - Raise errors for sentry if we use the sentinel context - Ensure we don't raise an error and carry on, but not recording stats - Include stack trace in the error case to lower Sean's blood pressure * Make mypy pass for synapse.http.request_metrics * Make synapse.http.connectproxyclient pass mypy Co-authored-by: reivilibre <oliverw@matrix.org>
* | docs/openid: Add Authentik documentation. (#11151)Skyler Mäntysaari2021-10-282-0/+35
| |
* | Update scripts to pass Shellcheck lints (#11166)Dan Callahan2021-10-2720-104/+120
|\ \
| * \ Merge remote-tracking branch 'origin/develop' into shellcheckDan Callahan2021-10-2799-324/+1336
| |\ \ | |/ / |/| | | | | | | | | | | Fixes a merge conflict with debian/changelog Signed-off-by: Dan Callahan <danc@element.io>
* | | Annotate `log_function` decorator (#10943)reivilibre2021-10-2712-18/+58
| | | | | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | | Fixed config parse bug in review_recent_signups (#11191)Samuel Philipp2021-10-272-2/+8
| | |
* | | Refactor `Filter` to handle fields according to data being filtered. (#11194)Patrick Cloke2021-10-275-69/+87
| | | | | | | | | | | | This avoids filtering against fields which cannot exist on an event source. E.g. presence updates don't have a room.
* | | Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-276-15/+256
| | | | | | | | | Fixes: #9346
* | | Include the stable identifier for MSC3288. (#11187)Patrick Cloke2021-10-272-0/+3
| | | | | | | | | | | | | | | Includes both the stable and unstable identifier to store-invite calls to the identity server. In the future we should remove the unstable identifier.
* | | Merge remote-tracking branch 'origin/release-v1.46' into developErik Johnston2021-10-2763-60/+77
|\ \ \ | | |/ | |/|
| * | Update release dateErik Johnston2021-10-271-1/+1
| | |
| * | Update changelog with new changes v1.46.0rc1Erik Johnston2021-10-273-4/+2
| | |
| * | Shut down the DNS threadpool (#11190)Sean Quah2021-10-272-0/+2
| | | | | | | | | | | | | | | The DNS threadpool must be explicitly stopped, otherwise Synapse will hang indefinitely when asked to shut down.
| * | Fix thread BG update to not seq scan event_json (#11192)Erik Johnston2021-10-272-1/+2
| | | | | | | | | | | | For some reason the query optimiser decided to seq scan both tables, rather than index scanning `event_json`.
| * | Update CHANGES.mdSean Quah2021-10-261-16/+8
| | |
| * | Move #10975 to bugfix section in changelogSean Quah2021-10-261-1/+1
| | |
| * | Update CHANGES.mdSean Quah2021-10-261-3/+3
| | |
| * | Update CHANGES.mdSean Quah2021-10-261-2/+4
| | |
| * | 1.46.0rc1Sean Quah2021-10-2661-59/+81
| | |
* | | Implement an `on_new_event` callback (#11126)Brendan Abolivier2021-10-268-12/+165
|/ / | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Move DNS lookups into separate thread pool (#11177)Erik Johnston2021-10-263-1/+149
| | | | | | | | | | | | This is to stop large bursts of lookups starving out other users of the thread pools. Fixes #11049.
* | Add a background update for updating MSC3440 relation threads. (#11181)Patrick Cloke2021-10-263-2/+102
| |
* | Document the version each module API method was added to Synapse (#11183)Brendan Abolivier2021-10-262-11/+89
| |
* | Enable changing user type via users admin API (#11174)Jason Robinson2021-10-265-2/+80
| | | | | | | | | | | | | | | | Users admin API can now also modify user type in addition to allowing it to be set on user creation. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Fix cyclic import in the module API (#11180)Brendan Abolivier2021-10-252-2/+5
| | | | | | | | | | Introduced in #10548 See https://github.com/matrix-org/synapse-email-account-validity/runs/3979337154?check_suite_focus=true for an example of a module's CI choking over this issue.
* | Don't set new room alias before potential 403 (#10930)AndrewFerr2021-10-254-12/+113
| | | | | | | | | | | | Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* | Ensure that we correctly auth events returned by `send_join` (#11012)Richard van der Hoff2021-10-252-86/+61
| | | | | | | | | | | | | | This is the final piece of the jigsaw for #9595. As with other changes before this one (eg #10771), we need to make sure that we auth the auth events in the right order, and actually check that their predecessors haven't been rejected. To do this I've reused the existing code we use when persisting outliers elsewhere. I've removed the code for attempting to fetch missing auth_events - the events should have been present in the send_join response, so the likely reason they are missing is that we couldn't verify them, so requesting them again is unlikely to help. Instead, we simply drop any state which relies on those auth events, as we do at a backwards-extremity. See also matrix-org/complement#216 for a test for this.
* | Fix module API's `get_user_ip_and_agents` function when run on workers (#11112)Sean Quah2021-10-253-40/+91
| |
| * ChangelogDan Callahan2021-10-222-0/+7
| | | | | | | | Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2006: Use $(...) notationDan Callahan2021-10-2210-15/+15
| | | | | | | | | | | | | | | | Use $(...) notation instead of legacy backticked `...`. https://github.com/koalaman/shellcheck/wiki/SC2006 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2129: Consider using {..} >> fileDan Callahan2021-10-221-60/+63
| | | | | | | | | | | | | | | | Consider using { cmd1; cmd2; } >> file instead of individual redirects. https://github.com/koalaman/shellcheck/wiki/SC2129 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2086: Quote to prevent splittingDan Callahan2021-10-2213-42/+42
| | | | | | | | | | | | | | | | Double quote to prevent globbing and word splitting. https://github.com/koalaman/shellcheck/wiki/SC2086 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2012: Use find instead of lsDan Callahan2021-10-221-1/+1
| | | | | | | | | | | | | | | | Use find instead of ls to better handle non-alphanumeric filenames. https://github.com/koalaman/shellcheck/wiki/SC2012 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2016: Single quotes don't expandDan Callahan2021-10-221-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Expressions don't expand in single quotes, use double quotes for that. https://github.com/koalaman/shellcheck/wiki/SC2016 This specifically warned about the '$aregis...' part of the sed script. Which is a relatively obscure use of sed. Splitting this into two commands makes its intent more obvious and avoids contravening Shellcheck's lints. Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC1091: Can't follow fileDan Callahan2021-10-222-0/+2
| | | | | | | | | | | | | | | | Not following: (error message here) https://github.com/koalaman/shellcheck/wiki/SC1091 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC1001: Meaningless char escapesDan Callahan2021-10-221-1/+1
| | | | | | | | | | | | | | | | This \o will be a regular 'o' in this context. https://github.com/koalaman/shellcheck/wiki/SC1001 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2089 and SC2090: Quotes in varsDan Callahan2021-10-221-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SC2089: Quotes/backslashes will be treated literally. Use an array. https://github.com/koalaman/shellcheck/wiki/SC2089 SC2090: Quotes/backslashes in this variable will not be respected. https://github.com/koalaman/shellcheck/wiki/SC2090 Putting literal JSON in a variable mistakenly triggers these warnings. Instead of adding ignore directives, this can be avoided by inlining the JSON data into the curl invocation. Since the variable is only used in this one location, inlining is fine. Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2155: Declare + export separatelyDan Callahan2021-10-221-1/+2
| | | | | | | | | | | | | | | | Declare and assign separately to avoid masking return values. https://github.com/koalaman/shellcheck/wiki/SC2155 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2166: test -a is not well definedDan Callahan2021-10-221-1/+1
| | | | | | | | | | | | | | | | Prefer [ p ] && [ q ] as [ p -a q ] is not well defined. https://github.com/koalaman/shellcheck/wiki/SC2166 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2154: variable possibly undefinedDan Callahan2021-10-221-1/+1
| | | | | | | | | | | | | | | | var is referenced but not assigned. https://github.com/koalaman/shellcheck/wiki/SC2154 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2064: Use single quotes on trapsDan Callahan2021-10-221-1/+1
| | | | | | | | | | | | | | | | Use single quotes, otherwise this expands now rather than when signalled. https://github.com/koalaman/shellcheck/wiki/SC2064 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2115: Ensure never expands to /*Dan Callahan2021-10-221-2/+2
| | | | | | | | | | | | | | | | Use "${var:?}" to ensure this never expands to /* . https://github.com/koalaman/shellcheck/wiki/SC2115 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2046: Quote to prevent word splitDan Callahan2021-10-228-8/+8
| | | | | | | | | | | | | | | | Quote this to prevent word splitting https://www.shellcheck.net/wiki/SC2046 Signed-off-by: Dan Callahan <danc@element.io>
| * Fix Shellcheck SC2164: exit in case cd fails.Dan Callahan2021-10-223-6/+6
|/ | | | | | | | Use `cd ... || exit` in case cd fails. https://github.com/koalaman/shellcheck/wiki/SC2164 Signed-off-by: Dan Callahan <danc@element.io>
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-2258-143/+342
|
* Fix synapse.config module "read" command (#11145)Jason Robinson2021-10-225-68/+138
| | | | | | | | | | `synapse.config.__main__` has the possibility to read a config item. This can be used to conveniently also validate the config is valid before trying to start Synapse. The "read" command broke in https://github.com/matrix-org/synapse/pull/10916 as it now requires passing in "server.server_name" for example. Also made the read command optional so one can just call this with just the confirm file reference and get a "Config parses OK" if things are ok. Signed-off-by: Jason Robinson <jasonr@matrix.org> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Add more information what happens when a user is deactivated (#11083)Dirk Klimpel2021-10-222-0/+11
|
* Add a thread relation type per MSC3440. (#11088)Patrick Cloke2021-10-218-8/+119
| | | | Adds experimental support for MSC3440's `io.element.thread` relation type (and the aggregation for it).
* Fix adding excluded users to the private room sharing tables when joining a ↵David Robertson2021-10-213-29/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | room (#11143) * We only need to fetch users in private rooms * Filter out `user_id` at the top * Discard excluded users in the top loop We weren't doing this in the "First, if they're our user" branch so this is a bugfix. * The caller must check that `user_id` is included This is in the docstring. There are two call sites: - one in `_handle_room_publicity_change`, which explicitly checks before calling; - and another in `_handle_room_membership_event`, which returns early if the user is excluded. So this change is safe. * Test joining a private room with an excluded user * Tweak an existing test * Changelog * test docstring * lint
* Improve docstrings for methods related to sending EDUs to application ↵Andrew Morgan2021-10-217-23/+148
| | | | services (#11138)
* Add missing type hints to synapse.crypto. (#11146)Patrick Cloke2021-10-215-18/+36
| | | And require type hints for this module.
* fix relative link in docker readme (#11144)Richard van der Hoff2021-10-212-1/+3
| | | relative links don't work when it's on dockerhub.
* Fix setting a user's external_id via the admin API returns 500 and deletes ↵Dirk Klimpel2021-10-214-37/+321
| | | | | users existing external mappings if that external ID is already mapped (#11051) Fixes #10846
* Update `sign_json` to support inline key config (#11139)Richard van der Hoff2021-10-202-7/+26
| | | | It's been possible to configure a key inline in the homeserver.yaml since 13bc1e0746aa0442aa5d43555cbbc2dc75e8ef43. Update `sign_json` to work with this.
* Consider IP whitelist for identity server resolution (#11120)Robert Edström2021-10-202-1/+4
| | | Signed-off-by: Robert Edström <github@legogris.se>
* Clean up `_update_auth_events_and_context_for_auth` (#11122)Richard van der Hoff2021-10-202-114/+38
| | | Remove some redundant code, and generally simplify.
* Show error when timestamp in seconds is provided to the /purge_media_cache ↵Aaron R2021-10-204-13/+133
| | | | API (#11101)
* Merge branch 'master' into developSean Quah2021-10-206-63/+19
|\
| * Remove link to #10947 from changelog v1.45.1Sean Quah2021-10-201-1/+1
| |
| * 1.45.1Sean Quah2021-10-204-2/+16
| |
| * Revert change to counting of deactivated users towards the monthly active ↵Sean Quah2021-10-204-62/+4
| | | | | | | | | | | | | | | | users limit (#11127) Temporarily revert "Add functionality to remove deactivated users from the monthly_active_users table (#10947)". This reverts commit eda8c88b84ee7506379a71ac2a7a88c08b759d43.
* | Remove false warning about copying the log config to a homeserver.yaml (#11092)Travis Ralston2021-10-202-6/+7
| | | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Document the version of Synapse each module callback was introduced in (#11132)Brendan Abolivier2021-10-206-0/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Mention callbacks introduced in v1.37.0 According to the documentation introduced in https://github.com/matrix-org/synapse/pull/10062 * Mention callbacks introduced in v1.39.0 According to https://github.com/matrix-org/synapse/pull/10386 and https://github.com/matrix-org/synapse/pull/9884 * Mention callbacks introduced in v1.42.0 According to https://github.com/matrix-org/synapse/pull/10524 * Mention callbacks introduced in v1.44.0 and v1.45.0 As per https://github.com/matrix-org/synapse/pull/10898, https://github.com/matrix-org/synapse/pull/10910 and https://github.com/matrix-org/synapse/pull/10894 * Mention callbacks introduced in v1.46.0 According to https://github.com/matrix-org/synapse/pull/10548
* | Add missing type hints to event fetching. (#11121)Patrick Cloke2021-10-192-61/+82
| | | | | | | | Updates the event rows returned from the database to be attrs classes instead of dictionaries.
* | Fix instances of [example]{.title-ref} in the upgrade notes (#11118)Andrew Morgan2021-10-192-27/+28
| |
* | Merge branch 'master' into developDavid Robertson2021-10-193-5/+40
|\|
| * Be less inconsistent about v1.2.3 versus 1.2.3 v1.45.0David Robertson2021-10-191-5/+5
| |
| * Duplicate known issues under 1.45 releaseDavid Robertson2021-10-191-0/+7
| |
| * 1.45.0David Robertson2021-10-194-2/+18
| |
| * Reword changelog regarding a suspected regression (#11117)Dan Callahan2021-10-192-3/+15
| | | | | | Signed-off-by: Dan Callahan <danc@element.io>
* | Move _persist_auth_tree into FederationEventHandler (#11115)Richard van der Hoff2021-10-193-125/+120
| | | | | | | | | | This is just a lift-and-shift, because it fits more naturally here. We do rename it to `process_remote_join` at the same time though.
* | Rename `_auth_and_persist_fetched_events` (#11116)Richard van der Hoff2021-10-192-14/+10
| | | | | | | | ... to `_auth_and_persist_outliers`, since that reflects its purpose better.
* | Include rejected status when we log events. (#11008)Richard van der Hoff2021-10-192-6/+11
| | | | | | | | | | | | If we find ourselves dealing with rejected events, we proably want to know about it. Let's include it in the stringification of the event so that it gets logged.
* | Add missing type hints to synapse.api. (#11109)Patrick Cloke2021-10-1810-99/+84
| | | | | | | | * Convert UserPresenceState to attrs. * Remove args/kwargs from error classes and explicitly pass msg/errorcode.
* | Check auth on received events' auth_events (#11001)Richard van der Hoff2021-10-182-2/+98
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, when we receive an event whose auth_events differ from those we expect, we state-resolve between the two state sets, and check that the event passes auth based on the resolved state. This means that it's possible for us to accept events which don't pass auth at their declared auth_events (or where the auth events themselves were rejected), leading to problems down the line like #10083. This change means we will: * ignore any events where we cannot find the auth events * reject any events whose auth events were rejected * reject any events which do not pass auth at their declared auth_events. Together with a whole raft of previous work, this is a partial fix to #9595. Fixes #6643. Based on #11009.
* | Check *all* auth events for room id and rejection (#11009)Richard van der Hoff2021-10-188-85/+122
| | | | | | | | | | | | | | | | | | | | | | This fixes a bug where we would accept an event whose `auth_events` include rejected events, if the rejected event was shadowed by another `auth_event` with same `(type, state_key)`. The approach is to pass a list of auth events into `check_auth_rules_for_event` instead of a dict, which of course means updating the call sites. This is an extension of #10956.
* | Document Synapse's behaviour when dealing with multiple modules (#11096)Brendan Abolivier2021-10-188-8/+154
| | | | | | | | | | | | Document Synapse's behaviour when multiple modules register the same callback/web resource/etc. Co-authored-by: reivilibre <oliverw@matrix.org>
* | `_run_push_actions_and_persist_event`: handle no min_depth (#11014)Richard van der Hoff2021-10-183-11/+20
| | | | | | | | | | Make sure that we correctly handle rooms where we do not yet have a `min_depth`, and also add some comments and logging.
* | Fix broken export-data admin command and add a test for it to CI (#11078)Hillery Shay2021-10-184-8/+93
| | | | | | Fix broken export-data admin command and add a test for it to CI
* | Don't remove local users from dir when the leave their last room (#11103)David Robertson2021-10-183-5/+59
| |
* | Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-154-83/+148
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Move experimental & retention config out of the server module. (#11070)Patrick Cloke2021-10-1510-255/+290
| |
* | Fix logging context warnings when losing replication connection (#10984)Sean Quah2021-10-153-10/+27
| | | | | | | | | | | | Instead of triggering `__exit__` manually on the replication handler's logging context, use it as a context manager so that there is an `__enter__` call to balance the `__exit__`.
* | Update doc of the allowed characters for registration tokens (#11093)Dirk Klimpel2021-10-152-1/+2
| | | | | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Fix 500 error on `/messages` when we accumulate more than 5 backward ↵Eric Eastwood2021-10-144-12/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | extremities (#11027) Found while working on the Gitter backfill script and noticed it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390 When there are more than 5 backward extremities for a given depth, backfill will throw an error because we sliced the extremity list to 5 but then try to iterate over the full list. This causes us to look for state that we never fetched and we get a `KeyError`. Before when calling `/messages` when there are more than 5 backward extremities: ``` Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper callback_return = await self._async_render(request) File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render callback_return = await raw_callback_return File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET msgs = await self.pagination_handler.get_messages( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages await self.hs.get_federation_handler().maybe_backfill( File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill return await self._maybe_backfill_inner(room_id, current_depth, limit) File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner likely_extremeties_domains = get_domains_from_state(states[e_id]) KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl' ```
* | Ensure each charset is attempted only once during media preview. (#11089)Patrick Cloke2021-10-143-14/+64
| | | | | | | | There's no point in trying more than once since it is guaranteed to continually fail.
* | Attempt different character encodings when previewing a URL. (#11077)Patrick Cloke2021-10-143-67/+80
| | | | | | | | This follows similar logic to BeautifulSoup where we attempt different character encodings until we find one which works.
* | Fix-up some type hints in the relations tests. (#11076)Patrick Cloke2021-10-145-51/+64
| |
* | Add a test for a workaround concerning the behaviour of third-party rule ↵reivilibre2021-10-142-6/+51
| | | | | | | | modules and `SynapseError`s. (#11071)
* | Merge tag 'v1.45.0rc2' into developDavid Robertson2021-10-1410-9/+30
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.45.0rc2 (2021-10-14) ============================== **Note:** This release candidate [fixes](https://github.com/matrix-org/synapse/issues/11053) the user directory [bug](https://github.com/matrix-org/synapse/issues/11025) present in 1.45.0rc1. However, the [performance issue](https://github.com/matrix-org/synapse/issues/11049) which appeared in v1.44.0 is yet to be resolved. Bugfixes -------- - Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](https://github.com/matrix-org/synapse/issues/11045)) - Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a user not in the `users` table. ([\#11053](https://github.com/matrix-org/synapse/issues/11053)) - Fix a bug introduced in Synapse v1.44.0 when logging errors during oEmbed processing. ([\#11061](https://github.com/matrix-org/synapse/issues/11061)) Internal Changes ---------------- - Add an 'approximate difference' method to `StateFilter`. ([\#10825](https://github.com/matrix-org/synapse/issues/10825)) - Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](https://github.com/matrix-org/synapse/issues/10970)) - Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly. ([\#10996](https://github.com/matrix-org/synapse/issues/10996)) - Ensure that cache config tests do not share state. ([\#11036](https://github.com/matrix-org/synapse/issues/11036))
| * it appeared in 1.44, not 45rc1 v1.45.0rc2David Robertson2021-10-141-1/+1
| |
| * mentioned -> which appearedDavid Robertson2021-10-141-1/+1
| |
| * Refer to the bugs mentioned in 1.45.0rc1 noteDavid Robertson2021-10-141-0/+2
| |
| * 1.45.0rc2David Robertson2021-10-1410-9/+28
| |
* | Resolve and share `state_groups` for all historical events in batch ↵Eric Eastwood2021-10-138-47/+114
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#10975) Resolve and share `state_groups` for all historical events in batch. This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch. This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room. For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch. --- How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example: **Before** (new `state_group` for every event 😬, very inefficient): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10 create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11 create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12 create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13 create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14 ``` **After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to): ``` # Tests from https://github.com/matrix-org/complement/pull/206 $ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9 create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10 create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10 create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10 create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10 create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10 ```
* | Fix upgrade dead links (#11069)David Robertson2021-10-132-4/+5
| |
* | Rearrange the user_directory's `_handle_deltas` function (#11035)David Robertson2021-10-132-57/+79
| | | | | | | | | | * Pull out `_handle_room_membership_event` * Discard excluded users early * Rearrange logic so the change is membership is effectively switched over. See PR for rationale.
* | Remove dead code from `MediaFilePaths` (#11056)Sean Quah2021-10-132-17/+1
| |
* | Merge remote-tracking branch 'origin/release-v1.45' into developDavid Robertson2021-10-134-10/+98
|\|
| * Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-1313-93/+921
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Add type hints to synapse.events.*. (#11066)Patrick Cloke2021-10-1311-145/+208
| | | | | | Except `synapse/events/__init__.py`, which will be done in a follow-up.
* | Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-1313-225/+790
| | | | | | | | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Be more lenient when parsing the version for oEmbed responses. (#11065)Patrick Cloke2021-10-135-8/+60
| |
* | Mark Module API error imports as re-exported and mark Synapse as containing ↵reivilibre2021-10-134-2/+11
| | | | | | | | type annotations (#11054)
* | Always dump logs from trial during CI. (#11068)Patrick Cloke2021-10-123-0/+9
| | | | | | Instead of only dumping them if trial passes.
* | Simplify the user admin API tests (#11048)Dirk Klimpel2021-10-122-255/+147
| |
* | Add support for ubuntu 21.10 "Impish Indri" (#11024)Hillery Shay2021-10-122-0/+2
| | | | | | | | | | | | | | | | | | | | | | * support ubuntu 21.10 indri * add changelog * update to correct codename Co-authored-by: Brendan Abolivier <github@brendanabolivier.com> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* | Merge remote-tracking branch 'origin/release-v1.45' into developPatrick Cloke2021-10-122-1/+2
|\|
| * Fix formatting string when oEmbed errors occur. (#11061)Patrick Cloke2021-10-122-1/+2
| |
* | Add tests for `MediaFilePaths` (#11057)Sean Quah2021-10-123-0/+240
| |
* | Update `_wrap_in_base_path` type hints to preserve function arguments (#11055)Sean Quah2021-10-122-3/+7
| |
* | Merge remote-tracking branch 'origin/release-v1.45' into developErik Johnston2021-10-124-28/+80
|\|
| * Fix race in `MultiWriterIdGenerator` (#11045)Erik Johnston2021-10-122-15/+68
| | | | | | | | | | | | | | | | | | | | The race allowed the current position to advance too far when stream IDs are still being persisted. This happened when it received a new stream ID from a remote write between a new stream ID being allocated and it being added to the set of unpersisted stream IDs. Fixes #9424.
| * Reset global cache state before cache tests. (#11036)Patrick Cloke2021-10-122-13/+12
| | | | | | | | | | | | This reverts #11019 and structures the code a bit more like it was before #10985. The global cache state must be reset before running the tests since other test cases might have configured caching (and thus touched the global state).
* | Add type hints to `synapse.storage.databases.main.client_ips` (#10972)Sean Quah2021-10-125-45/+121
| |
* | Merge tag 'v1.45.0rc1' into developBrendan Abolivier2021-10-1267-66/+97
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.45.0rc1 (2021-10-12) ============================== **Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them. Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information. Features -------- - Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655)) - Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822)) - Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894)) - Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910)) - Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954)) - Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028)) Bugfixes -------- - Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922)) - Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924)) - Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927)) - Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947)) - Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956)) - Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960)) - Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962)) - Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981)) - Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982)) - Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995)) - Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002)) - Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003)) - Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042)) - Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877)) Improved Documentation ---------------------- - Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971)) - Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973)) - Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990)) - Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991)) Internal Changes ---------------- - Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888)) - Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892)) - Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895)) - Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902)) - Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903)) - Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915)) - Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916), [\#10959](https://github.com/matrix-org/synapse/issues/10959), [\#10985](https://github.com/matrix-org/synapse/issues/10985)) - Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926), [\#10940](https://github.com/matrix-org/synapse/issues/10940), [\#10986](https://github.com/matrix-org/synapse/issues/10986), [\#10987](https://github.com/matrix-org/synapse/issues/10987), [\#10988](https://github.com/matrix-org/synapse/issues/10988), [\#11010](https://github.com/matrix-org/synapse/issues/11010), [\#11011](https://github.com/matrix-org/synapse/issues/11011)) - Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934)) - Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935)) - Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936)) - Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939)) - Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945)) - Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958)) - Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961)) - Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963)) - Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966)) - Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974)) - Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983)) - Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992)) - Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993)) - Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994)) - Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004)) - Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005)) - Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006)) - Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017)) - Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019)) - Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021)) - Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023)) - When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034)) - Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043))
| * Fixup changelog v1.45.0rc1Brendan Abolivier2021-10-121-2/+2
| |
| * TypoBrendan Abolivier2021-10-121-1/+1
| |
| * Add a link to the upgrade notesBrendan Abolivier2021-10-121-0/+2
| |
| * Update upgrade notesBrendan Abolivier2021-10-121-0/+9
| |
| * Add warning about known issuesBrendan Abolivier2021-10-121-0/+2
| |
| * Fixup changelogBrendan Abolivier2021-10-121-7/+7
| |
| * 1.45.0rc1Brendan Abolivier2021-10-1266-66/+84
| |
* | Fix inconsistent behavior of `get_last_client_by_ip` (#10970)Sean Quah2021-10-123-4/+53
| | | | | | | | | | | | | | Make `get_last_client_by_ip` return the same dictionary structure regardless of whether the data has been persisted to the database. This change will allow slightly cleaner type hints to be applied later on.
* | Fix opentracing and Prometheus metrics for replication requests (#10996)Sean Quah2021-10-123-76/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit fixes two bugs to do with decorators not instrumenting `ReplicationEndpoint`'s `send_request` correctly. There are two decorators on `send_request`: Prometheus' `Gauge.track_inprogress()` and Synapse's `opentracing.trace`. `Gauge.track_inprogress()` does not have any support for async functions when used as a decorator. Since async functions behave like regular functions that return coroutines, only the creation of the coroutine was covered by the metric and none of the actual body of `send_request`. `Gauge.track_inprogress()` returns a regular, non-async function wrapping `send_request`, which is the source of the next bug. The `opentracing.trace` decorator would normally handle async functions correctly, but since the wrapped `send_request` is a non-async function, the decorator ends up suffering from the same issue as `Gauge.track_inprogress()`: the opentracing span only measures the creation of the coroutine and none of the actual function body. Using `Gauge.track_inprogress()` as a context manager instead of a decorator resolves both bugs.
* | Add an approximate difference method to StateFilters (#10825)reivilibre2021-10-123-3/+683
|/
* disallow-untyped-defs for synapse.push (#11023)David Robertson2021-10-117-10/+28
|
* Include the requirements for [mypy,lint] in [dev] (#11034)reivilibre2021-10-114-11/+17
|
* Pass through `SynapseError`s that are raised from experimental ↵reivilibre2021-10-112-0/+10
| | | | | `check_event_allowed` callback of the module API (#11042) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Remove unnecessary list comprehension in `synapse_port_db` to fix linting in ↵reivilibre2021-10-112-1/+2
| | | | CI (#11043)
* Release script improvements (#10966)reivilibre2021-10-112-4/+33
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Refactor MSC2716 `/batch_send` endpoint into separate handler functions (#10974)Eric Eastwood2021-10-084-283/+485
|
* Autodiscover oEmbed endpoint from returned HTML (#10822)Patrick Cloke2021-10-085-55/+224
| | | | | Searches the returned HTML for an oEmbed endpoint using the autodiscovery mechanism (`<link rel=...>`), and will request it to generate the preview.
* Revert accidental push to develop.Olivier Wilkinson (reivilibre)2021-10-083-16/+10
|
* NewsfileOlivier Wilkinson (reivilibre)2021-10-081-0/+1
| | | | Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Update contributing guide to use [all,dev]Olivier Wilkinson (reivilibre)2021-10-081-1/+1
|
* Include the requirements for [mypy,lint] in [dev]Olivier Wilkinson (reivilibre)2021-10-081-9/+14
|
* Annotate synapse.storage.util (#10892)David Robertson2021-10-088-65/+124
| | | | | Also mark `synapse.streams` as having has no untyped defs Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-0812-80/+100
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix overwriting profile when making room public (#11003)David Robertson2021-10-083-31/+104
| | | This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
* Remove the deprecated BaseHandler. (#11005)Patrick Cloke2021-10-0824-215/+166
| | | | | | | | The shared ratelimit function was replaced with a dedicated RequestRatelimiter class (accessible from the HomeServer object). Other properties were copied to each sub-class that inherited from BaseHandler.
* Fix long-standing bug where `ReadWriteLock` could drop logging contexts (#10993)Sean Quah2021-10-082-2/+5
| | | | | | | | | | | Use `PreserveLoggingContext()` to ensure that logging contexts are not lost when exiting a read/write lock. When exiting a read/write lock, callbacks on a `Deferred` are triggered as a signal to any waiting coroutines. Any waiting coroutine that becomes runnable is likely to follow the Synapse logging context rules and will restore its own logging context, then either run to completion or await another `Deferred`, resetting the logging context in the process.
* Include exception in json logging (#11028)Nick Barrett2021-10-083-0/+35
|
* Fix CI to run the unit tests without optional deps (#11017)Erik Johnston2021-10-082-3/+7
| | | | | This also turns off calculating code coverage, as we didn't use it and it was a lot of noise
* disallow-untyped-defs for synapse.server_notices (#11021)David Robertson2021-10-073-6/+6
|
* Ensure each cache config test uses separate state. (#11019)Patrick Cloke2021-10-072-8/+13
| | | Hopefully this fixes these tests sometimes failing in CI.
* Don't alter directory entries for local users when setting a per-room ↵David Robertson2021-10-073-7/+48
| | | | | nickname (#11002) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Improve the logging in _auth_and_persist_outliers (#11010)Richard van der Hoff2021-10-072-1/+5
| | | Include the event ids being peristed
* Add a comment in _process_received_pdu (#11011)Richard van der Hoff2021-10-072-0/+4
|
* Catch AttributeErrors when calling registerProducer (#10995)Brendan Abolivier2021-10-072-1/+5
| | | Looks like the wrong exception type was caught in #10932.
* `disallow-untyped-defs` for `synapse.state` (#11004)David Robertson2021-10-065-4/+8
| | | | | * `disallow-untyped-defs` for `synapse.state` Much smaller than I was expecting!
* Add content to the Synapse documentation intro page (#10990)Andrew Morgan2021-10-063-1/+76
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add a spamchecker method to allow or deny 3pid invites (#10894)Brendan Abolivier2021-10-065-0/+153
| | | | | This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline. Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-0631-160/+124
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Add a spamchecker callback to allow or deny room joins (#10910)Brendan Abolivier2021-10-066-0/+174
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-14 21:15:20 +0000