summary refs log tree commit diff
path: root/changelog.d (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Revert "Pin to frozendict<2.1.2 (#11625)" and allow frozendict>2.1.2 (#11969)Pascal Bach2022-02-141-0/+1
| | | | | This reverts commit 2bf31f7807c7a0c229170803c97090d612dc16f9. Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
* Fix M_WRONG_ROOM_KEYS_VERSION error not including `current_version` field ↵Michael Telatynski2022-02-141-0/+1
| | | | | (#11988) Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
* Note when unstable MSC3283 prefixes will be removed (#11989)Andrew Morgan2022-02-141-0/+1
|
* remove empty file, reword/rename newsfragment, and add line to upgrade docH. Shay2022-02-142-1/+1
|
* Require that modules register their callbacks using keyword arguments. (#11975)Patrick Cloke2022-02-141-0/+1
| | | | To allow for easier backwards/forwards compatibility by making it easier to add/remove callbacks.
* Use version string helper from matrix-common (#11979)David Robertson2022-02-141-0/+1
| | | | * Require latest matrix-common * Use the common function
* Notify users, rather than rooms, of device list updates (#11905)Andrew Morgan2022-02-121-0/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Implement MSC3706: partial state in `/send_join` response (#11967)Richard van der Hoff2022-02-121-0/+1
| | | | | | | | | | | | * Make `get_auth_chain_ids` return a Set It has a set internally, and a set is often useful where it gets used, so let's avoid converting to an intermediate list. * Minor refactors in `on_send_join_request` A little bit of non-functional groundwork * Implement MSC3706: partial state in /send_join response
* Enable cache time-based expiry by default (#11849)Shay2022-02-111-0/+1
|
* Support the MSC3715 for `/relations`. (#11941)Patrick Cloke2022-02-111-0/+1
| | | | This adds an unstable org.matrix.msc3715.dir parameter which acts like dir on /mesages.
* Fetch thread summaries for multiple events in a single query (#11752)Patrick Cloke2022-02-111-0/+1
| | | | | This should reduce database usage when fetching bundled aggregations as the number of individual queries (and round trips to the database) are reduced.
* Prepare for rename of default complement branch (#11971)Richard van der Hoff2022-02-111-0/+1
| | | use `HEAD` rather than hardcoding `master`
* Remove deprecated user_may_create_room_with_invites callback (#11950)Brendan Abolivier2022-02-111-0/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix NewsfileErik Johnston2022-02-111-0/+1
| | | | | \#11966 fixes a bug introduced by #11215, which isn't in a released version.
* Fix NewsfileErik Johnston2022-02-111-1/+0
| | | | | \#11966 fixes a bug introduced by #11215, which isn't in a released version.
* Fix to-device being dropped in limited sync in SQLite. (#11966)Erik Johnston2022-02-111-0/+1
| | | | | | | | | If ther are more than 100 to-device messages pending for a device `/sync` will only return the first 100, however the next batch token was incorrectly calculated and so all other pending messages would be dropped. This is due to `txn.rowcount` only returning the number of rows that *changed*, rather than the number *selected* in SQLite.
* fix import cycle (#11965)Richard van der Hoff2022-02-111-0/+1
|
* Adds misc missing type hints (#11953)Patrick Cloke2022-02-111-0/+1
|
* Tests: replace mocked Authenticator with the real thing (#11913)Richard van der Hoff2022-02-111-0/+1
| | | | | | | | | | | | If we prepopulate the test homeserver with a key for a remote homeserver, we can make federation requests to it without having to stub out the authenticator. This has two advantages: * means that what we are testing is closer to reality (ie, we now have complete tests for the incoming-request-authorisation flow) * some tests require that other objects be signed by the remote server (eg, the event in `/send_join`), and doing that would require a whole separate set of mocking out. It's much simpler just to use real keys.
* Support the stable API endpoint for MSC3283: new settings in `/capabilities` ↵Dirk Klimpel2022-02-111-0/+1
| | | | endpoint (#11933)
* Drop support for EOL Ubuntu 21.04 (#11961)Shay2022-02-101-0/+1
|
* Support pagination tokens from /sync and /messages in the relations API. ↵Patrick Cloke2022-02-101-0/+1
| | | | (#11952)
* Implement a content type allow list for URL previews (#11936)Denis Kasak2022-02-101-0/+1
| | | | | | | This implements an allow list for content types for which Synapse will attempt URL preview. If a URL resolves to a resource with a content type which isn't in the list, the download will terminate immediately. This makes sense given that Synapse would never successfully generate a URL preview for such files in the first place, and helps prevent issues with streaming media servers, such as #8302. Signed-off-by: Denis Kasak dkasak@termina.org.uk
* Fix broken link in README to admin API. (#11955)Alexander Mnich2022-02-101-0/+1
| | | From when the documentation was converted from rst -> md.
* Basic documentation for the release cycle. (#11954)Patrick Cloke2022-02-101-0/+1
|
* Fix example for structured logging. (#11946)Brad Jones2022-02-091-0/+1
| | | The StreamHandler takes a stream parameter, not location.
* Update changelog from #11867 to be a single line.Patrick Cloke2022-02-091-5/+1
|
* Add missing type hints to synapse.replication. (#11938)Patrick Cloke2022-02-081-0/+1
|
* Experimental support to include bundled aggregations in search results ↵Patrick Cloke2022-02-081-0/+1
| | | | (MSC3666) (#11837)
* Remove unnecessary ignores due to Twisted upgrade. (#11939)Patrick Cloke2022-02-081-0/+1
| | | | Twisted 22.1.0 fixed some internal type hints, allowing Synapse to remove ignore calls for parameters to connectTCP.
* Add missing type hints to synapse.replication.http. (#11856)Patrick Cloke2022-02-081-0/+1
|
* Fetch edits for multiple events in a single query. (#11660)Patrick Cloke2022-02-081-0/+1
| | | | | This should reduce database usage when fetching bundled aggregations as the number of individual queries (and round trips to the database) are reduced.
* Return JSON errors for unknown resources under /matrix/client. (#11930)Patrick Cloke2022-02-081-0/+1
| | | | | Re-applies the changes from 3e0cfd447e17658a937fe62555db9e968f00b15b (#11602), reverting d93ec0a0ba5f6d2fbf2bc321086d4ad4c03136e0 (#11764) now that the conflict with the webclient listener was fixed in 119edf51eb3e4f5ed5139dc370f5d7aed46edc1c (#11895).
* Add a docstring to `add_device_change_to_streams` and fix some nearby types ↵Andrew Morgan2022-02-081-0/+1
| | | | (#11912)
* Add a callback to allow modules to deny 3PID (#11854)Brendan Abolivier2022-02-081-0/+1
| | | | | Part of the Tchap Synapse mainlining. This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
* Fix historical messages backfilling in random order on remote homeservers ↵Eric Eastwood2022-02-071-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (MSC2716) (#11114) Fix https://github.com/matrix-org/synapse/issues/11091 Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`) 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`) - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)). - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort. - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date. - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order. 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls. - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793 Before | After --- | --- ![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png) #### Why aren't we sorting topologically when receiving backfill events? > The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway. > > As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering. > > -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138 See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
* Remove optional state of `ApplicationService.is_interested`'s `store` ↵Andrew Morgan2022-02-071-0/+1
| | | | parameter (#11911)
* Pass the proper type when uploading files. (#11927)Patrick Cloke2022-02-071-0/+1
| | | | The Content-Length header should be treated as an int, not a string. This shouldn't have any user-facing change.
* Clean up an indirect reference to the homeserver datastore (#11914)Andrew Morgan2022-02-071-0/+1
|
* Add a unit test for users receiving their own device list updates (#11909)Andrew Morgan2022-02-071-0/+1
|
* Clarify that users' media are also preview images (#11862)Dirk Klimpel2022-02-041-0/+1
|
* Run Complement tests sequentially (#11910)kegsay2022-02-041-0/+1
| | | | | | | | | | | Since #11811 there has been general Complement flakiness around networking. It seems like tests are hitting the wrong containers. In an effort to diagnose the cause of this, as well as reduce its impact on this project, set the parallelsim to 1 (no parallelism) when running tests. If this fixes the flakiness then this indicates the cause and I can diagnose this further. If this doesn't fix the flakiness then that implies some kind of test pollution which also helps to diagnose this further.
* Stabilise MSC3231 (Token Based Registration) (#11867)Jonathan de Jong2022-02-041-0/+5
|
* Delete MSC1711_certificates_FAQ.md (#11907)Richard van der Hoff2022-02-041-0/+1
| | | This document isn't really relevant any more, and its existence is more confusing than helpful.
* Remove support for the webclient listener. (#11895)Patrick Cloke2022-02-031-0/+1
| | | Also remove support for non-HTTP(S) web_client_location.
* Fix typo: unpind -> unbind (#11859)Christian Paul2022-02-031-0/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Add a ratelimiter for 3pid invite (#11892)Brendan Abolivier2022-02-031-0/+1
|
* Allow specifying the application service-specific `user_id` parameter in the ↵reivilibre2022-02-032-1/+2
| | | | `join` test helper. (#11616)
* Improve opentracing for federation requests (#11870)Richard van der Hoff2022-02-031-0/+1
| | | | | | | | | The idea here is to set the parent span for incoming federation requests to the *outgoing* span on the other end. That means that you can see (most of) the full end-to-end flow when you have a process that includes federation requests. However, in order not to lose information, we still want a link to the `incoming-federation-request` span from the servlet, so we have to create another span to do exactly that.
* Fixes for opentracing scopes (#11869)Richard van der Hoff2022-02-021-0/+1
| | | | | | | | | | | | | | | | | | | | `start_active_span` was inconsistent as to whether it would activate the span immediately, or wait for `scope.__enter__` to happen (it depended on whether the current logcontext already had an associated scope). The inconsistency was rather confusing if you were hoping to set up a couple of separate spans before activating either. Looking at the other implementations of opentracing `ScopeManager`s, the intention is that it *should* be activated immediately, as the name implies. Indeed, the idea is that you don't have to use the scope as a contextmanager at all - you can just call `.close` on the result. Hence, our cleanup has to happen in `.close` rather than `.__exit__`. So, the main change here is to ensure that `start_active_span` does activate the span, and that `scope.close()` does close the scope. We also add some tests, which requires a `tracer` param so that we don't have to rely on the global variable in unit tests.
* Invalidate the get_users_in_room{_with_profile} caches only when necessary. ↵Patrick Cloke2022-02-021-0/+1
| | | | | | | (#11878) The get_users_in_room and get_users_in_room_with_profiles are now only invalidated when the membership of a room changes, instead of during any state change in the room.
* Fix type errors introduced by new annotations in the Prometheus Client ↵reivilibre2022-02-021-0/+1
| | | | | library. (#11832) Co-authored-by: David Robertson <davidr@element.io>
* Fix losing incoming EDUs if debug logging enabled (#11890)David Robertson2022-02-021-0/+1
| | | | | | | | | * Fix losing incoming EDUs if debug logging enabled Fixes #11889. Homeservers should only be affected if the `synapse.8631_debug` logger was enabled for DEBUG mode. I am not sure if this merits a bugfix release: I think the logging can be disabled in config if anyone is affected? But it is still pretty bad.
* Disable coverage calculation for olddeps build. (#11888)Richard van der Hoff2022-02-021-0/+1
| | | | We disabled coverage calculation for most of CI in #11017, but the olddeps build uses a separate script and got forgotten.
* Describe `prune_unread_entries` in docstrings (#11876)David Robertson2022-02-021-0/+1
| | | Should have been caught in #10826.
* Revert experimental push rules from #7997. (#11884)Patrick Cloke2022-02-021-0/+1
| | | Manually reverts the merge from cdbb8e6d6e36e0b6bc36e676d8fe66c96986b399.
* Add a background database update to purge account data for deactivated ↵reivilibre2022-02-021-0/+1
| | | | users. (#11655)
* Expose the registered device ID from the `register_appservice_user` test ↵reivilibre2022-02-021-0/+1
| | | | helper. (#11615)
* Convert `ApplicationServiceTestCase` to use `simple_async_mock` (#11880)Andrew Morgan2022-02-011-0/+1
|
* Allow modules to retrieve server and worker names (#11868)Brendan Abolivier2022-02-011-0/+1
| | | Fixes #10701
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-0/+1
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* 1.52.0rc1David Robertson2022-02-0137-37/+0
|
* Add type hints to `tests/rest/admin` (#11851)Dirk Klimpel2022-01-311-0/+1
|
* Consolidate the `access_token` information in the admin api (#11861)Dirk Klimpel2022-01-311-0/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Remove the obsolete MSC1849 configuration flag. (#11843)Patrick Cloke2022-01-311-0/+1
| | | | | MSC1849 was replaced by MSC2675, which was merged. The configuration flag, which defaulted to true, is no longer useful.
* Remove not needed old table of contents in documentation (#11860)Dirk Klimpel2022-01-311-0/+1
|
* Configurable limits on avatars (#11846)Brendan Abolivier2022-01-281-0/+1
| | | | | | Only allow files which file size and content types match configured limits to be set as avatar. Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
* Pass `isolation_level` to `runWithConnection` (#11847)Brendan Abolivier2022-01-271-0/+1
| | | This was missed in https://github.com/matrix-org/synapse/pull/11799
* Create singletons for `StateFilter.{all,none}()` (#11836)Richard van der Hoff2022-01-271-0/+1
| | | No point recreating these for each call, since they are frozen
* Fix some indentation inconsistencies in the sample config (modules) (#11838)Dirk Klimpel2022-01-271-0/+1
|
* Update installation docs to indicate that we support Python 3.10 (#11820)Shay2022-01-261-0/+1
|
* Include `prev_content` field in AS events (#11798)Vaishnav Nair2022-01-261-0/+1
| | | | | | * Include 'prev_content' field in AS events Signed-off-by: Vaishnav Nair <nairvaishnav007@icloud.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Add a module callback to set username at registration (#11790)Brendan Abolivier2022-01-261-0/+1
| | | | | | This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step). This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not. Co-authored-by: David Robertson <davidr@element.io>
* Improvements to bundling aggregations. (#11815)Patrick Cloke2022-01-261-0/+1
| | | | | | | | | | | This is some odds and ends found during the review of #11791 and while continuing to work in this code: * Return attrs classes instead of dictionaries from some methods to improve type safety. * Call `get_bundled_aggregations` fewer times. * Adds a missing assertion in the tests. * Do not return empty bundled aggregations for an event (preferring to not include the bundle at all, as the docstring states).
* Don't print HTTPStatus.* in "Processed..." logs (#11827)David Robertson2022-01-261-0/+1
| | | | | | | | | * Don't print HTTPStatus.* in "Processed..." logs Fixes #11812. See also #7118 and https://github.com/matrix-org/synapse/pull/7188#r401719326 in particular. Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Avoid type annotation problems in prom-client (#11834)David Robertson2022-01-261-0/+1
|
* Add a config flag to inhibit `M_USER_IN_USE` during registration (#11743)Brendan Abolivier2022-01-261-0/+1
| | | | | | | This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work). This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique. More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
* Fix another jsonschema typecheck error (#11830)David Robertson2022-01-252-1/+2
| | | | | | | | | | | | | | | | | | | | | Similar to #11817. In `_create_power_level_validator` we - retrieve `validator`. This is a class implementing the `jsonschema.protocols.Validator` interface. In other words, `validator: Type[jsonschema.protocols.Validator]`. - we then create an second validator class by modifying the original `validator`. We return that class, which is also of type `Type[jsonschema.protocols.Validator]`. So the original annotation was incorrect: it claimed we were returning an instance of jsonSchema.Draft7Validator, not the class (or a subclass) itself. (Strictly speaking this is incorrect, because `POWER_LEVELS_SCHEMA` isn't pinned to a particular version of JSON Schema. But there are other complications with the type stubs if you try to fix this; I felt like the change herein was a decent compromise that better expresses intent). (I suspect/hope the typeshed project would welcome an effort to improve the jsonschema stubs. Let's see if I get some spare time.)
* Check that `gc` method is available before using in `synapse/app/_base` ↵Shay2022-01-251-0/+1
| | | | | | | | | | | | | | | (#11816) * add check that gc.freeze is available before calling * newsfragment * lint * Update comment Co-authored-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* Add admin API to get a list of federated rooms (#11658)Dirk Klimpel2022-01-251-0/+1
|
* CI: run Complement on the VM, not inside Docker (#11811)kegsay2022-01-251-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * CI: run Complement on the VM, not inside Docker This requires https://github.com/matrix-org/complement/pull/289 We now run Complement on the VM instead of inside a Docker container. This is to allow Complement to bind to any high-numbered port when it starts up its own federation servers. We want to do this to allow for more concurrency when running complement tests. Previously, Complement only ever bound to `:8448` when running its own federation server. This prevented multiple federation tests running at the same time as they would fight each other on the port. This did however allow Complement to run in Docker, as the host could just port forward `:8448` to allow homeserver containers to communicate to Complement. Now that we are using random ports however, we cannot use Docker to run Complement. This ends up being a good thing because: - Running Complement tests locally is closer to how they run in CI. - Allows the `CI` env var to be removed in Complement. - Slightly speeds up runs as we don't need to pull down the Complement image prior to running tests. This assumes GHA caches actions sensibly. * Changelog * Full stop * Update .github/workflows/tests.yml Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Review comments * Update .github/workflows/tests.yml Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Log modules at startup (#11813)Brendan Abolivier2022-01-251-0/+1
|
* Db txn set isolation level (#11799)Nick Barrett2022-01-251-0/+1
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Minor updates, and docs, for schema delta files (#11823)Richard van der Hoff2022-01-251-0/+1
| | | | | | | | | * Make functions in python deltas optional It's annoying to always have to write stubs for these. * Documentation for delta files * changelog
* Docs: add missing PR submission process how-tos (#11821)Forest Johnson2022-01-251-0/+1
| | | | | | | * Docs: add missing PR submission process how-tos The documentation says that in order to submit a pull request you have to run the linter and links to [Run the linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters). IMO "Run the linters" should explain that development dependencies are a pre-requisite. I also included `pip install wheel` which I had to run inside my virtual environment on ubuntu before I `pip install -e ".[all,dev]"` would succeed.
* Add admin API to reset connection timeouts for remote server (#11639)Dirk Klimpel2022-01-251-0/+1
| | | * Fix get federation status of destination if no error occured
* Ignore the jsonschema type. (#11817)Patrick Cloke2022-01-251-0/+1
|
* Skip the initial amd64-only Docker build (#11810)Richard van der Hoff2022-01-241-0/+1
| | | | | PyNaCl's recent 1.5.0 release on PyPi includes arm64 wheels, which means our arm64 docker images now build in a sensible amount of time, so we can skip the amd64-only build.
* Apply a timeout to reading the body when fetching a file. (#11784)Patrick Cloke2022-01-241-0/+1
| | | | This prevents the URL preview code from reading a stream forever.
* Support rendering previews with data: URLs in them (#11767)Patrick Cloke2022-01-241-0/+1
| | | | | Images which are data URLs will no longer break URL previews and will properly be "downloaded" and thumbnailed.
* Remove account data (including client config, push rules and ignored users) ↵reivilibre2022-01-241-0/+1
| | | | | upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Drop support for and remove references to EOL Python 3.6 (#11683)Shay2022-01-211-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove reference in comments to python3.6 * upgrade tox python env in script * bump python version in example for completeness * upgrade python version requirement in setup doc * upgrade necessary python version in __init__.py * upgrade python version in setup.py * newsfragment * drops refs to bionic and replace with focal * bump refs to postgres 9.6 to 10 * fix hanging ci * try installing tzdata first * revert change made in b979f336 * ignore new random mypy error while debugging other error * fix lint error for temporary workaround * revert change to install list * try passing env var * export debian frontend var? * move line and add comment * bump pillow dependency * bump lxml depenency * install libjpeg-dev for pillow * bump automat version to one compatible with py3.8 * add libwebp for pillow * bump twisted trunk python version * change suffix of newsfragment * remove redundant python 3.7 checks * lint
* Remove obsolete newsfileOlivier Wilkinson (reivilibre)2022-01-211-1/+0
| | | | The PR was cherrypicked into v1.51.0rc1.
* Merge tag 'v1.51.0rc1' into developOlivier Wilkinson (reivilibre)2022-01-2154-55/+0
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.51.0rc1 (2022-01-21) ============================== Features -------- - Add `track_puppeted_user_ips` config flag to record client IP addresses against puppeted users, and include the puppeted users in monthly active user counts. ([\#11561](https://github.com/matrix-org/synapse/issues/11561), [\#11749](https://github.com/matrix-org/synapse/issues/11749), [\#11757](https://github.com/matrix-org/synapse/issues/11757)) - Include whether the requesting user has participated in a thread when generating a summary for [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11577](https://github.com/matrix-org/synapse/issues/11577)) - Return an `M_FORBIDDEN` error code instead of `M_UNKNOWN` when a spam checker module prevents a user from creating a room. ([\#11672](https://github.com/matrix-org/synapse/issues/11672)) - Add a flag to the `synapse_review_recent_signups` script to ignore and filter appservice users. ([\#11675](https://github.com/matrix-org/synapse/issues/11675), [\#11770](https://github.com/matrix-org/synapse/issues/11770)) Bugfixes -------- - Fix a long-standing issue which could cause Synapse to incorrectly accept data in the unsigned field of events received over federation. ([\#11530](https://github.com/matrix-org/synapse/issues/11530)) - Fix a long-standing bug where Synapse wouldn't cache a response indicating that a remote user has no devices. ([\#11587](https://github.com/matrix-org/synapse/issues/11587)) - Fix an error that occurs whilst trying to get the federation status of a destination server that was working normally. This admin API was newly introduced in Synapse v1.49.0. ([\#11593](https://github.com/matrix-org/synapse/issues/11593)) - Fix bundled aggregations not being included in the `/sync` response, per [MSC2675](https://github.com/matrix-org/matrix-doc/pull/2675). ([\#11612](https://github.com/matrix-org/synapse/issues/11612), [\#11659](https://github.com/matrix-org/synapse/issues/11659), [\#11791](https://github.com/matrix-org/synapse/issues/11791)) - Fix the `/_matrix/client/v1/room/{roomId}/hierarchy` endpoint returning incorrect fields which have been present since Synapse 1.49.0. ([\#11667](https://github.com/matrix-org/synapse/issues/11667)) - Fix preview of some GIF URLs (like tenor.com). Contributed by Philippe Daouadi. ([\#11669](https://github.com/matrix-org/synapse/issues/11669)) - Fix a bug where only the first 50 rooms from a space were returned from the `/hierarchy` API. This has existed since the introduction of the API in Synapse v1.41.0. ([\#11695](https://github.com/matrix-org/synapse/issues/11695)) - Fix a bug introduced in Synapse v1.18.0 where password reset and address validation emails would not be sent if their subject was configured to use the 'app' template variable. Contributed by @br4nnigan. ([\#11710](https://github.com/matrix-org/synapse/issues/11710), [\#11745](https://github.com/matrix-org/synapse/issues/11745)) - Make the 'List Rooms' Admin API sort stable. Contributed by Daniël Sonck. ([\#11737](https://github.com/matrix-org/synapse/issues/11737)) - Fix a long-standing bug where space hierarchy over federation would only work correctly some of the time. ([\#11775](https://github.com/matrix-org/synapse/issues/11775)) - Fix a bug introduced in Synapse v1.46.0 that prevented `on_logged_out` module callbacks from being correctly awaited by Synapse. ([\#11786](https://github.com/matrix-org/synapse/issues/11786)) Improved Documentation ---------------------- - Warn against using a Let's Encrypt certificate for TLS/DTLS TURN server client connections, and suggest using ZeroSSL certificate instead. This works around client-side connectivity errors caused by WebRTC libraries that reject Let's Encrypt certificates. Contibuted by @AndrewFerr. ([\#11686](https://github.com/matrix-org/synapse/issues/11686)) - Document the new `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable in the contributing guide. ([\#11715](https://github.com/matrix-org/synapse/issues/11715)) - Document that the minimum supported PostgreSQL version is now 10. ([\#11725](https://github.com/matrix-org/synapse/issues/11725)) - Fix typo in demo docs: differnt. ([\#11735](https://github.com/matrix-org/synapse/issues/11735)) - Update room spec URL in config files. ([\#11739](https://github.com/matrix-org/synapse/issues/11739)) - Mention `python3-venv` and `libpq-dev` dependencies in the contribution guide. ([\#11740](https://github.com/matrix-org/synapse/issues/11740)) - Update documentation for configuring login with Facebook. ([\#11755](https://github.com/matrix-org/synapse/issues/11755)) - Update installation instructions to note that Python 3.6 is no longer supported. ([\#11781](https://github.com/matrix-org/synapse/issues/11781)) Deprecations and Removals ------------------------- - Remove the unstable `/send_relation` endpoint. ([\#11682](https://github.com/matrix-org/synapse/issues/11682)) - Remove `python_twisted_reactor_pending_calls` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724)) - Remove the `password_hash` field from the response dictionaries of the [Users Admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html). ([\#11576](https://github.com/matrix-org/synapse/issues/11576)) - Deprecate support for `webclient` listeners and non-HTTP(S) `web_client_location` configuration. ([\#11774](https://github.com/matrix-org/synapse/issues/11774), [\#11783](https://github.com/matrix-org/synapse/issues/11783)) Internal Changes ---------------- - Run `pyupgrade --py37-plus --keep-percent-format` on Synapse. ([\#11685](https://github.com/matrix-org/synapse/issues/11685)) - Use buildkit's cache feature to speed up docker builds. ([\#11691](https://github.com/matrix-org/synapse/issues/11691)) - Use `auto_attribs` and native type hints for attrs classes. ([\#11692](https://github.com/matrix-org/synapse/issues/11692), [\#11768](https://github.com/matrix-org/synapse/issues/11768)) - Remove debug logging for #4422, which has been closed since Synapse 0.99. ([\#11693](https://github.com/matrix-org/synapse/issues/11693)) - Remove fallback code for Python 2. ([\#11699](https://github.com/matrix-org/synapse/issues/11699)) - Add a test for [an edge case](https://github.com/matrix-org/synapse/pull/11532#discussion_r769104461) in the `/sync` logic. ([\#11701](https://github.com/matrix-org/synapse/issues/11701)) - Add the option to write SQLite test dbs to disk when running tests. ([\#11702](https://github.com/matrix-org/synapse/issues/11702)) - Improve Complement test output for Gitub Actions. ([\#11707](https://github.com/matrix-org/synapse/issues/11707)) - Fix docstring on `add_account_data_for_user`. ([\#11716](https://github.com/matrix-org/synapse/issues/11716)) - Complement environment variable name change and update `.gitignore`. ([\#11718](https://github.com/matrix-org/synapse/issues/11718)) - Simplify calculation of Prometheus metrics for garbage collection. ([\#11723](https://github.com/matrix-org/synapse/issues/11723)) - Improve accuracy of `python_twisted_reactor_tick_time` Prometheus metric. ([\#11724](https://github.com/matrix-org/synapse/issues/11724), [\#11771](https://github.com/matrix-org/synapse/issues/11771)) - Minor efficiency improvements when inserting many values into the database. ([\#11742](https://github.com/matrix-org/synapse/issues/11742)) - Invite PR authors to give themselves credit in the changelog. ([\#11744](https://github.com/matrix-org/synapse/issues/11744)) - Add optional debugging to investigate [issue 8631](https://github.com/matrix-org/synapse/issues/8631). ([\#11760](https://github.com/matrix-org/synapse/issues/11760)) - Remove `log_function` utility function and its uses. ([\#11761](https://github.com/matrix-org/synapse/issues/11761)) - Add a unit test that checks both `client` and `webclient` resources will function when simultaneously enabled. ([\#11765](https://github.com/matrix-org/synapse/issues/11765)) - Allow overriding complement commit using `COMPLEMENT_REF`. ([\#11766](https://github.com/matrix-org/synapse/issues/11766)) - Add some comments and type annotations for `_update_outliers_txn`. ([\#11776](https://github.com/matrix-org/synapse/issues/11776))
| * 1.51.0rc1Olivier Wilkinson (reivilibre)2022-01-2156-57/+0
| |
| * Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke2022-01-213-1/+2
| |
* | Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff2022-01-211-0/+1
| | | | | | | | ... and start populating them for new events
* | Do not try to serialize raw aggregations dict. (#11791)Patrick Cloke2022-01-213-1/+2
| |
* | Drop unused table `public_room_list_stream`. (#11795)Richard van der Hoff2022-01-211-0/+1
| | | | | | This is a follow-up to #10565.
* | Stop reading from `event_reference_hashes` (#11794)Richard van der Hoff2022-01-211-0/+1
| | | | | | | | Preparation for dropping this table altogether. Part of #6574.
* | Add `FrozenEvent.get_state_key` and use it in a couple of places (#11793)Richard van der Hoff2022-01-211-0/+1
| | | | | | | | This is more efficient, since we only have to look up `state_key` in the event dict once, rather than three (!) times.
* | Make the `get_global_account_data_by_type_for_user` cache be a tree-cache ↵reivilibre2022-01-211-0/+1
| | | | | | | | whose key is prefixed with the user ID (#11788)
* | Make `get_account_data_for_room_and_type` a tree cache (#11789)reivilibre2022-01-211-0/+1
|/
* Correctly await on_logged_out callbacks (#11786)Brendan Abolivier2022-01-201-0/+1
|
* Fix redirecting to the webclient for non-HTTP(S) web_client_location. (#11783)Patrick Cloke2022-01-201-0/+1
| | | | | To not change the behaviour during the deprecation period. Follow-up to #11774.
* Add a regression test for using both webclient and client resources ↵Andrew Morgan2022-01-201-0/+1
| | | | simultaneously (#11765)
* Partially revert #11675; prevent attempting to create pushers on workers ↵Andrew Morgan2022-01-201-0/+1
| | | | (#11770)
* installation.md: drop python 3.6 support (#11781)Richard van der Hoff2022-01-201-0/+1
| | | #11595 dropped support for python 3.6, but forgot to update this doc.
* Add deprecation warnings for `webclient` listener and non-HTTP(S) ↵Patrick Cloke2022-01-201-0/+1
| | | | | | | `web_client_location`. (#11774) This changes the behaviour of the root endpoint to redirect directly to the configuration of `web_client_location` if it is given an HTTP(S) URL.
* Debug for device lists updates (#11760)David Robertson2022-01-201-0/+1
| | | | | | | | | | | | | | | | | | Debug for #8631. I'm having a hard time tracking down what's going wrong in that issue. In the reported example, I could see server A sending federation traffic to server B and all was well. Yet B reports out-of-sync device updates from A. I couldn't see what was _in_ the events being sent from A to B. So I have added some crude logging to track - when we have updates to send to a remote HS - the edus we actually accumulate to send - when a federation transaction includes a device list update edu - when such an EDU is received This is a bit of a sledgehammer.
* Allow overriding the complement ref. (#11766)Nicolas Werner2022-01-201-0/+1
| | | | | Updates complement.sh to read the ref from an environment variable (defaulting to master) when downloading a complement bundle for testing.
* Fix a bug that corrupted the cache of federated space hierarchies (#11775)Sean Quah2022-01-201-0/+1
| | | | `FederationClient.get_room_hierarchy()` caches its return values, so refactor the code to avoid modifying the returned room summary.
* Comments and typing for `_update_outliers_txn` (#11776)Richard van der Hoff2022-01-191-0/+1
| | | | A couple of surprises for me here, so thought I'd document them
* Fix mypy for platforms without epoll support. (#11771)Patrick Cloke2022-01-191-0/+1
|
* Add missing `auto_attribs=True` to the `_WrappedRustReporter` class (#11768)Andrew Morgan2022-01-191-0/+1
|
* Fix preview of imgur and Tenor URLs. (#11669)Philippe Daouadi2022-01-181-0/+1
| | | | | | By scraping Open Graph information from the HTML even when an autodiscovery endpoint is found. The results are then combined to capture as much information as possible from the page.
* Include whether the requesting user has participated in a thread. (#11577)Patrick Cloke2022-01-181-0/+1
| | | | | | Per updates to MSC3440. This is implement as a separate method since it needs to be cached on a per-user basis, instead of a per-thread basis.
* Remove `log_function` and its uses (#11761)Richard van der Hoff2022-01-181-0/+1
| | | | | | | I've never found this terribly useful. I think it was added in the early days of Synapse, without much thought as to what would actually be useful to log, and has just been cargo-culted ever since. Rather, it tends to clutter up debug logs with useless information.
* Add a flag to the `synapse_review_recent_signups` script to ignore and ↵lukasdenk2022-01-171-0/+1
| | | | filter appservice users. (#11675)
* Warn against using Let's Encrypt certs for encrypted TURN (#11686)AndrewFerr2022-01-171-0/+1
| | | | | | | | | * Warn against using Let's Encrypt certs for encrypted TURN This helps to avoid client-side issues: * https://github.com/vector-im/element-android/issues/1533 * https://github.com/vector-im/element-ios/issues/2712 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Fix up docs for `track_puppeted_user_ips` (again) (#11757)Richard van der Hoff2022-01-173-2/+3
| | | Fixes #11741
* Update documentation for configuring facebook login (#11755)Richard van der Hoff2022-01-171-0/+1
| | | ... and a minor thinko fix in the sample config.
* Improve `reactor_tick_time` metric (#11724)Richard van der Hoff2022-01-172-0/+2
| | | | | | | | | | | The existing implementation of the `python_twisted_reactor_tick_time` metric is pretty useless, because it *only* measures the time taken to execute timed calls and callbacks from threads. That neglects everything that happens off the back of I/O, which is obviously quite a lot for us. To improve this, I've hooked into a different place in the reactor - in particular, where it calls `epoll`. That call is the only place it should wait for something to happen - the rest of the loop *should* be quick. I've also removed `python_twisted_reactor_pending_calls`, because I don't believe anyone ever looks at it, and it's a nuisance to populate.
* Make pagination of rooms in admin api stable (#11737)Daniel Sonck2022-01-171-0/+1
| | | | | | | | | | | | | | Always add state.room_id after the configurable ORDER BY. Otherwise, for any sort, certain pages can contain results from other pages. (Especially when sorting by creator, since there may be many rooms by the same creator) * Document different order direction of numerical fields "joined_members", "joined_local_members", "version" and "state_events" are ordered in descending direction by default (dir=f). Added a note in tests to explain the differences in ordering. Signed-off-by: Daniël Sonck <daniel@sonck.nl>
* Remove the 'password_hash' from the Users Admin API endpoint response ↵Andrew Morgan2022-01-141-0/+1
| | | | dictionary (#11576)
* Fix sample_config.yaml in regards track_puppeted_user_ips (#11749)Jason Robinson2022-01-141-0/+1
| | | | | | | * Fix sample_config.yaml in regards track_puppeted_user_ips Closes #11741 Signed-off-by: Jason Robinson <jasonr@matrix.org>
* Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke2022-01-131-0/+1
| | | | This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
* Fix missing app variable in mail subject (#11745)David Robertson2022-01-132-0/+2
| | | | | documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500 Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>
* Invite PR submitters to credit themselves (#11744)David Robertson2022-01-131-0/+1
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* update room spec url in config files (#11739)qwertyforce2022-01-131-0/+1
| | | | | | | * change spec url in config files * Create 11739.txt * .txt -> .doc
* Mention python3-venv and libpq-dev dependencies in contribution guide (#11740)Andy Balaam2022-01-131-0/+1
|
* Include bundled aggregations in the sync response cache. (#11659)Patrick Cloke2022-01-131-0/+1
|
* Simplify GC prometheus metrics (#11723)Richard van der Hoff2022-01-131-0/+1
| | | | | Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection. Part 1 of a quest to simplify the reactor monkey-patching.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-131-0/+1
|
* Fix typo in demo docs: differnt (#11735)Andy Balaam2022-01-131-0/+1
|
* Allow tracking puppeted users for MAU (#11561)Jason Robinson2022-01-121-0/+1
| | | | | | | | | | | | | | | Currently when puppeting another user, the user doing the puppeting is tracked for client IPs and MAU (if configured). When tracking MAU is important, it becomes necessary to be possible to also track the client IPs and MAU of puppeted users. As an example a client that manages user creation and creation of tokens via the Synapse admin API, passing those tokens for the client to use. This PR adds optional configuration to enable tracking of puppeted users into monthly active users. The default behaviour stays the same. Signed-off-by: Jason Robinson <jasonr@matrix.org>
* Fix documentation of supported PostgreSQL version (#11725)haslersn2022-01-121-0/+1
| | | Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>
* Use buildkit's cache feature to speed up docker builds (#11691)Richard van der Hoff2022-01-121-0/+1
| | | Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
* Complement environment variable name change and update .gitignore. (#11718)Michael Kaye2022-01-101-0/+1
|
* Document the `SYNAPSE_TEST_PERSIST_SQLITE_DB` unit test env var (#11715)Andrew Morgan2022-01-101-0/+1
|
* Fix docstring on `add_account_data_for_user`. (#11716)reivilibre2022-01-101-0/+1
|
* Deal with mypy errors w/ type-hinted pynacl 1.5.0 (#11714)David Robertson2022-01-101-0/+1
| | | | | | | | | | | | * Deal with mypy errors w/ type-hinted pynacl 1.5.0 Fixes #11644. I really don't like that we're monkey patching pynacl SignedKey instances with alg and version objects. But I'm too scared to make the changes necessary right now. (Ideally I would replace `signedjson.types.SingingKey` with a runtime class which wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
* Prettier complement logs (#11707)kegsay2022-01-101-0/+1
| | | | | * Prettier complement logs * Changelog
* Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)Patrick Cloke2022-01-071-0/+1
| | | | | | | By returning all of the m.space.child state of the space, not just the first 50. The number of rooms returned is still capped at 50. For the federation API this implies that the requesting server will need to individually query for any other rooms it is not joined to.
* Optionally use an on-disk sqlite db in tests (#11702)David Robertson2022-01-071-0/+1
| | | | | | | | | | | | | | | * Optionally use an on-disk sqlite db in tests When debugging a test it is sometimes useful to inspect the state of the DB. This is not easy when the db is in-memory: one cannot attach the sqlite CLI to another process's DB. With this change, if SYNAPSE_TEST_PERSIST_SQLITE_DB is set, we use `_trial_temp/test.db` as our sqlite database. One can then use `sqlite3 _trial_temp/test.db` and query to your heart's content. The DB is destroyed and recreated between different test cases. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Test that bans win a join against a race when computing `/sync` response ↵David Robertson2022-01-071-0/+1
| | | | (#11701)
* Merge branch 'release-v1.50' into developOlivier Wilkinson (reivilibre)2022-01-0772-72/+0
|\
| * Re-run Towncrier to add in the changelog entry for the Mjolnir workaround v1.50.0rc1Olivier Wilkinson (reivilibre)2022-01-061-1/+0
| |
| * Work around Mjolnir compatibility issue by adding an import for ↵reivilibre2022-01-061-0/+1
| | | | | | | | `glob_to_regex` in `synapse.util`, where it moved from. (#11696)
| * Re-run towncrier.Patrick Cloke2022-01-051-1/+0
| |
| * Include `io.element.thread` capability for MSC3440. (#11690)Patrick Cloke2022-01-051-0/+1
| |
| * 1.50.0rc1Olivier Wilkinson (reivilibre)2022-01-0572-72/+0
| |
* | Bundle aggregations outside of the serialization method. (#11612)Patrick Cloke2022-01-071-0/+1
| | | | | | | | | | | | | | | | This makes the serialization of events synchronous (and it no longer access the database), but we must manually calculate and provide the bundled aggregations. Overall this should cause no change in behavior, but is prep work for other improvements.
* | Remove the /send_relation endpoint. (#11682)Patrick Cloke2022-01-061-0/+1
| | | | | | | | This was removed from MSC2674 before that was approved and is not used by any known clients.
* | Remove a Python 2-ism and improve type hints. (#11699)Patrick Cloke2022-01-061-0/+1
| | | | | | | | On Python 2, indexing a byte-string gives back a byte-string, while on Python 3 it gives back the ASCII equivalent as an int.
* | Strip unauthorized fields from `unsigned` object in events received over ↵Shay2022-01-061-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | federation (#11530) * add some tests to verify we are stripping unauthorized fields out of unsigned * add function to strip unauthorized fields from the unsigned object of event * newsfragment * update newsfragment number * add check to on_send_membership_event * refactor tests * fix lint error * slightly refactor tests and add some comments * slight refactor * refactor tests * fix import error * slight refactor * remove unsigned filtration code from synapse/handlers/federation_event.py * lint * move unsigned filtering code to event base * refactor tests * update newsfragment * requested changes * remove unused retun values
* | Make room creations denied by `user_may_create_room` cause an `M_FORBIDDEN` ↵lukasdenk2022-01-061-0/+1
| | | | | | | | | | error to be returned, not `M_UNKNOWN` (#11672) Co-authored-by: reivilibre <olivier@librepush.net>
* | Remove debug logging for #4422 (#11693)David Robertson2022-01-061-0/+1
| | | | | | | | as per https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
* | Fix space hierarchy endpoint to match MSC2946 (#11667)Travis Ralston2022-01-051-0/+1
| | | | | | | | | | | | | | Fixes minor discrepancies between the /hierarchy endpoint described in MSC2946 and the implementation. Note that the changes impact the stable and unstable /hierarchy and unstable /spaces endpoints for both client and federation APIs.
* | Fix get federation status of destination if no error occured (#11593)Dirk Klimpel2022-01-051-0/+1
| |
* | Run `pyupgrade --py37-plus --keep-percent-format` on Synapse (#11685)Shay2022-01-051-0/+1
| | | | | | | | | | | | | | | | | | * newsfragment * fix newsfragment number * update changelog * remove extra space
* | Cache empty responses from `/user/devices` (#11587)David Robertson2022-01-051-0/+1
|/ | | If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
* Refactor the way we set `outlier` (#11634)Richard van der Hoff2022-01-051-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * `_auth_and_persist_outliers`: mark persisted events as outliers Mark any events that get persisted via `_auth_and_persist_outliers` as, well, outliers. Currently this will be a no-op as everything will already be flagged as an outlier, but I'm going to change that. * `process_remote_join`: stop flagging as outlier The events are now flagged as outliers later on, by `_auth_and_persist_outliers`. * `send_join`: remove `outlier=True` The events created here are returned in the result of `send_join` to `FederationHandler.do_invite_join`. From there they are passed into `FederationEventHandler.process_remote_join`, which passes them to `_auth_and_persist_outliers`... which sets the `outlier` flag. * `get_event_auth`: remove `outlier=True` stop flagging the events returned by `get_event_auth` as outliers. This method is only called by `_get_remote_auth_chain_for_event`, which passes the results into `_auth_and_persist_outliers`, which will flag them as outliers. * `_get_remote_auth_chain_for_event`: remove `outlier=True` we pass all the events into `_auth_and_persist_outliers`, which will now flag the events as outliers. * `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter This param is now never set to True, so we can remove it. * `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param This is no longer set anywhere, so we can remove it. * `get_pdu`: remove unused `outlier` parameter ... and chase it down into `get_pdu_from_destination_raw`. * `event_from_pdu_json`: remove redundant `outlier` param This is never set to `True`, so can be removed. * changelog * update docstring
* Fix link from generated configuration file to documentation (#11678)Philipp Matthias Schäfer2022-01-051-0/+1
| | | | Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: reivilibre <oliverw@matrix.org>
* Fix SimpleHttpClient not sending Accept header in `get_json` (#11677)Fr3shTea2022-01-051-0/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Add admin API to get users' account data (#11664)Dirk Klimpel2022-01-051-0/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre2022-01-052-0/+2
| | | | | #11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
* Clarify SSO mapping provider documentation by writing `def` or `async def` ↵reivilibre2022-01-051-0/+1
| | | | before the names of methods, as appropriate. (#11681)
* Improve Docker docs for use with Postgres (#11640)Callum Macdonald2022-01-051-0/+1
|
* Correct Synapse install command for FreeBSD. (#11267)Donny Johnson2022-01-051-0/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* update ngnix reverse-proxy example (#11680)Richard van der Hoff2022-01-041-0/+1
| | | this should not be a case-insensitive match.
* Fix AssertionErrors after purging events (#11642)Richard van der Hoff2022-01-043-1/+2
| | | | | | | | | | | * Fix AssertionErrors after purging events If you purged a bunch of events from your database, and then restarted synapse without receiving more events, then you would get a bunch of AssertionErrors on restart. This fixes the situation by rewinding the stream processors. * `check-newsfragment`: ignore deleted newsfiles
* `FederationClient.backfill`: stop flagging events as outliers (#11632)Richard van der Hoff2022-01-041-0/+1
| | | | | | | | | | | | | | | | | | | | | Events returned by `backfill` should not be flagged as outliers. Fixes: ``` AssertionError: null File "synapse/handlers/federation.py", line 313, in try_backfill dom, room_id, limit=100, extremities=extremities File "synapse/handlers/federation_event.py", line 517, in backfill await self._process_pulled_events(dest, events, backfilled=True) File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events await self._process_pulled_event(origin, ev, backfilled=backfilled) File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event assert not event.internal_metadata.is_outlier() ``` See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992 Fixes #8894.
* Remove redundant `get_current_events_token` (#11643)Richard van der Hoff2022-01-041-0/+1
| | | | | | | | | | | | | | | | | * Push `get_room_{min,max_stream_ordering}` into StreamStore Both implementations of this are identical, so we may as well push it down and get rid of the abstract base class nonsense. * Remove redundant `StreamStore` class This is empty now * Remove redundant `get_current_events_token` This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid of it. * newsfile
* Better error messages from `get_create_event_for_room` (#11638)Richard van der Hoff2022-01-041-0/+1
| | | | "Unknown room" can mean a multitude of things here. To help with debugging, add some more words to the exception text.
* Include the topic event in the prejoin state, per MSC3173. (#11666)Patrick Cloke2022-01-041-0/+1
| | | | Invites and knocks will now include the topic in the stripped state send to clients before joining the room.
* Drop Bionic from Debian builds (#11633)Shay2022-01-031-0/+1
| | | | | | | | | | | | | | | | | * update Trove classifiers to remove py36 * stop building bionic * update dh-virtualenv * newsfragment * fix newsfragment * update version refs * another try at correct tag * Update changelog
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-0/+1
| | | To improve type hints throughout the code.
* Add type hints to `synapse/storage/databases/main/events_bg_updates.py` (#11654)Dirk Klimpel2021-12-301-0/+1
|
* Fix a type annotation in `test_account_data.py` and remove it from the Mypy ↵reivilibre2021-12-291-0/+1
| | | | | exclusion list. (#11657) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Improve type hints in storage classes. (#11652)Dirk Klimpel2021-12-291-0/+1
| | | By using cast and making ignores more specific.
* Do not attempt to bundled aggregations for /members and /state. (#11623)Patrick Cloke2021-12-291-0/+1
| | | | Both of those APIs return state events, which will not have bundled aggregations added anyway.
* Add type hints to `synapse/storage/databases/main/stats.py` (#11653)Dirk Klimpel2021-12-291-0/+1
|
* Update to the current version of Black and run it on Synapse codebase (#11596)Shay2021-12-231-0/+1
| | | | | | | | | * update black version * run updated version of black on code * newsfragment * enumerate python versions
* Fix mypy error with opentracing.tags. (#11622)Patrick Cloke2021-12-231-0/+1
|
* Add details for how to set up TURN behind NAT (#11553)AndrewFerr2021-12-231-0/+1
| | | | Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Drop EOL python 3.6 from CI (#11595)Shay2021-12-211-0/+1
| | | | | | | | | | | * remove python 3.6 and postgres 9.6 from github workflow * remove python 3.6 env from tox * newsfragment * correct postgres version * add py310 to tox env list
* Merge branch 'master' into developRichard van der Hoff2021-12-211-1/+0
|\
* | Refactor `tests.util.setup_test_homeserver` and ↵reivilibre2021-12-211-0/+1
| | | | | | | | `tests.server.setup_test_homeserver`. (#11503)
* | Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-211-0/+1
| |
* | Various opentracing enhancements (#11619)Richard van der Hoff2021-12-211-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Wrap `auth.get_user_by_req` in an opentracing span give `get_user_by_req` its own opentracing span, since it can result in a non-trivial number of sub-spans which it is useful to group together. This requires a bit of reorganisation because it also sets some tags (and may force tracing) on the servlet span. * Emit opentracing span for encoding json responses This can be a significant time sink. * Rename all sync spans with a prefix * Write an opentracing span for encoding sync response * opentracing span to group generate_room_entries * opentracing spans within sync.encode_response * changelog * Use the `trace` decorator instead of context managers
* | Do not bundle aggregations for APIs which shouldn't include them. (#11592)Patrick Cloke2021-12-201-0/+1
| | | | | | | | | | And make bundling aggregations opt-in, instead of opt-out to avoid having APIs to include extraneous data (and being much heavier than necessary).
* | Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff2021-12-201-0/+1
| | | | | | This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
* | Improve opentracing for incoming HTTP requests (#11618)Richard van der Hoff2021-12-201-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove `start_active_span_from_request` Instead, pull out a separate function, `span_context_from_request`, to extract the parent span, which we can then pass into `start_active_span` as normal. This seems to be clearer all round. * Remove redundant tags from `incoming-federation-request` These are all wrapped up inside a parent span generated in AsyncResource, so there's no point duplicating all the tags that are set there. * Leave request spans open until the request completes It may take some time for the response to be encoded into JSON, and that JSON to be streamed back to the client, and really we want that inside the top-level span, so let's hand responsibility for closure to the SynapseRequest. * opentracing logs for HTTP request events * changelog
* | Merge remote-tracking branch 'origin/release-v1.49' into developRichard van der Hoff2021-12-201-0/+1
|\ \
| * | Disable aggregation bundling on `/sync` responses (#11583)Richard van der Hoff2021-12-201-0/+1
| |/ | | | | | | | | | | | | | | | | | | | | * Disable aggregation bundling on `/sync` responses A partial revert of #11478. This turns out to have had a significant CPU impact on initial-sync handling. For now, let's disable it, until we find a more efficient way of achieving this. * Fix tests. Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* | Return JSON errors for unknown resources under /matrix/client. (#11602)Patrick Cloke2021-12-201-0/+1
| | | | | | | | Instead of returning 404 errors with HTML bodies when an unknown prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
* | Use mock from standard library (#11588)V024602021-12-201-0/+1
| | | | | | Instead of the backported version.
* | Add opentracing types (#11603)Shay2021-12-201-0/+1
| |
* | Add type hints to `synapse/tests/rest/admin` (#11590)Dirk Klimpel2021-12-161-0/+1
| |
* | Add type hints to `synapse/storage/databases/main/transactions.py` (#11589)Dirk Klimpel2021-12-161-0/+1
| |
* | Add MSC2716 and MSC3030 to `/versions` -> `unstable_features` (#11582)Eric Eastwood2021-12-161-0/+1
| | | | | | | | As suggested in https://github.com/matrix-org/matrix-react-sdk/pull/7372#discussion_r769523369
* | Add type hints to `synapse/storage/databases/main/room.py` (#11575)Sean Quah2021-12-151-0/+1
| |
* | Require Collections as the parameters for simple_* methods. (#11580)Patrick Cloke2021-12-151-0/+1
| | | | | | | | Instead of Iterable since the generators are not allowed due to the potential for their re-use.
* | Convert EventStreamResult to attrs. (#11574)Patrick Cloke2021-12-151-0/+1
| |
* | Add experimental support for MSC3202: allowing application services to ↵reivilibre2021-12-151-0/+1
| | | | | | | | masquerade as specific devices. (#11538)
* | Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah2021-12-141-0/+1
| |
* | Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-141-0/+1
| |
* | Add missing type hints to `synapse.appservice` (#11360)Patrick Cloke2021-12-141-0/+1
| |
* | Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah2021-12-141-0/+1
| | | | | | | | | | `COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
* | Add missing type hints to synapse.http. (#11571)Patrick Cloke2021-12-141-0/+1
| |
* | checks for generators in database functions (#11564)Richard van der Hoff2021-12-131-0/+1
| | | | | | | | | | A couple of safety-checks to hopefully stop people doing what I just did, and create a storage function which only works the first time it is called (and not when it is re-run due to a database concurrency error or similar).
* | Move HTML parsing to a separate file for URL previews. (#11566)Patrick Cloke2021-12-131-0/+1
| | | | | | | | | | | | | | * Splits the logic for parsing HTML from the resource handling code. * Fix a circular import in the oEmbed code (which uses the HTML parsing code). * Renames some of the HTML parsing methods to: * Make it clear which methods are "internal" to the module. * Clarify what the methods do.
* | Type hint the constructors of the data store classes (#11555)Sean Quah2021-12-131-0/+1
| |
* | Add type hints to `synapse/storage/databases/main/end_to_end_keys.py` (#11551)Sean Quah2021-12-131-0/+1
| |
* | Add type hints to `synapse/storage/databases/main/account_data.py` (#11546)Sean Quah2021-12-131-0/+1
| |
* | Make `get_device` return None if the device doesn't exist rather than ↵reivilibre2021-12-131-0/+1
| | | | | | | | | | raising an exception. (#11565) Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* | Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-101-0/+1
| | | | | | | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* | Test to ensure we share the same `state_group` across the whole historical ↵Eric Eastwood2021-12-101-0/+1
| | | | | | | | | | | | | | | | batch (MSC2716) (#11487) Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716 We did some work on making sure the `state_groups` were shared in https://github.com/matrix-org/synapse/pull/10975
* | Adjust _get_rooms_changed comments (#11550)David Robertson2021-12-101-0/+1
| | | | | | C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
* | skip some dict munging in event persistence (#11560)Richard van der Hoff2021-12-101-0/+1
| | | | | | | | | | | | Create a new dict helper method `simple_insert_many_values_txn`, which takes raw row values, rather than {key=>value} dicts. This saves us a bunch of dict munging, and makes it easier to use generators rather than creating intermediate lists and dicts.
* | Stop populating `state_events.prev_state` (#11558)Richard van der Hoff2021-12-101-0/+1
| | | | | | this field is never read, so we may as well stop populating it.
* | Ensure emails are canonicalized before fetching associated user. (#11547)Patrick Cloke2021-12-101-0/+1
| | | | | | | | This should fix pushers with an email in non-canonical form is used as the pushkey.
* | Do not allow cross-room relations, per MSC2674. (#11516)Patrick Cloke2021-12-091-0/+1
| |
* | Additional type hints for the config module, part 2. (#11480)Patrick Cloke2021-12-091-0/+1
| |
* | Use HTTPStatus constants in place of literals in `synapse.http` (#11543)Dirk Klimpel2021-12-091-0/+1
| |
* | Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi2021-12-091-0/+1
| |
* | Add missing `errcode` to `parse_string` and `parse_boolean` (#11542)Dirk Klimpel2021-12-091-0/+1
| |
* | Allow guests to send state events (#11378)Robert Long2021-12-091-0/+1
| |
* | Add a constant for receipt types (m.read). (#11531)Patrick Cloke2021-12-081-0/+1
| | | | | | And expand some type hints in the receipts storage module.
* | Clean up `synapse.rest.admin` (#11535)Dirk Klimpel2021-12-081-0/+1
| |
* | Document the usage of refresh tokens. (#11427)reivilibre2021-12-081-0/+1
| | | | | | Co-authored-by: David Robertson <davidr@element.io>
* | Use HTTPStatus constants in place of literals in ↵reivilibre2021-12-081-0/+1
| | | | | | | | `tests.rest.client.test_auth`. (#11520)
* | Improve log messages for stream ids (#11536)Richard van der Hoff2021-12-081-0/+1
| | | | | | | | Somehow I'd managed to get my database in a pickle with stream ids. These changes were useful to debug.
* | Send and handle cross-signing messages using the stable prefix. (#10520)Hubert Chathi2021-12-081-0/+1
| |
* | Fix case in `wait_for_background_updates` where `self.store` does not exist ↵Andrew Morgan2021-12-071-0/+1
|/ | | | | | (#11331) Pull the DataStore from the HomeServer instance, which always exists.
* 1.49.0rc1Sean Quah2021-12-0762-62/+0
|
* Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah2021-12-071-0/+1
| | | | | | `matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
* Refactors in `_generate_sync_entry_for_rooms` (#11515)David Robertson2021-12-072-1/+2
| | | | | | | | * Move sync_token up to the top * Pull out _get_ignored_users * Try to signpost the body of `_generate_sync_entry_for_rooms` * Pull out _calculate_user_changes Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Correctly register shutdown handler for presence workers (#11518)David Robertson2021-12-071-0/+1
| | | Fixes #11517
* Fix `ModuleApi.looping_background_call` for non-async functions (#11524)Sean Quah2021-12-071-0/+1
| | | | | | After #10847, `looping_background_call` would print an error in the logs every time a non-async function was called. Since the error would be caught and ignored immediately, there were no other side effects.
* Fix 'delete room' admin api to work on incomplete rooms (#11523)Richard van der Hoff2021-12-071-0/+1
| | | | | If, for some reason, we don't have the create event, we should still be able to purge a room.
* Correctly ignore invites from ignored users (#11511)David Robertson2021-12-071-0/+1
|
* Fix the test breakage introduced by #11435 as a result of concurrent PRs ↵reivilibre2021-12-071-0/+1
| | | | (#11522)
* Stabilise support for MSC2918 refresh tokens as they have now been merged ↵reivilibre2021-12-061-0/+1
| | | | into the Matrix specification. (#11435)
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-061-0/+1
| | | As a step towards allowing back-channel logout for OIDC.
* Add admin API to get some information about federation status (#11407)Dirk Klimpel2021-12-061-0/+1
|
* Include bundled aggregations in /sync and related fixes (#11478)Patrick Cloke2021-12-061-0/+1
| | | | | | | | Due to updates to MSC2675 this includes a few fixes: * Include bundled aggregations for /sync. * Do not include bundled aggregations for /initialSync and /events. * Do not bundle aggregations for state events. * Clarifies comments and variable names.
* Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)Sean Quah2021-12-061-0/+1
|
* Update backward extremity docs to make it clear that it does not indicate ↵Eric Eastwood2021-12-031-0/+1
| | | | | | | whether we have fetched an events' `prev_events` (#11469) Spawning from https://github.com/matrix-org/synapse/pull/9445#discussion_r758958181 Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-30 20:42:33 +0000