summary refs log tree commit diff
path: root/changelog.d (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'erikj/cache_mem_size' into erikj/test_sendErik Johnston2021-05-051-0/+1
|\
| * Merge remote-tracking branch 'origin/develop' into erikj/cache_mem_sizeErik Johnston2021-05-0435-27/+8
| |\
| * | NewsfileErik Johnston2021-04-261-0/+1
| | |
* | | Merge branch 'erikj/fix_presence_joined' into erikj/test_sendErik Johnston2021-05-042-0/+1
|\ \ \
| * | | Bugfix newsfileErik Johnston2021-05-041-0/+1
| | | |
| * | | Move newsfileErik Johnston2021-05-041-0/+0
| | | |
* | | | Merge remote-tracking branch 'origin/develop' into erikj/test_sendErik Johnston2021-05-042-0/+2
|\ \ \ \
| * | | | Time external cache response time (#9904)Erik Johnston2021-05-041-0/+1
| | |_|/ | |/| |
| * | | Add port parameter to the sample config for psycopg2 args (#9911)Andrew Morgan2021-05-041-0/+1
| | | | | | | | | | | | Adds the `port` option with the default value to the sample config file.
* | | | Merge branch 'erikj/efficient_presence_join' into erikj/test_sendErik Johnston2021-04-301-0/+1
|\ \ \ \
| * | | | NewsfileErik Johnston2021-04-301-0/+1
| | |/ / | |/| |
* | | | Merge branch 'erikj/limit_how_often_gc' into erikj/test_sendErik Johnston2021-04-301-0/+1
|\ \ \ \
| * | | | NewsfileErik Johnston2021-04-281-0/+1
| | | | |
* | | | | Merge branch 'erikj/fix_presence_joined' into erikj/test_sendErik Johnston2021-04-301-0/+1
|\ \ \ \ \ | | |/ / / | |/| | |
| * | | | NewsfileErik Johnston2021-04-291-0/+1
| | |/ / | |/| |
* | | | Merge remote-tracking branch 'origin/develop' into erikj/test_sendErik Johnston2021-04-2939-32/+7
|\| | |
| * | | Merge tag 'v1.33.0rc2' into developAndrew Morgan2021-04-291-1/+0
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.33.0rc2 (2021-04-29) ============================== Bugfixes -------- - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))
| | * | | 1.33.0rc2Andrew Morgan2021-04-291-1/+0
| | | | |
| * | | | Add missing type hints to handlers and fix a Spam Checker type hint. (#9896)Patrick Cloke2021-04-292-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | The user_may_create_room_alias method on spam checkers declared the room_alias parameter as a str when in reality it is passed a RoomAlias object.
| * | | | Delete room endpoint (#9889)ThibF2021-04-292-0/+2
| | |/ / | |/| | | | | | | | | | | | | | | | | | Support the delete of a room through DELETE request and mark previous request as deprecated through documentation. Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
| * | | Merge remote-tracking branch 'origin/release-v1.33.0' into developErik Johnston2021-04-281-0/+1
| |\| |
| | * | Fix tight loop handling presence replication. (#9900)Erik Johnston2021-04-281-0/+1
| | | | | | | | | | | | | | | | | | | | Only affects workers. Introduced in #9819. Fixes #9899.
| * | | Use the parent's logging context name for runWithConnection. (#9895)Patrick Cloke2021-04-281-0/+1
| | | | | | | | | | | | | | | | | | | | This fixes a regression where the logging context for runWithConnection was reported as runWithConnection instead of the connection name, e.g. "POST-XYZ".
| * | | Merge tag 'v1.33.0rc1' into developAndrew Morgan2021-04-2835-35/+0
| |\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.33.0rc1 (2021-04-28) ============================== Features -------- - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814)) - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802)) - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162)) - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786)) - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821)) - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845)) - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879)) - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
| | * | 1.33.0rc1Andrew Morgan2021-04-2834-34/+0
| | | |
| | * | Revert "Experimental Federation Speedup (#9702)"Andrew Morgan2021-04-281-1/+0
| | | | | | | | | | | | | | | | This reverts commit 05e8c70c059f8ebb066e029bc3aa3e0cefef1019.
| * | | Reduce memory footprint of caches (#9886)Erik Johnston2021-04-281-0/+1
| | | |
| * | | Add type hints to presence handler (#9885)Erik Johnston2021-04-281-0/+1
| |/ /
| * | Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-0/+1
| | | | | | | | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
| * | Pass errors back to the client when trying multiple federation destinations. ↵Patrick Cloke2021-04-271-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | (#9868) This ensures that something like an auth error (403) will be returned to the requester instead of attempting to try more servers, which will likely result in the same error, and then passing back a generic 400 error.
| * | Use current state table for `presence.get_interested_remotes` (#9887)Erik Johnston2021-04-271-0/+1
| | | | | | | | | This should be a lot quicker than asking the state handler.
* | | NewsfileErik Johnston2021-04-261-0/+1
|/ /
* | Improved validation for received requests (#9817)Richard van der Hoff2021-04-231-0/+1
| | | | | | | | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* | Kill off `_PushHTTPChannel`. (#9878)Richard van der Hoff2021-04-231-0/+1
| | | | | | | | | | First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation. Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
* | Allow OIDC cookies to work on non-root public baseurls (#9726)Andrew Morgan2021-04-231-0/+1
| | | | | | | | | | Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574. As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.
* | pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-231-0/+1
| |
* | Add type hints to auth and auth_blocking. (#9876)Patrick Cloke2021-04-231-0/+1
|/
* Make DomainSpecificString an attrs class (#9875)Erik Johnston2021-04-231-0/+1
|
* Remove room and user invite ratelimits in default unit test config (#9871)Andrew Morgan2021-04-231-0/+1
|
* Split presence out of master (#9820)Erik Johnston2021-04-231-0/+1
|
* Check for space membership during a remote join of a restricted room (#9814)Patrick Cloke2021-04-231-0/+1
| | | | | | When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Only store data in caches, not "smart" objects (#9845)Erik Johnston2021-04-231-0/+1
|
* Limit the size of HTTP responses read over federation. (#9833)Richard van der Hoff2021-04-231-0/+1
|
* Handle all new rate limits in demo scripts (#9858)manuroe2021-04-221-0/+1
|
* Limit length of accepted email addresses (#9855)Erik Johnston2021-04-221-0/+1
|
* Clear the resync bit after resyncing device lists (#9867)Richard van der Hoff2021-04-221-0/+1
| | | Fixes #9866.
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-221-0/+1
| | | This is no longer required, since we have dropped support for Python 3.5.
* Rename handler and config modules which end in handler/config. (#9816)Patrick Cloke2021-04-201-0/+1
|
* Add presence federation stream (#9819)Erik Johnston2021-04-201-0/+1
|
* Fix bug where we sent remote presence states to remote servers (#9850)Erik Johnston2021-04-201-0/+1
|
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-0/+1
|
* Port "Allow users to click account renewal links multiple times without ↵Andrew Morgan2021-04-191-0/+1
| | | | | hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
* Sanity check identity server passed to bind/unbind. (#9802)Denis Kasak2021-04-191-0/+1
| | | | Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* Don't send normal presence updates over federation replication stream (#9828)Erik Johnston2021-04-191-0/+1
|
* User directory: use calculated room membership state instead (#9821)Andrew Morgan2021-04-161-0/+1
| | | | | Fixes: #9797. Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
* Small speed up joining large remote rooms (#9825)Erik Johnston2021-04-161-0/+1
| | | | | There are a couple of points in `persist_events` where we are doing a query per event in series, which we can replace.
* remove `HomeServer.get_config` (#9815)Richard van der Hoff2021-04-141-0/+1
| | | | Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.
* Separate creating an event context from persisting it in the federation ↵Patrick Cloke2021-04-141-0/+1
| | | | | | handler (#9800) This refactoring allows adding logic that uses the event context before persisting it.
* Revert "Check for space membership during a remote join of a restricted ↵Patrick Cloke2021-04-142-2/+0
| | | | | | | | room. (#9763)" This reverts commit cc51aaaa7adb0ec2235e027b5184ebda9b660ec4. The PR was prematurely merged and not yet approved.
* Check for space membership during a remote join of a restricted room. (#9763)Patrick Cloke2021-04-142-0/+2
| | | | | | | When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* Experimental Federation Speedup (#9702)Jonathan de Jong2021-04-141-0/+1
| | | | | This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max). Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Move some replication processing out of generic_worker (#9796)Erik Johnston2021-04-141-0/+1
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* More robust handling of the Content-Type header for thumbnail generation (#9788)rkfg2021-04-141-0/+1
| | | | Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-0/+1
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Add a dockerfile for running a set of Synapse worker processes (#9162)Andrew Morgan2021-04-141-0/+1
| | | | | | | | | | | | | | | | | This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with: * A Synapse main process. * Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime. * A redis for worker communication. * A nginx for routing traffic. * A supervisord to start all worker processes and monitor them if any go down. Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite. `configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI. Huge thanks to @MatMaul for helping get this all working :tada: This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62. Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
* Add note to docker docs explaining platform support (#9801)Andrew Morgan2021-04-141-0/+1
| | | | | | | Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894. I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support. The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
* 1.32.0rc1Andrew Morgan2021-04-1335-35/+0
|
* Add release helper script (#9713)Erik Johnston2021-04-131-0/+1
| | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-131-0/+1
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Add an admin API to manage ratelimit for a specific user (#9648)Dirk Klimpel2021-04-131-0/+1
|
* Drop Python 3.5 from Trove classifier metadata. (#9782)Dan Callahan2021-04-121-0/+1
| | | | | * Drop Python 3.5 from Trove classifier metadata. Signed-off-by: Dan Callahan <danc@element.io>
* Add option to skip unit tests when building debs (#9793)Dan Callahan2021-04-121-0/+1
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Require AppserviceRegistrationType (#9548)Will Hunt2021-04-121-0/+1
| | | This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-091-0/+1
|
* Fix duplicate logging of exceptions in transaction processing (#9780)Richard van der Hoff2021-04-091-0/+1
| | | There's no point logging this twice.
* Enable complement tests for MSC2946. (#9771)Patrick Cloke2021-04-091-0/+1
| | | By providing the additional build tag for `msc2946`.
* Proof of concept for GitHub Actions (#9661)Dan Callahan2021-04-091-0/+1
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Remove old admin API `GET /_synapse/admin/v1/users/<user_id>` (#9401)Dirk Klimpel2021-04-091-0/+1
| | | | | | | | | | | | | Related: #8334 Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25) `GET /_synapse/admin/v1/users/<user_id>` has no - unit tests - documentation API in v2 is available (#5925 - 12/2019, v1.7.0). API is misleading. It expects `user_id` and returns a list of all users. Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-0/+1
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Merge remote-tracking branch 'origin/develop' into rav/drop_py35Richard van der Hoff2021-04-0813-2/+11
|\
| * Merge pull request #9769 from matrix-org/rav/fix_bionicRichard van der Hoff2021-04-081-0/+1
| |\ | | | | | | Fix incompatibility with bionic
| * \ Merge branch 'erikj/fix_stalled_catchup' into developErik Johnston2021-04-081-0/+1
| |\ \
| | * | Fix sharded federation sender sometimes using 100% CPU.Erik Johnston2021-04-081-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | We pull all destinations requiring catchup from the DB in batches. However, if all those destinations get filtered out (due to the federation sender being sharded), then the `last_processed` destination doesn't get updated, and we keep requesting the same set repeatedly.
| * | | Put opencontainers labels to the final image (#9765)Johannes Wienke2021-04-081-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | They don't make any sense on the intermediate builder image. The final images needs them to be of use for anyone. Signed-off-by: Johannes Wienke <languitar@semipol.de>
| * | | MSC3083: Check for space membership during a local join of restricted rooms. ↵Patrick Cloke2021-04-081-0/+1
| |/ / | | | | | | | | | | | | | | | | | | | | | (#9735) When joining a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
| * | Record more information into structured logs. (#9654)Patrick Cloke2021-04-081-0/+1
| | | | | | | | | | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
| * | Don't report anything from GaugeBucketCollector metrics until data is ↵Andrew Morgan2021-04-061-0/+1
| | | | | | | | | | | | | | | present (#8926) This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.
| * | Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan2021-04-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
| * | Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-061-0/+1
| | |
| * | Fix reported bugbear: too broad exception assertion (#9753)Andrew Morgan2021-04-061-0/+1
| | |
| * | Remove outdated constraint on remote_media_cache_thumbnails (#9725)Richard van der Hoff2021-04-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key` constraint is superceded by `remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds `thumbnail_method` to the unique key). PR #7124 made an attempt to remove the old constraint, but got the name wrong, so it didn't work. Here we update the bg update and rerun it. Fixes #8649.
| * | Merge branch 'master' into developErik Johnston2021-04-062-2/+0
| |\ \
| | * | 1.31.0 v1.31.0 github/release-v1.31.0 release-v1.31.0Erik Johnston2021-04-063-3/+0
| | | |
| | * | Add deprecation policy doc (#9723)Erik Johnston2021-04-061-0/+1
| | | | | | | | | | | | | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
| | * | Fix version for bugbear (#9734)Jonathan de Jong2021-04-061-0/+1
| | | |
| * | | Add type hints to the federation handler and server. (#9743)Patrick Cloke2021-04-061-0/+1
| | | |
| * | | Convert storage test cases to HomeserverTestCase. (#9736)Patrick Cloke2021-04-061-0/+1
| | | |
* | | | Drop support for sqlite<3.22 as wellRichard van der Hoff2021-04-081-1/+1
| | | |
* | | | Require py36 and Postgres 9.6Richard van der Hoff2021-04-081-0/+1
| |_|/ |/| |
* | | Fix incompatibility with tox 2.5Richard van der Hoff2021-04-081-0/+1
|/ / | | | | | | | | Apparently on tox 2.5, `usedevelop` overrides `skip_install`, so we end up trying to install the full dependencies even for the `-old` environment.
* | Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-0/+1
| |
* | Fix version for bugbear (#9734)Jonathan de Jong2021-04-021-0/+1
| |
* | Improve tracing for to device messages (#9686)Erik Johnston2021-04-011-0/+1
| |
* | Add `order_by` to list user admin API (#9691)Dirk Klimpel2021-04-011-0/+1
| |
* | Add an experimental room version to support restricted join rules. (#9717)Patrick Cloke2021-03-311-0/+1
| | | | | | Per MSC3083.
* | Merge branch 'release-v1.31.0' into developPatrick Cloke2021-03-311-0/+1
|\|
| * Revert "Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)" (#9720)Patrick Cloke2021-03-311-0/+1
| |
* | Make sample config allowed_local_3pids regex stricter. (#9719)Denis Kasak2021-03-311-0/+1
| | | | | | | | | | | | | | The regex should be terminated so that subdomain matches of another domain are not accepted. Just ensuring that someone doesn't shoot themselves in the foot by copying our example. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* | Deprecate imp (#9718)Cristina2021-03-311-0/+1
| | | | | | | | | | Fixes #9642. Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
* | Rewrite complement.sh (#9685)Andrew Morgan2021-03-311-0/+1
| | | | | | | | | | | | | | This PR rewrites the original complement.sh script with a number of improvements: * We can now use a local checkout of Complement (configurable with `COMPLEMENT_DIR`), though the default behaviour still downloads the master branch. * You can now specify a regex of test names to run, or just run all tests. * We now use the Synapse test blacklist tag (so all tests will pass).
* | Include m.room.create in invite_room_state for Spaces (#9710)Richard van der Hoff2021-03-301-0/+1
| |
* | Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff2021-03-301-0/+1
| | | | | | | | | | | | | | `room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
* | Make RateLimiter class check for ratelimit overrides (#9711)Erik Johnston2021-03-301-0/+1
|/ | | | | | | This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited. We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits. Fixes #9663
* 1.31.0rc1Erik Johnston2021-03-3043-43/+0
|
* Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)Andrew Morgan2021-03-301-0/+1
| | | For it's obvious performance benefits. `dmypy` support landed in #9692.
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-0/+1
|
* Clarify that register_new_matrix_user is present also when installed via ↵blakehawkins2021-03-291-0/+1
| | | | | non-pip package (#9074) Signed-off-by: blakehawkins blake.hawkins.11@gmail.com
* Add type hints for the federation sender. (#9681)Patrick Cloke2021-03-291-0/+1
| | | | Includes an abstract base class which both the FederationSender and the FederationRemoteSendQueue must implement.
* Update the OIDC sample config (#9695)Richard van der Hoff2021-03-291-0/+1
| | | | | | | I've reiterated the advice about using `oidc` to migrate, since I've seen a few people caught by this. I've also removed a couple of the examples as they are duplicating the OIDC documentation, and I think they might be leading people astray.
* Fix CI by ignore type for None module import (#9709)Andrew Morgan2021-03-291-0/+1
|
* Fix `re.Pattern` mypy error on 3.6 (#9703)Jonathan de Jong2021-03-291-0/+1
|
* Fix the suggested pip incantation for cryptography (#9699)Richard van der Hoff2021-03-291-0/+1
| | | | | | | | | | | If you have the wrong version of `cryptography` installed, synapse suggests: ``` To install run: pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6'' ``` However, the use of ' inside '...' doesn't work, so when you run this, you get an error.
* Make pip install faster in Docker build for Complement testing (#9610)Eric Eastwood2021-03-261-0/+1
| | | | | | | | | Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing. If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘 As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`. Fix https://github.com/matrix-org/synapse/issues/9364
* Suppress CryptographyDeprecationWarning (#9698)Richard van der Hoff2021-03-261-0/+1
| | | This warning is somewhat confusing to users, so let's suppress it
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-0/+1
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Use interpreter from $PATH instead of absolute paths in various scripts ↵Quentin Gliech2021-03-251-0/+1
| | | | | | | | | using /usr/bin/env (#9689) On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work. `/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* platform specific prerequisites in source install (#9667)Serban Constantin2021-03-251-0/+1
| | | | | | Make it clearer in the source install step that the platform specific prerequisites must be installed first. Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
* Add a storage method for returning all current presence from all users (#9650)Andrew Morgan2021-03-251-0/+1
| | | | | | | Split off from https://github.com/matrix-org/synapse/pull/9491 Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133 Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
* Fixed undefined variable error in catchup (#9664)Erik Johnston2021-03-241-0/+1
| | | | | Broke in #9640 Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix typo in changelog.Patrick Cloke2021-03-242-2/+2
|
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-241-0/+1
|
* Spaces summary: call out to other servers (#9653)Richard van der Hoff2021-03-241-0/+1
| | | | | When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data. Fixes: #9447
* docs: fallback/web endpoint does not appear to be mounted on workers (#9679)Ben Banfield-Zanin2021-03-241-0/+1
|
* Bump mypy-zope to 0.2.13. (#9678)Patrick Cloke2021-03-241-0/+1
| | | | This fixes an error ("Cannot determine consistent method resolution order (MRO)") when running mypy with a cache.
* Add type hints to misc. files. (#9676)Patrick Cloke2021-03-241-0/+1
|
* Add a type hints for service notices to the HomeServer object. (#9675)Patrick Cloke2021-03-241-0/+1
|
* Increase default join burst ratelimiting (#9674)Erik Johnston2021-03-231-0/+1
| | | It's legitimate behaviour to try and join a bunch of rooms at once.
* Fix federation stall on concurrent access errors (#9639)Jonathan de Jong2021-03-231-0/+1
|
* Federation API for Space summary (#9652)Richard van der Hoff2021-03-231-0/+1
| | | | | Builds on the work done in #9643 to add a federation API for space summaries. There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-231-0/+1
|
* Allow providing credentials to HTTPS_PROXY (#9657)Andrew Morgan2021-03-221-0/+1
| | | | | | | Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.
* Include opencontainers labels in Docker image (#9612)Johannes Wienke2021-03-221-0/+1
| | | | | | Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys Signed-off-by: Johannes Wienke <languitar@semipol.de>
* Fixed code misc. quality issues (#9649)Ankit Dobhal2021-03-221-0/+1
| | | | | - Merge 'isinstance' calls. - Remove unnecessary dict call outside of comprehension. - Use 'sys.exit()' calls.
* Merge branch 'develop' into babolivier/msc3026Brendan Abolivier2021-03-195-0/+5
|\
| * federation_client: handle inline signing_keys in hs.yaml (#9647)Richard van der Hoff2021-03-181-0/+1
| |
| * federation_client: stop adding URL prefix (#9645)Richard van der Hoff2021-03-181-0/+1
| |
| * Initial spaces summary API (#9643)Richard van der Hoff2021-03-181-0/+1
| | | | | | This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
| * Consistently check whether a password may be set for a user. (#9636)Dirk Klimpel2021-03-181-0/+1
| |
| * Make federation catchup send last event from any server. (#9640)Erik Johnston2021-03-181-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently federation catchup will send the last *local* event that we failed to send to the remote. This can cause issues for large rooms where lots of servers have sent events while the remote server was down, as when it comes back up again it'll be flooded with events from various points in the DAG. Instead, let's make it so that all the servers send the most recent events, even if its not theirs. The remote should deduplicate the events, so there shouldn't be much overhead in doing this. Alternatively, the servers could only send local events if they were also extremities and hope that the other server will send the event over, but that is a bit risky.
* | Implement MSC3026: busy presence stateBrendan Abolivier2021-03-181-0/+1
|/
* Ensure we use a copy of the event content dict before modifying it in ↵Andrew Morgan2021-03-171-0/+1
| | | | | | | | | serialize_event (#9585) This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](https://github.com/matrix-org/synapse/blob/17cd48fe5171d50da4cb59db647b993168e7dfab/synapse/events/third_party_rules.py#L73-L76) - will raise a 500 if you try to a edit a reply to a message. `serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised. To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
* Fix up types for the typing handler. (#9638)Patrick Cloke2021-03-171-0/+1
| | | | By splitting this to two separate methods the callers know what methods they can expect on the handler.
* only save remote cross-signing keys if they're different from the current ↵Hubert Chathi2021-03-171-0/+1
| | | | | ones (#9634) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix bad naming of storage function (#9637)Erik Johnston2021-03-171-0/+1
| | | | | | We had two functions named `get_forward_extremities_for_room` and `get_forward_extremeties_for_room` that took different paramters. We rename one of them to avoid confusion.
* Prep work for removing `outlier` from `internal_metadata` (#9411)Richard van der Hoff2021-03-171-0/+1
| | | | | | | | | | | | * Populate `internal_metadata.outlier` based on `events` table Rather than relying on `outlier` being in the `internal_metadata` column, populate it based on the `events.outlier` column. * Move `outlier` out of InternalMetadata._dict Ultimately, this will allow us to stop writing it to the database. For now, we have to grandfather it back in so as to maintain compatibility with older versions of Synapse.
* Add type hints to the room member handler. (#9631)Patrick Cloke2021-03-171-0/+1
|
* Enable flake8-bugbear, but disable most checks. (#9499)Jonathan de Jong2021-03-161-0/+1
| | | | * Adds B00 to ignored checks. * Fixes remaining issues.
* Add SSO attribute requirements for OIDC providers (#9609)Hubbe2021-03-161-0/+1
| | | | Allows limiting who can login using OIDC via the claims made from the IdP.
* Return m.change_password.enabled=false if local database is disabled (#9588)Dirk Klimpel2021-03-161-0/+1
| | | | | Instead of if the user does not have a password hash. This allows a SSO user to add a password to their account, but only if the local password database is configured.
* 1.30.0rc1Andrew Morgan2021-03-1644-44/+0
|
* Pass SSO IdP information to spam checker's registration function (#9626)Andrew Morgan2021-03-161-0/+1
| | | | | | | Fixes https://github.com/matrix-org/synapse/issues/9572 When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time. This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
* Install jemalloc in docker image (#8553)Mathieu Velten2021-03-161-0/+1
| | | | Co-authored-by: Will Hunt <willh@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Handle an empty cookie as an invalid macaroon. (#9620)Patrick Cloke2021-03-161-0/+1
| | | | | * Handle an empty cookie as an invalid macaroon. * Newsfragment
* Add support for stable MSC2858 API (#9617)Richard van der Hoff2021-03-161-0/+1
| | | | | The stable format uses different brand identifiers, so we need to support two identifiers for each IdP.
* Clean up config settings for stats (#9604)Richard van der Hoff2021-03-162-0/+2
| | | ... and complain if people try to turn it off.
* Prevent bundling aggregations for state events (#9619)Andrew Morgan2021-03-161-0/+1
| | | There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
* Fix Internal Server Error on `GET /saml2/authn_response` (#9623)Richard van der Hoff2021-03-161-0/+1
| | | | | | | * Fix Internal Server Error on `GET /saml2/authn_response` Seems to have been introduced in #8765 (Synapse 1.24.0) * Fix newsfile
* Revert requiring a specific version of Twisted for mypy checks. (#9618)Patrick Cloke2021-03-151-0/+1
|
* Fix remaining mypy issues due to Twisted upgrade. (#9608)Patrick Cloke2021-03-151-0/+1
|
* Don't go into federation catch up mode so easily (#9561)Erik Johnston2021-03-151-0/+1
| | | | | | | | | | Federation catch up mode is very inefficient if the number of events that the remote server has missed is small, since handling gaps can be very expensive, c.f. #9492. Instead of going into catch up mode whenever we see an error, we instead do so only if we've backed off from trying the remote for more than an hour (the assumption being that in such a case it is more than a transient failure).
* Optimise missing prev_event handling (#9601)Richard van der Hoff2021-03-151-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | Background: When we receive incoming federation traffic, and notice that we are missing prev_events from the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events, we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote server for the state at those prev_events, and then we may need to then ask the remote server for any events in that state which we don't already have, as well as the auth events for those missing state events, so that we can auth them. This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the observation that we are currently loading all of those auth events into memory at the start of the operation, but we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're actually going to need, but it doesn't.) The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about 60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache churn. (NB I've already tripled the size of the cache from its default of 10K). Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different (ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting that we clean it up.
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-0/+1
|
* Reject concurrent transactions (#9597)Richard van der Hoff2021-03-121-0/+1
| | | | | | If more transactions arrive from an origin while we're still processing the first one, reject them. Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
* Improve logging when processing incoming transactions (#9596)Richard van der Hoff2021-03-121-0/+1
| | | Put the room id in the logcontext, to make it easier to understand what's going on.
* Add logging for redis connection setup (#9590)Richard van der Hoff2021-03-111-0/+1
|
* Add tests for blacklisting reactor/agent. (#9563)Patrick Cloke2021-03-111-0/+1
|
* Re-Activating account when local passwords are disabled (#9587)Dirk Klimpel2021-03-111-0/+1
| | | Fixes: #8393
* Convert Requester to attrs (#9586)Richard van der Hoff2021-03-101-0/+1
| | | | | | ... because namedtuples suck Fix up a couple of other annotations to keep mypy happy.
* Fix the auth provider on the logins metric (#9573)Richard van der Hoff2021-03-101-0/+1
| | | | | We either need to pass the auth provider over the replication api, or make sure we report the auth provider on the worker that received the request. I've gone with the latter.
* Fix spam checker modules documentation example (#9580)Jason Robinson2021-03-101-0/+1
| | | | Mention that parse_config must exist and note the check_media_file_for_spam method.
* Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-0/+1
| | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* Fix a bug in the background task for purging chain cover. (#9583)Patrick Cloke2021-03-101-0/+1
|
* Do not ignore the unpaddedbase64 module when type checking. (#9568)Patrick Cloke2021-03-091-0/+1
|
* Add a background task to purge unused chain IDs. (#9542)Patrick Cloke2021-03-091-0/+1
| | | | | This is a companion change to apply the fix in #9498 / 922788c6043138165c025c78effeda87de842bab to previously purged rooms.
* Link to the List user's media admin API from media Admin API docs (#9571)Andrew Morgan2021-03-091-0/+1
| | | | | Earlier [I was convinced](https://github.com/matrix-org/synapse/issues/9565) that we didn't have an Admin API for listing media uploaded by a user. Foolishly I was looking under the Media Admin API documentation, instead of the User Admin API documentation. I thought it'd be helpful to link to the latter so others don't hit the same dead end :)
* JWT OIDC secrets for Sign in with Apple (#9549)Richard van der Hoff2021-03-091-0/+1
| | | | | Apple had to be special. They want a client secret which is generated from an EC key. Fixes #9220. Also fixes #9212 while I'm here.
* Retry 5xx errors in federation client (#9567)Erik Johnston2021-03-091-0/+1
| | | Fixes #8915
* Fix additional type hints. (#9543)Patrick Cloke2021-03-091-0/+1
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Handle image transparency better when thumbnailing. (#9473)Patrick Cloke2021-03-091-0/+1
| | | | Properly uses RGBA mode for 1- and 8-bit images with transparency (instead of RBG mode).
* Add a list of hashes to ignore during git blame. (#9560)Patrick Cloke2021-03-091-0/+1
| | | | | | | The hashes are from commits due to auto-formatting, e.g. running black. git can be configured to use this automatically by running the following: git config blame.ignoreRevsFile .git-blame-ignore-revs
* Prevent the config-lint script erroring out on any sample_config changes (#9562)Andrew Morgan2021-03-091-0/+1
| | | | | | | | | I noticed that I'd occasionally have `scripts-dev/lint.sh` fail when messing about with config options in my PR. The script calls `scripts-dev/config-lint.sh`, which attempts some validation on the sample config. It does this by using `sed` to edit the sample_config, and then seeing if the file changed using `git diff`. The problem is: if you changed the sample_config as part of your commit, this script will error regardless. This PR attempts to change the check so that existing, unstaged changes to the sample_config will not cause the script to report an invalid file.
* Add logging to ObservableDeferred callbacks (#9523)Jonathan de Jong2021-03-091-0/+1
|
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-0/+1
|
* Warn that /register will soon require a type when called with an access ↵Will Hunt2021-03-081-0/+1
| | | | | token (#9559) This notice is giving a heads up to the planned spec compliance fix https://github.com/matrix-org/synapse/pull/9548.
* Add type hints to purge room and server notice admin API. (#9520)Dirk Klimpel2021-03-081-0/+1
|
* Add a basic test for purging rooms. (#9541)Patrick Cloke2021-03-081-0/+1
| | | | | Unfortunately this doesn't test re-joining the room since that requires having another homeserver to query over federation, which isn't easily doable in unit tests.
* Create a SynapseReactor type which incorporates the necessary reactor ↵Patrick Cloke2021-03-081-0/+1
| | | | | interfaces. (#9528) This helps fix some type hints when running with Twisted 21.2.0.
* Update reverse proxy to add OpenBSD relayd example configuration. (#9508)Leo Bärring2021-03-061-0/+1
| | | | | Update reverse proxy to add OpenBSD relayd example configuration. Signed-off-by: Leo Bärring <leo.barring@protonmail.com>
* Add additional SAML2 upgrade notes (#9550)Ben Banfield-Zanin2021-03-051-0/+1
|
* Replace `last_*_pdu_age` metrics with timestamps (#9540)Richard van der Hoff2021-03-042-0/+2
| | | | | | | | Following the advice at https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since, it's preferable to export unix timestamps, not ages. There doesn't seem to be any particular naming convention for timestamp metrics.
* Prometheus metrics for logins and registrations (#9511)Richard van der Hoff2021-03-041-0/+1
| | | Add prom metrics for number of users successfully registering and logging in, by SSO provider.
* Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff2021-03-041-0/+1
| | | This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
* 1.29.0rc1Erik Johnston2021-03-0440-40/+0
|
* Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-031-0/+1
|
* Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539)Richard van der Hoff2021-03-031-0/+1
| | | Should fix some remaining warnings
* Fix 'rejected_events_metadata' background update (#9537)Erik Johnston2021-03-031-0/+1
| | | | Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
* Purge chain cover tables when purging events. (#9498)Patrick Cloke2021-03-031-0/+1
|
* Add type hints to user admin API. (#9521)Dirk Klimpel2021-03-031-0/+1
|
* Bump the mypy and mypy-zope versions. (#9529)Patrick Cloke2021-03-031-0/+1
|
* Make deleting stale pushers a background update (#9536)Erik Johnston2021-03-031-0/+1
|
* Update nginx reverse-proxy docs (#9512)Richard van der Hoff2021-03-031-0/+1
| | | Turns out nginx overwrites the Host header by default.
* Prevent presence background jobs from running when presence is disabled (#9530)Aaron Raimist2021-03-031-0/+1
| | | | | Prevent presence background jobs from running when presence is disabled Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"Patrick Cloke2021-03-021-1/+0
| | | | | | | This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46. This is being backed out due to a regression (#9507) and additional review feedback being provided.
* Re-run rejected metadata background update. (#9503)Erik Johnston2021-03-021-0/+1
| | | | | | | It landed in schema version 58 after 59 had been created, causing some servers to not run it. The main effect of was that not all rooms had their chain cover calculated correctly. After the BG updates complete the chain covers will get fixed when a new state event in the affected rooms is received.
* Fix SQL delta file taking a long time to run (#9516)Erik Johnston2021-03-021-0/+1
| | | | Fixes #9504
* Add type hints to device and event report admin API (#9519)Dirk Klimpel2021-03-021-0/+1
|
* Fix a bug when a room alias is given to the admin join endpoint (#9506)Patrick Cloke2021-03-011-0/+1
|
* (Hopefully) stop leaking file descriptors in media repo. (#9497)Patrick Cloke2021-03-011-0/+1
| | | | By consuming the response if the headers imply that the content is too large.
* Use the proper Request in type hints. (#9515)Patrick Cloke2021-03-011-0/+1
| | | | This also pins the Twisted version in the mypy job for CI until proper type hints are fixed throughout Synapse.
* Allow bytecode again (#9502)Jonathan de Jong2021-02-261-0/+1
| | | | | | | In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again. Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-261-0/+1
| | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
* SSO: redirect to public URL before setting cookies (#9436)Richard van der Hoff2021-02-261-0/+1
| | | ... otherwise, we don't get the cookie back.
* Call out the need for an X-Forwarded-Proto in the upgrade notes (#9501)Richard van der Hoff2021-02-262-1/+2
|
* Test that we require validated email for email pushers (#9496)Erik Johnston2021-02-251-0/+1
|
* Ensure pushers are deleted for deactivated accounts (#9285)Erik Johnston2021-02-251-0/+1
|
* Add support for X-Forwarded-Proto (#9472)Richard van der Hoff2021-02-241-0/+1
| | | | | rewrite XForwardedForRequest to set `isSecure()` based on `X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
* Add SQL delta for deleting stale pushers (#9479)Erik Johnston2021-02-241-0/+1
|
* Fix #8518 (sync requests being cached wrongly on timeout) (#9358)Jonathan de Jong2021-02-241-0/+1
| | | | | | | This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Clean up `ShardedWorkerHandlingConfig` (#9466)Erik Johnston2021-02-241-0/+1
| | | | | | | | | | | | | | | | | * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.
* Refactor to ensure we call check_consistency (#9470)Erik Johnston2021-02-241-0/+1
| | | The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
* Add a comment about systemd-python. (#9464)Richard van der Hoff2021-02-231-0/+1
| | | | This confused me for a while.
* Include newly added sequences in the port DB script. (#9449)Patrick Cloke2021-02-231-0/+1
| | | And ensure the consistency of `event_auth_chain_id`.
* Fix deleting pushers when using sharded pushers. (#9465)Erik Johnston2021-02-221-0/+1
|
* Remove vestiges of uploads_path config (#9462)Richard van der Hoff2021-02-221-0/+1
| | | | `uploads_path` was a thing that was never used; most of it was removed in #6628 but a few vestiges remained.
* Add an `order_by` field to list users' media admin API. (#8978)Dirk Klimpel2021-02-221-0/+1
|
* example systemd config: propagate reloads to units (#9463)Richard van der Hoff2021-02-221-0/+1
| | | | | It should be possible to reload `synapse.target` to have the reload propagate to all the synapse units.
* Remove cache for get_shared_rooms_for_users (#9416)Andrew Morgan2021-02-221-0/+1
| | | | | | | This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed. The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately. Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
* Clean up the user directory sample config section (#9385)Andrew Morgan2021-02-221-0/+1
| | | | | The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format). This PR cleans that up a bit.
* Ratelimit cross-user key sharing requests. (#8957)Patrick Cloke2021-02-191-0/+1
|
* Regenerate exact thumbnails if missing (#9438)Erik Johnston2021-02-191-0/+1
|\
| * NewsfileErik Johnston2021-02-181-0/+1
| |
* | Add documentation and type hints to parse_duration. (#9432)Patrick Cloke2021-02-191-0/+1
| |
* | Merge tag 'v1.28.0rc1' into developPatrick Cloke2021-02-1943-43/+0
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.28.0rc1 (2021-02-19) ============================== Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails). This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. Removal warning --------------- The v1 list accounts API is deprecated and will be removed in a future release. This API was undocumented and misleading. It can be replaced by the [v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts), which has been available since Synapse 1.7.0 (2019-12-13). Please check if you're using any scripts which use the admin API and replace `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`. Features -------- - New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150)) - Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301)) - Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311)) - Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376)) - Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400)) Bugfixes -------- - Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257)) - Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296)) - Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393)) - Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333)) - Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361)) - Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391)) - Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395)) - Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421)) - Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425)) - The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426)) - Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428)) Updates to the Docker image --------------------------- - Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433)) Improved Documentation ---------------------- - Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281)) - Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291)) - Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308)) - Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317)) - Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322)) - Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390)) - Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404)) - Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407)) - Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434)) Deprecations and Removals ------------------------- - Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429)) Internal Changes ---------------- - Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003)) - Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123)) - Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240)) - Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299)) - Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305)) - Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307)) - Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326)) - Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362)) - Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396)) - Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381)) - Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384)) - Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394)) - Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
| * | 1.28.0rc1Patrick Cloke2021-02-1843-43/+0
| | |
* | | Support not providing an IdP icon when choosing a username. (#9440)Patrick Cloke2021-02-191-0/+1
| | |
* | | Be smarter about which hosts to send presence to when processing room joins ↵Andrew Morgan2021-02-191-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#9402) This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually. --- When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed. It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw. This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence: * If it was a local user join, send that user's latest presence to all servers in the room * If it was a remote user join, send the presence for all local users in the room to that homeserver We deduplicate by inserting all of those pending updates into a dictionary of the form: ``` { server_name1: {presence_update1, ...}, server_name2: {presence_update1, presence_update2, ...} } ``` Only after building this dict do we then start sending out presence updates.
* | | Add a config option to prioritise local users in user directory search ↵Andrew Morgan2021-02-191-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | results (#9383) This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default. Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical. This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
* | | Add configs to make profile data more private (#9203)AndrewFerr2021-02-191-0/+1
|/ / | | | | | | | | | | | | Add off-by-default configuration settings to: - disable putting an invitee's profile info in invite events - disable profile lookup via federation Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* / Remove deprecated SAML2 callback URL since it does not work. (#9434)Patrick Cloke2021-02-181-0/+1
|/ | | | Updates documentation from #9289 and removes a deprecated endpoint which didn't work as expected.
* Revert "Newsfragment", which was meant to be part of #9434.Patrick Cloke2021-02-181-1/+0
| | | | This reverts commit 8ad4676f35ec065853a5eb335b5566fdcf320460.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-04 18:34:08 +0000