summary refs log tree commit diff
path: root/tests (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke2021-07-131-15/+176
| | | | | | | | | | Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
* Use inline type hints in `tests/` (#10350)Jonathan de Jong2021-07-1317-63/+61
| | | | | | | | This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)
* [pyupgrade] `tests/` (#10347)Jonathan de Jong2021-07-1319-58/+44
|
* Additional unit tests for spaces summary. (#10305)Patrick Cloke2021-07-121-1/+203
|
* Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-082-194/+10
| | | Fixes #9602
* Ignore EDUs for rooms we're not in (#10317)Dagfinn Ilmari Mannsåker2021-07-061-0/+37
|
* Fix deactivate a user if he does not have a profile (#10252)Dirk Klimpel2021-07-061-18/+68
|
* Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-051-1/+45
|
* Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-011-2/+2
| | | Instead of mixing them with user authentication methods.
* Add SSO `external_ids` to Query User Account admin API (#10261)Dirk Klimpel2021-07-011-84/+140
| | | Related to #10251
* Merge branch 'release-v1.37' into developRichard van der Hoff2021-06-291-0/+100
|\
| * Add a distributed lock (#10269)Erik Johnston2021-06-291-0/+100
| | | | | | This adds a simple best effort locking mechanism that works cross workers.
* | Do not recurse into non-spaces in the spaces summary. (#10256)Patrick Cloke2021-06-292-24/+27
| | | | | | | | | | Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.
* | Return errors from `send_join` etc if the event is rejected (#10243)Richard van der Hoff2021-06-241-3/+1
| | | | | | Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
* | Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-242-2/+2
| | | | | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
* | MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-243-2/+221
| | | | | | | | | | | | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* | Merge tag 'v1.37.0rc1' into developBrendan Abolivier2021-06-241-0/+76
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))
| * Fix wrapping of legacy check_registration_for_spam (#10238)Brendan Abolivier2021-06-231-0/+76
| | | | | | Fixes #10234
* | Re-introduce "Leave out optional keys from /sync" change (#10214)Andrew Morgan2021-06-232-32/+6
| | | | | | Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.
* | Improve the reliability of auto-joining remote rooms (#10237)Brendan Abolivier2021-06-231-1/+48
|/ | | | | | If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do. Fixes #10233 (at least to some extent)
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-223-4/+17
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Fix a missing await when in the spaces summary. (#10208)Patrick Cloke2021-06-181-1/+98
| | | | | | | This could cause a minor data leak if someone defined a non-restricted join rule with an allow key or used a restricted join rule in an older room version, but this is unlikely. Additionally this starts adding unit tests to the spaces summary handler.
* Standardise the module interface (#10062)Brendan Abolivier2021-06-183-48/+96
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Remove support for ACME v1 (#10194)Brendan Abolivier2021-06-171-97/+0
| | | | | Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
* Ensure that we do not cache empty sync responses after a timeout (#10158)Richard van der Hoff2021-06-172-5/+53
| | | Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
* update black to 21.6b0 (#10197)Marcus2021-06-1710-17/+17
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* Remove the experimental flag for knocking and use stable prefixes / ↵Patrick Cloke2021-06-152-17/+13
| | | | | | | endpoints. (#10167) * Room version 7 for knocking. * Stable prefixes and endpoints (both client and federation) for knocking. * Removes the experimental configuration flag.
* Extend `ResponseCache` to pass a context object into the callback (#10157)Richard van der Hoff2021-06-141-7/+68
| | | | | This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
* Implement knock feature (#6739)Sorunome2021-06-092-2/+395
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Limit the number of in-flight /keys/query requests from a single device. ↵Patrick Cloke2021-06-091-3/+10
| | | | (#10144)
* Always update AS last_pos, even on no events (#10107)14mRh4X0r2021-06-071-4/+2
| | | | | | | | | | Fixes #1834. `get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether. Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list. Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
* r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224)Travis Ralston2021-06-031-2/+1
| | | | | | [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. This is progress towards https://github.com/matrix-org/synapse/issues/8334
* Add new admin APIs to remove media by media ID from quarantine. (#10044)Dirk Klimpel2021-06-021-0/+128
| | | | | Related to: #6681, #5956, #10040 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Rewrite the KeyRing (#10035)Erik Johnston2021-06-023-114/+111
|
* add a cache to have_seen_event (#9953)Richard van der Hoff2021-06-013-0/+122
| | | Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
* Make reason and score optional for report_event (#10077)Callum Brown2021-05-272-1/+97
| | | | | | Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414 See #8551 Signed-off-by: Callum Brown <callum@calcuode.com>
* Fix up `BatchingQueue` (#10078)Erik Johnston2021-05-271-2/+76
| | | Fixes #10068
* Remove redundant code to reload tls cert (#10054)Richard van der Hoff2021-05-271-2/+1
| | | | | we don't need to reload the tls cert if we don't have any tls listeners. Follow-up to #9280.
* Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-271-3/+3
| | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* Add an admin API for unprotecting local media from quarantine (#10040)Dirk Klimpel2021-05-261-0/+99
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Add missing type hints to synapse.util (#9982)Patrick Cloke2021-05-241-2/+2
|
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-242-4/+6
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-213-16/+18
|
* Add a test for update_presence (#10033)Andrew Morgan2021-05-211-1/+46
| | | | | | | https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale. The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug. The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.
* Add a batching queue implementation. (#10017)Erik Johnston2021-05-211-0/+169
|
* Use a database table to hold the users that should have full presence sent ↵Andrew Morgan2021-05-183-92/+228
| | | | to them, instead of something in-memory (#9823)
* Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-141-3/+14
| | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* Correctly ratelimit invites when creating a room (#9968)Brendan Abolivier2021-05-122-0/+94
| | | | | * Correctly ratelimit invites when creating a room Also allow ratelimiting for more than one action at a time.
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-124-92/+34
|
* Tests for to-device messages (#9965)Richard van der Hoff2021-05-111-0/+201
|
* Sort child events according to MSC1772 for the spaces summary API. (#9954)Patrick Cloke2021-05-111-0/+81
| | | | | | | | | This should help ensure that equivalent results are achieved between homeservers querying for the summary of a space. This implements modified MSC1772 rules, according to MSC2946. The different is that the origin_server_ts of the m.room.create event is not used as a tie-breaker since this might not be known if the homeserver is not part of the room.
* Merge branch 'master' into developBrendan Abolivier2021-05-113-0/+244
|\
| * Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff2021-05-113-0/+244
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* | Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-071-3/+1
| | | | | | | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* | Revert "Leave out optional keys from /sync (#9919)" (#9940)Erik Johnston2021-05-062-6/+32
| | | | | | This reverts commit e9eb3549d32a6f93d07de8dbd5e1ebe54c8d8278.
* | Increase perf of handling presence when joining large rooms. (#9916)Erik Johnston2021-05-051-10/+4
| |
* | Leave out optional keys from /sync (#9919)DeepBlueV7.X2021-05-052-32/+6
| | | | | | | | | | | | This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.) Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
* | Delete room endpoint (#9889)ThibF2021-04-291-19/+26
|/ | | | | | Support the delete of a room through DELETE request and mark previous request as deprecated through documentation. Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
* Fix tight loop handling presence replication. (#9900)Erik Johnston2021-04-281-0/+22
| | | | | Only affects workers. Introduced in #9819. Fixes #9899.
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-273-11/+9
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Improved validation for received requests (#9817)Richard van der Hoff2021-04-234-0/+86
| | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* Kill off `_PushHTTPChannel`. (#9878)Richard van der Hoff2021-04-232-121/+19
| | | | | First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation. Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
* pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-233-0/+3
|
* Remove room and user invite ratelimits in default unit test config (#9871)Andrew Morgan2021-04-231-0/+4
|
* Split presence out of master (#9820)Erik Johnston2021-04-232-85/+3
|
* Limit the size of HTTP responses read over federation. (#9833)Richard van der Hoff2021-04-231-0/+59
|
* Limit length of accepted email addresses (#9855)Erik Johnston2021-04-221-0/+51
|
* Rename handler and config modules which end in handler/config. (#9816)Patrick Cloke2021-04-202-5/+5
|
* Merge branch 'master' into developAndrew Morgan2021-04-203-7/+7
|\
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-203-7/+7
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Add presence federation stream (#9819)Erik Johnston2021-04-201-8/+171
| |
* | Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-2011-23/+23
| |
* | Port "Allow users to click account renewal links multiple times without ↵Andrew Morgan2021-04-191-15/+37
| | | | | | | | | | hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
* | remove `HomeServer.get_config` (#9815)Richard van der Hoff2021-04-141-1/+1
| | | | | | | | Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.
* | Separate creating an event context from persisting it in the federation ↵Patrick Cloke2021-04-141-2/+4
| | | | | | | | | | | | handler (#9800) This refactoring allows adding logic that uses the event context before persisting it.
* | Revert "Check for space membership during a remote join of a restricted ↵Patrick Cloke2021-04-141-4/+2
| | | | | | | | | | | | | | | | room. (#9763)" This reverts commit cc51aaaa7adb0ec2235e027b5184ebda9b660ec4. The PR was prematurely merged and not yet approved.
* | Check for space membership during a remote join of a restricted room. (#9763)Patrick Cloke2021-04-141-2/+4
| | | | | | | | | | | | | | When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
* | Move some replication processing out of generic_worker (#9796)Erik Johnston2021-04-141-5/+3
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-14200-200/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-134-6/+6
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Add an admin API to manage ratelimit for a specific user (#9648)Dirk Klimpel2021-04-131-0/+284
|
* Require AppserviceRegistrationType (#9548)Will Hunt2021-04-122-16/+38
| | | This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-0978-121/+82
|
* Remove old admin API `GET /_synapse/admin/v1/users/<user_id>` (#9401)Dirk Klimpel2021-04-091-2/+2
| | | | | | | | | | | | | Related: #8334 Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25) `GET /_synapse/admin/v1/users/<user_id>` has no - unit tests - documentation API in v2 is available (#5925 - 12/2019, v1.7.0). API is misleading. It expects `user_id` and returns a list of all users. Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-0811-33/+68
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Merge remote-tracking branch 'origin/develop' into rav/drop_py35Richard van der Hoff2021-04-0819-559/+928
|\
| * Record more information into structured logs. (#9654)Patrick Cloke2021-04-085-45/+92
| | | | | | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
| * Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan2021-04-063-13/+569
| | | | | | | | | | | | | | | | | | | | | | | | At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
| * Fix reported bugbear: too broad exception assertion (#9753)Andrew Morgan2021-04-061-2/+3
| |
| * Convert storage test cases to HomeserverTestCase. (#9736)Patrick Cloke2021-04-0610-499/+264
| |
* | remove unused param on `make_tuple_comparison_clause`Richard van der Hoff2021-04-081-2/+1
| |
* | Drop support for sqlite<3.22 as wellRichard van der Hoff2021-04-081-11/+1
|/
* Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-2/+5
|
* Add `order_by` to list user admin API (#9691)Dirk Klimpel2021-04-011-1/+120
|
* Add an experimental room version to support restricted join rules. (#9717)Patrick Cloke2021-03-311-3/+243
| | | Per MSC3083.
* Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff2021-03-301-1/+0
| | | | | | | `room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
* Make RateLimiter class check for ratelimit overrides (#9711)Erik Johnston2021-03-301-60/+108
| | | | | | | This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited. We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits. Fixes #9663
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-292-15/+11
|
* Fix CI by ignore type for None module import (#9709)Andrew Morgan2021-03-291-1/+1
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-263-10/+23
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-241-1/+1
|
* Allow providing credentials to HTTPS_PROXY (#9657)Andrew Morgan2021-03-221-0/+40
| | | | | | | Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.
* Merge branch 'develop' into babolivier/msc3026Brendan Abolivier2021-03-192-55/+167
|\
| * Consistently check whether a password may be set for a user. (#9636)Dirk Klimpel2021-03-181-55/+118
| |
| * Make federation catchup send last event from any server. (#9640)Erik Johnston2021-03-181-0/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently federation catchup will send the last *local* event that we failed to send to the remote. This can cause issues for large rooms where lots of servers have sent events while the remote server was down, as when it comes back up again it'll be flooded with events from various points in the DAG. Instead, let's make it so that all the servers send the most recent events, even if its not theirs. The remote should deduplicate the events, so there shouldn't be much overhead in doing this. Alternatively, the servers could only send local events if they were also extremities and hope that the other server will send the event over, but that is a bit risky.
* | Implement MSC3026: busy presence stateBrendan Abolivier2021-03-181-0/+20
|/
* Ensure we use a copy of the event content dict before modifying it in ↵Andrew Morgan2021-03-173-0/+134
| | | | | | | | | serialize_event (#9585) This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](https://github.com/matrix-org/synapse/blob/17cd48fe5171d50da4cb59db647b993168e7dfab/synapse/events/third_party_rules.py#L73-L76) - will raise a 500 if you try to a edit a reply to a message. `serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised. To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
* Enable flake8-bugbear, but disable most checks. (#9499)Jonathan de Jong2021-03-161-1/+1
| | | | * Adds B00 to ignored checks. * Fixes remaining issues.
* Add SSO attribute requirements for OIDC providers (#9609)Hubbe2021-03-161-0/+132
| | | | Allows limiting who can login using OIDC via the claims made from the IdP.
* Return m.change_password.enabled=false if local database is disabled (#9588)Dirk Klimpel2021-03-161-4/+32
| | | | | Instead of if the user does not have a password hash. This allows a SSO user to add a password to their account, but only if the local password database is configured.
* Pass SSO IdP information to spam checker's registration function (#9626)Andrew Morgan2021-03-161-0/+31
| | | | | | | Fixes https://github.com/matrix-org/synapse/issues/9572 When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time. This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
* Add support for stable MSC2858 API (#9617)Richard van der Hoff2021-03-161-16/+27
| | | | | The stable format uses different brand identifiers, so we need to support two identifiers for each IdP.
* Fix remaining mypy issues due to Twisted upgrade. (#9608)Patrick Cloke2021-03-152-28/+18
|
* Don't go into federation catch up mode so easily (#9561)Erik Johnston2021-03-151-1/+2
| | | | | | | | | | Federation catch up mode is very inefficient if the number of events that the remote server has missed is small, since handling gaps can be very expensive, c.f. #9492. Instead of going into catch up mode whenever we see an error, we instead do so only if we've backed off from trying the remote for more than an hour (the assumption being that in such a case it is more than a transient failure).
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-3/+5
|
* Add tests for blacklisting reactor/agent. (#9563)Patrick Cloke2021-03-111-2/+124
|
* Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-3/+73
| | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* JWT OIDC secrets for Sign in with Apple (#9549)Richard van der Hoff2021-03-093-24/+166
| | | | | Apple had to be special. They want a client secret which is generated from an EC key. Fixes #9220. Also fixes #9212 while I'm here.
* Fix additional type hints. (#9543)Patrick Cloke2021-03-093-12/+19
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Handle image transparency better when thumbnailing. (#9473)Patrick Cloke2021-03-091-8/+21
| | | | Properly uses RGBA mode for 1- and 8-bit images with transparency (instead of RBG mode).
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-0/+131
|
* Add a basic test for purging rooms. (#9541)Patrick Cloke2021-03-081-26/+45
| | | | | Unfortunately this doesn't test re-joining the room since that requires having another homeserver to query over federation, which isn't easily doable in unit tests.
* Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff2021-03-044-50/+55
| | | This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
* Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-031-11/+24
|
* (Hopefully) stop leaking file descriptors in media repo. (#9497)Patrick Cloke2021-03-011-36/+55
| | | | By consuming the response if the headers imply that the content is too large.
* Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-261-44/+73
| | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
* SSO: redirect to public URL before setting cookies (#9436)Richard van der Hoff2021-02-264-27/+65
| | | ... otherwise, we don't get the cookie back.
* Test that we require validated email for email pushers (#9496)Erik Johnston2021-02-251-2/+32
|
* Clean up `ShardedWorkerHandlingConfig` (#9466)Erik Johnston2021-02-244-4/+4
| | | | | | | | | | | | | | | | | * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.
* Remove vestiges of uploads_path config (#9462)Richard van der Hoff2021-02-221-1/+0
| | | | `uploads_path` was a thing that was never used; most of it was removed in #6628 but a few vestiges remained.
* Add an `order_by` field to list users' media admin API. (#8978)Dirk Klimpel2021-02-221-21/+225
|
* Remove cache for get_shared_rooms_for_users (#9416)Andrew Morgan2021-02-221-34/+41
| | | | | | | This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed. The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately. Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
* Regenerate exact thumbnails if missing (#9438)Erik Johnston2021-02-191-3/+66
|\
| * Add testErik Johnston2021-02-191-3/+66
| |
* | Fix style checking due to updated black.Patrick Cloke2021-02-191-1/+4
| |
* | Be smarter about which hosts to send presence to when processing room joins ↵Andrew Morgan2021-02-191-4/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#9402) This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually. --- When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed. It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw. This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence: * If it was a local user join, send that user's latest presence to all servers in the room * If it was a remote user join, send the presence for all local users in the room to that homeserver We deduplicate by inserting all of those pending updates into a dictionary of the form: ``` { server_name1: {presence_update1, ...}, server_name2: {presence_update1, presence_update2, ...} } ``` Only after building this dict do we then start sending out presence updates.
* | Add a config option to prioritise local users in user directory search ↵Andrew Morgan2021-02-191-0/+94
| | | | | | | | | | | | | | | | | | results (#9383) This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default. Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical. This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
* | Add configs to make profile data more private (#9203)AndrewFerr2021-02-191-1/+5
|/ | | | | | | Add off-by-default configuration settings to: - disable putting an invitee's profile info in invite events - disable profile lookup via federation Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Parse ui_auth.session_timeout as a duration (instead of treating it as ms) ↵Rishabh Arya2021-02-181-1/+1
| | | | (#9426)
* Reduce the memory usage of previewing media files. (#9421)Patrick Cloke2021-02-181-4/+5
| | | | | | | | This reduces the memory usage of previewing media files which end up larger than the `max_spider_size` by avoiding buffering content internally in treq. It also checks the `Content-Length` header in additional places instead of streaming the content to check the body length.
* Add the shadow-banning status to the display user admin API. (#9400)Dirk Klimpel2021-02-172-0/+3
|
* Support for form_post in OIDC responses (#9376)Richard van der Hoff2021-02-171-13/+13
| | | Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1681-635/+1407
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-162-26/+206
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Handle missing data in power levels events during room upgrade. (#9395)Patrick Cloke2021-02-161-0/+153
|
* Convert additional test-cases to homeserver test case. (#9396)Patrick Cloke2021-02-164-331/+213
| | | And convert some inlineDeferreds to async-friendly functions.
* Merge tag 'v1.27.0rc2' into developPatrick Cloke2021-02-113-1/+13
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.27.0rc2 (2021-02-11) ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
| * Backout changes for automatically calculating the public baseurl. (#9313)Patrick Cloke2021-02-113-1/+13
| | | | | | | | This breaks some people's configurations (if their Client-Server API is not accessed via port 443).
* | Convert some test cases to use HomeserverTestCase. (#9377)Patrick Cloke2021-02-115-516/+301
| | | | | | | | This has the side-effect of being able to remove use of `inlineCallbacks` in the test-cases for cleaner tracebacks.
* | Combine the CAS & SAML implementations for required attributes. (#9326)Patrick Cloke2021-02-112-3/+105
| |
* | Handle additional errors when previewing URLs. (#9333)Patrick Cloke2021-02-081-11/+92
| | | | | | | | * Handle the case of lxml not finding a document tree. * Parse the document encoding from the XML tag.
* | Merge pull request #9150 from Yoric/develop-contextDavid Teller2021-02-081-0/+84
|\ \ | | | | | | New API /_synapse/admin/rooms/{roomId}/context/{eventId}
| * | FIXUP: linterDavid Teller2021-01-281-1/+3
| | |
| * | FIXUP: Now testing that the user is admin!David Teller2021-01-281-1/+35
| | |
| * | New API /_synapse/admin/rooms/{roomId}/context/{eventId}David Teller2021-01-281-0/+48
| | | | | | | | | | | | Signed-off-by: David Teller <davidt@element.io>
* | | Merge branch 'erikj/media_spam_checker' into developErik Johnston2021-02-041-0/+94
|\ \ \
| * | | Add check_media_file_for_spam spam checker hookErik Johnston2021-02-041-0/+94
| | |/ | |/|
* | | Handle empty rooms when generating email notifications. (#9257)Patrick Cloke2021-02-041-2/+49
| | | | | | | | | | | | | | | | | | Fixes some exceptions if the room state isn't quite as expected. If the expected state events aren't found, try to find them in the historical room state. If they still aren't found, fallback to a reasonable, although ugly, value.
* | | Merge branch 'social_login_hotfixes' into developRichard van der Hoff2021-02-031-4/+12
|\ \ \
| * | | Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixesRichard van der Hoff2021-02-031-47/+0
| |\| |
| * | | Social login UI polish (#9301)Richard van der Hoff2021-02-031-4/+12
| | | |
* | | | Convert blacklisted IPv4 addresses to compatible IPv6 addresses. (#9240)Patrick Cloke2021-02-031-1/+60
| |/ / |/| | | | | Also add a few more IP ranges to the default blacklist.
* | | Honour ratelimit flag for application services for invite ratelimiting (#9302)Erik Johnston2021-02-031-47/+0
|/ /
* | Add an admin API to get the current room state (#9168)Travis Ralston2021-02-021-0/+15
| | | | | | | | | | This could arguably replace the existing admin API for `/members`, however that is out of scope of this change. This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
* | Put OIDC callback URI under /_synapse/client. (#9288)Richard van der Hoff2021-02-011-9/+6
| |
* | Refactor email summary generation. (#9260)Patrick Cloke2021-02-011-0/+30
| | | | | | | | | | | | * Fixes a case where no summary text was returned. * The use of messages_from_person vs. messages_from_person_and_others was tweaked to depend on whether there was 1 sender or multiple senders, not based on if there was 1 room or multiple rooms.
* | Merge branch 'social_login' into developRichard van der Hoff2021-02-015-36/+40
|\ \
| * | Replace username picker with a template (#9275)Richard van der Hoff2021-02-011-3/+2
| | | | | | | | | | | | | | | There's some prelimiary work here to pull out the construction of a jinja environment to a separate function. I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
| * | Improve styling and wording of SSO redirect confirm template (#9272)Richard van der Hoff2021-02-013-18/+22
| | |
| * | Fix SSO on workers (#9271)Richard van der Hoff2021-02-012-16/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #8966. * Factor out build_synapse_client_resource_tree Start a function which will mount resources common to all workers. * Move sso init into build_synapse_client_resource_tree ... so that we don't have to do it for each worker * Fix SSO-login-via-a-worker Expose the SSO login endpoints on workers, like the documentation says. * Update workers config for new endpoints Add documentation for endpoints recently added (#8942, #9017, #9262) * remove submit_token from workers endpoints list this *doesn't* work on workers (yet). * changelog * Add a comment about the odd path for SAML2Resource
| * | Split out a separate endpoint to complete SSO registration (#9262)Richard van der Hoff2021-02-011-1/+13
| | | | | | | | | There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
* | | Ratelimit invites by room and target user (#9258)Erik Johnston2021-01-292-1/+127
| | |
* | | Ratelimit 3PID /requestToken API (#9238)Erik Johnston2021-01-284-8/+97
| |/ |/|
* | Merge branch 'social_login' into developRichard van der Hoff2021-01-272-2/+93
|\|
| * Implement MSC2858 support (#9183)Richard van der Hoff2021-01-272-2/+93
| | | | | | Fixes #8928.
* | Precompute joined hosts and store in Redis (#9198)Erik Johnston2021-01-261-16/+25
| |
* | Try to recover from unknown encodings when previewing media. (#9164)Patrick Cloke2021-01-261-0/+29
| | | | | | | | | | | | Treat unknown encodings (according to lxml) as UTF-8 when generating a preview for HTML documents. This isn't fully accurate, but will hopefully give a reasonable title and summary.
* | Add an admin API for shadow-banning users. (#9209)Patrick Cloke2021-01-252-6/+66
| | | | | | | | | | | | | | | | | | | | This expands the current shadow-banning feature to be usable via the admin API and adds documentation for it. A shadow-banned users receives successful responses to their client-server API requests, but the events are not propagated into rooms. Shadow-banning a user should be used as a tool of last resort and may lead to confusing or broken behaviour for the client.
* | Merge tag 'v1.26.0rc2' into developPatrick Cloke2021-01-251-0/+12
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
| * Fix chain cover update to handle events with duplicate auth events (#9210)Erik Johnston2021-01-221-0/+12
| |
* | Handle missing content keys when calculating presentable names. (#9165)Patrick Cloke2021-01-252-1/+230
| | | | | | | | Treat the content as untrusted and do not assume it is of the proper form.
* | Return a 404 if no valid thumbnail is found. (#9163)Patrick Cloke2021-01-211-1/+24
| | | | | | | | | | | | If no thumbnail of the requested type exists, return a 404 instead of erroring. This doesn't quite match the spec (which does not define what happens if no thumbnail can be found), but is consistent with what Synapse already does.
* | Add tests for List Users Admin API (#9045)Dirk Klimpel2021-01-211-27/+196
|/
* Prefix idp_id with "oidc-" (#9189)Richard van der Hoff2021-01-211-1/+1
| | | ... to avoid clashes with other SSO mechanisms
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-1/+1
|
* Give `public_baseurl` a default value (#9159)Richard van der Hoff2021-01-202-10/+0
|
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-181-6/+106
|
* Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128)Richard van der Hoff2021-01-184-82/+181
| | | | | | | | | | | | | | | | * Factor out a common TestHtmlParser Looks like I'm doing this in a few different places. * Improve OIDC login test Complete the OIDC login flow, rather than giving up halfway through. * Ensure that OIDC login works with multiple OIDC providers * Fix bugs in handling clientRedirectUrl - don't drop duplicate query-params, or params with no value - allow utf-8 in query-params
* Fix test failure due to bad mergeRichard van der Hoff2021-01-151-1/+3
| | | | | | | 0dd2649c1 (#9112) changed the signature of `auth_via_oidc`. Meanwhile, 26d10331e (#9091) introduced a new test which relied on the old signature of `auth_via_oidc`. The two branches were never tested together until they landed in develop.
* Make chain cover index bg update go faster (#9124)Erik Johnston2021-01-151-31/+186
| | | | | | We do this by allowing a single iteration to process multiple rooms at a time, as there are often a lot of really tiny rooms, which can massively slow things down.
* Land support for multiple OIDC providers (#9110)Richard van der Hoff2021-01-151-2/+2
| | | | | | | | | | | | | | | | | | | | | | | This is the final step for supporting multiple OIDC providers concurrently. First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before: oidc_config: enabled: true issuer: "https://oidc_provider" # etc After: oidc_providers: - idp_id: prov1 issuer: "https://oidc_provider" - idp_id: prov2 issuer: "https://another_oidc_provider" The old format is still grandfathered in. With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
* Add an admin API endpoint to protect media. (#9086)Patrick Cloke2021-01-151-3/+5
| | | | | | Protecting media stops it from being quarantined when e.g. all media in a room is quarantined. This is useful for sticker packs and other media that is uploaded by server administrators, but used by many people.
* Avoid raising the body exceeded error multiple times. (#9108)Patrick Cloke2021-01-152-2/+103
| | | | | Previously this code generated unreferenced `Deferred` instances which caused "Unhandled Deferreds" errors to appear in error situations.
* Improve UsernamePickerTestCase (#9112)Richard van der Hoff2021-01-154-125/+113
| | | | | * make the OIDC bits of the test work at a higher level - via the REST api instead of poking the OIDCHandler directly. * Move it to test_login.py, where I think it fits better.
* Store an IdP ID in the OIDC session (#9109)Richard van der Hoff2021-01-151-1/+2
| | | | | Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP. As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
* Merge pull request #9091 from matrix-org/rav/error_on_bad_ssoRichard van der Hoff2021-01-151-0/+27
|\ | | | | Give the user a better error when they present bad SSO creds
| * Add a test for wrong user returned by SSORichard van der Hoff2021-01-131-0/+27
| |
* | Fix event chain bg update. (#9118)Erik Johnston2021-01-141-0/+8
| | | | | | | | We passed in a graph to `sorted_topologically` which didn't have an entry for each node (as we dropped nodes with no edges).
* | Add background update for add chain cover index (#9029)Erik Johnston2021-01-141-0/+114
| |
* | Split OidcProvider out of OidcHandler (#9107)Richard van der Hoff2021-01-141-45/+48
|/ | | | | | | The idea here is that we will have an instance of OidcProvider for each configured IdP, with OidcHandler just doing the marshalling of them. For now it's still hardcoded with a single provider.
* Add a test for UI-Auth-via-SSO (#9082)Richard van der Hoff2021-01-133-42/+226
| | | | | * Add complete test for UI-Auth-via-SSO. * review comments
* Add basic domain validation for `DomainSpecificString.is_valid`. (#9071)Patrick Cloke2021-01-131-0/+4
| | | | | This checks that the domain given to `DomainSpecificString.is_valid` (e.g. `UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation was done on the localpart (e.g. the sigil), but not the domain portion.
* Merge branch 'master' into developErik Johnston2021-01-131-0/+130
|\
| * Don't apply the IP range blacklist to proxy connections (#9084)Marcus2021-01-121-0/+130
| | | | | | | | | | It is expected that the proxy would be on a private IP address so the configured proxy should be connected to regardless of the IP range blacklist.
* | Preparatory refactors of OidcHandler (#9067)Richard van der Hoff2021-01-131-24/+37
| | | | | | | | | | | | | | | | Some light refactoring of OidcHandler, in preparation for bigger things: * remove inheritance from deprecated BaseHandler * add an object to hold the things that go into a session cookie * factor out a separate class for manipulating said cookies
* | Remove user's avatar URL and displayname when deactivated. (#8932)Dirk Klimpel2021-01-125-3/+284
| | | | | | This only applies if the user's data is to be erased.
* | Handle bad JSON data being returned from the federation API. (#9070)Patrick Cloke2021-01-121-1/+1
| |
* | Remove SynapseRequest.get_user_agent (#9069)Richard van der Hoff2021-01-123-4/+3
| | | | | | | | | | | | | | | | | | | | | | SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests", which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the LoggingContext and write the right entries to the request log). Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a SynapseRequest when there is nothing synapse-specific about the Request at all, and any old twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult. In short: move get_user_agent out to a utility function.
* | Also support remote users on the joined_rooms admin API. (#8948)David Teller2021-01-111-6/+52
| | | | | | | | For remote users, only the rooms which the server knows about are returned. Local users have all of their joined rooms returned.
* | Removes unnecessary declarations in the tests for the admin API. (#9063)Dirk Klimpel2021-01-116-17/+0
| |
* | Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-113-25/+737
| |
* | Merge pull request #9036 from matrix-org/rav/multi_idp/testsRichard van der Hoff2021-01-082-36/+249
|\ \ | | | | | | Add tests for the IdP picker
| * | Add some tests for the IDP picker flowRichard van der Hoff2021-01-072-3/+191
| | |
| * | Fix type hints in test_login.pyRichard van der Hoff2021-01-071-24/+54
| | |
| * | black-format tests/rest/client/v1/test_login.pyRichard van der Hoff2021-01-071-11/+6
| | | | | | | | | | | | | | | black seems to want to reformat this, despite `black --check` being happy with it :/
* | | Fix validate_config on nested objects (#9054)Richard van der Hoff2021-01-081-0/+53
| | |
* | | Improve the performance of calculating ignored users in large rooms (#9024)Patrick Cloke2021-01-071-0/+120
|/ / | | | | | | | | | | This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>
* | Skip unit tests which require optional dependencies (#9031)Richard van der Hoff2021-01-076-13/+89
| | | | | | If we are lacking an optional dependency, skip the tests that rely on it.
* | Empty iterables should count towards cache usage. (#9028)Patrick Cloke2021-01-061-23/+50
| |
* | Implement MSC2176: Updated redaction rules (#8984)Patrick Cloke2021-01-051-21/+164
| | | | | | | | An experimental room version ("org.matrix.msc2176") contains the new redaction rules for testing.
* | Combine the SSO Redirect Servlets (#9015)Richard van der Hoff2021-01-041-1/+1
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | * Implement CasHandler.handle_redirect_request ... to make it match OidcHandler and SamlHandler * Clean up interface for OidcHandler.handle_redirect_request Make it accept `client_redirect_url=None`. * Clean up interface for `SamlHandler.handle_redirect_request` ... bring it into line with CAS and OIDC by making it take a Request parameter, move the magic for `client_redirect_url` for UIA into the handler, and fix the return type to be a `str` rather than a `bytes`. * Define a common protocol for SSO auth provider impls * Give SsoIdentityProvider an ID and register them * Combine the SSO Redirect servlets Now that the SsoHandler knows about the identity providers, we can combine the various *RedirectServlets into a single implementation which delegates to the right IdP. * changelog
* Add type hints to the crypto module. (#8999)Patrick Cloke2021-01-041-5/+5
|
* Use the SSO handler helpers for CAS registration/login. (#8856)Patrick Cloke2021-01-031-0/+121
|
* Fix RoomDirectoryFederationTests and make them actually run (#8998)Andrew Morgan2020-12-302-21/+18
| | | | | | | | | | The `RoomDirectoryFederationTests` tests were not being run unless explicitly called as an `__init__.py` file was not present in `tests/federation/transport/`. Thus the folder was not a python module, and `trial` did not look inside for any test cases to run. This was found while working on #6739. This PR adds a `__init__.py` and also fixes the test in a couple ways: - Switch to subclassing `unittest.FederatingHomeserverTestCase` instead, which sets up federation endpoints for us. - Supply a `federation_auth_origin` to `make_request` in order to more act like the request is coming from another server, instead of just an unauthenicated client requesting a federation endpoint. I found that the second point makes no difference to the test passing, but felt like the right thing to do if we're testing over federation.
* Drop the unused local_invites table. (#8979)Patrick Cloke2020-12-291-1/+0
| | | This table has been unused since Synapse v1.17.0.
* Support PyJWT v2.0.0. (#8986)Patrick Cloke2020-12-221-4/+12
| | | | Tests were broken due to an API changing. The code used in Synapse proper should be compatible with both versions already.
* Allow server admin to get admin bit in rooms where local user is an admin ↵Erik Johnston2020-12-181-0/+138
| | | | | | | (#8756) This adds an admin API that allows a server admin to get power in a room if a local user has power in a room. Will also invite the user if they're not in the room and its a private room. Can specify another user (rather than the admin user) to be granted power. Co-authored-by: Matthew Hodgson <matthew@matrix.org>
* Fix mainline ordering in state res v2 (#8971)Erik Johnston2020-12-181-1/+56
| | | | | | This had two effects 1) it'd give the wrong answer and b) would iterate *all* power levels in the auth chain of each event. The latter of which can be *very* expensive for certain types of IRC bridge rooms that have large numbers of power level changes.
* Implement a username picker for synapse (#8942)Richard van der Hoff2020-12-182-2/+149
| | | | | | | | | | | | | | The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow `UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser back to the client, we redirect to a username-picker resource, which allows the user to enter a username. We *then* complete the SSO flow (including doing the client permission checks). The static resources for the username picker itself (in https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker) are essentially lifted wholesale from https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. As the comment says, we might want to think about making them customisable, but that can be a follow-up. Fixes #8876.
* Allow re-using a UI auth validation for a period of time (#8970)Patrick Cloke2020-12-181-31/+63
|
* Merge remote-tracking branch 'origin/erikj/as_mau_block' into developErik Johnston2020-12-181-2/+44
|\
| * Update tests/test_mau.pyErik Johnston2020-12-171-1/+2
| | | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
| * Correctly handle AS registerations and add testErik Johnston2020-12-171-2/+43
| |
* | Try and drop stale extremities. (#8929)Erik Johnston2020-12-181-0/+334
| | | | | | | | If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.
* | Fix `UsersListTestCase` (#8964)Richard van der Hoff2020-12-171-6/+4
| |
* | Use the simple dictionary in full text search for the user directory (#8959)Brendan Abolivier2020-12-171-0/+23
| | | | | | | | | | * Use the simple dictionary in fts for the user directory * Clarify naming
* | Fix a bug that deactivated users appear in the directory (#8933)Dirk Klimpel2020-12-172-2/+88
| | | | | | | | | | | | | | | | | | | | Fixes a bug that deactivated users appear in the directory when their profile information was updated. To change profile information of deactivated users is neccesary for example you will remove displayname or avatar. But they should not appear in directory. They are deactivated. Co-authored-by: Erik Johnston <erikj@jki.re>
* | Make search statement in List Room and User Admin API case-insensitive (#8931)Dirk Klimpel2020-12-173-3/+112
| |
* | Add a maximum size for well-known lookups. (#8950)Patrick Cloke2020-12-161-0/+27
| |
* | Push login completion down into SsoHandler (#8941)Richard van der Hoff2020-12-161-4/+4
| | | | | | This is another part of my work towards fixing #8876. It moves some of the logic currently in the SAML and OIDC handlers - in particular the call to `AuthHandler.complete_sso_login` down into the `SsoHandler`.
* | Convert internal pusher dicts to attrs classes. (#8940)Patrick Cloke2020-12-163-9/+9
| | | | | | This improves type hinting and should use less memory.
* | Merge pull request #8951 from matrix-org/rav/username_picker_2Richard van der Hoff2020-12-162-82/+105
|\ \ | | | | | | More preparatory refactoring of the OidcHandler tests
| * | Make `_make_callback_with_userinfo` asyncRichard van der Hoff2020-12-151-68/+83
| | | | | | | | | | | | | | | | | | ... so that we can test its behaviour when it raises. Also pull it out to the top level so that I can use it from other test classes.
| * | Remove spurious mocking of complete_sso_loginRichard van der Hoff2020-12-151-2/+0
| | | | | | | | | | | | The tests that need this all do it already.
| * | Test `get_extra_attributes` fallbackRichard van der Hoff2020-12-152-12/+22
| | | | | | | | | | | | | | | despite the warnings saying "don't implement get_extra_attributes", we had implemented it, so the tests weren't doing what we thought they were.
* | | lintRichard van der Hoff2020-12-155-7/+3
| | |
* | | Remove spurious "SynapseRequest" result from `make_request"Richard van der Hoff2020-12-1559-983/+742
| | | | | | | | | | | | This was never used, so let's get rid of it.
* | | Remove redundant reading of SynapseRequest.argsRichard van der Hoff2020-12-151-9/+1
| | | | | | | | | | | | this didn't seem to be doing a lot, so remove it.
* | | Skip redundant check on `request.args`Richard van der Hoff2020-12-151-2/+1
| | |
* | | remove 'response' result from `_get_shared_rooms`Richard van der Hoff2020-12-151-8/+9
| | |