summary refs log tree commit diff
path: root/synapse/http (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge tag 'v1.38.0' into babolivier/dinsic_1.41.0Brendan Abolivier2021-09-012-5/+47
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.38.0 (2021-07-13) =========================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. No significant changes since 1.38.0rc3. Synapse 1.38.0rc3 (2021-07-13) ============================== Internal Changes ---------------- - Build the Debian packages in CI. ([\#10247](https://github.com/matrix-org/synapse/issues/10247), [\#10379](https://github.com/matrix-org/synapse/issues/10379)) Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331)) Synapse 1.38.0rc1 (2021-07-06) ============================== Features -------- - Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918). ([\#9450](https://github.com/matrix-org/synapse/issues/9450)) - Add support for evicting cache entries based on last access time. ([\#10205](https://github.com/matrix-org/synapse/issues/10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](https://github.com/matrix-org/synapse/issues/10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](https://github.com/matrix-org/synapse/issues/10225), [\#10243](https://github.com/matrix-org/synapse/issues/10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\#10261](https://github.com/matrix-org/synapse/issues/10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](https://github.com/matrix-org/synapse/issues/10263)) - Add metrics for new inbound federation staging area. ([\#10284](https://github.com/matrix-org/synapse/issues/10284)) - Add script to print information about recently registered users. ([\#10290](https://github.com/matrix-org/synapse/issues/10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](https://github.com/matrix-org/synapse/issues/10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](https://github.com/matrix-org/synapse/issues/10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](https://github.com/matrix-org/synapse/issues/10264), [\#10267](https://github.com/matrix-org/synapse/issues/10267), [\#10282](https://github.com/matrix-org/synapse/issues/10282), [\#10286](https://github.com/matrix-org/synapse/issues/10286), [\#10291](https://github.com/matrix-org/synapse/issues/10291), [\#10314](https://github.com/matrix-org/synapse/issues/10314), [\#10326](https://github.com/matrix-org/synapse/issues/10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](https://github.com/matrix-org/synapse/issues/10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](https://github.com/matrix-org/synapse/issues/10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](https://github.com/matrix-org/synapse/issues/10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](https://github.com/matrix-org/synapse/issues/10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](https://github.com/matrix-org/synapse/issues/10258)) - Fix homeserver config option name in presence router documentation. ([\#10288](https://github.com/matrix-org/synapse/issues/10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\#10302](https://github.com/matrix-org/synapse/issues/10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](https://github.com/matrix-org/synapse/issues/10114)) - Add type hints to the federation servlets. ([\#10213](https://github.com/matrix-org/synapse/issues/10213)) - Improve the reliability of auto-joining remote rooms. ([\#10237](https://github.com/matrix-org/synapse/issues/10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](https://github.com/matrix-org/synapse/issues/10239)) - Fix type hints for computing auth events. ([\#10253](https://github.com/matrix-org/synapse/issues/10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](https://github.com/matrix-org/synapse/issues/10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](https://github.com/matrix-org/synapse/issues/10268)) - Re-enable a SyTest after it has been fixed. ([\#10292](https://github.com/matrix-org/synapse/issues/10292))
| * Add additional types to the federation transport server. (#10213)Patrick Cloke2021-06-281-4/+46
| |
| * Drop Origin & Accept from Access-Control-Allow-Headers value (#10114)Michael[tm] Smith2021-06-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Drop Origin & Accept from Access-Control-Allow-Headers value This change drops the Origin and Accept header names from the value of the Access-Control-Allow-Headers response header sent by Synapse. Per the CORS protocol, it’s not necessary or useful to include those header names. Details: Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin is a “forbidden header name” set by the browser and that frontend JavaScript code is never allowed to set. So the value of Access-Control-Allow-Headers isn’t relevant to Origin or in general to other headers set by the browser itself — the browser never ever consults the Access-Control-Allow-Headers value to confirm that it’s OK for the request to include an Origin header. And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header, Accept is a “CORS-safelisted request-header”, which means that browsers allow requests to contain the Accept header regardless of whether the Access-Control-Allow-Headers value contains "Accept". So it’s unnecessary for the Access-Control-Allow-Headers to explicitly include Accept. Browsers will not perform a CORS preflight for requests containing an Accept request header. Related: https://github.com/matrix-org/matrix-doc/pull/3225 Signed-off-by: Michael[tm] Smith <mike@w3.org>
* | Merge tag 'v1.37.0' into babolivier/dinsic_1.41.0Brendan Abolivier2021-09-012-39/+31
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.37.0 (2021-06-29) =========================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Synapse 1.37.0rc1 (2021-06-24) ============================== Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))
| * update black to 21.6b0 (#10197)Marcus2021-06-171-1/+1
| | | | | | | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
| * Fix 'ip_range_whitelist' not working for federation servers (#10115)Michael Kutzner2021-06-151-1/+3
| | | | | | | | | | Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers. Signed-off-by: Michael Kutzner 1mikure@gmail.com
| * Implement knock feature (#6739)Sorunome2021-06-091-1/+0
| | | | | | | | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
| * Clean up the interface for injecting opentracing over HTTP (#10143)Richard van der Hoff2021-06-091-7/+3
| | | | | | | | | | | | | | * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog
| * Add type hints to the federation server transport. (#10080)Patrick Cloke2021-06-081-0/+24
| |
* | Merge tag 'v1.36.0' into babolivier/dinsic_1.41.0Brendan Abolivier2021-08-311-70/+201
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.36.0 (2021-06-15) =========================== No significant changes. Synapse 1.36.0rc2 (2021-06-11) ============================== Bugfixes -------- - Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149)) - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163)) - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165)) Synapse 1.36.0rc1 (2021-06-08) ============================== Features -------- - Add new endpoint `/_matrix/client/r0/rooms/{roomId}/aliases` from Client-Server API r0.6.1 (previously [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432)). ([\#9224](https://github.com/matrix-org/synapse/issues/9224)) - Improve performance of incoming federation transactions in large rooms. ([\#9953](https://github.com/matrix-org/synapse/issues/9953), [\#9973](https://github.com/matrix-org/synapse/issues/9973)) - Rewrite logic around verifying JSON object and fetching server keys to be more performant and use less memory. ([\#10035](https://github.com/matrix-org/synapse/issues/10035)) - Add new admin APIs for unprotecting local media from quarantine. Contributed by @dklimpel. ([\#10040](https://github.com/matrix-org/synapse/issues/10040)) - Add new admin APIs to remove media by media ID from quarantine. Contributed by @dklimpel. ([\#10044](https://github.com/matrix-org/synapse/issues/10044)) - Make reason and score parameters optional for reporting content. Implements [MSC2414](https://github.com/matrix-org/matrix-doc/pull/2414). Contributed by Callum Brown. ([\#10077](https://github.com/matrix-org/synapse/issues/10077)) - Add support for routing more requests to workers. ([\#10084](https://github.com/matrix-org/synapse/issues/10084)) - Report OpenTracing spans for database activity. ([\#10113](https://github.com/matrix-org/synapse/issues/10113), [\#10136](https://github.com/matrix-org/synapse/issues/10136), [\#10141](https://github.com/matrix-org/synapse/issues/10141)) - Significantly reduce memory usage of joining large remote rooms. ([\#10117](https://github.com/matrix-org/synapse/issues/10117)) Bugfixes -------- - Fixed a bug causing replication requests to fail when receiving a lot of events via federation. ([\#10082](https://github.com/matrix-org/synapse/issues/10082)) - Fix a bug in the `force_tracing_for_users` option introduced in Synapse v1.35 which meant that the OpenTracing spans produced were missing most tags. ([\#10092](https://github.com/matrix-org/synapse/issues/10092)) - Fixed a bug that could cause Synapse to stop notifying application services. Contributed by Willem Mulder. ([\#10107](https://github.com/matrix-org/synapse/issues/10107)) - Fix bug where the server would attempt to fetch the same history in the room from a remote server multiple times in parallel. ([\#10116](https://github.com/matrix-org/synapse/issues/10116)) - Fix a bug introduced in Synapse 1.33.0 which caused replication requests to fail when receiving a lot of very large events via federation. ([\#10118](https://github.com/matrix-org/synapse/issues/10118)) - Fix bug when using workers where pagination requests failed if a remote server returned zero events from `/backfill`. Introduced in 1.35.0. ([\#10133](https://github.com/matrix-org/synapse/issues/10133)) Improved Documentation ---------------------- - Clarify security note regarding hosting Synapse on the same domain as other web applications. ([\#9221](https://github.com/matrix-org/synapse/issues/9221)) - Update CAPTCHA documentation to mention turning off the verify origin feature. Contributed by @aaronraimist. ([\#10046](https://github.com/matrix-org/synapse/issues/10046)) - Tweak wording of database recommendation in `INSTALL.md`. Contributed by @aaronraimist. ([\#10057](https://github.com/matrix-org/synapse/issues/10057)) - Add initial infrastructure for rendering Synapse documentation with mdbook. ([\#10086](https://github.com/matrix-org/synapse/issues/10086)) - Convert the remaining Admin API documentation files to markdown. ([\#10089](https://github.com/matrix-org/synapse/issues/10089)) - Make a link in docs use HTTPS. Contributed by @RhnSharma. ([\#10130](https://github.com/matrix-org/synapse/issues/10130)) - Fix broken link in Docker docs. ([\#10132](https://github.com/matrix-org/synapse/issues/10132)) Deprecations and Removals ------------------------- - Remove the experimental `spaces_enabled` flag. The spaces features are always available now. ([\#10063](https://github.com/matrix-org/synapse/issues/10063)) Internal Changes ---------------- - Tell CircleCI to build Docker images from `main` branch. ([\#9906](https://github.com/matrix-org/synapse/issues/9906)) - Simplify naming convention for release branches to only include the major and minor version numbers. ([\#10013](https://github.com/matrix-org/synapse/issues/10013)) - Add `parse_strings_from_args` for parsing an array from query parameters. ([\#10048](https://github.com/matrix-org/synapse/issues/10048), [\#10137](https://github.com/matrix-org/synapse/issues/10137)) - Remove some dead code regarding TLS certificate handling. ([\#10054](https://github.com/matrix-org/synapse/issues/10054)) - Remove redundant, unmaintained `convert_server_keys` script. ([\#10055](https://github.com/matrix-org/synapse/issues/10055)) - Improve the error message printed by synctl when synapse fails to start. ([\#10059](https://github.com/matrix-org/synapse/issues/10059)) - Fix GitHub Actions lint for newsfragments. ([\#10069](https://github.com/matrix-org/synapse/issues/10069)) - Update opentracing to inject the right context into the carrier. ([\#10074](https://github.com/matrix-org/synapse/issues/10074)) - Fix up `BatchingQueue` implementation. ([\#10078](https://github.com/matrix-org/synapse/issues/10078)) - Log method and path when dropping request due to size limit. ([\#10091](https://github.com/matrix-org/synapse/issues/10091)) - In Github Actions workflows, summarize the Sytest results in an easy-to-read format. ([\#10094](https://github.com/matrix-org/synapse/issues/10094)) - Make `/sync` do fewer state resolutions. ([\#10102](https://github.com/matrix-org/synapse/issues/10102)) - Add missing type hints to the admin API servlets. ([\#10105](https://github.com/matrix-org/synapse/issues/10105)) - Improve opentracing annotations for `Notifier`. ([\#10111](https://github.com/matrix-org/synapse/issues/10111)) - Enable Prometheus metrics for the jaeger client library. ([\#10112](https://github.com/matrix-org/synapse/issues/10112)) - Work to improve the responsiveness of `/sync` requests. ([\#10124](https://github.com/matrix-org/synapse/issues/10124)) - OpenTracing: use a consistent name for background processes. ([\#10135](https://github.com/matrix-org/synapse/issues/10135))
| * Correct type hints for parse_string(s)_from_args. (#10137)Patrick Cloke2021-06-081-68/+111
| |
| * Merge branch 'master' into developAndrew Morgan2021-06-011-1/+13
| |\
| * | Log method and path when dropping request due to size limit (#10091)Erik Johnston2021-05-281-1/+3
| | |
| * | Add `parse_strings_from_args` to get `prev_events` array (#10048)Eric Eastwood2021-05-281-43/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Split out from https://github.com/matrix-org/synapse/pull/9247 Strings: - `parse_string` - `parse_string_from_args` - `parse_strings_from_args` For comparison with ints: - `parse_integer` - `parse_integer_from_args` Previous discussions: - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791 Signed-off-by: Eric Eastwood <erice@element.io>
* | | Merge tag 'v1.35.0' into babolivier/dinsic_1.41.0Brendan Abolivier2021-08-313-37/+146
|\ \ \ | | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.35.0 (2021-06-01) =========================== Note that [the tag](https://github.com/matrix-org/synapse/releases/tag/v1.35.0rc3) and [docker images](https://hub.docker.com/layers/matrixdotorg/synapse/v1.35.0rc3/images/sha256-34ccc87bd99a17e2cbc0902e678b5937d16bdc1991ead097eee6096481ecf2c4?context=explore) for `v1.35.0rc3` were incorrectly built. If you are experiencing issues with either, it is recommended to upgrade to the equivalent tag or docker image for the `v1.35.0` release. Deprecations and Removals ------------------------- - The core Synapse development team plan to drop support for the [unstable API of MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option, in August 2021. Client authors should ensure that their clients are updated to use the stable API (which has been supported since Synapse 1.30) well before that time, to give their users time to upgrade. ([\#10101](https://github.com/matrix-org/synapse/issues/10101)) Bugfixes -------- - Fixed a bug causing replication requests to fail when receiving a lot of events via federation. Introduced in v1.33.0. ([\#10082](https://github.com/matrix-org/synapse/issues/10082)) - Fix HTTP response size limit to allow joining very large rooms over federation. Introduced in v1.33.0. ([\#10093](https://github.com/matrix-org/synapse/issues/10093)) Internal Changes ---------------- - Log method and path when dropping request due to size limit. ([\#10091](https://github.com/matrix-org/synapse/issues/10091)) Synapse 1.35.0rc2 (2021-05-27) ============================== Bugfixes -------- - Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079)) Synapse 1.35.0rc1 (2021-05-25) ============================== Features -------- - Add experimental support to allow a user who could join a restricted room to view it in the spaces summary. ([\#9922](https://github.com/matrix-org/synapse/issues/9922), [\#10007](https://github.com/matrix-org/synapse/issues/10007), [\#10038](https://github.com/matrix-org/synapse/issues/10038)) - Reduce memory usage when joining very large rooms over federation. ([\#9958](https://github.com/matrix-org/synapse/issues/9958)) - Add a configuration option which allows enabling opentracing by user id. ([\#9978](https://github.com/matrix-org/synapse/issues/9978)) - Enable experimental support for [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946) (spaces summary API) and [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) (restricted join rules) by default. ([\#10011](https://github.com/matrix-org/synapse/issues/10011)) Bugfixes -------- - Fix a bug introduced in v1.26.0 which meant that `synapse_port_db` would not correctly initialise some postgres sequences, requiring manual updates afterwards. ([\#9991](https://github.com/matrix-org/synapse/issues/9991)) - Fix `synctl`'s `--no-daemonize` parameter to work correctly with worker processes. ([\#9995](https://github.com/matrix-org/synapse/issues/9995)) - Fix a validation bug introduced in v1.34.0 in the ordering of spaces in the space summary API. ([\#10002](https://github.com/matrix-org/synapse/issues/10002)) - Fixed deletion of new presence stream states from database. ([\#10014](https://github.com/matrix-org/synapse/issues/10014), [\#10033](https://github.com/matrix-org/synapse/issues/10033)) - Fixed a bug with very high resolution image uploads throwing internal server errors. ([\#10029](https://github.com/matrix-org/synapse/issues/10029)) Updates to the Docker image --------------------------- - Fix bug introduced in Synapse 1.33.0 which caused a `Permission denied: '/homeserver.log'` error when starting Synapse with the generated log configuration. Contributed by Sergio Miguéns Iglesias. ([\#10045](https://github.com/matrix-org/synapse/issues/10045)) Improved Documentation ---------------------- - Add hardened systemd files as proposed in [#9760](https://github.com/matrix-org/synapse/issues/9760) and added them to `contrib/`. Change the docs to reflect the presence of these files. ([\#9803](https://github.com/matrix-org/synapse/issues/9803)) - Clarify documentation around SSO mapping providers generating unique IDs and localparts. ([\#9980](https://github.com/matrix-org/synapse/issues/9980)) - Updates to the PostgreSQL documentation (`postgres.md`). ([\#9988](https://github.com/matrix-org/synapse/issues/9988), [\#9989](https://github.com/matrix-org/synapse/issues/9989)) - Fix broken link in user directory documentation. Contributed by @junquera. ([\#10016](https://github.com/matrix-org/synapse/issues/10016)) - Add missing room state entry to the table of contents of room admin API. ([\#10043](https://github.com/matrix-org/synapse/issues/10043)) Deprecations and Removals ------------------------- - Removed support for the deprecated `tls_fingerprints` configuration setting. Contributed by Jerin J Titus. ([\#9280](https://github.com/matrix-org/synapse/issues/9280)) Internal Changes ---------------- - Allow sending full presence to users via workers other than the one that called `ModuleApi.send_local_online_presence_to`. ([\#9823](https://github.com/matrix-org/synapse/issues/9823)) - Update comments in the space summary handler. ([\#9974](https://github.com/matrix-org/synapse/issues/9974)) - Minor enhancements to the `@cachedList` descriptor. ([\#9975](https://github.com/matrix-org/synapse/issues/9975)) - Split multipart email sending into a dedicated handler. ([\#9977](https://github.com/matrix-org/synapse/issues/9977)) - Run `black` on files in the `scripts` directory. ([\#9981](https://github.com/matrix-org/synapse/issues/9981)) - Add missing type hints to `synapse.util` module. ([\#9982](https://github.com/matrix-org/synapse/issues/9982)) - Simplify a few helper functions. ([\#9984](https://github.com/matrix-org/synapse/issues/9984), [\#9985](https://github.com/matrix-org/synapse/issues/9985), [\#9986](https://github.com/matrix-org/synapse/issues/9986)) - Remove unnecessary property from SQLBaseStore. ([\#9987](https://github.com/matrix-org/synapse/issues/9987)) - Remove `keylen` param on `LruCache`. ([\#9993](https://github.com/matrix-org/synapse/issues/9993)) - Update the Grafana dashboard in `contrib/`. ([\#10001](https://github.com/matrix-org/synapse/issues/10001)) - Add a batching queue implementation. ([\#10017](https://github.com/matrix-org/synapse/issues/10017)) - Reduce memory usage when verifying signatures on large numbers of events at once. ([\#10018](https://github.com/matrix-org/synapse/issues/10018)) - Properly invalidate caches for destination retry timings every (instead of expiring entries every 5 minutes). ([\#10036](https://github.com/matrix-org/synapse/issues/10036)) - Fix running complement tests with Synapse workers. ([\#10039](https://github.com/matrix-org/synapse/issues/10039)) - Fix typo in `get_state_ids_for_event` docstring where the return type was incorrect. ([\#10050](https://github.com/matrix-org/synapse/issues/10050))
| * | Log method and path when dropping request due to size limit (#10091)Erik Johnston2021-05-281-1/+3
| | |
| * | Allow response of `/send_join` to be larger. (#10093)Erik Johnston2021-05-281-1/+13
| |/ | | | | Fixes #10087.
| * Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958)Erik Johnston2021-05-202-36/+131
| | | | | | Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
* | Merge tag 'v1.33.0' into babolivier/dinsic_1.41.0Brendan Abolivier2021-08-3114-35/+105
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.33.0 (2021-05-05) =========================== Features -------- - Build Debian packages for Ubuntu 21.04 (Hirsute Hippo). ([\#9909](https://github.com/matrix-org/synapse/issues/9909)) Synapse 1.33.0rc2 (2021-04-29) ============================== Bugfixes -------- - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900)) Synapse 1.33.0rc1 (2021-04-28) ============================== Features -------- - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814)) - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802)) - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162)) - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786)) - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821)) - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845)) - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879)) - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
| * Improved validation for received requests (#9817)Richard van der Hoff2021-04-231-5/+27
| | | | | | | | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
| * pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-231-9/+28
| |
| * Limit the size of HTTP responses read over federation. (#9833)Richard van der Hoff2021-04-232-8/+50
| |
| * Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1413-13/+0
| | | | | | | | | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* | Merge tag 'v1.32.0' into babolivier/dinsic_1.41.0Brendan Abolivier2021-08-314-31/+97
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.32.0 (2021-04-20) =========================== **Note:** This release requires Python 3.6+ and Postgres 9.6+ or SQLite 3.22+. This release removes the deprecated `GET /_synapse/admin/v1/users/<user_id>` admin API. Please use the [v2 API](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#query-user-account) instead, which has improved capabilities. This release requires Application Services to use type `m.login.application_service` when registering users via the `/_matrix/client/r0/register` endpoint to comply with the spec. Please ensure your Application Services are up to date. Bugfixes -------- - Fix the log lines of nested logging contexts. Broke in 1.32.0rc1. ([\#9829](https://github.com/matrix-org/synapse/issues/9829)) Synapse 1.32.0rc1 (2021-04-13) ============================== Features -------- - Add a Synapse module for routing presence updates between users. ([\#9491](https://github.com/matrix-org/synapse/issues/9491)) - Add an admin API to manage ratelimit for a specific user. ([\#9648](https://github.com/matrix-org/synapse/issues/9648)) - Include request information in structured logging output. ([\#9654](https://github.com/matrix-org/synapse/issues/9654)) - Add `order_by` to the admin API `GET /_synapse/admin/v2/users`. Contributed by @dklimpel. ([\#9691](https://github.com/matrix-org/synapse/issues/9691)) - Replace the `room_invite_state_types` configuration setting with `room_prejoin_state`. ([\#9700](https://github.com/matrix-org/synapse/issues/9700)) - Add experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9717](https://github.com/matrix-org/synapse/issues/9717), [\#9735](https://github.com/matrix-org/synapse/issues/9735)) - Update experimental support for Spaces: include `m.room.create` in the room state sent with room-invites. ([\#9710](https://github.com/matrix-org/synapse/issues/9710)) - Synapse now requires Python 3.6 or later. It also requires Postgres 9.6 or later or SQLite 3.22 or later. ([\#9766](https://github.com/matrix-org/synapse/issues/9766)) Bugfixes -------- - Prevent `synapse_forward_extremities` and `synapse_excess_extremity_events` Prometheus metrics from initially reporting zero-values after startup. ([\#8926](https://github.com/matrix-org/synapse/issues/8926)) - Fix recently added ratelimits to correctly honour the application service `rate_limited` flag. ([\#9711](https://github.com/matrix-org/synapse/issues/9711)) - Fix longstanding bug which caused `duplicate key value violates unique constraint "remote_media_cache_thumbnails_media_origin_media_id_thumbna_key"` errors. ([\#9725](https://github.com/matrix-org/synapse/issues/9725)) - Fix bug where sharded federation senders could get stuck repeatedly querying the DB in a loop, using lots of CPU. ([\#9770](https://github.com/matrix-org/synapse/issues/9770)) - Fix duplicate logging of exceptions thrown during federation transaction processing. ([\#9780](https://github.com/matrix-org/synapse/issues/9780)) Updates to the Docker image --------------------------- - Move opencontainers labels to the final Docker image such that users can inspect them. ([\#9765](https://github.com/matrix-org/synapse/issues/9765)) Improved Documentation ---------------------- - Make the `allowed_local_3pids` regex example in the sample config stricter. ([\#9719](https://github.com/matrix-org/synapse/issues/9719)) Deprecations and Removals ------------------------- - Remove old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9401](https://github.com/matrix-org/synapse/issues/9401)) - Make `/_matrix/client/r0/register` expect a type of `m.login.application_service` when an Application Service registers a user, to align with [the relevant spec](https://spec.matrix.org/unstable/application-service-api/#server-admin-style-permissions). ([\#9548](https://github.com/matrix-org/synapse/issues/9548)) Internal Changes ---------------- - Replace deprecated `imp` module with successor `importlib`. Contributed by Cristina Muñoz. ([\#9718](https://github.com/matrix-org/synapse/issues/9718)) - Experiment with GitHub Actions for CI. ([\#9661](https://github.com/matrix-org/synapse/issues/9661)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9682](https://github.com/matrix-org/synapse/issues/9682)) - Update `scripts-dev/complement.sh` to use a local checkout of Complement, allow running a subset of tests and have it use Synapse's Complement test blacklist. ([\#9685](https://github.com/matrix-org/synapse/issues/9685)) - Improve Jaeger tracing for `to_device` messages. ([\#9686](https://github.com/matrix-org/synapse/issues/9686)) - Add release helper script for automating part of the Synapse release process. ([\#9713](https://github.com/matrix-org/synapse/issues/9713)) - Add type hints to expiring cache. ([\#9730](https://github.com/matrix-org/synapse/issues/9730)) - Convert various testcases to `HomeserverTestCase`. ([\#9736](https://github.com/matrix-org/synapse/issues/9736)) - Start linting mypy with `no_implicit_optional`. ([\#9742](https://github.com/matrix-org/synapse/issues/9742)) - Add missing type hints to federation handler and server. ([\#9743](https://github.com/matrix-org/synapse/issues/9743)) - Check that a `ConfigError` is raised, rather than simply `Exception`, when appropriate in homeserver config file generation tests. ([\#9753](https://github.com/matrix-org/synapse/issues/9753)) - Fix incompatibility with `tox` 2.5. ([\#9769](https://github.com/matrix-org/synapse/issues/9769)) - Enable Complement tests for [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946): Spaces Summary API. ([\#9771](https://github.com/matrix-org/synapse/issues/9771)) - Use mock from the standard library instead of a separate package. ([\#9772](https://github.com/matrix-org/synapse/issues/9772)) - Update Black configuration to target Python 3.6. ([\#9781](https://github.com/matrix-org/synapse/issues/9781)) - Add option to skip unit tests when building Debian packages. ([\#9793](https://github.com/matrix-org/synapse/issues/9793))
| * Bump black configuration to target py36 (#9781)Dan Callahan2021-04-132-2/+2
| | | | | | Signed-off-by: Dan Callahan <danc@element.io>
| * Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-082-4/+6
| | | | | | | | | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
| * Record more information into structured logs. (#9654)Patrick Cloke2021-04-081-24/+88
| | | | | | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
| * Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-1/+1
| |
* | Merge commit '78e48f61b' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-4/+6
|\|
| * Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-4/+6
| |
* | Merge commit '4ecba9bd5' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-1/+1
|\|
| * Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-231-1/+1
| |
| * Allow providing credentials to HTTPS_PROXY (#9657)Andrew Morgan2021-03-222-34/+143
| | | | | | | | | | | | | | Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.
* | Merge commit '1c8a2541d' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-2/+10
|\|
| * Fix remaining mypy issues due to Twisted upgrade. (#9608)Patrick Cloke2021-03-151-2/+10
| |
* | Merge commit '55da8df07' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-13/+22
|\|
| * Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-1/+8
| |
| * Add tests for blacklisting reactor/agent. (#9563)Patrick Cloke2021-03-111-12/+14
| |
* | Merge commit '918f6ed82' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-3/+4
|\|
| * Retry 5xx errors in federation client (#9567)Erik Johnston2021-03-091-3/+4
| | | | | | Fixes #8915
* | Merge commit '7fdc6cefb' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-1/+2
|\|
| * Fix additional type hints. (#9543)Patrick Cloke2021-03-091-1/+2
| | | | | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* | Merge commit '22db45bd4' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-233-7/+9
|\|
| * Create a SynapseReactor type which incorporates the necessary reactor ↵Patrick Cloke2021-03-083-7/+9
| | | | | | | | | | interfaces. (#9528) This helps fix some type hints when running with Twisted 21.2.0.
* | Merge commit '33a02f0f5' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-234-35/+53
|\|
| * Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-034-35/+53
| |
* | Merge commit 'f00c4e7af' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-2/+29
|\|
| * (Hopefully) stop leaking file descriptors in media repo. (#9497)Patrick Cloke2021-03-011-2/+29
| | | | | | | | By consuming the response if the headers imply that the content is too large.
* | Merge commit 'e12077a78' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-232-16/+106
|\|
| * Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-262-10/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
| * SSO: redirect to public URL before setting cookies (#9436)Richard van der Hoff2021-02-261-1/+36
| | | | | | ... otherwise, we don't get the cookie back.
| * Add support for X-Forwarded-Proto (#9472)Richard van der Hoff2021-02-241-15/+70
| | | | | | | | | | rewrite XForwardedForRequest to set `isSecure()` based on `X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
* | Merge commit 'd9f1dccba' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-1/+2
|\|
* | Merge commit '8ec221710' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-14/+12
|\|
| * Reduce the memory usage of previewing media files. (#9421)Patrick Cloke2021-02-181-14/+12
| | | | | | | | | | | | | | | | This reduces the memory usage of previewing media files which end up larger than the `max_spider_size` by avoiding buffering content internally in treq. It also checks the `Content-Length` header in additional places instead of streaming the content to check the body length.
* | Merge commit 'd2f0ec12d' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-239-51/+74
|\|
| * Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-169-51/+74
| | | | | | | | | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* | Merge commit '2ab6e67ab' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-0/+7
|\|
| * Add debug logging to DNS SRV requests. (#9305)Marcus2021-02-031-0/+7
| |
* | Merge commit '9c715a5f1' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-0/+7
|\|
| * Split out a separate endpoint to complete SSO registration (#9262)Richard van der Hoff2021-02-011-0/+7
| | | | | | There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
* | Merge commit '2e537a028' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-8/+36
|\|
| * Implement MSC2858 support (#9183)Richard van der Hoff2021-01-271-8/+36
| | | | | | Fixes #8928.
* | Merge commit '0cd2938bc' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-79/+0
|\|
| * Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-79/+0
| |
* | Merge commit '2b467d0b6' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-222-2/+2
|\|
| * Properly raise an exception when the body exceeds the max size. (#9145)Patrick Cloke2021-01-182-2/+2
| | | | | | ...instead of just creating the exception object and doing nothing with it.
* | Merge commit '3e4cdfe5d' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-1/+11
|\|
| * Avoid raising the body exceeded error multiple times. (#9108)Patrick Cloke2021-01-151-1/+11
| | | | | | | | | | Previously this code generated unreferenced `Deferred` instances which caused "Unhandled Deferreds" errors to appear in error situations.
* | Merge commit '12702be95' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-1/+0
|\|
| * Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098)Tim Leung2021-01-141-1/+0
| | | | | | | | | | | | A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper used by the WellyKnownResolver. This coulld cause exceptions when attempting to connect to IP addresses that are blacklisted, but in reality this did not have any observable affect since this code is not used for IP literals.
* | Merge commit '233c8b9fc' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-5/+5
|\|
| * Reduce scope of exception handler. (#9106)Patrick Cloke2021-01-131-5/+5
| | | | | | | | Removes a bare `except Exception` clause and replaces it with catching a specific exception around the portion that might throw.
* | Merge commit '98a64b7f7' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-223-16/+27
|\|
| * Merge branch 'master' into developErik Johnston2021-01-132-3/+14
| |\
| | * Don't apply the IP range blacklist to proxy connections (#9084)Marcus2021-01-122-3/+14
| | | | | | | | | | | | | | | It is expected that the proxy would be on a private IP address so the configured proxy should be connected to regardless of the IP range blacklist.
| * | Handle bad JSON data being returned from the federation API. (#9070)Patrick Cloke2021-01-121-0/+10
| | |
| * | Remove SynapseRequest.get_user_agent (#9069)Richard van der Hoff2021-01-122-16/+17
| |/ | | | | | | | | | | | | | | | | | | | | SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests", which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the LoggingContext and write the right entries to the request log). Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a SynapseRequest when there is nothing synapse-specific about the Request at all, and any old twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult. In short: move get_user_agent out to a utility function.
* | Merge commit 'ff5c4da12' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-203-18/+52
|\|
| * Add a maximum size for well-known lookups. (#8950)Patrick Cloke2020-12-163-18/+52
| |
* | Merge commit '01333681b' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-161-2/+1
|\|
| * Various clean-ups to the logging context code (#8935)Patrick Cloke2020-12-141-2/+1
| |
* | Merge commit 'c64002e1c' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-161-3/+5
|\|
| * Fix buglet in DirectRenderJsonResource (#8897)Richard van der Hoff2020-12-101-3/+5
| | | | | | | | this was using `canonical_json` without setting it, so when you used it as a standalone class, you would get exceptions.
| * Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-023-36/+52
| | | | | | | | | | | | | | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* | Port "Allow providing credentials to HTTPS_PROXY (#9657)" from mainline (#95)Andrew Morgan2021-03-232-34/+143
| | | | | | | | | | | | | | | | | | | | | | * Allow providing credentials to HTTPS_PROXY (#9657) Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints. * lint
* | Port "Add support for no_proxy and case insensitive env variables" from ↵Andrew Morgan2021-03-224-49/+102
| | | | | | | | | | | | | | mainline to dinsic (#93) This PR is simply porting https://github.com/matrix-org/synapse/pull/9372 to dinsic. I also had to bring in https://github.com/matrix-org/synapse/pull/8821 and https://github.com/matrix-org/synapse/pull/9084 for this code to work properly - a sign that we should merge mainline into dinsic again soon.
* | Add knocking support (#81)Andrew Morgan2021-02-091-8/+59
|/ | | | | | | | | Implement knocking as defined by https://github.com/matrix-org/matrix-doc/pull/2403 This is the base knocking stuff, taken from https://github.com/matrix-org/synapse/pull/6739 and does not include any public room directory changes. While knocking hasn't merged yet on mainline due to waiting on getting Complement into Synapse's CI, the code has been well-tested.
* Allow Date header through CORS (#8804)Nicolas Chamo2020-12-011-1/+1
|
* Remove special case of pretty printing JSON responses for curl (#8833)Tulir Asokan2020-11-271-24/+5
| | | | | * Remove special case of pretty printing JSON responses for curl Signed-off-by: Tulir Asokan <tulir@maunium.net>
* Add additional type hints to HTTP client. (#8812)Patrick Cloke2020-11-252-147/+138
| | | | This also removes some duplicated code between the simple HTTP client and matrix federation client.
* Add type hints to matrix federation client / agent. (#8806)Patrick Cloke2020-11-253-194/+226
|
* Better error message when a remote resource uses invalid Content-Type (#8719)Andrew Morgan2020-11-111-2/+8
|
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-7/+23
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-281-3/+1
|\ | | | | Fix serialisation errors when using third-party event rules.
| * remove unused importsRichard van der Hoff2020-10-281-2/+0
| |
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-281-1/+1
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
| | | | | | | | | | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* | Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-235-9/+11
| |
* | Fix handling of User-Agent headers with bad utf-8. (#8632)Erik Johnston2020-10-231-7/+9
| |
* | Remove unused OPTIONS handlers. (#8621)Patrick Cloke2020-10-221-2/+1
|/ | | | The handling of OPTIONS requests was consolidated in #7534, but the endpoint specific handlers were not removed.
* Don't bother responding to client requests that have already disconnected ↵Andrew Morgan2020-10-061-0/+5
| | | | | | | | | | (#8465) This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so. This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003. Fixes https://github.com/matrix-org/synapse/issues/6700 Fixes https://github.com/matrix-org/synapse/issues/5304
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-2/+2
|
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-294-50/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* A pair of tiny cleanups in the federation request code. (#8401)Richard van der Hoff2020-09-281-2/+0
|
* Add type annotations to SimpleHttpClient (#8372)Richard van der Hoff2020-09-241-56/+131
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-142-2/+2
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-101-1/+4
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-049-18/+18
|
* Convert the well known resolver to async (#8214)Patrick Cloke2020-09-012-28/+33
|
* Switch the JSON byte producer from a pull to a push producer. (#8116)Patrick Cloke2020-08-191-32/+43
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-193-11/+10
|
* Iteratively encode JSON responses to avoid blocking the reactor. (#8013)Patrick Cloke2020-08-181-8/+89
|
* Reduce INFO logging (#8050)Erik Johnston2020-08-113-25/+73
| | | | | | | | | | c.f. #8021 A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line. Some notes: 1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps. 2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
* Don't log OPTIONS request at INFO (#8049)Erik Johnston2020-08-071-1/+7
|
* Add health check endpoint (#8048)Erik Johnston2020-08-071-1/+8
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-071-2/+3
|
* Convert run_as_background_process inner function to async. (#8032)Patrick Cloke2020-08-061-3/+2
|
* Convert the SimpleHttpClient to async. (#8016)Patrick Cloke2020-08-041-31/+24
|
* Implement handling of HTTP HEAD requests. (#7999)Patrick Cloke2020-08-031-5/+11
|
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-40/+32
|
* Ensure the msg property of HttpResponseException is a string. (#7979)Patrick Cloke2020-07-292-7/+16
|
* Return an empty body for OPTIONS requests. (#7886)Patrick Cloke2020-07-241-19/+5
|
* Downgrade warning on client disconnect to INFO (#7928)Richard van der Hoff2020-07-241-3/+1
| | | | Clients disconnecting before we finish processing the request happens from time to time. We don't need to yell about it
* Convert the federation agent and related code to async/await. (#7874)Patrick Cloke2020-07-232-16/+10
|
* fix an incorrect commentRichard van der Hoff2020-07-221-2/+2
|
* Ensure that calls to `json.dumps` are compatible with the standard library ↵Patrick Cloke2020-07-152-3/+5
| | | | json. (#7836)
* Fix client reader sharding tests (#7853)Erik Johnston2020-07-151-1/+23
| | | | | | | | | | | | | | | * Fix client reader sharding tests * Newsfile * Fix typing * Update changelog.d/7853.misc Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Move mocking of http_client to tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802)Patrick Cloke2020-07-102-19/+9
|
* Add types to the server code and remove unused parameter (#7813)Patrick Cloke2020-07-101-30/+41
|
* Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-1/+1
| | | ... instead of duplicating `config.signing_key[0]` everywhere
* Stop passing bytes when dumping JSON (#7799)Patrick Cloke2020-07-081-3/+7
|
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-032-189/+195
|
* Merge branch 'master' into developPatrick Cloke2020-07-021-8/+68
|\
| * Ensure that HTML pages served from Synapse include headers to avoid embedding.Patrick Cloke2020-07-021-8/+68
| |
* | Include a user agent in federation requests. (#7677)Patrick Cloke2020-06-163-4/+32
| |
* | Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-163-14/+10
| |
* | Create a ListenerConfig object (#7681)Richard van der Hoff2020-06-161-2/+4
|/ | | | | | | | | | This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.
* Clean up exception handling in SAML2ResponseResource (#7614)Richard van der Hoff2020-06-031-12/+31
| | | | | | | | | | | | | * Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string * Clean up exception handling in SAML2ResponseResource * use the existing code in `return_html_error` instead of re-implementing it (giving it a jinja2 template rather than inventing a new form of template) * do the exception-catching in the REST layer rather than in the handler layer, to make sure we catch all exceptions.
* Fix missing CORS headers on OPTION responses (#7560)Erik Johnston2020-05-221-1/+1
| | | Broke in #7534.
* Return 200 OK for all OPTIONS requests (#7534)Patrick Cloke2020-05-221-3/+20
|
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-0/+7
| | | | These are business as usual errors, rather than stuff we want to log at error.
* mypy for synapse.http.site (#7553)Richard van der Hoff2020-05-221-3/+6
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-6/+2
|
* Workaround for failure to wrap reason in Failure (#7473)Andrew Morgan2020-05-141-0/+7
|
* Fix b'GET' in prometheus metrics (#7503)Richard van der Hoff2020-05-141-2/+4
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-2/+4
| | | | variables (#6391)
* Implement OpenID Connect-based login (#7256)Quentin Gliech2020-05-081-0/+7
|
* Reduce federation logging on success (#7321)Michael Kaye2020-04-221-8/+14
| | | | Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
* Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181)Richard van der Hoff2020-03-311-7/+6
| | | | Fixes #7179.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-241-3/+3
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Share SSL contexts for non-federation requests (#7094)Richard van der Hoff2020-03-172-4/+1
| | | | | | | Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests. Fixes #7092.
* Tiny optimisation for _get_handler_for_request (#6950)Richard van der Hoff2020-02-191-1/+3
| | | | we have hundreds of path_regexes (see #5118), so let's not convert the same bytes to str for each of them.
* Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-1/+1
| | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* Fix outbound federation request metrics (#6795)Erik Johnston2020-01-281-0/+4
|
* Implement RedirectException (#6687)Richard van der Hoff2020-01-151-4/+9
| | | | | Allow REST endpoint implemnentations to raise a RedirectException, which will redirect the user's browser to a given location.
* Fix exceptions on requests for non-ascii urls (#6682)Richard van der Hoff2020-01-131-1/+1
| | | Fixes #6402
* Kill off redundant SynapseRequestFactory (#6619)Richard van der Hoff2020-01-031-15/+3
| | | | We already get the Site via the Channel, so there's no need for a dedicated RequestFactory: we can just use the right constructor.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
|
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-013-2/+405
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-317-14/+14
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-1/+1
|
* Merge branch 'master' into developAndrew Morgan2019-10-031-2/+4
|\
| * Replace client_secret with <redacted> in server logs (#6158)Andrew Morgan2019-10-031-2/+4
| | | | | | Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
* | Edit SimpleHttpClient to reference that header keys can be passed as str or ↵Andrew Morgan2019-09-271-6/+6
|/ | | | bytes (#6077)
* Fix well-known lookups with the federation certificate whitelist (#5997)Amber Brown2019-09-141-1/+1
|
* Trace how long it takes for the send trasaction to complete, including ↵Jorik Schellekens2019-09-052-37/+51
| | | | retrys (#5986)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-052-6/+13
|
* Remove unused methods from c/s api v1 in register.py (#5963)Andrew Morgan2019-09-021-33/+1
| | | These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Merge pull request #5864 from matrix-org/erikj/reliable_lookupsErik Johnston2019-08-272-208/+217
|\ | | | | Refactor MatrixFederationAgent to retry SRV.
| * Fix off by one error in SRV result shufflingErik Johnston2019-08-271-8/+13
| |
| * Fixup commentsErik Johnston2019-08-231-1/+9
| |
| * Fix logcontextsErik Johnston2019-08-231-4/+8
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-08-223-47/+128
| |\ | | | | | | | | | erikj/reliable_lookups
| * | Remove now unused pick_server_from_listErik Johnston2019-08-201-30/+0
| | |
| * | Fixup _sort_server_list to be slightly more efficientErik Johnston2019-08-201-2/+11
| | | | | | | | | | | | | | | Also document that we are using the algorithm described in RFC2782 and ensure we handle zero weight correctly.
| * | Refactor MatrixFederationAgent to retry SRV.Erik Johnston2019-08-152-189/+202
| | | | | | | | | | | | | | | | | | | | | | | | This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool
* | | Opentracing across workers (#5771)Jorik Schellekens2019-08-221-1/+1
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* | Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_failErik Johnston2019-08-222-37/+115
|\ \ | | | | | | Retry well known on fail
| * | Change jitter to be a factor rather than absolute valueErik Johnston2019-08-201-11/+12
| | |
| * | Fixup changelog and remove debug loggingErik Johnston2019-08-161-4/+1
| | |
| * | Retry well known on fail.Erik Johnston2019-08-152-34/+114
| |/ | | | | | | | | | | If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.
* / Opentracing misc (#5856)Jorik Schellekens2019-08-161-10/+13
|/ | | | | | | | | | | Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-22/+60
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Make default well known cache global again.Erik Johnston2019-08-131-1/+4
|
* Move well known lookup into a separate clasErik Johnston2019-08-072-153/+197
|
* Add a lower bound for TTL on well known results.Erik Johnston2019-08-061-0/+4
| | | | | | | It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".
* Handle incorrectly encoded query params correctlyErik Johnston2019-08-011-1/+6
|
* Share SSL options for well-known requestsErik Johnston2019-07-311-11/+5
|
* Merge tag 'v1.2.0rc2' into developAndrew Morgan2019-07-242-14/+31
|\ | | | | | | | | | | | | Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
| * Fix servlet metric names (#5734)Jorik Schellekens2019-07-242-14/+31
| | | | | | | | | | | | | | | | | | | | * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths
* | Replace returnValue with return (#5736)Amber Brown2019-07-234-53/+45
|/
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-112-5/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-047-7/+7
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-10/+16
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-291-36/+41
|
* Run Black. (#5482)Amber Brown2019-06-2010-287/+321
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-4/+9
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Improve docstrings on MatrixFederationClient. (#5332)Richard van der Hoff2019-06-041-16/+55
|
* Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) v0.99.5.1.dev0Richard van der Hoff2019-05-291-4/+0
| | | This is just unhelpful spam
* Fix error code for invalid parameterAaron Raimist2019-05-241-1/+1
| | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-132-14/+40
|
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-20/+25
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Config option for verifying federation certificates (MSC 1711) (#4967)Andrew Morgan2019-04-251-1/+1
|
* Make federation endpoints more tolerant of trailing slashes v2 (#4935)Andrew Morgan2019-03-261-0/+1
| | | Redo of https://github.com/matrix-org/synapse/pull/4840
* Clean up backoff_on_404 and metehod callsAndrew Morgan2019-03-211-9/+13
|
* kwargs doesn't like commas on calling funcs either. TILAndrew Morgan2019-03-201-2/+2
|
* lintAndrew Morgan2019-03-201-2/+6
|
* New test, fix issuesAndrew Morgan2019-03-201-49/+28
|
* Fix comments. v0.99.2 -> v0.99.3Andrew Morgan2019-03-201-5/+5
|
* Just return if not doing any trailing slash shennanigansAndrew Morgan2019-03-201-0/+2
|
* lintAndrew Morgan2019-03-181-1/+1
|
* Better exception handlingAndrew Morgan2019-03-181-16/+17
|
* Correct var nameAndrew Morgan2019-03-131-2/+2
|
* receiving a 400 caused an exception. handle itAndrew Morgan2019-03-131-5/+12
|
* is this what purgatory feels likeAndrew Morgan2019-03-131-3/+3
|
* i should have given up x2Andrew Morgan2019-03-131-18/+13
|
* i should have given upAndrew Morgan2019-03-131-1/+1
|
* there comes a time when you should give up. but you dontAndrew Morgan2019-03-131-1/+1
|
* as aboveAndrew Morgan2019-03-131-2/+4
|
* no kwargs todayAndrew Morgan2019-03-131-7/+12
|
* Or perhaps I was the one who was drunkAndrew Morgan2019-03-131-6/+3
|
* go home python, you're drunkAndrew Morgan2019-03-131-4/+2
|
* Syntax checker is borkAndrew Morgan2019-03-131-6/+5
|
* Destructure againAndrew Morgan2019-03-131-7/+8
|
* Are you happy nowAndrew Morgan2019-03-131-2/+2
|
* Syntax testAndrew Morgan2019-03-131-6/+5
|
* Remove testing codeAndrew Morgan2019-03-131-3/+0
|
* Add missing docstring detailAndrew Morgan2019-03-131-1/+1
|
* Fix syntax issuesAndrew Morgan2019-03-131-7/+10
|
* Switch to wrapper function around _send_requestAndrew Morgan2019-03-131-32/+71
|
* Retry on 400:M_UNRECOGNIZEDAndrew Morgan2019-03-131-6/+10
|
* Fix paranthesis indentAndrew Morgan2019-03-131-1/+2
|
* Add workaround noteAndrew Morgan2019-03-131-10/+4
|
* Cleaner way of implementing trailing slashesAndrew Morgan2019-03-121-63/+52
|
* Retry certain federation requests on 404Andrew Morgan2019-03-081-0/+45
|
* Minor docstring fixes for MatrixFederationAgent (#4765)Andrew Morgan2019-02-281-1/+5
|
* Log tracebacks correctlyErik Johnston2019-02-251-4/+4
|
* Fix TaskStopped exceptions when outbound requests time out (#4690)Richard van der Hoff2019-02-203-18/+29
|
* Cleanup top level request exception loggingErik Johnston2019-02-181-3/+3
| | | | | | | | | | | Firstly, we always logged that the request was being handled via `JsonResource._async_render`, so we change that to use the servlet name we add to the request. Secondly, we pass the exception information to the logger rather than formatting it manually. This makes it consistent with other exception logging, allwoing logging hooks and formatters to access the exception information.
* Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)Richard van der Hoff2019-02-111-2/+2
| | | | | It's nothing to do with refreshing the certificates. No idea why it was here.
* Merge pull request #4546 from ↵Richard van der Hoff2019-02-011-2/+15
|\ | | | | | | | | matrix-org/rav/silence_critical_error_from_federation Fix noisy "twisted.internet.task.TaskStopped" errors in logs
| * Fix noisy "twisted.internet.task.TaskStopped" errors in logsRichard van der Hoff2019-02-011-2/+15
| | | | | | | | Fixes #4003
* | Merge pull request #4544 from matrix-org/rav/skip_invalid_well_knownRichard van der Hoff2019-02-011-19/+6
|\ \ | | | | | | Treat an invalid .well-known the same as an absent one
| * | Treat an invalid .well-known the same as an absent oneRichard van der Hoff2019-02-011-19/+6
| | | | | | | | | | | | ... basically, carry on and fall back to SRV etc.
* | | Fix b'ab' noise in logsRichard van der Hoff2019-02-011-1/+1
|/ /
* | Cache failures to parse .well-knownRichard van der Hoff2019-02-011-13/+43
| | | | | | | | Also add a Measure block around the .well-known fetch