summary refs log tree commit diff
path: root/synapse/http (follow)
Commit message (Collapse)AuthorAgeFilesLines
* A second batch of Pydantic models for rest/client/account.py (#13687)David Robertson2022-09-071-2/+17
|
* Generalise the `@cancellable` annotation so it can be used on functions ↵reivilibre2022-08-311-65/+3
| | | | other than just servlet methods. (#13662)
* `synapse.api.auth.Auth` cleanup: make permission-related methods use ↵Quentin Gliech2022-08-221-1/+1
| | | | | | | | | `Requester` instead of the `UserID` (#13024) Part of #13019 This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it. It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
* Use Pydantic to systematically validate a first batch of endpoints in ↵David Robertson2022-08-151-0/+25
| | | | `synapse.rest.client.account`. (#13188)
* Implement MSC3848: Introduce errcodes for specific event sending failures ↵Will Hunt2022-07-271-4/+14
| | | | | (#13343) Implements MSC3848
* Validate federation destinations and log an error if server name is invalid. ↵Shay2022-07-201-0/+9
| | | | (#13318)
* Add Cross-Origin-Resource-Policy header to thumbnail and download media ↵Robert Long2022-06-271-0/+11
| | | | endpoints (#12944)
* Fix `destination_is` errors seen in sentry. (#13041)David Robertson2022-06-141-2/+5
| | | | | | * Rename test_fedclient to match its source file * Require at least one destination to be truthy * Explicitly validate user ID in profile endpoint GETs Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Merge tag 'v1.60.0rc2' into developSean Quah2022-05-271-0/+11
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.60.0rc2 (2022-05-27) ============================== This release of Synapse adds a unique index to the `state_group_edges` table, in order to prevent accidentally introducing duplicate information (for example, because a database backup was restored multiple times). If your Synapse database already has duplicate rows in this table, this could fail with an error and require manual remediation. Additionally, the signature of the `check_event_for_spam` module callback has changed. The previous signature has been deprecated and remains working for now. Module authors should update their modules to use the new signature where possible. See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1600) for more details. Features -------- - Add an option allowing users to use their password to reauthenticate for privileged actions even though password login is disabled. ([\#12883](https://github.com/matrix-org/synapse/issues/12883)) Bugfixes -------- - Explicitly close `ijson` coroutines once we are done with them, instead of leaving the garbage collector to close them. ([\#12875](https://github.com/matrix-org/synapse/issues/12875)) Internal Changes ---------------- - Improve URL previews by not including the content of media tags in the generated description. ([\#12887](https://github.com/matrix-org/synapse/issues/12887))
| * Close `ijson` coroutines ourselves instead of letting the GC close them (#12875)Sean Quah2022-05-271-0/+11
| | | | | | | | | | | | | | Hopefully this means that exceptions raised due to truncated JSON get a sensible logging context and stack. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Allow bigger responses to `/federation/v1/state` (#12877)Richard van der Hoff2022-05-251-22/+7
|/ | | | | | | | | | | * Refactor HTTP response size limits Rather than passing a separate `max_response_size` down the stack, make it an attribute of the `parser`. * Allow bigger responses on `federation/v1/state` `/state` can return huge responses, so we need to handle that.
* Make handling of federation Authorization header (more) compliant with ↵Hubert Chathi2022-05-181-1/+1
| | | | | | | | | | | | RFC7230 (#12774) The main differences are: - values with delimiters (such as colons) should be quoted, so always quote the origin, since it could contain a colon followed by a port number - should allow more than one space after "X-Matrix" - quoted values with backslash-escaped characters should be unescaped - names should be case insensitive
* Another batch of type annotations (#12726)David Robertson2022-05-132-17/+24
|
* Reduce the number of "untyped defs" (#12716)David Robertson2022-05-126-29/+40
|
* Enable cancellation of `GET /members` and `GET /state` requests (#12708)Sean Quah2022-05-111-1/+3
| | | | | | | | Enable cancellation of `GET /rooms/$room_id/members`, `GET /rooms/$room_id/state` and `GET /rooms/$room_id/state/$state_key/*` requests. Signed-off-by: Sean Quah <seanq@element.io>
* Reload cache factors from disk on SIGHUP (#12673)David Robertson2022-05-111-1/+1
|
* Respect the `@cancellable` flag for `RestServlet`s and ↵Sean Quah2022-05-111-0/+5
| | | | | | | | | | | | | | | `BaseFederationServlet`s (#12699) Both `RestServlet`s and `BaseFederationServlet`s register their handlers with `HttpServer.register_paths` / `JsonResource.register_paths`. Update `JsonResource` to respect the `@cancellable` flag on handlers registered in this way. Although `ReplicationEndpoint` also registers itself using `register_paths`, it does not pass the handler method that would have the `@cancellable` flag directly, and so needs separate handling. Signed-off-by: Sean Quah <seanq@element.io>
* Respect the `@cancellable` flag for `DirectServe{Html,Json}Resource`s (#12698)Sean Quah2022-05-111-0/+2
| | | | | | | | | | | | `DirectServeHtmlResource` and `DirectServeJsonResource` both inherit from `_AsyncResource`. These classes expect to be subclassed with `_async_render_*` methods. This commit has no effect on `JsonResource`, despite inheriting from `_AsyncResource`. `JsonResource` has its own `_async_render` override which will need to be updated separately. Signed-off-by: Sean Quah <seanq@element.io>
* Capture the `Deferred` for request cancellation in `_AsyncResource` (#12694)Sean Quah2022-05-102-5/+8
| | | | | | | | | | | | | All async request processing goes through `_AsyncResource`, so this is the only place where a `Deferred` needs to be captured for cancellation. Unfortunately, the same isn't true for determining whether a request can be cancelled. Each of `RestServlet`, `BaseFederationServlet`, `DirectServe{Html,Json}Resource` and `ReplicationEndpoint` have different wrappers around the method doing the request handling and they all need to be handled separately. Signed-off-by: Sean Quah <seanq@element.io>
* Add ability to cancel disconnected requests to `SynapseRequest` (#12588)Sean Quah2022-05-101-1/+23
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Add `@cancellable` decorator, for use on request handlers (#12586)Sean Quah2022-05-101-0/+61
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Immediately retry any requests that have backed off when a server comes back ↵Erik Johnston2022-05-101-2/+13
| | | | | online. (#12500) Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
* Use `getClientAddress` instead of `getClientIP`. (#12599)Patrick Cloke2022-05-041-3/+3
| | | | | getClientIP was deprecated in Twisted 18.4.0, which also added getClientAddress. The Synapse minimum version for Twisted is currently 18.9.0, so all supported versions have the new API.
* Improve logging for cancelled requests (#12587)Sean Quah2022-05-041-0/+30
| | | | | | | Don't log stack traces for cancelled requests and use a custom HTTP status code of 499. Signed-off-by: Sean Quah <seanq@element.io>
* Fix logging of incorrect status codes for disconnected requests (#12580)Sean Quah2022-04-281-3/+9
| | | | | | | | | | | | | | The status code of requests must always be set, regardless of client disconnection, otherwise they will always be logged as 200!. Broken for `respond_with_json` in f48792eec43f893f4f893ffdcbf00f8958b6f6b5. Broken for `respond_with_json_bytes` in 3e58ce72b42f2ae473c1e76a967548cd6fa7e2e6. Broken for `respond_with_html_bytes` in ea26e9a98b0541fc886a1cb826a38352b7599dbe. Signed-off-by: Sean Quah <seanq@element.io>
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-2/+2
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Implement MSC3383: include destination in X-Matrix auth header (#11398)Jan Christian Grünhage2022-04-191-2/+10
| | | | Co-authored-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz> Co-authored-by: Marcus Hoffmann <bubu@bubu1.eu>
* Unify HTTP query parameter type hints (#12415)David Robertson2022-04-083-35/+37
| | | | | | * Pull out query param types to `synapse.http.types` * Use QueryParams everywhere * Simplify `encode_query_args` * Add annotation which would have caught #12410
* Bump `black` and `click` versions (#12320)David Robertson2022-03-291-1/+1
|
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-112-3/+0
|
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Faster joins: Support for calling `/federation/v1/state` (#12013)Richard van der Hoff2022-02-221-1/+49
| | | | This is an endpoint that we have server-side support for, but no client-side support. It's going to be useful for resyncing partial-stated rooms, so let's introduce it.
* Adds misc missing type hints (#11953)Patrick Cloke2022-02-112-8/+6
|
* Implement a content type allow list for URL previews (#11936)Denis Kasak2022-02-101-0/+18
| | | | | | | This implements an allow list for content types for which Synapse will attempt URL preview. If a URL resolves to a resource with a content type which isn't in the list, the download will terminate immediately. This makes sense given that Synapse would never successfully generate a URL preview for such files in the first place, and helps prevent issues with streaming media servers, such as #8302. Signed-off-by: Denis Kasak dkasak@termina.org.uk
* Don't print HTTPStatus.* in "Processed..." logs (#11827)David Robertson2022-01-261-1/+4
| | | | | | | | | * Don't print HTTPStatus.* in "Processed..." logs Fixes #11812. See also #7118 and https://github.com/matrix-org/synapse/pull/7188#r401719326 in particular. Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Apply a timeout to reading the body when fetching a file. (#11784)Patrick Cloke2022-01-241-3/+12
| | | | This prevents the URL preview code from reading a stream forever.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-133-13/+13
|
* Fix SimpleHttpClient not sending Accept header in `get_json` (#11677)Fr3shTea2022-01-051-1/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-4/+6
| | | To improve type hints throughout the code.
* Various opentracing enhancements (#11619)Richard van der Hoff2021-12-211-2/+17
| | | | | | | | | | | | | | | | | | | | | | | | | * Wrap `auth.get_user_by_req` in an opentracing span give `get_user_by_req` its own opentracing span, since it can result in a non-trivial number of sub-spans which it is useful to group together. This requires a bit of reorganisation because it also sets some tags (and may force tracing) on the servlet span. * Emit opentracing span for encoding json responses This can be a significant time sink. * Rename all sync spans with a prefix * Write an opentracing span for encoding sync response * opentracing span to group generate_room_entries * opentracing spans within sync.encode_response * changelog * Use the `trace` decorator instead of context managers
* Improve opentracing for incoming HTTP requests (#11618)Richard van der Hoff2021-12-201-1/+29
| | | | | | | | | | | | | | | | | | | | | | * remove `start_active_span_from_request` Instead, pull out a separate function, `span_context_from_request`, to extract the parent span, which we can then pass into `start_active_span` as normal. This seems to be clearer all round. * Remove redundant tags from `incoming-federation-request` These are all wrapped up inside a parent span generated in AsyncResource, so there's no point duplicating all the tags that are set there. * Leave request spans open until the request completes It may take some time for the response to be encoded into JSON, and that JSON to be streamed back to the client, and really we want that inside the top-level span, so let's hand responsibility for closure to the SynapseRequest. * opentracing logs for HTTP request events * changelog
* Return JSON errors for unknown resources under /matrix/client. (#11602)Patrick Cloke2021-12-201-3/+3
| | | | Instead of returning 404 errors with HTML bodies when an unknown prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
* Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-141-2/+5
|
* Add missing type hints to synapse.http. (#11571)Patrick Cloke2021-12-145-49/+70
|
* Use HTTPStatus constants in place of literals in `synapse.http` (#11543)Dirk Klimpel2021-12-093-19/+46
|
* Add missing `errcode` to `parse_string` and `parse_boolean` (#11542)Dirk Klimpel2021-12-091-2/+2
|
* Add MSC3030 experimental client and federation API endpoints to get the ↵Eric Eastwood2021-12-021-0/+29
| | | | | | | | | | | | | | | | | | | | | | | | | closest event to a given timestamp (#9445) MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030 Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about. ``` GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction> { "event_id": ... "origin_server_ts": ... } ``` Federation API endpoint: ``` GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction> { "event_id": ... "origin_server_ts": ... } ``` Co-authored-by: Erik Johnston <erik@matrix.org>
* Add more type hints to synapse.util. (#11321)Patrick Cloke2021-11-121-3/+3
|
* Type hints for the remaining two files in `synapse.http`. (#11164)David Robertson2021-10-282-20/+42
| | | | | | | | | | | | | | | | | | | * Teach MyPy that the sentinel context is False This means that if `ctx: LoggingContextOrSentinel` then `bool(ctx)` narrows us to `ctx:LoggingContext`, which is a really neat find! * Annotate RequestMetrics - Raise errors for sentry if we use the sentinel context - Ensure we don't raise an error and carry on, but not recording stats - Include stack trace in the error case to lower Sean's blood pressure * Make mypy pass for synapse.http.request_metrics * Make synapse.http.connectproxyclient pass mypy Co-authored-by: reivilibre <oliverw@matrix.org>
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-222-9/+18
|
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-081-1/+1
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Catch AttributeErrors when calling registerProducer (#10995)Brendan Abolivier2021-10-071-1/+4
| | | Looks like the wrong exception type was caught in #10932.
* Merge tag 'v1.44.0rc1' into developOlivier Wilkinson (reivilibre)2021-09-291-3/+11
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc1 (2021-09-29) ============================== Features -------- - Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event. ([\#10776](https://github.com/matrix-org/synapse/issues/10776)) - Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814), [\#10819](https://github.com/matrix-org/synapse/issues/10819)) - Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868), [\#10905](https://github.com/matrix-org/synapse/issues/10905)) - Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes. ([\#10898](https://github.com/matrix-org/synapse/issues/10898)) Bugfixes -------- - Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690)) - Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782)) - Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807)) - Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810)) - Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters. ([\#10827](https://github.com/matrix-org/synapse/issues/10827)) - Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843)) - Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859)) - Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database. ([\#10873](https://github.com/matrix-org/synapse/issues/10873)) - Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper. ([\#10875](https://github.com/matrix-org/synapse/issues/10875)) - Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked. ([\#10881](https://github.com/matrix-org/synapse/issues/10881)) - Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907)) - Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911)) - Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913)) Improved Documentation ---------------------- - Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845)) - Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865)) - Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869)) Internal Changes ---------------- - Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659)) - Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777)) - Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785), [\#10817](https://github.com/matrix-org/synapse/issues/10817)) - Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796)) - Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812), [\#10885](https://github.com/matrix-org/synapse/issues/10885), [\#10893](https://github.com/matrix-org/synapse/issues/10893), [\#10897](https://github.com/matrix-org/synapse/issues/10897)) - Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815)) - Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816)) - Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820)) - Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823)) - Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826)) - Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829)) - Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831), [\#10856](https://github.com/matrix-org/synapse/issues/10856)) - Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833)) - Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834)) - Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835)) - Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint. ([\#10838](https://github.com/matrix-org/synapse/issues/10838)) - Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`. ([\#10839](https://github.com/matrix-org/synapse/issues/10839)) - Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867)) - Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879)) - Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829). ([\#10880](https://github.com/matrix-org/synapse/issues/10880)) - Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883), [\#10884](https://github.com/matrix-org/synapse/issues/10884), [\#10896](https://github.com/matrix-org/synapse/issues/10896), [\#10901](https://github.com/matrix-org/synapse/issues/10901)) - Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231). ([\#10887](https://github.com/matrix-org/synapse/issues/10887)) - Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889)) - Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891)) - Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906)) - Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917)) - Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925)) - Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931)) F124520CEEE062448FE1C8442D2EFA2F32FBE047 <olivier@librepush.net>" [ultimate]
| * Fix exception responding to request that has been closed (#10932)Erik Johnston2021-09-281-3/+11
| | | | | | Introduced in #10905
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-5/+5
|/
* Encode JSON responses on a thread in C, mk2 (#10905)Erik Johnston2021-09-281-15/+57
| | | | | | | | | | Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library. Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this: 1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or 2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types. I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
* Add reactor to `SynapseRequest` and fix up types. (#10868)Erik Johnston2021-09-242-15/+26
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-2/+3
|
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-231-1/+1
|
* Add types to http.site (#10867)Erik Johnston2021-09-211-18/+22
|
* Use direct references for some configuration variables (part 2) (#10812)Patrick Cloke2021-09-151-2/+5
|
* Prevent logging context going missing on federation request timeout (#10810)Sean Quah2021-09-141-4/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | In `MatrixFederationHttpClient._send_request()`, we make a HTTP request using an `Agent`, wrap that request in a timeout and await the resulting `Deferred`. On its own, the `Agent` performing the HTTP request correctly stashes and restores the logging context while waiting. The addition of the timeout introduces a path where the logging context is not restored when execution resumes. To address this, we wrap the timeout `Deferred` in a `make_deferred_yieldable()` to stash the logging context and restore it on completion of the `await`. However this is not sufficient, since by the time we construct the timeout `Deferred`, the `Agent` has already stashed and cleared the logging context when using `make_deferred_yieldable()` to produce its `Deferred` for the request. Hence, we wrap the `Agent` request in a `run_in_background()` to "fork" and preserve the logging context so that we can stash and restore it when `await`ing the timeout `Deferred`. This approach is similar to the one used with `defer.gatherResults`. Note that the code is still not fully correct. When a timeout occurs, the request remains running in the background (existing behavior which is nothing to do with the new call to `run_in_background`) and may re-start the logging context after it has finished.
* Change logging of puppeted requests to better differentiate users (#10779)Erik Johnston2021-09-081-1/+1
| | | This used to be a comma and got accidentally changed to a period in #9654, but a pipe character is more easier to parse visually.
* Additional type hints for client REST servlets (part 5) (#10736)Patrick Cloke2021-09-031-0/+19
| | | | Additionally this enforce type hints on all function signatures inside of the synapse.rest.client package.
* Additional type hints for the proxy agent and SRV resolver modules. (#10608)Dirk Klimpel2021-08-183-25/+37
|
* Experimental support for MSC3266 Room Summary API. (#10394)Michael Telatynski2021-08-161-2/+56
|
* support federation queries through http connect proxy (#10475)Dirk Klimpel2021-08-114-79/+152
| | | | | Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu> Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Fix up type hints for Twisted 21.7 (#10490)Richard van der Hoff2021-07-281-2/+2
| | | Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
* allow specifying https:// proxy (#10411)Dirk Klimpel2021-07-271-75/+109
|
* Fix a handful of type annotations. (#10446)Richard van der Hoff2021-07-221-2/+2
| | | | | | | | | | * switch from `types.CoroutineType` to `typing.Coroutine` these should be identical semantically, and since `defer.ensureDeferred` is defined to take a `typing.Coroutine`, will keep mypy happy * Fix some annotations on inlineCallbacks functions * changelog
* Add type hints to additional servlet functions (#10437)Patrick Cloke2021-07-211-51/+169
| | | | | | | | | Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.
* Add a return type to parse_string. (#10438)Patrick Cloke2021-07-211-1/+37
| | | | And set the required attribute in a few places which will error if a parameter is not provided.
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-192-2/+2
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-151-7/+6
|
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-156-28/+28
|
* Allow providing credentials to `http_proxy` (#10360)Dirk Klimpel2021-07-151-1/+11
|
* Fix a number of logged errors caused by remote servers being down. (#10400)Erik Johnston2021-07-151-0/+28
|
* Add additional types to the federation transport server. (#10213)Patrick Cloke2021-06-281-4/+46
|
* Drop Origin & Accept from Access-Control-Allow-Headers value (#10114)Michael[tm] Smith2021-06-231-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Drop Origin & Accept from Access-Control-Allow-Headers value This change drops the Origin and Accept header names from the value of the Access-Control-Allow-Headers response header sent by Synapse. Per the CORS protocol, it’s not necessary or useful to include those header names. Details: Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin is a “forbidden header name” set by the browser and that frontend JavaScript code is never allowed to set. So the value of Access-Control-Allow-Headers isn’t relevant to Origin or in general to other headers set by the browser itself — the browser never ever consults the Access-Control-Allow-Headers value to confirm that it’s OK for the request to include an Origin header. And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header, Accept is a “CORS-safelisted request-header”, which means that browsers allow requests to contain the Accept header regardless of whether the Access-Control-Allow-Headers value contains "Accept". So it’s unnecessary for the Access-Control-Allow-Headers to explicitly include Accept. Browsers will not perform a CORS preflight for requests containing an Accept request header. Related: https://github.com/matrix-org/matrix-doc/pull/3225 Signed-off-by: Michael[tm] Smith <mike@w3.org>
* update black to 21.6b0 (#10197)Marcus2021-06-171-1/+1
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* Fix 'ip_range_whitelist' not working for federation servers (#10115)Michael Kutzner2021-06-151-1/+3
| | | | | Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers. Signed-off-by: Michael Kutzner 1mikure@gmail.com
* Implement knock feature (#6739)Sorunome2021-06-091-1/+0
| | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* Clean up the interface for injecting opentracing over HTTP (#10143)Richard van der Hoff2021-06-091-7/+3
| | | | | | | * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog
* Add type hints to the federation server transport. (#10080)Patrick Cloke2021-06-081-0/+24
|
* Correct type hints for parse_string(s)_from_args. (#10137)Patrick Cloke2021-06-081-68/+111
|
* Merge branch 'master' into developAndrew Morgan2021-06-011-1/+13
|\
| * Log method and path when dropping request due to size limit (#10091)Erik Johnston2021-05-281-1/+3
| |
| * Allow response of `/send_join` to be larger. (#10093)Erik Johnston2021-05-281-1/+13
| | | | | | Fixes #10087.
* | Log method and path when dropping request due to size limit (#10091)Erik Johnston2021-05-281-1/+3
| |
* | Add `parse_strings_from_args` to get `prev_events` array (#10048)Eric Eastwood2021-05-281-43/+153
|/ | | | | | | | | | | | | | | | | | | | | | Split out from https://github.com/matrix-org/synapse/pull/9247 Strings: - `parse_string` - `parse_string_from_args` - `parse_strings_from_args` For comparison with ints: - `parse_integer` - `parse_integer_from_args` Previous discussions: - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791 Signed-off-by: Eric Eastwood <erice@element.io>
* Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958)Erik Johnston2021-05-202-36/+131
| | | Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
* Improved validation for received requests (#9817)Richard van der Hoff2021-04-231-5/+27
| | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-231-9/+28
|
* Limit the size of HTTP responses read over federation. (#9833)Richard van der Hoff2021-04-232-8/+50
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1413-13/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-132-2/+2
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-082-4/+6
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Record more information into structured logs. (#9654)Patrick Cloke2021-04-081-24/+88
| | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
* Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-1/+1
|
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-4/+6
|
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-231-1/+1
|
* Allow providing credentials to HTTPS_PROXY (#9657)Andrew Morgan2021-03-222-34/+143
| | | | | | | Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.
* Fix remaining mypy issues due to Twisted upgrade. (#9608)Patrick Cloke2021-03-151-2/+10
|
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-1/+8
|
* Add tests for blacklisting reactor/agent. (#9563)Patrick Cloke2021-03-111-12/+14
|
* Retry 5xx errors in federation client (#9567)Erik Johnston2021-03-091-3/+4
| | | Fixes #8915
* Fix additional type hints. (#9543)Patrick Cloke2021-03-091-1/+2
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Create a SynapseReactor type which incorporates the necessary reactor ↵Patrick Cloke2021-03-083-7/+9
| | | | | interfaces. (#9528) This helps fix some type hints when running with Twisted 21.2.0.
* Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-034-35/+53
|
* (Hopefully) stop leaking file descriptors in media repo. (#9497)Patrick Cloke2021-03-011-2/+29
| | | | By consuming the response if the headers imply that the content is too large.
* Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-262-10/+37
| | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
* SSO: redirect to public URL before setting cookies (#9436)Richard van der Hoff2021-02-261-1/+36
| | | ... otherwise, we don't get the cookie back.
* Add support for X-Forwarded-Proto (#9472)Richard van der Hoff2021-02-241-15/+70
| | | | | rewrite XForwardedForRequest to set `isSecure()` based on `X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
* Reduce the memory usage of previewing media files. (#9421)Patrick Cloke2021-02-181-14/+12
| | | | | | | | This reduces the memory usage of previewing media files which end up larger than the `max_spider_size` by avoiding buffering content internally in treq. It also checks the `Content-Length` header in additional places instead of streaming the content to check the body length.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-169-51/+74
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add debug logging to DNS SRV requests. (#9305)Marcus2021-02-031-0/+7
|
* Split out a separate endpoint to complete SSO registration (#9262)Richard van der Hoff2021-02-011-0/+7
| | | There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
* Implement MSC2858 support (#9183)Richard van der Hoff2021-01-271-8/+36
| | | Fixes #8928.
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-79/+0
|
* Properly raise an exception when the body exceeds the max size. (#9145)Patrick Cloke2021-01-182-2/+2
| | | ...instead of just creating the exception object and doing nothing with it.
* Avoid raising the body exceeded error multiple times. (#9108)Patrick Cloke2021-01-151-1/+11
| | | | | Previously this code generated unreferenced `Deferred` instances which caused "Unhandled Deferreds" errors to appear in error situations.
* Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098)Tim Leung2021-01-141-1/+0
| | | | | | A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper used by the WellyKnownResolver. This coulld cause exceptions when attempting to connect to IP addresses that are blacklisted, but in reality this did not have any observable affect since this code is not used for IP literals.
* Reduce scope of exception handler. (#9106)Patrick Cloke2021-01-131-5/+5
| | | | Removes a bare `except Exception` clause and replaces it with catching a specific exception around the portion that might throw.
* Merge branch 'master' into developErik Johnston2021-01-132-3/+14
|\
| * Don't apply the IP range blacklist to proxy connections (#9084)Marcus2021-01-122-3/+14
| | | | | | | | | | It is expected that the proxy would be on a private IP address so the configured proxy should be connected to regardless of the IP range blacklist.
* | Handle bad JSON data being returned from the federation API. (#9070)Patrick Cloke2021-01-121-0/+10
| |
* | Remove SynapseRequest.get_user_agent (#9069)Richard van der Hoff2021-01-122-16/+17
|/ | | | | | | | | | | SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests", which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the LoggingContext and write the right entries to the request log). Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a SynapseRequest when there is nothing synapse-specific about the Request at all, and any old twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult. In short: move get_user_agent out to a utility function.
* Add a maximum size for well-known lookups. (#8950)Patrick Cloke2020-12-163-18/+52
|
* Various clean-ups to the logging context code (#8935)Patrick Cloke2020-12-141-2/+1
|
* Fix buglet in DirectRenderJsonResource (#8897)Richard van der Hoff2020-12-101-3/+5
| | | | this was using `canonical_json` without setting it, so when you used it as a standalone class, you would get exceptions.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-023-36/+52
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Allow Date header through CORS (#8804)Nicolas Chamo2020-12-011-1/+1
|
* Remove special case of pretty printing JSON responses for curl (#8833)Tulir Asokan2020-11-271-24/+5
| | | | | * Remove special case of pretty printing JSON responses for curl Signed-off-by: Tulir Asokan <tulir@maunium.net>
* Add additional type hints to HTTP client. (#8812)Patrick Cloke2020-11-252-147/+138
| | | | This also removes some duplicated code between the simple HTTP client and matrix federation client.
* Add type hints to matrix federation client / agent. (#8806)Patrick Cloke2020-11-253-194/+226
|
* Better error message when a remote resource uses invalid Content-Type (#8719)Andrew Morgan2020-11-111-2/+8
|
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-7/+23
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-281-3/+1
|\ | | | | Fix serialisation errors when using third-party event rules.
| * remove unused importsRichard van der Hoff2020-10-281-2/+0
| |
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-281-1/+1
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
| | | | | | | | | | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* | Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-235-9/+11
| |
* | Fix handling of User-Agent headers with bad utf-8. (#8632)Erik Johnston2020-10-231-7/+9
| |
* | Remove unused OPTIONS handlers. (#8621)Patrick Cloke2020-10-221-2/+1
|/ | | | The handling of OPTIONS requests was consolidated in #7534, but the endpoint specific handlers were not removed.
* Don't bother responding to client requests that have already disconnected ↵Andrew Morgan2020-10-061-0/+5
| | | | | | | | | | (#8465) This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so. This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003. Fixes https://github.com/matrix-org/synapse/issues/6700 Fixes https://github.com/matrix-org/synapse/issues/5304
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-2/+2
|
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-294-50/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* A pair of tiny cleanups in the federation request code. (#8401)Richard van der Hoff2020-09-281-2/+0
|
* Add type annotations to SimpleHttpClient (#8372)Richard van der Hoff2020-09-241-56/+131
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-142-2/+2
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-101-1/+4
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-049-18/+18
|
* Convert the well known resolver to async (#8214)Patrick Cloke2020-09-012-28/+33
|
* Switch the JSON byte producer from a pull to a push producer. (#8116)Patrick Cloke2020-08-191-32/+43
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-193-11/+10
|
* Iteratively encode JSON responses to avoid blocking the reactor. (#8013)Patrick Cloke2020-08-181-8/+89
|
* Reduce INFO logging (#8050)Erik Johnston2020-08-113-25/+73
| | | | | | | | | | c.f. #8021 A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line. Some notes: 1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps. 2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
* Don't log OPTIONS request at INFO (#8049)Erik Johnston2020-08-071-1/+7
|
* Add health check endpoint (#8048)Erik Johnston2020-08-071-1/+8
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-071-2/+3
|
* Convert run_as_background_process inner function to async. (#8032)Patrick Cloke2020-08-061-3/+2
|
* Convert the SimpleHttpClient to async. (#8016)Patrick Cloke2020-08-041-31/+24
|
* Implement handling of HTTP HEAD requests. (#7999)Patrick Cloke2020-08-031-5/+11
|
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-40/+32
|
* Ensure the msg property of HttpResponseException is a string. (#7979)Patrick Cloke2020-07-292-7/+16
|
* Return an empty body for OPTIONS requests. (#7886)Patrick Cloke2020-07-241-19/+5
|
* Downgrade warning on client disconnect to INFO (#7928)Richard van der Hoff2020-07-241-3/+1
| | | | Clients disconnecting before we finish processing the request happens from time to time. We don't need to yell about it
* Convert the federation agent and related code to async/await. (#7874)Patrick Cloke2020-07-232-16/+10
|
* fix an incorrect commentRichard van der Hoff2020-07-221-2/+2
|
* Ensure that calls to `json.dumps` are compatible with the standard library ↵Patrick Cloke2020-07-152-3/+5
| | | | json. (#7836)
* Fix client reader sharding tests (#7853)Erik Johnston2020-07-151-1/+23
| | | | | | | | | | | | | | | * Fix client reader sharding tests * Newsfile * Fix typing * Update changelog.d/7853.misc Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Move mocking of http_client to tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802)Patrick Cloke2020-07-102-19/+9
|
* Add types to the server code and remove unused parameter (#7813)Patrick Cloke2020-07-101-30/+41
|
* Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-1/+1
| | | ... instead of duplicating `config.signing_key[0]` everywhere
* Stop passing bytes when dumping JSON (#7799)Patrick Cloke2020-07-081-3/+7
|
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-032-189/+195
|
* Merge branch 'master' into developPatrick Cloke2020-07-021-8/+68
|\
| * Ensure that HTML pages served from Synapse include headers to avoid embedding.Patrick Cloke2020-07-021-8/+68
| |
* | Include a user agent in federation requests. (#7677)Patrick Cloke2020-06-163-4/+32
| |
* | Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-163-14/+10
| |
* | Create a ListenerConfig object (#7681)Richard van der Hoff2020-06-161-2/+4
|/ | | | | | | | | | This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.
* Clean up exception handling in SAML2ResponseResource (#7614)Richard van der Hoff2020-06-031-12/+31
| | | | | | | | | | | | | * Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string * Clean up exception handling in SAML2ResponseResource * use the existing code in `return_html_error` instead of re-implementing it (giving it a jinja2 template rather than inventing a new form of template) * do the exception-catching in the REST layer rather than in the handler layer, to make sure we catch all exceptions.
* Fix missing CORS headers on OPTION responses (#7560)Erik Johnston2020-05-221-1/+1
| | | Broke in #7534.
* Return 200 OK for all OPTIONS requests (#7534)Patrick Cloke2020-05-221-3/+20
|
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-0/+7
| | | | These are business as usual errors, rather than stuff we want to log at error.
* mypy for synapse.http.site (#7553)Richard van der Hoff2020-05-221-3/+6
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-6/+2
|
* Workaround for failure to wrap reason in Failure (#7473)Andrew Morgan2020-05-141-0/+7
|
* Fix b'GET' in prometheus metrics (#7503)Richard van der Hoff2020-05-141-2/+4
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-2/+4
| | | | variables (#6391)
* Implement OpenID Connect-based login (#7256)Quentin Gliech2020-05-081-0/+7
|
* Reduce federation logging on success (#7321)Michael Kaye2020-04-221-8/+14
| | | | Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
* Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181)Richard van der Hoff2020-03-311-7/+6
| | | | Fixes #7179.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-241-3/+3
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Share SSL contexts for non-federation requests (#7094)Richard van der Hoff2020-03-172-4/+1
| | | | | | | Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests. Fixes #7092.
* Tiny optimisation for _get_handler_for_request (#6950)Richard van der Hoff2020-02-191-1/+3
| | | | we have hundreds of path_regexes (see #5118), so let's not convert the same bytes to str for each of them.
* Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-1/+1
| | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* Fix outbound federation request metrics (#6795)Erik Johnston2020-01-281-0/+4
|
* Implement RedirectException (#6687)Richard van der Hoff2020-01-151-4/+9
| | | | | Allow REST endpoint implemnentations to raise a RedirectException, which will redirect the user's browser to a given location.
* Fix exceptions on requests for non-ascii urls (#6682)Richard van der Hoff2020-01-131-1/+1
| | | Fixes #6402
* Kill off redundant SynapseRequestFactory (#6619)Richard van der Hoff2020-01-031-15/+3
| | | | We already get the Site via the Channel, so there's no need for a dedicated RequestFactory: we can just use the right constructor.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
|
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-013-2/+405
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-317-14/+14
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-1/+1
|
* Merge branch 'master' into developAndrew Morgan2019-10-031-2/+4
|\
| * Replace client_secret with <redacted> in server logs (#6158)Andrew Morgan2019-10-031-2/+4
| | | | | | Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
* | Edit SimpleHttpClient to reference that header keys can be passed as str or ↵Andrew Morgan2019-09-271-6/+6
|/ | | | bytes (#6077)
* Fix well-known lookups with the federation certificate whitelist (#5997)Amber Brown2019-09-141-1/+1
|
* Trace how long it takes for the send trasaction to complete, including ↵Jorik Schellekens2019-09-052-37/+51
| | | | retrys (#5986)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-052-6/+13
|
* Remove unused methods from c/s api v1 in register.py (#5963)Andrew Morgan2019-09-021-33/+1
| | | These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Merge pull request #5864 from matrix-org/erikj/reliable_lookupsErik Johnston2019-08-272-208/+217
|\ | | | | Refactor MatrixFederationAgent to retry SRV.
| * Fix off by one error in SRV result shufflingErik Johnston2019-08-271-8/+13
| |
| * Fixup commentsErik Johnston2019-08-231-1/+9
| |
| * Fix logcontextsErik Johnston2019-08-231-4/+8
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-08-223-47/+128
| |\ | | | | | | | | | erikj/reliable_lookups
| * | Remove now unused pick_server_from_listErik Johnston2019-08-201-30/+0
| | |
| * | Fixup _sort_server_list to be slightly more efficientErik Johnston2019-08-201-2/+11
| | | | | | | | | | | | | | | Also document that we are using the algorithm described in RFC2782 and ensure we handle zero weight correctly.
| * | Refactor MatrixFederationAgent to retry SRV.Erik Johnston2019-08-152-189/+202
| | | | | | | | | | | | | | | | | | | | | | | | This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool
* | | Opentracing across workers (#5771)Jorik Schellekens2019-08-221-1/+1
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* | Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_failErik Johnston2019-08-222-37/+115
|\ \ | | | | | | Retry well known on fail
| * | Change jitter to be a factor rather than absolute valueErik Johnston2019-08-201-11/+12
| | |
| * | Fixup changelog and remove debug loggingErik Johnston2019-08-161-4/+1
| | |
| * | Retry well known on fail.Erik Johnston2019-08-152-34/+114
| |/ | | | | | | | | | | If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.
* / Opentracing misc (#5856)Jorik Schellekens2019-08-161-10/+13
|/ | | | | | | | | | | Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-22/+60
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Make default well known cache global again.Erik Johnston2019-08-131-1/+4
|
* Move well known lookup into a separate clasErik Johnston2019-08-072-153/+197
|
* Add a lower bound for TTL on well known results.Erik Johnston2019-08-061-0/+4
| | | | | | | It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".
* Handle incorrectly encoded query params correctlyErik Johnston2019-08-011-1/+6
|
* Share SSL options for well-known requestsErik Johnston2019-07-311-11/+5
|
* Merge tag 'v1.2.0rc2' into developAndrew Morgan2019-07-242-14/+31
|\ | | | | | | | | | | | | Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
| * Fix servlet metric names (#5734)Jorik Schellekens2019-07-242-14/+31
| | | | | | | | | | | | | | | | | | | | * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths
* | Replace returnValue with return (#5736)Amber Brown2019-07-234-53/+45
|/
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-112-5/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-047-7/+7
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-10/+16
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-291-36/+41
|
* Run Black. (#5482)Amber Brown2019-06-2010-287/+321
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-4/+9
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Improve docstrings on MatrixFederationClient. (#5332)Richard van der Hoff2019-06-041-16/+55
|
* Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) v0.99.5.1.dev0Richard van der Hoff2019-05-291-4/+0
| | | This is just unhelpful spam
* Fix error code for invalid parameterAaron Raimist2019-05-241-1/+1
| | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-132-14/+40
|
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-20/+25
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Config option for verifying federation certificates (MSC 1711) (#4967)Andrew Morgan2019-04-251-1/+1
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-26 02:10:49 +0000