summary refs log tree commit diff
path: root/synapse/http/matrixfederationclient.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-231-0/+1
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Merge remote-tracking branch 'gitlab/clokep/license-license' into new_developErik Johnston2023-12-131-10/+16
|\
| * Update license headersPatrick Cloke2023-11-211-10/+16
| |
* | Request & follow redirects for /media/v3/download (#16701)Patrick Cloke2023-11-291-20/+57
|/ | | | | | Implement MSC3860 to follow redirects for federated media downloads. Note that the Client-Server API doesn't support this (yet) since the media repository in Synapse doesn't have a way of supporting redirects.
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-151-6/+8
|
* Don't reset retry timers on "valid" error codes (#16221)Erik Johnston2023-09-041-0/+8
|
* Bump ruff from 0.0.277 to 0.0.286 (#16198)dependabot[bot]2023-08-291-1/+1
| | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-181-10/+132
| | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-101-122/+10
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Federation outbound proxy (#15773)Eric Eastwood2023-07-051-10/+122
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Allow for the configuration of max request retries and min/max retry delays ↵Mathieu Velten2023-06-211-21/+38
| | | | in the matrix federation client (#15783)
* Switch from `matrix://` to `matrix-federation://` scheme for internal ↵Eric Eastwood2023-06-201-1/+8
| | | | | | | Synapse routing of outbound federation traffic (#15806) `matrix://` is a registered specced scheme nowadays and doesn't make sense for our internal to Synapse use case anymore. ([discussion] (https://github.com/matrix-org/synapse/pull/15773#discussion_r1227598679))
* Revert "Allow for the configuration of max request retries and min/max retry ↵Mathieu Velten2023-06-141-12/+9
| | | | | | delays in the matrix federation client (#12504)" This reverts commit d84e66144dc12dacf71c987a2ba802dd59c0b68e.
* Allow for the configuration of max request retries and min/max retry delays ↵Shay2023-06-091-9/+12
| | | | | | | in the matrix federation client (#12504) Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Add context for when/why to use the `long_retries` option when sending ↵Eric Eastwood2023-06-061-2/+9
| | | | Federation requests (#15721)
* Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-191-7/+7
| | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* Finish type hints for federation client HTTP code. (#15465)Patrick Cloke2023-04-241-18/+58
|
* Small fixes to `MatrixFederationHttpClient` docstrings (#15148)Andrew Morgan2023-02-271-7/+9
|
* Use mypy 1.0 (#15052)David Robertson2023-02-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Remove redundant types from comments. (#14412)Patrick Cloke2022-11-161-2/+1
| | | | | | | Remove type hints from comments which have been added as Python type hints. This helps avoid drift between comments and reality, as well as removing redundant information. Also adds some missing type hints which were simple to fill in.
* Validate federation destinations and log an error if server name is invalid. ↵Shay2022-07-201-0/+9
| | | | (#13318)
* Fix `destination_is` errors seen in sentry. (#13041)David Robertson2022-06-141-2/+5
| | | | | | * Rename test_fedclient to match its source file * Require at least one destination to be truthy * Explicitly validate user ID in profile endpoint GETs Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Merge tag 'v1.60.0rc2' into developSean Quah2022-05-271-0/+11
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.60.0rc2 (2022-05-27) ============================== This release of Synapse adds a unique index to the `state_group_edges` table, in order to prevent accidentally introducing duplicate information (for example, because a database backup was restored multiple times). If your Synapse database already has duplicate rows in this table, this could fail with an error and require manual remediation. Additionally, the signature of the `check_event_for_spam` module callback has changed. The previous signature has been deprecated and remains working for now. Module authors should update their modules to use the new signature where possible. See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1600) for more details. Features -------- - Add an option allowing users to use their password to reauthenticate for privileged actions even though password login is disabled. ([\#12883](https://github.com/matrix-org/synapse/issues/12883)) Bugfixes -------- - Explicitly close `ijson` coroutines once we are done with them, instead of leaving the garbage collector to close them. ([\#12875](https://github.com/matrix-org/synapse/issues/12875)) Internal Changes ---------------- - Improve URL previews by not including the content of media tags in the generated description. ([\#12887](https://github.com/matrix-org/synapse/issues/12887))
| * Close `ijson` coroutines ourselves instead of letting the GC close them (#12875)Sean Quah2022-05-271-0/+11
| | | | | | | | | | | | | | Hopefully this means that exceptions raised due to truncated JSON get a sensible logging context and stack. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Allow bigger responses to `/federation/v1/state` (#12877)Richard van der Hoff2022-05-251-22/+7
|/ | | | | | | | | | | * Refactor HTTP response size limits Rather than passing a separate `max_response_size` down the stack, make it an attribute of the `parser`. * Allow bigger responses on `federation/v1/state` `/state` can return huge responses, so we need to handle that.
* Make handling of federation Authorization header (more) compliant with ↵Hubert Chathi2022-05-181-1/+1
| | | | | | | | | | | | RFC7230 (#12774) The main differences are: - values with delimiters (such as colons) should be quoted, so always quote the origin, since it could contain a colon followed by a port number - should allow more than one space after "X-Matrix" - quoted values with backslash-escaped characters should be unescaped - names should be case insensitive
* Reduce the number of "untyped defs" (#12716)David Robertson2022-05-121-12/+19
|
* Immediately retry any requests that have backed off when a server comes back ↵Erik Johnston2022-05-101-2/+13
| | | | | online. (#12500) Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
* Implement MSC3383: include destination in X-Matrix auth header (#11398)Jan Christian Grünhage2022-04-191-2/+10
| | | | Co-authored-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz> Co-authored-by: Marcus Hoffmann <bubu@bubu1.eu>
* Unify HTTP query parameter type hints (#12415)David Robertson2022-04-081-22/+13
| | | | | | * Pull out query param types to `synapse.http.types` * Use QueryParams everywhere * Simplify `encode_query_args` * Add annotation which would have caught #12410
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-2/+0
|
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Faster joins: Support for calling `/federation/v1/state` (#12013)Richard van der Hoff2022-02-221-1/+49
| | | | This is an endpoint that we have server-side support for, but no client-side support. It's going to be useful for resyncing partial-stated rooms, so let's introduce it.
* Adds misc missing type hints (#11953)Patrick Cloke2022-02-111-2/+1
|
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-131-9/+9
|
* Use HTTPStatus constants in place of literals in `synapse.http` (#11543)Dirk Klimpel2021-12-091-1/+2
|
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-221-2/+6
|
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-5/+5
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-2/+3
|
* Remove unnecessary parentheses around tuples returned from methods (#10889)Andrew Morgan2021-09-231-1/+1
|
* Prevent logging context going missing on federation request timeout (#10810)Sean Quah2021-09-141-4/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | In `MatrixFederationHttpClient._send_request()`, we make a HTTP request using an `Agent`, wrap that request in a timeout and await the resulting `Deferred`. On its own, the `Agent` performing the HTTP request correctly stashes and restores the logging context while waiting. The addition of the timeout introduces a path where the logging context is not restored when execution resumes. To address this, we wrap the timeout `Deferred` in a `make_deferred_yieldable()` to stash the logging context and restore it on completion of the `await`. However this is not sufficient, since by the time we construct the timeout `Deferred`, the `Agent` has already stashed and cleared the logging context when using `make_deferred_yieldable()` to produce its `Deferred` for the request. Hence, we wrap the `Agent` request in a `run_in_background()` to "fork" and preserve the logging context so that we can stash and restore it when `await`ing the timeout `Deferred`. This approach is similar to the one used with `defer.gatherResults`. Note that the code is still not fully correct. When a timeout occurs, the request remains running in the background (existing behavior which is nothing to do with the new call to `run_in_background`) and may re-start the logging context after it has finished.
* support federation queries through http connect proxy (#10475)Dirk Klimpel2021-08-111-9/+3
| | | | | Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu> Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-151-6/+6
|
* Fix a number of logged errors caused by remote servers being down. (#10400)Erik Johnston2021-07-151-0/+28
|
* Fix 'ip_range_whitelist' not working for federation servers (#10115)Michael Kutzner2021-06-151-1/+3
| | | | | Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers. Signed-off-by: Michael Kutzner 1mikure@gmail.com
* Clean up the interface for injecting opentracing over HTTP (#10143)Richard van der Hoff2021-06-091-7/+3
| | | | | | | * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog
* Allow response of `/send_join` to be larger. (#10093)Erik Johnston2021-05-281-1/+13
| | | Fixes #10087.
* Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958)Erik Johnston2021-05-201-35/+125
| | | Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
* Limit the size of HTTP responses read over federation. (#9833)Richard van der Hoff2021-04-231-6/+37
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bump black configuration to target py36 (#9781)Dan Callahan2021-04-131-1/+1
| | | Signed-off-by: Dan Callahan <danc@element.io>
* Retry 5xx errors in federation client (#9567)Erik Johnston2021-03-091-3/+4
| | | Fixes #8915
* Create a SynapseReactor type which incorporates the necessary reactor ↵Patrick Cloke2021-03-081-4/+4
| | | | | interfaces. (#9528) This helps fix some type hints when running with Twisted 21.2.0.
* Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-031-3/+3
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-6/+14
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Properly raise an exception when the body exceeds the max size. (#9145)Patrick Cloke2021-01-181-1/+1
| | | ...instead of just creating the exception object and doing nothing with it.
* Handle bad JSON data being returned from the federation API. (#9070)Patrick Cloke2021-01-121-0/+10
|
* Add a maximum size for well-known lookups. (#8950)Patrick Cloke2020-12-161-2/+11
|
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-18/+8
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Add additional type hints to HTTP client. (#8812)Patrick Cloke2020-11-251-64/+10
| | | | This also removes some duplicated code between the simple HTTP client and matrix federation client.
* Add type hints to matrix federation client / agent. (#8806)Patrick Cloke2020-11-251-147/+157
|
* Better error message when a remote resource uses invalid Content-Type (#8719)Andrew Morgan2020-11-111-2/+8
|
* Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-231-3/+3
|
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-291-12/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* A pair of tiny cleanups in the federation request code. (#8401)Richard van der Hoff2020-09-281-2/+0
|
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-1/+1
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-101-1/+4
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-3/+3
|
* Reduce INFO logging (#8050)Erik Johnston2020-08-111-23/+71
| | | | | | | | | | c.f. #8021 A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line. Some notes: 1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps. 2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-40/+32
|
* Ensure the msg property of HttpResponseException is a string. (#7979)Patrick Cloke2020-07-291-3/+4
|
* Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-1/+1
| | | ... instead of duplicating `config.signing_key[0]` everywhere
* Stop passing bytes when dumping JSON (#7799)Patrick Cloke2020-07-081-3/+7
|
* Include a user agent in federation requests. (#7677)Patrick Cloke2020-06-161-1/+8
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-7/+5
|
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-0/+7
| | | | These are business as usual errors, rather than stuff we want to log at error.
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-6/+2
|
* Fix b'GET' in prometheus metrics (#7503)Richard van der Hoff2020-05-141-2/+4
|
* Reduce federation logging on success (#7321)Michael Kaye2020-04-221-8/+14
| | | | Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
* Fix outbound federation request metrics (#6795)Erik Johnston2020-01-281-0/+4
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
|
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-5/+5
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace how long it takes for the send trasaction to complete, including ↵Jorik Schellekens2019-09-051-1/+0
| | | | retrys (#5986)
* Opentracing misc (#5856)Jorik Schellekens2019-08-161-10/+13
| | | | | | | | | | | Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.
* Replace returnValue with return (#5736)Amber Brown2019-07-231-8/+8
|
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-111-4/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Run Black. (#5482)Amber Brown2019-06-201-121/+121
|
* Improve docstrings on MatrixFederationClient. (#5332)Richard van der Hoff2019-06-041-16/+55
|
* Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) v0.99.5.1.dev0Richard van der Hoff2019-05-291-4/+0
| | | This is just unhelpful spam
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-131-10/+38
|
* Make federation endpoints more tolerant of trailing slashes v2 (#4935)Andrew Morgan2019-03-261-0/+1
| | | Redo of https://github.com/matrix-org/synapse/pull/4840
* Clean up backoff_on_404 and metehod callsAndrew Morgan2019-03-211-9/+13
|
* kwargs doesn't like commas on calling funcs either. TILAndrew Morgan2019-03-201-2/+2
|
* lintAndrew Morgan2019-03-201-2/+6
|
* New test, fix issuesAndrew Morgan2019-03-201-49/+28
|
* Fix comments. v0.99.2 -> v0.99.3Andrew Morgan2019-03-201-5/+5
|
* Just return if not doing any trailing slash shennanigansAndrew Morgan2019-03-201-0/+2
|
* lintAndrew Morgan2019-03-181-1/+1
|
* Better exception handlingAndrew Morgan2019-03-181-16/+17
|
* Correct var nameAndrew Morgan2019-03-131-2/+2
|
* receiving a 400 caused an exception. handle itAndrew Morgan2019-03-131-5/+12
|
* is this what purgatory feels likeAndrew Morgan2019-03-131-3/+3
|
* i should have given up x2Andrew Morgan2019-03-131-18/+13
|
* i should have given upAndrew Morgan2019-03-131-1/+1
|
* there comes a time when you should give up. but you dontAndrew Morgan2019-03-131-1/+1
|
* as aboveAndrew Morgan2019-03-131-2/+4
|
* no kwargs todayAndrew Morgan2019-03-131-7/+12
|
* Or perhaps I was the one who was drunkAndrew Morgan2019-03-131-6/+3
|
* go home python, you're drunkAndrew Morgan2019-03-131-4/+2
|
* Syntax checker is borkAndrew Morgan2019-03-131-6/+5
|
* Destructure againAndrew Morgan2019-03-131-7/+8
|
* Are you happy nowAndrew Morgan2019-03-131-2/+2
|
* Syntax testAndrew Morgan2019-03-131-6/+5
|
* Remove testing codeAndrew Morgan2019-03-131-3/+0
|
* Add missing docstring detailAndrew Morgan2019-03-131-1/+1
|
* Fix syntax issuesAndrew Morgan2019-03-131-7/+10
|
* Switch to wrapper function around _send_requestAndrew Morgan2019-03-131-32/+71
|
* Retry on 400:M_UNRECOGNIZEDAndrew Morgan2019-03-131-6/+10
|
* Fix paranthesis indentAndrew Morgan2019-03-131-1/+2
|
* Add workaround noteAndrew Morgan2019-03-131-10/+4
|
* Cleaner way of implementing trailing slashesAndrew Morgan2019-03-121-63/+52
|
* Retry certain federation requests on 404Andrew Morgan2019-03-081-0/+45
|
* Fix TaskStopped exceptions when outbound requests time out (#4690)Richard van der Hoff2019-02-201-15/+2
|
* Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)Richard van der Hoff2019-02-111-2/+2
| | | | | It's nothing to do with refreshing the certificates. No idea why it was here.
* Fix noisy "twisted.internet.task.TaskStopped" errors in logsRichard van der Hoff2019-02-011-2/+15
| | | | Fixes #4003
* Fix Host header sent by MatrixFederationAgent (#4468)Richard van der Hoff2019-01-251-1/+0
| | | | | | Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.
* Make MatrixFederationClient use MatrixFederationAgentRichard van der Hoff2019-01-221-27/+10
| | | | ... instead of the matrix_federation_endpoint
* Remove redundant WrappedConnection (#4409)Richard van der Hoff2019-01-181-15/+15
| | | | | | | | * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.
* sign_request -> build_auth_headers (#4408)Richard van der Hoff2019-01-171-12/+11
| | | | | Just got very confused about the fact that the headers are only an output, not an input.
* Fixup docstrings for matrixfederationclientErik Johnston2019-01-091-68/+73
|
* Use RequestSendFailed when fail to parse content type headersErik Johnston2019-01-081-5/+5
|
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-33/+72
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Merge pull request #3969 from turt2live/travis/fix-federated-group-requestsTravis Ralston2018-10-231-6/+6
|\ | | | | Handle HttpResponseException more safely for federated groups
| * Fix exception documentation in matrixfederationclient.pyTravis Ralston2018-09-261-6/+6
| |
* | Various cleanups in the federation client code (#4031)Richard van der Hoff2018-10-161-38/+40
|/ | | | | | | | | | | | | | - Improve logging: log things in the right order, include destination and txids in all log lines, don't log successful responses twice - Fix the docstring on TransportLayerClient.send_transaction - Don't use treq.request, which is overcomplicated for our purposes: just use a twisted.web.client.Agent. - simplify the logic for setting up the bodyProducer - fix bytes/str confusions
* Update to use new timeout function everywhere.Erik Johnston2018-09-191-10/+15
| | | | | | | The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.
* Destination is a stringTravis Ralston2018-09-181-4/+4
|
* Refactor matrixfederationclient to fix logging (#3906)Richard van der Hoff2018-09-181-149/+236
| | | | | | | | We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.
* Fix some b'abcd' noise in logs and metricsRichard van der Hoff2018-09-171-1/+4
| | | | | Python 3 compatibility: make sure that we decode some byte sequences before we use them to create log lines and metrics labels.
* Add an awful secondary timeout to fix wedged requestsErik Johnston2018-09-141-0/+11
| | | | This is an attempt to mitigate #3842 by adding yet-another-timeout
* Merge pull request #3872 from matrix-org/hawkowl/timeouts-2Erik Johnston2018-09-141-1/+3
|\ | | | | timeouts 2: electric boogaloo
| * we do that higher upAmber Brown2018-09-141-0/+1
| |
| * buffer?Amber Brown2018-09-141-1/+2
| |
* | Measure outbound requestsErik Johnston2018-09-141-3/+6
|/
* Attempt to figure out what's going on with timeouts (#3857)Amber Brown2018-09-141-56/+42
|
* Log outbound requests when we retryErik Johnston2018-09-121-75/+74
|
* Timeout reading body for outbound HTTP requestsErik Johnston2018-09-121-7/+45
|
* Port http/ to Python 3 (#3771)Amber Brown2018-09-061-120/+84
|
* Fix bug when federation_domain_whitelist is an emtpy listErik Johnston2018-08-241-1/+1
| | | | | Outbound federation were incorrectly allowed when the config option was set to an empty list
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
|
* Merge pull request #3439 from vojeroen/send_sni_for_federation_requestsRichard van der Hoff2018-08-101-2/+2
|\ | | | | send SNI for federation requests
| * Merge remote-tracking branch 'upstream/develop' into ↵Jeroen2018-07-141-24/+24
| |\ | | | | | | | | | | | | | | | | | | send_sni_for_federation_requests # Conflicts: # synapse/crypto/context_factory.py
| * \ Merge branch 'develop' into send_sni_for_federation_requestsJeroen2018-07-091-2/+1
| |\ \ | | | | | | | | | | | | | | | | # Conflicts: # synapse/http/endpoint.py
| * | | formatting changes for pep8Jeroen2018-06-251-1/+1
| | | |
| * | | send SNI for federation requestsJeroen2018-06-241-2/+2
| | | |
* | | | include known room versions in outgoing make_joinsRichard van der Hoff2018-08-061-2/+5
| |_|/ |/| |
* | | run isortAmber Brown2018-07-091-24/+24
| |/ |/|
* | Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-2/+1
|/
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-221-2/+3
|
* Merge pull request #3276 from matrix-org/dbkr/unbindDavid Baker2018-06-111-2/+23
|\ | | | | Remove email addresses / phone numbers from ID servers when they're removed from synapse
| * pep8David Baker2018-06-061-3/+3
| |
| * More doc fixesDavid Baker2018-06-061-2/+2
| |
| * fix pep8David Baker2018-06-051-3/+5
| |
| * doc fixesDavid Baker2018-06-051-6/+6
| |
| * DocstringDavid Baker2018-06-041-0/+14
| |
| * Merge remote-tracking branch 'origin/develop' into dbkr/unbindDavid Baker2018-05-241-3/+5
| |\
| * | Hit the 3pid unbind endpoint on deactivationDavid Baker2018-05-231-2/+7
| | |
* | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-281-3/+5
|\ \ \ | | |/ | |/|
| * | Replace some more comparisons with sixAdrian Tschira2018-05-191-3/+5
| |/ | | | | | | | | | | plus a bonus b"" string I missed last time Signed-off-by: Adrian Tschira <nota@notafile.com>
* | cleanupAmber Brown2018-05-221-2/+4
| |
* | replacing portionsAmber Brown2018-05-211-10/+4
|/
* Merge pull request #3108 from NotAFile/py3-six-urlparseRichard van der Hoff2018-04-301-2/+1
|\ | | | | Use six.moves.urlparse
| * Use six.moves.urlparseAdrian Tschira2018-04-151-2/+1
| | | | | | | | | | | | The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Backport deferred.addTimeoutRichard van der Hoff2018-04-271-3/+4
| | | | | | | | Twisted 16.0 doesn't have addTimeout, so let's backport it.
* | Use deferred.addTimeout instead of time_bound_deferredRichard van der Hoff2018-04-231-17/+18
|/ | | | This doesn't feel like a wheel we need to reinvent.
* use PUT instead of POST for federating groups/m.join_policyKrombel2018-04-061-1/+5
|
* Add federation_domain_whitelist option (#2820)Matthew Hodgson2018-01-221-1/+27
| | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-231-1/+1
| | | | what could possibly go wrong
* Merge branch 'develop' into erikj/groups_mergedDavid Baker2017-10-021-7/+6
|\
| * Improve logging of failures in matrixfederationclientRichard van der Hoff2017-09-281-7/+6
| | | | | | | | | | | | * don't log exception types twice * not all exceptions have a meaningful 'message'. Use the repr rather than attempting to build a string ourselves.
* | Add DELETEErik Johnston2017-07-181-0/+46
| |
* | Add 'args' param to post_jsonErik Johnston2017-07-181-9/+15
|/
* Fix some lies, and other clarifications, in docstringsRichard van der Hoff2017-04-211-7/+14
| | | | | The documentation on get_json has been wrong ever since the very first commit to synapse...
* try not to drop context after federation requestsRichard van der Hoff2017-03-231-12/+16
| | | | | | | preserve_context_over_fn uses a ContextPreservingDeferred, which only restores context for the duration of its callbacks, which isn't really correct, and means that subsequent operations in the same request can end up without their logcontexts.
* Ignore backoff history for invites, aliases, and roomdirsRichard van der Hoff2017-03-231-7/+26
| | | | | Add a param to the federation client which lets us ignore historical backoff data for federation queries, and set it for a handful of operations.
* push federation retry limiter down to matrixfederationclientRichard van der Hoff2017-03-231-101/+127
| | | | | rather than having to instrument everywhere we make a federation call, make the MatrixFederationHttpClient manage the retry limiter.
* MatrixFederationHttpClient: clean upRichard van der Hoff2017-03-231-17/+24
| | | | | rename _create_request to _request, and push ascii-encoding of `destination` and `path` down into it
* Fix routing loop when fetching remote mediaRichard van der Hoff2017-03-131-3/+12
| | | | | | | | | | | When we proxy a media request to a remote server, add a query-param, which will tell the remote server to 404 if it doesn't recognise the server_name. This should fix a routing loop where the server keeps forwarding back to itself. Also improves the error handling on remote media fetches, so that we don't always return a rather obscure 502.
* Respect long_retries param and default to offErik Johnston2016-12-291-2/+2
|
* Clean upErik Johnston2016-12-291-5/+0
|
* Wrap connections in an N minute timeout to ensure they get reaped correctlyErik Johnston2016-12-291-1/+7
|
* More intelligent Content-Type parsingRichard van der Hoff2016-11-301-18/+30
| | | | | Content-Type is allowed to contain options (`; charset=utf-8`, for instance). We should allow that.
* Add a timeout parameter for end2end key queries.Mark Haines2016-09-121-2/+9
| | | | | | | | | | Add a timeout parameter for controlling how long synapse will wait for responses from remote servers. For servers that fail include how they failed to make it easier to debug. Fetch keys from different servers in parallel rather than in series. Set the default timeout to 10s.
* Clean up TransactionQueueErik Johnston2016-08-101-3/+1
|
* Fix flake8 warnings for new flake8Daniel Wagner-Hall2016-02-021-1/+1
|
* copyrightsMatthew Hodgson2016-01-071-1/+1
|
* Fix typoErik Johnston2015-11-201-2/+2
|
* Don't limit connections to perspective serversErik Johnston2015-11-201-1/+4
|
* Use min and not max to set an upper bound on retry intervalErik Johnston2015-11-201-2/+2
|
* Add commentErik Johnston2015-11-171-0/+2
|
* Only retry federation requests for a long time for background requestsErik Johnston2015-11-171-7/+19
|
* Implement required method 'resumeProducing'Erik Johnston2015-11-171-0/+3
|
* Slightly more aggressive retry timers at HTTP levelErik Johnston2015-11-171-2/+3
|
* Retry dead servers a lot less oftenErik Johnston2015-11-021-2/+8
|
* Allow configuration to ignore invalid SSL certsDaniel Wagner-Hall2015-09-091-2/+2
| | | | | This will be useful for sytest, and sytest only, hence the aggressive config key name.
* Remove syutil dependency in favour of smaller single-purpose librariesMark Haines2015-08-241-2/+2
|
* PEP8Erik Johnston2015-08-131-1/+0
|
* Merge branch 'develop' into markjh/twisted-15Mark Haines2015-08-121-68/+77
|\ | | | | | | | | Conflicts: synapse/http/matrixfederationclient.py
| * Fix log context when sending requestsErik Johnston2015-06-191-28/+30
| |
| * Remove stale debug linesErik Johnston2015-06-191-10/+0
| |
| * Add IDs to outbound transactionsErik Johnston2015-06-191-53/+71
| |
* | Use Twisted-15.2.1, Use Agent.usingEndpointFactory rather than implement our ↵Mark Haines2015-06-011-50/+25
|/ | | | own Agent
* Correctly pass connection pool parameterErik Johnston2015-05-281-1/+1
|
* Up maxPersistentPerHost countErik Johnston2015-05-281-0/+1
|
* Use connection pool for federation connectionsErik Johnston2015-05-281-2/+3
|
* Add doc stringsErik Johnston2015-05-221-0/+3
|
* Add a timeout param to get_eventErik Johnston2015-05-191-5/+8
|
* Change the way we do logging contexts so that they survive divergencesErik Johnston2015-05-081-16/+16
|
* Appease pep8Paul "LeoNerd" Evans2015-03-121-3/+7
|
* Use _ instead of . as a metric namespacing separator, for PrometheusPaul "LeoNerd" Evans2015-03-121-2/+2
|
* Rename Metrics' "keys" to "labels"Paul "LeoNerd" Evans2015-03-121-2/+2
|
* Add vector counters to HTTP clients and servers; count the requests by ↵Paul "LeoNerd" Evans2015-03-121-3/+25
| | | | method and responses by method and response code
* Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.7.1Erik Johnston2015-02-181-2/+2
|\
| * Use git aware version string in User-Agent and Server headersErik Johnston2015-02-181-2/+2
| |
* | Don't convert DNSLookupError to a 4xx SynapseErrorErik Johnston2015-02-181-1/+1
|/
* Add .__name__ after type(e)Erik Johnston2015-02-181-2/+2
|
* Fix up ResponseNeverReceived to strErik Johnston2015-02-181-9/+11
|
* strings.join() expects iterable of stringsErik Johnston2015-02-181-1/+1
|
* Remove unused importErik Johnston2015-02-171-1/+0
|
* Make matrixfederationclient log more nicelyErik Johnston2015-02-171-11/+12
|
* Merge pull request #61 from matrix-org/timeout-federation-requestsErik Johnston2015-02-111-1/+7
|\ | | | | Timeout federation requests
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2015-02-111-5/+73
| |\ | | | | | | | | | timeout-federation-requests
| * | PEP8Erik Johnston2015-02-111-1/+0
| | |
| * | Move time_bound_deferred into ClockErik Johnston2015-02-111-3/+2
| | |
| * | Time out HTTP federation requestsErik Johnston2015-02-101-2/+9
| | |
* | | Blunty replace json with simplejsonErik Johnston2015-02-111-1/+1
| |/ |/|
* | Return body of response in HttpResponseExceptionErik Johnston2015-02-061-1/+2
| |
* | Apply sanity to the transport client interface. Convert 'make_join' and ↵Erik Johnston2015-02-041-6/+36
| | | | | | | | 'send_join' to accept iterables of destinations
* | Make post_json(...) actually send data.Erik Johnston2015-01-291-1/+1
| |
* | Add post_json(...) method to federation clientErik Johnston2015-01-291-0/+37
| |
* | Finish renaming "context" to "room_id" in federation codebaseMark Haines2015-01-161-1/+0
|/
* Update copyright noticesMark Haines2015-01-061-1/+1
|
* Include version in User-Agent and Server headersMark Haines2014-12-221-1/+3
|
* Limit the size of images that are thumbnailed serverside. Limit the size of ↵Mark Haines2014-12-111-6/+19
| | | | file that a server will download from a remote server
* Merge branch 'develop' into media_repositoryMark Haines2014-12-101-9/+26
|\
| * turn back on per-request transaction retries, so that every time we try to ↵Matthew Hodgson2014-12-101-4/+3
| | | | | | | | hit a dead server we actually end up hammering 5 times :|
| * Code style.Erik Johnston2014-12-101-8/+21
| |
| * add errbacks to enqueue_pdu deferreds; change logging for failed federation ↵Matthew Hodgson2014-12-101-2/+2
| | | | | | | | sends to warn rather than exception
| * squidge to 79 columns as per pep8Matthew Hodgson2014-12-101-3/+4
| |