summary refs log tree commit diff
path: root/synapse/http (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-034-35/+53
|
* (Hopefully) stop leaking file descriptors in media repo. (#9497)Patrick Cloke2021-03-011-2/+29
| | | | By consuming the response if the headers imply that the content is too large.
* Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-262-10/+37
| | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
* SSO: redirect to public URL before setting cookies (#9436)Richard van der Hoff2021-02-261-1/+36
| | | ... otherwise, we don't get the cookie back.
* Add support for X-Forwarded-Proto (#9472)Richard van der Hoff2021-02-241-15/+70
| | | | | rewrite XForwardedForRequest to set `isSecure()` based on `X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
* Reduce the memory usage of previewing media files. (#9421)Patrick Cloke2021-02-181-14/+12
| | | | | | | | This reduces the memory usage of previewing media files which end up larger than the `max_spider_size` by avoiding buffering content internally in treq. It also checks the `Content-Length` header in additional places instead of streaming the content to check the body length.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-169-51/+74
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add debug logging to DNS SRV requests. (#9305)Marcus2021-02-031-0/+7
|
* Split out a separate endpoint to complete SSO registration (#9262)Richard van der Hoff2021-02-011-0/+7
| | | There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
* Implement MSC2858 support (#9183)Richard van der Hoff2021-01-271-8/+36
| | | Fixes #8928.
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-79/+0
|
* Properly raise an exception when the body exceeds the max size. (#9145)Patrick Cloke2021-01-182-2/+2
| | | ...instead of just creating the exception object and doing nothing with it.
* Avoid raising the body exceeded error multiple times. (#9108)Patrick Cloke2021-01-151-1/+11
| | | | | Previously this code generated unreferenced `Deferred` instances which caused "Unhandled Deferreds" errors to appear in error situations.
* Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098)Tim Leung2021-01-141-1/+0
| | | | | | A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper used by the WellyKnownResolver. This coulld cause exceptions when attempting to connect to IP addresses that are blacklisted, but in reality this did not have any observable affect since this code is not used for IP literals.
* Reduce scope of exception handler. (#9106)Patrick Cloke2021-01-131-5/+5
| | | | Removes a bare `except Exception` clause and replaces it with catching a specific exception around the portion that might throw.
* Merge branch 'master' into developErik Johnston2021-01-132-3/+14
|\
| * Don't apply the IP range blacklist to proxy connections (#9084)Marcus2021-01-122-3/+14
| | | | | | | | | | It is expected that the proxy would be on a private IP address so the configured proxy should be connected to regardless of the IP range blacklist.
* | Handle bad JSON data being returned from the federation API. (#9070)Patrick Cloke2021-01-121-0/+10
| |
* | Remove SynapseRequest.get_user_agent (#9069)Richard van der Hoff2021-01-122-16/+17
|/ | | | | | | | | | | SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests", which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the LoggingContext and write the right entries to the request log). Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a SynapseRequest when there is nothing synapse-specific about the Request at all, and any old twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult. In short: move get_user_agent out to a utility function.
* Add a maximum size for well-known lookups. (#8950)Patrick Cloke2020-12-163-18/+52
|
* Various clean-ups to the logging context code (#8935)Patrick Cloke2020-12-141-2/+1
|
* Fix buglet in DirectRenderJsonResource (#8897)Richard van der Hoff2020-12-101-3/+5
| | | | this was using `canonical_json` without setting it, so when you used it as a standalone class, you would get exceptions.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-023-36/+52
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Allow Date header through CORS (#8804)Nicolas Chamo2020-12-011-1/+1
|
* Remove special case of pretty printing JSON responses for curl (#8833)Tulir Asokan2020-11-271-24/+5
| | | | | * Remove special case of pretty printing JSON responses for curl Signed-off-by: Tulir Asokan <tulir@maunium.net>
* Add additional type hints to HTTP client. (#8812)Patrick Cloke2020-11-252-147/+138
| | | | This also removes some duplicated code between the simple HTTP client and matrix federation client.
* Add type hints to matrix federation client / agent. (#8806)Patrick Cloke2020-11-253-194/+226
|
* Better error message when a remote resource uses invalid Content-Type (#8719)Andrew Morgan2020-11-111-2/+8
|
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-7/+23
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-281-3/+1
|\ | | | | Fix serialisation errors when using third-party event rules.
| * remove unused importsRichard van der Hoff2020-10-281-2/+0
| |
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-281-1/+1
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
| | | | | | | | | | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* | Fix typos and spelling errors. (#8639)Patrick Cloke2020-10-235-9/+11
| |
* | Fix handling of User-Agent headers with bad utf-8. (#8632)Erik Johnston2020-10-231-7/+9
| |
* | Remove unused OPTIONS handlers. (#8621)Patrick Cloke2020-10-221-2/+1
|/ | | | The handling of OPTIONS requests was consolidated in #7534, but the endpoint specific handlers were not removed.
* Don't bother responding to client requests that have already disconnected ↵Andrew Morgan2020-10-061-0/+5
| | | | | | | | | | (#8465) This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so. This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003. Fixes https://github.com/matrix-org/synapse/issues/6700 Fixes https://github.com/matrix-org/synapse/issues/5304
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-2/+2
|
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-294-50/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* A pair of tiny cleanups in the federation request code. (#8401)Richard van der Hoff2020-09-281-2/+0
|
* Add type annotations to SimpleHttpClient (#8372)Richard van der Hoff2020-09-241-56/+131
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-142-2/+2
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-101-1/+4
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-049-18/+18
|
* Convert the well known resolver to async (#8214)Patrick Cloke2020-09-012-28/+33
|
* Switch the JSON byte producer from a pull to a push producer. (#8116)Patrick Cloke2020-08-191-32/+43
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-193-11/+10
|
* Iteratively encode JSON responses to avoid blocking the reactor. (#8013)Patrick Cloke2020-08-181-8/+89
|
* Reduce INFO logging (#8050)Erik Johnston2020-08-113-25/+73
| | | | | | | | | | c.f. #8021 A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line. Some notes: 1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps. 2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
* Don't log OPTIONS request at INFO (#8049)Erik Johnston2020-08-071-1/+7
|
* Add health check endpoint (#8048)Erik Johnston2020-08-071-1/+8
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-071-2/+3
|
* Convert run_as_background_process inner function to async. (#8032)Patrick Cloke2020-08-061-3/+2
|
* Convert the SimpleHttpClient to async. (#8016)Patrick Cloke2020-08-041-31/+24
|
* Implement handling of HTTP HEAD requests. (#7999)Patrick Cloke2020-08-031-5/+11
|
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-40/+32
|
* Ensure the msg property of HttpResponseException is a string. (#7979)Patrick Cloke2020-07-292-7/+16
|
* Return an empty body for OPTIONS requests. (#7886)Patrick Cloke2020-07-241-19/+5
|
* Downgrade warning on client disconnect to INFO (#7928)Richard van der Hoff2020-07-241-3/+1
| | | | Clients disconnecting before we finish processing the request happens from time to time. We don't need to yell about it
* Convert the federation agent and related code to async/await. (#7874)Patrick Cloke2020-07-232-16/+10
|
* fix an incorrect commentRichard van der Hoff2020-07-221-2/+2
|
* Ensure that calls to `json.dumps` are compatible with the standard library ↵Patrick Cloke2020-07-152-3/+5
| | | | json. (#7836)
* Fix client reader sharding tests (#7853)Erik Johnston2020-07-151-1/+23
| | | | | | | | | | | | | | | * Fix client reader sharding tests * Newsfile * Fix typing * Update changelog.d/7853.misc Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Move mocking of http_client to tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802)Patrick Cloke2020-07-102-19/+9
|
* Add types to the server code and remove unused parameter (#7813)Patrick Cloke2020-07-101-30/+41
|
* Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-1/+1
| | | ... instead of duplicating `config.signing_key[0]` everywhere
* Stop passing bytes when dumping JSON (#7799)Patrick Cloke2020-07-081-3/+7
|
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-032-189/+195
|
* Merge branch 'master' into developPatrick Cloke2020-07-021-8/+68
|\
| * Ensure that HTML pages served from Synapse include headers to avoid embedding.Patrick Cloke2020-07-021-8/+68
| |
* | Include a user agent in federation requests. (#7677)Patrick Cloke2020-06-163-4/+32
| |
* | Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-163-14/+10
| |
* | Create a ListenerConfig object (#7681)Richard van der Hoff2020-06-161-2/+4
|/ | | | | | | | | | This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.
* Clean up exception handling in SAML2ResponseResource (#7614)Richard van der Hoff2020-06-031-12/+31
| | | | | | | | | | | | | * Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string * Clean up exception handling in SAML2ResponseResource * use the existing code in `return_html_error` instead of re-implementing it (giving it a jinja2 template rather than inventing a new form of template) * do the exception-catching in the REST layer rather than in the handler layer, to make sure we catch all exceptions.
* Fix missing CORS headers on OPTION responses (#7560)Erik Johnston2020-05-221-1/+1
| | | Broke in #7534.
* Return 200 OK for all OPTIONS requests (#7534)Patrick Cloke2020-05-221-3/+20
|
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-0/+7
| | | | These are business as usual errors, rather than stuff we want to log at error.
* mypy for synapse.http.site (#7553)Richard van der Hoff2020-05-221-3/+6
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-6/+2
|
* Workaround for failure to wrap reason in Failure (#7473)Andrew Morgan2020-05-141-0/+7
|
* Fix b'GET' in prometheus metrics (#7503)Richard van der Hoff2020-05-141-2/+4
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-2/+4
| | | | variables (#6391)
* Implement OpenID Connect-based login (#7256)Quentin Gliech2020-05-081-0/+7
|
* Reduce federation logging on success (#7321)Michael Kaye2020-04-221-8/+14
| | | | Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
* Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181)Richard van der Hoff2020-03-311-7/+6
| | | | Fixes #7179.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-241-3/+3
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Share SSL contexts for non-federation requests (#7094)Richard van der Hoff2020-03-172-4/+1
| | | | | | | Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests. Fixes #7092.
* Tiny optimisation for _get_handler_for_request (#6950)Richard van der Hoff2020-02-191-1/+3
| | | | we have hundreds of path_regexes (see #5118), so let's not convert the same bytes to str for each of them.
* Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-1/+1
| | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* Fix outbound federation request metrics (#6795)Erik Johnston2020-01-281-0/+4
|
* Implement RedirectException (#6687)Richard van der Hoff2020-01-151-4/+9
| | | | | Allow REST endpoint implemnentations to raise a RedirectException, which will redirect the user's browser to a given location.
* Fix exceptions on requests for non-ascii urls (#6682)Richard van der Hoff2020-01-131-1/+1
| | | Fixes #6402
* Kill off redundant SynapseRequestFactory (#6619)Richard van der Hoff2020-01-031-15/+3
| | | | We already get the Site via the Channel, so there's no need for a dedicated RequestFactory: we can just use the right constructor.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
|
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-013-2/+405
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-317-14/+14
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-1/+1
|
* Merge branch 'master' into developAndrew Morgan2019-10-031-2/+4
|\
| * Replace client_secret with <redacted> in server logs (#6158)Andrew Morgan2019-10-031-2/+4
| | | | | | Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
* | Edit SimpleHttpClient to reference that header keys can be passed as str or ↵Andrew Morgan2019-09-271-6/+6
|/ | | | bytes (#6077)
* Fix well-known lookups with the federation certificate whitelist (#5997)Amber Brown2019-09-141-1/+1
|
* Trace how long it takes for the send trasaction to complete, including ↵Jorik Schellekens2019-09-052-37/+51
| | | | retrys (#5986)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-052-6/+13
|
* Remove unused methods from c/s api v1 in register.py (#5963)Andrew Morgan2019-09-021-33/+1
| | | These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Merge pull request #5864 from matrix-org/erikj/reliable_lookupsErik Johnston2019-08-272-208/+217
|\ | | | | Refactor MatrixFederationAgent to retry SRV.
| * Fix off by one error in SRV result shufflingErik Johnston2019-08-271-8/+13
| |
| * Fixup commentsErik Johnston2019-08-231-1/+9
| |
| * Fix logcontextsErik Johnston2019-08-231-4/+8
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-08-223-47/+128
| |\ | | | | | | | | | erikj/reliable_lookups
| * | Remove now unused pick_server_from_listErik Johnston2019-08-201-30/+0
| | |
| * | Fixup _sort_server_list to be slightly more efficientErik Johnston2019-08-201-2/+11
| | | | | | | | | | | | | | | Also document that we are using the algorithm described in RFC2782 and ensure we handle zero weight correctly.
| * | Refactor MatrixFederationAgent to retry SRV.Erik Johnston2019-08-152-189/+202
| | | | | | | | | | | | | | | | | | | | | | | | This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool
* | | Opentracing across workers (#5771)Jorik Schellekens2019-08-221-1/+1
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* | Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_failErik Johnston2019-08-222-37/+115
|\ \ | | | | | | Retry well known on fail
| * | Change jitter to be a factor rather than absolute valueErik Johnston2019-08-201-11/+12
| | |
| * | Fixup changelog and remove debug loggingErik Johnston2019-08-161-4/+1
| | |
| * | Retry well known on fail.Erik Johnston2019-08-152-34/+114
| |/ | | | | | | | | | | If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.
* / Opentracing misc (#5856)Jorik Schellekens2019-08-161-10/+13
|/ | | | | | | | | | | Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-22/+60
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Make default well known cache global again.Erik Johnston2019-08-131-1/+4
|
* Move well known lookup into a separate clasErik Johnston2019-08-072-153/+197
|
* Add a lower bound for TTL on well known results.Erik Johnston2019-08-061-0/+4
| | | | | | | It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".
* Handle incorrectly encoded query params correctlyErik Johnston2019-08-011-1/+6
|
* Share SSL options for well-known requestsErik Johnston2019-07-311-11/+5
|
* Merge tag 'v1.2.0rc2' into developAndrew Morgan2019-07-242-14/+31
|\ | | | | | | | | | | | | Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
| * Fix servlet metric names (#5734)Jorik Schellekens2019-07-242-14/+31
| | | | | | | | | | | | | | | | | | | | * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths
* | Replace returnValue with return (#5736)Amber Brown2019-07-234-53/+45
|/
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-112-5/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-047-7/+7
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-10/+16
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-291-36/+41
|
* Run Black. (#5482)Amber Brown2019-06-2010-287/+321
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-4/+9
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Improve docstrings on MatrixFederationClient. (#5332)Richard van der Hoff2019-06-041-16/+55
|
* Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) v0.99.5.1.dev0Richard van der Hoff2019-05-291-4/+0
| | | This is just unhelpful spam
* Fix error code for invalid parameterAaron Raimist2019-05-241-1/+1
| | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-132-14/+40
|
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-20/+25
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Config option for verifying federation certificates (MSC 1711) (#4967)Andrew Morgan2019-04-251-1/+1
|
* Make federation endpoints more tolerant of trailing slashes v2 (#4935)Andrew Morgan2019-03-261-0/+1
| | | Redo of https://github.com/matrix-org/synapse/pull/4840
* Clean up backoff_on_404 and metehod callsAndrew Morgan2019-03-211-9/+13
|
* kwargs doesn't like commas on calling funcs either. TILAndrew Morgan2019-03-201-2/+2
|
* lintAndrew Morgan2019-03-201-2/+6
|
* New test, fix issuesAndrew Morgan2019-03-201-49/+28
|
* Fix comments. v0.99.2 -> v0.99.3Andrew Morgan2019-03-201-5/+5
|
* Just return if not doing any trailing slash shennanigansAndrew Morgan2019-03-201-0/+2
|
* lintAndrew Morgan2019-03-181-1/+1
|
* Better exception handlingAndrew Morgan2019-03-181-16/+17
|
* Correct var nameAndrew Morgan2019-03-131-2/+2
|
* receiving a 400 caused an exception. handle itAndrew Morgan2019-03-131-5/+12
|
* is this what purgatory feels likeAndrew Morgan2019-03-131-3/+3
|
* i should have given up x2Andrew Morgan2019-03-131-18/+13
|
* i should have given upAndrew Morgan2019-03-131-1/+1
|
* there comes a time when you should give up. but you dontAndrew Morgan2019-03-131-1/+1
|
* as aboveAndrew Morgan2019-03-131-2/+4
|
* no kwargs todayAndrew Morgan2019-03-131-7/+12
|
* Or perhaps I was the one who was drunkAndrew Morgan2019-03-131-6/+3
|
* go home python, you're drunkAndrew Morgan2019-03-131-4/+2
|
* Syntax checker is borkAndrew Morgan2019-03-131-6/+5
|
* Destructure againAndrew Morgan2019-03-131-7/+8
|
* Are you happy nowAndrew Morgan2019-03-131-2/+2
|
* Syntax testAndrew Morgan2019-03-131-6/+5
|
* Remove testing codeAndrew Morgan2019-03-131-3/+0
|
* Add missing docstring detailAndrew Morgan2019-03-131-1/+1
|
* Fix syntax issuesAndrew Morgan2019-03-131-7/+10
|
* Switch to wrapper function around _send_requestAndrew Morgan2019-03-131-32/+71
|
* Retry on 400:M_UNRECOGNIZEDAndrew Morgan2019-03-131-6/+10
|
* Fix paranthesis indentAndrew Morgan2019-03-131-1/+2
|
* Add workaround noteAndrew Morgan2019-03-131-10/+4
|
* Cleaner way of implementing trailing slashesAndrew Morgan2019-03-121-63/+52
|
* Retry certain federation requests on 404Andrew Morgan2019-03-081-0/+45
|
* Minor docstring fixes for MatrixFederationAgent (#4765)Andrew Morgan2019-02-281-1/+5
|
* Log tracebacks correctlyErik Johnston2019-02-251-4/+4
|
* Fix TaskStopped exceptions when outbound requests time out (#4690)Richard van der Hoff2019-02-203-18/+29
|
* Cleanup top level request exception loggingErik Johnston2019-02-181-3/+3
| | | | | | | | | | | Firstly, we always logged that the request was being handled via `JsonResource._async_render`, so we change that to use the servlet name we add to the request. Secondly, we pass the exception information to the logger rather than formatting it manually. This makes it consistent with other exception logging, allwoing logging hooks and formatters to access the exception information.
* Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)Richard van der Hoff2019-02-111-2/+2
| | | | | It's nothing to do with refreshing the certificates. No idea why it was here.
* Merge pull request #4546 from ↵Richard van der Hoff2019-02-011-2/+15
|\ | | | | | | | | matrix-org/rav/silence_critical_error_from_federation Fix noisy "twisted.internet.task.TaskStopped" errors in logs
| * Fix noisy "twisted.internet.task.TaskStopped" errors in logsRichard van der Hoff2019-02-011-2/+15
| | | | | | | | Fixes #4003
* | Merge pull request #4544 from matrix-org/rav/skip_invalid_well_knownRichard van der Hoff2019-02-011-19/+6
|\ \ | | | | | | Treat an invalid .well-known the same as an absent one
| * | Treat an invalid .well-known the same as an absent oneRichard van der Hoff2019-02-011-19/+6
| | | | | | | | | | | | ... basically, carry on and fall back to SRV etc.
* | | Fix b'ab' noise in logsRichard van der Hoff2019-02-011-1/+1
|/ /
* | Cache failures to parse .well-knownRichard van der Hoff2019-02-011-13/+43
| | | | | | | | Also add a Measure block around the .well-known fetch
* | better logging for federation connectionsRichard van der Hoff2019-01-311-5/+16
| |
* | Update federation routing logic to check .well-known before SRVRichard van der Hoff2019-01-311-5/+5
|/
* Merge pull request #4521 from matrix-org/rav/fed_routing/cleanupsRichard van der Hoff2019-01-301-9/+11
|\ | | | | Tiny .well-known fixes
| * fix exception textRichard van der Hoff2019-01-301-1/+1
| |
| * Also jitter the invalid cache periodRichard van der Hoff2019-01-301-9/+11
| |
* | Follow redirects on .well-known (#4520)Richard van der Hoff2019-01-301-2/+4
|/
* Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-2/+88
|
* Relax requirement for a content-type on .well-known (#4511)Richard van der Hoff2019-01-291-20/+13
|
* Implement MSC1708 (.well-known lookups for server routing) (#4489)Richard van der Hoff2019-01-291-4/+110
|
* docstrings for _RoutingResultRichard van der Hoff2019-01-281-0/+36
|
* Handle IP literals explicitlyRichard van der Hoff2019-01-281-0/+19
| | | | We don't want to be doing .well-known lookups on these guys.
* MatrixFederationAgent: factor out routing logicRichard van der Hoff2019-01-281-18/+62
| | | | This is going to get too big and unmanageable.
* Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487)Richard van der Hoff2019-01-281-11/+12
| | | | | | | | | | | | | | | | Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.
* Fix Host header sent by MatrixFederationAgent (#4468)Richard van der Hoff2019-01-252-1/+10
| | | | | | Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.
* Merge pull request #4464 from matrix-org/rav/fix_srv_lookupRichard van der Hoff2019-01-241-1/+2
|\ | | | | MatrixFederationAgent: Look up the right SRV record
| * Look up the right SRV recordRichard van der Hoff2019-01-241-1/+2
| |
* | Fix UnboundLocalError in post_urlencoded_get_json (#4460)Richard van der Hoff2019-01-241-2/+3
|/ | | This could cause exceptions if the id server returned 4xx responses.
* put resolve_service in an objectRichard van der Hoff2019-01-222-63/+86
| | | | this makes it easier to stub things out for tests.
* Require that service_name be a byte stringRichard van der Hoff2019-01-221-4/+4
| | | | it is only ever a bytes now, so let's enforce that.
* Kill off matrix_federation_endpointRichard van der Hoff2019-01-222-145/+0
| | | | this thing is now redundant.
* Make MatrixFederationClient use MatrixFederationAgentRichard van der Hoff2019-01-221-27/+10
| | | | ... instead of the matrix_federation_endpoint
* MatrixFederationAgentRichard van der Hoff2019-01-222-0/+147
| | | | | Pull the magic that is currently in matrix_federation_endpoint and friends into an agent-like thing
* Refactor and bugfix for resove_service (#4427)Richard van der Hoff2019-01-223-72/+141
|
* Remove redundant WrappedConnection (#4409)Richard van der Hoff2019-01-182-86/+19
| | | | | | | | * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.
* sign_request -> build_auth_headers (#4408)Richard van der Hoff2019-01-171-12/+11
| | | | | Just got very confused about the fact that the headers are only an output, not an input.
* Fixup docstrings for matrixfederationclientErik Johnston2019-01-091-68/+73
|
* Use RequestSendFailed when fail to parse content type headersErik Johnston2019-01-081-5/+5
|
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-33/+72
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Fix IP URL previews on Python 3 (#4215)Amber Brown2018-12-222-169/+243
|
* fix parse_string docstringRichard van der Hoff2018-11-081-5/+4
|
* Fix URL preview bugs (type error when loading cache from db, content-type ↵Amber Brown2018-11-081-4/+4
| | | | including quotes) (#4157)
* Merge pull request #3969 from turt2live/travis/fix-federated-group-requestsTravis Ralston2018-10-231-6/+6
|\ | | | | Handle HttpResponseException more safely for federated groups
| * Merge branch 'develop' into travis/fix-federated-group-requestsTravis Ralston2018-10-121-6/+21
| |\
| * | Fix exception documentation in matrixfederationclient.pyTravis Ralston2018-09-261-6/+6
| | |
* | | Make the metrics less racy (#4061)Amber Brown2018-10-191-13/+18
| | |
* | | Various cleanups in the federation client code (#4031)Richard van der Hoff2018-10-161-38/+40
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | - Improve logging: log things in the right order, include destination and txids in all log lines, don't log successful responses twice - Fix the docstring on TransportLayerClient.send_transaction - Don't use treq.request, which is overcomplicated for our purposes: just use a twisted.web.client.Agent. - simplify the logic for setting up the bodyProducer - fix bytes/str confusions
* | Fix error when logging incomplete requestsErik Johnston2018-10-021-6/+21
|/ | | | | | | | | | If a connection is lost before a request is read from Request, Twisted sets `method` (and `uri`) attributes to dummy values. These dummy values have incorrect types (i.e. they're not bytes), and so things like `__repr__` would raise an exception. To fix this we had a helper method to return the method with a consistent type.
* Merge pull request #3925 from matrix-org/erikj/fix_producers_unregisteredRichard van der Hoff2018-09-251-14/+35
|\ | | | | Fix spurious exceptions when client closes conncetion
| * Fix spurious exceptions when client closes conncetionErik Johnston2018-09-201-14/+35
| | | | | | | | | | | | | | | | | | If a HTTP handler throws an exception while processing a request we automatically write a JSON error response. If the handler had already started writing a response twisted throws an exception. We should check for this case and simple abort the connection if there was an error after the response had started being written.
* | Fix client IPs being broken on Python 3 (#3908)Amber Brown2018-09-201-1/+1
|/
* Add a regression test for logging on failed connections (#3912)Amber Brown2018-09-201-2/+2
|
* Update to use new timeout function everywhere.Erik Johnston2018-09-192-12/+17
| | | | | | | The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.
* Merge pull request #3909 from turt2live/travis/fix-logging-1Amber Brown2018-09-191-4/+4
|\ | | | | Fix matrixfederationclient.py logging: Destination is a string
| * Destination is a stringTravis Ralston2018-09-181-4/+4
| |
* | Merge pull request #3907 from matrix-org/rav/set_sni_to_server_nameAmber Brown2018-09-191-3/+10
|\ \ | |/ |/| Set SNI to the server_name, not whatever was in the SRV record
| * Set SNI to the server_name, not whatever was in the SRV recordRichard van der Hoff2018-09-181-3/+10
| | | | | | | | Fixes #3843
* | Refactor matrixfederationclient to fix logging (#3906)Richard van der Hoff2018-09-181-149/+236
|/ | | | | | | | We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.
* Fix more b'abcd' noise in metricsRichard van der Hoff2018-09-172-12/+12
|
* Fix some b'abcd' noise in logs and metricsRichard van der Hoff2018-09-172-5/+8
| | | | | Python 3 compatibility: make sure that we decode some byte sequences before we use them to create log lines and metrics labels.
* Add an awful secondary timeout to fix wedged requestsErik Johnston2018-09-141-0/+11
| | | | This is an attempt to mitigate #3842 by adding yet-another-timeout
* Merge pull request #3872 from matrix-org/hawkowl/timeouts-2Erik Johnston2018-09-144-7/+12
|\ | | | | timeouts 2: electric boogaloo
| * fixAmber Brown2018-09-151-3/+3
| |
| * fixAmber Brown2018-09-141-1/+4
| |
| * fixAmber Brown2018-09-141-1/+1
| |
| * redact betterAmber Brown2018-09-141-2/+2
| |
| * we do that higher upAmber Brown2018-09-141-0/+1
| |
| * Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2Amber Brown2018-09-141-4/+10
| |\
| * | buffer?Amber Brown2018-09-141-1/+2
| | |
* | | Measure outbound requestsErik Johnston2018-09-141-3/+6
| |/ |/|
* | merge (#3576)Amber Brown2018-09-141-4/+10
|/
* Attempt to figure out what's going on with timeouts (#3857)Amber Brown2018-09-141-56/+42
|
* Log outbound requests when we retryErik Johnston2018-09-121-75/+74
|
* Timeout reading body for outbound HTTP requestsErik Johnston2018-09-121-7/+45
|
* Port http/ to Python 3 (#3771)Amber Brown2018-09-063-173/+117
|
* Fix bug when federation_domain_whitelist is an emtpy listErik Johnston2018-08-241-1/+1
| | | | | Outbound federation were incorrectly allowed when the config option was set to an empty list