summary refs log tree commit diff
path: root/synapse/http (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-2/+2
|
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-294-50/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* A pair of tiny cleanups in the federation request code. (#8401)Richard van der Hoff2020-09-281-2/+0
|
* Add type annotations to SimpleHttpClient (#8372)Richard van der Hoff2020-09-241-56/+131
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-1/+1
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-142-2/+2
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-101-1/+4
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-049-18/+18
|
* Convert the well known resolver to async (#8214)Patrick Cloke2020-09-012-28/+33
|
* Switch the JSON byte producer from a pull to a push producer. (#8116)Patrick Cloke2020-08-191-32/+43
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-193-11/+10
|
* Iteratively encode JSON responses to avoid blocking the reactor. (#8013)Patrick Cloke2020-08-181-8/+89
|
* Reduce INFO logging (#8050)Erik Johnston2020-08-113-25/+73
| | | | | | | | | | c.f. #8021 A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line. Some notes: 1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps. 2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
* Don't log OPTIONS request at INFO (#8049)Erik Johnston2020-08-071-1/+7
|
* Add health check endpoint (#8048)Erik Johnston2020-08-071-1/+8
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-071-2/+3
|
* Convert run_as_background_process inner function to async. (#8032)Patrick Cloke2020-08-061-3/+2
|
* Convert the SimpleHttpClient to async. (#8016)Patrick Cloke2020-08-041-31/+24
|
* Implement handling of HTTP HEAD requests. (#7999)Patrick Cloke2020-08-031-5/+11
|
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-40/+32
|
* Ensure the msg property of HttpResponseException is a string. (#7979)Patrick Cloke2020-07-292-7/+16
|
* Return an empty body for OPTIONS requests. (#7886)Patrick Cloke2020-07-241-19/+5
|
* Downgrade warning on client disconnect to INFO (#7928)Richard van der Hoff2020-07-241-3/+1
| | | | Clients disconnecting before we finish processing the request happens from time to time. We don't need to yell about it
* Convert the federation agent and related code to async/await. (#7874)Patrick Cloke2020-07-232-16/+10
|
* fix an incorrect commentRichard van der Hoff2020-07-221-2/+2
|
* Ensure that calls to `json.dumps` are compatible with the standard library ↵Patrick Cloke2020-07-152-3/+5
| | | | json. (#7836)
* Fix client reader sharding tests (#7853)Erik Johnston2020-07-151-1/+23
| | | | | | | | | | | | | | | * Fix client reader sharding tests * Newsfile * Fix typing * Update changelog.d/7853.misc Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Move mocking of http_client to tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802)Patrick Cloke2020-07-102-19/+9
|
* Add types to the server code and remove unused parameter (#7813)Patrick Cloke2020-07-101-30/+41
|
* Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-1/+1
| | | ... instead of duplicating `config.signing_key[0]` everywhere
* Stop passing bytes when dumping JSON (#7799)Patrick Cloke2020-07-081-3/+7
|
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-032-189/+195
|
* Merge branch 'master' into developPatrick Cloke2020-07-021-8/+68
|\
| * Ensure that HTML pages served from Synapse include headers to avoid embedding.Patrick Cloke2020-07-021-8/+68
| |
* | Include a user agent in federation requests. (#7677)Patrick Cloke2020-06-163-4/+32
| |
* | Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-163-14/+10
| |
* | Create a ListenerConfig object (#7681)Richard van der Hoff2020-06-161-2/+4
|/ | | | | | | | | | This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.
* Clean up exception handling in SAML2ResponseResource (#7614)Richard van der Hoff2020-06-031-12/+31
| | | | | | | | | | | | | * Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string * Clean up exception handling in SAML2ResponseResource * use the existing code in `return_html_error` instead of re-implementing it (giving it a jinja2 template rather than inventing a new form of template) * do the exception-catching in the REST layer rather than in the handler layer, to make sure we catch all exceptions.
* Fix missing CORS headers on OPTION responses (#7560)Erik Johnston2020-05-221-1/+1
| | | Broke in #7534.
* Return 200 OK for all OPTIONS requests (#7534)Patrick Cloke2020-05-221-3/+20
|
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-0/+7
| | | | These are business as usual errors, rather than stuff we want to log at error.
* mypy for synapse.http.site (#7553)Richard van der Hoff2020-05-221-3/+6
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-6/+2
|
* Workaround for failure to wrap reason in Failure (#7473)Andrew Morgan2020-05-141-0/+7
|
* Fix b'GET' in prometheus metrics (#7503)Richard van der Hoff2020-05-141-2/+4
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-2/+4
| | | | variables (#6391)
* Implement OpenID Connect-based login (#7256)Quentin Gliech2020-05-081-0/+7
|
* Reduce federation logging on success (#7321)Michael Kaye2020-04-221-8/+14
| | | | Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
* Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181)Richard van der Hoff2020-03-311-7/+6
| | | | Fixes #7179.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-241-3/+3
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Share SSL contexts for non-federation requests (#7094)Richard van der Hoff2020-03-172-4/+1
| | | | | | | Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests. Fixes #7092.
* Tiny optimisation for _get_handler_for_request (#6950)Richard van der Hoff2020-02-191-1/+3
| | | | we have hundreds of path_regexes (see #5118), so let's not convert the same bytes to str for each of them.
* Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-1/+1
| | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* Fix outbound federation request metrics (#6795)Erik Johnston2020-01-281-0/+4
|
* Implement RedirectException (#6687)Richard van der Hoff2020-01-151-4/+9
| | | | | Allow REST endpoint implemnentations to raise a RedirectException, which will redirect the user's browser to a given location.
* Fix exceptions on requests for non-ascii urls (#6682)Richard van der Hoff2020-01-131-1/+1
| | | Fixes #6402
* Kill off redundant SynapseRequestFactory (#6619)Richard van der Hoff2020-01-031-15/+3
| | | | We already get the Site via the Channel, so there's no need for a dedicated RequestFactory: we can just use the right constructor.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
|
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-013-2/+405
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-317-14/+14
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-1/+1
|
* Merge branch 'master' into developAndrew Morgan2019-10-031-2/+4
|\
| * Replace client_secret with <redacted> in server logs (#6158)Andrew Morgan2019-10-031-2/+4
| | | | | | Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
* | Edit SimpleHttpClient to reference that header keys can be passed as str or ↵Andrew Morgan2019-09-271-6/+6
|/ | | | bytes (#6077)
* Fix well-known lookups with the federation certificate whitelist (#5997)Amber Brown2019-09-141-1/+1
|
* Trace how long it takes for the send trasaction to complete, including ↵Jorik Schellekens2019-09-052-37/+51
| | | | retrys (#5986)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-052-6/+13
|
* Remove unused methods from c/s api v1 in register.py (#5963)Andrew Morgan2019-09-021-33/+1
| | | These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Merge pull request #5864 from matrix-org/erikj/reliable_lookupsErik Johnston2019-08-272-208/+217
|\ | | | | Refactor MatrixFederationAgent to retry SRV.
| * Fix off by one error in SRV result shufflingErik Johnston2019-08-271-8/+13
| |
| * Fixup commentsErik Johnston2019-08-231-1/+9
| |
| * Fix logcontextsErik Johnston2019-08-231-4/+8
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-08-223-47/+128
| |\ | | | | | | | | | erikj/reliable_lookups
| * | Remove now unused pick_server_from_listErik Johnston2019-08-201-30/+0
| | |
| * | Fixup _sort_server_list to be slightly more efficientErik Johnston2019-08-201-2/+11
| | | | | | | | | | | | | | | Also document that we are using the algorithm described in RFC2782 and ensure we handle zero weight correctly.
| * | Refactor MatrixFederationAgent to retry SRV.Erik Johnston2019-08-152-189/+202
| | | | | | | | | | | | | | | | | | | | | | | | This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool
* | | Opentracing across workers (#5771)Jorik Schellekens2019-08-221-1/+1
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* | Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_failErik Johnston2019-08-222-37/+115
|\ \ | | | | | | Retry well known on fail
| * | Change jitter to be a factor rather than absolute valueErik Johnston2019-08-201-11/+12
| | |
| * | Fixup changelog and remove debug loggingErik Johnston2019-08-161-4/+1
| | |
| * | Retry well known on fail.Erik Johnston2019-08-152-34/+114
| |/ | | | | | | | | | | If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.
* / Opentracing misc (#5856)Jorik Schellekens2019-08-161-10/+13
|/ | | | | | | | | | | Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-22/+60
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Make default well known cache global again.Erik Johnston2019-08-131-1/+4
|
* Move well known lookup into a separate clasErik Johnston2019-08-072-153/+197
|
* Add a lower bound for TTL on well known results.Erik Johnston2019-08-061-0/+4
| | | | | | | It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".
* Handle incorrectly encoded query params correctlyErik Johnston2019-08-011-1/+6
|
* Share SSL options for well-known requestsErik Johnston2019-07-311-11/+5
|
* Merge tag 'v1.2.0rc2' into developAndrew Morgan2019-07-242-14/+31
|\ | | | | | | | | | | | | Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
| * Fix servlet metric names (#5734)Jorik Schellekens2019-07-242-14/+31
| | | | | | | | | | | | | | | | | | | | * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths
* | Replace returnValue with return (#5736)Amber Brown2019-07-234-53/+45
|/
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-112-5/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-047-7/+7
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-10/+16
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-291-36/+41
|
* Run Black. (#5482)Amber Brown2019-06-2010-287/+321
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-4/+9
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Improve docstrings on MatrixFederationClient. (#5332)Richard van der Hoff2019-06-041-16/+55
|
* Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) v0.99.5.1.dev0Richard van der Hoff2019-05-291-4/+0
| | | This is just unhelpful spam
* Fix error code for invalid parameterAaron Raimist2019-05-241-1/+1
| | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-132-14/+40
|
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-20/+25
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Config option for verifying federation certificates (MSC 1711) (#4967)Andrew Morgan2019-04-251-1/+1
|
* Make federation endpoints more tolerant of trailing slashes v2 (#4935)Andrew Morgan2019-03-261-0/+1
| | | Redo of https://github.com/matrix-org/synapse/pull/4840
* Clean up backoff_on_404 and metehod callsAndrew Morgan2019-03-211-9/+13
|
* kwargs doesn't like commas on calling funcs either. TILAndrew Morgan2019-03-201-2/+2
|
* lintAndrew Morgan2019-03-201-2/+6
|
* New test, fix issuesAndrew Morgan2019-03-201-49/+28
|
* Fix comments. v0.99.2 -> v0.99.3Andrew Morgan2019-03-201-5/+5
|
* Just return if not doing any trailing slash shennanigansAndrew Morgan2019-03-201-0/+2
|
* lintAndrew Morgan2019-03-181-1/+1
|
* Better exception handlingAndrew Morgan2019-03-181-16/+17
|
* Correct var nameAndrew Morgan2019-03-131-2/+2
|
* receiving a 400 caused an exception. handle itAndrew Morgan2019-03-131-5/+12
|
* is this what purgatory feels likeAndrew Morgan2019-03-131-3/+3
|
* i should have given up x2Andrew Morgan2019-03-131-18/+13
|
* i should have given upAndrew Morgan2019-03-131-1/+1
|
* there comes a time when you should give up. but you dontAndrew Morgan2019-03-131-1/+1
|
* as aboveAndrew Morgan2019-03-131-2/+4
|
* no kwargs todayAndrew Morgan2019-03-131-7/+12
|
* Or perhaps I was the one who was drunkAndrew Morgan2019-03-131-6/+3
|
* go home python, you're drunkAndrew Morgan2019-03-131-4/+2
|
* Syntax checker is borkAndrew Morgan2019-03-131-6/+5
|
* Destructure againAndrew Morgan2019-03-131-7/+8
|
* Are you happy nowAndrew Morgan2019-03-131-2/+2
|
* Syntax testAndrew Morgan2019-03-131-6/+5
|
* Remove testing codeAndrew Morgan2019-03-131-3/+0
|
* Add missing docstring detailAndrew Morgan2019-03-131-1/+1
|
* Fix syntax issuesAndrew Morgan2019-03-131-7/+10
|
* Switch to wrapper function around _send_requestAndrew Morgan2019-03-131-32/+71
|
* Retry on 400:M_UNRECOGNIZEDAndrew Morgan2019-03-131-6/+10
|
* Fix paranthesis indentAndrew Morgan2019-03-131-1/+2
|
* Add workaround noteAndrew Morgan2019-03-131-10/+4
|
* Cleaner way of implementing trailing slashesAndrew Morgan2019-03-121-63/+52
|
* Retry certain federation requests on 404Andrew Morgan2019-03-081-0/+45
|
* Minor docstring fixes for MatrixFederationAgent (#4765)Andrew Morgan2019-02-281-1/+5
|
* Log tracebacks correctlyErik Johnston2019-02-251-4/+4
|
* Fix TaskStopped exceptions when outbound requests time out (#4690)Richard van der Hoff2019-02-203-18/+29
|
* Cleanup top level request exception loggingErik Johnston2019-02-181-3/+3
| | | | | | | | | | | Firstly, we always logged that the request was being handled via `JsonResource._async_render`, so we change that to use the servlet name we add to the request. Secondly, we pass the exception information to the logger rather than formatting it manually. This makes it consistent with other exception logging, allwoing logging hooks and formatters to access the exception information.
* Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)Richard van der Hoff2019-02-111-2/+2
| | | | | It's nothing to do with refreshing the certificates. No idea why it was here.
* Merge pull request #4546 from ↵Richard van der Hoff2019-02-011-2/+15
|\ | | | | | | | | matrix-org/rav/silence_critical_error_from_federation Fix noisy "twisted.internet.task.TaskStopped" errors in logs
| * Fix noisy "twisted.internet.task.TaskStopped" errors in logsRichard van der Hoff2019-02-011-2/+15
| | | | | | | | Fixes #4003
* | Merge pull request #4544 from matrix-org/rav/skip_invalid_well_knownRichard van der Hoff2019-02-011-19/+6
|\ \ | | | | | | Treat an invalid .well-known the same as an absent one
| * | Treat an invalid .well-known the same as an absent oneRichard van der Hoff2019-02-011-19/+6
| | | | | | | | | | | | ... basically, carry on and fall back to SRV etc.
* | | Fix b'ab' noise in logsRichard van der Hoff2019-02-011-1/+1
|/ /
* | Cache failures to parse .well-knownRichard van der Hoff2019-02-011-13/+43
| | | | | | | | Also add a Measure block around the .well-known fetch
* | better logging for federation connectionsRichard van der Hoff2019-01-311-5/+16
| |
* | Update federation routing logic to check .well-known before SRVRichard van der Hoff2019-01-311-5/+5
|/
* Merge pull request #4521 from matrix-org/rav/fed_routing/cleanupsRichard van der Hoff2019-01-301-9/+11
|\ | | | | Tiny .well-known fixes
| * fix exception textRichard van der Hoff2019-01-301-1/+1
| |
| * Also jitter the invalid cache periodRichard van der Hoff2019-01-301-9/+11
| |
* | Follow redirects on .well-known (#4520)Richard van der Hoff2019-01-301-2/+4
|/
* Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-2/+88
|
* Relax requirement for a content-type on .well-known (#4511)Richard van der Hoff2019-01-291-20/+13
|
* Implement MSC1708 (.well-known lookups for server routing) (#4489)Richard van der Hoff2019-01-291-4/+110
|
* docstrings for _RoutingResultRichard van der Hoff2019-01-281-0/+36
|
* Handle IP literals explicitlyRichard van der Hoff2019-01-281-0/+19
| | | | We don't want to be doing .well-known lookups on these guys.
* MatrixFederationAgent: factor out routing logicRichard van der Hoff2019-01-281-18/+62
| | | | This is going to get too big and unmanageable.
* Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487)Richard van der Hoff2019-01-281-11/+12
| | | | | | | | | | | | | | | | Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.
* Fix Host header sent by MatrixFederationAgent (#4468)Richard van der Hoff2019-01-252-1/+10
| | | | | | Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.
* Merge pull request #4464 from matrix-org/rav/fix_srv_lookupRichard van der Hoff2019-01-241-1/+2
|\ | | | | MatrixFederationAgent: Look up the right SRV record
| * Look up the right SRV recordRichard van der Hoff2019-01-241-1/+2
| |
* | Fix UnboundLocalError in post_urlencoded_get_json (#4460)Richard van der Hoff2019-01-241-2/+3
|/ | | This could cause exceptions if the id server returned 4xx responses.
* put resolve_service in an objectRichard van der Hoff2019-01-222-63/+86
| | | | this makes it easier to stub things out for tests.
* Require that service_name be a byte stringRichard van der Hoff2019-01-221-4/+4
| | | | it is only ever a bytes now, so let's enforce that.
* Kill off matrix_federation_endpointRichard van der Hoff2019-01-222-145/+0
| | | | this thing is now redundant.
* Make MatrixFederationClient use MatrixFederationAgentRichard van der Hoff2019-01-221-27/+10
| | | | ... instead of the matrix_federation_endpoint
* MatrixFederationAgentRichard van der Hoff2019-01-222-0/+147
| | | | | Pull the magic that is currently in matrix_federation_endpoint and friends into an agent-like thing
* Refactor and bugfix for resove_service (#4427)Richard van der Hoff2019-01-223-72/+141
|
* Remove redundant WrappedConnection (#4409)Richard van der Hoff2019-01-182-86/+19
| | | | | | | | * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.
* sign_request -> build_auth_headers (#4408)Richard van der Hoff2019-01-171-12/+11
| | | | | Just got very confused about the fact that the headers are only an output, not an input.
* Fixup docstrings for matrixfederationclientErik Johnston2019-01-091-68/+73
|
* Use RequestSendFailed when fail to parse content type headersErik Johnston2019-01-081-5/+5
|
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-33/+72
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Fix IP URL previews on Python 3 (#4215)Amber Brown2018-12-222-169/+243
|
* fix parse_string docstringRichard van der Hoff2018-11-081-5/+4
|
* Fix URL preview bugs (type error when loading cache from db, content-type ↵Amber Brown2018-11-081-4/+4
| | | | including quotes) (#4157)
* Merge pull request #3969 from turt2live/travis/fix-federated-group-requestsTravis Ralston2018-10-231-6/+6
|\ | | | | Handle HttpResponseException more safely for federated groups
| * Merge branch 'develop' into travis/fix-federated-group-requestsTravis Ralston2018-10-121-6/+21
| |\
| * | Fix exception documentation in matrixfederationclient.pyTravis Ralston2018-09-261-6/+6
| | |
* | | Make the metrics less racy (#4061)Amber Brown2018-10-191-13/+18
| | |
* | | Various cleanups in the federation client code (#4031)Richard van der Hoff2018-10-161-38/+40
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | - Improve logging: log things in the right order, include destination and txids in all log lines, don't log successful responses twice - Fix the docstring on TransportLayerClient.send_transaction - Don't use treq.request, which is overcomplicated for our purposes: just use a twisted.web.client.Agent. - simplify the logic for setting up the bodyProducer - fix bytes/str confusions
* | Fix error when logging incomplete requestsErik Johnston2018-10-021-6/+21
|/ | | | | | | | | | If a connection is lost before a request is read from Request, Twisted sets `method` (and `uri`) attributes to dummy values. These dummy values have incorrect types (i.e. they're not bytes), and so things like `__repr__` would raise an exception. To fix this we had a helper method to return the method with a consistent type.
* Merge pull request #3925 from matrix-org/erikj/fix_producers_unregisteredRichard van der Hoff2018-09-251-14/+35
|\ | | | | Fix spurious exceptions when client closes conncetion
| * Fix spurious exceptions when client closes conncetionErik Johnston2018-09-201-14/+35
| | | | | | | | | | | | | | | | | | If a HTTP handler throws an exception while processing a request we automatically write a JSON error response. If the handler had already started writing a response twisted throws an exception. We should check for this case and simple abort the connection if there was an error after the response had started being written.
* | Fix client IPs being broken on Python 3 (#3908)Amber Brown2018-09-201-1/+1
|/
* Add a regression test for logging on failed connections (#3912)Amber Brown2018-09-201-2/+2
|
* Update to use new timeout function everywhere.Erik Johnston2018-09-192-12/+17
| | | | | | | The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.
* Merge pull request #3909 from turt2live/travis/fix-logging-1Amber Brown2018-09-191-4/+4
|\ | | | | Fix matrixfederationclient.py logging: Destination is a string
| * Destination is a stringTravis Ralston2018-09-181-4/+4
| |
* | Merge pull request #3907 from matrix-org/rav/set_sni_to_server_nameAmber Brown2018-09-191-3/+10
|\ \ | |/ |/| Set SNI to the server_name, not whatever was in the SRV record
| * Set SNI to the server_name, not whatever was in the SRV recordRichard van der Hoff2018-09-181-3/+10
| | | | | | | | Fixes #3843
* | Refactor matrixfederationclient to fix logging (#3906)Richard van der Hoff2018-09-181-149/+236
|/ | | | | | | | We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.
* Fix more b'abcd' noise in metricsRichard van der Hoff2018-09-172-12/+12
|
* Fix some b'abcd' noise in logs and metricsRichard van der Hoff2018-09-172-5/+8
| | | | | Python 3 compatibility: make sure that we decode some byte sequences before we use them to create log lines and metrics labels.
* Add an awful secondary timeout to fix wedged requestsErik Johnston2018-09-141-0/+11
| | | | This is an attempt to mitigate #3842 by adding yet-another-timeout
* Merge pull request #3872 from matrix-org/hawkowl/timeouts-2Erik Johnston2018-09-144-7/+12
|\ | | | | timeouts 2: electric boogaloo
| * fixAmber Brown2018-09-151-3/+3
| |
| * fixAmber Brown2018-09-141-1/+4
| |
| * fixAmber Brown2018-09-141-1/+1
| |
| * redact betterAmber Brown2018-09-141-2/+2
| |
| * we do that higher upAmber Brown2018-09-141-0/+1
| |
| * Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2Amber Brown2018-09-141-4/+10
| |\
| * | buffer?Amber Brown2018-09-141-1/+2
| | |
* | | Measure outbound requestsErik Johnston2018-09-141-3/+6
| |/ |/|
* | merge (#3576)Amber Brown2018-09-141-4/+10
|/
* Attempt to figure out what's going on with timeouts (#3857)Amber Brown2018-09-141-56/+42
|
* Log outbound requests when we retryErik Johnston2018-09-121-75/+74
|
* Timeout reading body for outbound HTTP requestsErik Johnston2018-09-121-7/+45
|
* Port http/ to Python 3 (#3771)Amber Brown2018-09-063-173/+117
|
* Fix bug when federation_domain_whitelist is an emtpy listErik Johnston2018-08-241-1/+1
| | | | | Outbound federation were incorrectly allowed when the config option was set to an empty list
* Merge tag 'v0.33.3rc2' into developRichard van der Hoff2018-08-211-1/+7
|\ | | | | | | | | | | | | | | Bugfixes -------- - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
| * Fix exceptions when a connection is closed before we read the headersRichard van der Hoff2018-08-201-1/+7
| | | | | | | | | | This fixes bugs introduced in #3700, by making sure that we behave sanely when an incoming connection is closed before the headers are read.
* | Fix typoErik Johnston2018-08-211-4/+4
| |
* | Make the in flight requests metrics thread safeErik Johnston2018-08-201-3/+10
|/
* Port over enough to get some sytests running on Python 3 (#3668)Amber Brown2018-08-202-16/+44
|
* Merge pull request #3701 from matrix-org/rav/use_producer_for_responsesRichard van der Hoff2018-08-171-4/+13
|\ | | | | Use a producer to stream back responses
| * Use a producer to stream back responsesRichard van der Hoff2018-08-151-4/+13
| | | | | | | | | | | | | | | | | | | | The problem with dumping all of the json response into the Request object at once is that doing so starts the timeout for the next request to be received: so if it takes longer than 60s to stream back the response to the client, the client never gets it. The correct solution is to use a Producer; then the timeout is only started once all of the content is sent over the TCP connection.
* | Refactor request logging codeRichard van der Hoff2018-08-152-92/+179
|/ | | | | | | | | | | | | | | This commit moves a bunch of the logic for deciding when to log the receipt and completion of HTTP requests into SynapseRequest, rather than in the request handling wrappers. Advantages of this are: * we get logs for *all* requests (including OPTIONS and HEADs), rather than just those that end up hitting handlers we've remembered to decorate correctly. * when a request handler wires up a Producer (as the media stuff does currently, and as other things will do soon), we log at the point that all of the traffic has been sent to the client.
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-102-2/+2
|
* Merge pull request #3439 from vojeroen/send_sni_for_federation_requestsRichard van der Hoff2018-08-102-8/+8
|\ | | | | send SNI for federation requests
| * Merge branch 'develop' into send_sni_for_federation_requestsRichard van der Hoff2018-07-273-8/+14
| |\
| * \ Merge remote-tracking branch 'upstream/develop' into ↵Jeroen2018-07-148-160/+105
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | send_sni_for_federation_requests # Conflicts: # synapse/crypto/context_factory.py
| * \ \ Merge branch 'develop' into send_sni_for_federation_requestsJeroen2018-07-096-21/+103
| |\ \ \ | | | | | | | | | | | | | | | | | | | | # Conflicts: # synapse/http/endpoint.py
| * | | | allow self-signed certificatesJeroen2018-06-261-1/+1
| | | | |
| * | | | formatting changes for pep8Jeroen2018-06-252-2/+3
| | | | |
| * | | | send SNI for federation requestsJeroen2018-06-242-8/+7
| | | | |
* | | | | include known room versions in outgoing make_joinsRichard van der Hoff2018-08-061-2/+5
| | | | |
* | | | | Kill off MatrixCodeMessageExceptionRichard van der Hoff2018-08-011-34/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.
* | | | | Be more careful which errors we send back over the C-S APIRichard van der Hoff2018-08-011-9/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We really shouldn't be sending all CodeMessageExceptions back over the C-S API; it will include things like 401s which we shouldn't proxy. That means that we need to explicitly turn a few HttpResponseExceptions into SynapseErrors in the federation layer. The effect of the latter is that the matrix errcode will get passed through correctly to calling clients, which might help with some of the random M_UNKNOWN errors when trying to join rooms.
* | | | | Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-022-11/+34
| |_|_|/ |/| | |
* | | | Move v1-only APIs into their own module & isolate deprecated ones (#3460)Amber Brown2018-07-191-2/+4
| | | |
* | | | pep8Erik Johnston2018-07-181-1/+2
| | | |
* | | | Add response code to response timer metricsErik Johnston2018-07-181-3/+5
| | | |
* | | | Refactor REST API tests to use explicit reactors (#3351)Amber Brown2018-07-171-2/+3
| | | |
* | | | rerun isort with latest versionKrombel2018-07-161-3/+1
| | | |
* | | | check isort by travisKrombel2018-07-162-2/+4
| |_|/ |/| |
* | | rename assert_params_in_request to assert_params_in_dictKrombel2018-07-131-1/+1
| | | | | | | | | | | | | | | | | | the method "assert_params_in_request" does handle dicts and not requests. A request body has to be parsed to json before this method can be used
* | | Refactor logcontext resource usage tracking (#3501)Richard van der Hoff2018-07-102-81/+26
| | | | | | | | | | | | | | | Factor out the resource usage tracking out to a separate object, which can be passed around and copied independently of the logcontext itself.
* | | run isortAmber Brown2018-07-098-78/+78
| |/ |/|
* | More server_name validationRichard van der Hoff2018-07-041-5/+42
| | | | | | | | | | | | | | | | We need to do a bit more validation when we get a server name, but don't want to be re-doing it all over the shop, so factor out a separate parse_and_validate_server_name, and do the extra validation. Also, use it to verify the server name in the config file.
* | Reject invalid server names (#3480)Richard van der Hoff2018-07-031-3/+31
| | | | | | | | | | Make sure that server_names used in auth headers are sane, and reject them with a sensible error code, before they disappear off into the depths of the system.
* | replace invalid utf8 with \ufffdMatthew Hodgson2018-07-021-2/+2
| |
* | a fix which doesn't NPE everywhereMatthew Hodgson2018-07-011-9/+17
| |
* | don't mix unicode strings with utf8-in-byte-stringsMatthew Hodgson2018-07-011-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | otherwise we explode with: ``` Traceback (most recent call last): File /usr/lib/python2.7/logging/handlers.py, line 78, in emit logging.FileHandler.emit(self, record) File /usr/lib/python2.7/logging/__init__.py, line 950, in emit StreamHandler.emit(self, record) File /usr/lib/python2.7/logging/__init__.py, line 887, in emit self.handleError(record) File /usr/lib/python2.7/logging/__init__.py, line 810, in handleError None, sys.stderr) File /usr/lib/python2.7/traceback.py, line 124, in print_exception _print(file, 'Traceback (most recent call last):') File /usr/lib/python2.7/traceback.py, line 13, in _print file.write(str+terminator) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_io.py, line 170, in write self.log.emit(self.level, format=u{log_io}, log_io=line) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 144, in emit self.observer(event) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 136, in __call__ errorLogger = self._errorLoggerForObserver(brokenObserver) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 156, in _errorLoggerForObserver if obs is not observer File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 81, in __init__ self.log = Logger(observer=self) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 64, in __init__ namespace = self._namespaceFromCallingContext() File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 42, in _namespaceFromCallingContext return currentframe(2).f_globals[__name__] File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/compat.py, line 93, in currentframe for x in range(n + 1): RuntimeError: maximum recursion depth exceeded while calling a Python object Logged from file site.py, line 129 File /usr/lib/python2.7/logging/__init__.py, line 859, in emit msg = self.format(record) File /usr/lib/python2.7/logging/__init__.py, line 732, in format return fmt.format(record) File /usr/lib/python2.7/logging/__init__.py, line 471, in format record.message = record.getMessage() File /usr/lib/python2.7/logging/__init__.py, line 335, in getMessage msg = msg % self.args UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128) Logged from file site.py, line 129 ``` ...where the logger apparently recurses whilst trying to log the error, hitting the maximum recursion depth and killing everything badly.
* | Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-284-8/+8
| |
* | Need to pass reactor to endpoint facErik Johnston2018-06-251-1/+1
| |
* | Remove all global reactor imports & pass it around explicitly (#3424)Amber Brown2018-06-251-7/+9
|/