summary refs log tree commit diff
path: root/synapse/http (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Kill off redundant SynapseRequestFactory (#6619)Richard van der Hoff2020-01-031-15/+3
| | | | We already get the Site via the Channel, so there's no need for a dedicated RequestFactory: we can just use the right constructor.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
|
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-013-2/+405
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-317-14/+14
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-1/+1
|
* Merge branch 'master' into developAndrew Morgan2019-10-031-2/+4
|\
| * Replace client_secret with <redacted> in server logs (#6158)Andrew Morgan2019-10-031-2/+4
| | | | | | Replace `client_secret` query parameter values with `<redacted>` in the logs. Prevents a scenario where a MITM of server traffic can horde 3pids on their account.
* | Edit SimpleHttpClient to reference that header keys can be passed as str or ↵Andrew Morgan2019-09-271-6/+6
|/ | | | bytes (#6077)
* Fix well-known lookups with the federation certificate whitelist (#5997)Amber Brown2019-09-141-1/+1
|
* Trace how long it takes for the send trasaction to complete, including ↵Jorik Schellekens2019-09-052-37/+51
| | | | retrys (#5986)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-052-6/+13
|
* Remove unused methods from c/s api v1 in register.py (#5963)Andrew Morgan2019-09-021-33/+1
| | | These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-1/+1
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Merge pull request #5864 from matrix-org/erikj/reliable_lookupsErik Johnston2019-08-272-208/+217
|\ | | | | Refactor MatrixFederationAgent to retry SRV.
| * Fix off by one error in SRV result shufflingErik Johnston2019-08-271-8/+13
| |
| * Fixup commentsErik Johnston2019-08-231-1/+9
| |
| * Fix logcontextsErik Johnston2019-08-231-4/+8
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-08-223-47/+128
| |\ | | | | | | | | | erikj/reliable_lookups
| * | Remove now unused pick_server_from_listErik Johnston2019-08-201-30/+0
| | |
| * | Fixup _sort_server_list to be slightly more efficientErik Johnston2019-08-201-2/+11
| | | | | | | | | | | | | | | Also document that we are using the algorithm described in RFC2782 and ensure we handle zero weight correctly.
| * | Refactor MatrixFederationAgent to retry SRV.Erik Johnston2019-08-152-189/+202
| | | | | | | | | | | | | | | | | | | | | | | | This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool
* | | Opentracing across workers (#5771)Jorik Schellekens2019-08-221-1/+1
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* | Merge pull request #5850 from matrix-org/erikj/retry_well_known_on_failErik Johnston2019-08-222-37/+115
|\ \ | | | | | | Retry well known on fail
| * | Change jitter to be a factor rather than absolute valueErik Johnston2019-08-201-11/+12
| | |
| * | Fixup changelog and remove debug loggingErik Johnston2019-08-161-4/+1
| | |
| * | Retry well known on fail.Erik Johnston2019-08-152-34/+114
| |/ | | | | | | | | | | If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.
* / Opentracing misc (#5856)Jorik Schellekens2019-08-161-10/+13
|/ | | | | | | | | | | Add authenticated_entity and servlet_names tags. Functionally: - Add a tag for authenticated_entity - Add a tag for servlet_names Stylistically: Moved to importing methods directly from opentracing.
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-22/+60
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Make default well known cache global again.Erik Johnston2019-08-131-1/+4
|
* Move well known lookup into a separate clasErik Johnston2019-08-072-153/+197
|
* Add a lower bound for TTL on well known results.Erik Johnston2019-08-061-0/+4
| | | | | | | It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".
* Handle incorrectly encoded query params correctlyErik Johnston2019-08-011-1/+6
|
* Share SSL options for well-known requestsErik Johnston2019-07-311-11/+5
|
* Merge tag 'v1.2.0rc2' into developAndrew Morgan2019-07-242-14/+31
|\ | | | | | | | | | | | | Bugfixes -------- - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
| * Fix servlet metric names (#5734)Jorik Schellekens2019-07-242-14/+31
| | | | | | | | | | | | | | | | | | | | * Fix servlet metric names Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Remove redundant check * Cover all return paths
* | Replace returnValue with return (#5736)Amber Brown2019-07-234-53/+45
|/
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-112-5/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-047-7/+7
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-10/+16
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-291-36/+41
|
* Run Black. (#5482)Amber Brown2019-06-2010-287/+321
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-4/+9
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Improve docstrings on MatrixFederationClient. (#5332)Richard van der Hoff2019-06-041-16/+55
|
* Remove spurious debug from MatrixFederationHttpClient.get_json (#5287) v0.99.5.1.dev0Richard van der Hoff2019-05-291-4/+0
| | | This is just unhelpful spam
* Fix error code for invalid parameterAaron Raimist2019-05-241-1/+1
| | | | Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-132-14/+40
|
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-20/+25
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Config option for verifying federation certificates (MSC 1711) (#4967)Andrew Morgan2019-04-251-1/+1
|
* Make federation endpoints more tolerant of trailing slashes v2 (#4935)Andrew Morgan2019-03-261-0/+1
| | | Redo of https://github.com/matrix-org/synapse/pull/4840
* Clean up backoff_on_404 and metehod callsAndrew Morgan2019-03-211-9/+13
|
* kwargs doesn't like commas on calling funcs either. TILAndrew Morgan2019-03-201-2/+2
|
* lintAndrew Morgan2019-03-201-2/+6
|
* New test, fix issuesAndrew Morgan2019-03-201-49/+28
|
* Fix comments. v0.99.2 -> v0.99.3Andrew Morgan2019-03-201-5/+5
|
* Just return if not doing any trailing slash shennanigansAndrew Morgan2019-03-201-0/+2
|
* lintAndrew Morgan2019-03-181-1/+1
|
* Better exception handlingAndrew Morgan2019-03-181-16/+17
|
* Correct var nameAndrew Morgan2019-03-131-2/+2
|
* receiving a 400 caused an exception. handle itAndrew Morgan2019-03-131-5/+12
|
* is this what purgatory feels likeAndrew Morgan2019-03-131-3/+3
|
* i should have given up x2Andrew Morgan2019-03-131-18/+13
|
* i should have given upAndrew Morgan2019-03-131-1/+1
|
* there comes a time when you should give up. but you dontAndrew Morgan2019-03-131-1/+1
|
* as aboveAndrew Morgan2019-03-131-2/+4
|
* no kwargs todayAndrew Morgan2019-03-131-7/+12
|
* Or perhaps I was the one who was drunkAndrew Morgan2019-03-131-6/+3
|
* go home python, you're drunkAndrew Morgan2019-03-131-4/+2
|
* Syntax checker is borkAndrew Morgan2019-03-131-6/+5
|
* Destructure againAndrew Morgan2019-03-131-7/+8
|
* Are you happy nowAndrew Morgan2019-03-131-2/+2
|
* Syntax testAndrew Morgan2019-03-131-6/+5
|
* Remove testing codeAndrew Morgan2019-03-131-3/+0
|
* Add missing docstring detailAndrew Morgan2019-03-131-1/+1
|
* Fix syntax issuesAndrew Morgan2019-03-131-7/+10
|
* Switch to wrapper function around _send_requestAndrew Morgan2019-03-131-32/+71
|
* Retry on 400:M_UNRECOGNIZEDAndrew Morgan2019-03-131-6/+10
|
* Fix paranthesis indentAndrew Morgan2019-03-131-1/+2
|
* Add workaround noteAndrew Morgan2019-03-131-10/+4
|
* Cleaner way of implementing trailing slashesAndrew Morgan2019-03-121-63/+52
|
* Retry certain federation requests on 404Andrew Morgan2019-03-081-0/+45
|
* Minor docstring fixes for MatrixFederationAgent (#4765)Andrew Morgan2019-02-281-1/+5
|
* Log tracebacks correctlyErik Johnston2019-02-251-4/+4
|
* Fix TaskStopped exceptions when outbound requests time out (#4690)Richard van der Hoff2019-02-203-18/+29
|
* Cleanup top level request exception loggingErik Johnston2019-02-181-3/+3
| | | | | | | | | | | Firstly, we always logged that the request was being handled via `JsonResource._async_render`, so we change that to use the servlet name we add to the request. Secondly, we pass the exception information to the logger rather than formatting it manually. This makes it consistent with other exception logging, allwoing logging hooks and formatters to access the exception information.
* Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)Richard van der Hoff2019-02-111-2/+2
| | | | | It's nothing to do with refreshing the certificates. No idea why it was here.
* Merge pull request #4546 from ↵Richard van der Hoff2019-02-011-2/+15
|\ | | | | | | | | matrix-org/rav/silence_critical_error_from_federation Fix noisy "twisted.internet.task.TaskStopped" errors in logs
| * Fix noisy "twisted.internet.task.TaskStopped" errors in logsRichard van der Hoff2019-02-011-2/+15
| | | | | | | | Fixes #4003
* | Merge pull request #4544 from matrix-org/rav/skip_invalid_well_knownRichard van der Hoff2019-02-011-19/+6
|\ \ | | | | | | Treat an invalid .well-known the same as an absent one
| * | Treat an invalid .well-known the same as an absent oneRichard van der Hoff2019-02-011-19/+6
| | | | | | | | | | | | ... basically, carry on and fall back to SRV etc.
* | | Fix b'ab' noise in logsRichard van der Hoff2019-02-011-1/+1
|/ /
* | Cache failures to parse .well-knownRichard van der Hoff2019-02-011-13/+43
| | | | | | | | Also add a Measure block around the .well-known fetch
* | better logging for federation connectionsRichard van der Hoff2019-01-311-5/+16
| |
* | Update federation routing logic to check .well-known before SRVRichard van der Hoff2019-01-311-5/+5
|/
* Merge pull request #4521 from matrix-org/rav/fed_routing/cleanupsRichard van der Hoff2019-01-301-9/+11
|\ | | | | Tiny .well-known fixes
| * fix exception textRichard van der Hoff2019-01-301-1/+1
| |
| * Also jitter the invalid cache periodRichard van der Hoff2019-01-301-9/+11
| |
* | Follow redirects on .well-known (#4520)Richard van der Hoff2019-01-301-2/+4
|/
* Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-2/+88
|
* Relax requirement for a content-type on .well-known (#4511)Richard van der Hoff2019-01-291-20/+13
|
* Implement MSC1708 (.well-known lookups for server routing) (#4489)Richard van der Hoff2019-01-291-4/+110
|
* docstrings for _RoutingResultRichard van der Hoff2019-01-281-0/+36
|
* Handle IP literals explicitlyRichard van der Hoff2019-01-281-0/+19
| | | | We don't want to be doing .well-known lookups on these guys.
* MatrixFederationAgent: factor out routing logicRichard van der Hoff2019-01-281-18/+62
| | | | This is going to get too big and unmanageable.
* Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487)Richard van der Hoff2019-01-281-11/+12
| | | | | | | | | | | | | | | | Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.
* Fix Host header sent by MatrixFederationAgent (#4468)Richard van der Hoff2019-01-252-1/+10
| | | | | | Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.
* Merge pull request #4464 from matrix-org/rav/fix_srv_lookupRichard van der Hoff2019-01-241-1/+2
|\ | | | | MatrixFederationAgent: Look up the right SRV record
| * Look up the right SRV recordRichard van der Hoff2019-01-241-1/+2
| |
* | Fix UnboundLocalError in post_urlencoded_get_json (#4460)Richard van der Hoff2019-01-241-2/+3
|/ | | This could cause exceptions if the id server returned 4xx responses.
* put resolve_service in an objectRichard van der Hoff2019-01-222-63/+86
| | | | this makes it easier to stub things out for tests.
* Require that service_name be a byte stringRichard van der Hoff2019-01-221-4/+4
| | | | it is only ever a bytes now, so let's enforce that.
* Kill off matrix_federation_endpointRichard van der Hoff2019-01-222-145/+0
| | | | this thing is now redundant.
* Make MatrixFederationClient use MatrixFederationAgentRichard van der Hoff2019-01-221-27/+10
| | | | ... instead of the matrix_federation_endpoint
* MatrixFederationAgentRichard van der Hoff2019-01-222-0/+147
| | | | | Pull the magic that is currently in matrix_federation_endpoint and friends into an agent-like thing
* Refactor and bugfix for resove_service (#4427)Richard van der Hoff2019-01-223-72/+141
|
* Remove redundant WrappedConnection (#4409)Richard van der Hoff2019-01-182-86/+19
| | | | | | | | * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.
* sign_request -> build_auth_headers (#4408)Richard van der Hoff2019-01-171-12/+11
| | | | | Just got very confused about the fact that the headers are only an output, not an input.
* Fixup docstrings for matrixfederationclientErik Johnston2019-01-091-68/+73
|
* Use RequestSendFailed when fail to parse content type headersErik Johnston2019-01-081-5/+5
|
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-33/+72
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Fix IP URL previews on Python 3 (#4215)Amber Brown2018-12-222-169/+243
|
* fix parse_string docstringRichard van der Hoff2018-11-081-5/+4
|
* Fix URL preview bugs (type error when loading cache from db, content-type ↵Amber Brown2018-11-081-4/+4
| | | | including quotes) (#4157)
* Merge pull request #3969 from turt2live/travis/fix-federated-group-requestsTravis Ralston2018-10-231-6/+6
|\ | | | | Handle HttpResponseException more safely for federated groups
| * Merge branch 'develop' into travis/fix-federated-group-requestsTravis Ralston2018-10-121-6/+21
| |\
| * | Fix exception documentation in matrixfederationclient.pyTravis Ralston2018-09-261-6/+6
| | |
* | | Make the metrics less racy (#4061)Amber Brown2018-10-191-13/+18
| | |
* | | Various cleanups in the federation client code (#4031)Richard van der Hoff2018-10-161-38/+40
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | - Improve logging: log things in the right order, include destination and txids in all log lines, don't log successful responses twice - Fix the docstring on TransportLayerClient.send_transaction - Don't use treq.request, which is overcomplicated for our purposes: just use a twisted.web.client.Agent. - simplify the logic for setting up the bodyProducer - fix bytes/str confusions
* | Fix error when logging incomplete requestsErik Johnston2018-10-021-6/+21
|/ | | | | | | | | | If a connection is lost before a request is read from Request, Twisted sets `method` (and `uri`) attributes to dummy values. These dummy values have incorrect types (i.e. they're not bytes), and so things like `__repr__` would raise an exception. To fix this we had a helper method to return the method with a consistent type.
* Merge pull request #3925 from matrix-org/erikj/fix_producers_unregisteredRichard van der Hoff2018-09-251-14/+35
|\ | | | | Fix spurious exceptions when client closes conncetion
| * Fix spurious exceptions when client closes conncetionErik Johnston2018-09-201-14/+35
| | | | | | | | | | | | | | | | | | If a HTTP handler throws an exception while processing a request we automatically write a JSON error response. If the handler had already started writing a response twisted throws an exception. We should check for this case and simple abort the connection if there was an error after the response had started being written.
* | Fix client IPs being broken on Python 3 (#3908)Amber Brown2018-09-201-1/+1
|/
* Add a regression test for logging on failed connections (#3912)Amber Brown2018-09-201-2/+2
|
* Update to use new timeout function everywhere.Erik Johnston2018-09-192-12/+17
| | | | | | | The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.
* Merge pull request #3909 from turt2live/travis/fix-logging-1Amber Brown2018-09-191-4/+4
|\ | | | | Fix matrixfederationclient.py logging: Destination is a string
| * Destination is a stringTravis Ralston2018-09-181-4/+4
| |
* | Merge pull request #3907 from matrix-org/rav/set_sni_to_server_nameAmber Brown2018-09-191-3/+10
|\ \ | |/ |/| Set SNI to the server_name, not whatever was in the SRV record
| * Set SNI to the server_name, not whatever was in the SRV recordRichard van der Hoff2018-09-181-3/+10
| | | | | | | | Fixes #3843
* | Refactor matrixfederationclient to fix logging (#3906)Richard van der Hoff2018-09-181-149/+236
|/ | | | | | | | We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.
* Fix more b'abcd' noise in metricsRichard van der Hoff2018-09-172-12/+12
|
* Fix some b'abcd' noise in logs and metricsRichard van der Hoff2018-09-172-5/+8
| | | | | Python 3 compatibility: make sure that we decode some byte sequences before we use them to create log lines and metrics labels.
* Add an awful secondary timeout to fix wedged requestsErik Johnston2018-09-141-0/+11
| | | | This is an attempt to mitigate #3842 by adding yet-another-timeout
* Merge pull request #3872 from matrix-org/hawkowl/timeouts-2Erik Johnston2018-09-144-7/+12
|\ | | | | timeouts 2: electric boogaloo
| * fixAmber Brown2018-09-151-3/+3
| |
| * fixAmber Brown2018-09-141-1/+4
| |
| * fixAmber Brown2018-09-141-1/+1
| |
| * redact betterAmber Brown2018-09-141-2/+2
| |
| * we do that higher upAmber Brown2018-09-141-0/+1
| |
| * Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2Amber Brown2018-09-141-4/+10
| |\
| * | buffer?Amber Brown2018-09-141-1/+2
| | |
* | | Measure outbound requestsErik Johnston2018-09-141-3/+6
| |/ |/|
* | merge (#3576)Amber Brown2018-09-141-4/+10
|/
* Attempt to figure out what's going on with timeouts (#3857)Amber Brown2018-09-141-56/+42
|
* Log outbound requests when we retryErik Johnston2018-09-121-75/+74
|
* Timeout reading body for outbound HTTP requestsErik Johnston2018-09-121-7/+45
|
* Port http/ to Python 3 (#3771)Amber Brown2018-09-063-173/+117
|
* Fix bug when federation_domain_whitelist is an emtpy listErik Johnston2018-08-241-1/+1
| | | | | Outbound federation were incorrectly allowed when the config option was set to an empty list
* Merge tag 'v0.33.3rc2' into developRichard van der Hoff2018-08-211-1/+7
|\ | | | | | | | | | | | | | | Bugfixes -------- - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
| * Fix exceptions when a connection is closed before we read the headersRichard van der Hoff2018-08-201-1/+7
| | | | | | | | | | This fixes bugs introduced in #3700, by making sure that we behave sanely when an incoming connection is closed before the headers are read.
* | Fix typoErik Johnston2018-08-211-4/+4
| |
* | Make the in flight requests metrics thread safeErik Johnston2018-08-201-3/+10
|/
* Port over enough to get some sytests running on Python 3 (#3668)Amber Brown2018-08-202-16/+44
|
* Merge pull request #3701 from matrix-org/rav/use_producer_for_responsesRichard van der Hoff2018-08-171-4/+13
|\ | | | | Use a producer to stream back responses
| * Use a producer to stream back responsesRichard van der Hoff2018-08-151-4/+13
| | | | | | | | | | | | | | | | | | | | The problem with dumping all of the json response into the Request object at once is that doing so starts the timeout for the next request to be received: so if it takes longer than 60s to stream back the response to the client, the client never gets it. The correct solution is to use a Producer; then the timeout is only started once all of the content is sent over the TCP connection.
* | Refactor request logging codeRichard van der Hoff2018-08-152-92/+179
|/ | | | | | | | | | | | | | | This commit moves a bunch of the logic for deciding when to log the receipt and completion of HTTP requests into SynapseRequest, rather than in the request handling wrappers. Advantages of this are: * we get logs for *all* requests (including OPTIONS and HEADs), rather than just those that end up hitting handlers we've remembered to decorate correctly. * when a request handler wires up a Producer (as the media stuff does currently, and as other things will do soon), we log at the point that all of the traffic has been sent to the client.
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-102-2/+2
|
* Merge pull request #3439 from vojeroen/send_sni_for_federation_requestsRichard van der Hoff2018-08-102-8/+8
|\ | | | | send SNI for federation requests
| * Merge branch 'develop' into send_sni_for_federation_requestsRichard van der Hoff2018-07-273-8/+14
| |\
| * \ Merge remote-tracking branch 'upstream/develop' into ↵Jeroen2018-07-148-160/+105
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | send_sni_for_federation_requests # Conflicts: # synapse/crypto/context_factory.py
| * \ \ Merge branch 'develop' into send_sni_for_federation_requestsJeroen2018-07-096-21/+103
| |\ \ \ | | | | | | | | | | | | | | | | | | | | # Conflicts: # synapse/http/endpoint.py
| * | | | allow self-signed certificatesJeroen2018-06-261-1/+1
| | | | |
| * | | | formatting changes for pep8Jeroen2018-06-252-2/+3
| | | | |
| * | | | send SNI for federation requestsJeroen2018-06-242-8/+7
| | | | |
* | | | | include known room versions in outgoing make_joinsRichard van der Hoff2018-08-061-2/+5
| | | | |
* | | | | Kill off MatrixCodeMessageExceptionRichard van der Hoff2018-08-011-34/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.
* | | | | Be more careful which errors we send back over the C-S APIRichard van der Hoff2018-08-011-9/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We really shouldn't be sending all CodeMessageExceptions back over the C-S API; it will include things like 401s which we shouldn't proxy. That means that we need to explicitly turn a few HttpResponseExceptions into SynapseErrors in the federation layer. The effect of the latter is that the matrix errcode will get passed through correctly to calling clients, which might help with some of the random M_UNKNOWN errors when trying to join rooms.
* | | | | Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-022-11/+34
| |_|_|/ |/| | |
* | | | Move v1-only APIs into their own module & isolate deprecated ones (#3460)Amber Brown2018-07-191-2/+4
| | | |
* | | | pep8Erik Johnston2018-07-181-1/+2
| | | |
* | | | Add response code to response timer metricsErik Johnston2018-07-181-3/+5
| | | |
* | | | Refactor REST API tests to use explicit reactors (#3351)Amber Brown2018-07-171-2/+3
| | | |
* | | | rerun isort with latest versionKrombel2018-07-161-3/+1
| | | |
* | | | check isort by travisKrombel2018-07-162-2/+4
| |_|/ |/| |
* | | rename assert_params_in_request to assert_params_in_dictKrombel2018-07-131-1/+1
| | | | | | | | | | | | | | | | | | the method "assert_params_in_request" does handle dicts and not requests. A request body has to be parsed to json before this method can be used
* | | Refactor logcontext resource usage tracking (#3501)Richard van der Hoff2018-07-102-81/+26
| | | | | | | | | | | | | | | Factor out the resource usage tracking out to a separate object, which can be passed around and copied independently of the logcontext itself.
* | | run isortAmber Brown2018-07-098-78/+78
| |/ |/|
* | More server_name validationRichard van der Hoff2018-07-041-5/+42
| | | | | | | | | | | | | | | | We need to do a bit more validation when we get a server name, but don't want to be re-doing it all over the shop, so factor out a separate parse_and_validate_server_name, and do the extra validation. Also, use it to verify the server name in the config file.
* | Reject invalid server names (#3480)Richard van der Hoff2018-07-031-3/+31
| | | | | | | | | | Make sure that server_names used in auth headers are sane, and reject them with a sensible error code, before they disappear off into the depths of the system.
* | replace invalid utf8 with \ufffdMatthew Hodgson2018-07-021-2/+2
| |
* | a fix which doesn't NPE everywhereMatthew Hodgson2018-07-011-9/+17
| |
* | don't mix unicode strings with utf8-in-byte-stringsMatthew Hodgson2018-07-011-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | otherwise we explode with: ``` Traceback (most recent call last): File /usr/lib/python2.7/logging/handlers.py, line 78, in emit logging.FileHandler.emit(self, record) File /usr/lib/python2.7/logging/__init__.py, line 950, in emit StreamHandler.emit(self, record) File /usr/lib/python2.7/logging/__init__.py, line 887, in emit self.handleError(record) File /usr/lib/python2.7/logging/__init__.py, line 810, in handleError None, sys.stderr) File /usr/lib/python2.7/traceback.py, line 124, in print_exception _print(file, 'Traceback (most recent call last):') File /usr/lib/python2.7/traceback.py, line 13, in _print file.write(str+terminator) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_io.py, line 170, in write self.log.emit(self.level, format=u{log_io}, log_io=line) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 144, in emit self.observer(event) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 136, in __call__ errorLogger = self._errorLoggerForObserver(brokenObserver) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 156, in _errorLoggerForObserver if obs is not observer File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 81, in __init__ self.log = Logger(observer=self) File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 64, in __init__ namespace = self._namespaceFromCallingContext() File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 42, in _namespaceFromCallingContext return currentframe(2).f_globals[__name__] File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/compat.py, line 93, in currentframe for x in range(n + 1): RuntimeError: maximum recursion depth exceeded while calling a Python object Logged from file site.py, line 129 File /usr/lib/python2.7/logging/__init__.py, line 859, in emit msg = self.format(record) File /usr/lib/python2.7/logging/__init__.py, line 732, in format return fmt.format(record) File /usr/lib/python2.7/logging/__init__.py, line 471, in format record.message = record.getMessage() File /usr/lib/python2.7/logging/__init__.py, line 335, in getMessage msg = msg % self.args UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128) Logged from file site.py, line 129 ``` ...where the logger apparently recurses whilst trying to log the error, hitting the maximum recursion depth and killing everything badly.
* | Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-284-8/+8
| |
* | Need to pass reactor to endpoint facErik Johnston2018-06-251-1/+1
| |
* | Remove all global reactor imports & pass it around explicitly (#3424)Amber Brown2018-06-251-7/+9
|/
* Merge pull request #3419 from matrix-org/rav/events_per_requestRichard van der Hoff2018-06-221-1/+4
|\ | | | | Log number of events fetched from DB
| * Log number of events fetched from DBRichard van der Hoff2018-06-211-1/+4
| | | | | | | | | | | | | | | | | | | | When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]
* | Pass around the reactor explicitly (#3385)Amber Brown2018-06-222-5/+6
|/
* Fix inflight requests metric (incorrect name & traceback) (#3413)Amber Brown2018-06-201-3/+7
|
* Merge pull request #3276 from matrix-org/dbkr/unbindDavid Baker2018-06-111-2/+23
|\ | | | | Remove email addresses / phone numbers from ID servers when they're removed from synapse
| * pep8David Baker2018-06-061-3/+3
| |
| * More doc fixesDavid Baker2018-06-061-2/+2
| |
| * fix pep8David Baker2018-06-051-3/+5
| |
| * doc fixesDavid Baker2018-06-051-6/+6
| |
| * DocstringDavid Baker2018-06-041-0/+14
| |
| * Merge remote-tracking branch 'origin/develop' into dbkr/unbindDavid Baker2018-05-241-3/+5
| |\
| * | Hit the 3pid unbind endpoint on deactivationDavid Baker2018-05-231-2/+7
| | |
* | | redact_uri in two missed log pathsMichael Telatynski2018-06-061-2/+2
| | |
* | | factor out uri redaction into a method on httpMichael Telatynski2018-06-053-13/+17
| | |
* | | Strip `access_token` from outgoing requests using existing regexMichael Telatynski2018-06-021-1/+6
| | |
* | | update to more consistently use seconds in any metrics or loggingAmber Brown2018-05-281-19/+19
| | |
* | | update metrics to be in secondsAmber Brown2018-05-282-13/+13
| | |
* | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-282-4/+6
|\ \ \
| * \ \ Merge pull request #3246 from NotAFile/py3-repr-stringAmber Brown2018-05-241-1/+1
| |\ \ \ | | |_|/ | |/| | use repr, not str
| | * | use repr, not strAdrian Tschira2018-05-191-1/+1
| | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | Merge pull request #3243 from NotAFile/py3-six-3Erik Johnston2018-05-241-3/+5
| |\ \ \ | | |_|/ | |/| | Replace some more comparisons with six
| | * | Replace some more comparisons with sixAdrian Tschira2018-05-191-3/+5
| | |/ | | | | | | | | | | | | | | | plus a bonus b"" string I missed last time Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | fixesAmber Brown2018-05-231-7/+7
| | |
* | | cleanupAmber Brown2018-05-222-3/+6
| | |
* | | cleanup pep8 errorsAmber Brown2018-05-221-28/+86
| | |
* | | cleanups, self-registrationAmber Brown2018-05-221-12/+3
| | |
* | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-223-4/+213
|\| |
| * | Don't store contextErik Johnston2018-05-221-8/+5
| | |
| * | Move in_flight_requests_count to be a callback metricErik Johnston2018-05-221-10/+21
| | |
| * | Add in flight request metricsErik Johnston2018-05-212-2/+154
| |/ | | | | | | | | This tracks CPU and DB usage while requests are in flight, rather than when we write the response.
| * Merge pull request #3213 from matrix-org/rav/consent_handlerRichard van der Hoff2018-05-161-2/+74
| |\ | | | | | | ConsentResource to gather policy consent from users
| | * ConsentResource to gather policy consent from usersRichard van der Hoff2018-05-151-2/+74
| | | | | | | | | | | | | | | Hopefully there are enough comments and docs in this that it makes sense on its own.
* | | rest of the changesAmber Brown2018-05-212-85/+25
| | |
* | | replacing portionsAmber Brown2018-05-212-24/+10
|/ /
* / Set Server header in SynapseRequestRichard van der Hoff2018-05-103-13/+15
|/ | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.
* Remove redundant request_handler decoratorRichard van der Hoff2018-05-102-9/+4
| | | | | | This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler.
* Factor wrap_request_handler_with_logging out of wrap_request_handlerRichard van der Hoff2018-05-101-54/+66
| | | | ... so that it can be used on non-JSON endpoints
* Remove include_metrics paramRichard van der Hoff2018-05-101-17/+7
| | | | | The metrics are now available via the request, so this is redundant and can go away at last.
* Move outgoing_responses_counter handling to RequestMetricsRichard van der Hoff2018-05-102-4/+2
| | | | it's much neater there.
* Bump requests_counter in wrapped_request_handlerRichard van der Hoff2018-05-101-4/+11
| | | | less magic
* Move RequestMetrics handling into SynapseRequest.processing()Richard van der Hoff2018-05-102-24/+64
| | | | | It fits quite nicely here, and opens the path to getting rid of the "include_metrics" mess.
* Make RequestMetrics take a raw time rather than a clockRichard van der Hoff2018-05-102-6/+6
| | | | ... which is going to make it easier to move around.
* Move request_id management into SynapseRequestRichard van der Hoff2018-05-102-15/+25
|
* Move RequestsMetrics to its own fileRichard van der Hoff2018-05-092-124/+151
| | | | | | This is useful in its own right, because server.py is full of stuff; but more importantly, I want to do some refactoring that will cause a circular reference as it is.
* Merge pull request #3182 from Half-Shot/hs/fix-twisted-shutdownRichard van der Hoff2018-05-031-3/+8
|\ | | | | Fix 'Unhandled Error' logs with Twisted 18.4
| * Don't abortConnection() if the transport connection has already closed.Will Hunt2018-05-031-3/+8
| |
* | add missing param to cancelled_to_request_timed_out_errorRichard van der Hoff2018-05-021-1/+1
| | | | | | | | This gets two arguments, not one.
* | Merge pull request #3154 from NotAFile/py3-stringioRichard van der Hoff2018-04-301-2/+2
|\ \ | | | | | | Replace stringIO imports with six
| * | replace stringIO importsAdrian Tschira2018-04-281-2/+2
| | |
* | | Merge pull request #3155 from NotAFile/py3-bytes-1Richard van der Hoff2018-04-302-2/+2
|\ \ \ | | | | | | | | more bytes strings
| * | | more bytes stringsAdrian Tschira2018-04-292-2/+2
| |/ / | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | Merge pull request #3108 from NotAFile/py3-six-urlparseRichard van der Hoff2018-04-301-2/+1
|\ \ \ | |/ / |/| | Use six.moves.urlparse
| * | Use six.moves.urlparseAdrian Tschira2018-04-151-2/+1
| | | | | | | | | | | | | | | | | | The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | Backport deferred.addTimeoutRichard van der Hoff2018-04-273-6/+9
| | | | | | | | | | | | Twisted 16.0 doesn't have addTimeout, so let's backport it.
* | | Use deferred.addTimeout instead of time_bound_deferredRichard van der Hoff2018-04-233-29/+48
|/ / | | | | | | This doesn't feel like a wheel we need to reinvent.
* | Merge pull request #3061 from NotAFile/add-some-byte-stringsRichard van der Hoff2018-04-092-5/+5
|\ \ | | | | | | Add b prefixes to some strings that are bytes in py3