summary refs log tree commit diff
path: root/tests/http (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1410-10/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-096-9/+6
|
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-4/+11
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Allow providing credentials to HTTPS_PROXY (#9657)Andrew Morgan2021-03-221-0/+40
| | | | | | | Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.
* Add tests for blacklisting reactor/agent. (#9563)Patrick Cloke2021-03-111-2/+124
|
* (Hopefully) stop leaking file descriptors in media repo. (#9497)Patrick Cloke2021-03-011-36/+55
| | | | By consuming the response if the headers imply that the content is too large.
* Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-261-44/+73
| | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
* Reduce the memory usage of previewing media files. (#9421)Patrick Cloke2021-02-181-4/+5
| | | | | | | | This reduces the memory usage of previewing media files which end up larger than the `max_spider_size` by avoiding buffering content internally in treq. It also checks the `Content-Length` header in additional places instead of streaming the content to check the body length.
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-161-4/+2
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-1/+1
|
* Avoid raising the body exceeded error multiple times. (#9108)Patrick Cloke2021-01-152-2/+103
| | | | | Previously this code generated unreferenced `Deferred` instances which caused "Unhandled Deferreds" errors to appear in error situations.
* Merge branch 'master' into developErik Johnston2021-01-131-0/+130
|\
| * Don't apply the IP range blacklist to proxy connections (#9084)Marcus2021-01-121-0/+130
| | | | | | | | | | It is expected that the proxy would be on a private IP address so the configured proxy should be connected to regardless of the IP range blacklist.
* | Handle bad JSON data being returned from the federation API. (#9070)Patrick Cloke2021-01-121-1/+1
|/
* Add a maximum size for well-known lookups. (#8950)Patrick Cloke2020-12-161-0/+27
|
* Remove spurious "SynapseRequest" result from `make_request"Richard van der Hoff2020-12-151-2/+2
| | | | This was never used, so let's get rid of it.
* Replace `request.code` with `channel.code`Richard van der Hoff2020-12-151-2/+2
| | | | | | The two are equivalent, but really we want to check the HTTP result that got returned to the channel, not the code that the Request object *intended* to return to the channel.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-0/+3
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Remove redundant calls to `render()`Richard van der Hoff2020-11-161-3/+1
|
* use global make_request() directly where we have a custom ResourceRichard van der Hoff2020-11-151-6/+7
| | | | | | Where we want to render a request against a specific Resource, call the global make_request() function rather than the one in HomeserverTestCase, allowing us to pass in an appropriate `Site`.
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-292-6/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Merge tag 'v1.20.0rc3' into developPatrick Cloke2020-09-112-0/+128
|\ | | | | | | | | | | | | | | | | | | Synapse 1.20.0rc3 (2020-09-11) ============================== Bugfixes -------- - Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
| * Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-102-0/+128
| |
* | Use TLSv1.2 for fake servers in tests (#8208)Dan Callaghan2020-09-101-1/+1
|/ | | | | | | | | | | | | | Some Linux distros have begun disabling TLSv1.0 and TLSv1.1 by default for security reasons, for example in Fedora 33 onwards: https://fedoraproject.org/wiki/Changes/StrongCryptoSettings2 Use TLSv1.2 for the fake TLS servers created in the test suite, to avoid failures due to OpenSSL disallowing TLSv1.0: <twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl_choose_client_version', 'unsupported protocol')]> Signed-off-by: Dan Callaghan <djc@djc.id.au>
* Stop sub-classing object (#8249)Patrick Cloke2020-09-042-2/+2
|
* Convert the well known resolver to async (#8214)Patrick Cloke2020-09-011-6/+18
|
* Reduce run-times of tests by advancing the reactor less (#7757)Andrew Morgan2020-08-271-1/+1
|
* Convert federation client to async/await. (#7975)Patrick Cloke2020-07-301-14/+36
|
* Convert the federation agent and related code to async/await. (#7874)Patrick Cloke2020-07-232-37/+40
|
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-031-0/+62
|
* Include a user agent in federation requests. (#7677)Patrick Cloke2020-06-161-0/+10
|
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-243-9/+9
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Share SSL contexts for non-federation requests (#7094)Richard van der Hoff2020-03-171-3/+3
| | | | | | | Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests. Fixes #7092.
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-013-3/+359
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-08-221-35/+48
|\ | | | | | | erikj/reliable_lookups
| * Change jitter to be a factor rather than absolute valueErik Johnston2019-08-201-2/+2
| |
| * Retry well known on fail.Erik Johnston2019-08-151-33/+46
| | | | | | | | | | | | If we have recently seen a valid well-known for a domain we want to retry on (non-final) errors a few times, to handle temporary blips in networking/etc.
* | Remove test debugsErik Johnston2019-08-201-1/+0
| |
* | Refactor MatrixFederationAgent to retry SRV.Erik Johnston2019-08-152-5/+66
|/ | | | | | | | This refactors MatrixFederationAgent to move the SRV lookup into the endpoint code, this has two benefits: 1. Its easier to retry different host/ports in the same way as HostnameEndpoint. 2. We avoid SRV lookups if we have a free connection in the pool
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-0/+69
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Move well known lookup into a separate clasErik Johnston2019-08-071-20/+19
|
* Add a lower bound for TTL on well known results.Erik Johnston2019-08-061-2/+2
| | | | | | | It costs both us and the remote server for us to fetch the well known for every single request we send, so we add a minimum cache period. This is set to 5m so that we still honour the basic premise of "refetch frequently".
* Share SSL options for well-known requestsErik Johnston2019-07-311-6/+6
|
* Replace returnValue with return (#5736)Amber Brown2019-07-233-4/+4
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-043-3/+3
|
* Pass config_dir_path and data_dir_path into Config.read_config. (#5522)Richard van der Hoff2019-06-241-1/+1
| | | | | | * Pull config_dir_path and data_dir_path calculation out of read_config_files * Pass config_dir_path and data_dir_path into read_config
* Run Black. (#5482)Amber Brown2019-06-204-125/+124
|
* fix ci on py2, againRichard van der Hoff2019-06-101-1/+1
|
* fix CI on python 2.7Richard van der Hoff2019-06-101-6/+4
|
* Tests for SSL certs for federation connectionsRichard van der Hoff2019-06-106-106/+343
| | | | Add some tests for bad certificates for federation and .well-known connections
* Stop hardcoding trust of old matrix.org key (#5374)Richard van der Hoff2019-06-061-0/+1
| | | | | | | | | | | | | | | | | There are a few changes going on here: * We make checking the signature on a key server response optional: if no verify_keys are specified, we trust to TLS to validate the connection. * We change the default config so that it does not require responses to be signed by the old key. * We replace the old 'perspectives' config with 'trusted_key_servers', which is also formatted slightly differently. * We emit a warning to the logs every time we trust a key server response signed by the old key.
* Validate federation server TLS certificates by default.Richard van der Hoff2019-06-051-3/+9
|
* Migrate all tests to use the dict-based config format instead of hanging ↵Amber Brown2019-05-131-1/+3
| | | | items off HomeserverConfig (#5171)
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-131-0/+71
|
* Run Black on the tests again (#5170)Amber Brown2019-05-104-172/+113
|
* Config option for verifying federation certificates (MSC 1711) (#4967)Andrew Morgan2019-04-251-1/+2
|
* lintAndrew Morgan2019-03-201-1/+1
|
* New test, fix issuesAndrew Morgan2019-03-201-0/+45
|
* Fix comments. v0.99.2 -> v0.99.3Andrew Morgan2019-03-201-1/+1
|
* Federation test fixed!Andrew Morgan2019-03-201-2/+5
|
* Better exception handlingAndrew Morgan2019-03-181-4/+1
|
* Correct var nameAndrew Morgan2019-03-131-0/+54
|
* i should have given up x3Andrew Morgan2019-03-131-39/+0
|
* i should have given up x2Andrew Morgan2019-03-131-0/+39
|
* Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)Richard van der Hoff2019-02-111-3/+1
| | | | | It's nothing to do with refreshing the certificates. No idea why it was here.
* Treat an invalid .well-known the same as an absent oneRichard van der Hoff2019-02-011-11/+70
| | | | ... basically, carry on and fall back to SRV etc.
* Update federation routing logic to check .well-known before SRVRichard van der Hoff2019-01-311-30/+21
|
* Follow redirects on .well-known (#4520)Richard van der Hoff2019-01-301-0/+97
|
* Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-4/+146
|
* Relax requirement for a content-type on .well-known (#4511)Richard van der Hoff2019-01-291-1/+0
|
* Implement MSC1708 (.well-known lookups for server routing) (#4489)Richard van der Hoff2019-01-293-7/+339
|
* Use SimpleResolverComplexifier in tests (#4497)Richard van der Hoff2019-01-291-4/+3
| | | | | two reasons for this. One, it saves a bunch of boilerplate. Two, it squashes unicode to IDNA-in-a-`str` (even on python 3) in a way that it turns out we rely on to give consistent behaviour between python 2 and 3.
* Handle IP literals explicitlyRichard van der Hoff2019-01-281-17/+2
| | | | We don't want to be doing .well-known lookups on these guys.
* Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487)Richard van der Hoff2019-01-281-1/+180
| | | | | | | | | | | | | | | | Turns out that the library does a better job of parsing URIs than our reinvented wheel. Who knew. There are two things going on here. The first is that, unlike parse_server_name, URI.fromBytes will strip off square brackets from IPv6 literals, which means that it is valid input to ClientTLSOptionsFactory and HostnameEndpoint. The second is that we stay in `bytes` throughout (except for the argument to ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up with idna-encoded values being held in `unicode` variables. TBH it probably would have been ok but it made the tests fragile.
* Fix Host header sent by MatrixFederationAgent (#4468)Richard van der Hoff2019-01-252-1/+17
| | | | | | Move the Host header logic down here so that (a) it is used if we reuse the agent elsewhere, and (b) we can mess about with it with .well-known.
* Look up the right SRV recordRichard van der Hoff2019-01-241-3/+9
|
* lots more tests for MatrixFederationAgentRichard van der Hoff2019-01-241-10/+79
|
* Don't send IP addresses as SNI (#4452)Richard van der Hoff2019-01-241-3/+60
| | | | | | The problem here is that we have cut-and-pasted an impl from Twisted, and then failed to maintain it. It was fixed in Twisted in https://github.com/twisted/twisted/pull/1047/files; let's do the same here.
* Add a test for MatrixFederationAgentRichard van der Hoff2019-01-221-0/+183
|
* put resolve_service in an objectRichard van der Hoff2019-01-221-20/+18
| | | | this makes it easier to stub things out for tests.
* Require that service_name be a byte stringRichard van der Hoff2019-01-221-4/+4
| | | | it is only ever a bytes now, so let's enforce that.
* Make MatrixFederationClient use MatrixFederationAgentRichard van der Hoff2019-01-221-0/+96
| | | | ... instead of the matrix_federation_endpoint
* Refactor and bugfix for resove_service (#4427)Richard van der Hoff2019-01-222-0/+223
|
* Remove redundant WrappedConnection (#4409)Richard van der Hoff2019-01-181-7/+47
| | | | | | | | * Remove redundant WrappedConnection The matrix federation client uses an HTTP connection pool, which times out its idle HTTP connections, so there is no need for any of this business.
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-3/+10
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Merge branch 'master' into developAmber Brown2018-09-251-2/+2
|\
| * Fix compatibility issue with older Twisted in tests.Oleg Girko2018-09-251-2/+2
| | | | | | | | | | | | | | | | Older Twisted (18.4.0) returns TimeoutError instead of ConnectingCancelledError when connection times out. This change allows tests to be compatible with this behaviour. Signed-off-by: Oleg Girko <ol@infoserver.lv>
* | Refactor matrixfederationclient to fix logging (#3906)Richard van der Hoff2018-09-181-5/+38
|/ | | | | | | | We want to wait until we have read the response body before we log the request as complete, otherwise a confusing thing happens where the request appears to have completed, but we later fail it. To do this, we factor the salient details of a request out to a separate object, which can then keep track of the txn_id, so that it can be logged.
* Attempt to figure out what's going on with timeouts (#3857)Amber Brown2018-09-141-0/+157
|
* Run black.black2018-08-101-4/+2
|
* run isortAmber Brown2018-07-091-4/+2
|
* More server_name validationRichard van der Hoff2018-07-041-4/+13
| | | | | | | | We need to do a bit more validation when we get a server name, but don't want to be re-doing it all over the shop, so factor out a separate parse_and_validate_server_name, and do the extra validation. Also, use it to verify the server name in the config file.
* Reject invalid server names (#3480)Richard van der Hoff2018-07-032-0/+46
Make sure that server_names used in auth headers are sane, and reject them with a sensible error code, before they disappear off into the depths of the system.