summary refs log tree commit diff
path: root/synapse/http/client.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add a maximum size for well-known lookups. (#8950)Patrick Cloke2020-12-161-14/+18
|
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-14/+32
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Add additional type hints to HTTP client. (#8812)Patrick Cloke2020-11-251-83/+128
| | | | This also removes some duplicated code between the simple HTTP client and matrix federation client.
* Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
| | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-291-21/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Add type annotations to SimpleHttpClient (#8372)Richard van der Hoff2020-09-241-56/+131
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-4/+4
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-191-5/+6
|
* Reduce INFO logging (#8050)Erik Johnston2020-08-111-1/+1
| | | | | | | | | | c.f. #8021 A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line. Some notes: 1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps. 2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
* Convert the SimpleHttpClient to async. (#8016)Patrick Cloke2020-08-041-31/+24
|
* Ensure the msg property of HttpResponseException is a string. (#7979)Patrick Cloke2020-07-291-4/+12
|
* Ensure that calls to `json.dumps` are compatible with the standard library ↵Patrick Cloke2020-07-151-2/+2
| | | | json. (#7836)
* Fix client reader sharding tests (#7853)Erik Johnston2020-07-151-1/+23
| | | | | | | | | | | | | | | * Fix client reader sharding tests * Newsfile * Fix typing * Update changelog.d/7853.misc Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Move mocking of http_client to tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802)Patrick Cloke2020-07-101-7/+7
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-5/+3
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-2/+4
| | | | variables (#6391)
* Implement OpenID Connect-based login (#7256)Quentin Gliech2020-05-081-0/+7
|
* Share SSL contexts for non-federation requests (#7094)Richard van der Hoff2020-03-171-3/+0
| | | | | | | Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests. Fixes #7092.
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-011-2/+15
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-2/+2
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Edit SimpleHttpClient to reference that header keys can be passed as str or ↵Andrew Morgan2019-09-271-6/+6
| | | | bytes (#6077)
* Trace how long it takes for the send trasaction to complete, including ↵Jorik Schellekens2019-09-051-36/+51
| | | | retrys (#5986)
* Remove unused methods from c/s api v1 in register.py (#5963)Andrew Morgan2019-09-021-33/+1
| | | These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
* Replace returnValue with return (#5736)Amber Brown2019-07-231-15/+13
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Run Black. (#5482)Amber Brown2019-06-201-18/+10
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-4/+9
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Add ability to blacklist ip ranges for federation traffic (#5043)Andrew Morgan2019-05-131-4/+2
|
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-20/+25
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Fix TaskStopped exceptions when outbound requests time out (#4690)Richard van der Hoff2019-02-201-3/+12
|
* Fix UnboundLocalError in post_urlencoded_get_json (#4460)Richard van der Hoff2019-01-241-2/+3
| | | This could cause exceptions if the id server returned 4xx responses.
* Fix IP URL previews on Python 3 (#4215)Amber Brown2018-12-221-134/+243
|
* Update to use new timeout function everywhere.Erik Johnston2018-09-191-2/+2
| | | | | | | The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.
* fixAmber Brown2018-09-151-3/+3
|
* merge (#3576)Amber Brown2018-09-141-4/+10
|
* Port http/ to Python 3 (#3771)Amber Brown2018-09-061-51/+31
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
|
* Kill off MatrixCodeMessageExceptionRichard van der Hoff2018-08-011-34/+27
| | | | | | | | | | | This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.
* Move v1-only APIs into their own module & isolate deprecated ones (#3460)Amber Brown2018-07-191-2/+4
|
* run isortAmber Brown2018-07-091-23/+25
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-1/+1
|
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-221-3/+3
|
* redact_uri in two missed log pathsMichael Telatynski2018-06-061-2/+2
|
* factor out uri redaction into a method on httpMichael Telatynski2018-06-051-6/+2
|
* Strip `access_token` from outgoing requests using existing regexMichael Telatynski2018-06-021-1/+6
|
* cleanupAmber Brown2018-05-221-1/+2
|
* replacing portionsAmber Brown2018-05-211-14/+6
|
* replace stringIO importsAdrian Tschira2018-04-281-2/+2
|
* Backport deferred.addTimeoutRichard van der Hoff2018-04-271-2/+4
| | | | Twisted 16.0 doesn't have addTimeout, so let's backport it.
* Use deferred.addTimeout instead of time_bound_deferredRichard van der Hoff2018-04-231-12/+8
| | | | This doesn't feel like a wheel we need to reinvent.
* Increase http conn pool sizeRichard van der Hoff2018-01-291-1/+6
|
* Use a connection pool for the SimpleHttpClientRichard van der Hoff2018-01-201-1/+8
| | | | | In particular I hope this will help the pusher, which makes many requests to sygnal, and is currently negotiating SSL for each one.
* Merge pull request #2586 from matrix-org/rav/frontend_proxy_auth_headerRichard van der Hoff2017-10-271-28/+80
|\ | | | | Front-end proxy: pass through auth header
| * Docstring for post_urlencoded_get_jsonRichard van der Hoff2017-10-271-0/+11
| |
| * SimpleHTTPClient: add support for headersRichard van der Hoff2017-10-261-28/+69
| | | | | | | | Sometimes we need to pass headers into these methods
* | Fix logcontext leaks in httpclientRichard van der Hoff2017-10-261-10/+9
|/ | | | `preserve_context_over_fn` is borked
* Remember how twisted worksKegan Dougal2017-05-051-2/+2
|
* Include the clockKegan Dougal2017-05-051-0/+1
|
* Rewrite SimpleHttpClient.request to include timeoutsKegan Dougal2017-05-051-13/+19
| | | | Fixes #2191
* No need for the exception variableDavid Baker2017-05-031-1/+1
|
* List caught expection typesDavid Baker2017-05-031-1/+1
|
* Remove debuggingDavid Baker2017-04-261-2/+1
|
* Fix get_jsonDavid Baker2017-04-261-5/+4
|
* Use CodeMessageException subclass insteadDavid Baker2017-04-251-44/+20
| | | | | Parse json errors from get_json client methods and throw special errors.
* Need the HTTP status codeDavid Baker2017-04-211-5/+5
|
* Do the same for get_jsonDavid Baker2017-04-211-8/+21
|
* Don't error for 3xx responsesDavid Baker2017-04-211-1/+1
|
* Propagate errors sensibly from proxied IS requestsDavid Baker2017-04-211-0/+30
| | | | | | When we're proxying Matrix endpoints, parse out Matrix error responses and turn them into SynapseErrors so they can be propagated sensibly upstream.
* Remove spurious newlineJohannes Löthberg2016-12-121-1/+0
| | | | | | Apparently I just removed the spaces instead... Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
* Fixup for #1689 and #1690Johannes Löthberg2016-12-121-4/+7
| | | | Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
* IPv6 support for client.pyGlyph2016-12-111-16/+11
| | | This is an (untested) general sketch of how to use wrapClientTLS to implement TLS over IPv6, as well as faster connections over IPv4.
* Work around TLS bug in twistedRichard van der Hoff2016-06-151-2/+26
| | | | | | Wrap up twisted's FileBodyProducer to work around https://twistedmatrix.com/trac/ticket/8473. Hopefully this fixes https://matrix.org/jira/browse/SYN-700.
* Clean up the blacklist/whitelist handling.Mark Haines2016-05-161-2/+1
| | | | | | | Always set the config key with an empty list, even if a list isn't specified. This means that the codepaths are the same for both the empty list and for a missing key. Since the behaviour is the same for both cases this makes the code somewhat easier to reason about.
* add a url_preview_ip_range_whitelist config param so we can whitelist the ↵Matthew Hodgson2016-05-011-2/+4
| | | | matrix.org IP space
* Make InsecureInterceptableContextFactory work with SpiderEndpointErik Johnston2016-04-191-1/+4
|
* actually throw meaningful errorsMatthew Hodgson2016-04-081-5/+16
|
* Add url_preview_enabled config option to turn on/off preview_url endpoint. ↵Matthew Hodgson2016-04-081-5/+39
| | | | | | | | | defaults to off. Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered. Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered. Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist Add commentary and generally address PR feedback
* pep8Matthew Hodgson2016-04-031-4/+10
|
* add a persistent cache of URL lookups, and fix up the in-memory one to workMatthew Hodgson2016-04-031-3/+3
|
* support gzip compression, and don't pass through error msgsMatthew Hodgson2016-04-021-3/+8
|
* handle spidered relative images correctlyMatthew Hodgson2016-03-311-1/+1
|
* sync in changes from matrixfederationclientMatthew Hodgson2016-03-311-2/+6
|
* handle requests with missing content-length headers (e.g. YouTube)Matthew Hodgson2016-03-311-7/+26
|
* implement redirectsMatthew Hodgson2016-03-291-3/+3
|
* make it workMatthew Hodgson2016-03-291-1/+2
|
* make it buildMatthew Hodgson2016-03-291-1/+1
|
* Merge branch 'develop' into matthew/preview_urlsMatthew Hodgson2016-03-271-2/+15
|\
| * pep8David Baker2016-03-081-2/+5
| |
| * Encode unicode objects given to post_urlencode* otherwise urllib.urlencode ↵David Baker2016-03-081-2/+12
| | | | | | | | chokes.
* | initial WIP of a tentative preview_url endpoint - incomplete, untested, ↵Matthew Hodgson2016-01-241-0/+81
|/ | | | experimental, etc. just putting it here for safekeeping for now
* copyrightsMatthew Hodgson2016-01-071-1/+1
|
* Verify third party ID server certificatesDaniel Wagner-Hall2015-10-161-4/+0
|
* Fix previous merge to s/version_string/user_agent/Erik Johnston2015-10-101-2/+2
|
* Add get_raw method to SimpleHttpClient, use this in CAS auth rather than ↵Steven Hammerton2015-10-101-19/+36
| | | | requests
* Use space not dash as delimiterDaniel Wagner-Hall2015-10-061-1/+1
|
* Use space not dash as delimiterDaniel Wagner-Hall2015-10-061-1/+1
|
* Preserve version string in user agentDaniel Wagner-Hall2015-10-051-3/+3
|
* Allow synapse's useragent to be customizedDaniel Wagner-Hall2015-10-021-6/+9
| | | | | This will allow me to write tests which verify which server made HTTP requests in a federation context.
* Rename context factoryDaniel Wagner-Hall2015-09-151-3/+3
| | | | Mjark is officially no fun.
* Fix random formattingDaniel Wagner-Hall2015-09-091-1/+2
|
* Allow configuration to ignore invalid SSL certsDaniel Wagner-Hall2015-09-091-2/+23
| | | | | This will be useful for sytest, and sytest only, hence the aggressive config key name.
* Remove syutil dependency in favour of smaller single-purpose librariesMark Haines2015-08-241-1/+2
|
* Fix log context when sending requestsErik Johnston2015-06-191-5/+5
|
* Log requests and responses sent via http.clientErik Johnston2015-06-151-3/+15
|
* Correctly pass connection pool parameterErik Johnston2015-05-281-1/+1
|
* Up maxPersistentPerHost countErik Johnston2015-05-281-0/+1
|
* Add connection pooling to SimpleHttpClientErik Johnston2015-05-281-2/+4
|
* Change the way we do logging contexts so that they survive divergencesErik Johnston2015-05-081-1/+5
|
* New registration for C/S API v2. Only ReCAPTCHA working currently.David Baker2015-03-301-0/+2
|
* Appease pep8Paul "LeoNerd" Evans2015-03-121-3/+7
|
* Use _ instead of . as a metric namespacing separator, for PrometheusPaul "LeoNerd" Evans2015-03-121-2/+2
|
* Rename Metrics' "keys" to "labels"Paul "LeoNerd" Evans2015-03-121-2/+2
|
* Add vector counters to HTTP clients and servers; count the requests by ↵Paul "LeoNerd" Evans2015-03-121-1/+22
| | | | method and responses by method and response code
* Pull out all uses of the underlying HTTP user agent .request() method into a ↵Paul "LeoNerd" Evans2015-03-121-5/+10
| | | | single wrapper function, to make adding metrics easier
* Fix bug which prevented the HS pushing events to the AS due to FrozenEventsKegan Dougal2015-03-021-1/+1
|
* Use git aware version string in User-Agent and Server headersErik Johnston2015-02-181-5/+5
|
* Merge pull request #50 from matrix-org/application-servicesMark Haines2015-02-131-10/+57
|\ | | | | Application Services
| * Minor tweaks based on PR feedback.Kegan Dougal2015-02-111-1/+2
| |
| * Add SimpleHttpClient.put_json with the same semantics as get_json.Kegan Dougal2015-02-041-2/+46
| |
| * Update user/alias query APIs to use new format of SimpleHttpClient.get_jsonKegan Dougal2015-02-041-0/+3
| |
| * Modify API for SimpleHttpClient.get_json and update usages.Kegan Dougal2015-02-041-10/+9
| | | | | | | | | | | | | | | | | | | | Previously, this would only return the HTTP body as JSON, and discard other response information (e.g. the HTTP response code). This has now been changed to throw a CodeMessageException on a non-2xx response, with the response code and body, which can then be parsed as JSON. Affected modules include: - Registration/Login (when using an email for IS auth)
* | Use encode_canonical_json for http clientErik Johnston2015-02-111-1/+3
| |
* | Blunty replace json with simplejsonErik Johnston2015-02-111-1/+1
|/
* Merge branch 'develop' into pushersDavid Baker2015-01-131-3/+9
|\
| * SYN-231: User agent header brokenMark Haines2015-01-081-3/+3
| |
| * Update copyright noticesMark Haines2015-01-061-1/+1
| |
| * Include version in User-Agent and Server headersMark Haines2014-12-221-2/+8
| |
* | More work on pushers. Attempt to do HTTP pokes. Not sure if the actual HTTP ↵David Baker2014-11-211-0/+19
|/ | | | pokes work or not yet but the retry semantics are pretty good.
* Fix pep8 and pyflakes warningsMark Haines2014-11-201-3/+4
|
* Separate out the matrix http client completely because just about all of its ↵David Baker2014-11-201-322/+29
| | | | code it now separate from the simple case we need for standard HTTP(S)
* Refactor the HTTP clients a little.David Baker2014-11-201-79/+75
|
* CaptchaServerHttpClient should extend the base, not matrix http client.David Baker2014-11-201-1/+1
|
* SYN-141: Encode query params as UTF-8.Mark Haines2014-11-181-1/+7
|
* Merge branch 'develop' into request_loggingMark Haines2014-10-301-3/+4
|\ | | | | | | | | Conflicts: synapse/config/logger.py
| * Fix pep8 warningsMark Haines2014-10-301-4/+7
| |
* | Add a request-id to each log lineMark Haines2014-10-301-11/+15
|/
* Add get_json method to 3pid http client. Better logging for errors in 3pid ↵Mark Haines2014-10-201-1/+39
| | | | requests
* remove debugging logging for signing requestsMark Haines2014-10-141-4/+0
|
* Verify signatures for server2server requestsMark Haines2014-10-131-3/+7
|
* SYN-75 sign at the request level rather than the transaction levelMark Haines2014-10-131-7/+45
|
* Replace on_send_callback with something a bit clearer so that we can sign ↵Mark Haines2014-10-131-11/+35
| | | | messages
* Fix pyflakes errorsMark Haines2014-10-021-8/+2
|
* Add a keyword argument to get_json to avoid retrying on DNS failures. Rather ↵Mark Haines2014-10-021-8/+1
| | | | than passing MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES as a fake query string parameter
* Split PlainHttpClient into separate clients for talking to Identity servers ↵Mark Haines2014-10-021-144/+146
| | | | and talking to Capatcha servers
* remove "red", "blue" and "green" server_name mappingsMark Haines2014-10-011-7/+0
|
* Fix 'age' key to update on retriesErik Johnston2014-09-151-3/+10
|
* Captchas now work on registration. Missing x-forwarded-for config arg ↵Kegan Dougal2014-09-051-1/+27
| | | | support. Missing reloading a new captcha on the web client / displaying a sensible error message.
* Merge branch 'develop' of github.com:matrix-org/synapse into developDavid Baker2014-09-031-4/+22
|\ | | | | | | | | Conflicts: synapse/http/client.py
| * fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org ↵Matthew Hodgson2014-09-031-1/+1
| | | | | | | | hasn't been incorporated in time for launch.
| * Make retrying requests on DNS failures configurable, and turn off retrying ↵Kegan Dougal2014-09-031-6/+19
| | | | | | | | only in directory.get_association
| * hs: Updated synapse.http.client to handle DNSLookupErrors and bail immediately.Kegan Dougal2014-09-031-1/+6
| |
* | Add support for registering with a threepid to the HS (get credentials from ↵David Baker2014-09-031-5/+35
|/ | | | the client and check them against an ID server).
* Fix SSL for federation http clientMark Haines2014-09-011-1/+1
|
* Enable SSL for s2s http clientMark Haines2014-09-011-2/+6
|
* fix http client GET parameters; somehow missing named param. how could this ↵Matthew Hodgson2014-08-141-2/+2
| | | | have ever worked!?
* Update get_json()'s documentation to match the actual observed behaviourPaul "LeoNerd" Evans2014-08-131-4/+3
|
* add in whitespace after copyright statements to improve legibilityMatthew Hodgson2014-08-131-0/+1
|
* Reference Matrix Home Servermatrix.org2014-08-121-0/+246
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-09-22 04:38:24 +0000