| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some small cleanups after Python3.8 became EOL.
- Move some type imports from `typing_extensions` to `typing`
- Remove the `abi3-py38` feature from pyo3
### Pull Request Checklist
<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->
* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
- Use markdown where necessary, mostly for `code blocks`.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
---------
Co-authored-by: Quentin Gliech <quenting@element.io>
|
| |
|
|
|
|
|
|
|
| |
Fixes various `mypy` errors associated with Twisted `24.11.0`.
Hopefully addresses https://github.com/element-hq/synapse/issues/17075,
though I've yet to test against `trunk`.
Changes should be compatible with our currently pinned Twisted version
of `24.7.0`.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
I thought ruff check would also format, but it doesn't.
This runs ruff format in CI and dev scripts. The first commit is just a
run of `ruff format .` in the root directory.
|
| | |
|
| |
|
|
| |
This is really spurious and causes a lot of spam. I don't think there is
a use for it even at DEBUG level.
|
| | |
|
| |
|
|
|
| |
Requests may require a User-Agent header, and the change in #16972
accidentally removed it, resulting in requests getting rejected causing
login to fail.
|
| |
|
|
|
| |
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
|
| | |
|
| | |
|
| |
|
|
|
| |
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).
This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
|
| |
|
|
|
|
| |
Revert "Federation outbound proxy (#15773)"
This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
|
| |
|
|
|
|
|
| |
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).
This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
|
| |
|
|
|
|
|
|
| |
All the information needed is already in the `instance_map`, so
use that instead of passing the hostname / IP & port manually
for each replication request.
This consolidates logic for future improvements of using e.g.
UNIX sockets for workers.
|
| |
|
|
| |
Avoid renaming configuration settings for now and rename internal code
to use blocklist and allowlist instead.
|
| |
|
|
|
|
| |
Separate out a HTTP client for replication in preparation for
also supporting using UNIX sockets. The major difference from
the base class is that this does not use treq to handle HTTP
requests.
|
| |
|
|
|
|
|
|
|
| |
#15514 introduced a regression where Synapse would encounter
`PartialDownloadError`s when fetching OpenID metadata for certain
providers on startup. Due to #8088, this prevents Synapse from starting
entirely.
Revert the change while we decide what to do about the regression.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pushers tend to make many connections to the same HTTP host
(e.g. a new event comes in, causes events to be pushed, and then
the homeserver connects to the same host many times). Due to this
the per-host HTTP connection pool size was increased, but this does
not make sense for other SimpleHttpClients.
Add a parameter for the connection pool and override it for pushers
(making a separate SimpleHttpClient for pushers with the increased
configuration).
This returns the HTTP connection pool settings to the default Twisted
ones for non-pusher HTTP clients.
|
| |
|
|
| |
Pulls out some methods to `BaseHttpClient` to eventually be
reused in other contexts.
|
| |
|
|
|
| |
This uses the specced /_matrix/app/v1/... paths instead of the
"legacy" paths. If the homeserver receives an error it will retry
using the legacy path.
|
| |
|
|
|
| |
If enabled, for users which are exclusively owned by an application
service then the appservice will be queried for devices in addition
to any information stored in the Synapse database.
|
| |
|
| |
And fix any issues from having those type hints.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Tweak http types in Synapse
AFACIS these are correct, and they make mypy happier on tests.http.
* Type hints for test_proxyagent
* type hints for test_srv_resolver
* test_matrix_federation_agent
* tests.http.server._base
* tests.http.__init__
* tests.http.test_additional_resource
* tests.http.test_client
* tests.http.test_endpoint
* tests.http.test_matrixfederationclient
* tests.http.test_servlet
* tests.http.test_simple_client
* tests.http.test_site
* One fixup in tests.server
* Untyped defs
* Changelog
* Fixup syntax for Python 3.7
* Fix olddeps syntax
* Use a twisted IPv4 addr for dummy_address
* Fix typo, thanks Sean
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Remove redundant `Optional`
---------
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
* Pull out query param types to `synapse.http.types`
* Use QueryParams everywhere
* Simplify `encode_query_args`
* Add annotation which would have caught #12410
|
| | |
|
| |
|
|
|
|
|
| |
This implements an allow list for content types for which Synapse will attempt URL preview. If a URL resolves to a resource with a content type which isn't in the list, the download will terminate immediately.
This makes sense given that Synapse would never successfully generate a URL preview for such files in the first place, and helps prevent issues with streaming media servers, such as #8302.
Signed-off-by: Denis Kasak dkasak@termina.org.uk
|
| |
|
|
| |
This prevents the URL preview code from reading
a stream forever.
|
| |
|
| |
Co-authored-by: reivilibre <olivier@librepush.net>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
mypy (#11006)
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py
Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.
The rest of this change consists of fixups to make the newer mypy + stubs pass CI.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| |
|
| |
Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
|
| | |
|
| |
|
| |
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
|
| | |
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| |
|
|
|
|
|
| |
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
interfaces. (#9528)
This helps fix some type hints when running with Twisted 21.2.0.
|
| |
|
|
| |
By consuming the response if the headers imply that the
content is too large.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Changes proposed in this PR
- Add support for the `no_proxy` and `NO_PROXY` environment variables
- Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519)
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment
This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`
Related to #9306 which also uses `ProxyAgent`
Signed-off-by: Timothy Leung tim95@hotmail.co.uk
|
| |
|
|
|
|
|
|
| |
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.
It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| |
|
| |
...instead of just creating the exception object and doing nothing with it.
|
| |
|
|
|
| |
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
|
| |
|
|
| |
Removes a bare `except Exception` clause and replaces it with
catching a specific exception around the portion that might throw.
|
| |
|
|
|
| |
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:
* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events
The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
|
| |
|
|
| |
This also removes some duplicated code between the simple
HTTP client and matrix federation client.
|
| |
|
|
|
|
|
|
| |
This allows trailing commas in multi-line arg lists.
Minor, but we might as well keep our formatting current with regard to
our minimum supported Python version.
Signed-off-by: Dan Callahan <danc@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Remove `on_timeout_cancel` from `timeout_deferred`
The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a
timeout (in particular if the canceller raised an exception), so it was
unreliable. It was also only used in one place, and to be honest it's easier to
do what it does a different way.
* Fix handling of connection timeouts in outgoing http requests
Turns out that if we get a timeout during connection, then a different
exception is raised, which wasn't always handled correctly.
To fix it, catch the exception in SimpleHttpClient and turn it into a
RequestTimedOutError (which is already a documented exception).
Also add a description to RequestTimedOutError so that we can see which stage
it failed at.
* Fix incorrect handling of timeouts reading federation responses
This was trapping the wrong sort of TimeoutError, so was never being hit.
The effect was relatively minor, but we should fix this so that it does the
expected thing.
* Fix inconsistent handling of `timeout` param between methods
`get_json`, `put_json` and `delete_json` were applying a different timeout to
the response body to `post_json`; bring them in line and test.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
c.f. #8021
A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line.
Some notes:
1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps.
2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
|
| | |
|
| | |
|
| |
|
|
| |
json. (#7836)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix client reader sharding tests
* Newsfile
* Fix typing
* Update changelog.d/7853.misc
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Move mocking of http_client to tests
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| | |
|
| |
|
|
| |
variables (#6391)
|
| | |
|
| |
|
|
|
|
|
| |
Extends #5794 etc to the SimpleHttpClient so that it also applies to non-federation requests.
Fixes #7092.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.
The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.
The proxy will then be used for
* push
* url previews
* phone-home stats
* recaptcha validation
* CAS auth validation
It will *not* be used for:
* Application Services
* Identity servers
* Outbound federation
* In worker configurations, connections from workers to masters
Fixes #4198.
|
| |
|
| |
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
|
| |
|
|
| |
bytes (#6077)
|
| |
|
|
| |
retrys (#5986)
|
| |
|
| |
These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
In particular, let's not log stack traces when we stop processing
becuase the response body was too large.
|
| | |
|
| |
|
| |
Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
|
| | |
|
| |
|
| |
This could cause exceptions if the id server returned 4xx responses.
|
| | |
|
| |
|
|
|
|
|
| |
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, #3842.
The new helper function doesn't suffer from this problem.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Twisted 16.0 doesn't have addTimeout, so let's backport it.
|
| |
|
|
| |
This doesn't feel like a wheel we need to reinvent.
|
| | |
|
| |
|
|
|
| |
In particular I hope this will help the pusher, which makes many requests to
sygnal, and is currently negotiating SSL for each one.
|
| |\
| |
| | |
Front-end proxy: pass through auth header
|
| | | |
|
| | |
| |
| |
| | |
Sometimes we need to pass headers into these methods
|
| |/
|
|
| |
`preserve_context_over_fn` is borked
|
| | |
|
| | |
|
| |
|
|
| |
Fixes #2191
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Parse json errors from get_json client methods and throw special
errors.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
|
| |
|
|
|
|
| |
Apparently I just removed the spaces instead...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
|
| |
|
|
| |
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
|
| |
|
| |
This is an (untested) general sketch of how to use wrapClientTLS to implement TLS over IPv6, as well as faster connections over IPv4.
|
| |
|
|
|
|
| |
Wrap up twisted's FileBodyProducer to work around
https://twistedmatrix.com/trac/ticket/8473. Hopefully this fixes
https://matrix.org/jira/browse/SYN-700.
|
| |
|
|
|
|
|
| |
Always set the config key with an empty list, even if a list isn't specified.
This means that the codepaths are the same for both the empty list and
for a missing key. Since the behaviour is the same for both cases this
makes the code somewhat easier to reason about.
|
| |
|
|
| |
matrix.org IP space
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
defaults to off.
Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered.
Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered.
Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist
Add commentary and generally address PR feedback
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| | |
| |
| |
| | |
chokes.
|
| |/
|
|
| |
experimental, etc. just putting it here for safekeeping for now
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
requests
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
This will allow me to write tests which verify which server made HTTP
requests in a federation context.
|
| |
|
|
| |
Mjark is officially no fun.
|
| | |
|
| |
|
|
|
| |
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
method and responses by method and response code
|
| |
|
|
| |
single wrapper function, to make adding metrics easier
|
| | |
|
| | |
|
| |\
| |
| | |
Application Services
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously, this would only return the HTTP body as JSON, and discard other
response information (e.g. the HTTP response code). This has now been changed
to throw a CodeMessageException on a non-2xx response, with the response code
and body, which can then be parsed as JSON.
Affected modules include:
- Registration/Login (when using an email for IS auth)
|
| | | |
|
| |/ |
|
| |\ |
|
| | | |
|
| | | |
|
| | | |
|
| |/
|
|
| |
pokes work or not yet but the retry semantics are pretty good.
|
| | |
|
| |
|
|
| |
code it now separate from the simple case we need for standard HTTP(S)
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
Conflicts:
synapse/config/logger.py
|
| | | |
|
| |/ |
|
| |
|
|
| |
requests
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
messages
|
| | |
|
| |
|
|
| |
than passing MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES as a fake query string parameter
|
| |
|
|
| |
and talking to Capatcha servers
|
| | |
|
| | |
|
| |
|
|
| |
support. Missing reloading a new captcha on the web client / displaying a sensible error message.
|
| |\
| |
| |
| |
| | |
Conflicts:
synapse/http/client.py
|
| | |
| |
| |
| | |
hasn't been incorporated in time for launch.
|
| | |
| |
| |
| | |
only in directory.get_association
|
| | | |
|
| |/
|
|
| |
the client and check them against an ID server).
|
| | |
|
| | |
|
| |
|
|
| |
have ever worked!?
|
| | |
|
| | |
|
| |
|
