| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
| |
Introduced in #15913
|
| |
|
|
|
| |
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).
This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
|
| | |
|
| |
|
|
|
|
| |
Revert "Federation outbound proxy (#15773)"
This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
|
| |
|
|
|
|
|
| |
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).
This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
|
| |
|
|
| |
Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any,
and disallow_incomplete_defs.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Respond with a 405 error if a request is received on a known endpoint,
but to an unknown method, per MSC3743.
|
| |
|
|
| |
Returns a proper 404 with an errcode of M_RECOGNIZED for
unknown endpoints per MSC3743.
|
| |
|
|
|
|
|
| |
Remove type hints from comments which have been added
as Python type hints. This helps avoid drift between comments
and reality, as well as removing redundant information.
Also adds some missing type hints which were simple to fill in.
|
| |
|
|
| |
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Andrew Morgan <andrewm@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Lockfile: update canonicaljson 1.6.0 -> 1.6.3
* Fix mypy errors with latest canonicaljson
The change to `_encode_json_bytes` definition wasn't sufficient:
```
synapse/http/server.py:751: error: Incompatible types in assignment (expression has type "Callable[[Arg(object, 'json_object')], bytes]", variable has type "Callable[[Arg(object, 'data')], bytes]") [assignment]
```
Which I think is mypy warning us that the two functions accept different
sets of kwargs. Fair enough!
* Changelog
|
| |
|
|
| |
other than just servlet methods. (#13662)
|
| |
|
|
|
| |
(#13343)
Implements MSC3848
|
| |
|
|
| |
endpoints (#12944)
|
| |
|
|
|
|
|
|
| |
Enable cancellation of `GET /rooms/$room_id/members`,
`GET /rooms/$room_id/state` and
`GET /rooms/$room_id/state/$state_key/*` requests.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`BaseFederationServlet`s (#12699)
Both `RestServlet`s and `BaseFederationServlet`s register their handlers
with `HttpServer.register_paths` / `JsonResource.register_paths`. Update
`JsonResource` to respect the `@cancellable` flag on handlers registered
in this way.
Although `ReplicationEndpoint` also registers itself using
`register_paths`, it does not pass the handler method that would have the
`@cancellable` flag directly, and so needs separate handling.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
`DirectServeHtmlResource` and `DirectServeJsonResource` both inherit
from `_AsyncResource`. These classes expect to be subclassed with
`_async_render_*` methods.
This commit has no effect on `JsonResource`, despite inheriting from
`_AsyncResource`. `JsonResource` has its own `_async_render` override
which will need to be updated separately.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
All async request processing goes through `_AsyncResource`, so this is
the only place where a `Deferred` needs to be captured for cancellation.
Unfortunately, the same isn't true for determining whether a request
can be cancelled. Each of `RestServlet`, `BaseFederationServlet`,
`DirectServe{Html,Json}Resource` and `ReplicationEndpoint` have
different wrappers around the method doing the request handling and they
all need to be handled separately.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
| |
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
| |
Don't log stack traces for cancelled requests and use a custom HTTP
status code of 499.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The status code of requests must always be set, regardless of client
disconnection, otherwise they will always be logged as 200!.
Broken for `respond_with_json` in
f48792eec43f893f4f893ffdcbf00f8958b6f6b5.
Broken for `respond_with_json_bytes` in
3e58ce72b42f2ae473c1e76a967548cd6fa7e2e6.
Broken for `respond_with_html_bytes` in
ea26e9a98b0541fc886a1cb826a38352b7599dbe.
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.
There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)
I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.
* Installing extras before typechecking
Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.
Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
|
| | |
|
| |
|
| |
To improve type hints throughout the code.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Wrap `auth.get_user_by_req` in an opentracing span
give `get_user_by_req` its own opentracing span, since it can result in a
non-trivial number of sub-spans which it is useful to group together.
This requires a bit of reorganisation because it also sets some tags (and may
force tracing) on the servlet span.
* Emit opentracing span for encoding json responses
This can be a significant time sink.
* Rename all sync spans with a prefix
* Write an opentracing span for encoding sync response
* opentracing span to group generate_room_entries
* opentracing spans within sync.encode_response
* changelog
* Use the `trace` decorator instead of context managers
|
| |
|
|
| |
Instead of returning 404 errors with HTML bodies when an unknown
prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
|
| | |
|
| | |
|
| | |
|
| |
|
| |
Looks like the wrong exception type was caught in #10932.
|
| |
|
| |
Introduced in #10905
|
| |
|
|
|
|
|
|
|
|
| |
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.
Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:
1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.
I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Drop Origin & Accept from Access-Control-Allow-Headers value
This change drops the Origin and Accept header names from the value of the
Access-Control-Allow-Headers response header sent by Synapse. Per the CORS
protocol, it’s not necessary or useful to include those header names.
Details:
Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin
is a “forbidden header name” set by the browser and that frontend
JavaScript code is never allowed to set.
So the value of Access-Control-Allow-Headers isn’t relevant to Origin or
in general to other headers set by the browser itself — the browser
never ever consults the Access-Control-Allow-Headers value to confirm
that it’s OK for the request to include an Origin header.
And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header,
Accept is a “CORS-safelisted request-header”, which means that browsers
allow requests to contain the Accept header regardless of whether the
Access-Control-Allow-Headers value contains "Accept".
So it’s unnecessary for the Access-Control-Allow-Headers to explicitly
include Accept. Browsers will not perform a CORS preflight for requests
containing an Accept request header.
Related: https://github.com/matrix-org/matrix-doc/pull/3225
Signed-off-by: Michael[tm] Smith <mike@w3.org>
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| | |
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| |
|
| |
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
|
| |
|
| |
Fixes #8928.
|
| |
|
|
| |
this was using `canonical_json` without setting it, so when you used it as a
standalone class, you would get exceptions.
|
| | |
|
| |
|
|
|
| |
* Remove special case of pretty printing JSON responses for curl
Signed-off-by: Tulir Asokan <tulir@maunium.net>
|
| |\
| |
| | |
Fix serialisation errors when using third-party event rules.
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
| |
(#8465)
This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so.
This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003.
Fixes https://github.com/matrix-org/synapse/issues/6700
Fixes https://github.com/matrix-org/synapse/issues/5304
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Expose `return_html_error`, and allow it to take a Jinja2 template instead of a raw string
* Clean up exception handling in SAML2ResponseResource
* use the existing code in `return_html_error` instead of re-implementing it
(giving it a jinja2 template rather than inventing a new form of template)
* do the exception-catching in the REST layer rather than in the handler
layer, to make sure we catch all exceptions.
|
| |
|
| |
Broke in #7534.
|
| | |
|
| |
|
|
| |
we have hundreds of path_regexes (see #5118), so let's not convert the same
bytes to str for each of them.
|
| |
|
|
|
| |
Allow REST endpoint implemnentations to raise a RedirectException, which will
redirect the user's browser to a given location.
|
| |
|
| |
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
* Fix servlet metric names
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove redundant check
* Cover all return paths
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Firstly, we always logged that the request was being handled via
`JsonResource._async_render`, so we change that to use the servlet name
we add to the request.
Secondly, we pass the exception information to the logger rather than
formatting it manually. This makes it consistent with other exception
logging, allwoing logging hooks and formatters to access the exception
information.
|
| |
|
|
| |
including quotes) (#4157)
|
| |
|
|
|
|
|
|
|
| |
If a HTTP handler throws an exception while processing a request we
automatically write a JSON error response. If the handler had already
started writing a response twisted throws an exception.
We should check for this case and simple abort the connection if there
was an error after the response had started being written.
|
| |\
| |
| | |
Use a producer to stream back responses
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The problem with dumping all of the json response into the Request object at
once is that doing so starts the timeout for the next request to be received:
so if it takes longer than 60s to stream back the response to the client, the
client never gets it.
The correct solution is to use a Producer; then the timeout is only started
once all of the content is sent over the TCP connection.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit moves a bunch of the logic for deciding when to log the receipt and
completion of HTTP requests into SynapseRequest, rather than in the request
handling wrappers.
Advantages of this are:
* we get logs for *all* requests (including OPTIONS and HEADs), rather than
just those that end up hitting handlers we've remembered to decorate
correctly.
* when a request handler wires up a Producer (as the media stuff does
currently, and as other things will do soon), we log at the point that all
of the traffic has been sent to the client.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.
That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.
The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | |\
| | |
| | | |
ConsentResource to gather policy consent from users
|
| | | |
| | |
| | |
| | |
| | | |
Hopefully there are enough comments and docs in this that it makes sense on its
own.
|
| |/ / |
|
| |/
|
|
|
|
|
|
|
|
|
|
| |
(instead of everywhere that writes a response. Or rather, the subset of places
which write responses where we haven't forgotten it).
This also means that we don't have to have the mysterious version_string
attribute in anything with a request handler.
Unfortunately it does mean that we have to pass the version string wherever we
instantiate a SynapseSite, which has been c&ped 150 times, but that is code
that ought to be cleaned up anyway really.
|
| |
|
|
|
|
| |
This is needless complexity; we might as well use the wrapper directly.
Also rename wrap_request_handler->wrap_json_request_handler.
|
| |
|
|
| |
... so that it can be used on non-JSON endpoints
|
| |
|
|
|
| |
The metrics are now available via the request, so this is redundant and can go
away at last.
|
| |
|
|
| |
it's much neater there.
|
| |
|
|
| |
less magic
|
| |
|
|
|
| |
It fits quite nicely here, and opens the path to getting rid of the
"include_metrics" mess.
|
| |
|
|
| |
... which is going to make it easier to move around.
|
| | |
|
| |
|
|
|
|
| |
This is useful in its own right, because server.py is full of stuff; but more
importantly, I want to do some refactoring that will cause a circular reference
as it is.
|
| |
|
|
| |
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| |\
| |
| | |
Add b prefixes to some strings that are bytes in py3
|
| | |
| |
| |
| |
| |
| | |
This has no effect on python2
Signed-off-by: Adrian Tschira <nota@notafile.com>
|
| | |
| |
| |
| |
| |
| |
| | |
We aren't ready to release this yet, so I'm reverting it for now.
This reverts commit d1679a4ed7947b0814e0f2af9b888a16c588f1a1, reversing
changes made to e089100c6231541c446e37e157dec8feed02d283.
|
| | | |
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
It is especially important that sync requests don't get cached, as if a
sync returns the same token given then the client will call sync with
the same parameters again. If the previous response was cached it will
get reused, resulting in the client tight looping making the same
request and never making any progress.
In general, clients will expect to get up to date data when requesting
APIs, and so its safer to do a blanket no cache policy than only
whitelisting APIs that we know will break things if they get cached.
|
| |\ |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
It's useful to know when there are peaks in incoming requests - which isn't
quite the same as there being peaks in outgoing responses, due to the time
taken to handle requests.
|
| | |
| |
| |
| |
| | |
rephrase the OPTIONS and unrecognised request handling so that they look
similar to the common flow.
|
| |\ \ |
|
| | |\ \
| | | |
| | | | |
Track db txn time in millisecs
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Avoid throwing a (harmless) exception when we try to write an error response to
an http request where the client has disconnected.
This comes up as a CRITICAL error in the logs which tends to mislead people
into thinking there's an actual problem
|
| | |/ /
|/| |
| | |
| | |
| | |
| | | |
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
|
| |/ /
| |
| |
| | |
... to reduce the amount of floating-point foo we do.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
In order to circumvent the number of duplicate foo:count metrics increasing
without bounds, it's time for a rearrangement.
The following are all deprecated, and replaced with synapse_util_metrics_block_count:
synapse_util_metrics_block_timer:count
synapse_util_metrics_block_ru_utime:count
synapse_util_metrics_block_ru_stime:count
synapse_util_metrics_block_db_txn_count:count
synapse_util_metrics_block_db_txn_duration:count
The following are all deprecated, and replaced with synapse_http_server_response_count:
synapse_http_server_requests
synapse_http_server_response_time:count
synapse_http_server_response_ru_utime:count
synapse_http_server_response_ru_stime:count
synapse_http_server_response_db_txn_count:count
synapse_http_server_response_db_txn_duration:count
The following are renamed (the old metrics are kept for now, but deprecated):
synapse_util_metrics_block_timer:total ->
synapse_util_metrics_block_time_seconds
synapse_util_metrics_block_ru_utime:total ->
synapse_util_metrics_block_ru_utime_seconds
synapse_util_metrics_block_ru_stime:total ->
synapse_util_metrics_block_ru_stime_seconds
synapse_util_metrics_block_db_txn_count:total ->
synapse_util_metrics_block_db_txn_count
synapse_util_metrics_block_db_txn_duration:total ->
synapse_util_metrics_block_db_txn_duration_seconds
synapse_http_server_response_time:total ->
synapse_http_server_response_time_seconds
synapse_http_server_response_ru_utime:total ->
synapse_http_server_response_ru_utime_seconds
synapse_http_server_response_ru_stime:total ->
synapse_http_server_response_ru_stime_seconds
synapse_http_server_response_db_txn_count:total ->
synapse_http_server_response_db_txn_count
synapse_http_server_response_db_txn_duration:total
synapse_http_server_response_db_txn_duration_seconds
|
| |/
|
|
|
| |
Make sure that we set the servlet name in the metrics object *before* calling
the servlet, in case the servlet throws an exception.
|
| |
|
|
|
| |
Use failure.Failure to recover our failure, which will give us a useful
stacktrace, unlike the rethrown exception.
|
| |
|
|
| |
what could possibly go wrong
|
| | |
|
| |
|
|
| |
Signed-off-by: Matthias Kesler <krombel@krombel.de>
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
JsonResource
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
and track the number of transactions that request started.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
ujson.loads
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
Conflicts:
synapse/http/server.py
|
| | |
| |
| |
| | |
wrapper decorator
|
| | |
| |
| |
| | |
separate wrapper function
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
* Now only the auth part goes to fallback, not the whole operation
* Auth fallback is a normal API endpoint, not a static page
* Params like the recaptcha pubkey can just live in the config
Involves a little engineering on JsonResource so its servlets aren't always forced to return JSON. I should document this more, in fact I'll do that now.
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
| |
work like bind_host: 'fe80::1%lo0'
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
method and responses by method and response code
|
| |
|
|
| |
response path, by 'continue'ing around a non-match or falling through
|
| | |
|
| | |
|
| |
|
|
| |
to process it.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
clients shouldn't cause ERROR level logging. Fix sql logging to use 'repr' rather than 'str'
|
| | |
|
| |
|
|
| |
hasn't been incorporated in time for launch.
|
| | |
|
| | |
|
| |
|
|
| |
hardcoding...
|
| | |
|
| |
|
|
| |
include the port due to SRV.
|
| |
|
|
| |
upload. Update the webclient to use new content repo api.
|
| |
|
|
| |
SynapseErrors) shouldn't have their full trace logged every time. Don't send responses to disconnected requests.
|
| | |
|
| |
|
|
| |
working.
|
| |
|
|
| |
base64 encoded user id of the uploader. Make a reasonable attempt to retry clashing upload paths. Try to guess a sensible file extension depending on the content type.
|
| |
|
|
| |
Added exception handling for content uploads. webclient: Only prefix the client API path on doRequest, not doBaseRequest (this would've broken the identity server auth too). Added matrixService.uploadContent. May not require mFileUpload anymore.
|
| |
|
|
| |
FileUploadResource. Added stub methods.
|
| |
|
|
| |
client path.
|
| |
|
|
| |
resource_for_federation or resource_for_client depending on what is being tested.
|
| |
|
|
| |
resource_for_client/federation/web_client to the HomeServer and hooked the C-S servlets to operate on resource_for_client. Dynamically construct the Resource tree.
|
| | |
|
| |
|
|
| |
all paths
|
| |
|
