| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
| |
Instead of storing and sending an ACK for every single row we send
synchronously, we instead do it asynchronously while batching up
updates.
|
| |
|
| |
Introduced in #7556
|
| |
|
|
|
|
|
|
| |
A couple of changes of significance:
* remove the `_last_ack < federation_position` condition, so that
updates will still be correctly processed after restart
* Correctly wire up send_federation_ack to the right class.
|
| | |
|
| | |
|
| |
|
|
| |
These are business as usual errors, rather than stuff we want to log at
error.
|
| |
|
|
|
| |
We don't really make any promises about returning accurate presence data when
presence is disabled, so we may as well just return a static response, rather
than making the master handle a request.
|
| |
|
| |
This allows workers to talk to each other over HTTP replication.
|
| |
|
|
|
| |
This is required as both event persistence and the background update needs access to this function. It should be perfectly safe for two workers to write to that table at the same time.
|
| |
|
| |
This is so that the logic can happen on both master and workers when we move event persistence out.
|
| |
|
| |
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
|
| | |
|
| |
|
|
| |
variables (#6391)
|
| | |
|
| |
|
| |
For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
|
| |
|
|
| |
We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
|
| |
|
|
|
| |
By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.
|
| |
|
|
|
| |
This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
|
| |
|
| |
Currently we never write to streams from workers, but that will change soon
|
| |
|
|
|
|
|
| |
Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication.
In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits.
Fixes (I hope) #7257.
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| | |
This is configured via the `redis` config options.
|
| | |
| |
| | |
The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
|
| | |
| |
| |
| |
| | |
Log warning when filesystem path is used.
Signed-off-by: Martin Milata <martin@martinmilata.cz>
|
| | |
| |
| |
| |
| |
| | |
By running this stuff with `run_in_background`, it won't be correctly reported
against the relevant CPU usage stats.
Fixes #7202
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Remove `conn_id` usage for UserSyncCommand.
Each tcp replication connection is assigned a "conn_id", which is used
to give an ID to a remotely connected worker. In a redis world, there
will no longer be a one to one mapping between connection and instance,
so instead we need to replace such usages with an ID generated by the
remote instances and included in the replicaiton commands.
This really only effects UserSyncCommand.
* Add CLEAR_USER_SYNCS command that is sent on shutdown.
This should help with the case where a synchrotron gets restarted
gracefully, rather than rely on 5 minute timeout.
|
| | |
| |
| | |
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
|
| |\ \
| | |
| | | |
Fix starting workers when federation sending not split out.
|
| | |/ |
|
| | |
| |
| |
| |
| | |
This just helps keep the rows closer to their streams, so that it's easier to
see what the format of each stream is.
|
| | |
| |
| |
| |
| |
| | |
`groups` != `receipts`
Introduced in #6964
|
| |\ \
| |/
|/|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Add 'device_lists_outbound_pokes' as extra table.
This makes sure we check all the relevant tables to get the current max
stream ID.
Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.
* Change device lists stream to have one row per id.
This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.
* Change device list replication to match new semantics.
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
* Newsfile
* Remove handling of multiple rows per ID
* Fix worker handling
* Comments from review
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
|
| | |
| |
| |
| |
| | |
This should be safe to do on all workers/masters because it is guarded by
a config option which will ensure it is only actually done on the worker
assigned as a pusher.
|
| |/
|
|
|
| |
* Break down monthly active users by appservice_id and emit via prometheus.
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| |
|
|
|
|
|
|
| |
Instead lets just warn if the worker has a media listener configured but
has the media repository disabled.
Previously non media repository workers would just ignore the media
listener.
|
| | |
|
| |
|
|
| |
Ensure good comprehension hygiene using flake8-comprehensions.
|
| |
|
|
|
| |
This may make gc go a bit faster as the gc will know things like
caches/data stores etc. are frozen without having to check.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
We were sending device updates down both the federation stream and
device streams. This mean there was a race if the federation sender
worker processed the federation stream first, as when the sender checked
if there were new device updates the slaved ID generator hadn't been
updated with the new stream IDs and so returned nothing.
This situation is correctly handled by events/receipts/etc by not
sending updates down the federation stream and instead having the
federation sender worker listen on the other streams and poke the
transaction queues as appropriate.
|
| | |
|
| |
|
|
|
| |
This will be used to retry outbound transactions to a remote server if
we think it might have come back up.
|
| |
|
|
|
|
|
|
|
|
| |
* Port synapse.replication.tcp to async/await
* Newsfile
* Correctly document type of on_<FOO> functions as async
* Don't be overenthusiastic with the asyncing....
|
| |
|
|
|
|
| |
AdditionalResource really doesn't add any value, and it gets in the way for
resources which want to support child resources or the like. So, if the
resource object already implements the IResource interface, don't bother
wrapping it.
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
This has caused some confusion for people who didn't notice it going away.
|
| |
|
|
|
|
| |
This looks like it got half-killed back in #888.
Fixes #6567.
|
| | |
|
| |
|
|
| |
`Failed to upgrade database` is not helpful, and it's unlikely that UPGRADE.rst
has anything useful.
|
| |
|
|
|
|
|
| |
If acme was enabled, the sdnotify startup hook would never be run because we
would try to add it to a hook which had already fired.
There's no need to delay it: we can sdnotify as soon as we've started the
listeners.
|
| |\
| |
| | |
Move database config from apps into HomeServer object
|
| | | |
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Synapse 1.7.0rc2 (2019-12-11)
=============================
Bugfixes
--------
- Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497))
- Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499))
- Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
|
| | |/
| |
| |
| | |
So that it has access to the get_retention_policy_for_room function which is required by filter_events_for_client.
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| | |
erikj/make_database_class
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| | |
Fix phone home stats
|
| |/ |
|
| | |
|
| |
|
| |
* remove psutil and replace with resource
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy.
The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`.
The proxy will then be used for
* push
* url previews
* phone-home stats
* recaptcha validation
* CAS auth validation
It will *not* be used for:
* Application Services
* Identity servers
* Outbound federation
* In worker configurations, connections from workers to masters
Fixes #4198.
|
| |
|
| |
Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
|
| |
|
|
|
| |
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
|
| | |
|
| |
|
|
|
|
| |
* type checking fixes
* changelog
|
| |
|
| |
This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
server to handle 3pid validation (#5987)
This is a combination of a few different PRs, finally all being merged into `develop`:
* #5875
* #5876
* #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](https://github.com/matrix-org/synapse/commit/891afb57cbdf9867f2848341b29c75d6f35eef5a#diff-e591d42d30690ffb79f63bb726200891) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future)
* #5835
* #5969
* #5940
Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged.
UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
|
| |
|
|
|
| |
Python will return a tuple whether there are parentheses around the returned values or not.
I'm just sick of my editor complaining about this all over the place :)
|
| | |
|
| |
|
|
|
|
| |
... to save OSes which don't use it from having to maintain a port.
Fixes #5865.
|
| |
|
|
| |
Signed-off-by: Chris Moos <chris@chrismoos.com>
|
| | |
|
| | |
|
| |
|
|
|
| |
This helps ensures that we only consider ourselves "up" once all the
startup functions have completed.
|
| |
|
|
| |
Fixes #5676.
|
| | |
|
| | |
|
| |
|
| |
Co-Authored-By: Aaron Raimist <aaron@raim.ist>
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Configure and initialise tracer
Includes config options for the tracer and sets up JaegerClient.
* Scope manager using LogContexts
We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.
* jaeger is a dependency now
* Carrier inject and extraction for Twisted Headers
* Trace federation requests on the way in and out.
The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.
* Create logcontext for new scope.
Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.
* Remove scope from logcontext if logcontext is top level
* Disable tracer if not configured
* typo
* Remove dependence on jaeger internals
* bools
* Set service name
* :Explicitely state that the tracer is disabled
* Black is the new black
* Newsfile
* Code style
* Use the new config setup.
* Generate config.
* Copyright
* Rename config to opentracing
* Remove user whitelisting
* Empty whitelist by default
* User ConfigError instead of RuntimeError
* Use isinstance
* Use tag constants for opentracing.
* Remove debug comment and no need to explicitely record error
* Two errors a "s(c)entry"
* Docstrings!
* Remove debugging brainslip
* Homeserver Whitlisting
* Better opentracing config comment
* linting
* Inclue worker name in service_name
* Make opentracing an optional dependency
* Neater config retreival
* Clean up dummy tags
* Instantiate tracing as object instead of global class
* Inlcude opentracing as a homeserver member.
* Thread opentracing to the request level
* Reference opetnracing through hs
* Instantiate dummy opentracin g for tests.
* About to revert, just keeping the unfinished changes just in case
* Revert back to global state, commit number:
9ce4a3d9067bf9889b86c360c05ac88618b85c4f
* Use class level methods in tracerutils
* Start and stop requests spans in a place where we
have access to the authenticated entity
* Seen it, isort it
* Make sure to close the active span.
* I'm getting black and blue from this.
* Logger formatting
Co-Authored-By: Erik Johnston <erik@matrix.org>
* Outdated comment
* Import opentracing at the top
* Return a contextmanager
* Start tracing client requests from the servlet
* Return noop context manager if not tracing
* Explicitely say that these are federation requests
* Include servlet name in client requests
* Use context manager
* Move opentracing to logging/
* Seen it, isort it again!
* Ignore twisted return exceptions on context exit
* Escape the scope
* Scopes should be entered to make them useful.
* Nicer decorator names
* Just one init, init?
* Don't need to close something that isn't open
* Docs make you smarter
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
* Fix 'utime went backwards' errors on daemonization.
Fixes #5608
* remove spurious debug
|
| | | |
|
| | | |
|
| | | |
|
| |/ |
|
| |
|
| |
This has no useful purpose on python3, and is generally a source of confusion.
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| |/
|
|
| |
* add monthly active users to phonehome stats
|
| |
|
|
|
|
|
|
|
|
|
|
| |
identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.
This PR is a culmination of 3 smaller PRs which have each been separately reviewed:
* #5308
* #5345
* #5368
|
| | |
|
| |
|
| |
Fixes #5271.
|
| |
|
| |
* expose SlavedProfileStore to ClientReaderSlavedStore
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit ce5bcefc609db40740c692bd53a1ef84ab675e8c.
This caused:
```
Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/synapse/src/synapse/app/client_reader.py", line 32, in <module>
from synapse.replication.slave.storage import SlavedProfileStore
ImportError: cannot import name 'SlavedProfileStore' from 'synapse.replication.slave.storage' (/home/synapse/src/synapse/replication/slave/storage/__init__.py)
error starting synapse.app.client_reader('/home/synapse/config/workers/client_reader.yaml') (exit code: 1); see above for logs
```
|
| |
|
| |
* expose SlavedProfileStore to ClientReaderSlavedStore
|
| |\
| |
| | |
Limit in flight DNS requests
|
| | |
| |
| |
| |
| |
| |
| | |
This is to work around a bug in twisted where a large number of
concurrent DNS requests cause it to tight loop forever.
c.f. https://twistedmatrix.com/trac/ticket/9620#ticket
|
| | |
| |
| |
| | |
It doesn't really belong under rest/client/v1 any more.
|
| | | |
|
| | | |
|
| |\ \
| |/
|/| |
Move some rest endpoints to client reader
|
| | | |
|
| |/
|
|
| |
add context to phonehome stats
|
| | |
|
| |
|
|
| |
... as a precursor to combining it with the CurrentStateDelta stream.
|
| | |
|
| |\
| |
| | |
Move client receipt processing to federation sender worker.
|
| | |
| |
| |
| |
| | |
This is mostly a prerequisite for #4730, but also fits with the general theme
of "move everything off the master that we possibly can".
|
| |\ \
| | |
| | | |
Allow passing --daemonize to workers
|
| | |/ |
|
| |/ |
|
| | |
|
| |\
| |
| | |
Move /account/3pid to client_reader
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |\
| |
| | |
Split /login into client_reader
|
| | | |
|
| |\ \
| | |
| | | |
Add basic optional sentry.io integration
|
| | | | |
|
| | | | |
|
| | | | |
|
| |\ \ \
| | |/
| |/| |
Split out registration to worker
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This allows registration to be handled by a worker, though the actual
write to the database still happens on master.
Note: due to the in-memory session map all registration requests must be
handled by the same worker.
|
| |/ /
| |
| |
| |
| |
| |
| |
| | |
When guest_access changes from allowed to forbidden all local guest
users should be kicked from the room. This did not happen when
revocation was received from federation on a worker.
Presumably broken in #4141
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
* Better logging for errors on startup
* Fix "TypeError: '>' not supported" when starting without an existing
certificate
* Fix a bug where an existing certificate would be reprovisoned every day
|
| | |
| |
| | |
Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>
|
| | |
| |
| |
| |
| | |
Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with
what we do elsewhere.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
I wanted to bring listen_tcp into line with listen_ssl in terms of returning a
list of ports, and wanted to check that was a safe thing to do - hence the
logging in `refresh_certificate`.
Also, pull the 'Synapse now listening' message up to homeserver.py, because it
was being duplicated everywhere else.
|
| |/
|
|
|
| |
turns out it doesn't really support ipv6, so let's hack around that by only
listening on ipv4 by default.
|
| |
|
|
|
|
| |
If TLS is disabled, it should not be an error if no cert is given.
Fixes #4554.
|
| |
|
|
|
| |
Rather than have to specify `no_tls` explicitly, infer whether we need to load
the TLS keys etc from whether we have any TLS-enabled listeners.
|
| |
|
|
| |
we aren't going to use them anyway.
|
| |
|
|
|
| |
Log which file we're reading keys and certs from, and refactor the code a bit
in preparation for other work
|
| |
|
|
|
| |
It's nothing to do with refreshing the certificates. No idea why it was here.
|
| | |
|
| |\
| |
| | |
New listener resource for the federation API "openid/userinfo" endpoint
|
| | |
| |
| |
| | |
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This allows the OpenID userinfo endpoint to be active even if the
federation resource is not active. The OpenID userinfo endpoint
is called by integration managers to verify user actions using the
client API OpenID access token. Without this verification, the
integration manager cannot know that the access token is valid.
The OpenID userinfo endpoint will be loaded in the case that either
"federation" or "openid" resource is defined. The new "openid"
resource is defaulted to active in default configuration.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
For all the homeserver classes, only the FrontendProxyServer passes
its reactor when doing the http listen. Looking at previous PR's looks
like this was introduced to make it possible to write a test, otherwise
when you try to run a test with the test homeserver it tries to
do a real bind to a port. Passing the reactor that the homeserver
is instantiated with should probably be the right thing to do anyway?
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
For all the homeserver classes, only the FrontendProxyServer passes
its reactor when doing the http listen. Looking at previous PR's looks
like this was introduced to make it possible to write a test, otherwise
when you try to run a test with the test homeserver it tries to
do a real bind to a port. Passing the reactor that the homeserver
is instantiated with should probably be the right thing to do anyway?
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Handle listening for ACME requests on IPv6 addresses
the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses
without extra quoting. Building a string which you are about to parse again
seems like a weird choice. Let's just use listenTCP, which is consistent with
what we do elsewhere.
* Clean up the default ACME config
make it look a bit more consistent with everything else, and tweak the defaults
to listen on port 80.
* newsfile
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
* load cert
* changelog
* fix
|
| |/ |
|
| |
|
|
|
|
|
|
| |
* Raise a ConfigError if an invalid resource is specified
* Require Jinja 2.9 for the consent resource
* changelog
|
| |
|
|
| |
optional dependencies to setuptools (#4298)
|
| |
|
| |
* ensure can report mau stats when hs.config.mau_stats_only is set
|
| |\ |
|
| | |\
| | |
| | | |
Stop installing Matrix Console by default
|
| | | |
| | |
| | |
| | | |
This is based on the work done by @krombel in #2601.
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is largely a precursor for the removal of the bundled webclient. The idea
is to present a page at / which reassures people that something is working, and
to give them some links for next steps.
The welcome page lives at `/_matrix/static/`, so is enabled alongside the other
`static` resources (which, in practice, means the client API is enabled). We'll
redirect to it from `/` if we have nothing better to display there.
It would be nice to have a way to disable it (in the same way that you might
disable the nginx welcome page), but I can't really think of a good way to do
that without a load of ickiness.
It's based on the work done by @krombel for #2601.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This implements both a SAML2 metadata endpoint (at
`/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at
`/_matrix/saml2/authn_response`). If the SAML2 response matches what's been
configured, we complete the SSO login flow by redirecting to the client url
(aka `RelayState` in SAML2 jargon) with a login token.
What we don't yet have is anything to build a SAML2 request and redirect the
user to the identity provider. That is left as an exercise for the reader.
|
| |/
|
|
| |
Sometimes it's useful for synapse to generate its own .well-known file.
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | |\
| | |
| | | |
Clean up the way logcontexts and threads work in the pushers
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
`on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher`
now always return synchronously, so we can remove the `defer.gatherResults` on
their results, and the `run_as_background_process` wrappers can be removed too
because the PusherPool methods will now complete quickly enough.
|
| | | |
| | |
| | |
| | |
| | | |
This is public (or at least, called from outside the class), so ought to have a
better name.
|
| |/ / |
|
| |\ \
| | |
| | | |
Run MAU queries as background processes
|
| | |/
| |
| |
| | |
Fixes #3820
|
| |\ \
| | |
| | | |
Make psutil an explicit dependency
|
| | |/
| |
| |
| |
| |
| |
| |
| | |
As of #4027, we require psutil to be installed, so it should be in our
dependency list. We can also remove some of the conditional import code
introduced by #992.
Fixes #4062.
|
| |/
|
|
| |
on py3) (#4068)
|
| | |
|
| | |
|
| |
|
|
|
| |
I think this got forgotten in #3932. We were getting away with it because it
was the last call in this function.
|
| |
|
|
| |
Signed-off-by: Schnuffle <schnuffle@github.com>
|
| |\
| |
| | |
Move synctl into top dir to avoid a symlink
|
| | |
| |
| |
| |
| |
| | |
symlinks apparently break setuptools on python3 and alpine
(https://bugs.python.org/issue31940), so let's stop using a symlink and just
use the file directly.
|
| |/
|
|
|
|
|
|
| |
ExpiringCache required that `start()` be called before it would actually
start expiring entries. A number of places didn't do that.
This PR removes `start` from ExpiringCache, and automatically starts
backround reaping process on creation instead.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
We should explicitly close any db connections we open, because failing to do so
can block other transactions as per
https://github.com/matrix-org/synapse/issues/3682.
Let's also try to factor out some of the boilerplate by having server classes
define their datastore class rather than duplicating the whole of `setup`.
|
| |\
| |
| | |
Allow profile updates to happen on workers
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Turns out that the user directory handling is fairly racey as a bunch
of stuff assumes that the processing happens on master, which it doesn't
when there is a synapse.app.user_dir worker. So lets just call the
function directly until we actually get round to fixing it, since it
doesn't make the situation any worse.
|
| | |\
| | |
| | |
| | | |
erikj/split_profiles
|
| | |\ \
| | | |
| | | |
| | | | |
erikj/split_profiles
|
| | | | | |
|
| |\ \ \ \
| | | | |
| | | | | |
Logcontexts for replication command handlers
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Run the handlers for replication commands as background processes. This should
improve the visibility in our metrics, and reduce the number of "running db
transaction from sentinel context" warnings.
Ideally it means converting the things that fire off deferreds into the night
into things that actually return a Deferred when they are done. I've made a bit
of a stab at this, but it will probably be leaky.
|
| |\ \ \ \ \
| | | | | |
| | | | | | |
Fix logcontexts for running pushers
|
| | |/ / / /
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
First of all, avoid resetting the logcontext before running the pushers, to fix
the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning.
Instead, give them their own "background process" logcontexts.
|
| | |_|_|/
|/| | | |
|
| | | | | |
|
| |/ / / |
|
| | | |
| | |
| | |
| | |
| | | |
Inbound federation requires calculating push, which in turn relies on
having access to account data.
|
| |\ \ \
| | | |
| | | |
| | | | |
erikj/split_federation
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | |\ \ \
| | |_|/
| |/| | |
send SNI for federation requests
|
| | | |\ \ |
|
| | | |\ \ \
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
send_sni_for_federation_requests
# Conflicts:
# synapse/crypto/context_factory.py
|
| | | | | | | |
|
| | | | | | | |
|
| |\| | | | |
| | | | | |
| | | | | |
| | | | | | |
erikj/split_federation
|
| | | | | | | |
|
| | | | | | | |
|
| | | | | | | |
|
| | |\ \ \ \ \
| | | |_|_|/
| | |/| | | |
|
| | | |_|_|/
| |/| | | |
|
| | | | | | |
|
| | |/ / /
|/| | | |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| |/ / / |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
It turns out that looping_call does check the deferred returned by its
callback, and (at least in the case of client_ips), we were relying on this,
and I broke it in #3604.
Update run_as_background_process to return the deferred, and make sure we
return it to clock.looping_call.
|
| | | |
| | |
| | |
| | |
| | | |
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This allows us to handle /context/ requests on the client_reader worker
without having to pull in all the various stream handlers (e.g.
precence, typing, pushers etc). The only thing the token gets used for
is pagination, and that ignores everything but the room portion of the
token.
|
| | | | |
|
