| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
(#15891)
|
| |
|
|
|
| |
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).
This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
|
| |
|
|
|
|
| |
Revert "Federation outbound proxy (#15773)"
This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
|
| |
|
|
|
|
|
| |
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).
This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
into existing rooms (#15748)
Context for why we're removing the implementation:
- https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010
- https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734
Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737
Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
|
| |
|
|
|
| |
R30v2 has been out since 2021-07-19 (https://github.com/matrix-org/synapse/pull/10332)
and we started collecting stats on 2021-08-16. Since it's been over a year now
(almost 2 years), this is enough grace period for us to now rip it out.
|
| |
|
|
| |
A bunch of comments and variables are out of date and use
obsolete terms.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
and 3.11 (#15599)
* Usage that is compatible with Python 3.8 and 3.11
> Since Python 3.10, instead of passing value and tb, an exception object can
be passed as the first argument. If value and tb are provided, the first
argument is ignored in order to provide backwards compatibility.
>
> -- https://docs.python.org/3/library/traceback.html
* Add changelog
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I found the error in the **Before** really vague and obtuse and didn't realize port `5432` corresponded to the Postgres port until searching the codebase. It says to check the logs but that wasn't my first instinct. It's just more obvious if we just print the full thing which gives context of the error type and the traceback to the relevant area of code.
#### Before
```
$ poetry run python -m synapse.app.homeserver -c homeserver.yaml
**********************************************************************************
Error during initialisation:
connection to server at "localhost" (::1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
There may be more information in the logs.
**********************************************************************************
```
#### After
```sh
$ poetry run python -m synapse.app.homeserver -c homeserver.yaml
**********************************************************************************
Error during initialisation:
Traceback (most recent call last):
File "/home/eric/Documents/github/element/synapse/synapse/app/homeserver.py", line 352, in setup
hs.setup()
File "/home/eric/Documents/github/element/synapse/synapse/server.py", line 337, in setup
self.datastores = Databases(self.DATASTORE_CLASS, self)
File "/home/eric/Documents/github/element/synapse/synapse/storage/databases/__init__.py", line 65, in __init__
with make_conn(database_config, engine, "startup") as db_conn:
File "/home/eric/Documents/github/element/synapse/synapse/storage/database.py", line 161, in make_conn
native_db_conn = engine.module.connect(**db_params)
File "/home/eric/.cache/pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.10/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "localhost" (::1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
There may be more information in the logs.
**********************************************************************************
```
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Add IReactorUNIX to ISynapseReactor type hint.
* Create listen_unix().
Two options, 'path' to the file and 'mode' of permissions(not umask, recommend 666 as default as
nginx/other reverse proxies write to it and it's setup as user www-data)
For the moment, leave the option to always create a PID lockfile turned on by default
* Create UnixListenerConfig and wire it up.
Rename ListenerConfig to TCPListenerConfig, then Union them together into ListenerConfig.
This spidered around a bit, but I think I got it all. Metrics and manhole have been placed
behind a conditional in case of accidental putting them onto a unix socket.
Use new helpers to get if a listener is configured for TLS, and to help create a site tag
for logging.
There are 2 TODO things in parse_listener_def() to finish up at a later point.
* Refactor SynapseRequest to handle logging correctly when using a unix socket.
This prevents an exception when an IP address can not be retrieved for a request.
* Make the 'Synapse now listening on Unix socket' log line a little prettier.
* No silent failures on generic workers when trying to use a unix socket with metrics or manhole.
* Inline variables in app/_base.py
* Update docstring for listen_unix() to remove reference to a hardcoded permission of 0o666 and add a few comments saying where the default IS declared.
* Disallow both a unix socket and a ip/port combo on the same listener resource
* Linting
* Changelog
* review: simplify how listen_unix returns(and get rid of a type: ignore)
* review: fix typo from ConfigError in app/homeserver.py
* review: roll conditional for http_options.tag into get_site_tag() helper(and add docstring)
* review: enhance the conditionals for checking if a port or path is valid, remove a TODO line
* review: Try updating comment in get_client_ip_if_available to clarify what is being retrieved and why
* Pretty up how 'Synapse now listening on Unix Socket' looks by decoding the byte string.
* review: In parse_listener_def(), raise ConfigError if neither socket_path nor port is declared(and fix a typo)
|
| | |
|
| | |
|
| |
|
|
| |
To directly write to the open file, instead of writing to an
in-memory string first.
|
| |
|
|
|
|
|
|
|
|
|
| |
* Add account data to to export command
* newsfile
* remove not needed function
* update newsfile
* adopt #14973
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
It's important that collections returned from `@cached` methods are not
modified, otherwise future retrievals from the cache will return the
modified collection.
This applies to the return values from `@cached` methods and the values
inside the dictionaries returned by `@cachedList` methods. It's not
necessary for the dictionaries returned by `@cachedList` methods
themselves to be read-only.
Signed-off-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
|
| |
|
| |
Signed-off-by: Jason Little realtyem@gmail.com
|
| |
|
|
|
|
| |
* The user's profile information.
* The user's devices.
* The user's connections / IP address information.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Enable Complement tests for Faster Remote Room Joins on worker-mode
* (dangerous) Add an override to allow Complement to use FRRJ under workers
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Fix race where we didn't send out replication notification
* MORE HACKS
* Fix get_un_partial_stated_rooms_token to take instance_name
* Fix bad merge
* Remove warning
* Correctly advance un_partial_stated_room_stream
* Fix merge
* Add another notify_replication
* Fixups
* Create a separate ReplicationNotifier
* Fix test
* Fix portdb
* Create a separate ReplicationNotifier
* Fix test
* Fix portdb
* Fix presence test
* Newsfile
* Apply suggestions from code review
* Update changelog.d/14752.misc
Co-authored-by: Erik Johnston <erik@matrix.org>
* lint
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
| |
Fixes: #8780
|
| |
|
|
|
|
|
| |
Add logic to ClientRestResource to decide whether to mount servlets
or not based on whether the current process is a worker.
This is clearer to see what a worker runs than the completely separate /
copy & pasted list of servlets being mounted for workers.
|
| |
|
|
| |
v1.69.0 and disabled by default in Synapse v1.71.0. (#14538)
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* Fix typechecking errors introduced in #14128
* Changelog
* Correct annotations
so that context_factory works if you don't use TLS
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix missing SSL support in worker endpoints.
* Add changelog
* SSL for Replication endpoint
* Remove unit test change
* Refactor listener creation to reduce duplicated code
* Fix the logger message
* Update synapse/app/_base.py
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Update synapse/app/_base.py
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Update synapse/app/_base.py
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add config documentation for new TLS option
Co-authored-by: Tuomas Ojamies <tojamies@palantir.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| | |
|
| |
|
|
|
| |
launcher. (#14084)
Co-authored-by: David Robertson <davidr@element.io>
|
| | |
|
| | |
|
| |
|
|
|
| |
To fix the response for unknown endpoints under that prefix.
See MSC3743.
|
| |
|
| |
Co-authored-by: Eric Eastwood <erice@element.io>
|
| |
|
| |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
| |
This reverts commit 6d543d6d9f56e39199b7e460d0081b02d61f12be.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Update mypy and mypy-zope
* Unignore assigning to LogRecord attributes
Presumably https://github.com/python/typeshed/pull/8064 makes this ok
Cherry-picked from #13521
* Remove unused ignores due to mypy ParamSpec fixes
https://github.com/python/mypy/pull/12668
Cherry-picked from #13521
* Remove additional unused ignores
* Fix new mypy complaints related to `assertGreater`
Presumably due to https://github.com/python/typeshed/pull/8077
* Changelog
* Reword changelog
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
| |
(#13885)
|
| |
|
|
|
|
|
|
|
|
| |
This should mean that logs from worker processes are flushed before shutdown.
When a test completes, Complement stops the docker container, which means that
synapse will receive a SIGTERM. Currently, the `complement_fork_starter` exits
immediately (without notifying the worker processes), which means that the
workers never get a chance to flush their logs before the whole container is
vaped. We can fix this by propagating the SIGTERM to the children.
|
| | |
|
| |
|
| |
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
|
| |
|
|
| |
(#13671)
|
| |
|
|
|
| |
metric names. (#13540)
Co-authored-by: David Robertson <davidr@element.io>
|
| | |
|
| |
|
|
|
|
| |
Closes #13415.
Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
|
| |
|
|
|
|
| |
Synapse does not currently support faster room joins on deployments with
workers.
Signed-off-by: Sean Quah <seanq@matrix.org>
|
| |
|
|
|
|
|
|
| |
This reverts commit f383b9b3eceaa082d5ae690550fe41460b711779. Other PRs
were seeing mypy failures that looked to be related to mypy-zope.
Confusingly, we didn't see this on #13521.
Revert this for now and investigate later.
|
| | |
|
| |
|
|
|
| |
Reverts commit fa71bb18b527d1a3e2629b48640ea67fff2f8c59, and tweaks documentation.
Signed-off-by: 3nprob <git@3n.anonaddy.com>
|
| | |
|
| |
|
| |
This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Drop support for delegating email validation
Delegating email validation to an IS is insecure (since it allows the owner of
the IS to do a password reset on your HS), and has long been deprecated. It
will now cause a config error at startup.
* Update unit test which checks for email verification
Give it an `email` config instead of a threepid delegate
* Remove unused method `requestEmailToken`
* Simplify config handling for email verification
Rather than an enum and a boolean, all we need here is a single bool, which
says whether we are or are not doing email verification.
* update docs
* changelog
* upgrade.md: fix typo
* update version number
this will be in 1.64, not 1.63
* update version number
this one too
|
| | |
|
| |
|
|
|
| |
in CPU-constrained environments. (#13127)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
The replication logic for groups is no longer used, so the message
passing infrastructure can be removed.
|
| |
|
|
|
|
|
|
|
| |
Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove:
* All API endpoints (including Client-Server, Server-Server, and admin).
* Documented configuration options (and the experimental flag, which is now unused).
* Special handling during room upgrades.
* The `groups` section of the `/sync` response.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
(superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. (#12654)
Co-authored-by: Shay <hillerys@element.io>
|
| | |
|
| |
|
|
| |
`notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.
There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)
I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.
* Installing extras before typechecking
Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.
Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
|
| | |
|
| |
|
|
| |
background worker. (#12251)
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
* Make it possible to enable compression for the metrics HTTP resource
This can provide significant bandwidth savings pulling metrics from
synapse instances.
* Add changelog file.
* Fix type hint
|
| |
|
|
| |
configuration flag. (#12200)
|
| |
|
|
|
|
| |
Since the object it returns is a ReplicationCommandHandler.
This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Pull runtime dep checks into their own module
* Reimplement `check_requirements` using `importlib`
I've tried to make this clearer. We start by working out which of
Synapse's requirements we need to be installed here and now. I was
surprised that there wasn't an easier way to see which packages were
installed by a given extra.
I've pulled out the error messages into functions that deal with "is
this for an extra or not". And I've rearranged the loop over two
different sets of requirements into one loop with a "must be instaled"
flag.
I hope you agree that this is clearer.
* Test cases
|
| |
|
|
|
|
|
| |
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
|
| |
|
|
| |
* Require latest matrix-common
* Use the common function
|
| |
|
|
|
| |
Re-applies the changes from 3e0cfd447e17658a937fe62555db9e968f00b15b (#11602),
reverting d93ec0a0ba5f6d2fbf2bc321086d4ad4c03136e0 (#11764) now that the conflict
with the webclient listener was fixed in 119edf51eb3e4f5ed5139dc370f5d7aed46edc1c (#11895).
|
| |
|
| |
Also remove support for non-HTTP(S) web_client_location.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#11816)
* add check that gc.freeze is available before calling
* newsfragment
* lint
* Update comment
Co-authored-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* remove reference in comments to python3.6
* upgrade tox python env in script
* bump python version in example for completeness
* upgrade python version requirement in setup doc
* upgrade necessary python version in __init__.py
* upgrade python version in setup.py
* newsfragment
* drops refs to bionic and replace with focal
* bump refs to postgres 9.6 to 10
* fix hanging ci
* try installing tzdata first
* revert change made in b979f336
* ignore new random mypy error while debugging other error
* fix lint error for temporary workaround
* revert change to install list
* try passing env var
* export debian frontend var?
* move line and add comment
* bump pillow dependency
* bump lxml depenency
* install libjpeg-dev for pillow
* bump automat version to one compatible with py3.8
* add libwebp for pillow
* bump twisted trunk python version
* change suffix of newsfragment
* remove redundant python 3.7 checks
* lint
|
| |
|
|
|
| |
To not change the behaviour during the deprecation period.
Follow-up to #11774.
|
| |
|
|
|
|
|
| |
`web_client_location`. (#11774)
This changes the behaviour of the root endpoint to redirect
directly to the configuration of `web_client_location` if it is
given an HTTP(S) URL.
|
| |\ |
|
| | | |
|
| |/
|
|
|
| |
Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection.
Part 1 of a quest to simplify the reactor monkey-patching.
|
| |
|
|
| |
Instead of returning 404 errors with HTML bodies when an unknown
prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
|
| |
|
|
| |
This adds some misc. type hints to helper methods used
in the `synapse.config` module.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Add check to catch syanpse master process starting when workers are configured
* add test to verify that starting master process with worker config raises error
* newsfragment
* specify config.worker.worker_app in check
* update test
* report specific config option that triggered the error
Co-authored-by: reivilibre <oliverw@matrix.org>
* clarify error message
Co-authored-by: reivilibre <oliverw@matrix.org>
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
| |
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
| |
This also makes additional updates where the implementation
had drifted from the approved MSC.
Unstable endpoints will be removed at a later data.
|
| |
|
|
| |
Signed-off-by: Tulir Asokan <tulir@beeper.com>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Add support for `/_matrix/media/v3` APIs
Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Update `workers.md` to use v3 client and media APIs
Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add changelog
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| |
|
|
|
| |
This is one of the changes required to support Matrix 1.1
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| | |
|
| |
|
|
| |
Fixes https://github.com/matrix-org/synapse/issues/8308
|
| | |
|
| |
|
| |
Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
|
| |
|
|
|
| |
The DNS threadpool must be explicitly stopped, otherwise Synapse will
hang indefinitely when asked to shut down.
|
| |
|
|
|
|
| |
This is to stop large bursts of lookups starving out other users of the
thread pools.
Fixes #11049.
|
| | |
|
| |
|
| |
Fix broken export-data admin command and add a test for it to CI
|
| |
|
|
|
|
| |
interface (#10548)
Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| |
|
|
|
|
| |
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.
It also improves type hints since mypy can now resolve the
types of the configuration variables.
|
| |
|
|
| |
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
|
| |
|
| |
Also refactors some of the registration of endpoints on workers.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Use `gc.freeze()` on exit to exclude all existing objects from the final GC.
In testing, this sped up shutdown by up to a few seconds.
`gc.freeze()` runs in constant time, so there is little chance of performance
regression.
Signed-off-by: Sean Quah <seanq@element.io>
|
| | |
|
| |
|
|
|
| |
Signed-off-by: Callum Brown <callum@calcuode.com>
This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
|
| |\ |
|
| | |
| |
| | |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |/
|
| |
Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
|
| | |
|
| |
|
| |
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
|
| |
|
| |
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
|
| |
|
|
|
| |
(#10386)
Port the third-party event rules interface to the generic module interface introduced in v1.37.0
|
| |
|
| |
Follow-up to #10332.
|
| |
|
|
|
| |
impression of retention given by the old R30 metric (#10332)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
|
| |
|
|
|
|
|
|
|
| |
This PR is tantamount to running
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```
Part of #9744
|
| | |
|
| | |
|
| |
|
| |
This adds a simple best effort locking mechanism that works cross workers.
|
| |
|
|
|
|
|
|
| |
(#10191)
* Defer stdio redirection until we are about to start the reactor
* Catch and handle exceptions during startup
|
| |
|
| |
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
|
| |
|
|
|
| |
Fixes #9778
ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
|
| |
|
| |
As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.
|
| | |
|
| | |
|
| |
|
|
|
| |
we don't need to reload the tls cert if we don't have any tls listeners.
Follow-up to #9280.
|
| | |
|
| | |
|
| |
|
|
|
| |
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.
This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.
Adds a `gc_min_seconds_between` config option to override the defaults.
Fixes #9890.
|
| |
|
|
|
|
| |
* Simplify `start_listening` callpath
* Correctly check the size of uploaded files
|
| | |
|
| | |
|
| |
|
|
| |
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
|
| |
|
| |
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| |
|
|
|
|
|
|
|
|
|
|
| |
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.
This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.
A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync.
The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:
* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
|
| |
|
|
| |
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
|
| |
|
| |
This warning is somewhat confusing to users, so let's suppress it
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
* Adds B00 to ignored checks.
* Fixes remaining issues.
|
| |
|
| |
Should fix some remaining warnings
|
| |
|
|
|
|
|
| |
In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again.
Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Split ShardedWorkerHandlingConfig
This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).
* Remove special cases in ShardedWorkerHandlingConfig.
`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.
To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
|
| | |
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes #8966.
* Factor out build_synapse_client_resource_tree
Start a function which will mount resources common to all workers.
* Move sso init into build_synapse_client_resource_tree
... so that we don't have to do it for each worker
* Fix SSO-login-via-a-worker
Expose the SSO login endpoints on workers, like the documentation says.
* Update workers config for new endpoints
Add documentation for endpoints recently added (#8942, #9017, #9262)
* remove submit_token from workers endpoints list
this *doesn't* work on workers (yet).
* changelog
* Add a comment about the odd path for SAML2Resource
|
| | |
| |
| | |
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
|
| | |
| |
| | |
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
|
| |/
|
|
|
|
|
|
|
|
|
|
| |
* synapse.app.base: only call gc.freeze() on CPython
gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.
Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.
Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
|
| | |
|
| |
|
|
|
|
|
| |
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.
For now it's still hardcoded with a single provider.
|
| | |
|
| | |
|
| |
|
|
| |
Factor out the exception handling in the startup code to a utility function,
and fix the some logging and exit code stuff.
|
| | |
|
| | |
|
| |
|
|
|
| |
During login, if there are multiple IdPs enabled, offer the user a choice of
IdPs.
|
| |
|
| |
Adds the redacts endpoint to workers that have the client listener.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow
`UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser
back to the client, we redirect to a username-picker resource, which allows the user to enter a username.
We *then* complete the SSO flow (including doing the client permission checks).
The static resources for the username picker itself (in
https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker)
are essentially lifted wholesale from
https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res.
As the comment says, we might want to think about making them customisable, but that can be a follow-up.
Fixes #8876.
|
| |
|
| |
Fixes #8892
|
| |
|
|
|
|
|
|
|
|
| |
The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError,
and it will be magically turned into a legible error message. There's a few components to it:
* Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it
into an absolute path.
* Generally improving the way ConfigErrors get printed.
* Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:
* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events
The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
|
| |
|
|
|
|
|
|
|
| |
We can get a SIGHUP at any point, including times where we are not in a
sane state. By deferring calling the handlers until the next reactor
tick we ensure that we don't get unexpected conflicts, e.g. trying to
flush logs from the signal handler while the code was in the process of
writing a log entry.
Fixes #8769.
|
| |
|
|
|
|
|
| |
Fixes:
```
builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given
```
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#8536)
* Fix outbound federaion with multiple event persisters.
We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.
Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.
* Change some interfaces to use RoomStreamToken.
By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
|
| | |
|
| |
|
| |
All handlers now available via get_*_handler() methods on the HomeServer.
|
| | |
|
| |
|
| |
By reporting the log level of the synapse logger as a string.
|
| |
|
|
|
| |
Lots of different module apis is not easy to maintain.
Rather than adding yet another ModuleApi(hs, hs.get_auth_handler()) incantation, first add an hs.get_module_api() method and use it where possible.
|
| |
|
|
|
| |
This is so we can tell what is going on when things are taking a while to start up.
The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
|
| | |
|
| |
|
|
|
|
|
| |
This converts calls like super(Foo, self) -> super().
Generated with:
sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
|
| | |
|
| |
|
|
|
| |
This PR adds a confirmation step to resetting your user password between clicking the link in your email and your password actually being reset.
This is to better align our password reset flow with the industry standard of requiring a confirmation from the user after email validation.
|
| |
|
|
|
| |
By importing from canonicaljson the simplejson module was still being used
in some situations. After this change the std lib json is consistenty used
throughout Synapse.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Duplicating function signatures between server.py and server.pyi is
silly. This commit changes that by changing all `build_*` methods to
`get_*` methods and changing the `_make_dependency_method` to work work
as a descriptor that caches the produced value.
There are some changes in other files that were made to fix the typing
in server.py.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This has long been something I've wanted to do. Basically the `Daemonize` code
is both too flexible and not flexible enough, in that it offers a bunch of
features that we don't use (changing UID, closing FDs in the child, logging to
syslog) and doesn't offer a bunch that we could do with (redirecting stdout/err
to a file instead of /dev/null; having the parent not exit until the child is
running).
As a first step, I've lifted the Daemonize code and removed the bits we don't
use. This should be a non-functional change. Fixing everything else will come
later.
|
| | |
|
| |\ |
|
| | | |
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Handling of incoming typing stream updates from replication was not
hooked up on master, effecting set ups where typing was handled on a
different worker.
This is really only a problem if the master process is also handling
sync requests, which is unlikely for those that are at the stage of
moving typing off.
The other observable effect is that if a worker restarts or a
replication connect drops then the typing worker will issue a
`POSITION typing`, triggering master process to try and stream *all*
typing updates from position 0.
Fixes #7907
|
| | |
|
| | |
|
| | |
|
| | |
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This ended up being a bit more invasive than I'd hoped for (not helped by
generic_worker duplicating some of the code from homeserver), but hopefully
it's an improvement.
The idea is that, rather than storing unstructured `dict`s in the config for
the listener configurations, we instead parse it into a structured
`ListenerConfig` object.
|
| | | |
|
| |/ |
|
| | |
|
| |
|
|
|
| |
Based on #7619
async's `get_user_id_by_threepid` and its call stack.
|
| | |
|
| |
|
|
|
|
| |
Instead of storing and sending an ACK for every single row we send
synchronously, we instead do it asynchronously while batching up
updates.
|
| |
|
| |
Introduced in #7556
|
| |
|
|
|
|
|
|
| |
A couple of changes of significance:
* remove the `_last_ack < federation_position` condition, so that
updates will still be correctly processed after restart
* Correctly wire up send_federation_ack to the right class.
|
| | |
|
| | |
|
| |
|
|
| |
These are business as usual errors, rather than stuff we want to log at
error.
|
| |
|
|
|
| |
We don't really make any promises about returning accurate presence data when
presence is disabled, so we may as well just return a static response, rather
than making the master handle a request.
|
| |
|
| |
This allows workers to talk to each other over HTTP replication.
|
| |
|
|
|
| |
This is required as both event persistence and the background update needs access to this function. It should be perfectly safe for two workers to write to that table at the same time.
|
| |
|
| |
This is so that the logic can happen on both master and workers when we move event persistence out.
|
| |
|
| |
This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
|
| | |
|
| |
|
|
| |
variables (#6391)
|
| | |
|
| |
|
| |
For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
|
| |
|
|
| |
We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
|
| |
|
|
|
| |
By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.
|
| |
|
|
|
| |
This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
|
| |
|
| |
Currently we never write to streams from workers, but that will change soon
|
| |
|
|
|
|
|
| |
Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication.
In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits.
Fixes (I hope) #7257.
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| | |
This is configured via the `redis` config options.
|
| | |
| |
| | |
The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
|
| | |
| |
| |
| |
| | |
Log warning when filesystem path is used.
Signed-off-by: Martin Milata <martin@martinmilata.cz>
|
| | |
| |
| |
| |
| |
| | |
By running this stuff with `run_in_background`, it won't be correctly reported
against the relevant CPU usage stats.
Fixes #7202
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Remove `conn_id` usage for UserSyncCommand.
Each tcp replication connection is assigned a "conn_id", which is used
to give an ID to a remotely connected worker. In a redis world, there
will no longer be a one to one mapping between connection and instance,
so instead we need to replace such usages with an ID generated by the
remote instances and included in the replicaiton commands.
This really only effects UserSyncCommand.
* Add CLEAR_USER_SYNCS command that is sent on shutdown.
This should help with the case where a synchrotron gets restarted
gracefully, rather than rely on 5 minute timeout.
|
| | |
| |
| | |
This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
|
| |\ \
| | |
| | | |
Fix starting workers when federation sending not split out.
|
| | |/ |
|
| | |
| |
| |
| |
| | |
This just helps keep the rows closer to their streams, so that it's easier to
see what the format of each stream is.
|
| | |
| |
| |
| |
| |
| | |
`groups` != `receipts`
Introduced in #6964
|
| |\ \
| |/
|/|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Add 'device_lists_outbound_pokes' as extra table.
This makes sure we check all the relevant tables to get the current max
stream ID.
Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.
* Change device lists stream to have one row per id.
This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.
* Change device list replication to match new semantics.
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
* Newsfile
* Remove handling of multiple rows per ID
* Fix worker handling
* Comments from review
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).
|
| | |
| |
| |
| |
| | |
This should be safe to do on all workers/masters because it is guarded by
a config option which will ensure it is only actually done on the worker
assigned as a pusher.
|
| |/
|
|
|
| |
* Break down monthly active users by appservice_id and emit via prometheus.
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
| |
|
|
|
|
|
|
| |
Instead lets just warn if the worker has a media listener configured but
has the media repository disabled.
Previously non media repository workers would just ignore the media
listener.
|
