summary refs log tree commit diff
path: root/synapse/app (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Make cached account data/tags/admin types immutable (#16325)Patrick Cloke2023-09-181-7/+7
|
* Add automatic purge after all users forget a room (#15488)Mathieu Velten2023-09-151-0/+2
| | | | | | Also add restore of purge/shutdown rooms after a synapse restart. Co-authored-by: Eric Eastwood <erice@matrix.org> Co-authored-by: Erik Johnston <erikj@matrix.org>
* Use StrCollection in additional places. (#16301)Patrick Cloke2023-09-131-7/+5
|
* Implements a task scheduler for resumable potentially long running tasks ↵Mathieu Velten2023-08-211-0/+2
| | | | (#15891)
* Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-183-0/+4
| | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-103-4/+0
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Federation outbound proxy (#15773)Eric Eastwood2023-07-053-0/+4
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-161-2/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* Remove old R30 because R30v2 supercedes it (#10428)reivilibre2023-05-191-4/+0
| | | | | R30v2 has been out since 2021-07-19 (https://github.com/matrix-org/synapse/pull/10332) and we started collecting stats on 2021-08-16. Since it's been over a year now (almost 2 years), this is enough grace period for us to now rip it out.
* Update code to refer to "workers". (#15606)Patrick Cloke2023-05-162-4/+4
| | | | A bunch of comments and variables are out of date and use obsolete terms.
* `traceback.format_exception(...)` usage that is compatible with Python 3.7 ↵Eric Eastwood2023-05-161-1/+1
| | | | | | | | | | | | | and 3.11 (#15599) * Usage that is compatible with Python 3.8 and 3.11 > Since Python 3.10, instead of passing value and tb, an exception object can be passed as the first argument. If value and tb are provided, the first argument is ignored in order to provide backwards compatibility. > > -- https://docs.python.org/3/library/traceback.html * Add changelog
* Print full startup/initialization error (#15569)Eric Eastwood2023-05-111-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I found the error in the **Before** really vague and obtuse and didn't realize port `5432` corresponded to the Postgres port until searching the codebase. It says to check the logs but that wasn't my first instinct. It's just more obvious if we just print the full thing which gives context of the error type and the traceback to the relevant area of code. #### Before ``` $ poetry run python -m synapse.app.homeserver -c homeserver.yaml ********************************************************************************** Error during initialisation: connection to server at "localhost" (::1), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? There may be more information in the logs. ********************************************************************************** ``` #### After ```sh $ poetry run python -m synapse.app.homeserver -c homeserver.yaml ********************************************************************************** Error during initialisation: Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/synapse/app/homeserver.py", line 352, in setup hs.setup() File "/home/eric/Documents/github/element/synapse/synapse/server.py", line 337, in setup self.datastores = Databases(self.DATASTORE_CLASS, self) File "/home/eric/Documents/github/element/synapse/synapse/storage/databases/__init__.py", line 65, in __init__ with make_conn(database_config, engine, "startup") as db_conn: File "/home/eric/Documents/github/element/synapse/synapse/storage/database.py", line 161, in make_conn native_db_conn = engine.module.connect(**db_params) File "/home/eric/.cache/pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.10/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) psycopg2.OperationalError: connection to server at "localhost" (::1), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? There may be more information in the logs. ********************************************************************************** ```
* Move ThirdPartyEventRules into module_api/callbacks (#15535)Andrew Morgan2023-05-041-1/+3
|
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-181-1/+1
|
* Experimental Unix socket support (#15353)Jason Little2023-04-033-54/+114
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add IReactorUNIX to ISynapseReactor type hint. * Create listen_unix(). Two options, 'path' to the file and 'mode' of permissions(not umask, recommend 666 as default as nginx/other reverse proxies write to it and it's setup as user www-data) For the moment, leave the option to always create a PID lockfile turned on by default * Create UnixListenerConfig and wire it up. Rename ListenerConfig to TCPListenerConfig, then Union them together into ListenerConfig. This spidered around a bit, but I think I got it all. Metrics and manhole have been placed behind a conditional in case of accidental putting them onto a unix socket. Use new helpers to get if a listener is configured for TLS, and to help create a site tag for logging. There are 2 TODO things in parse_listener_def() to finish up at a later point. * Refactor SynapseRequest to handle logging correctly when using a unix socket. This prevents an exception when an IP address can not be retrieved for a request. * Make the 'Synapse now listening on Unix socket' log line a little prettier. * No silent failures on generic workers when trying to use a unix socket with metrics or manhole. * Inline variables in app/_base.py * Update docstring for listen_unix() to remove reference to a hardcoded permission of 0o666 and add a few comments saying where the default IS declared. * Disallow both a unix socket and a ip/port combo on the same listener resource * Linting * Changelog * review: simplify how listen_unix returns(and get rid of a type: ignore) * review: fix typo from ConfigError in app/homeserver.py * review: roll conditional for http_options.tag into get_site_tag() helper(and add docstring) * review: enhance the conditionals for checking if a port or path is valid, remove a TODO line * review: Try updating comment in get_client_ip_if_available to clarify what is being retrieved and why * Pretty up how 'Synapse now listening on Unix Socket' looks by decoding the byte string. * review: In parse_listener_def(), raise ConfigError if neither socket_path nor port is declared(and fix a typo)
* Add information on uploaded media to user export command. (#15107)Dirk Klimpel2023-02-231-0/+10
|
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-224-4/+2
|
* Use `json.dump` in `FileExfiltrationWriter` (#15095)Dirk Klimpel2023-02-221-8/+8
| | | | To directly write to the open file, instead of writing to an in-memory string first.
* Add account data to export command (#14969)Dirk Klimpel2023-02-171-1/+14
| | | | | | | | | | | * Add account data to to export command * newsfile * remove not needed function * update newsfile * adopt #14973
* Return read-only collections from `@cached` methods (#13755)Sean Quah2023-02-101-2/+2
| | | | | | | | | | | | | It's important that collections returned from `@cached` methods are not modified, otherwise future retrievals from the cache will return the modified collection. This applies to the return values from `@cached` methods and the values inside the dictionaries returned by `@cachedList` methods. It's not necessary for the dictionaries returned by `@cachedList` methods themselves to be read-only. Signed-off-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Allow enabling the asyncio reactor in complement (#14858)realtyem2023-02-011-2/+19
| | | Signed-off-by: Jason Little realtyem@gmail.com
* Add more user information to export-data command. (#14894)Dirk Klimpel2023-02-011-1/+31
| | | | | | * The user's profile information. * The user's devices. * The user's connections / IP address information.
* Enable Faster Remote Room Joins against worker-mode Synapse. (#14752)reivilibre2023-01-221-7/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Enable Complement tests for Faster Remote Room Joins on worker-mode * (dangerous) Add an override to allow Complement to use FRRJ under workers * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Fix race where we didn't send out replication notification * MORE HACKS * Fix get_un_partial_stated_rooms_token to take instance_name * Fix bad merge * Remove warning * Correctly advance un_partial_stated_room_stream * Fix merge * Add another notify_replication * Fixups * Create a separate ReplicationNotifier * Fix test * Fix portdb * Create a separate ReplicationNotifier * Fix test * Fix portdb * Fix presence test * Newsfile * Apply suggestions from code review * Update changelog.d/14752.misc Co-authored-by: Erik Johnston <erik@matrix.org> * lint Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Add listener `health` (#14747)Dirk Klimpel2023-01-112-0/+6
| | | Fixes: #8780
* Use ClientRestResource on both the main process and workers. (#14528)Patrick Cloke2022-12-021-72/+2
| | | | | | | Add logic to ClientRestResource to decide whether to mount servlets or not based on whether the current process is a worker. This is clearer to see what a worker runs than the completely separate / copy & pasted list of servlets being mounted for workers.
* Remove legacy Prometheus metrics names. They were deprecated in Synapse ↵reivilibre2022-11-243-14/+4
| | | | v1.69.0 and disabled by default in Synapse v1.71.0. (#14538)
* Remove need for `worker_main_http_uri` setting to use /keys/upload. (#14400)realtyem2022-11-161-101/+2
|
* Fix typechecking errors introduced in #14128 (#14455)David Robertson2022-11-151-2/+2
| | | | | | | | | * Fix typechecking errors introduced in #14128 * Changelog * Correct annotations so that context_factory works if you don't use TLS
* Support using SSL on worker endpoints. (#14128)Tuomas Ojamies2022-11-153-52/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix missing SSL support in worker endpoints. * Add changelog * SSL for Replication endpoint * Remove unit test change * Refactor listener creation to reduce duplicated code * Fix the logger message * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Add config documentation for new TLS option Co-authored-by: Tuomas Ojamies <tojamies@palantir.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Merge/remove `Slaved*` stores into `WorkerStores` (#14375)Nick Mills-Barrett2022-11-112-21/+59
|
* Fix type annotation causing import time error in the Complement forking ↵reivilibre2022-11-011-2/+2
| | | | | launcher. (#14084) Co-authored-by: David Robertson <davidr@element.io>
* Improve `RawHeaders` type hints (#14303)David Robertson2022-10-281-4/+4
|
* Fix logging error on SIGHUP (#14258)Richard van der Hoff2022-10-211-1/+1
|
* Use servlets for /key/ endpoints. (#14229)Patrick Cloke2022-10-202-29/+17
| | | | | To fix the response for unknown endpoints under that prefix. See MSC3743.
* Mark /relations endpoint as usable on workers. (#14028)Patrick Cloke2022-10-121-0/+2
| | | Co-authored-by: Eric Eastwood <erice@element.io>
* Update mypy and mypy-zope, attempt 3 (#13993)David Robertson2022-09-301-3/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Revert "Update mypy and mypy-zope (#13925)"David Robertson2022-09-301-1/+3
| | | | This reverts commit 6d543d6d9f56e39199b7e460d0081b02d61f12be.
* Update mypy and mypy-zope (#13925)David Robertson2022-09-301-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Unignore assigning to LogRecord attributes Presumably https://github.com/python/typeshed/pull/8064 makes this ok Cherry-picked from #13521 * Remove unused ignores due to mypy ParamSpec fixes https://github.com/python/mypy/pull/12668 Cherry-picked from #13521 * Remove additional unused ignores * Fix new mypy complaints related to `assertGreater` Presumably due to https://github.com/python/typeshed/pull/8077 * Changelog * Reword changelog Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Handle the case of remote users leaving a partial join room for device lists ↵Erik Johnston2022-09-271-1/+1
| | | | (#13885)
* Complement image: propagate SIGTERM to all workers (#13914)Richard van der Hoff2022-09-261-2/+30
| | | | | | | | | | This should mean that logs from worker processes are flushed before shutdown. When a test completes, Complement stops the docker container, which means that synapse will receive a SIGTERM. Currently, the `complement_fork_starter` exits immediately (without notifying the worker processes), which means that the workers never get a chance to flush their logs before the whole container is vaped. We can fix this by propagating the SIGTERM to the children.
* Fix Prometheus recording rules to not use legacy metric names. (#13718)reivilibre2022-09-081-3/+3
|
* Remove configuration options for direct TCP replication. (#13647)Patrick Cloke2022-09-061-11/+0
| | | Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
* Share some metrics between the Prometheus exporter and the phone home stats ↵Brendan Abolivier2022-09-052-3/+15
| | | | (#13671)
* Add experimental configuration option to allow disabling legacy Prometheus ↵reivilibre2022-08-243-5/+46
| | | | | metric names. (#13540) Co-authored-by: David Robertson <davidr@element.io>
* Drop support for delegating email validation, round 2 (#13596)David Robertson2022-08-231-2/+1
|
* Add support for compression to federation responses (#13537)Ayush Anand2022-08-181-1/+4
| | | | | | Closes #13415. Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
* Refuse to start when `faster_joins` is enabled on a worker deployment (#13531)Sean Quah2022-08-161-0/+7
| | | | | | Synapse does not currently support faster room joins on deployments with workers. Signed-off-by: Sean Quah <seanq@matrix.org>
* Revert "Update locked versions of mypy and mypy-zope (#13521)"David Robertson2022-08-151-1/+3
| | | | | | | | This reverts commit f383b9b3eceaa082d5ae690550fe41460b711779. Other PRs were seeing mypy failures that looked to be related to mypy-zope. Confusingly, we didn't see this on #13521. Revert this for now and investigate later.
* Update locked versions of mypy and mypy-zope (#13521)David Robertson2022-08-151-3/+1
|
* Revert "Drop support for delegating email validation (#13192)" (#13406)3nprob2022-07-291-1/+2
| | | | | Reverts commit fa71bb18b527d1a3e2629b48640ea67fff2f8c59, and tweaks documentation. Signed-off-by: 3nprob <git@3n.anonaddy.com>
* Remove old empty/redundant slaved stores. (#13349)Nick Mills-Barrett2022-07-212-24/+36
|
* Use cache store remove base slaved (#13329)Nick Mills-Barrett2022-07-212-4/+0
| | | This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
* Drop support for delegating email validation (#13192)Richard van der Hoff2022-07-121-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Drop support for delegating email validation Delegating email validation to an IS is insecure (since it allows the owner of the IS to do a password reset on your HS), and has long been deprecated. It will now cause a config error at startup. * Update unit test which checks for email verification Give it an `email` config instead of a threepid delegate * Remove unused method `requestEmailToken` * Simplify config handling for email verification Rather than an enum and a boolean, all we need here is a single bool, which says whether we are or are not doing email verification. * update docs * changelog * upgrade.md: fix typo * update version number this will be in 1.64, not 1.63 * update version number this one too
* Add a `filter_event_for_clients_with_state` function (#13222)Erik Johnston2022-07-111-1/+12
|
* Improve startup times in Complement test runs against workers, particularly ↵reivilibre2022-06-302-2/+196
| | | | | in CPU-constrained environments. (#13127) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Don't use keyword arguments when initialising modules (#13060)Brendan Abolivier2022-06-151-1/+1
|
* Refactor entry points so that they all have a `main` function. (#13052)reivilibre2022-06-1410-10/+60
|
* Fix Synapse git info missing in version strings (#12973)David Robertson2022-06-074-12/+8
|
* Remove groups replication code. (#12900)Patrick Cloke2022-05-312-4/+0
| | | | The replication logic for groups is no longer used, so the message passing infrastructure can be removed.
* Remove user-visible groups/communities code (#12553)Patrick Cloke2022-05-251-4/+0
| | | | | | | | | Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove: * All API endpoints (including Client-Server, Server-Server, and admin). * Documented configuration options (and the experimental flag, which is now unused). * Special handling during room upgrades. * The `groups` section of the `/sync` response.
* Add the `batch_send` endpoint to generic workers (#12868)Nick Mills-Barrett2022-05-251-0/+2
|
* Add the `/account/whoami` endpoint to generic workers (#12866)Nick Mills-Barrett2022-05-251-1/+2
|
* Reload cache factors from disk on SIGHUP (#12673)David Robertson2022-05-112-34/+46
|
* Add the `update_user_directory_from_worker` configuration option ↵reivilibre2022-05-102-17/+1
| | | | | (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. (#12654) Co-authored-by: Shay <hillerys@element.io>
* Use `ParamSpec` in a few places (#12667)David Robertson2022-05-091-4/+10
|
* Add the `notify_appservices_from_worker` configuration option (superseding ↵reivilibre2022-05-061-16/+0
| | | | `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-2/+2
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Add missing type hints to config classes. (#12402)Patrick Cloke2022-04-111-1/+3
|
* Move `update_client_ip` background job from the main process to the ↵reivilibre2022-04-012-4/+0
| | | | background worker. (#12251)
* Add restrictions by default to open registration in Synapse (#12091)Shay2022-03-251-0/+17
|
* Make it possible to enable compression for the metrics HTTP resource (#12258)Nick Mills-Barrett2022-03-211-1/+4
| | | | | | | | | | * Make it possible to enable compression for the metrics HTTP resource This can provide significant bandwidth savings pulling metrics from synapse instances. * Add changelog file. * Fix type hint
* Deprecate the groups/communities endpoints and add an experimental ↵Patrick Cloke2022-03-121-1/+2
| | | | configuration flag. (#12200)
* Rename get_tcp_replication to get_replication_command_handler. (#12192)Patrick Cloke2022-03-102-2/+2
| | | | | | Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.
* Use importlib.metadata to read requirements (#12088)David Robertson2022-03-012-4/+4
| | | | | | | | | | | | | | | | | | * Pull runtime dep checks into their own module * Reimplement `check_requirements` using `importlib` I've tried to make this clearer. We start by working out which of Synapse's requirements we need to be installed here and now. I was surprised that there wasn't an easier way to see which packages were installed by a given extra. I've pulled out the error messages into functions that deal with "is this for an extra or not". And I've rearranged the loop over two different sets of requirements into one loop with a "must be instaled" flag. I hope you agree that this is clearer. * Test cases
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-234-8/+12
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Use version string helper from matrix-common (#11979)David Robertson2022-02-144-8/+12
| | | | * Require latest matrix-common * Use the common function
* Return JSON errors for unknown resources under /matrix/client. (#11930)Patrick Cloke2022-02-081-7/+2
| | | | | Re-applies the changes from 3e0cfd447e17658a937fe62555db9e968f00b15b (#11602), reverting d93ec0a0ba5f6d2fbf2bc321086d4ad4c03136e0 (#11764) now that the conflict with the webclient listener was fixed in 119edf51eb3e4f5ed5139dc370f5d7aed46edc1c (#11895).
* Remove support for the webclient listener. (#11895)Patrick Cloke2022-02-031-31/+3
| | | Also remove support for non-HTTP(S) web_client_location.
* Check that `gc` method is available before using in `synapse/app/_base` ↵Shay2022-01-251-6/+8
| | | | | | | | | | | | | | | (#11816) * add check that gc.freeze is available before calling * newsfragment * lint * Update comment Co-authored-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* Log modules at startup (#11813)Brendan Abolivier2022-01-251-1/+2
|
* Drop support for and remove references to EOL Python 3.6 (#11683)Shay2022-01-211-8/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove reference in comments to python3.6 * upgrade tox python env in script * bump python version in example for completeness * upgrade python version requirement in setup doc * upgrade necessary python version in __init__.py * upgrade python version in setup.py * newsfragment * drops refs to bionic and replace with focal * bump refs to postgres 9.6 to 10 * fix hanging ci * try installing tzdata first * revert change made in b979f336 * ignore new random mypy error while debugging other error * fix lint error for temporary workaround * revert change to install list * try passing env var * export debian frontend var? * move line and add comment * bump pillow dependency * bump lxml depenency * install libjpeg-dev for pillow * bump automat version to one compatible with py3.8 * add libwebp for pillow * bump twisted trunk python version * change suffix of newsfragment * remove redundant python 3.7 checks * lint
* Fix redirecting to the webclient for non-HTTP(S) web_client_location. (#11783)Patrick Cloke2022-01-201-1/+8
| | | | | To not change the behaviour during the deprecation period. Follow-up to #11774.
* Add deprecation warnings for `webclient` listener and non-HTTP(S) ↵Patrick Cloke2022-01-201-5/+7
| | | | | | | `web_client_location`. (#11774) This changes the behaviour of the root endpoint to redirect directly to the configuration of `web_client_location` if it is given an HTTP(S) URL.
* Merge branch 'master' into developAndrew Morgan2022-01-181-2/+7
|\
| * Partially revert #11602 to prevent webclient overriding client resource (#11764)Andrew Morgan2022-01-181-2/+7
| |
* | Simplify GC prometheus metrics (#11723)Richard van der Hoff2022-01-131-1/+2
|/ | | | | Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection. Part 1 of a quest to simplify the reactor monkey-patching.
* Return JSON errors for unknown resources under /matrix/client. (#11602)Patrick Cloke2021-12-201-7/+2
| | | | Instead of returning 404 errors with HTML bodies when an unknown prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
* Additional type hints for config module. (#11465)Patrick Cloke2021-12-011-1/+2
| | | | This adds some misc. type hints to helper methods used in the `synapse.config` module.
* Don't start Synapse master process if `worker_app` is set (#11416)Shay2021-11-301-0/+7
| | | | | | | | | | | | | | | | | | | | | * Add check to catch syanpse master process starting when workers are configured * add test to verify that starting master process with worker config raises error * newsfragment * specify config.worker.worker_app in check * update test * report specific config option that triggered the error Co-authored-by: reivilibre <oliverw@matrix.org> * clarify error message Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* Expose worker & homeserver as entrypoints in `setup.py` (#11449)Maximilian Bosch2021-11-301-1/+5
| | | Co-authored-by: reivilibre <oliverw@matrix.org>
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-0/+1
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Fix using MSC2716 batch sending with event persistence workers (#11220)Tulir Asokan2021-11-291-0/+2
| | | | Signed-off-by: Tulir Asokan <tulir@beeper.com>
* Add type annotations to `synapse.metrics` (#10847)Sean Quah2021-11-171-1/+1
|
* Add support for `/_matrix/media/v3` APIs (#11371)Aaron R2021-11-172-4/+11
| | | | | | | | | | | | | * Add support for `/_matrix/media/v3` APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Update `workers.md` to use v3 client and media APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Add changelog Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add support for `/_matrix/client/v3` APIs (#11318)Aaron R2021-11-161-0/+1
| | | | | This is one of the changes required to support Matrix 1.1 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-106-151/+191
|
* Support for serving server well-known files (#11211)Richard van der Hoff2021-11-012-2/+5
| | | | Fixes https://github.com/matrix-org/synapse/issues/8308
* Add metrics to the threadpools (#11178)Erik Johnston2021-11-011-0/+5
|
* Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-281-0/+14
| | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* Shut down the DNS threadpool (#11190)Sean Quah2021-10-271-0/+1
| | | | | The DNS threadpool must be explicitly stopped, otherwise Synapse will hang indefinitely when asked to shut down.
* Move DNS lookups into separate thread pool (#11177)Erik Johnston2021-10-261-1/+12
| | | | | | This is to stop large bursts of lookups starving out other users of the thread pools. Fixes #11049.
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-225-11/+15
|
* Fix broken export-data admin command and add a test for it to CI (#11078)Hillery Shay2021-10-181-8/+6
| | | Fix broken export-data admin command and add a test for it to CI
* Port the Password Auth Providers module interface to the new generic ↵Azrenbeth2021-10-131-0/+2
| | | | | | interface (#10548) Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-063-4/+4
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-295-18/+18
|
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-244-9/+9
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-235-13/+17
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-231-1/+1
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-135-15/+19
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Document that /spaces & /hierarchy can be routed to workers. (#10648)Patrick Cloke2021-09-081-24/+18
| | | Also refactors some of the registration of endpoints on workers.
* Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-063-4/+16
|
* Skip the final GC on shutdown to improve restart times (#10712)Sean2021-09-011-0/+7
| | | | | | | | | | Use `gc.freeze()` on exit to exclude all existing objects from the final GC. In testing, this sped up shutdown by up to a few seconds. `gc.freeze()` runs in constant time, so there is little chance of performance regression. Signed-off-by: Sean Quah <seanq@element.io>
* Persist room hierarchy pagination sessions to the database. (#10613)Patrick Cloke2021-08-241-0/+2
|
* Implement MSC3231: Token authenticated registration (#10142)Callum Brown2021-08-211-1/+5
| | | | | Signed-off-by: Callum Brown <callum@calcuode.com> This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
* Merge branch 'release-v1.41' into developErik Johnston2021-08-182-4/+2
|\
| * Remove the unused public_room_list_stream (#10565)Andrew Morgan2021-08-172-4/+2
| | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Port the PresenceRouter module interface to the new generic interface (#10524)Azrenbeth2021-08-171-0/+2
|/ | | Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-171-20/+20
|
* Remove shebang line from module files (#10415)V024602021-07-2913-13/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* Fix typo that causes R30v2 to actually be old R30 (#10486)reivilibre2021-07-271-1/+1
| | | Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* Port the ThirdPartyEventRules module interface to the new generic interface ↵Brendan Abolivier2021-07-201-0/+2
| | | | | (#10386) Port the third-party event rules interface to the generic module interface introduced in v1.37.0
* Factorise `get_datastore` calls in phone_stats_home. (#10427)reivilibre2021-07-191-16/+18
| | | Follow-up to #10332.
* Add a new version of the R30 phone-home metric, which removes a false ↵reivilibre2021-07-191-0/+4
| | | | | impression of retention given by the old R30 metric (#10332) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-192-8/+4
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-151-1/+1
|
* Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-051-2/+9
|
* Add a distributed lock (#10269)Erik Johnston2021-06-291-0/+2
| | | This adds a simple best effort locking mechanism that works cross workers.
* Ensure that errors during startup are written to the logs and the console. ↵Richard van der Hoff2021-06-213-13/+53
| | | | | | | | (#10191) * Defer stdio redirection until we are about to start the reactor * Catch and handle exceptions during startup
* Standardise the module interface (#10062)Brendan Abolivier2021-06-183-0/+17
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Remove support for ACME v1 (#10194)Brendan Abolivier2021-06-172-50/+1
| | | | | Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
* Rip out the DNS lookup limiter (#10190)Richard van der Hoff2021-06-171-104/+0
| | | As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.
* Clean up a broken import in admin_cmd.py (#10154)Andrew Morgan2021-06-111-2/+0
|
* Move some more endpoints off master (#10084)Erik Johnston2021-05-271-2/+2
|
* Remove redundant code to reload tls cert (#10054)Richard van der Hoff2021-05-271-4/+1
| | | | | we don't need to reload the tls cert if we don't have any tls listeners. Follow-up to #9280.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-2/+0
|
* Export jemalloc stats to prometheus when used (#9882)Erik Johnston2021-05-061-0/+2
|
* Optionally track memory usage of each LruCache (#9881)Erik Johnston2021-05-052-0/+2
| | | | | This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
* Limit how often GC happens by time. (#9902)Erik Johnston2021-05-052-0/+6
| | | | | | | | | | | | | | | Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.
* Improved validation for received requests (#9817)Richard van der Hoff2021-04-234-21/+45
| | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-232-15/+11
|
* Split presence out of master (#9820)Erik Johnston2021-04-231-27/+4
|
* remove `HomeServer.get_config` (#9815)Richard van der Hoff2021-04-142-10/+10
| | | | Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.
* Move some replication processing out of generic_worker (#9796)Erik Johnston2021-04-141-468/+2
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1415-15/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan2021-04-061-1/+2
| | | | | | | | | | | | At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
* Add type hints for the federation sender. (#9681)Patrick Cloke2021-03-291-7/+0
| | | | Includes an abstract base class which both the FederationSender and the FederationRemoteSendQueue must implement.
* Suppress CryptographyDeprecationWarning (#9698)Richard van der Hoff2021-03-263-14/+25
| | | This warning is somewhat confusing to users, so let's suppress it
* Fix lintBrendan Abolivier2021-03-191-4/+4
|
* Move support for MSC3026 behind an experimental flagBrendan Abolivier2021-03-181-1/+6
|
* Implement MSC3026: busy presence stateBrendan Abolivier2021-03-181-0/+1
|
* Enable flake8-bugbear, but disable most checks. (#9499)Jonathan de Jong2021-03-161-1/+3
| | | | * Adds B00 to ignored checks. * Fixes remaining issues.
* Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539)Richard van der Hoff2021-03-031-2/+13
| | | Should fix some remaining warnings
* Allow bytecode again (#9502)Jonathan de Jong2021-02-261-2/+0
| | | | | | | In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again. Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Clean up `ShardedWorkerHandlingConfig` (#9466)Erik Johnston2021-02-242-32/+2
| | | | | | | | | | | | | | | | | * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.
* Fix deleting pushers when using sharded pushers. (#9465)Erik Johnston2021-02-221-3/+0
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-162-11/+6
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Merge branch 'social_login' into developRichard van der Hoff2021-02-012-18/+9
|\
| * Fix SSO on workers (#9271)Richard van der Hoff2021-02-012-20/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #8966. * Factor out build_synapse_client_resource_tree Start a function which will mount resources common to all workers. * Move sso init into build_synapse_client_resource_tree ... so that we don't have to do it for each worker * Fix SSO-login-via-a-worker Expose the SSO login endpoints on workers, like the documentation says. * Update workers config for new endpoints Add documentation for endpoints recently added (#8942, #9017, #9262) * remove submit_token from workers endpoints list this *doesn't* work on workers (yet). * changelog * Add a comment about the odd path for SAML2Resource
| * Split out a separate endpoint to complete SSO registration (#9262)Richard van der Hoff2021-02-011-0/+2
| | | | | | There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
* | Add phone home stats for encrypted messages. (#9283)Jan Christian Grünhage2021-02-011-2/+7
| | | | | | Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
* | Fixes for PyPy compatibility (#9270)Ivan Shapovalov2021-01-301-1/+2
|/ | | | | | | | | | | | * synapse.app.base: only call gc.freeze() on CPython gc.freeze() is an implementation detail of CPython garbage collector, and notably does not exist on PyPy. Rather than playing whack-a-mole and skipping the call when under PyPy, simply restrict it to CPython because the whole gc module is implementation-defined. Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-181-1/+14
|
* Split OidcProvider out of OidcHandler (#9107)Richard van der Hoff2021-01-141-1/+0
| | | | | | | The idea here is that we will have an instance of OidcProvider for each configured IdP, with OidcHandler just doing the marshalling of them. For now it's still hardcoded with a single provider.
* Register the /devices endpoint on workers. (#9092)Patrick Cloke2021-01-131-0/+2
|
* Move more encryption endpoints off master (#9068)Erik Johnston2021-01-111-2/+10
|
* Clean up exception handling in the startup code (#9059)Richard van der Hoff2021-01-113-100/+119
| | | | Factor out the exception handling in the startup code to a utility function, and fix the some logging and exit code stuff.
* Remove unused SynapseService (#9058)Richard van der Hoff2021-01-111-22/+0
|
* Allow running sendToDevice on workers (#9044)Erik Johnston2021-01-071-0/+3
|
* Add initial support for a "pick your IdP" page (#9017)Richard van der Hoff2021-01-051-0/+2
| | | | | During login, if there are multiple IdPs enabled, offer the user a choice of IdPs.
* Allow redacting events on workers (#8994)Patrick Cloke2020-12-291-27/+4
| | | Adds the redacts endpoint to workers that have the client listener.
* Implement a username picker for synapse (#8942)Richard van der Hoff2020-12-181-0/+2
| | | | | | | | | | | | | | The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow `UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser back to the client, we redirect to a username-picker resource, which allows the user to enter a username. We *then* complete the SSO flow (including doing the client permission checks). The static resources for the username picker itself (in https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker) are essentially lifted wholesale from https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. As the comment says, we might want to think about making them customisable, but that can be a follow-up. Fixes #8876.
* Fix deadlock on SIGHUP (#8918)Erik Johnston2020-12-101-1/+5
| | | Fixes #8892
* Better formatting for config errors from modules (#8874)Richard van der Hoff2020-12-081-4/+42
| | | | | | | | | | The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError, and it will be magically turned into a legible error message. There's a few components to it: * Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it into an absolute path. * Generally improving the way ConfigErrors get printed. * Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-1/+0
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Defer SIGHUP handlers to reactor. (#8817)Erik Johnston2020-11-261-1/+9
| | | | | | | | | We can get a SIGHUP at any point, including times where we are not in a sane state. By deferring calling the handlers until the next reactor tick we ensure that we don't get unexpected conflicts, e.g. trying to flush logs from the signal handler while the code was in the process of writing a log entry. Fixes #8769.
* Fix SIGHUP handler (#8697)Richard van der Hoff2020-11-061-3/+2
| | | | | | | Fixes: ``` builtins.TypeError: _reload_logging_config() takes 1 positional argument but 2 were given ```
* Fix not sending events over federation when using sharded event persisters ↵Erik Johnston2020-10-141-4/+0
| | | | | | | | | | | | | | | | | (#8536) * Fix outbound federaion with multiple event persisters. We incorrectly notified federation senders that the minimum persisted stream position had advanced when we got an `RDATA` from an event persister. Notifying of federation senders already correctly happens in the notifier, so we just delete the offending line. * Change some interfaces to use RoomStreamToken. By enforcing use of `RoomStreamTokens` we make it less likely that people pass in random ints that they got from somewhere random.
* Move additional tasks to the background worker, part 3 (#8489)Patrick Cloke2020-10-091-11/+3
|
* Remove the deprecated Handlers object (#8494)Patrick Cloke2020-10-091-1/+1
| | | All handlers now available via get_*_handler() methods on the HomeServer.
* Move additional tasks to the background worker (#8458)Patrick Cloke2020-10-072-22/+15
|
* Include the configured log level in phone home stats. (#8477)Patrick Cloke2020-10-071-0/+7
| | | By reporting the log level of the synapse logger as a string.
* Combine `SpamCheckerApi` with the more generic `ModuleApi`. (#8464)Richard van der Hoff2020-10-071-2/+1
| | | | | Lots of different module apis is not easy to maintain. Rather than adding yet another ModuleApi(hs, hs.get_auth_handler()) incantation, first add an hs.get_module_api() method and use it where possible.
* Add logging on startup/shutdown (#8448)Erik Johnston2020-10-021-0/+5
| | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* Allow background tasks to be run on a separate worker. (#8369)Patrick Cloke2020-10-025-182/+213
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-181-3/+3
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Remove obsolete __future__ imports (#8337)Jonathan de Jong2020-09-171-2/+0
|
* Show a confirmation page during user password reset (#8004)Andrew Morgan2020-09-101-0/+10
| | | | | This PR adds a confirmation step to resetting your user password between clicking the link in your email and your password actually being reset. This is to better align our password reset flow with the industry standard of requiring a confirmation from the user after email validation.
* Directly import json from the standard library. (#8259)Patrick Cloke2020-09-081-2/+1
| | | | | By importing from canonicaljson the simplejson module was still being used in some situations. After this change the std lib json is consistenty used throughout Synapse.
* Stop sub-classing object (#8249)Patrick Cloke2020-09-042-3/+3
|
* Convert the main methods run by the reactor to async. (#8213)Patrick Cloke2020-09-022-21/+17
|
* Add a comment about _LimitedHostnameResolverRichard van der Hoff2020-08-291-0/+7
|
* Change HomeServer definition to work with typing.Erik Johnston2020-08-111-3/+5
| | | | | | | | | | Duplicating function signatures between server.py and server.pyi is silly. This commit changes that by changing all `build_*` methods to `get_*` methods and changing the `_make_dependency_method` to work work as a descriptor that caches the produced value. There are some changes in other files that were made to fix the typing in server.py.
* Add health check endpoint (#8048)Erik Johnston2020-08-072-2/+9
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-053-11/+11
|
* re-implement daemonize (#8011)Richard van der Hoff2020-08-041-13/+3
| | | | | | | | | | | | | This has long been something I've wanted to do. Basically the `Daemonize` code is both too flexible and not flexible enough, in that it offers a bunch of features that we don't use (changing UID, closing FDs in the child, logging to syslog) and doesn't offer a bunch that we could do with (redirecting stdout/err to a file instead of /dev/null; having the parent not exit until the child is running). As a first step, I've lifted the Daemonize code and removed the bits we don't use. This should be a non-functional change. Fixing everything else will come later.
* Convert ACME code to async/await. (#7989)Patrick Cloke2020-08-031-7/+6
|
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2020-07-301-3/+3
|\
| * Update worker docs with recent enhancements (#7969)Erik Johnston2020-07-291-3/+3
| |
* | Ensure that remove_pusher is always async (#7981)Patrick Cloke2020-07-301-1/+1
|/
* Fix typing replication not being handled on master (#7959)Erik Johnston2020-07-271-7/+0
| | | | | | | | | | | | | | | | Handling of incoming typing stream updates from replication was not hooked up on master, effecting set ups where typing was handled on a different worker. This is really only a problem if the master process is also handling sync requests, which is unlikely for those that are at the stage of moving typing off. The other observable effect is that if a worker restarts or a replication connect drops then the typing worker will issue a `POSITION typing`, triggering master process to try and stream *all* typing updates from position 0. Fixes #7907
* Convert synapse.app to async/await. (#7868)Patrick Cloke2020-07-172-20/+17
|
* Allow moving typing off master (#7869)Erik Johnston2020-07-161-34/+2
|
* Add ability to shard the federation sender (#7798)Erik Johnston2020-07-101-43/+16
|
* Pass original request headers from workers to the main process. (#7797)Patrick Cloke2020-07-091-5/+25
|
* Merge branch 'master' into developPatrick Cloke2020-07-021-1/+2
|\
| * Ensure that HTML pages served from Synapse include headers to avoid embedding.Patrick Cloke2020-07-021-1/+2
| |
* | Create a ListenerConfig object (#7681)Richard van der Hoff2020-06-163-41/+53
| | | | | | | | | | | | | | | | | | | | This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.
* | Discard RDATA from already seen positions. (#7648)Patrick Cloke2020-06-151-0/+5
| |
* | Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-4/+2
|/
* Attempt to fix PhoneHomeStatsTestCase.test_performance_100 being flaky. (#7634)Patrick Cloke2020-06-051-19/+23
|
* async/await get_user_id_by_threepid (#7620)Andrew Morgan2020-06-031-5/+4
| | | | | Based on #7619 async's `get_user_id_by_threepid` and its call stack.
* Ensure we persist and ack the same tokenErik Johnston2020-05-271-5/+7
|
* Speed up processing of federation stream RDATA rows.Erik Johnston2020-05-271-2/+17
| | | | | | Instead of storing and sending an ACK for every single row we send synchronously, we instead do it asynchronously while batching up updates.
* Fix incorrect exception handling in KeyUploadServlet.on_POST (#7563)Richard van der Hoff2020-05-261-1/+1
| | | Introduced in #7556
* Fix recording of federation stream token (#7564)Richard van der Hoff2020-05-261-24/+44
| | | | | | | | A couple of changes of significance: * remove the `_last_ack < federation_position` condition, so that updates will still be correctly processed after restart * Correctly wire up send_federation_ack to the right class.
* Add option to move event persistence off master (#7517)Erik Johnston2020-05-221-5/+48
|
* Return 200 OK for all OPTIONS requests (#7534)Patrick Cloke2020-05-222-8/+11
|
* Fix exception reporting due to HTTP request errors. (#7556)Erik Johnston2020-05-221-4/+9
| | | | These are business as usual errors, rather than stuff we want to log at error.
* Stub out GET presence requests in the frontend proxy (#7545)Richard van der Hoff2020-05-211-17/+4
| | | | | We don't really make any promises about returning accurate presence data when presence is disabled, so we may as well just return a static response, rather than making the master handle a request.
* Allow ReplicationRestResource to be added to workers (#7515)Erik Johnston2020-05-181-0/+4
| | | This allows workers to talk to each other over HTTP replication.
* Add a worker store for search insertion. (#7516)Erik Johnston2020-05-151-0/+2
| | | | | This is required as both event persistence and the background update needs access to this function. It should be perfectly safe for two workers to write to that table at the same time.
* Move EventStream handling into default ReplicationDataHandler (#7493)Erik Johnston2020-05-141-31/+2
| | | This is so that the logic can happen on both master and workers when we move event persistence out.
* Allow censoring of events to happen on workers. (#7492)Erik Johnston2020-05-131-0/+2
| | | This is safe as we can now write to cache invalidation stream on workers, and is required for when we move event persistence off master.
* Fix new flake8 errors (#7470)Erik Johnston2020-05-121-2/+3
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-3/+2
| | | | variables (#6391)
* Implement OpenID Connect-based login (#7256)Quentin Gliech2020-05-081-0/+12
|
* Thread through instance name to replication client. (#7369)Erik Johnston2020-05-011-6/+4
| | | For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.
* Use `stream.current_token()` and remove `stream_positions()` (#7172)Erik Johnston2020-05-011-16/+0
| | | | We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.
* Persist user interactive authentication sessions (#7302)Patrick Cloke2020-04-301-0/+2
| | | | | By persisting the user interactive authentication sessions to the database, this fixes situations where a user hits different works throughout their auth session and also allows sessions to persist through restarts of Synapse.
* Add instance name to RDATA/POSITION commands (#7364)Erik Johnston2020-04-291-2/+2
| | | | | This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
* Run replication streamers on workers (#7146)Erik Johnston2020-04-281-4/+9
| | | Currently we never write to streams from workers, but that will change soon
* Stop the master relaying USER_SYNC for other workers (#7318)Richard van der Hoff2020-04-221-36/+49
| | | | | | | Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication. In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits. Fixes (I hope) #7257.
* Merge branch 'release-v1.12.4' into developRichard van der Hoff2020-04-221-0/+6
|\
| * Support GET account_data requests on a worker (#7311)Richard van der Hoff2020-04-211-0/+6
| |
| * Fix starting workers when federation sending not split out.Erik Johnston2020-03-311-0/+12
| |
* | Add ability to run replication protocol over redis. (#7040)Erik Johnston2020-04-221-0/+6
| | | | | | This is configured via the `redis` config options.
* | Move client command handling out of TCP protocol (#7185)Erik Johnston2020-04-062-18/+3
| | | | | | The aim here is to move the command handling out of the TCP protocol classes and to also merge the client and server command handling (so that we can reuse them for redis protocol). This PR simply moves the client paths to the new `ReplicationCommandHandler`, a future PR will move the server paths too.
* | Extend web_client_location to handle absolute URLs (#7006)Martin Milata2020-04-031-3/+13
| | | | | | | | | | Log warning when filesystem path is used. Signed-off-by: Martin Milata <martin@martinmilata.cz>
* | Remove some `run_in_background` calls in replication code (#7203)Richard van der Hoff2020-04-031-8/+8
| | | | | | | | | | | | By running this stuff with `run_in_background`, it won't be correctly reported against the relevant CPU usage stats. Fixes #7202
* | Remove usage of "conn_id" for presence. (#7128)Erik Johnston2020-03-301-4/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `conn_id` usage for UserSyncCommand. Each tcp replication connection is assigned a "conn_id", which is used to give an ID to a remotely connected worker. In a redis world, there will no longer be a one to one mapping between connection and instance, so instead we need to replace such usages with an ID generated by the remote instances and included in the replicaiton commands. This really only effects UserSyncCommand. * Add CLEAR_USER_SYNCS command that is sent on shutdown. This should help with the case where a synchrotron gets restarted gracefully, rather than rely on 5 minute timeout.
* | Move catchup of replication streams to worker. (#7024)Erik Johnston2020-03-251-0/+3
| | | | | | This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
* | Merge pull request #7133 from matrix-org/erikj/fix_worker_startupErik Johnston2020-03-251-0/+12
|\ \ | | | | | | Fix starting workers when federation sending not split out.
| * | Fix starting workers when federation sending not split out.Erik Johnston2020-03-241-0/+12
| |/
* | Convert `*StreamRow` classes to inner classes (#7116)Richard van der Hoff2020-03-231-1/+1
| | | | | | | | | | This just helps keep the rows closer to their streams, so that it's easier to see what the format of each stream is.
* | Fix processing of `groups` stream, and use symbolic names for streams (#7117)Richard van der Hoff2020-03-231-12/+23
| | | | | | | | | | | | `groups` != `receipts` Introduced in #6964
* | Change device list streams to have one row per ID (#7010)Erik Johnston2020-03-191-3/+7
|\ \ | |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add 'device_lists_outbound_pokes' as extra table. This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change. * Change device lists stream to have one row per id. This will make it possible to process the streams more incrementally, avoiding having to process large chunks at once. * Change device list replication to match new semantics. Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host). * Newsfile * Remove handling of multiple rows per ID * Fix worker handling * Comments from review
| * Comments from reviewErik Johnston2020-03-181-0/+3
| |
| * Fix worker handlingErik Johnston2020-03-021-2/+3
| |
| * Change device list replication to match new semantics.Erik Johnston2020-02-281-1/+1
| | | | | | | | | | Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host).
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-06 16:25:30 +0000