summary refs log tree commit diff
path: root/synapse/app/homeserver.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-181-0/+1
| | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-101-1/+0
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Federation outbound proxy (#15773)Eric Eastwood2023-07-051-0/+1
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Experimental Unix socket support (#15353)Jason Little2023-04-031-16/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add IReactorUNIX to ISynapseReactor type hint. * Create listen_unix(). Two options, 'path' to the file and 'mode' of permissions(not umask, recommend 666 as default as nginx/other reverse proxies write to it and it's setup as user www-data) For the moment, leave the option to always create a PID lockfile turned on by default * Create UnixListenerConfig and wire it up. Rename ListenerConfig to TCPListenerConfig, then Union them together into ListenerConfig. This spidered around a bit, but I think I got it all. Metrics and manhole have been placed behind a conditional in case of accidental putting them onto a unix socket. Use new helpers to get if a listener is configured for TLS, and to help create a site tag for logging. There are 2 TODO things in parse_listener_def() to finish up at a later point. * Refactor SynapseRequest to handle logging correctly when using a unix socket. This prevents an exception when an IP address can not be retrieved for a request. * Make the 'Synapse now listening on Unix socket' log line a little prettier. * No silent failures on generic workers when trying to use a unix socket with metrics or manhole. * Inline variables in app/_base.py * Update docstring for listen_unix() to remove reference to a hardcoded permission of 0o666 and add a few comments saying where the default IS declared. * Disallow both a unix socket and a ip/port combo on the same listener resource * Linting * Changelog * review: simplify how listen_unix returns(and get rid of a type: ignore) * review: fix typo from ConfigError in app/homeserver.py * review: roll conditional for http_options.tag into get_site_tag() helper(and add docstring) * review: enhance the conditionals for checking if a port or path is valid, remove a TODO line * review: Try updating comment in get_client_ip_if_available to clarify what is being retrieved and why * Pretty up how 'Synapse now listening on Unix Socket' looks by decoding the byte string. * review: In parse_listener_def(), raise ConfigError if neither socket_path nor port is declared(and fix a typo)
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-221-1/+0
|
* Add listener `health` (#14747)Dirk Klimpel2023-01-111-0/+3
| | | Fixes: #8780
* Remove legacy Prometheus metrics names. They were deprecated in Synapse ↵reivilibre2022-11-241-1/+0
| | | | v1.69.0 and disabled by default in Synapse v1.71.0. (#14538)
* Support using SSL on worker endpoints. (#14128)Tuomas Ojamies2022-11-151-30/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix missing SSL support in worker endpoints. * Add changelog * SSL for Replication endpoint * Remove unit test change * Refactor listener creation to reduce duplicated code * Fix the logger message * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update synapse/app/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Add config documentation for new TLS option Co-authored-by: Tuomas Ojamies <tojamies@palantir.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Use servlets for /key/ endpoints. (#14229)Patrick Cloke2022-10-201-17/+9
| | | | | To fix the response for unknown endpoints under that prefix. See MSC3743.
* Remove configuration options for direct TCP replication. (#13647)Patrick Cloke2022-09-061-11/+0
| | | Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
* Add experimental configuration option to allow disabling legacy Prometheus ↵reivilibre2022-08-241-1/+5
| | | | | metric names. (#13540) Co-authored-by: David Robertson <davidr@element.io>
* Drop support for delegating email validation, round 2 (#13596)David Robertson2022-08-231-2/+1
|
* Add support for compression to federation responses (#13537)Ayush Anand2022-08-181-1/+4
| | | | | | Closes #13415. Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
* Revert "Drop support for delegating email validation (#13192)" (#13406)3nprob2022-07-291-1/+2
| | | | | Reverts commit fa71bb18b527d1a3e2629b48640ea67fff2f8c59, and tweaks documentation. Signed-off-by: 3nprob <git@3n.anonaddy.com>
* Drop support for delegating email validation (#13192)Richard van der Hoff2022-07-121-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Drop support for delegating email validation Delegating email validation to an IS is insecure (since it allows the owner of the IS to do a password reset on your HS), and has long been deprecated. It will now cause a config error at startup. * Update unit test which checks for email verification Give it an `email` config instead of a threepid delegate * Remove unused method `requestEmailToken` * Simplify config handling for email verification Rather than an enum and a boolean, all we need here is a single bool, which says whether we are or are not doing email verification. * update docs * changelog * upgrade.md: fix typo * update version number this will be in 1.64, not 1.63 * update version number this one too
* Fix Synapse git info missing in version strings (#12973)David Robertson2022-06-071-4/+2
|
* Reload cache factors from disk on SIGHUP (#12673)David Robertson2022-05-111-34/+2
|
* Add restrictions by default to open registration in Synapse (#12091)Shay2022-03-251-0/+17
|
* Make it possible to enable compression for the metrics HTTP resource (#12258)Nick Mills-Barrett2022-03-211-1/+4
| | | | | | | | | | * Make it possible to enable compression for the metrics HTTP resource This can provide significant bandwidth savings pulling metrics from synapse instances. * Add changelog file. * Fix type hint
* Rename get_tcp_replication to get_replication_command_handler. (#12192)Patrick Cloke2022-03-101-1/+1
| | | | | | Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.
* Use importlib.metadata to read requirements (#12088)David Robertson2022-03-011-1/+1
| | | | | | | | | | | | | | | | | | * Pull runtime dep checks into their own module * Reimplement `check_requirements` using `importlib` I've tried to make this clearer. We start by working out which of Synapse's requirements we need to be installed here and now. I was surprised that there wasn't an easier way to see which packages were installed by a given extra. I've pulled out the error messages into functions that deal with "is this for an extra or not". And I've rearranged the loop over two different sets of requirements into one loop with a "must be instaled" flag. I hope you agree that this is clearer. * Test cases
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-1/+1
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Use version string helper from matrix-common (#11979)David Robertson2022-02-141-2/+3
| | | | * Require latest matrix-common * Use the common function
* Return JSON errors for unknown resources under /matrix/client. (#11930)Patrick Cloke2022-02-081-7/+2
| | | | | Re-applies the changes from 3e0cfd447e17658a937fe62555db9e968f00b15b (#11602), reverting d93ec0a0ba5f6d2fbf2bc321086d4ad4c03136e0 (#11764) now that the conflict with the webclient listener was fixed in 119edf51eb3e4f5ed5139dc370f5d7aed46edc1c (#11895).
* Remove support for the webclient listener. (#11895)Patrick Cloke2022-02-031-31/+3
| | | Also remove support for non-HTTP(S) web_client_location.
* Fix redirecting to the webclient for non-HTTP(S) web_client_location. (#11783)Patrick Cloke2022-01-201-1/+8
| | | | | To not change the behaviour during the deprecation period. Follow-up to #11774.
* Add deprecation warnings for `webclient` listener and non-HTTP(S) ↵Patrick Cloke2022-01-201-5/+7
| | | | | | | `web_client_location`. (#11774) This changes the behaviour of the root endpoint to redirect directly to the configuration of `web_client_location` if it is given an HTTP(S) URL.
* Partially revert #11602 to prevent webclient overriding client resource (#11764)Andrew Morgan2022-01-181-2/+7
|
* Return JSON errors for unknown resources under /matrix/client. (#11602)Patrick Cloke2021-12-201-7/+2
| | | | Instead of returning 404 errors with HTML bodies when an unknown prefix was requested (e.g. /matrix/client/v1 before Synapse v1.49.0).
* Don't start Synapse master process if `worker_app` is set (#11416)Shay2021-11-301-0/+7
| | | | | | | | | | | | | | | | | | | | | * Add check to catch syanpse master process starting when workers are configured * add test to verify that starting master process with worker config raises error * newsfragment * specify config.worker.worker_app in check * update test * report specific config option that triggered the error Co-authored-by: reivilibre <oliverw@matrix.org> * clarify error message Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: reivilibre <oliverw@matrix.org>
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-0/+1
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Add support for `/_matrix/media/v3` APIs (#11371)Aaron R2021-11-171-2/+7
| | | | | | | | | | | | | * Add support for `/_matrix/media/v3` APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Update `workers.md` to use v3 client and media APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Add changelog Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add support for `/_matrix/client/v3` APIs (#11318)Aaron R2021-11-161-0/+1
| | | | | This is one of the changes required to support Matrix 1.1 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-101-54/+38
|
* Support for serving server well-known files (#11211)Richard van der Hoff2021-11-011-2/+2
| | | | Fixes https://github.com/matrix-org/synapse/issues/8308
* Add type hints for most `HomeServer` parameters (#11095)Sean Quah2021-10-221-1/+1
|
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-061-1/+1
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-7/+7
|
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-241-1/+1
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-5/+5
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-231-1/+1
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-1/+1
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-061-1/+4
|
* Remove shebang line from module files (#10415)V024602021-07-291-1/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-4/+2
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Ensure that errors during startup are written to the logs and the console. ↵Richard van der Hoff2021-06-211-7/+9
| | | | | | | | (#10191) * Defer stdio redirection until we are about to start the reactor * Catch and handle exceptions during startup
* Standardise the module interface (#10062)Brendan Abolivier2021-06-181-0/+4
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Remove support for ACME v1 (#10194)Brendan Abolivier2021-06-171-48/+0
| | | | | Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
* Optionally track memory usage of each LruCache (#9881)Erik Johnston2021-05-051-0/+1
| | | | | This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
* Limit how often GC happens by time. (#9902)Erik Johnston2021-05-051-0/+3
| | | | | | | | | | | | | | | Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.
* Improved validation for received requests (#9817)Richard van der Hoff2021-04-231-5/+12
| | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* pass a reactor into SynapseSite (#9874)Richard van der Hoff2021-04-231-15/+10
|
* remove `HomeServer.get_config` (#9815)Richard van der Hoff2021-04-141-9/+9
| | | | Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-141-1/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Suppress CryptographyDeprecationWarning (#9698)Richard van der Hoff2021-03-261-7/+2
| | | This warning is somewhat confusing to users, so let's suppress it
* Fix SSO on workers (#9271)Richard van der Hoff2021-02-011-16/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #8966. * Factor out build_synapse_client_resource_tree Start a function which will mount resources common to all workers. * Move sso init into build_synapse_client_resource_tree ... so that we don't have to do it for each worker * Fix SSO-login-via-a-worker Expose the SSO login endpoints on workers, like the documentation says. * Update workers config for new endpoints Add documentation for endpoints recently added (#8942, #9017, #9262) * remove submit_token from workers endpoints list this *doesn't* work on workers (yet). * changelog * Add a comment about the odd path for SAML2Resource
* Split out a separate endpoint to complete SSO registration (#9262)Richard van der Hoff2021-02-011-0/+2
| | | There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
* Split OidcProvider out of OidcHandler (#9107)Richard van der Hoff2021-01-141-1/+0
| | | | | | | The idea here is that we will have an instance of OidcProvider for each configured IdP, with OidcHandler just doing the marshalling of them. For now it's still hardcoded with a single provider.
* Clean up exception handling in the startup code (#9059)Richard van der Hoff2021-01-111-37/+25
| | | | Factor out the exception handling in the startup code to a utility function, and fix the some logging and exit code stuff.
* Remove unused SynapseService (#9058)Richard van der Hoff2021-01-111-22/+0
|
* Add initial support for a "pick your IdP" page (#9017)Richard van der Hoff2021-01-051-0/+2
| | | | | During login, if there are multiple IdPs enabled, offer the user a choice of IdPs.
* Implement a username picker for synapse (#8942)Richard van der Hoff2020-12-181-0/+2
| | | | | | | | | | | | | | The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow `UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser back to the client, we redirect to a username-picker resource, which allows the user to enter a username. We *then* complete the SSO flow (including doing the client permission checks). The static resources for the username picker itself (in https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker) are essentially lifted wholesale from https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. As the comment says, we might want to think about making them customisable, but that can be a follow-up. Fixes #8876.
* Better formatting for config errors from modules (#8874)Richard van der Hoff2020-12-081-4/+42
| | | | | | | | | | The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError, and it will be magically turned into a legible error message. There's a few components to it: * Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it into an absolute path. * Generally improving the way ConfigErrors get printed. * Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
* Combine `SpamCheckerApi` with the more generic `ModuleApi`. (#8464)Richard van der Hoff2020-10-071-2/+1
| | | | | Lots of different module apis is not easy to maintain. Rather than adding yet another ModuleApi(hs, hs.get_auth_handler()) incantation, first add an hs.get_module_api() method and use it where possible.
* Allow background tasks to be run on a separate worker. (#8369)Patrick Cloke2020-10-021-182/+0
|
* Remove obsolete __future__ imports (#8337)Jonathan de Jong2020-09-171-2/+0
|
* Show a confirmation page during user password reset (#8004)Andrew Morgan2020-09-101-0/+10
| | | | | This PR adds a confirmation step to resetting your user password between clicking the link in your email and your password actually being reset. This is to better align our password reset flow with the industry standard of requiring a confirmation from the user after email validation.
* Convert the main methods run by the reactor to async. (#8213)Patrick Cloke2020-09-021-10/+8
|
* Add health check endpoint (#8048)Erik Johnston2020-08-071-1/+4
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-3/+3
|
* Convert ACME code to async/await. (#7989)Patrick Cloke2020-08-031-7/+6
|
* Convert synapse.app to async/await. (#7868)Patrick Cloke2020-07-171-13/+12
|
* Merge branch 'master' into developPatrick Cloke2020-07-021-1/+2
|\
| * Ensure that HTML pages served from Synapse include headers to avoid embedding.Patrick Cloke2020-07-021-1/+2
| |
* | Create a ListenerConfig object (#7681)Richard van der Hoff2020-06-161-23/+27
| | | | | | | | | | | | | | | | | | | | This ended up being a bit more invasive than I'd hoped for (not helped by generic_worker duplicating some of the code from homeserver), but hopefully it's an improvement. The idea is that, rather than storing unstructured `dict`s in the config for the listener configurations, we instead parse it into a structured `ListenerConfig` object.
* | Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-4/+2
|/
* Attempt to fix PhoneHomeStatsTestCase.test_performance_100 being flaky. (#7634)Patrick Cloke2020-06-051-19/+23
|
* async/await get_user_id_by_threepid (#7620)Andrew Morgan2020-06-031-5/+4
| | | | | Based on #7619 async's `get_user_id_by_threepid` and its call stack.
* Return 200 OK for all OPTIONS requests (#7534)Patrick Cloke2020-05-221-5/+9
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-111-3/+2
| | | | variables (#6391)
* Implement OpenID Connect-based login (#7256)Quentin Gliech2020-05-081-0/+12
|
* Add ability to run replication protocol over redis. (#7040)Erik Johnston2020-04-221-0/+6
| | | This is configured via the `redis` config options.
* Extend web_client_location to handle absolute URLs (#7006)Martin Milata2020-04-031-3/+13
| | | | | Log warning when filesystem path is used. Signed-off-by: Martin Milata <martin@martinmilata.cz>
* Move pusherpool startup into _base.setup (#7104)Richard van der Hoff2020-03-191-1/+0
| | | | | This should be safe to do on all workers/masters because it is guarded by a config option which will ensure it is only actually done on the worker assigned as a pusher.
* Break down monthly active users by appservice_id (#7030)Neil Johnson2020-03-061-0/+13
| | | | | * Break down monthly active users by appservice_id and emit via prometheus. Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Allow additional_resources to implement Resource directly (#6686)Richard van der Hoff2020-01-131-2/+11
| | | | | | AdditionalResource really doesn't add any value, and it gets in the way for resources which want to support child resources or the like. So, if the resource object already implements the IResource interface, don't bother wrapping it.
* Raise an error if someone tries to use the log_file config option (#6626)Richard van der Hoff2020-01-031-1/+1
| | | | This has caused some confusion for people who didn't notice it going away.
* Remove unused, undocumented "content repo" resource (#6628)Richard van der Hoff2020-01-031-9/+1
| | | | | | This looks like it got half-killed back in #888. Fixes #6567.
* Improve diagnostics on database upgrade failure (#6570)Richard van der Hoff2019-12-191-7/+2
| | | | `Failed to upgrade database` is not helpful, and it's unlikely that UPGRADE.rst has anything useful.
* Merge pull request #6511 from matrix-org/erikj/remove_db_config_from_appsErik Johnston2019-12-121-6/+1
|\ | | | | Move database config from apps into HomeServer object
| * Remove database config parsing from apps.Erik Johnston2019-12-101-6/+1
| |
* | Phone home stats DB reporting should not assume a single DB.Erik Johnston2019-12-101-2/+4
|/
* Move start up DB checks to main data store.Erik Johnston2019-12-061-18/+5
|
* Move are_all_users_on_domain checks to main data store.Erik Johnston2019-12-061-11/+1
|
* Move background update handling out of storeErik Johnston2019-12-051-1/+1
|
* Don't call SQLBaseStore methods from outside storesErik Johnston2019-12-041-2/+2
|
* Fix phone home stats (#6418)Erik Johnston2019-11-261-1/+1
| | | | Fix phone home stats
* Fix phone home statsErik Johnston2019-11-051-2/+2
|
* Remove the psutil dependency (#6318)Amber Brown2019-11-051-82/+92
| | | * remove psutil and replace with resource
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-011-1/+1
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-3/+3
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Fix MAU reaping where reserved users are specified. (#6168)Neil Johnson2019-10-111-3/+3
|
* add report_stats_endpoint config option (#6012)Sorunome2019-09-121-2/+4
| | | This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
* Implement a structured logging output system. (#5680)Amber Brown2019-08-281-2/+2
|
* Fix issue with Synapse not starting up. Fixes #5866.Chris Moos2019-08-161-1/+1
| | | | Signed-off-by: Chris Moos <chris@chrismoos.com>
* Replace returnValue with return (#5736)Amber Brown2019-07-231-1/+1
|
* Call startup commands as system triggers.Erik Johnston2019-07-221-1/+1
| | | | | This helps ensures that we only consider ourselves "up" once all the startup functions have completed.
* Support Prometheus_client 0.4.0+ (#5636)Amber Brown2019-07-181-2/+1
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Drop support for cpu_affinity (#5525)Richard van der Hoff2019-06-221-1/+0
| | | This has no useful purpose on python3, and is generally a source of confusion.
* Run Black. (#5482)Amber Brown2019-06-201-84/+79
|
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2019-06-111-1/+0
|\
| * Change password reset links to /_matrix.Erik Johnston2019-06-111-1/+0
| |
* | add monthly active users to phonehome stats (#5252)Neil Johnson2019-06-101-0/+1
|/ | | | * add monthly active users to phonehome stats
* Add ability to perform password reset via email without trusting the ↵Andrew Morgan2019-06-061-0/+1
| | | | | | | | | | | | identity server (#5377) Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368
* Move admin api impl to its own packageRichard van der Hoff2019-05-011-1/+1
| | | | It doesn't really belong under rest/client/v1 any more.
* Move admin API to a new prefixRichard van der Hoff2019-05-011-0/+2
|
* add context to phonehome stats (#5020)Neil Johnson2019-04-081-1/+1
| | | | add context to phonehome stats
* Merge pull request #4853 from matrix-org/erikj/worker_docker_ciErik Johnston2019-03-151-9/+7
|\ | | | | Allow passing --daemonize to workers
| * Correctly handle all command line optionsErik Johnston2019-03-141-9/+7
| |
* | master startupAmber Brown2019-03-111-0/+1
|/
* Add database version to phonehome stats. (#4753)Richard van der Hoff2019-02-271-0/+3
|
* Fix errors in acme provisioning (#4648)Richard van der Hoff2019-02-141-6/+13
| | | | | | | | | * Better logging for errors on startup * Fix "TypeError: '>' not supported" when starting without an existing certificate * Fix a bug where an existing certificate would be reprovisoned every day
* Use `listen_tcp` for the replication listenerRichard van der Hoff2019-02-131-7/+7
| | | | | Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with what we do elsewhere.
* Improve logging around listening servicesRichard van der Hoff2019-02-131-2/+6
| | | | | | | | | I wanted to bring listen_tcp into line with listen_ssl in terms of returning a list of ports, and wanted to check that was a safe thing to do - hence the logging in `refresh_certificate`. Also, pull the 'Synapse now listening' message up to homeserver.py, because it was being duplicated everywhere else.
* Infer no_tls from presence of TLS listenersRichard van der Hoff2019-02-111-5/+0
| | | | | Rather than have to specify `no_tls` explicitly, infer whether we need to load the TLS keys etc from whether we have any TLS-enabled listeners.
* ACME Reprovisioning (#4522)Amber Brown2019-02-111-24/+55
|
* Merge pull request #4420 from matrix-org/jaywink/openid-listenerErik Johnston2019-02-111-1/+12
|\ | | | | New listener resource for the federation API "openid/userinfo" endpoint
| * Split federation OpenID userinfo endpoint out of the federation resourceJason Robinson2019-01-231-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the OpenID userinfo endpoint to be active even if the federation resource is not active. The OpenID userinfo endpoint is called by integration managers to verify user actions using the client API OpenID access token. Without this verification, the integration manager cannot know that the access token is valid. The OpenID userinfo endpoint will be loaded in the case that either "federation" or "openid" resource is defined. The new "openid" resource is defaulted to active in default configuration. Signed-off-by: Jason Robinson <jasonr@matrix.org>
| * Make SynapseHomeServer _http_listener use self.get_reactor()Jason Robinson2019-01-231-1/+3
| | | | | | | | | | | | | | | | | | | | | | For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Deduplicate some code in synapse.app (#4567)Amber Brown2019-02-081-50/+4
| |
* | Raise ConfigError insteadErik Johnston2019-01-301-1/+3
| |
* | _listener_http should return a listErik Johnston2019-01-301-1/+1
| |
* | SIGHUP for TLS cert reloading (#4495)Amber Brown2019-01-301-5/+46
| |
* | Support ACME for certificate provisioning (#4384)Amber Brown2019-01-231-9/+47
|/
* Make the dependencies more like a standard Python project and hook up the ↵Amber Brown2018-12-221-3/+0
| | | | optional dependencies to setuptools (#4298)
* ensure can report mau stats when hs.config.mau_stats_only is set (#4305)Neil Johnson2018-12-181-1/+1
| | | * ensure can report mau stats when hs.config.mau_stats_only is set
* Merge branch 'release-v0.34.0' into developRichard van der Hoff2018-12-111-32/+14
|\
| * Merge pull request #4290 from matrix-org/rav/remove_webclientRichard van der Hoff2018-12-111-32/+11
| |\ | | | | | | Stop installing Matrix Console by default
| | * Stop installing Matrix Console by defaultRichard van der Hoff2018-12-111-32/+11
| | | | | | | | | | | | This is based on the work done by @krombel in #2601.
| * | Add a welcome page to the static resourcesRichard van der Hoff2018-12-111-0/+3
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is largely a precursor for the removal of the bundled webclient. The idea is to present a page at / which reassures people that something is working, and to give them some links for next steps. The welcome page lives at `/_matrix/static/`, so is enabled alongside the other `static` resources (which, in practice, means the client API is enabled). We'll redirect to it from `/` if we have nothing better to display there. It would be nice to have a way to disable it (in the same way that you might disable the nginx welcome page), but I can't really think of a good way to do that without a load of ickiness. It's based on the work done by @krombel for #2601.
* | Implement SAML2 authentication (#4267)Richard van der Hoff2018-12-071-0/+4
| | | | | | | | | | | | | | | | | | | | | | This implements both a SAML2 metadata endpoint (at `/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at `/_matrix/saml2/authn_response`). If the SAML2 response matches what's been configured, we complete the SSO login flow by redirecting to the client url (aka `RelayState` in SAML2 jargon) with a login token. What we don't yet have is anything to build a SAML2 request and redirect the user to the identity provider. That is left as an exercise for the reader.
* | Implement .well-known handling (#4262)Richard van der Hoff2018-12-051-0/+2
|/ | | | Sometimes it's useful for synapse to generate its own .well-known file.
* Add option to track MAU stats (but not limit people) (#3830)Travis Ralston2018-11-151-1/+1
|
* Remove deprecated v1 key exchange endpoint (#4119)Amber Brown2018-10-311-6/+1
|
* fix race condiftion in calling initialise_reserved_usersNeil Johnson2018-10-231-8/+0
|
* Merge pull request #4076 from matrix-org/rav/fix_init_logcontextsRichard van der Hoff2018-10-221-8/+22
|\ | | | | Run MAU queries as background processes
| * Run MAU queries as background processesRichard van der Hoff2018-10-201-8/+22
| | | | | | | | Fixes #3820
* | Merge pull request #4073 from matrix-org/rav/require_psutilRichard van der Hoff2018-10-221-7/+4
|\ \ | | | | | | Make psutil an explicit dependency
| * | Make psutil an explicit dependencyRichard van der Hoff2018-10-191-7/+4
| |/ | | | | | | | | | | | | | | As of #4027, we require psutil to be installed, so it should be in our dependency list. We can also remove some of the conditional import code introduced by #992. Fixes #4062.
* / Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase ↵Amber Brown2018-10-201-1/+1
|/ | | | on py3) (#4068)
* Remove redundant call to start_get_pdu_cacheRichard van der Hoff2018-09-281-1/+0
| | | | | I think this got forgotten in #3932. We were getting away with it because it was the last call in this function.
* Replaced all occurences of e.message with str(e)Schnuffle2018-09-271-2/+2
| | | | Signed-off-by: Schnuffle <schnuffle@github.com>
* Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* typoWill Hunt2018-09-171-1/+1
|
* make pip happy?Will Hunt2018-09-171-2/+4
|
* Use a string for versionsWill Hunt2018-09-171-1/+1
|
* Add python_version phone home statWill Hunt2018-09-171-0/+2
|
* improve namingNeil Johnson2018-09-121-5/+9
|
* expose number of real reserved usersNeil Johnson2018-09-121-3/+7
|
* Make sure that we close db connections opened during initRichard van der Hoff2018-08-281-6/+8
| | | | | | | | | We should explicitly close any db connections we open, because failing to do so can block other transactions as per https://github.com/matrix-org/synapse/issues/3682. Let's also try to factor out some of the boilerplate by having server classes define their datastore class rather than duplicating the whole of `setup`.
* remove errant yieldNeil Johnson2018-08-171-1/+1
|
* call reap on start up and fix under reaping bugNeil Johnson2018-08-161-0/+1
|
* pep8Neil Johnson2018-08-141-0/+1
|
* combine mau metrics into one groupNeil Johnson2018-08-141-4/+3
|
* Merge pull request #3439 from vojeroen/send_sni_for_federation_requestsRichard van der Hoff2018-08-101-0/+2
|\ | | | | send SNI for federation requests
| * Merge branch 'develop' into send_sni_for_federation_requestsRichard van der Hoff2018-07-271-5/+14
| |\
| * \ Merge remote-tracking branch 'upstream/develop' into ↵Jeroen2018-07-141-13/+20
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | send_sni_for_federation_requests # Conflicts: # synapse/crypto/context_factory.py
| * | | send SNI for federation requestsJeroen2018-06-241-0/+2
| | | |
* | | | implement reserved users for mau limitsNeil Johnson2018-08-071-0/+6
| | | |
* | | | fix caching and testsNeil Johnson2018-08-031-1/+0
| | | |
* | | | update generate_monthly_active_users, and reap_monthly_active_usersNeil Johnson2018-08-021-2/+2
| | | |
* | | | Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_trackerNeil Johnson2018-08-011-0/+19
|\ \ \ \
| * | | | count_monthly_users is now asyncNeil Johnson2018-08-011-1/+2
| | | | |
| * | | | remove need to plot limit_usage_by_mauNeil Johnson2018-08-011-4/+0
| | | | |
| * | | | only need to loop if mau limiting is enabledNeil Johnson2018-08-011-1/+2
| | | | |
| * | | | coding styleNeil Johnson2018-07-311-1/+5
| | | | |
| * | | | remove errant loggingNeil Johnson2018-07-301-1/+0
| | | | |
| * | | | factor out metrics from __init__ to app/homeserverNeil Johnson2018-07-301-1/+19
| | |_|/ | |/| |
* / | | api into monthly_active_users tableNeil Johnson2018-07-311-0/+4
|/ / /
* | | Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-261-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
* | | Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-251-3/+10
| | | | | | | | | | | | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
* | | Make the rest of the .iterwhatever go away (#3562)Amber Brown2018-07-211-2/+4
| |/ |/|
* | run isortAmber Brown2018-07-091-13/+20
|/
* Write a clear restart indicator in logsRichard van der Hoff2018-06-201-6/+1
| | | | | I'm fed up with never being able to find the point a server restarted in the logs.
* The flag is named enable_metrics, not collect_metricsTravis Ralston2018-06-121-1/+1
| | | Signed-off-by: Travis Ralston <travpc@gmail.com>
* Merge pull request #3264 from matrix-org/neil/sign-up-statsNeil Johnson2018-06-011-0/+4
|\ | | | | daily user type phone home stats
| * daily user type phone home statsNeil Johnson2018-05-221-0/+4
| |
* | Run Prometheus on a different port, optionally. (#3274)Amber Brown2018-05-311-4/+9
| |
* | fixesAmber Brown2018-05-221-2/+4
| |
* | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-221-0/+17
|\|
| * Fix dependency on jinja2Richard van der Hoff2018-05-221-1/+1
| | | | | | | | | | | | | | Delay the import of ConsentResource, so that we can get away without jinja2 if people don't have the consent resource enabled. Fixes #3259
| * Merge pull request #3163 from matrix-org/cohort_analyticsNeil Johnson2018-05-161-0/+8
| |\ | | | | | | user visit data
| | * remove unused method recurring_user_daily_visit_statsNeil Johnson2018-05-161-3/+0
| | |
| | * pep8Neil Johnson2018-05-151-1/+0
| | |
| | * Limit query load of generate_user_daily_visitsNeil Johnson2018-05-151-14/+7
| | | | | | | | | | | | The aim is to keep track of when it was last called and only query from that point in time
| | * instead of inserting user daily visit data at the end of the day, instead ↵Neil Johnson2018-05-141-3/+16
| | | | | | | | | | | | insert incrementally through the day
| | * Merge branch 'develop' of https://github.com/matrix-org/synapse into ↵Neil Johnson2018-05-141-0/+2
| | |\ | | | | | | | | | | | | cohort_analytics
| | * | 10 mins seems more reasonable that every minuteNeil Johnson2018-05-011-1/+1
| | | |
| | * | Generate user daily statsNeil Johnson2018-04-251-0/+6
| | | |
| * | | Merge pull request #3213 from matrix-org/rav/consent_handlerRichard van der Hoff2018-05-161-0/+9
| |\ \ \ | | |_|/ | |/| | ConsentResource to gather policy consent from users
| | * | ConsentResource to gather policy consent from usersRichard van der Hoff2018-05-151-0/+9
| | |/ | | | | | | | | | | | | Hopefully there are enough comments and docs in this that it makes sense on its own.
* | / look at the Prometheus metrics insteadAmber Brown2018-05-211-5/+3
|/ /
* / Set Server header in SynapseRequestRichard van der Hoff2018-05-101-0/+2
|/ | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.
* Merge pull request #3041 from matrix-org/r30_statsNeil Johnson2018-04-051-0/+4
|\ | | | | R30 stats
| * Review commentsNeil Johnson2018-04-051-1/+1
| | | | | | | | | | Use iteritems over item to loop over dict formatting
| * Support multi client R30 for psqlNeil Johnson2018-03-281-1/+3
| |
| * rename stat to future proofNeil Johnson2018-03-281-1/+1
| |
| * query and call for r30 statsNeil Johnson2018-03-281-0/+2
| |
* | phone home cache size configurationsJan Christian Grünhage2018-04-041-0/+3
| |
* | Handle review commentsMichael Kaye2018-03-281-6/+9
| |
* | As daemonizing will make a new process, defer call to init.Michael Kaye2018-03-281-13/+23
| |
* | Include coarse CPU and Memory use in stats callbacks.Michael Kaye2018-03-271-0/+20
|/ | | | | This requires the psutil module, and is still opt-in based on the report_stats config option.
* 404 correctly on missing paths via NoResourceMatthew Hodgson2018-03-231-2/+2
| | | | fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029
* s/replication_client/federation_client/Erik Johnston2018-03-131-1/+1
|
* Split replication layer into twoErik Johnston2018-03-131-1/+1
|
* Add replication http endpoint for event sendingErik Johnston2018-02-071-0/+4
|
* Factor out get_db_conn to HomeServer base classRichard van der Hoff2018-01-261-13/+0
| | | | | This function is identical to all subclasses, so we may as well push it up to the base class to reduce duplication (and make use of it in the tests)
* Implement listen_tcp method in remaining workersSilke2017-12-181-3/+1
| | | | Signed-off-by: Silke <silke@slxh.eu>
* Remove logger argument and do not catch replication listenerSilke2017-12-181-18/+11
| | | | Signed-off-by: Silke <silke@slxh.eu>
* Add methods for listening on multiple addressesSilke Hofstra2017-12-171-50/+34
| | | | | | | Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP and reactor.listenSSL for multiple addresses. Signed-off-by: Silke Hofstra <silke@slxh.eu>
* Allow binds to both :: and 0.0.0.0Silke Hofstra2017-12-171-37/+58
| | | | | | | | | | Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed. This causes a warning explaining the behaviour. Configuration changed to match. See #2232 Signed-off-by: Silke Hofstra <silke@slxh.eu>
* Add config option to disable media_repo on main synapseRichard van der Hoff2017-11-221-8/+13
| | | | ... to stop us doing the cache cleanup jobs on the master.
* Build MediaRepositoryResource as a homeserver dependencyRichard van der Hoff2017-11-221-2/+1
| | | | | | This avoids the scenario where we have four different PreviewUrlResources configured on a single app, each of which have their own caches and cache clearing jobs.
* Merge pull request #2627 from matrix-org/rav/custom_rest_endpointsDavid Baker2017-11-021-0/+12
|\ | | | | Add a hook for custom rest endpoints
| * Add a hook for custom rest endpointsRichard van der Hoff2017-11-021-0/+12
| | | | | | | | | | Let the user specify custom modules which can be used for implementing extra endpoints.
* | Factor out _configure_named_resourceRichard van der Hoff2017-11-021-46/+64
|/ | | | This was a bit of a code vomit, so let's factor it out to preserve some sanity
* Move quit_with_errorErik Johnston2017-10-021-10/+1
|
* Allow configuration of CPU affinityRichard van der Hoff2017-08-151-0/+1
| | | | | Make it possible to set the CPU affinity in the config file, so that we don't need to remember to do it manually every time.
* Factor out common application startRichard van der Hoff2017-08-151-72/+39
| | | | | We have 10 copies of this code, and I don't really want to update each one separately.
* TypoErik Johnston2017-06-151-1/+3
|
* Add some more statsErik Johnston2017-06-151-0/+7
|
* Fix phone home statsErik Johnston2017-06-141-36/+13
|
* Queried CONDITIONAL_REQUIREMENTSMatthew Wolff2017-04-181-2/+5
|
* Fixed travis build failureMatthew Wolff2017-04-171-3/+1
| | | | Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
* web_server_root documentation fixMatthew Wolff2017-04-171-2/+1
| | | | Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
* Remove HTTP replication APIsErik Johnston2017-04-111-4/+0
|
* Add tcp replication listener type and hook it upErik Johnston2017-03-301-0/+11
|
* Run the reactor with the sentinel logcontextRichard van der Hoff2017-03-181-2/+7
| | | | | | This fixes a class of 'Unexpected logcontext' messages, which were happening because the logcontext was somewhat arbitrarily swapping between the sentinel and the `run` logcontext.
* Refactor logger config for workersRichard van der Hoff2017-03-101-1/+3
| | | | - to make it easier to add more config options.
* Restore default bind addressErik Johnston2017-01-101-10/+2
|
* Fix check for bind_addressJohannes Löthberg2016-12-201-2/+2
| | | | | | | The empty string is a valid setting for the bind_address option, so explicitly check for None here instead. Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
* Add support for specifying multiple bind addressesJohannes Löthberg2016-12-181-31/+45
| | | | Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
* Update commentMark Haines2016-08-221-1/+2
|
* Add usage stats to prometheus monitoringMark Haines2016-08-221-2/+25
|
* Add federation /version APIErik Johnston2016-08-051-1/+1
|
* Add metrics for psutil derived memory usageErik Johnston2016-07-201-0/+3
|
* Remove the legacy v0 content upload API.Mark Haines2016-06-211-2/+1
| | | | | | The existing content can still be downloaded. The last upload to the matrix.org server was in January 2015, so it is probably safe to remove the upload API.
* Add function to load config without generating itMark Haines2016-06-091-2/+1
| | | | | | | | | | | | | | | | Renames ``load_config`` to ``load_or_generate_config`` Adds a method called ``load_config`` that just loads the config. The main synapse.app.homeserver will continue to use ``load_or_generate_config`` to retain backwards compat. However new worker processes can use ``load_config`` to load the config avoiding some of the cruft needed to generate the config. As the new ``load_config`` method is expected to be used by new configs it removes support for the legacy commandline overrides that ``load_or_generate_config`` supports
* Allow setting of gc.set_thresholdsErik Johnston2016-06-071-0/+5
|