summary refs log tree commit diff
path: root/synapse/app (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Move start up DB checks to main data store.Erik Johnston2019-12-061-18/+5
|
* Move are_all_users_on_domain checks to main data store.Erik Johnston2019-12-061-11/+1
|
* Change DataStores to accept 'database' param.Erik Johnston2019-12-062-4/+6
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-12-061-1/+1
|\ | | | | | | erikj/make_database_class
| * Port rest/v1 to async/awaitErik Johnston2019-12-051-1/+1
| |
* | Move background update handling out of storeErik Johnston2019-12-051-1/+1
| |
* | Move DB pool and helper functions into dedicated Database classErik Johnston2019-12-052-2/+2
|/
* Remove underscore from SQLBaseStore functionsErik Johnston2019-12-041-1/+1
|
* Don't call SQLBaseStore methods from outside storesErik Johnston2019-12-041-2/+2
|
* Merge branch 'master' into developAndrew Morgan2019-11-261-1/+1
|\
| * Fix phone home stats (#6418)Erik Johnston2019-11-261-1/+1
| | | | | | | | Fix phone home stats
* | Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|/
* Fix phone home statsErik Johnston2019-11-051-2/+2
|
* Remove the psutil dependency (#6318)Amber Brown2019-11-051-82/+92
| | | * remove psutil and replace with resource
* Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-011-1/+1
| | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-3112-24/+26
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Move storage classes into a main "data store".Erik Johnston2019-10-214-4/+4
| | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* Fix MAU reaping where reserved users are specified. (#6168)Neil Johnson2019-10-111-3/+3
|
* Fix up some typechecking (#6150)Amber Brown2019-10-021-1/+3
| | | | | | * type checking fixes * changelog
* add report_stats_endpoint config option (#6012)Sorunome2019-09-121-2/+4
| | | This PR adds the optional `report_stats_endpoint` to configure where stats are reported to, if enabled.
* Allow Synapse to send registration emails + choose Synapse or an external ↵Andrew Morgan2019-09-061-1/+1
| | | | | | | | | | | | | | | | server to handle 3pid validation (#5987) This is a combination of a few different PRs, finally all being merged into `develop`: * #5875 * #5876 * #5868 (This one added the `/versions` flag but the flag itself was actually [backed out](https://github.com/matrix-org/synapse/commit/891afb57cbdf9867f2848341b29c75d6f35eef5a#diff-e591d42d30690ffb79f63bb726200891) in #5969. What's left is just giving /versions access to the config file, which could be useful in the future) * #5835 * #5969 * #5940 Clients should not actually use the new registration functionality until https://github.com/matrix-org/synapse/pull/5972 is merged. UPGRADE.rst, changelog entries and config file changes should all be reviewed closely before this PR is merged.
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-301-4/+4
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Implement a structured logging output system. (#5680)Amber Brown2019-08-2813-29/+31
|
* Drop dependency on sdnotify (#5871)Richard van der Hoff2019-08-171-10/+37
| | | | | | ... to save OSes which don't use it from having to maintain a port. Fixes #5865.
* Fix issue with Synapse not starting up. Fixes #5866.Chris Moos2019-08-161-1/+1
| | | | Signed-off-by: Chris Moos <chris@chrismoos.com>
* Don't load the media repo when configured to use an external media repo (#5754)Amber Brown2019-08-131-0/+9
|
* Replace returnValue with return (#5736)Amber Brown2019-07-232-5/+5
|
* Call startup commands as system triggers.Erik Johnston2019-07-2211-11/+29
| | | | | This helps ensures that we only consider ourselves "up" once all the startup functions have completed.
* Add process hooks to tell systemd our state.Erik Johnston2019-07-221-0/+29
| | | | Fixes #5676.
* Support Prometheus_client 0.4.0+ (#5636)Amber Brown2019-07-1812-24/+12
|
* Remove pointless descriptionErik Johnston2019-07-161-1/+0
|
* Fix typo in synapse/app/admin_cmd.pyErik Johnston2019-07-161-1/+1
| | | Co-Authored-By: Aaron Raimist <aaron@raim.ist>
* s/exfiltrate_user_data/export_user_data/Erik Johnston2019-07-151-1/+1
|
* Add FileExfiltrationWriterErik Johnston2019-07-151-1/+69
|
* Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_api_cmdErik Johnston2019-07-1512-40/+45
|\
| * Add basic opentracing support (#5544)Jorik Schellekens2019-07-111-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
| * Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-0412-12/+12
| |
| * Fix 'utime went backwards' errors on daemonization. (#5609)Richard van der Hoff2019-07-031-27/+30
| | | | | | | | | | | | | | | | * Fix 'utime went backwards' errors on daemonization. Fixes #5608 * remove spurious debug
* | Use set_defaults(func=) styleErik Johnston2019-07-151-11/+8
| |
* | Move creation of ArgumentParser to callerErik Johnston2019-07-151-1/+3
| |
* | Fix up commentsErik Johnston2019-07-151-2/+2
| |
* | Add basic admin cmd appErik Johnston2019-07-022-3/+212
|/
* Drop support for cpu_affinity (#5525)Richard van der Hoff2019-06-222-26/+1
| | | This has no useful purpose on python3, and is generally a source of confusion.
* Fix /messages on worker when no token suppliedErik Johnston2019-06-211-0/+2
|
* Support pagination API in client_reader workerErik Johnston2019-06-211-0/+2
|
* Run Black. (#5482)Amber Brown2019-06-2013-368/+358
|
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2019-06-111-1/+0
|\
| * Change password reset links to /_matrix.Erik Johnston2019-06-111-1/+0
| |
* | add monthly active users to phonehome stats (#5252)Neil Johnson2019-06-101-0/+1
|/ | | | * add monthly active users to phonehome stats
* Add ability to perform password reset via email without trusting the ↵Andrew Morgan2019-06-061-0/+1
| | | | | | | | | | | | identity server (#5377) Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368
* Unify v1 and v2 REST client APIs (#5226)Amber Brown2019-06-031-6/+5
|
* Fix dropped logcontexts during high outbound traffic. (#5277)Richard van der Hoff2019-05-291-7/+13
| | | Fixes #5271.
* Neilj/ensure get profileinfo available in client reader slaved store (#5213)Neil Johnson2019-05-221-0/+2
| | | * expose SlavedProfileStore to ClientReaderSlavedStore
* Revert "expose SlavedProfileStore to ClientReaderSlavedStore (#5200)"Richard van der Hoff2019-05-201-2/+0
| | | | | | | | | | | | | | | | | | This reverts commit ce5bcefc609db40740c692bd53a1ef84ab675e8c. This caused: ``` Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/synapse/src/synapse/app/client_reader.py", line 32, in <module> from synapse.replication.slave.storage import SlavedProfileStore ImportError: cannot import name 'SlavedProfileStore' from 'synapse.replication.slave.storage' (/home/synapse/src/synapse/replication/slave/storage/__init__.py) error starting synapse.app.client_reader('/home/synapse/config/workers/client_reader.yaml') (exit code: 1); see above for logs ```
* expose SlavedProfileStore to ClientReaderSlavedStore (#5200)Neil Johnson2019-05-171-0/+2
| | | * expose SlavedProfileStore to ClientReaderSlavedStore
* Merge pull request #5037 from matrix-org/erikj/limit_inflight_dnsErik Johnston2019-05-081-1/+82
|\ | | | | Limit in flight DNS requests
| * Limit in flight DNS requestsErik Johnston2019-04-091-1/+82
| | | | | | | | | | | | | | This is to work around a bug in twisted where a large number of concurrent DNS requests cause it to tight loop forever. c.f. https://twistedmatrix.com/trac/ticket/9620#ticket
* | Move admin api impl to its own packageRichard van der Hoff2019-05-011-1/+1
| | | | | | | | It doesn't really belong under rest/client/v1 any more.
* | Move admin API to a new prefixRichard van der Hoff2019-05-011-0/+2
| |
* | VersionRestServlet doesn't take a paramErik Johnston2019-04-151-1/+1
| |
* | Merge pull request #5063 from matrix-org/erikj/move_endpointsErik Johnston2019-04-151-4/+7
|\ \ | |/ |/| Move some rest endpoints to client reader
| * Move some rest endpoints to client readerErik Johnston2019-04-151-4/+7
| |
* | add context to phonehome stats (#5020)Neil Johnson2019-04-081-1/+1
|/ | | | add context to phonehome stats
* Combine the CurrentStateDeltaStream into the EventStreamRichard van der Hoff2019-03-271-7/+10
|
* Make EventStream rows have a typeRichard van der Hoff2019-03-271-1/+4
| | | | ... as a precursor to combining it with the CurrentStateDelta stream.
* Move replication.tcp.streams into a packageRichard van der Hoff2019-03-271-1/+1
|
* Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_workerRichard van der Hoff2019-03-151-0/+30
|\ | | | | Move client receipt processing to federation sender worker.
| * Move client receipt processing to federation sender worker.Richard van der Hoff2019-03-131-0/+30
| | | | | | | | | | This is mostly a prerequisite for #4730, but also fits with the general theme of "move everything off the master that we possibly can".
* | Merge pull request #4853 from matrix-org/erikj/worker_docker_ciErik Johnston2019-03-152-15/+19
|\ \ | | | | | | Allow passing --daemonize to workers
| * | Correctly handle all command line optionsErik Johnston2019-03-142-15/+19
| |/
* / master startupAmber Brown2019-03-111-0/+1
|/
* Allow /keys/{changes,query} API to run on workerErik Johnston2019-03-041-0/+11
|
* Merge pull request #4759 from matrix-org/erikj/3pid_client_readerErik Johnston2019-02-271-0/+2
|\ | | | | Move /account/3pid to client_reader
| * Move /account/3pid to client_readerErik Johnston2019-02-271-0/+2
| |
* | Move server key queries to federation readerErik Johnston2019-02-271-1/+5
|/
* Add database version to phonehome stats. (#4753)Richard van der Hoff2019-02-271-0/+3
|
* Correctly proxy exception in frontend_proxy workerErik Johnston2019-02-231-5/+10
|
* Merge pull request #4670 from matrix-org/erikj/register_login_splitErik Johnston2019-02-191-0/+2
|\ | | | | Split /login into client_reader
| * Split /login into client_readerErik Johnston2019-02-181-0/+2
| |
* | Merge pull request #4632 from matrix-org/erikj/basic_sentryErik Johnston2019-02-181-0/+30
|\ \ | | | | | | Add basic optional sentry.io integration
| * | Fixup comments and add warningErik Johnston2019-02-131-3/+3
| | |
| * | CommentsErik Johnston2019-02-121-0/+8
| | |
| * | Basic sentry integrationErik Johnston2019-02-121-0/+22
| | |
* | | Merge pull request #4666 from matrix-org/erikj/register_login_splitErik Johnston2019-02-181-0/+2
|\ \ \ | | |/ | |/| Split out registration to worker
| * | Split out registration to workerErik Johnston2019-02-181-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | This allows registration to be handled by a worker, though the actual write to the database still happens on master. Note: due to the in-memory session map all registration requests must be handled by the same worker.
* | | Fix kicking guest users in worker modeErik Johnston2019-02-181-0/+2
|/ / | | | | | | | | | | | | | | When guest_access changes from allowed to forbidden all local guest users should be kicked from the room. This did not happen when revocation was received from federation on a worker. Presumably broken in #4141
* | Fix errors in acme provisioning (#4648)Richard van der Hoff2019-02-141-6/+13
| | | | | | | | | | | | | | | | | | * Better logging for errors on startup * Fix "TypeError: '>' not supported" when starting without an existing certificate * Fix a bug where an existing certificate would be reprovisoned every day
* | Update synapse/app/_base.pyErik Johnston2019-02-131-1/+1
| | | | | | Co-Authored-By: richvdh <1389908+richvdh@users.noreply.github.com>
* | Use `listen_tcp` for the replication listenerRichard van der Hoff2019-02-131-7/+7
| | | | | | | | | | Fixes the "can't listen on 0.0.0.0" error. Also makes it more consistent with what we do elsewhere.
* | Improve logging around listening servicesRichard van der Hoff2019-02-132-11/+20
| | | | | | | | | | | | | | | | | | I wanted to bring listen_tcp into line with listen_ssl in terms of returning a list of ports, and wanted to check that was a safe thing to do - hence the logging in `refresh_certificate`. Also, pull the 'Synapse now listening' message up to homeserver.py, because it was being duplicated everywhere else.
* | Special-case the default bind_addresses for metrics listenerRichard van der Hoff2019-02-131-3/+2
|/ | | | | turns out it doesn't really support ipv6, so let's hack around that by only listening on ipv4 by default.
* Fix error when loading cert if tls is disabled (#4618)Richard van der Hoff2019-02-121-2/+3
| | | | | | If TLS is disabled, it should not be an error if no cert is given. Fixes #4554.
* Infer no_tls from presence of TLS listenersRichard van der Hoff2019-02-112-6/+1
| | | | | Rather than have to specify `no_tls` explicitly, infer whether we need to load the TLS keys etc from whether we have any TLS-enabled listeners.
* Don't create server contexts when TLS is disabledRichard van der Hoff2019-02-111-0/+5
| | | | we aren't going to use them anyway.
* Logging improvements around TLS certsRichard van der Hoff2019-02-111-4/+2
| | | | | Log which file we're reading keys and certs from, and refactor the code a bit in preparation for other work
* Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)Richard van der Hoff2019-02-111-3/+0
| | | | | It's nothing to do with refreshing the certificates. No idea why it was here.
* ACME Reprovisioning (#4522)Amber Brown2019-02-112-24/+74
|
* Merge pull request #4420 from matrix-org/jaywink/openid-listenerErik Johnston2019-02-112-2/+24
|\ | | | | New listener resource for the federation API "openid/userinfo" endpoint
| * Fix flake8 issuesJason Robinson2019-01-231-1/+4
| | | | | | | | Signed-off-by: Jason Robinson <jasonr@matrix.org>
| * Split federation OpenID userinfo endpoint out of the federation resourceJason Robinson2019-01-232-0/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the OpenID userinfo endpoint to be active even if the federation resource is not active. The OpenID userinfo endpoint is called by integration managers to verify user actions using the client API OpenID access token. Without this verification, the integration manager cannot know that the access token is valid. The OpenID userinfo endpoint will be loaded in the case that either "federation" or "openid" resource is defined. The new "openid" resource is defaulted to active in default configuration. Signed-off-by: Jason Robinson <jasonr@matrix.org>
| * Make SynapseHomeServer _http_listener use self.get_reactor()Jason Robinson2019-01-231-1/+3
| | | | | | | | | | | | | | | | | | | | | | For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>
| * Make FederationReaderServer _http_listen use self.get_reactor()Jason Robinson2019-01-231-1/+2
| | | | | | | | | | | | | | | | | | | | | | For all the homeserver classes, only the FrontendProxyServer passes its reactor when doing the http listen. Looking at previous PR's looks like this was introduced to make it possible to write a test, otherwise when you try to run a test with the test homeserver it tries to do a real bind to a port. Passing the reactor that the homeserver is instantiated with should probably be the right thing to do anyway? Signed-off-by: Jason Robinson <jasonr@matrix.org>
* | Deduplicate some code in synapse.app (#4567)Amber Brown2019-02-0812-147/+77
| |
* | ACME config cleanups (#4525)Richard van der Hoff2019-01-302-22/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Handle listening for ACME requests on IPv6 addresses the weird url-but-not-actually-a-url-string doesn't handle IPv6 addresses without extra quoting. Building a string which you are about to parse again seems like a weird choice. Let's just use listenTCP, which is consistent with what we do elsewhere. * Clean up the default ACME config make it look a bit more consistent with everything else, and tweak the defaults to listen on port 80. * newsfile
* | Raise ConfigError insteadErik Johnston2019-01-301-1/+3
| |
* | _listener_http should return a listErik Johnston2019-01-301-1/+1
| |
* | SIGHUP for TLS cert reloading (#4495)Amber Brown2019-01-302-12/+68
| |
* | Fix worker TLS (#4492)Amber Brown2019-01-287-48/+48
| | | | | | | | | | | | | | | | * load cert * changelog * fix
* | Support ACME for certificate provisioning (#4384)Amber Brown2019-01-231-9/+47
|/
* Check jinja version for consent resource (#4327)Richard van der Hoff2019-01-071-8/+1
| | | | | | | | * Raise a ConfigError if an invalid resource is specified * Require Jinja 2.9 for the consent resource * changelog
* Make the dependencies more like a standard Python project and hook up the ↵Amber Brown2018-12-222-6/+3
| | | | optional dependencies to setuptools (#4298)
* ensure can report mau stats when hs.config.mau_stats_only is set (#4305)Neil Johnson2018-12-181-1/+1
| | | * ensure can report mau stats when hs.config.mau_stats_only is set
* Merge branch 'release-v0.34.0' into developRichard van der Hoff2018-12-111-32/+14
|\
| * Merge pull request #4290 from matrix-org/rav/remove_webclientRichard van der Hoff2018-12-111-32/+11
| |\ | | | | | | Stop installing Matrix Console by default
| | * Stop installing Matrix Console by defaultRichard van der Hoff2018-12-111-32/+11
| | | | | | | | | | | | This is based on the work done by @krombel in #2601.
| * | Add a welcome page to the static resourcesRichard van der Hoff2018-12-111-0/+3
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is largely a precursor for the removal of the bundled webclient. The idea is to present a page at / which reassures people that something is working, and to give them some links for next steps. The welcome page lives at `/_matrix/static/`, so is enabled alongside the other `static` resources (which, in practice, means the client API is enabled). We'll redirect to it from `/` if we have nothing better to display there. It would be nice to have a way to disable it (in the same way that you might disable the nginx welcome page), but I can't really think of a good way to do that without a load of ickiness. It's based on the work done by @krombel for #2601.
* | Implement SAML2 authentication (#4267)Richard van der Hoff2018-12-071-0/+4
| | | | | | | | | | | | | | | | | | | | | | This implements both a SAML2 metadata endpoint (at `/_matrix/saml2/metadata.xml`), and a SAML2 response receiver (at `/_matrix/saml2/authn_response`). If the SAML2 response matches what's been configured, we complete the SSO login flow by redirecting to the client url (aka `RelayState` in SAML2 jargon) with a login token. What we don't yet have is anything to build a SAML2 request and redirect the user to the identity provider. That is left as an exercise for the reader.
* | Implement .well-known handling (#4262)Richard van der Hoff2018-12-051-0/+2
|/ | | | Sometimes it's useful for synapse to generate its own .well-known file.
* Add option to track MAU stats (but not limit people) (#3830)Travis Ralston2018-11-151-1/+1
|
* Fix typing being reset causing infinite syncs (#4127)Amber Brown2018-11-031-0/+14
|
* Remove deprecated v1 key exchange endpoint (#4119)Amber Brown2018-10-311-6/+1
|
* Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_initNeil Johnson2018-10-241-3/+3
|\
| * Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontextsRichard van der Hoff2018-10-241-3/+3
| |\ | | | | | | Clean up the way logcontexts and threads work in the pushers
| | * Remove redundant run_as_background_process() from pusherpoolRichard van der Hoff2018-10-221-2/+2
| | | | | | | | | | | | | | | | | | | | | `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.
| | * Rename _refresh_pusherRichard van der Hoff2018-10-221-1/+1
| | | | | | | | | | | | | | | This is public (or at least, called from outside the class), so ought to have a better name.
* | | fix race condiftion in calling initialise_reserved_usersNeil Johnson2018-10-231-8/+0
|/ /
* | Merge pull request #4076 from matrix-org/rav/fix_init_logcontextsRichard van der Hoff2018-10-221-8/+22
|\ \ | | | | | | Run MAU queries as background processes
| * | Run MAU queries as background processesRichard van der Hoff2018-10-201-8/+22
| |/ | | | | | | Fixes #3820
* | Merge pull request #4073 from matrix-org/rav/require_psutilRichard van der Hoff2018-10-221-7/+4
|\ \ | | | | | | Make psutil an explicit dependency
| * | Make psutil an explicit dependencyRichard van der Hoff2018-10-191-7/+4
| |/ | | | | | | | | | | | | | | As of #4027, we require psutil to be installed, so it should be in our dependency list. We can also remove some of the conditional import code introduced by #992. Fixes #4062.
* / Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase ↵Amber Brown2018-10-201-1/+1
|/ | | | on py3) (#4068)
* synapse/app: frontend_proxy.py: actually make workers work on py3Ivan Shapovalov2018-10-141-2/+2
|
* Make workers work on Py3 (#4027)Amber Brown2018-10-134-28/+32
|
* Remove redundant call to start_get_pdu_cacheRichard van der Hoff2018-09-281-1/+0
| | | | | I think this got forgotten in #3932. We were getting away with it because it was the last call in this function.
* Replaced all occurences of e.message with str(e)Schnuffle2018-09-2712-13/+13
| | | | Signed-off-by: Schnuffle <schnuffle@github.com>
* Merge pull request #3948 from matrix-org/rav/no_symlink_synctlAmber Brown2018-09-261-284/+0
|\ | | | | Move synctl into top dir to avoid a symlink
| * Move synctl into top dir to avoid a symlinkRichard van der Hoff2018-09-251-284/+0
| | | | | | | | | | | | symlinks apparently break setuptools on python3 and alpine (https://bugs.python.org/issue31940), so let's stop using a symlink and just use the file directly.
* | Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-2111-11/+0
|/ | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* typoWill Hunt2018-09-171-1/+1
|
* make pip happy?Will Hunt2018-09-171-2/+4
|
* Use a string for versionsWill Hunt2018-09-171-1/+1
|
* Add python_version phone home statWill Hunt2018-09-171-0/+2
|
* improve namingNeil Johnson2018-09-121-5/+9
|
* expose number of real reserved usersNeil Johnson2018-09-121-3/+7
|
* Make sure that we close db connections opened during initRichard van der Hoff2018-08-2811-46/+18
| | | | | | | | | We should explicitly close any db connections we open, because failing to do so can block other transactions as per https://github.com/matrix-org/synapse/issues/3682. Let's also try to factor out some of the boilerplate by having server classes define their datastore class rather than duplicating the whole of `setup`.
* Merge pull request #3659 from matrix-org/erikj/split_profilesErik Johnston2018-08-221-0/+12
|\ | | | | Allow profile updates to happen on workers
| * Add some fixmesErik Johnston2018-08-171-0/+2
| |
| * Call UserDirectoryHandler methods directlyErik Johnston2018-08-171-0/+2
| | | | | | | | | | | | | | | | Turns out that the user directory handling is fairly racey as a bunch of stuff assumes that the processing happens on master, which it doesn't when there is a synapse.app.user_dir worker. So lets just call the function directly until we actually get round to fixing it, since it doesn't make the situation any worse.
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-178-14/+43
| |\ | | | | | | | | | erikj/split_profiles
| * \ Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-091-1/+10
| |\ \ | | | | | | | | | | | | erikj/split_profiles
| * | | Allow profile changes to happen on workersErik Johnston2018-08-071-0/+8
| | | |
* | | | Merge pull request #3709 from matrix-org/rav/logcontext_for_replication_commandsRichard van der Hoff2018-08-175-5/+10
|\ \ \ \ | | | | | | | | | | Logcontexts for replication command handlers
| * | | | Logcontexts for replication command handlersRichard van der Hoff2018-08-175-5/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Run the handlers for replication commands as background processes. This should improve the visibility in our metrics, and reduce the number of "running db transaction from sentinel context" warnings. Ideally it means converting the things that fire off deferreds into the night into things that actually return a Deferred when they are done. I've made a bit of a stab at this, but it will probably be leaky.
* | | | | Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updatesRichard van der Hoff2018-08-171-2/+2
|\ \ \ \ \ | | | | | | | | | | | | Fix logcontexts for running pushers
| * | | | | Fix logcontexts for running pushersRichard van der Hoff2018-08-171-2/+2
| |/ / / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.
* | | | / Integrate presence from hotfixes (#3694)Amber Brown2018-08-183-8/+53
| |_|_|/ |/| | |
* | | | remove errant yieldNeil Johnson2018-08-171-1/+1
| | | |
* | | | call reap on start up and fix under reaping bugNeil Johnson2018-08-161-0/+1
|/ / /
* | | Fix inbound federation on reader workerErik Johnston2018-08-161-0/+2
| | | | | | | | | | | | | | | Inbound federation requires calculating push, which in turn relies on having access to account data.
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-158-4/+20
|\ \ \ | | | | | | | | | | | | erikj/split_federation
| * | | pep8Neil Johnson2018-08-141-0/+1
| | | |
| * | | combine mau metrics into one groupNeil Johnson2018-08-141-4/+3
| | | |
| * | | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-101-1/+1
| | | |
| * | | Merge pull request #3439 from vojeroen/send_sni_for_federation_requestsRichard van der Hoff2018-08-108-0/+16
| |\ \ \ | | |_|/ | |/| | send SNI for federation requests
| | * | Merge branch 'develop' into send_sni_for_federation_requestsRichard van der Hoff2018-07-274-11/+33
| | |\ \
| | * \ \ Merge remote-tracking branch 'upstream/develop' into ↵Jeroen2018-07-1414-50/+71
| | |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | send_sni_for_federation_requests # Conflicts: # synapse/crypto/context_factory.py
| | * | | | send SNI for federation requestsJeroen2018-06-248-0/+16
| | | | | |
* | | | | | Rename slave TransactionStore to SlaveTransactionStoreErik Johnston2018-08-155-10/+10
| | | | | |
* | | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-08-091-1/+10
|\| | | | | | | | | | | | | | | | | | | | | | | erikj/split_federation
| * | | | | implement reserved users for mau limitsNeil Johnson2018-08-071-0/+6
| | | | | |
| * | | | | fix caching and testsNeil Johnson2018-08-031-1/+0
| | | | | |
| * | | | | update generate_monthly_active_users, and reap_monthly_active_usersNeil Johnson2018-08-021-2/+2
| | | | | |
| * | | | | Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_trackerNeil Johnson2018-08-011-0/+19
| |\ \ \ \ \ | | | |_|_|/ | | |/| | |
| * | | | | api into monthly_active_users tableNeil Johnson2018-07-311-0/+4
| | |_|_|/ | |/| | |
* | | | | Pull in necessary stores in federation_readerErik Johnston2018-08-061-0/+2
| | | | |
* | | | | Add replication APIs for persisting federation eventsErik Johnston2018-08-061-0/+8
| |/ / / |/| | |
* | | | count_monthly_users is now asyncNeil Johnson2018-08-011-1/+2
| | | |
* | | | remove need to plot limit_usage_by_mauNeil Johnson2018-08-011-4/+0
| | | |
* | | | only need to loop if mau limiting is enabledNeil Johnson2018-08-011-1/+2
| | | |
* | | | coding styleNeil Johnson2018-07-311-1/+5
| | | |
* | | | remove errant loggingNeil Johnson2018-07-301-1/+0
| | | |
* | | | factor out metrics from __init__ to app/homeserverNeil Johnson2018-07-301-1/+19
|/ / /
* | | Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-261-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
* | | Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-251-3/+10
| | | | | | | | | | | | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
* | | Replace usage of get_current_toke with StreamToken.STARTErik Johnston2018-07-241-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | This allows us to handle /context/ requests on the client_reader worker without having to pull in all the various stream handlers (e.g. precence, typing, pushers etc). The only thing the token gets used for is pagination, and that ignores everything but the room portion of the token.
* | | Pull out did_forget to worker storeErik Johnston2018-07-241-4/+1
| | |
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-07-232-3/+7
|\ \ \ | | | | | | | | | | | | erikj/client_apis_move
| * | | Make the rest of the .iterwhatever go away (#3562)Amber Brown2018-07-212-3/+7
| | |/ | |/|
* / | Add some room read only APIs to client_readerErik Johnston2018-07-181-1/+13
|/ /
* / run isortAmber Brown2018-07-0914-50/+71
|/
* Write a clear restart indicator in logsRichard van der Hoff2018-06-201-6/+1
| | | | | I'm fed up with never being able to find the point a server restarted in the logs.
* Merge pull request #3389 from turt2live/travis/name_metricsAmber Brown2018-06-1311-11/+11
|\ | | | | Use the correct flag (enable_metrics) when warning about an incorrect metrics setup
| * The flag is named enable_metrics, not collect_metricsTravis Ralston2018-06-1211-11/+11
| | | | | | Signed-off-by: Travis Ralston <travpc@gmail.com>
* | Use the RegistryProxy for appservices tooTravis Ralston2018-06-121-1/+2
|/ | | Signed-off-by: Travis Ralston <travpc@gmail.com>
* Add hacky cache factor override systemErik Johnston2018-06-041-0/+4
|
* Merge pull request #3264 from matrix-org/neil/sign-up-statsNeil Johnson2018-06-011-0/+4
|\ | | | | daily user type phone home stats
| * daily user type phone home statsNeil Johnson2018-05-221-0/+4
| |
* | Run Prometheus on a different port, optionally. (#3274)Amber Brown2018-05-3112-14/+110
| |
* | fixesAmber Brown2018-05-221-2/+4
| |
* | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-221-0/+17
|\|
| * Fix dependency on jinja2Richard van der Hoff2018-05-221-1/+1
| | | | | | | | | | | | | | Delay the import of ConsentResource, so that we can get away without jinja2 if people don't have the consent resource enabled. Fixes #3259
| * Merge pull request #3163 from matrix-org/cohort_analyticsNeil Johnson2018-05-161-0/+8
| |\ | | | | | | user visit data
| | * remove unused method recurring_user_daily_visit_statsNeil Johnson2018-05-161-3/+0
| | |
| | * pep8Neil Johnson2018-05-151-1/+0
| | |
| | * Limit query load of generate_user_daily_visitsNeil Johnson2018-05-151-14/+7
| | | | | | | | | | | | The aim is to keep track of when it was last called and only query from that point in time
| | * instead of inserting user daily visit data at the end of the day, instead ↵Neil Johnson2018-05-141-3/+16
| | | | | | | | | | | | insert incrementally through the day
| | * Merge branch 'develop' of https://github.com/matrix-org/synapse into ↵Neil Johnson2018-05-1411-87/+119
| | |\ | | | | | | | | | | | | cohort_analytics
| | * | 10 mins seems more reasonable that every minuteNeil Johnson2018-05-011-1/+1
| | | |
| | * | Generate user daily statsNeil Johnson2018-04-251-0/+6
| | | |
| * | | Merge pull request #3213 from matrix-org/rav/consent_handlerRichard van der Hoff2018-05-161-0/+9
| |\ \ \ | | |_|/ | |/| | ConsentResource to gather policy consent from users
| | * | ConsentResource to gather policy consent from usersRichard van der Hoff2018-05-151-0/+9
| | | | | | | | | | | | | | | | | | | | Hopefully there are enough comments and docs in this that it makes sense on its own.
* | | | look at the Prometheus metrics insteadAmber Brown2018-05-211-5/+3
|/ / /
* / / Set Server header in SynapseRequestRichard van der Hoff2018-05-1011-0/+12
|/ / | | | | | | | | | | | | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.
* | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-275-80/+101
|\ \
| * | Improve exception handling for background processesRichard van der Hoff2018-04-275-80/+101
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* / Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-273-7/+6
|/ | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* Use six.itervalues in some placesAdrian Tschira2018-04-151-1/+3
| | | | | | There's more where that came from Signed-off-by: Adrian Tschira <nota@notafile.com>
* Merge pull request #3061 from NotAFile/add-some-byte-stringsRichard van der Hoff2018-04-091-1/+1
|\ | | | | Add b prefixes to some strings that are bytes in py3
| * Add b prefixes to some strings that are bytes in py3Adrian Tschira2018-04-041-1/+1
| | | | | | | | | | | | This has no effect on python2 Signed-off-by: Adrian Tschira <nota@notafile.com>
* | more verbosity in synctlRichard van der Hoff2018-04-061-0/+1
| |
* | Merge pull request #3041 from matrix-org/r30_statsNeil Johnson2018-04-051-0/+4
|\ \ | | | | | | R30 stats
| * | Review commentsNeil Johnson2018-04-051-1/+1
| | | | | | | | | | | | | | | Use iteritems over item to loop over dict formatting
| * | Support multi client R30 for psqlNeil Johnson2018-03-281-1/+3
| | |
| * | rename stat to future proofNeil Johnson2018-03-281-1/+1
| | |
| * | query and call for r30 statsNeil Johnson2018-03-281-0/+2
| |/
* | phone home cache size configurationsJan Christian Grünhage2018-04-041-0/+3
| |
* | Merge pull request #3000 from NotAFile/change-except-styleRichard van der Hoff2018-04-041-2/+2
|\ \ | | | | | | Replace old style error catching with 'as' keyword
| * | replace old style error catching with 'as' keywordNotAFile2018-03-151-2/+2
| | | | | | | | | | | | | | | | | | | | | This is both easier to read and compatible with python3 (not that that matters) Signed-off-by: Adrian Tschira <nota@notafile.com>
* | | Handle review commentsMichael Kaye2018-03-281-6/+9
| | |
* | | As daemonizing will make a new process, defer call to init.Michael Kaye2018-03-281-13/+23
| | |
* | | Include coarse CPU and Memory use in stats callbacks.Michael Kaye2018-03-271-0/+20
| |/ |/| | | | | | | This requires the psutil module, and is still opt-in based on the report_stats config option.
* | 404 correctly on missing paths via NoResourceMatthew Hodgson2018-03-2311-22/+22
|/ | | | fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029
* Register membership/state servlets in event_creatorErik Johnston2018-03-141-1/+13
|
* Merge pull request #2978 from matrix-org/erikj/refactor_replication_layerErik Johnston2018-03-131-1/+1
|\ | | | | Remove ReplicationLayer and user Client/Server directly
| * s/replication_client/federation_client/Erik Johnston2018-03-131-1/+1
| |
* | Don't build handlers on workers unnecessarilyErik Johnston2018-03-135-5/+0
|/
* Split replication layer into twoErik Johnston2018-03-131-1/+1
|
* Calculate push actions on workerErik Johnston2018-02-281-0/+8
|
* Move storage functions for push calculationsErik Johnston2018-02-272-11/+2
| | | | This will allow push actions for an event to be calculated on workers.
* Add event_creator workerErik Johnston2018-02-071-0/+170
|
* Add replication http endpoint for event sendingErik Johnston2018-02-071-0/+4
|
* Factor out get_db_conn to HomeServer base classRichard van der Hoff2018-01-2610-130/+0
| | | | | This function is identical to all subclasses, so we may as well push it up to the base class to reduce duplication (and make use of it in the tests)
* Actually make it workErik Johnston2018-01-121-2/+10
|
* RefactorErik Johnston2018-01-121-9/+4
|
* When using synctl with workers, don't start the main synapse automaticallyErik Johnston2018-01-111-8/+26
|
* Implement listen_tcp method in remaining workersSilke2017-12-1810-192/+155
| | | | Signed-off-by: Silke <silke@slxh.eu>
* Remove logger argument and do not catch replication listenerSilke2017-12-182-24/+18
| | | | Signed-off-by: Silke <silke@slxh.eu>
* Add methods for listening on multiple addressesSilke Hofstra2017-12-172-51/+90
| | | | | | | Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP and reactor.listenSSL for multiple addresses. Signed-off-by: Silke Hofstra <silke@slxh.eu>
* Allow binds to both :: and 0.0.0.0Silke Hofstra2017-12-171-37/+58
| | | | | | | | | | Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed. This causes a warning explaining the behaviour. Configuration changed to match. See #2232 Signed-off-by: Silke Hofstra <silke@slxh.eu>
* Add a comment which might save some confusionRichard van der Hoff2017-11-241-0/+1
|