summary refs log tree commit diff
path: root/docker/configure_workers_and_start.py (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Route `/make_knock` and `/send_knock` to workers in Complement docker image ↵Andrew Morgan2024-05-161-0/+2
| | | | (#17195)
* Fixups to new push stream (#17038)Erik Johnston2024-03-281-0/+8
| | | Follow on from #17037
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-231-0/+1
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Add a `--generate-only` option to the Complement launcher. (#16828)reivilibre2024-01-221-1/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pulled out of #16803 since the drive-by cleanup was maybe not as drive-by as I had hoped. <!-- Fixes: # <!-- --> <!-- Supersedes: # <!-- --> <!-- Follows: # <!-- --> <!-- Part of: # <!-- --> Base: `develop` <!-- git-stack-base-branch:develop --> <!-- This pull request is commit-by-commit review friendly. <!-- --> <!-- This pull request is intended for commit-by-commit review. <!-- --> Original commit schedule, with full messages: <ol> <li> Add a --generate-only option </li> </ol> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Update license headersPatrick Cloke2023-11-211-10/+16
|
* Add `/notifications` endpoint to workers (#16265)Erik Johnston2023-09-071-0/+1
|
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-151-1/+1
|
* Unix Sockets for HTTP Replication (#15708)Jason Little2023-07-111-26/+78
| | | | | | | | | Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at. --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Eric Eastwood <erice@element.io>
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-161-1/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* Log when events are (unexpectedly) filtered out of responses in tests (#14213)Eric Eastwood2023-06-011-0/+3
| | | | | See https://github.com/matrix-org/synapse/pull/14095#discussion_r990335492 This is useful because when see that a relevant event is an `outlier` or `soft-failed`, then that's a good unexpected indicator explaining why it's not showing up. `filter_events_for_client` is used in `/sync`, `/messages`, `/context` which are all common end-to-end assertion touch points (also notifications, relations).
* Remove `worker_replication_*` settings (#15491)Jason Little2023-05-111-2/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add master to the instance_map as part of Complement, have ReplicationEndpoint look at instance_map for master. * Fix typo in drive by. * Remove unnecessary worker_replication_* bits from unit tests and add master to instance_map(hopefully in the right place) * Several updates: 1. Switch from master to main for naming the main process in the instance_map. Add useful constants for easier adjustment of names in the future. 2. Add backwards compatibility for worker_replication_* to allow time to transition to new style. Make sure to prioritize declaring main directly on the instance_map. 3. Clean up old comments/commented out code. 4. Adjust unit tests to match with new code. 5. Adjust Complement setup infrastructure to only add main to the instance_map if workers are used and remove now unused options from the worker.yaml template. * Initial Docs upload * Changelog * Missed some commented out code that can go now * Remove TODO comment that no longer holds true. * Fix links in docs * More docs * Remove debug logging * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Update version to latest, include completeish before/after examples in upgrade notes. * Fix up and docs too --------- Co-authored-by: reivilibre <olivier@librepush.net>
* Load `/capabilities` endpoint on workers (#15436)Dirk Klimpel2023-04-141-0/+1
|
* Load `/directory/room/{roomAlias}` endpoint on workers (#15333)Dirk Klimpel2023-04-141-0/+1
| | | | | | | | | * Enable `directory` * move to worker store * newsfile * disable `ClientDirectoryListServer` and `ClientAppserviceDirectoryListServer` for workers
* Load `/password_policy` endpoint on workers. (#15331)Dirk Klimpel2023-03-271-0/+1
|
* Load `/register/available` endpoint on workers (#15268)Jason Little2023-03-171-0/+1
|
* Make `configure_workers_and_start` script used in Complement tests ↵reivilibre2023-03-171-1/+1
| | | | compatible with older versions of Python. (#15275)
* Additional functionality for declaring worker types in Complement (#14921)Jason Little2023-03-141-112/+409
| | | | Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Add support for knocking to workers. (#15133)Dirk Klimpel2023-03-021-0/+1
|
* Dockerfile-workers: spell out when config isn't generated (#15186)David Robertson2023-03-021-1/+5
| | | | | * Complement: Spell out when config isn't generated * Changelog
* Allow use of the `/filter` Client-Server APIs on workers. (#15134)reivilibre2023-02-281-0/+1
|
* Move MSC3030 `/timestamp_to_event` endpoint to stable v1 location (#14471)Eric Eastwood2022-11-281-0/+2
| | | | | | | | Fix https://github.com/matrix-org/synapse/issues/14390 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` Complement test changes: https://github.com/matrix-org/complement/pull/559
* Remove need for `worker_main_http_uri` setting to use /keys/upload. (#14400)realtyem2022-11-161-4/+1
|
* Modernize configure_workers_and_start.py bootstrapping script for ↵realtyem2022-11-091-15/+14
| | | | Dockerfile-workers. (#14294)
* Add all Stream Writer worker types to configure_workers_and_start.py (#14197)realtyem2022-11-081-7/+69
| | | Co-authored-by: reivilibre <oliverw@matrix.org>
* Flush stdout/err in Dockerfile-workers before replacing the current process ↵realtyem2022-10-181-12/+8
| | | | | | | (#14195) Also update `subprocess.check_output` to the slightly newer `subprocess.run`. Signed-off-by: Jason Little <realtyem@gmail.com>
* Strip whitespace from worker types in Dockerfile-workers (#14165)realtyem2022-10-141-4/+2
|
* Set LD_PRELOAD to load jemalloc in Dockerfile-workers. (#14182)realtyem2022-10-141-1/+11
|
* Stabilize the threads API. (#14175)Patrick Cloke2022-10-141-0/+1
| | | | | | | Stabilize the threads API (MSC3856) by supporting (only) the v1 path for the endpoint. This also marks the API as safe for workers since it is a read-only API.
* Mark /relations endpoint as usable on workers. (#14028)Patrick Cloke2022-10-121-0/+27
| | | Co-authored-by: Eric Eastwood <erice@element.io>
* Add the ability to set the log level using the `SYNAPSE_TEST_LOG_LEVEL` ↵reivilibre2022-07-051-6/+14
| | | | environment when using `complement.sh`. (#13152)
* Improve startup times in Complement test runs against workers, particularly ↵reivilibre2022-06-301-0/+7
| | | | | in CPU-constrained environments. (#13127) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Refactor the Dockerfile-workers configuration script to use Jinja2 templates ↵reivilibre2022-06-271-24/+12
| | | | | in Synapse workers' Supervisord blocks. (#13054) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Use updated `update_user_directory_from_worker` options in Complement tests. ↵reivilibre2022-06-151-3/+3
| | | | (#13069)
* Merge the Complement testing Docker images into a single, multi-purpose ↵reivilibre2022-06-081-8/+14
| | | | | image. (#12881) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add the `batch_send` endpoint to generic workers (#12868)Nick Mills-Barrett2022-05-251-0/+1
|
* Support registering Application Services when running with workers under ↵reivilibre2022-05-231-0/+15
| | | | | Complement. (#12826) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Remove Caddy from the Synapse workers image used in Complement. (#12818)reivilibre2022-05-231-0/+5
|
* Add the `notify_appservices_from_worker` configuration option (superseding ↵reivilibre2022-05-061-2/+2
| | | | `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)
* Use supervisord to supervise Postgres and Caddy in the Complement image. ↵reivilibre2022-04-271-1/+1
| | | | | (#12480) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* disallow-untyped-defs in `docker` and `stubs` directories (#12528)David Robertson2022-04-251-11/+13
|
* Dockerfile-workers: reduce the amount we install (#12464)Richard van der Hoff2022-04-141-10/+9
| | | | This is an attempt to reduce the rebuild time. In short, we reduce the amount of stuff that the dockerfile installs, so as to give a faster startup.
* Dockerfile-workers: give the master its own log config (#12466)Richard van der Hoff2022-04-131-17/+31
| | | | | | | | | | | When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example: ``` 2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches ``` - note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading. To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
* Fix up healthcheck generation for workers docker image (#12405)Richard van der Hoff2022-04-111-7/+9
| | | This wasn't quite generating the right thing.
* Add missing type definitions for scripts in docker folder (#12280)Jorge Florian2022-04-081-4/+5
| | | | Signed-off-by: Jorge Florian <jafn28@gmail.com>
* Create healthcheck script for synapse-workers container (#11429)Michael Kaye2021-11-261-0/+13
| | | | | The intent is to iterate through all the worker ports and only report healthy when all are healthy, starting with the main process.
* Add support for `/_matrix/media/v3` APIs (#11371)Aaron R2021-11-171-11/+11
| | | | | | | | | | | | | * Add support for `/_matrix/media/v3` APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Update `workers.md` to use v3 client and media APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Add changelog Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Fix frontend_proxy jinja script in docker workers (#10783)kegsay2021-09-081-1/+1
|
* Run complement with Synapse workers manually. (#10039)Patrick Cloke2021-05-241-4/+4
| | | | Adds an option to complement.sh to run Synapse in worker mode (instead of the default monolith mode).
* Add a dockerfile for running a set of Synapse worker processes (#9162)Andrew Morgan2021-04-141-0/+558
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with: * A Synapse main process. * Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime. * A redis for worker communication. * A nginx for routing traffic. * A supervisord to start all worker processes and monitor them if any go down. Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite. `configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI. Huge thanks to @MatMaul for helping get this all working :tada: This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62. Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-22 00:06:28 +0000