| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
can't exist.
|
| |
|
|
|
|
|
|
| |
1. Update a few comments for clarity.
2. Apply a replace() to remove whitespace in a requested worker name and use an underscore instead.
3. Stop-error if requesting a worker name, but forgetting to actually put one in.
4. If a worker name isn't requested, just use the set of worker types, sorted and concatenated with a hyphen. This avoids having to sanitize the worker name in this case.
5. Add regex to avoid nasty name surprises.
|
| |
|
|
|
| |
...so that even with a single port, a worker gets it's own upstream. Nginx says this can
lead to performance improvements if some features are enabled. Revisit this later.
|
| |
|
|
| |
update comments to match.
|
| |
|
|
|
| |
1. Update comments and edit(or create) docstrings and a single type hint that was incomplete(split_and_strip_string()).
2. Rename parse_worker_types_from_env to parse_worker_types as it's not coming from the environment.
|
| |
|
|
| |
'worker_type', and similar and that should have been 'worker_types' when in use by sets and a couple other places.
|
| | |
|
| |\ |
|
| | | |
|
| | |
| |
| |
| |
| | |
* Complement: Spell out when config isn't generated
* Changelog
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
1. Factor processing of all worker types from the environment out and away from
generating config files.
2. Fix up early templating of 'worker_config' to remove boilerplate.
3. Wrangle nginx upstream processing to accommodate odd combinations for overlapping
workers. e.g. 'user_dir, user_dir+presence' should be able to handle the(in this case
single) endpoint over either worker. I think this was promised in a previous commit,
consider it delivered.
4. Update a bunch of comments, and adjust some pre-existing to fit inside the line
length count of 88(aka, make the green squiggles go away)
5. Move processing of unusable characters for worker names to a later position so it
will check the name produced if one was not requested as well.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
1. Turn 'placeholder_name' into a magic constant.
2. Fix casting on a worker_type_list from 'list' to 'List[str]'.
3. In 'insert_worker_name_for_shared_config', use new WORKER_PLACEHOLDER_NAME constant and reduce complexity by using value from items() directly.
4. In 'is_sharding_allowed_for_worker_type', simplify the return value.
5. Adjust 'split_string_and_strip' to allow for using the 'maxsplit' kwarg on 'strip' and update comment.
6. Remove function 'increment_counter' and use 'defaultdict' instead to reduce complexity.
7. Remove function 'is_name_allowed_for_worker', as this can be accomplished with an 'if' statement and only checked for a worker base name in a
dict. Adjust 'generate_worker_files' to accommadate.
8. Rename a few dict's to remove misnomer's and adjust comments for extra clarity.
9. In 'generate_worker_files':
a. Use 'ValueError' exception handling to catch additional errors when parsing an integer and simplify counting of appended worker_type's.
b. Add additional error cases to check(single and double quotes, spaces, and last character being an integer) for potential file name/nginx issues.
c. Adjust various variable names to improve readability.
10. Update various comments throughout.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
In the Synapse-worker docker image, multiple worker types are defined but only allowed to be that one single definition.
This makes it so that any two(or more) worker types can be merged into a single worker. For example:
1. If you wish to have one worker with both client_reader and federation_sender functions enabled, you can set
SYNAPSE_WORKER_TYPES='client_reader+federation_sender'
2. If you wish to have all stream_writers(excepting event_persister) you can set
SYNAPSE_WORKER_TYPES='account_data+presence+receipts+to_device+typing'
Multiple types can be combined, but some error checking to dis-allow multiples of worker types that shouldn't be
enabled more than once has been added. For example:
SYNAPSE_WORKER_TYPES='background_worker+event_persister, background_worker+event_persister' will not work as
background_worker is only allowed to have a single worker for the entire deployment.
Giving a worker or a combination of workers a custom name is as simple as adding the name then an equal sign in front
of the given worker type(s). For example:
SYNAPSE_WORKER_TYPES='alice=federation_reader'
or
SYNAPSE_WORKER_TYPES='bob=federation_inbound+federation_sender'
or
SYNAPSE_WORKER_TYPES='charlie=event_persister:2, derek=media_repository + pusher + user_dir + appservice + event_creator'
|
| |/
|
|
|
|
|
| |
This allows for appending a ':' and a positive integer to a requested worker type for the SYNAPSE_WORKER_TYPES
environment variable(and likewise, the WORKER_TYPES) used in Complement. For example:
'event_persister:2' would be yield identical results of 'event_persister, event_persister'
|
| |
|
|
|
|
|
|
| |
Fix https://github.com/matrix-org/synapse/issues/14390
- Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`
- Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`
Complement test changes: https://github.com/matrix-org/complement/pull/559
|
| | |
|
| |
|
|
| |
Dockerfile-workers. (#14294)
|
| |
|
| |
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
|
| |
(#14195)
Also update `subprocess.check_output` to the slightly newer `subprocess.run`.
Signed-off-by: Jason Little <realtyem@gmail.com>
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.
This also marks the API as safe for workers since it is a read-only
API.
|
| |
|
| |
Co-authored-by: Eric Eastwood <erice@element.io>
|
| |
|
|
| |
environment when using `complement.sh`. (#13152)
|
| |
|
|
|
| |
in CPU-constrained environments. (#13127)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
|
| |
in Synapse workers' Supervisord blocks. (#13054)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
| |
(#13069)
|
| |
|
|
|
| |
image. (#12881)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
|
| |
|
|
|
| |
Complement. (#12826)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| |
|
|
| |
`notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)
|
| |
|
|
|
| |
(#12480)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
|
| |
|
|
| |
This is an attempt to reduce the rebuild time. In short, we reduce the amount
of stuff that the dockerfile installs, so as to give a faster startup.
|
| |
|
|
|
|
|
|
|
|
|
| |
When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example:
```
2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches
```
- note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading.
To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
|
| |
|
| |
This wasn't quite generating the right thing.
|
| |
|
|
| |
Signed-off-by: Jorge Florian <jafn28@gmail.com>
|
| |
|
|
|
| |
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Add support for `/_matrix/media/v3` APIs
Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Update `workers.md` to use v3 client and media APIs
Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add changelog
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| | |
|
| |
|
|
| |
Adds an option to complement.sh to run Synapse in worker
mode (instead of the default monolith mode).
|
|
|
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with:
* A Synapse main process.
* Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime.
* A redis for worker communication.
* A nginx for routing traffic.
* A supervisord to start all worker processes and monitor them if any go down.
Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite.
`configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI.
Huge thanks to @MatMaul for helping get this all working :tada: This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62.
Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
|
