summary refs log tree commit diff
path: root/docker (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add release note and update doc regarding ICU (#14712)Mathieu Velten2022-12-201-0/+1
| | | | | Fixes #14704. Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* Add optional ICU support for user search (#14464)Brendan Abolivier2022-12-122-0/+4
| | | | | | | Fixes #13655 This change uses ICU (International Components for Unicode) to improve boundary detection in user search. This change also adds a new dependency on libicu-dev and pkg-config for the Debian packages, which are available in all supported distros.
* Add `--editable` flag to `complement.sh` which uses an editable install of ↵reivilibre2022-12-073-2/+79
| | | | | Synapse for faster turn-around times whilst developing iteratively. (#14548) Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
* docker: remove useless cargo install with apt (#14636)Mathieu Velten2022-12-071-1/+1
| | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* Move MSC3030 `/timestamp_to_event` endpoint to stable v1 location (#14471)Eric Eastwood2022-11-282-2/+2
| | | | | | | | Fix https://github.com/matrix-org/synapse/issues/14390 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` Complement test changes: https://github.com/matrix-org/complement/pull/559
* Remove need for `worker_main_http_uri` setting to use /keys/upload. (#14400)realtyem2022-11-161-4/+1
|
* Modernize configure_workers_and_start.py bootstrapping script for ↵realtyem2022-11-091-15/+14
| | | | Dockerfile-workers. (#14294)
* Add all Stream Writer worker types to configure_workers_and_start.py (#14197)realtyem2022-11-081-7/+69
| | | Co-authored-by: reivilibre <oliverw@matrix.org>
* Allow override for requesting specific worker types for Complement on ↵realtyem2022-11-081-1/+8
| | | | | | | | | | | | | | | | | | | | | | | command line. (#14324) * Expose getting SYNAPSE_WORKER_TYPES from external, allowing override of workers requested. * Add WORKER_TYPES variable option to complement.sh script that passes requested workers into start_for_complement.sh entrypoint. * Update docs to reflect this new ability. * Changelog * Don't rely on soft wrapping to format long strings Good idea dklimpel. Thanks for catching that. Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com> * Small nits just noticed in docs. * Fixup new line in docs. Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
* Enable testing MSC3874 in complement. (#14339)Patrick Cloke2022-11-011-2/+2
|
* Enable URL previews in complement homeserver config. (#14198)Jonathan de Jong2022-10-181-0/+2
|
* Flush stdout/err in Dockerfile-workers before replacing the current process ↵realtyem2022-10-182-17/+21
| | | | | | | (#14195) Also update `subprocess.check_output` to the slightly newer `subprocess.run`. Signed-off-by: Jason Little <realtyem@gmail.com>
* Cache Rust build cache when building docker images (#14130)Erik Johnston2022-10-171-1/+3
|
* Strip whitespace from worker types in Dockerfile-workers (#14165)realtyem2022-10-141-4/+2
|
* Merge remote-tracking branch 'origin/release-v1.69' into developPatrick Cloke2022-10-141-1/+7
|\
| * Fix docker build OOMing in CI for arm64 builds (#14173)Erik Johnston2022-10-131-0/+6
| | | | | | Co-authored-by: David Robertson <davidr@element.io>
| * Fix pinning Rust deps in docker images (#14129)Erik Johnston2022-10-111-1/+1
| |
* | Set LD_PRELOAD to load jemalloc in Dockerfile-workers. (#14182)realtyem2022-10-141-1/+11
| |
* | Stabilize the threads API. (#14175)Patrick Cloke2022-10-141-0/+1
| | | | | | | | | | | | | | Stabilize the threads API (MSC3856) by supporting (only) the v1 path for the endpoint. This also marks the API as safe for workers since it is a read-only API.
* | Mark /relations endpoint as usable on workers. (#14028)Patrick Cloke2022-10-122-0/+28
| | | | | | Co-authored-by: Eric Eastwood <erice@element.io>
* | Use minimal Rust installation in docker images and CI (#14141)David Robertson2022-10-122-2/+2
| |
* | Fix broken links to README (#14093)David Robertson2022-10-061-1/+1
| |
* | Complement test image: capture logs from nginx (#14063)Richard van der Hoff2022-10-051-1/+5
|/ | | | | Have nginx send its logs to stderr/out, so that we can debug https://github.com/matrix-org/synapse/issues/13334.
* Dockerfile for tests: align comments with current behavior (#13867)Mathieu Velten2022-09-271-13/+9
| | | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix Docker build when Rust .so has been build locally first (#13811)Mathieu Velten2022-09-161-3/+5
| | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* complement: init postgres DB directly inside the target image (#13819)Mathieu Velten2022-09-161-11/+9
| | | | | | | Doing so in the base postgres image doesn't work with buildah because changes in a declared VOLUME in the Dockerfile is supposed to be discarded, cf https://docs.docker.com/engine/reference/builder/#volume Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* Add a stub Rust crate (#12595)Erik Johnston2022-09-062-2/+22
|
* Bump docker image to use stable poetry version (#13688)Erik Johnston2022-09-011-10/+1
|
* Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker ↵reivilibre2022-08-121-1/+1
| | | | image. (#13515)
* synapse-workers docker: copy nginx and redis in from base images (#13447)Richard van der Hoff2022-08-042-36/+59
| | | Part of my continuing quest to make the docker images build quicker: copy nginx and redis in from base docker images, rather than apt installing each time.
* Install cryptography build dependencies in requirements image. (#13372)Jasper Spaans2022-08-031-1/+2
|
* Enable Complement CI tests in the 'latest deps' test run. (#13213)reivilibre2022-08-011-2/+22
| | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Call out buildkit is required when building test docker images (#13338)Brendan Abolivier2022-07-213-0/+6
| | | Co-authored-by: David Robertson <davidr@element.io>
* Rate limit joins per-room (#13276)David Robertson2022-07-191-0/+4
|
* Use and recommend poetry 1.1.14, up from 1.1.12 (#13285)David Robertson2022-07-151-1/+1
|
* Docker: copy postgres from base image (#13279)Richard van der Hoff2022-07-152-34/+50
| | | | | | When building the docker images for complement testing, copy a preinstalled complement over from a base image, rather than apt installing it. This avoids network traffic and is much faster.
* Add the ability to set the log level using the `SYNAPSE_TEST_LOG_LEVEL` ↵reivilibre2022-07-053-6/+27
| | | | environment when using `complement.sh`. (#13152)
* [Complement] Allow device_name lookup over federation (#13167)Till2022-07-041-0/+2
|
* Enable Complement testing in the 'Twisted Trunk' CI runs. (#13079)reivilibre2022-07-011-1/+8
|
* Improve startup times in Complement test runs against workers, particularly ↵reivilibre2022-06-305-3/+43
| | | | | in CPU-constrained environments. (#13127) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Refactor the Dockerfile-workers configuration script to use Jinja2 templates ↵reivilibre2022-06-273-38/+42
| | | | | in Synapse workers' Supervisord blocks. (#13054) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add a Subject Alternative Name to the certificate generated for Complement ↵reivilibre2022-06-161-3/+19
| | | | tests. (#13071)
* Complement: use SQLite by default (#13075)Richard van der Hoff2022-06-162-2/+2
| | | | | If no database is configured explicitly, use sqlite. This means that you don't have to pass any variables into the image.
* Change default `sync_response_cache_duration` (#13042)Erik Johnston2022-06-151-0/+6
|
* Use updated `update_user_directory_from_worker` options in Complement tests. ↵reivilibre2022-06-151-3/+3
| | | | (#13069)
* Collapse Docker build commands in Complement CI runs to make the logs easier ↵reivilibre2022-06-153-6/+6
| | | | to read. (#13058)
* Merge the Complement testing Docker images into a single, multi-purpose ↵reivilibre2022-06-0815-361/+247
| | | | | image. (#12881) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Add the `batch_send` endpoint to generic workers (#12868)Nick Mills-Barrett2022-05-251-0/+1
|
* Pin poetry.core in Docker images (#12853)Erik Johnston2022-05-241-1/+1
|
* Support registering Application Services when running with workers under ↵reivilibre2022-05-233-1/+30
| | | | | Complement. (#12826) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix Complement `TestCanRegisterAdmin` with workers, by adding Complement's ↵reivilibre2022-05-231-0/+6
| | | | shared registration secret. (#12819)
* Remove Caddy from the Synapse workers image used in Complement. (#12818)reivilibre2022-05-236-93/+37
|
* Update configs used by Complement to allow more invites (#12731)Till2022-05-132-0/+24
|
* Tweaks to workers-under-complement (#12637)Richard van der Hoff2022-05-091-3/+3
| | | | | | | * Bump the HS startup timeout * Log prefixes for more processes * Bump the overall timeout
* Add the `notify_appservices_from_worker` configuration option (superseding ↵reivilibre2022-05-061-2/+2
| | | | `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)
* Use supervisord to supervise Postgres and Caddy in the Complement image. ↵reivilibre2022-04-278-12/+43
| | | | | (#12480) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Mark Dockerfile as requiring BuildKit (#12541)Jason Robinson2022-04-261-0/+1
| | | Co-authored-by: David Robertson <davidr@element.io>
* disallow-untyped-defs in `docker` and `stubs` directories (#12528)David Robertson2022-04-252-28/+36
|
* Await un-partial-stating after a partial-state join (#12399)Richard van der Hoff2022-04-211-1/+3
| | | | | | When we join a room via the faster-joins mechanism, we end up with "partial state" at some points on the event DAG. Many parts of the codebase need to wait for the full state to load. So, we implement a mechanism to keep track of which events have partial state, and wait for them to be fully-populated.
* Dump setuptools; correct pyproject version number (#12478)David Robertson2022-04-201-4/+2
|
* Dockerfile-workers: reduce the amount we install (#12464)Richard van der Hoff2022-04-143-13/+26
| | | | This is an attempt to reduce the rebuild time. In short, we reduce the amount of stuff that the dockerfile installs, so as to give a faster startup.
* complement-synapse-workers: factor out separate entry point script (#12467)Richard van der Hoff2022-04-142-33/+57
| | | ... with a bit more verbosity.
* Dockerfile-workers: give the master its own log config (#12466)Richard van der Hoff2022-04-131-17/+31
| | | | | | | | | | | When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example: ``` 2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches ``` - note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading. To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
* Workaround pip bug installing latest treq and not-latest twisted from hashes ↵David Robertson2022-04-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | (#12439) The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0. When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true > ``` > #15 9.204 Collecting Twisted[tls]>=18.7.0 > #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not: > #15 9.205 Twisted[tls]>=18.7.0 from https://files.pythonhosted.org/packages/db/99/38622ff95bb740bcc991f548eb46295bba62fcb6e907db1987c4d92edd09/Twisted-22.4.0-py3-none-any.whl#sha256=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724)) > #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1 > ``` The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so. (At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
* Enable certificate checking during complement tests (#12435)Richard van der Hoff2022-04-112-12/+0
|
* Fix up healthcheck generation for workers docker image (#12405)Richard van der Hoff2022-04-112-10/+9
| | | This wasn't quite generating the right thing.
* Move complement setup stuff into the Synapse repo (#12404)Richard van der Hoff2022-04-119-23/+442
| | | | | | Fixes matrix-org/complement#330 (or it will, once we remove the old files). It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
* Add missing type definitions for scripts in docker folder (#12280)Jorge Florian2022-04-081-4/+5
| | | | Signed-off-by: Jorge Florian <jafn28@gmail.com>
* Poetry: use locked environment in Docker images (#12385)David Robertson2022-04-072-25/+63
|
* README-testing.md: fix minor errorRichard van der Hoff2022-03-311-1/+1
|
* Remove `dockerfile-pgtests` (#12336)David Robertson2022-03-312-49/+0
|
* Move synctl into `synapse._scripts` and expose as an entrypoint (#12140)David Robertson2022-03-041-1/+1
|
* Move scripts directory inside synapse, exposing as setuptools entry_points ↵David Robertson2022-03-021-1/+0
| | | | | | | | | | | | (#12118) * Two scripts are basically entry_points already * Move and rename scripts/* to synapse/_scripts/*.py * Delete sync_room_to_group.pl * Expose entry points in setup.py * Update linter script and config * Fixup scripts & docs mentioning scripts that moved Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Use Python 3.9 in Synapse dockerfiles by default (#12112)Andrew Morgan2022-03-011-2/+2
|
* Docker: remove `VOLUME` directive (#11997)Richard van der Hoff2022-02-151-2/+0
| | | | | | | | | | | The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway. I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes. That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data. So... this wasn't doing anything, except annoying Complement?
* Drop support for and remove references to EOL Python 3.6 (#11683)Shay2022-01-212-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * remove reference in comments to python3.6 * upgrade tox python env in script * bump python version in example for completeness * upgrade python version requirement in setup doc * upgrade necessary python version in __init__.py * upgrade python version in setup.py * newsfragment * drops refs to bionic and replace with focal * bump refs to postgres 9.6 to 10 * fix hanging ci * try installing tzdata first * revert change made in b979f336 * ignore new random mypy error while debugging other error * fix lint error for temporary workaround * revert change to install list * try passing env var * export debian frontend var? * move line and add comment * bump pillow dependency * bump lxml depenency * install libjpeg-dev for pillow * bump automat version to one compatible with py3.8 * add libwebp for pillow * bump twisted trunk python version * change suffix of newsfragment * remove redundant python 3.7 checks * lint
* Use buildkit's cache feature to speed up docker builds (#11691)Richard van der Hoff2022-01-121-5/+21
| | | Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
* Improve Docker docs for use with Postgres (#11640)Callum Macdonald2022-01-051-0/+4
|
* Drop Bionic from Debian builds (#11633)Shay2022-01-031-5/+4
| | | | | | | | | | | | | | | | | * update Trove classifiers to remove py36 * stop building bionic * update dh-virtualenv * newsfragment * fix newsfragment * update version refs * another try at correct tag * Update changelog
* Create healthcheck script for synapse-workers container (#11429)Michael Kaye2021-11-263-0/+22
| | | | | The intent is to iterate through all the worker ports and only report healthy when all are healthy, starting with the main process.
* Remove code invalidated by deprecated config flag ↵Shay2021-11-231-8/+0
| | | | | | | | | | | | | | | | | | | | | 'trust_identity_servers_for_password_resets' (#11395) * remove background update code related to deprecated config flag * changelog entry * update changelog * Delete 11394.removal Duplicate, wrong number * add no-op background update and change newfragment so it will be consolidated with associated work * remove unused code * Remove code associated with deprecated flag from legacy docker dynamic config file Co-authored-by: reivilibre <oliverw@matrix.org>
* Add support for `/_matrix/media/v3` APIs (#11371)Aaron R2021-11-171-11/+11
| | | | | | | | | | | | | * Add support for `/_matrix/media/v3` APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Update `workers.md` to use v3 client and media APIs Signed-off-by: Aaron Raimist <aaron@raim.ist> * Add changelog Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Docker: avoid changing userid unnecessarily (#11209)Richard van der Hoff2021-11-012-23/+27
| | | | | | | | | | | | * Docker image: avoid changing user during `generate` The intention was always that the config files get written as the initial user (normally root) - only the data directory needs to be writable by Synapse. This got changed in https://github.com/matrix-org/synapse/pull/5970, but that seems to have been a mistake. * Avoid changing user if no explicit UID is given * changelog
* Fix Shellcheck SC2006: Use $(...) notationDan Callahan2021-10-221-2/+2
| | | | | | | | Use $(...) notation instead of legacy backticked `...`. https://github.com/koalaman/shellcheck/wiki/SC2006 Signed-off-by: Dan Callahan <danc@element.io>
* Fix Shellcheck SC2086: Quote to prevent splittingDan Callahan2021-10-221-1/+1
| | | | | | | | Double quote to prevent globbing and word splitting. https://github.com/koalaman/shellcheck/wiki/SC2086 Signed-off-by: Dan Callahan <danc@element.io>
* Fix Shellcheck SC2154: variable possibly undefinedDan Callahan2021-10-221-1/+1
| | | | | | | | var is referenced but not assigned. https://github.com/koalaman/shellcheck/wiki/SC2154 Signed-off-by: Dan Callahan <danc@element.io>
* fix relative link in docker readme (#11144)Richard van der Hoff2021-10-211-1/+2
| | | relative links don't work when it's on dockerhub.
* Fix debian package builds. (#10931)Erik Johnston2021-09-281-2/+3
| | | | This was due to dh-virtualenv builds being broken due to Shpinx removing deprecated APIs.
* Update postgresql testing script (#10906)David Robertson2021-09-242-7/+24
| | | | | | | | | | | | | | | | | | - Use sytest:bionic. Sytest:latest is two years old (do we want CI to push out latest at all?) and comes with Python 3.5, which we explictly no longer support. The script now runs under PostgreSQL 10 as a result. - Advertise script in the docs - Move pg testing script to scripts-dev directory - Write to host as the script's exector, not root A few changes to make it speedier to re-run the tests: - Create blank DB in the container, not the script, so we don't have to `initdb` each time - Use a named volume to persist the tox environment, so we don't have to fetch and install a bunch of packages from PyPI each time Co-authored-by: reivilibre <olivier@librepush.net>
* Fix frontend_proxy jinja script in docker workers (#10783)kegsay2021-09-081-1/+1
|
* Make `PeriodicallyFlushingMemoryHandler` the default logging handler. (#10518)reivilibre2021-08-171-7/+20
|
* Fix deb build script to set prerelease flag correctly (#10500)Richard van der Hoff2021-08-011-5/+4
|
* Drop xenial-support hacks (#10429)Richard van der Hoff2021-07-211-7/+11
|
* Set section for prerelease debs (#10391)Richard van der Hoff2021-07-141-0/+14
| | | This is part of fixing #6116: we want to put RC debs into a different place than release debs, so reprepro has to be able to tell them apart.
* Fix broken links in INSTALL.md (#10331)Dirk Klimpel2021-07-081-2/+2
| | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Remove support for ACME v1 (#10194)Brendan Abolivier2021-06-171-6/+0
| | | | | Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
* Fix broken link to README at root of repo (#10132)Chris Castle2021-06-071-1/+1
| | | Signed-off-by: Chris Castle chris@crc.io
* Run complement with Synapse workers manually. (#10039)Patrick Cloke2021-05-241-4/+4
| | | | Adds an option to complement.sh to run Synapse in worker mode (instead of the default monolith mode).
* Fix docker image to not log at `/homeserver.log` (#10045)Sergio Miguéns2021-05-241-1/+3
| | | | | Fixes #9970 Signed-off-by: Sergio Miguéns Iglesias lonyelon@lony.xyz
* Docker healthcheck timings - add startup delay and changed interval (#9913)Christopher May-Townsend2021-05-052-4/+15
| | | | | * Add healthcheck startup delay by 5secs and reduced interval check to 15s to reduce waiting time for docker aware edge routers bringing an instance online
* Add a dockerfile for running a set of Synapse worker processes (#9162)Andrew Morgan2021-04-1410-6/+866
| | | | | | | | | | | | | | | | | This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with: * A Synapse main process. * Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime. * A redis for worker communication. * A nginx for routing traffic. * A supervisord to start all worker processes and monitor them if any go down. Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite. `configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI. Huge thanks to @MatMaul for helping get this all working :tada: This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62. Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
* Add note to docker docs explaining platform support (#9801)Andrew Morgan2021-04-141-3/+6
| | | | | | | Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894. I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support. The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
* Put opencontainers labels to the final image (#9765)Johannes Wienke2021-04-081-5/+5
| | | | | | They don't make any sense on the intermediate builder image. The final images needs them to be of use for anyone. Signed-off-by: Johannes Wienke <languitar@semipol.de>
* Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff2021-03-301-8/+0
| | | | | | | `room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
* Make pip install faster in Docker build for Complement testing (#9610)Eric Eastwood2021-03-261-43/+41
| | | | | | | | | Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing. If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘 As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`. Fix https://github.com/matrix-org/synapse/issues/9364
* Merge branch 'master' into developErik Johnston2021-03-261-20/+23
|\
| * Explicitly upgrade openssl in docker file and enforce new version of ↵Erik Johnston2021-03-261-20/+23
| | | | | | | | cryptography (#9697)
* | Use interpreter from $PATH instead of absolute paths in various scripts ↵Quentin Gliech2021-03-252-2/+2
| | | | | | | | | | | | | | | | | | using /usr/bin/env (#9689) On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work. `/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* | Include opencontainers labels in Docker image (#9612)Johannes Wienke2021-03-221-0/+5
|/ | | | | | Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys Signed-off-by: Johannes Wienke <languitar@semipol.de>
* Install jemalloc in docker image (#8553)Mathieu Velten2021-03-163-2/+16
| | | | Co-authored-by: Will Hunt <willh@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Remove vestiges of uploads_path config (#9462)Richard van der Hoff2021-02-222-2/+0
| | | | `uploads_path` was a thing that was never used; most of it was removed in #6628 but a few vestiges remained.
* Add rustc to Docker image build environment (#9405)Dan Callahan2021-02-151-0/+2
| | | | | | | | This is needed to build the cryptography library, since it does not provide wheels for ARMv7. Fixes #9403 Signed-off-by: Dan Callahan <danc@element.io>
* Fix Debian builds on Xenial (#9254)Dan Callahan2021-01-291-0/+1
| | | | | | | | Adds note about updating dh-virtualenv once we drop support for Xenial. We can't update now, because it needs debhelper 12, while Xenial only backports 10. Signed-off-by: Dan Callahan <danc@element.io>
* Replace 'perspectives' config block with 'trusted_key_servers' in docker ↵Andrew Morgan2021-01-191-6/+4
| | | | homeserver.yaml template (#9157)
* Fix failures in Debian packaging (#9079)Dan Callahan2021-01-121-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Debian package builds were failing for two reasons: 1. Python versions prior to 3.7 throw exceptions when attempting to print Unicode characters under a "C" locale. (#9076) 2. We depended on `dh-systemd` which no longer exists in Debian Bullseye, but is necessary in Ubuntu Xenial. (#9073) Setting `LANG="C.UTF-8"` in the build environment fixes the first issue. See also: https://bugs.python.org/issue19846 The second issue is a bit trickier. The dh-systemd package was merged into debhelper version 9.20160709 and a transitional package left in its wake. The transitional dh-systemd package was removed in Debian Bullseye. However, Ubuntu Xenial ships an older debhelper, and still needs dh-systemd. Thus, builds were failing on Bullseye since we depended on a package which had ceased existing, but we couldn't remove it from the debian/control file and our build scripts because we still needed it for Ubuntu Xenial. We can fix the debian/control issue by listing dh-systemd as an alternative to the newer versions of debhelper. Since dh-systemd declares that it depends on debhelper, Ubuntu Xenial will select its older dh-systemd which will in turn pull in its older debhelper, resulting in no change from the status quo. All other supported releases will satisfy the debhelper dependency constraint and skip the dh-systemd alternative. Build scripts were fixed by unconditionally attempting to install dh-systemd on all releases and suppressing failures. Once we drop support for Ubuntu Xenial, we can revert most of this commit and rely on the version constraint on debhelper in debian/control. Fixes #9076 Fixes #9073 Signed-off-by: Dan Callahan <danc@element.io>
* Add 'xmlsec1' to dependency list github/release-v1.24.0 release-v1.24.0Erik Johnston2020-12-091-1/+2
|
* Remove version pin prometheus_client dependency (#8875)Jordan Bancino2020-12-041-1/+1
| | | | | | | This removes the version pin of the `prometheus_client` dependency, in direct response to #8831. If merged, this will close #8831 As far as I can tell, no other changes are needed, but as I'm no synapse expert, I'm relying heavily on CI and maintainer reviews for this. My very primitive test of synapse with prometheus_client v0.9.0 on my home server didn't bring up any issues, so we'll see what happens. Signed-off-by: Jordan Bancino
* Cap the version of prometheus_client to <v0.9.0 in the dockerfile (#8767)Andrew Morgan2020-11-171-1/+2
| | | Short-term fix for https://github.com/matrix-org/synapse/issues/8766.
* Use Python 3.8 in Docker images by default (#8698)Dan Callahan2020-11-021-1/+1
| | | | | | | This bumps us closer to current Python without going all the way to 3.9. Fixes #8674 Signed-off-by: Dan Callahan <danc@element.io>
* Docker: support passing additional commandline args to synapse (#8390)Samuel Philipp2020-10-112-18/+43
|
* Increase default max_upload_size from 10M to 50M (#8502)Mateusz Przybyłowicz2020-10-091-1/+1
| | | Signed-off-by: Mateusz Przybyłowicz <uamfhq@gmail.com>
* Add required Debian dependencies to allow docker builds on the arm platform ↵Christopher May-Townsend2020-08-261-3/+10
| | | | | | (#8144) Signed-off-by: Christopher May-Townsend <chris@maytownsend.co.uk>
* Add healthcheck for default localhost 8008 port on /health endpoint. (#8147)Christopher May-Townsend2020-08-242-0/+33
|
* Move setting of Filter into code.Erik Johnston2020-08-111-6/+0
| | | | | | | | | | | | | | We do this to prevent foot guns. The default config uses a MemoryFilter, but users are free to change to logging to files directly. If they do then they have to ensure to set the `filters: [context]` on the right handler, otherwise records get written with the wrong context. Instead we move the logic to happen when we generate a record, which is when we *log* rather than *handle*. (It's possible to add filters to loggers in the config, however they don't apply to descendant loggers and so they have to be manually set on *every* logger used in the code base)
* Add help for creating a user via docker (#7885)Michael Kaye2020-07-171-0/+15
|
* Switch to Debian:Slim from Alpine for the docker image (#7839)Christopher May-Townsend2020-07-172-40/+29
| | | As mentioned in #7397, switching to a debian base should help with multi-arch work to save time on compiling. This is unashamedly based on #6373, but without the extra functionality. Switch python version back to generic 3.7 to always pull the latest. Essentially, keeping this as small as possible. The image is bigger though unfortunately.
* Add libwebp dependency to Dockerfile (#7791)Juho Vanhanen2020-07-061-0/+2
| | | | | * Add libwebp dependency to Dockerfile Signed-off-by: Juho Vanhanen <juho@vanhanen.io>
* Switch back to upstream dh-virtualenv (#7621)Richard van der Hoff2020-06-031-1/+1
| | | | | | Upstream have merged our changes (https://github.com/spotify/dh-virtualenv/pull/300), so let's switch back to it instead of using our fork.
* Refresh apt cache when building dh_virtualenv docker image (#7555)Richard van der Hoff2020-05-221-2/+4
| | | | | | When we tried to build debs for 1.13.0, the build failed because docker used a base docker image which had a stale apt cache. Fixes: #7540
* Merge branch 'master' into developPatrick Cloke2020-05-191-7/+8
|\
| * update dh-virtualenv (#7526)Richard van der Hoff2020-05-191-7/+8
| |
* | Update runtime docker image to Alpine v3.11Cédric Laubacher2020-05-031-1/+1
|/
* Kill off deprecated "config-on-the-fly" docker mode (#6918)Richard van der Hoff2020-02-182-40/+32
| | | | | Lots of people seem to get confused by this mode, and it's been deprecated since Synapse 1.1.0. It's time for it to go.
* bump the version of Alpine Linux used in the docker images (#6897)Richard van der Hoff2020-02-121-1/+1
|
* Add working build command for docker image (#6390)Andrew Morgan2019-11-231-0/+12
| | | | | | * Add working build command for docker image * Add changelog
* Docker: Change permissions for data dir before attempting to write to it (#6389)Andrew Morgan2019-11-201-3/+3
|
* Merge remote-tracking branch 'origin/master' into developRichard van der Hoff2019-10-311-2/+3
|\
| * Clarify environment variable usage when running in Docker (#6181)Anton Lazarev2019-10-301-2/+3
| |
* | Fix broken URL in docker/README.md (#6264)Tobia De Koninck2019-10-281-1/+1
|/ | | Signed-off-by: Tobia De Koninck <LEDfan@users.noreply.github.com>
* Fix logging config for the docker image (#6197)Richard van der Hoff2019-10-181-0/+2
| | | | | | | | Turns out that loggers that are instantiated before the config is loaded get turned off. Also bring the logging config that is generated by --generate-config into line. Fixes #6194.
* Docker: support SYNAPSE_WORKER envvar (#6058)Michael Kaye2019-09-192-1/+4
| | | | | | | | | | | | | | | | * Allow passing SYNAPSE_WORKER envvar * changelog.d * Document SYNAPSE_WORKER. Attempting to imply that you don't need to change this default unless you're in worker mode. Also aware that there's a bigger problem of attempting to document a complete working configuration of workers using docker, as we currently only document to use `synctl` for worker mode, and synctl doesn't work that way in docker.
* Docker packaging should not su-exec or chmod if already running as UID/GID ↵Michael Kaye2019-09-031-35/+49
| | | | | | | | (#5970) Adjust su-exec to only be used if needed. If UID == getuid() and GID == getgid() then we do not need to su-exec, and chmod will not work.
* Typographical corrections in docker/README (#5921)Victor Goff2019-08-281-5/+5
|
* Fix debian packages for sid being called buster. (#5775)Erik Johnston2019-07-302-1/+7
| | | | | | | | | | * Fix debian packages for sid being called buster. I don't know why the sid images return buster as its codename in `lsb_release` but it does, so lets just grab the codename from the distro we pass into dockerfile * Newsfile
* Improve `Depends` specs in debian package. (#5675)Richard van der Hoff2019-07-171-0/+3
| | | | | | | | | | | | | | | | | | | This is basically a contrived way of adding a `Recommends` on `libpq5`, to fix #5653. The way this is supposed to happen in debhelper is to run `dh_shlibdeps`, which in turn runs `dpkg-shlibdeps`, which spits things out into `debian/<package>.substvars` whence they can later be included by `control`. Previously, we had disabled `dh_shlibdeps`, mostly because `dpkg-shlibdeps` gets confused about PIL's interdependent objects, but that's not really the right thing to do and there is another way to work around that. Since we don't always use postgres, we don't necessarily want a hard Depends on libpq5, so I've actually ended up adding an explicit invocation of `dpkg-shlibdeps` for `psycopg2`. I've also updated the build-depends list for the package, which was missing a couple of entries.
* Add missing space in default logging file format generated by the Docker ↵Slavi Pantaleev2019-07-121-1/+1
| | | | | | | image (#5620) This adds a missing space, without which log lines appear uglier. Signed-off-by: Slavi Pantaleev <slavi@devture.com>
* Upgrade Alpine Linux used in the Docker image (3.8 -> 3.10) (#5619)Slavi Pantaleev2019-07-121-2/+2
| | | | | | | | | Alpine Linux 3.8 is still supported, but it seems like it's quite outdated now. While Python should be the same on both, all other libraries, etc., are much newer in Alpine 3.9 and 3.10. Signed-off-by: Slavi Pantaleev <slavi@devture.com>
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-041-1/+1
|
* Remove SMTP_* env var functionality from docker conf (#5596)Andrew Morgan2019-07-031-19/+0
| | | | | Removes any `SMTP_*` docker container environment variables from having any effect on the default config. Fixes https://github.com/matrix-org/synapse/issues/5430
* Add ability to set timezone for Docker container (#5383)Amir Zarrinkafsh2019-07-022-1/+2
| | | | Signed-off-by: Amir Zarrinkafsh <nightah@me.com>
* Docker image: Add a migrate_config mode (#5567)Richard van der Hoff2019-06-273-17/+57
| | | ... to help people escape env var hell
* Docker image: open the non-TLS port by default. (#5568)Richard van der Hoff2019-06-271-0/+1
| | | There's not much point in binding to localhost when it's in a docker container.
* Deprecate the env var way of running the docker image (#5566)Richard van der Hoff2019-06-272-154/+94
| | | | | This is mostly a documentation change, but also adds a default value for SYNAPSE_CONFIG_PATH, so that running from the generated config is the default, and will Just Work provided your config is in the right place.
* Merge pull request #5565 from matrix-org/rav/docker/fix_log_configRichard van der Hoff2019-06-272-1/+13
|\ | | | | Docker: generate our own log config
| * Docker: generate our own log configRichard van der Hoff2019-06-261-1/+8
| | | | | | | | | | | | When running under docker, we want to use docker's own logging stuff rather than losing the logs somewhere on the container's filesystem, so let's use log configs that spit logs out to stdout instead.
| * Add support for SYNAPSE_CONFIG_DIRRichard van der Hoff2019-06-262-0/+5
| |
* | Merge pull request #5563 from matrix-org/rav/docker/data_dirRichard van der Hoff2019-06-272-4/+24
|\| | | | | Docker image: add support for SYNAPSE_DATA_DIR parameter
| * Docker image: add support for SYNAPSE_DATA_DIR parameterRichard van der Hoff2019-06-262-4/+17
| | | | | | | | Fixes #4830.
| * document supported env vars for docker 'generate' optionRichard van der Hoff2019-06-261-0/+7
| |
* | Docker: only run --generate-keys when generating config on-the-fly.Richard van der Hoff2019-06-261-7/+21
| | | | | | | | | | | | | | | | | | We don't want to generate any missing configs when running from a precanned config. (There's a strong argument that we don't want to do this at all, since generating a new signing key on each invocation sounds disasterous, but I don't fancy unpicking that for now.)
* | improve logging for generate_config_from_templateRichard van der Hoff2019-06-261-1/+2
|/
* isortRichard van der Hoff2019-06-251-4/+5
|
* Add a main() functionRichard van der Hoff2019-06-251-8/+11
|
* Factor out a run_generate_config functionRichard van der Hoff2019-06-251-17/+28
|
* Factor out "generate_config_from_template"Richard van der Hoff2019-06-251-41/+81
| | | | ... and inline generate_secrets
* Increase default log level for docker image to INFO. (#5547)Richard van der Hoff2019-06-251-5/+2
| | | Fixes #3370.
* Update docker image to use Python 3.7. (#5546)Richard van der Hoff2019-06-251-1/+1
| | | Python 3.7 is apparently faster than 3.6, and should be mature enough.
* Run Black. (#5482)Amber Brown2019-06-201-19/+39
|
* Include xmlsec in the docker imageRichard van der Hoff2019-06-181-1/+3
| | | | Fixes #5467.
* Remove Postgres 9.4 support (#5448)Amber Brown2019-06-182-3/+3
|
* Change to absolute path for contrib/dockerNeil Johnson2019-06-131-1/+1
| | | because this file is reproduced on dockerhub and relative paths don't work
* Update README.md (#5222)Steffen2019-05-221-1/+1
| | | Add missing backslash
* Merge branch 'master' into developRichard van der Hoff2019-05-031-1/+2
|\
| * Debian: we now need libpq-dev.Richard van der Hoff2019-05-031-1/+2
| | | | | | | | | | | | | | | | psycopg 2.8 is now out, which means that the C library gets built from source, so we now need libpq-dev when building. Turns out the need for this package is already documented in docs/postgres.rst.
* | Fix handling of SYNAPSE_NO_TLS in docker image (#5005)*=0=1=4=*2019-04-252-2/+15
| |
* | Add systemd-python to optional dependencies (#4339)Silke Hofstra2019-04-161-0/+2
| | | | | | | | | | | | | | Using systemd-python allows for logging to the systemd journal, as is documented in: `synapse/contrib/systemd/log_config.yaml`. Signed-off-by: Silke Hofstra <silke@slxh.eu>
* | Correct default POSTGRES_USER in Docker README (#4987)*=0=1=4=*2019-04-041-2/+3
| | | | | | Correct default POSTGRES_USER in Docker README
* | Add -p argument for docker run command exampleJurrie Overgoor2019-03-291-0/+1
| | | | | | Signed-off-by: Jurrie Overgoor <1213142+Jurrie@users.noreply.github.com>
* | Fix typo in TLS filenamesJurrie Overgoor2019-03-291-2/+2
|/ | | Signed-off-by: Jurrie Overgoor <1213142+Jurrie@users.noreply.github.com>
* Document the `generate` option for the docker image.Richard van der Hoff2019-03-071-4/+37
|
* Fix debian build dockerfile github/release-v0.99.1 release-v0.99.1Richard van der Hoff2019-02-141-1/+5
| | | | Make sure it refreshes the apt cache before trying to install stuff
* Remove redundant entries from docker configRichard van der Hoff2019-02-112-24/+1
| | | | | * no_tls is now redundant (#4613) * we don't need a dummy cert any more (#4618)
* Enable ACME support in the docker image (#4566)Richard van der Hoff2019-02-055-41/+115
| | | | | | | Also: * Fix wrapping in docker readme * Clean up some docs on the docker image * a workaround for #4554
* Docker: only copy what we need to the build image (#4562)Richard van der Hoff2019-02-051-1/+4
| | | | | | | | | | There are two reasons this is a good thing: * first, it means that you don't end up with stuff kicking around your working copy ending up in the build image by mistake (which can upset the pip install process) * second: it means that the docker image cache is more effective, and we can reuse docker images when iterating on the docker stuff.
* Require ECDH key exchange & remove dh_params (#4429)Amber Brown2019-01-221-1/+0
| | | * remove dh_params and set better cipher string
* Rewrite build_debian_packages github/release-v0.34.1.1 release-v0.34.1.1Richard van der Hoff2019-01-121-46/+0
| | | | Rewrite this in python so that it can be run in parallel.
* fix docker build to install optional deps github/release-v0.34.1 release-v0.34.1Richard van der Hoff2019-01-091-3/+1
|
* Install the optional dependencies into the debian package (#4325)Richard van der Hoff2019-01-022-18/+33
| | | | | | | | since #4298, the optional dependencies are no longer installed with a simple `pip install .`, which meant that they were not being included in the debian package. The easy fix to that is dh_virtualenv --extras, but that needs dh_virtualenv 1.1...
* buster is a thingRichard van der Hoff2018-12-201-1/+8
|
* Debian packaging via dh_virtualenv (#4285)Richard van der Hoff2018-12-203-0/+115
|
* Remove obsolete settings from docker homeserver.yamlRichard van der Hoff2018-12-041-3/+1
| | | | These aren't used, because we have a `log_config` setting.
* Remove riot.im from the list of trusted Identity Servers in the default ↵Neil Johnson2018-11-201-1/+3
| | | | configuration (#4207)
* Make Docker image listening on ipv6 as well as ipv4Cédric Laudrel2018-10-251-2/+2
| | | | Signed-off-by: Cédric Laudrel <dek@iono.me>
* Ship the email templates as package_dataRichard van der Hoff2018-10-171-1/+3
| | | | | | | | | | move the example email templates into the synapse package so that they can be used as package data, which should mean that all of the packaging mechanisms (pip, docker, debian, arch, etc) should now come with the example templates. In order to grandfather in people who relied on the templates being in the old place, check for that situation and fall back to using the defaults if the templates directory does not exist.
* Further reduce the size of the docker image (#3972)Richard van der Hoff2018-10-011-25/+46
| | | | Rewrite the dockerfile as a multistage build: this means we can get rid of a whole load of cruft which we don't need.
* Update DockerfileJan Christian Grünhage2018-09-251-1/+0
|
* make python 3 work in the docker containerJan Christian Grünhage2018-09-202-2/+5
|
* Add a way to run tests in PostgreSQL in Docker (#3699)Amber Brown2018-09-202-0/+32
|
* Added description for "SYNAPSE_MAX_UPLOAD_SIZE" variable.Simon Dwyer2018-09-161-1/+1
|
* Added description for "SYNAPSE_MAX_UPLOAD_SIZE" variable.Simon Dwyer2018-09-161-0/+1
|
* Added 'MAX_UPLOAD_SIZE' variable and set default to "10M"Simon Dwyer2018-09-161-1/+1
|
* add runtime dependenciesJan Christian Grünhage2018-09-101-5/+10
|
* Move COPY before RUN and merge RUNsMathijs van Gorcum2018-09-101-5/+5
|
* Remove build requirements after buildingMathijs van Gorcum2018-09-101-2/+3
|
* update docker base-image to alpine 3.8Jan Christian Grünhage2018-08-091-1/+1
|
* Update README.mdMichael Kaye2018-08-031-1/+1
| | | Link to contrib/docker
* Address SPAG issuesMichael Kaye2018-08-021-2/+4
|
* Refactor docker locations and README.Michael Kaye2018-08-025-0/+471
This addresses #3224