| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
| |
Follow on from #17037
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
when it's already set (#16985)
I have a use case where I'd like the Synapse image to start up a
postgres instance that I can use, but don't want to force Synapse to use
postgres as well.
This commit prevents postgres from being started when it has already
been explicitly enabled elsewhere.
|
| |
|
|
|
|
|
|
|
|
|
| |
Background: we have a `matrixdotorg/synapse-workers` docker image, which
is intended for running multiple workers within the same container. That
image includes a `prefix-log` script which, for each line printed to
stdout or stderr by one of the processes, prepends the name of the
process.
This commit disables buffering in that script, so that lines are logged
quickly after they are printed. This makes it much easier to understand
the output, since they then come out in a natural order.
|
| | |
|
| |
|
|
|
| |
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pulled out of #16803 since the drive-by cleanup was maybe not as
drive-by as I had hoped.
<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->
<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->
Original commit schedule, with full messages:
<ol>
<li>
Add a --generate-only option
</li>
</ol>
---------
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* complement: enable dirty runs
* Add changelog
* Set a low connpool limit when running in Complement
Dirty runs can cause many containers to be running concurrently,
which seems to easily exhaust resources on the host. The increased
speedup from dirty runs also seems to use more db connections on
workers, which are misconfigured currently to have
`SUM(workers * cp_max) > max_connections`, causing
```
FATAL: sorry, too many clients already
```
which results in tests failing.
* Try p=2 concurrency to restrict slowness of servers which causes partial state join tests to flake
* Debug logging
* Only run flakey tests
* Only adjust connection pool limits in worker mode
* Move cp vars to somewhere where they get executed in CI
* Move cp values back to where they actually work
* Debug logging
* Try p=1 to see if this makes worker mode happier
* Remove debug logging
|
| |\ |
|
| | | |
|
| | |
| |
| | |
Enable additional checks & clean-up unneeded configuration.
|
| |/ |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at.
---------
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
|
| |
|
| |
Fix https://github.com/matrix-org/synapse/issues/15836
|
| |
|
|
|
|
|
|
|
|
|
|
| |
into existing rooms (#15748)
Context for why we're removing the implementation:
- https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010
- https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734
Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737
Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Bumps library/redis from 6-bullseye to 7-bullseye.
---
updated-dependencies:
- dependency-name: library/redis
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
| |
|
|
|
| |
See https://github.com/matrix-org/synapse/pull/14095#discussion_r990335492
This is useful because when see that a relevant event is an `outlier` or `soft-failed`, then that's a good unexpected indicator explaining why it's not showing up. `filter_events_for_client` is used in `/sync`, `/messages`, `/context` which are all common end-to-end assertion touch points (also notifications, relations).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fully qualified docker image names for the main Dockerfile and Complement related.
* Fully qualified docker image names for Dockerfiles associated with building Debian release artifacts.
This one is harder and is separate from the other commit in case it wasn't correct or was unwanted. I decided to
do the expansion on the docker images in the Dockerfile itself, instead of the various source places that build
which distribution that is selected, as it would have been more invasive with the scripts breaking up the string
for tagging and such. This one is untested.
* Changelog
* Update docker/Dockerfile-workers
* Update docker/complement/Dockerfile
---------
Co-authored-by: reivilibre <olivier@librepush.net>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Add master to the instance_map as part of Complement, have ReplicationEndpoint look at instance_map for master.
* Fix typo in drive by.
* Remove unnecessary worker_replication_* bits from unit tests and add master to instance_map(hopefully in the right place)
* Several updates:
1. Switch from master to main for naming the main process in the instance_map. Add useful constants for easier adjustment of names in the future.
2. Add backwards compatibility for worker_replication_* to allow time to transition to new style. Make sure to prioritize declaring main directly on the instance_map.
3. Clean up old comments/commented out code.
4. Adjust unit tests to match with new code.
5. Adjust Complement setup infrastructure to only add main to the instance_map if workers are used and remove now unused options from the worker.yaml template.
* Initial Docs upload
* Changelog
* Missed some commented out code that can go now
* Remove TODO comment that no longer holds true.
* Fix links in docs
* More docs
* Remove debug logging
* Apply suggestions from code review
Co-authored-by: reivilibre <olivier@librepush.net>
* Apply suggestions from code review
Co-authored-by: reivilibre <olivier@librepush.net>
* Update version to latest, include completeish before/after examples in upgrade notes.
* Fix up and docs too
---------
Co-authored-by: reivilibre <olivier@librepush.net>
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* Enable `directory`
* move to worker store
* newsfile
* disable `ClientDirectoryListServer` and `ClientAppserviceDirectoryListServer` for workers
|
| | |
|
| | |
|
| |
|
|
| |
compatible with older versions of Python. (#15275)
|
| |
|
|
| |
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
| |
* Install rust during Stage 0 of docker build
Thanks to @atomdmac for spotting the fix.
Fixes #15179.
* Changelog
|
| | |
|
| |
|
|
|
| |
* Complement: Spell out when config isn't generated
* Changelog
|
| | |
|
| |
|
| |
Signed-off-by: Jason Little realtyem@gmail.com
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Enable Complement tests for Faster Remote Room Joins on worker-mode
* (dangerous) Add an override to allow Complement to use FRRJ under workers
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Fix race where we didn't send out replication notification
* MORE HACKS
* Fix get_un_partial_stated_rooms_token to take instance_name
* Fix bad merge
* Remove warning
* Correctly advance un_partial_stated_room_stream
* Fix merge
* Add another notify_replication
* Fixups
* Create a separate ReplicationNotifier
* Fix test
* Fix portdb
* Create a separate ReplicationNotifier
* Fix test
* Fix portdb
* Fix presence test
* Newsfile
* Apply suggestions from code review
* Update changelog.d/14752.misc
Co-authored-by: Erik Johnston <erik@matrix.org>
* lint
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
|
|
|
|
| |
Closes https://github.com/matrix-org/synapse/issues/13234
Signed-off-by: Katia Esposito <1695469+katlol@users.noreply.github.com>
Signed-off-by: Katia Esposito <1695469+katlol@users.noreply.github.com>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Upgrade to new lockfile format
Now requires poetry >= 1.2.2 to read and poetry >= 1.3.0 to write.
Cheat sheet:
```
poetry --version
poetry show > scratch/before
pipx upgrade poetry
poetry --version
poetry show > scratch/after
diff scratch{before,after} && echo "no change!"
```
* Use Poetry 1.3.2 when reading or writing lockfile
* Remove unneeded(?) poetry dep for cibuildwheel
* Update docs
* Remove redundant call to setup-python
* Remove outdated comments related to Poetry 1.x
* Remove outdated docs line
was fixed in #13082
* Minor improvements to poetry cheat sheet
* Invoke setup-python-poetry with explicit version
Not sure about this. It's hardcoding versions everywhere.
* Changelog
* Check the lockfile is version 2.0
Might one day incorporate other checks like #14742
* Typo fixes, thanks Sean
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| |
|
|
|
| |
Serving partial join responses is no longer experimental. They will only be served under the stable identifier if the the undocumented config flag experimental.msc3706_enabled is set to true.
Synapse continues to request a partial join only if the undocumented config flag experimental.faster_joins is set to true; this setting remains present and unaffected.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
Fixes #14704.
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
|
| |
|
|
|
|
|
| |
Fixes #13655
This change uses ICU (International Components for Unicode) to improve boundary detection in user search.
This change also adds a new dependency on libicu-dev and pkg-config for the Debian packages, which are available in all supported distros.
|
| |
|
|
|
| |
Synapse for faster turn-around times whilst developing iteratively. (#14548)
Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
|
| |
|
| |
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
|
| |
|
|
|
|
|
|
| |
Fix https://github.com/matrix-org/synapse/issues/14390
- Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`
- Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`
Complement test changes: https://github.com/matrix-org/complement/pull/559
|
| | |
|
| |
|
|
| |
Dockerfile-workers. (#14294)
|
| |
|
| |
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
command line. (#14324)
* Expose getting SYNAPSE_WORKER_TYPES from external, allowing override of workers requested.
* Add WORKER_TYPES variable option to complement.sh script that passes requested workers into start_for_complement.sh entrypoint.
* Update docs to reflect this new ability.
* Changelog
* Don't rely on soft wrapping to format long strings
Good idea dklimpel. Thanks for catching that.
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
* Small nits just noticed in docs.
* Fixup new line in docs.
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
(#14195)
Also update `subprocess.check_output` to the slightly newer `subprocess.run`.
Signed-off-by: Jason Little <realtyem@gmail.com>
|
| | |
|
| | |
|
| |\ |
|
| | |
| |
| | |
Co-authored-by: David Robertson <davidr@element.io>
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.
This also marks the API as safe for workers since it is a read-only
API.
|
| | |
| |
| | |
Co-authored-by: Eric Eastwood <erice@element.io>
|
| | | |
|
| | | |
|
| |/
|
|
|
| |
Have nginx send its logs to stderr/out, so that we can debug
https://github.com/matrix-org/synapse/issues/13334.
|
| |
|
|
| |
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
| |
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
|
| |
|
|
|
|
|
| |
Doing so in the base postgres image doesn't work with buildah because
changes in a declared VOLUME in the Dockerfile is supposed to be
discarded, cf https://docs.docker.com/engine/reference/builder/#volume
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
|
| | |
|
| | |
|
| |
|
|
| |
image. (#13515)
|
| |
|
| |
Part of my continuing quest to make the docker images build quicker: copy nginx and redis in from base docker images, rather than apt installing each time.
|
| | |
|
| |
|
| |
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
| |
|
| |
Co-authored-by: David Robertson <davidr@element.io>
|
| | |
|
| | |
|
| |
|
|
|
|
| |
When building the docker images for complement testing, copy a preinstalled
complement over from a base image, rather than apt installing it. This avoids
network traffic and is much faster.
|
| |
|
|
| |
environment when using `complement.sh`. (#13152)
|
| | |
|
| | |
|
| |
|
|
|
| |
in CPU-constrained environments. (#13127)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
|
| |
in Synapse workers' Supervisord blocks. (#13054)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
| |
tests. (#13071)
|
| |
|
|
|
| |
If no database is configured explicitly, use sqlite.
This means that you don't have to pass any variables into the image.
|
| | |
|
| |
|
|
| |
(#13069)
|
| |
|
|
| |
to read. (#13058)
|
| |
|
|
|
| |
image. (#12881)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
|
| | |
|
| |
|
|
|
| |
Complement. (#12826)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
| |
shared registration secret. (#12819)
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
* Bump the HS startup timeout
* Log prefixes for more processes
* Bump the overall timeout
|
| |
|
|
| |
`notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452)
|
| |
|
|
|
| |
(#12480)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
| |
Co-authored-by: David Robertson <davidr@element.io>
|
| | |
|
| |
|
|
|
|
| |
When we join a room via the faster-joins mechanism, we end up with "partial
state" at some points on the event DAG. Many parts of the codebase need to
wait for the full state to load. So, we implement a mechanism to keep track of
which events have partial state, and wait for them to be fully-populated.
|
| | |
|
| |
|
|
| |
This is an attempt to reduce the rebuild time. In short, we reduce the amount
of stuff that the dockerfile installs, so as to give a faster startup.
|
| |
|
| |
... with a bit more verbosity.
|
| |
|
|
|
|
|
|
|
|
|
| |
When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example:
```
2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches
```
- note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading.
To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(#12439)
The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.
When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true
> ```
> #15 9.204 Collecting Twisted[tls]>=18.7.0
> #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
> #15 9.205 Twisted[tls]>=18.7.0 from https://files.pythonhosted.org/packages/db/99/38622ff95bb740bcc991f548eb46295bba62fcb6e907db1987c4d92edd09/Twisted-22.4.0-py3-none-any.whl#sha256=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
> #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```
The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.
(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
|
| | |
|
| |
|
| |
This wasn't quite generating the right thing.
|
| |
|
|
|
|
| |
Fixes matrix-org/complement#330 (or it will, once we remove the old files).
It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
|
| |
|
|
| |
Signed-off-by: Jorge Florian <jafn28@gmail.com>
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
(#12118)
* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway.
I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes.
That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data.
So... this wasn't doing anything, except annoying Complement?
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* remove reference in comments to python3.6
* upgrade tox python env in script
* bump python version in example for completeness
* upgrade python version requirement in setup doc
* upgrade necessary python version in __init__.py
* upgrade python version in setup.py
* newsfragment
* drops refs to bionic and replace with focal
* bump refs to postgres 9.6 to 10
* fix hanging ci
* try installing tzdata first
* revert change made in b979f336
* ignore new random mypy error while debugging other error
* fix lint error for temporary workaround
* revert change to install list
* try passing env var
* export debian frontend var?
* move line and add comment
* bump pillow dependency
* bump lxml depenency
* install libjpeg-dev for pillow
* bump automat version to one compatible with py3.8
* add libwebp for pillow
* bump twisted trunk python version
* change suffix of newsfragment
* remove redundant python 3.7 checks
* lint
|
| |
|
| |
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* update Trove classifiers to remove py36
* stop building bionic
* update dh-virtualenv
* newsfragment
* fix newsfragment
* update version refs
* another try at correct tag
* Update changelog
|
| |
|
|
|
| |
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
'trust_identity_servers_for_password_resets' (#11395)
* remove background update code related to deprecated config flag
* changelog entry
* update changelog
* Delete 11394.removal
Duplicate, wrong number
* add no-op background update and change newfragment so it will be consolidated with associated work
* remove unused code
* Remove code associated with deprecated flag from legacy docker dynamic config file
Co-authored-by: reivilibre <oliverw@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Add support for `/_matrix/media/v3` APIs
Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Update `workers.md` to use v3 client and media APIs
Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Add changelog
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* Docker image: avoid changing user during `generate`
The intention was always that the config files get written as the initial user
(normally root) - only the data directory needs to be writable by Synapse. This
got changed in https://github.com/matrix-org/synapse/pull/5970, but that seems
to have been a mistake.
* Avoid changing user if no explicit UID is given
* changelog
|
| |
|
|
|
|
|
|
| |
Use $(...) notation instead of legacy backticked `...`.
https://github.com/koalaman/shellcheck/wiki/SC2006
Signed-off-by: Dan Callahan <danc@element.io>
|
| |
|
|
|
|
|
|
| |
Double quote to prevent globbing and word splitting.
https://github.com/koalaman/shellcheck/wiki/SC2086
Signed-off-by: Dan Callahan <danc@element.io>
|
| |
|
|
|
|
|
|
| |
var is referenced but not assigned.
https://github.com/koalaman/shellcheck/wiki/SC2154
Signed-off-by: Dan Callahan <danc@element.io>
|
| |
|
| |
relative links don't work when it's on dockerhub.
|
| |
|
|
| |
This was due to dh-virtualenv builds being broken due to Shpinx removing
deprecated APIs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Use sytest:bionic. Sytest:latest is two years old (do we want
CI to push out latest at all?) and comes with Python 3.5, which we
explictly no longer support. The script now runs under PostgreSQL 10
as a result.
- Advertise script in the docs
- Move pg testing script to scripts-dev directory
- Write to host as the script's exector, not root
A few changes to make it speedier to re-run the tests:
- Create blank DB in the container, not the script, so we don't have to
`initdb` each time
- Use a named volume to persist the tox environment, so we don't have to
fetch and install a bunch of packages from PyPI each time
Co-authored-by: reivilibre <olivier@librepush.net>
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
| |
This is part of fixing #6116: we want to put RC debs into a different place than release debs, so reprepro has to be able to tell them apart.
|
| |
|
| |
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
| |
|
|
|
| |
Fixes #9778
ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
|
| |
|
| |
Signed-off-by: Chris Castle chris@crc.io
|
| |
|
|
| |
Adds an option to complement.sh to run Synapse in worker
mode (instead of the default monolith mode).
|
| |
|
|
|
| |
Fixes #9970
Signed-off-by: Sergio Miguéns Iglesias lonyelon@lony.xyz
|
| |
|
|
|
| |
* Add healthcheck startup delay by 5secs and reduced interval check to 15s
to reduce waiting time for docker aware edge routers bringing an
instance online
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with:
* A Synapse main process.
* Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime.
* A redis for worker communication.
* A nginx for routing traffic.
* A supervisord to start all worker processes and monitor them if any go down.
Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite.
`configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI.
Huge thanks to @MatMaul for helping get this all working :tada: This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62.
Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
|
| |
|
|
|
|
|
| |
Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894.
I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support.
The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
|
| |
|
|
|
|
| |
They don't make any sense on the intermediate builder image. The final
images needs them to be of use for anyone.
Signed-off-by: Johannes Wienke <languitar@semipol.de>
|
| |
|
|
|
|
|
| |
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
|
| |
|
|
|
|
|
|
|
| |
Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing.
If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘
As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`.
Fix https://github.com/matrix-org/synapse/issues/9364
|
| |\ |
|
| | |
| |
| |
| | |
cryptography (#9697)
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
using /usr/bin/env (#9689)
On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work.
`/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters.
Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
|
| |/
|
|
|
|
| |
Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys
Signed-off-by: Johannes Wienke <languitar@semipol.de>
|
| |
|
|
| |
Co-authored-by: Will Hunt <willh@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
|
| |
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
|
| |
|
|
|
|
|
|
| |
This is needed to build the cryptography library, since it does not
provide wheels for ARMv7.
Fixes #9403
Signed-off-by: Dan Callahan <danc@element.io>
|
| |
|
|
|
|
|
|
| |
Adds note about updating dh-virtualenv once we drop support for Xenial.
We can't update now, because it needs debhelper 12, while Xenial only
backports 10.
Signed-off-by: Dan Callahan <danc@element.io>
|
| |
|
|
| |
homeserver.yaml template (#9157)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Debian package builds were failing for two reasons:
1. Python versions prior to 3.7 throw exceptions when attempting to print
Unicode characters under a "C" locale. (#9076)
2. We depended on `dh-systemd` which no longer exists in Debian Bullseye, but
is necessary in Ubuntu Xenial. (#9073)
Setting `LANG="C.UTF-8"` in the build environment fixes the first issue.
See also: https://bugs.python.org/issue19846
The second issue is a bit trickier. The dh-systemd package was merged into
debhelper version 9.20160709 and a transitional package left in its wake.
The transitional dh-systemd package was removed in Debian Bullseye.
However, Ubuntu Xenial ships an older debhelper, and still needs dh-systemd.
Thus, builds were failing on Bullseye since we depended on a package which had
ceased existing, but we couldn't remove it from the debian/control file and our
build scripts because we still needed it for Ubuntu Xenial.
We can fix the debian/control issue by listing dh-systemd as an alternative to
the newer versions of debhelper. Since dh-systemd declares that it depends on
debhelper, Ubuntu Xenial will select its older dh-systemd which will in turn
pull in its older debhelper, resulting in no change from the status quo. All
other supported releases will satisfy the debhelper dependency constraint and
skip the dh-systemd alternative.
Build scripts were fixed by unconditionally attempting to install dh-systemd on
all releases and suppressing failures.
Once we drop support for Ubuntu Xenial, we can revert most of this commit and
rely on the version constraint on debhelper in debian/control.
Fixes #9076
Fixes #9073
Signed-off-by: Dan Callahan <danc@element.io>
|
| | |
|
| |
|
|
|
|
|
| |
This removes the version pin of the `prometheus_client` dependency, in direct response to #8831. If merged, this will close #8831
As far as I can tell, no other changes are needed, but as I'm no synapse expert, I'm relying heavily on CI and maintainer reviews for this. My very primitive test of synapse with prometheus_client v0.9.0 on my home server didn't bring up any issues, so we'll see what happens.
Signed-off-by: Jordan Bancino
|
| |
|
| |
Short-term fix for https://github.com/matrix-org/synapse/issues/8766.
|
| |
|
|
|
|
|
| |
This bumps us closer to current Python without going all the way to 3.9.
Fixes #8674
Signed-off-by: Dan Callahan <danc@element.io>
|
| | |
|
| |
|
| |
Signed-off-by: Mateusz Przybyłowicz <uamfhq@gmail.com>
|
| |
|
|
|
|
| |
(#8144)
Signed-off-by: Christopher May-Townsend <chris@maytownsend.co.uk>
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
We do this to prevent foot guns. The default config uses a MemoryFilter,
but users are free to change to logging to files directly. If they do
then they have to ensure to set the `filters: [context]` on the right
handler, otherwise records get written with the wrong context.
Instead we move the logic to happen when we generate a record, which is
when we *log* rather than *handle*.
(It's possible to add filters to loggers in the config, however they
don't apply to descendant loggers and so they have to be manually set on
*every* logger used in the code base)
|
| | |
|
| |
|
| |
As mentioned in #7397, switching to a debian base should help with multi-arch work to save time on compiling. This is unashamedly based on #6373, but without the extra functionality. Switch python version back to generic 3.7 to always pull the latest. Essentially, keeping this as small as possible. The image is bigger though unfortunately.
|
| |
|
|
|
| |
* Add libwebp dependency to Dockerfile
Signed-off-by: Juho Vanhanen <juho@vanhanen.io>
|
| |
|
|
|
|
| |
Upstream have merged our changes
(https://github.com/spotify/dh-virtualenv/pull/300), so let's switch back to it
instead of using our fork.
|
| |
|
|
|
|
| |
When we tried to build debs for 1.13.0, the build failed because docker used a
base docker image which had a stale apt cache.
Fixes: #7540
|
| |\ |
|
| | | |
|
| |/ |
|
| |
|
|
|
| |
Lots of people seem to get confused by this mode, and it's been deprecated
since Synapse 1.1.0. It's time for it to go.
|
| | |
|
| |
|
|
|
|
| |
* Add working build command for docker image
* Add changelog
|
| | |
|
| |\ |
|
| | | |
|
| |/
|
| |
Signed-off-by: Tobia De Koninck <LEDfan@users.noreply.github.com>
|
| |
|
|
|
|
|
|
| |
Turns out that loggers that are instantiated before the config is loaded get
turned off.
Also bring the logging config that is generated by --generate-config into line.
Fixes #6194.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Allow passing SYNAPSE_WORKER envvar
* changelog.d
* Document SYNAPSE_WORKER.
Attempting to imply that you don't need to change this default
unless you're in worker mode.
Also aware that there's a bigger problem of attempting to document
a complete working configuration of workers using docker, as we
currently only document to use `synctl` for worker mode, and synctl
doesn't work that way in docker.
|
| |
|
|
|
|
|
|
| |
(#5970)
Adjust su-exec to only be used if needed.
If UID == getuid() and GID == getgid() then we do not need to su-exec, and chmod will not work.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
* Fix debian packages for sid being called buster.
I don't know why the sid images return buster as its codename in
`lsb_release` but it does, so lets just grab the codename from the
distro we pass into dockerfile
* Newsfile
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is basically a contrived way of adding a `Recommends` on `libpq5`, to fix #5653.
The way this is supposed to happen in debhelper is to run
`dh_shlibdeps`, which in turn runs `dpkg-shlibdeps`, which spits things out
into `debian/<package>.substvars` whence they can later be included by
`control`.
Previously, we had disabled `dh_shlibdeps`, mostly because `dpkg-shlibdeps`
gets confused about PIL's interdependent objects, but that's not really the
right thing to do and there is another way to work around that.
Since we don't always use postgres, we don't necessarily want a hard Depends on
libpq5, so I've actually ended up adding an explicit invocation of
`dpkg-shlibdeps` for `psycopg2`.
I've also updated the build-depends list for the package, which was missing a
couple of entries.
|
| |
|
|
|
|
|
| |
image (#5620)
This adds a missing space, without which log lines appear uglier.
Signed-off-by: Slavi Pantaleev <slavi@devture.com>
|
| |
|
|
|
|
|
|
|
| |
Alpine Linux 3.8 is still supported, but it seems like
it's quite outdated now.
While Python should be the same on both, all other libraries, etc.,
are much newer in Alpine 3.9 and 3.10.
Signed-off-by: Slavi Pantaleev <slavi@devture.com>
|
| | |
|
| |
|
|
|
| |
Removes any `SMTP_*` docker container environment variables from having any effect on the default config.
Fixes https://github.com/matrix-org/synapse/issues/5430
|
| |
|
|
| |
Signed-off-by: Amir Zarrinkafsh <nightah@me.com>
|
| |
|
| |
... to help people escape env var hell
|
| |
|
| |
There's not much point in binding to localhost when it's in a docker container.
|
| |
|
|
|
| |
This is mostly a documentation change, but also adds a default value for
SYNAPSE_CONFIG_PATH, so that running from the generated config is the default,
and will Just Work provided your config is in the right place.
|
| |\
| |
| | |
Docker: generate our own log config
|
| | |
| |
| |
| |
| |
| | |
When running under docker, we want to use docker's own logging stuff rather
than losing the logs somewhere on the container's filesystem, so let's use log
configs that spit logs out to stdout instead.
|
| | | |
|
| |\|
| |
| | |
Docker image: add support for SYNAPSE_DATA_DIR parameter
|
| | |
| |
| |
| | |
Fixes #4830.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
We don't want to generate any missing configs when running from a precanned
config.
(There's a strong argument that we don't want to do this at all, since
generating a new signing key on each invocation sounds disasterous, but I don't
fancy unpicking that for now.)
|
| |/ |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
... and inline generate_secrets
|
| |
|
| |
Fixes #3370.
|
| |
|
| |
Python 3.7 is apparently faster than 3.6, and should be mature enough.
|
| | |
|
| |
|
|
| |
Fixes #5467.
|
| | |
|
| |
|
| |
because this file is reproduced on dockerhub and relative paths don't work
|
| |
|
| |
Add missing backslash
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
psycopg 2.8 is now out, which means that the C library gets built from source,
so we now need libpq-dev when building.
Turns out the need for this package is already documented in
docs/postgres.rst.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
Using systemd-python allows for logging to the systemd journal,
as is documented in: `synapse/contrib/systemd/log_config.yaml`.
Signed-off-by: Silke Hofstra <silke@slxh.eu>
|
| | |
| |
| | |
Correct default POSTGRES_USER in Docker README
|
| | |
| |
| | |
Signed-off-by: Jurrie Overgoor <1213142+Jurrie@users.noreply.github.com>
|
| |/
|
| |
Signed-off-by: Jurrie Overgoor <1213142+Jurrie@users.noreply.github.com>
|
| | |
|
| |
|
|
| |
Make sure it refreshes the apt cache before trying to install stuff
|
| |
|
|
|
| |
* no_tls is now redundant (#4613)
* we don't need a dummy cert any more (#4618)
|
| |
|
|
|
|
|
| |
Also:
* Fix wrapping in docker readme
* Clean up some docs on the docker image
* a workaround for #4554
|
| |
|
|
|
|
|
|
|
|
| |
There are two reasons this is a good thing:
* first, it means that you don't end up with stuff kicking around your working
copy ending up in the build image by mistake (which can upset the pip
install process)
* second: it means that the docker image cache is more effective, and we can
reuse docker images when iterating on the docker stuff.
|
| |
|
| |
* remove dh_params and set better cipher string
|
| |
|
|
| |
Rewrite this in python so that it can be run in parallel.
|
| | |
|
| |
|
|
|
|
|
|
| |
since #4298, the optional dependencies are no longer installed with a simple
`pip install .`, which meant that they were not being included in the debian
package.
The easy fix to that is dh_virtualenv --extras, but that needs dh_virtualenv
1.1...
|
| | |
|
| | |
|
| |
|
|
| |
These aren't used, because we have a `log_config` setting.
|
| |
|
|
| |
configuration (#4207)
|
| |
|
|
| |
Signed-off-by: Cédric Laudrel <dek@iono.me>
|
| |
|
|
|
|
|
|
|
|
| |
move the example email templates into the synapse package so that they can be
used as package data, which should mean that all of the packaging mechanisms
(pip, docker, debian, arch, etc) should now come with the example templates.
In order to grandfather in people who relied on the templates being in the old
place, check for that situation and fall back to using the defaults if the
templates directory does not exist.
|
| |
|
|
| |
Rewrite the dockerfile as a multistage build: this means we can get rid of a whole load of cruft which we don't need.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
| |
Link to contrib/docker
|
