summary refs log tree commit diff
path: root/docker/Dockerfile-pgtests (unfollow)
Commit message (Collapse)AuthorFilesLines
2021-04-22Note regression was in 1.32.0 and 1.32.1Andrew Morgan1-2/+2
2021-04-21Update dates in changelogsAndrew Morgan2-2/+2
2021-04-211.32.2Andrew Morgan4-2/+18
2021-04-21Note LoggingContext signature change incompatibility in 1.32.0 (#9859) github/release-v1.32.1 release-v1.32.1Andrew Morgan2-9/+16
1.32.0 also introduced an incompatibility with Synapse modules that make use of `synapse.logging.context.LoggingContext`, such as [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider). This PR adds a note to the 1.32.0 changelog and upgrade notes about it.
2021-04-21Make LoggingContext's name optional (#9857)Richard van der Hoff2-3/+13
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
2021-04-21Clarify 1.32.0/1 changelog and upgrade notes v1.32.1Andrew Morgan2-8/+7
2021-04-21Add regression notes to CHANGES.md; fix link in 1.32.0 changelogAndrew Morgan1-2/+6
2021-04-21Add link to fixing prometheus to 1.32.0 upgrade notes; 1.32.1 has a fixAndrew Morgan2-2/+6
2021-04-21Fix typo in link to regression in 1.32.0 upgrade notesAndrew Morgan1-1/+1
2021-04-211.32.1Andrew Morgan4-2/+16
2021-04-21Mention Prometheus metrics regression in v1.32.0Andrew Morgan2-0/+15
2021-04-21Stop BackgroundProcessLoggingContext making new prometheus timeseries (#9854)Richard van der Hoff3-5/+18
This undoes part of b076bc276e881b262048307b6a226061d96c4a8d.
2021-04-20Further tweaking on gpg signing key noticeAndrew Morgan1-2/+5
2021-04-20Add note about expired Debian gpg signing keys to CHANGES.mdAndrew Morgan1-0/+6
2021-04-20Update v1.32.0 changelog. It's m.login.application_service, not plural v1.32.0Andrew Morgan1-1/+1
2021-04-20 1.32.0Andrew Morgan4-6/+18
2021-04-20Add Application Service registration type requirement + py35, pg95 ↵Andrew Morgan1-0/+18
deprecation notices to v1.32.0 upgrade notes (#9849) Fixes https://github.com/matrix-org/synapse/issues/9846. Adds important removal information from the top of https://github.com/matrix-org/synapse/releases/tag/v1.32.0rc1 into UPGRADE.rst.
2021-04-20Always use the name as the log ID. (#9829)Patrick Cloke8-34/+26
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-13Update changelog for v1.32.0 v1.32.0rc1Andrew Morgan1-1/+1
2021-04-13 1.32.0rc1Andrew Morgan37-36/+73
2021-04-13Add release helper script (#9713)Erik Johnston3-0/+252
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-04-13Bump black configuration to target py36 (#9781)Dan Callahan11-15/+16
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13Add an admin API to manage ratelimit for a specific user (#9648)Dirk Klimpel6-6/+573
2021-04-12Drop Python 3.5 from Trove classifier metadata. (#9782)Dan Callahan2-1/+1
* Drop Python 3.5 from Trove classifier metadata. Signed-off-by: Dan Callahan <danc@element.io>
2021-04-12Add option to skip unit tests when building debs (#9793)Dan Callahan4-13/+34
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-12Require AppserviceRegistrationType (#9548)Will Hunt5-23/+60
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
2021-04-09Use mock from the stdlib. (#9772)Patrick Cloke82-126/+86
2021-04-09Fix duplicate logging of exceptions in transaction processing (#9780)Richard van der Hoff2-7/+4
There's no point logging this twice.
2021-04-09Enable complement tests for MSC2946. (#9771)Patrick Cloke2-1/+2
By providing the additional build tag for `msc2946`.
2021-04-09Proof of concept for GitHub Actions (#9661)Dan Callahan2-0/+323
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-09Remove old admin API `GET /_synapse/admin/v1/users/<user_id>` (#9401)Dirk Klimpel5-27/+16
Related: #8334 Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25) `GET /_synapse/admin/v1/users/<user_id>` has no - unit tests - documentation API in v2 is available (#5925 - 12/2019, v1.7.0). API is misleading. It expects `user_id` and returns a list of all users. Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-04-08Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong38-113/+224
Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08remove unused param on `make_tuple_comparison_clause`Richard van der Hoff5-9/+3
2021-04-08Fix sharded federation sender sometimes using 100% CPU.Erik Johnston2-2/+5
We pull all destinations requiring catchup from the DB in batches. However, if all those destinations get filtered out (due to the federation sender being sharded), then the `last_processed` destination doesn't get updated, and we keep requesting the same set repeatedly.
2021-04-08update test_old_deps scriptRichard van der Hoff1-1/+1
2021-04-08Update tox.ini to remove py35Richard van der Hoff1-8/+6
2021-04-08drop support for stretch and xenialRichard van der Hoff1-2/+0
2021-04-08Drop support for sqlite<3.22 as wellRichard van der Hoff6-92/+14
2021-04-08Require py36 and Postgres 9.6Richard van der Hoff3-3/+4
2021-04-08unpin olddeps build from py36Richard van der Hoff2-5/+5
2021-04-08Fix incompatibility with tox 2.5Richard van der Hoff2-6/+13
Apparently on tox 2.5, `usedevelop` overrides `skip_install`, so we end up trying to install the full dependencies even for the `-old` environment.
2021-04-08Put opencontainers labels to the final image (#9765)Johannes Wienke2-5/+6
They don't make any sense on the intermediate builder image. The final images needs them to be of use for anyone. Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-04-08MSC3083: Check for space membership during a local join of restricted rooms. ↵Patrick Cloke3-2/+76
(#9735) When joining a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
2021-04-08Record more information into structured logs. (#9654)Patrick Cloke10-88/+255
Records additional request information into the structured logs, e.g. the requester, IP address, etc.
2021-04-06Don't report anything from GaugeBucketCollector metrics until data is ↵Andrew Morgan2-3/+14
present (#8926) This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.
2021-04-06Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan14-64/+1282
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06Add type hints to expiring cache. (#9730)Patrick Cloke8-54/+65
2021-04-06Fix reported bugbear: too broad exception assertion (#9753)Andrew Morgan2-2/+4
2021-04-06Remove outdated constraint on remote_media_cache_thumbnails (#9725)Richard van der Hoff3-3/+41
The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key` constraint is superceded by `remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds `thumbnail_method` to the unique key). PR #7124 made an attempt to remove the old constraint, but got the name wrong, so it didn't work. Here we update the bg update and rerun it. Fixes #8649.
2021-04-06 1.31.0 v1.31.0 github/release-v1.31.0 release-v1.31.0Erik Johnston6-7/+27
2021-04-06Add deprecation policy doc (#9723)Erik Johnston4-2/+46
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-04-06Fix version for bugbear (#9734)Jonathan de Jong2-1/+2
2021-04-06Add type hints to the federation handler and server. (#9743)Patrick Cloke4-95/+97
2021-04-06Convert storage test cases to HomeserverTestCase. (#9736)Patrick Cloke11-499/+265
2021-04-05Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong10-11/+21
2021-04-02Fix version for bugbear (#9734)Jonathan de Jong2-1/+2
2021-04-01Improve tracing for to device messages (#9686)Erik Johnston7-19/+102
2021-04-01Add `order_by` to list user admin API (#9691)Dirk Klimpel6-31/+248
2021-03-31Add an experimental room version to support restricted join rules. (#9717)Patrick Cloke6-11/+297
Per MSC3083.
2021-03-31Revert "Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)" (#9720)Patrick Cloke2-1/+2
2021-03-31Make sample config allowed_local_3pids regex stricter. (#9719)Denis Kasak3-4/+5
The regex should be terminated so that subdomain matches of another domain are not accepted. Just ensuring that someone doesn't shoot themselves in the foot by copying our example. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-03-31Deprecate imp (#9718)Cristina2-3/+9
Fixes #9642. Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
2021-03-31Rewrite complement.sh (#9685)Andrew Morgan2-11/+39
This PR rewrites the original complement.sh script with a number of improvements: * We can now use a local checkout of Complement (configurable with `COMPLEMENT_DIR`), though the default behaviour still downloads the master branch. * You can now specify a regex of test names to run, or just run all tests. * We now use the Synapse test blacklist tag (so all tests will pass).
2021-03-30Include m.room.create in invite_room_state for Spaces (#9710)Richard van der Hoff2-0/+5
2021-03-30Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff8-43/+144
`room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
2021-03-30Make RateLimiter class check for ratelimit overrides (#9711)Erik Johnston16-154/+241
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited. We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits. Fixes #9663
2021-03-30Update changelog v1.31.0rc1Erik Johnston1-3/+7
2021-03-30 1.31.0rc1Erik Johnston45-44/+65
2021-03-30Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)Andrew Morgan2-1/+2
For it's obvious performance benefits. `dmypy` support landed in #9692.
2021-03-29Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke7-67/+96
2021-03-29Clarify that register_new_matrix_user is present also when installed via ↵blakehawkins2-8/+19
non-pip package (#9074) Signed-off-by: blakehawkins blake.hawkins.11@gmail.com
2021-03-29Add type hints for the federation sender. (#9681)Patrick Cloke7-59/+177
Includes an abstract base class which both the FederationSender and the FederationRemoteSendQueue must implement.
2021-03-29Update the OIDC sample config (#9695)Richard van der Hoff3-62/+7
I've reiterated the advice about using `oidc` to migrate, since I've seen a few people caught by this. I've also removed a couple of the examples as they are duplicating the OIDC documentation, and I think they might be leading people astray.
2021-03-29Fix CI by ignore type for None module import (#9709)Andrew Morgan2-1/+2
2021-03-29Fix `re.Pattern` mypy error on 3.6 (#9703)Jonathan de Jong2-2/+3
2021-03-29Fix the suggested pip incantation for cryptography (#9699)Richard van der Hoff2-2/+16
If you have the wrong version of `cryptography` installed, synapse suggests: ``` To install run: pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6'' ``` However, the use of ' inside '...' doesn't work, so when you run this, you get an error.
2021-03-26Make pip install faster in Docker build for Complement testing (#9610)Eric Eastwood2-43/+42
Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing. If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘 As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`. Fix https://github.com/matrix-org/synapse/issues/9364
2021-03-26Suppress CryptographyDeprecationWarning (#9698)Richard van der Hoff4-14/+26
This warning is somewhat confusing to users, so let's suppress it
2021-03-26Make it possible to use dmypy (#9692)Erik Johnston16-17/+56
Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
2021-03-26Update cahngelog v1.30.1 github/release-v1.30.1 release-v1.30.1Erik Johnston1-3/+2
2021-03-26Update cahngelogErik Johnston1-2/+14
2021-03-26 1.30.1Erik Johnston5-3/+26
2021-03-26Explicitly upgrade openssl in docker file and enforce new version of ↵Erik Johnston4-20/+28
cryptography (#9697)
2021-03-26Preserve host in example apache config (#9696)Paul Tötterman2-5/+11
Fixes redirect loop Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2021-03-25Use interpreter from $PATH instead of absolute paths in various scripts ↵Quentin Gliech17-17/+18
using /usr/bin/env (#9689) On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work. `/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-03-25platform specific prerequisites in source install (#9667)Serban Constantin2-7/+9
Make it clearer in the source install step that the platform specific prerequisites must be installed first. Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
2021-03-25Add a storage method for returning all current presence from all users (#9650)Andrew Morgan3-3/+69
Split off from https://github.com/matrix-org/synapse/pull/9491 Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133 Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
2021-03-24Fixed undefined variable error in catchup (#9664)Erik Johnston2-0/+3
Broke in #9640 Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-24Fix typo in changelog.Patrick Cloke2-2/+2
2021-03-24Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong8-8/+9
2021-03-24Spaces summary: call out to other servers (#9653)Richard van der Hoff4-27/+324
When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data. Fixes: #9447
2021-03-24docs: fallback/web endpoint does not appear to be mounted on workers (#9679)Ben Banfield-Zanin2-2/+2
2021-03-24Bump mypy-zope to 0.2.13. (#9678)Patrick Cloke2-1/+2
This fixes an error ("Cannot determine consistent method resolution order (MRO)") when running mypy with a cache.
2021-03-24Add type hints to misc. files. (#9676)Patrick Cloke6-54/+57
2021-03-24Add a type hints for service notices to the HomeServer object. (#9675)Patrick Cloke11-40/+52
2021-03-23Increase default join burst ratelimiting (#9674)Erik Johnston3-6/+7
It's legitimate behaviour to try and join a bunch of rooms at once.
2021-03-23Fix federation stall on concurrent access errors (#9639)Jonathan de Jong2-36/+10
2021-03-23Federation API for Space summary (#9652)Richard van der Hoff3-54/+197
Builds on the work done in #9643 to add a federation API for space summaries. There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
2021-03-23Import HomeServer from the proper module. (#9665)Patrick Cloke59-58/+59
2021-03-22Allow providing credentials to HTTPS_PROXY (#9657)Andrew Morgan4-34/+184
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.
2021-03-22Include opencontainers labels in Docker image (#9612)Johannes Wienke2-0/+6
Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-03-22Fixed code misc. quality issues (#9649)Ankit Dobhal4-3/+4
- Merge 'isinstance' calls. - Remove unnecessary dict call outside of comprehension. - Use 'sys.exit()' calls.
2021-03-22 1.30.0 v1.30.0 github/release-v1.30.0 release-v1.30.0Erik Johnston3-3/+16
2021-03-19Incorporate reviewBrendan Abolivier2-2/+2
2021-03-19Fix lintBrendan Abolivier2-11/+10
2021-03-19fix mypyRichard van der Hoff1-4/+7
2021-03-18federation_client: handle inline signing_keys in hs.yaml (#9647)Richard van der Hoff2-54/+18
2021-03-18federation_client: stop adding URL prefix (#9645)Richard van der Hoff2-2/+3
2021-03-18Fix type-hints from bad merge.Patrick Cloke1-2/+2
2021-03-18Initial spaces summary API (#9643)Richard van der Hoff6-3/+277
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
2021-03-18Move support for MSC3026 behind an experimental flagBrendan Abolivier3-3/+18
2021-03-18Consistently check whether a password may be set for a user. (#9636)Dirk Klimpel5-57/+122
2021-03-18Make federation catchup send last event from any server. (#9640)Erik Johnston4-38/+141
Currently federation catchup will send the last *local* event that we failed to send to the remote. This can cause issues for large rooms where lots of servers have sent events while the remote server was down, as when it comes back up again it'll be flooded with events from various points in the DAG. Instead, let's make it so that all the servers send the most recent events, even if its not theirs. The remote should deduplicate the events, so there shouldn't be much overhead in doing this. Alternatively, the servers could only send local events if they were also extremities and hope that the other server will send the event over, but that is a bit risky.
2021-03-18Implement MSC3026: busy presence stateBrendan Abolivier6-1/+27
2021-03-17Ensure we use a copy of the event content dict before modifying it in ↵Andrew Morgan5-2/+147
serialize_event (#9585) This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](https://github.com/matrix-org/synapse/blob/17cd48fe5171d50da4cb59db647b993168e7dfab/synapse/events/third_party_rules.py#L73-L76) - will raise a 500 if you try to a edit a reply to a message. `serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised. To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
2021-03-17Fix up types for the typing handler. (#9638)Patrick Cloke4-14/+30
By splitting this to two separate methods the callers know what methods they can expect on the handler.
2021-03-17only save remote cross-signing keys if they're different from the current ↵Hubert Chathi2-4/+19
ones (#9634) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-17Fix bad naming of storage function (#9637)Erik Johnston4-4/+7
We had two functions named `get_forward_extremities_for_room` and `get_forward_extremeties_for_room` that took different paramters. We rename one of them to avoid confusion.
2021-03-17Prep work for removing `outlier` from `internal_metadata` (#9411)Richard van der Hoff7-7/+36
* Populate `internal_metadata.outlier` based on `events` table Rather than relying on `outlier` being in the `internal_metadata` column, populate it based on the `events.outlier` column. * Move `outlier` out of InternalMetadata._dict Ultimately, this will allow us to stop writing it to the database. For now, we have to grandfather it back in so as to maintain compatibility with older versions of Synapse.
2021-03-17Add type hints to the room member handler. (#9631)Patrick Cloke5-6/+17
2021-03-16Enable flake8-bugbear, but disable most checks. (#9499)Jonathan de Jong12-10/+29
* Adds B00 to ignored checks. * Fixes remaining issues.
2021-03-16Add SSO attribute requirements for OIDC providers (#9609)Hubbe5-1/+209
Allows limiting who can login using OIDC via the claims made from the IdP.
2021-03-16Return m.change_password.enabled=false if local database is disabled (#9588)Dirk Klimpel4-15/+58
Instead of if the user does not have a password hash. This allows a SSO user to add a password to their account, but only if the local password database is configured.
2021-03-16Fix jemalloc changelog entry wordingAndrew Morgan1-1/+1
2021-03-16Changelog typo v1.30.0rc1Andrew Morgan1-1/+1
2021-03-16Pull up appservice login deprecation noticeAndrew Morgan1-6/+7
2021-03-161.30.0rc1Andrew Morgan46-45/+71
2021-03-16Pass SSO IdP information to spam checker's registration function (#9626)Andrew Morgan5-6/+67
Fixes https://github.com/matrix-org/synapse/issues/9572 When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time. This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
2021-03-16Install jemalloc in docker image (#8553)Mathieu Velten4-2/+17
Co-authored-by: Will Hunt <willh@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
2021-03-16Handle an empty cookie as an invalid macaroon. (#9620)Patrick Cloke2-1/+3
* Handle an empty cookie as an invalid macaroon. * Newsfragment
2021-03-16Add support for stable MSC2858 API (#9617)Richard van der Hoff10-28/+88
The stable format uses different brand identifiers, so we need to support two identifiers for each IdP.
2021-03-16Clean up config settings for stats (#9604)Richard van der Hoff4-29/+43
... and complain if people try to turn it off.
2021-03-16Prevent bundling aggregations for state events (#9619)Andrew Morgan3-2/+9
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
2021-03-16Fix Internal Server Error on `GET /saml2/authn_response` (#9623)Richard van der Hoff2-2/+9
* Fix Internal Server Error on `GET /saml2/authn_response` Seems to have been introduced in #8765 (Synapse 1.24.0) * Fix newsfile
2021-03-15Revert requiring a specific version of Twisted for mypy checks. (#9618)Patrick Cloke2-2/+1
2021-03-15Fix remaining mypy issues due to Twisted upgrade. (#9608)Patrick Cloke8-34/+42
2021-03-15Don't go into federation catch up mode so easily (#9561)Erik Johnston5-159/+190
Federation catch up mode is very inefficient if the number of events that the remote server has missed is small, since handling gaps can be very expensive, c.f. #9492. Instead of going into catch up mode whenever we see an error, we instead do so only if we've backed off from trying the remote for more than an hour (the assumption being that in such a case it is more than a transient failure).
2021-03-15Optimise missing prev_event handling (#9601)Richard van der Hoff3-28/+137
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events, we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote server for the state at those prev_events, and then we may need to then ask the remote server for any events in that state which we don't already have, as well as the auth events for those missing state events, so that we can auth them. This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the observation that we are currently loading all of those auth events into memory at the start of the operation, but we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're actually going to need, but it doesn't.) The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about 60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache churn. (NB I've already tripled the size of the cache from its default of 10K). Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different (ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting that we clean it up.
2021-03-12Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke18-119/+187
2021-03-12Reject concurrent transactions (#9597)Richard van der Hoff2-35/+43
If more transactions arrive from an origin while we're still processing the first one, reject them. Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
2021-03-12Improve logging when processing incoming transactions (#9596)Richard van der Hoff3-73/+51
Put the room id in the logcontext, to make it easier to understand what's going on.
2021-03-11Add logging for redis connection setup (#9590)Richard van der Hoff3-1/+39
2021-03-11Add tests for blacklisting reactor/agent. (#9563)Patrick Cloke3-14/+139
2021-03-11Re-Activating account when local passwords are disabled (#9587)Dirk Klimpel2-1/+5
Fixes: #8393
2021-03-10Convert Requester to attrs (#9586)Richard van der Hoff5-35/+37
... because namedtuples suck Fix up a couple of other annotations to keep mypy happy.
2021-03-10Fix the auth provider on the logins metric (#9573)Richard van der Hoff3-18/+33
We either need to pass the auth provider over the replication api, or make sure we report the auth provider on the worker that received the request. I've gone with the latter.
2021-03-10Fix spam checker modules documentation example (#9580)Jason Robinson2-0/+11
Mention that parse_config must exist and note the check_media_file_for_spam method.
2021-03-10Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke5-11/+226
This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
2021-03-10Fix a bug in the background task for purging chain cover. (#9583)Patrick Cloke2-1/+2
2021-03-09Do not ignore the unpaddedbase64 module when type checking. (#9568)Patrick Cloke2-3/+1
2021-03-09Add a background task to purge unused chain IDs. (#9542)Patrick Cloke4-6/+99
This is a companion change to apply the fix in #9498 / 922788c6043138165c025c78effeda87de842bab to previously purged rooms.
2021-03-09Link to the List user's media admin API from media Admin API docs (#9571)Andrew Morgan2-2/+15
Earlier [I was convinced](https://github.com/matrix-org/synapse/issues/9565) that we didn't have an Admin API for listing media uploaded by a user. Foolishly I was looking under the Media Admin API documentation, instead of the User Admin API documentation. I thought it'd be helpful to link to the latter so others don't hit the same dead end :)
2021-03-09JWT OIDC secrets for Sign in with Apple (#9549)Richard van der Hoff11-47/+444
Apple had to be special. They want a client secret which is generated from an EC key. Fixes #9220. Also fixes #9212 while I'm here.
2021-03-09Retry 5xx errors in federation client (#9567)Erik Johnston2-3/+5
Fixes #8915
2021-03-09Fix additional type hints. (#9543)Patrick Cloke9-18/+32
Type hint fixes due to Twisted 21.2.0 adding type hints.
2021-03-09Handle image transparency better when thumbnailing. (#9473)Patrick Cloke3-11/+30
Properly uses RGBA mode for 1- and 8-bit images with transparency (instead of RBG mode).
2021-03-09Add a list of hashes to ignore during git blame. (#9560)Patrick Cloke3-0/+10
The hashes are from commits due to auto-formatting, e.g. running black. git can be configured to use this automatically by running the following: git config blame.ignoreRevsFile .git-blame-ignore-revs
2021-03-09Fixup sample configErik Johnston1-2/+1
After 0764d0c6e575793ca506cf021aff3c4b9e0a5972
2021-03-09Prevent the config-lint script erroring out on any sample_config changes (#9562)Andrew Morgan2-2/+8
I noticed that I'd occasionally have `scripts-dev/lint.sh` fail when messing about with config options in my PR. The script calls `scripts-dev/config-lint.sh`, which attempts some validation on the sample config. It does this by using `sed` to edit the sample_config, and then seeing if the file changed using `git diff`. The problem is: if you changed the sample_config as part of your commit, this script will error regardless. This PR attempts to change the check so that existing, unstaged changes to the sample_config will not cause the script to report an invalid file.
2021-03-09Add logging to ObservableDeferred callbacks (#9523)Jonathan de Jong2-8/+19
2021-03-08quick config comment tweak to clarify allow_profile_lookup_over_federationMatthew Hodgson1-2/+1
2021-03-08Add ResponseCache tests. (#9458)Jonathan de Jong10-20/+156
2021-03-08Warn that /register will soon require a type when called with an access ↵Will Hunt2-0/+7
token (#9559) This notice is giving a heads up to the planned spec compliance fix https://github.com/matrix-org/synapse/pull/9548.
2021-03-08Add type hints to purge room and server notice admin API. (#9520)Dirk Klimpel3-15/+24
2021-03-08Add a basic test for purging rooms. (#9541)Patrick Cloke2-26/+46
Unfortunately this doesn't test re-joining the room since that requires having another homeserver to query over federation, which isn't easily doable in unit tests.
2021-03-08Fixup changelog v1.29.0 github/release-v1.29.0 release-v1.29.0Erik Johnston1-0/+3
2021-03-08 1.29.0Erik Johnston3-5/+11
2021-03-08Create a SynapseReactor type which incorporates the necessary reactor ↵Patrick Cloke8-12/+32
interfaces. (#9528) This helps fix some type hints when running with Twisted 21.2.0.
2021-03-06Update reverse proxy to add OpenBSD relayd example configuration. (#9508)Leo Bärring3-4/+53
Update reverse proxy to add OpenBSD relayd example configuration. Signed-off-by: Leo Bärring <leo.barring@protonmail.com>
2021-03-05Add additional SAML2 upgrade notes (#9550)Ben Banfield-Zanin2-0/+8
2021-03-04Replace `last_*_pdu_age` metrics with timestamps (#9540)Richard van der Hoff4-12/+11
Following the advice at https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since, it's preferable to export unix timestamps, not ages. There doesn't seem to be any particular naming convention for timestamp metrics.
2021-03-04Prometheus metrics for logins and registrations (#9511)Richard van der Hoff4-4/+43
Add prom metrics for number of users successfully registering and logging in, by SSO provider.
2021-03-04Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff13-151/+258
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
2021-03-04Fix link in UPGRADES v1.29.0rc1Erik Johnston1-3/+3
2021-03-04Fix changelogErik Johnston1-6/+3
2021-03-04 1.29.0rc1Erik Johnston42-41/+55
2021-03-03Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke12-61/+96
2021-03-03Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539)Richard van der Hoff2-2/+14
Should fix some remaining warnings
2021-03-03Fix 'rejected_events_metadata' background update (#9537)Erik Johnston2-1/+4
Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
2021-03-03Purge chain cover tables when purging events. (#9498)Patrick Cloke3-10/+38
2021-03-03Add type hints to user admin API. (#9521)Dirk Klimpel4-35/+63
2021-03-03Bump the mypy and mypy-zope versions. (#9529)Patrick Cloke4-3/+4
2021-03-03Make deleting stale pushers a background update (#9536)Erik Johnston3-1/+55
2021-03-03Update nginx reverse-proxy docs (#9512)Richard van der Hoff2-0/+3
Turns out nginx overwrites the Host header by default.
2021-03-03Prevent presence background jobs from running when presence is disabled (#9530)Aaron Raimist2-14/+18
Prevent presence background jobs from running when presence is disabled Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-03-02Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"Patrick Cloke3-35/+3
This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46. This is being backed out due to a regression (#9507) and additional review feedback being provided.
2021-03-02Re-run rejected metadata background update. (#9503)Erik Johnston2-0/+10
It landed in schema version 58 after 59 had been created, causing some servers to not run it. The main effect of was that not all rooms had their chain cover calculated correctly. After the BG updates complete the chain covers will get fixed when a new state event in the affected rooms is received.
2021-03-02Fix SQL delta file taking a long time to run (#9516)Erik Johnston4-5/+60
Fixes #9504
2021-03-02Add type hints to device and event report admin API (#9519)Dirk Klimpel3-16/+40
2021-03-01Fix a bug when a room alias is given to the admin join endpoint (#9506)Patrick Cloke2-58/+75
2021-03-01(Hopefully) stop leaking file descriptors in media repo. (#9497)Patrick Cloke3-38/+85
By consuming the response if the headers imply that the content is too large.
2021-03-01Use the proper Request in type hints. (#9515)Patrick Cloke18-31/+38
This also pins the Twisted version in the mypy job for CI until proper type hints are fixed throughout Synapse.
2021-02-26Allow bytecode again (#9502)Jonathan de Jong8-10/+16
In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again. Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-02-26Add support for no_proxy and case insensitive env variables (#9372)Tim Leung6-64/+114
### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26SSO: redirect to public URL before setting cookies (#9436)Richard van der Hoff7-28/+130
... otherwise, we don't get the cookie back.
2021-02-26Call out the need for an X-Forwarded-Proto in the upgrade notes (#9501)Richard van der Hoff4-1/+28
2021-02-25Test that we require validated email for email pushers (#9496)Erik Johnston3-2/+39
2021-02-25Ensure pushers are deleted for deactivated accounts (#9285)Erik Johnston4-0/+70
2021-02-25Fixup changelog v1.28.0 github/release-v1.28.0 release-v1.28.0Erik Johnston1-1/+1
2021-02-25Fixup changelogErik Johnston1-4/+5
2021-02-25 1.28.0Erik Johnston4-2/+16
2021-02-24Add support for X-Forwarded-Proto (#9472)Richard van der Hoff3-28/+94
rewrite XForwardedForRequest to set `isSecure()` based on `X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
2021-02-24Fix typo in spam checker documentationAndrew Morgan1-1/+1
2021-02-24Add SQL delta for deleting stale pushers (#9479)Erik Johnston2-0/+20
2021-02-24Fix #8518 (sync requests being cached wrongly on timeout) (#9358)Jonathan de Jong3-3/+35
This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-02-24Clean up `ShardedWorkerHandlingConfig` (#9466)Erik Johnston14-63/+128
* Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.
2021-02-24Refactor to ensure we call check_consistency (#9470)Erik Johnston8-28/+72
The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
2021-02-23Add back the deprecated SAML endpoint. (#9474)Patrick Cloke2-1/+7
2021-02-23Add a comment about systemd-python. (#9464)Richard van der Hoff2-0/+4
This confused me for a while.
2021-02-23Include newly added sequences in the port DB script. (#9449)Patrick Cloke4-26/+55
And ensure the consistency of `event_auth_chain_id`.
2021-02-22Fix deleting pushers when using sharded pushers. (#9465)Erik Johnston10-67/+94
2021-02-22Remove vestiges of uploads_path config (#9462)Richard van der Hoff5-4/+1
`uploads_path` was a thing that was never used; most of it was removed in #6628 but a few vestiges remained.
2021-02-22Add an `order_by` field to list users' media admin API. (#8978)Dirk Klimpel5-29/+325
2021-02-22example systemd config: propagate reloads to units (#9463)Richard van der Hoff3-0/+3
It should be possible to reload `synapse.target` to have the reload propagate to all the synapse units.
2021-02-22Remove cache for get_shared_rooms_for_users (#9416)Andrew Morgan3-37/+43
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed. The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately. Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
2021-02-22Clean up the user directory sample config section (#9385)Andrew Morgan3-50/+67
The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format). This PR cleans that up a bit.
2021-02-19Ratelimit cross-user key sharing requests. (#8957)Patrick Cloke9-17/+67
2021-02-19Add testErik Johnston1-3/+66
2021-02-19Add documentation and type hints to parse_duration. (#9432)Patrick Cloke2-2/+16
2021-02-19Fix style checking due to updated black.Patrick Cloke2-3/+5
2021-02-19Update release date. v1.28.0rc1Patrick Cloke1-1/+1
2021-02-19Support not providing an IdP icon when choosing a username. (#9440)Patrick Cloke2-1/+2
2021-02-19Be smarter about which hosts to send presence to when processing room joins ↵Andrew Morgan4-19/+54
(#9402) This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually. --- When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed. It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw. This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence: * If it was a local user join, send that user's latest presence t