| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
|
| |
|
|
|
|
|
| |
I've reiterated the advice about using `oidc` to migrate, since I've seen a few
people caught by this.
I've also removed a couple of the examples as they are duplicating the OIDC
documentation, and I think they might be leading people astray.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
If you have the wrong version of `cryptography` installed, synapse suggests:
```
To install run:
pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6''
```
However, the use of ' inside '...' doesn't work, so when you run this, you get
an error.
|
| |
|
|
|
|
|
|
|
| |
Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing.
If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘
As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`.
Fix https://github.com/matrix-org/synapse/issues/9364
|
| |
|
| |
This warning is somewhat confusing to users, so let's suppress it
|
| |
|
|
|
|
|
|
|
| |
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.
`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
|
| |
|
|
|
|
|
|
|
| |
using /usr/bin/env (#9689)
On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work.
`/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters.
Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
|
| |
|
|
|
|
| |
Make it clearer in the source install step that the platform specific
prerequisites must be installed first.
Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
|
| |
|
|
|
|
|
| |
Split off from https://github.com/matrix-org/synapse/pull/9491
Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133
Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
|
| |
|
|
|
| |
Broke in #9640
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
|
| | |
|
| |
|
|
|
| |
When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data.
Fixes: #9447
|
| | |
|
| |
|
|
| |
This fixes an error ("Cannot determine consistent method resolution order (MRO)")
when running mypy with a cache.
|
| | |
|
| | |
|
| |
|
| |
It's legitimate behaviour to try and join a bunch of rooms at once.
|
| | |
|
| |
|
|
|
| |
Builds on the work done in #9643 to add a federation API for space summaries.
There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
|
| | |
|
| |
|
|
|
|
|
| |
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70
This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.
Also added some type hints.
|
| |
|
|
|
|
| |
Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys
Signed-off-by: Johannes Wienke <languitar@semipol.de>
|
| |
|
|
|
| |
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| | |
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.
Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
|
| |/ |
|
| |
|
|
|
|
|
|
|
| |
serialize_event (#9585)
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](https://github.com/matrix-org/synapse/blob/17cd48fe5171d50da4cb59db647b993168e7dfab/synapse/events/third_party_rules.py#L73-L76) - will raise a 500 if you try to a edit a reply to a message.
`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.
To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
|
| |
|
|
| |
By splitting this to two separate methods the callers know
what methods they can expect on the handler.
|
| |
|
|
|
| |
ones (#9634)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
|
|
| |
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* Populate `internal_metadata.outlier` based on `events` table
Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.
* Move `outlier` out of InternalMetadata._dict
Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
|
| | |
|
| |
|
|
| |
* Adds B00 to ignored checks.
* Fixes remaining issues.
|
| |
|
|
| |
Allows limiting who can login using OIDC via the claims
made from the IdP.
|
| |
|
|
|
| |
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
|
| | |
|
| |
|
|
|
|
|
| |
Fixes https://github.com/matrix-org/synapse/issues/9572
When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.
This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
|
| |
|
|
| |
Co-authored-by: Will Hunt <willh@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
|
|
| |
* Handle an empty cookie as an invalid macaroon.
* Newsfragment
|
| |
|
|
|
| |
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
|
| |
|
| |
... and complain if people try to turn it off.
|
| |
|
| |
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
|
| |
|
|
|
|
|
| |
* Fix Internal Server Error on `GET /saml2/authn_response`
Seems to have been introduced in #8765 (Synapse 1.24.0)
* Fix newsfile
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.
Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.
This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)
The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).
Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
|
| | |
|
| |
|
|
|
|
| |
If more transactions arrive from an origin while we're still processing the
first one, reject them.
Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
|
| |
|
| |
Put the room id in the logcontext, to make it easier to understand what's going on.
|
| | |
|
| | |
|
| |
|
| |
Fixes: #8393
|
| |
|
|
|
|
| |
... because namedtuples suck
Fix up a couple of other annotations to keep mypy happy.
|
| |
|
|
|
| |
We either need to pass the auth provider over the replication api, or make sure
we report the auth provider on the worker that received the request. I've gone
with the latter.
|
| |
|
|
| |
Mention that parse_config must exist and note the
check_media_file_for_spam method.
|
| |
|
|
| |
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
|
| | |
|
| | |
|
| |
|
|
|
| |
This is a companion change to apply the fix in #9498 /
922788c6043138165c025c78effeda87de842bab to previously
purged rooms.
|
| |
|
|
|
| |
Earlier [I was convinced](https://github.com/matrix-org/synapse/issues/9565) that we didn't have an Admin API for listing media uploaded by a user. Foolishly I was looking under the Media Admin API documentation, instead of the User Admin API documentation.
I thought it'd be helpful to link to the latter so others don't hit the same dead end :)
|
| |
|
|
|
| |
Apple had to be special. They want a client secret which is generated from an EC key.
Fixes #9220. Also fixes #9212 while I'm here.
|
| |
|
| |
Fixes #8915
|
| |
|
| |
Type hint fixes due to Twisted 21.2.0 adding type hints.
|
| |
|
|
| |
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
|
| |
|
|
|
|
|
| |
The hashes are from commits due to auto-formatting, e.g. running black.
git can be configured to use this automatically by running the following:
git config blame.ignoreRevsFile .git-blame-ignore-revs
|
| |
|
|
|
|
|
|
|
| |
I noticed that I'd occasionally have `scripts-dev/lint.sh` fail when messing about with config options in my PR. The script calls `scripts-dev/config-lint.sh`, which attempts some validation on the sample config.
It does this by using `sed` to edit the sample_config, and then seeing if the file changed using `git diff`.
The problem is: if you changed the sample_config as part of your commit, this script will error regardless.
This PR attempts to change the check so that existing, unstaged changes to the sample_config will not cause the script to report an invalid file.
|
| | |
|
| | |
|
| |
|
|
|
| |
token (#9559)
This notice is giving a heads up to the planned spec compliance fix https://github.com/matrix-org/synapse/pull/9548.
|
| | |
|
| |
|
|
|
| |
Unfortunately this doesn't test re-joining the room since
that requires having another homeserver to query over
federation, which isn't easily doable in unit tests.
|
| |
|
|
|
| |
interfaces. (#9528)
This helps fix some type hints when running with Twisted 21.2.0.
|
| |
|
|
|
| |
Update reverse proxy to add OpenBSD relayd example configuration.
Signed-off-by: Leo Bärring <leo.barring@protonmail.com>
|
| | |
|
| |
|
|
|
|
|
|
| |
Following the advice at
https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since,
it's preferable to export unix timestamps, not ages.
There doesn't seem to be any particular naming convention for timestamp
metrics.
|
| |
|
| |
Add prom metrics for number of users successfully registering and logging in, by SSO provider.
|
| |
|
| |
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
|
| | |
|
| | |
|
| |
|
| |
Should fix some remaining warnings
|
| |
|
|
| |
Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
| |
Turns out nginx overwrites the Host header by default.
|
| |
|
|
|
| |
Prevent presence background jobs from running when presence is disabled
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| |
|
|
|
|
|
| |
This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46.
This is being backed out due to a regression (#9507) and additional
review feedback being provided.
|
| |
|
|
|
|
|
| |
It landed in schema version 58 after 59 had been created, causing some
servers to not run it. The main effect of was that not all rooms had
their chain cover calculated correctly. After the BG updates complete
the chain covers will get fixed when a new state event in the affected
rooms is received.
|
| |
|
|
| |
Fixes #9504
|
| | |
|
| | |
|
| |
|
|
| |
By consuming the response if the headers imply that the
content is too large.
|
| |
|
|
| |
This also pins the Twisted version in the mypy job for CI until
proper type hints are fixed throughout Synapse.
|
| |
|
|
|
|
|
| |
In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again.
Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
### Changes proposed in this PR
- Add support for the `no_proxy` and `NO_PROXY` environment variables
- Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519)
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment
This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`
Related to #9306 which also uses `ProxyAgent`
Signed-off-by: Timothy Leung tim95@hotmail.co.uk
|
| |
|
| |
... otherwise, we don't get the cookie back.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
rewrite XForwardedForRequest to set `isSecure()` based on
`X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
|
| | |
|
| |
|
|
|
|
|
| |
This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false".
This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Split ShardedWorkerHandlingConfig
This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).
* Remove special cases in ShardedWorkerHandlingConfig.
`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.
To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
|
| |
|
| |
The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
|
| |
|
|
| |
This confused me for a while.
|
| |
|
| |
And ensure the consistency of `event_auth_chain_id`.
|
| | |
|
| |
|
|
| |
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
|
| | |
|
| |
|
|
|
| |
It should be possible to reload `synapse.target` to have the reload propagate
to all the synapse units.
|
| |
|
|
|
|
|
| |
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.
The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.
Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
|
| |
|
|
|
| |
The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).
This PR cleans that up a bit.
|
| | |
|
| |\ |
|
| | | |
|
| | | |
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Synapse 1.28.0rc1 (2021-02-19)
==============================
Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).
This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
Removal warning
---------------
The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
which has been available since Synapse 1.7.0 (2019-12-13).
Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.
Features
--------
- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376))
- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400))
Bugfixes
--------
- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257))
- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428))
Updates to the Docker image
---------------------------
- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433))
Improved Documentation
----------------------
- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291))
- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317))
- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390))
- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404))
- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434))
Deprecations and Removals
-------------------------
- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429))
Internal Changes
----------------
- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003))
- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123))
- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305))
- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362))
- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396))
- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381))
- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384))
- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394))
- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
(#9402)
This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually.
---
When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed.
It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw.
This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence:
* If it was a local user join, send that user's latest presence to all servers in the room
* If it was a remote user join, send the presence for all local users in the room to that homeserver
We deduplicate by inserting all of those pending updates into a dictionary of the form:
```
{
server_name1: {presence_update1, ...},
server_name2: {presence_update1, presence_update2, ...}
}
```
Only after building this dict do we then start sending out presence updates.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
results (#9383)
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.
Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.
This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
|
| |/ /
| |
| |
| |
| |
| |
| | |
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
|
| |/
|
|
| |
Updates documentation from #9289 and removes a deprecated
endpoint which didn't work as expected.
|
| |
|
|
| |
This reverts commit 8ad4676f35ec065853a5eb335b5566fdcf320460.
|
| | |
|
| |
|
| |
It's proving incredibly hard to build in CircleCI infra.
|
| |
|
|
| |
(#9426)
|
| |
|
| |
This API was undocumented and nonsensical.
|
| |
|
|
|
|
|
|
| |
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.
It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
|
| | |
|
| |
|
|
|
|
| |
(#9428)
As the comment says, this guard was there for when the
initial user directory update has yet to happen.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a small bug that I noticed while working on #8956.
We have a for-loop which attempts to strip all presence changes for each user except for the final one, as we don't really care about older presence:
https://github.com/matrix-org/synapse/blob/9e19c6aab4b5a99039f2ddc7d3120dd3b26c274b/synapse/handlers/presence.py#L368-L371
`new_states_dict` stores this stripped copy of latest presence state for each user, before it is... put into a new variable `new_state`, which is just overridden by the subsequent for loop.
I believe this was instead meant to override `new_states`. Without doing so, it effectively meant:
1. The for loop had no effect.
2. We were still processing old presence state for users.
|
| | |
|
| | |
|
| |
|
| |
Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
|
| |
|
| |
Fixes #9347
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| |
|
|
| |
Fixes a "conflict" from 846b9d3df033be1043710e49e89bcba68722071e
and d1f13c7485768b45bfc1a2a674830c681c52e2a1.
|
| |
|
|
| |
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
|
| | |
|
| | |
|
| |
|
| |
And convert some inlineDeferreds to async-friendly functions.
|
| |
|
|
|
| |
This PR removes a set that was created and [initially used](https://github.com/matrix-org/synapse/commit/1d2a0040cff8d04cdc7d7d09d8f04a5d628fa9dd#diff-0bc92da3d703202f5b9be2d3f845e375f5b1a6bc6ba61705a8af9be1121f5e42R435-R436), but is no longer today.
May help cut down a bit on the time it takes to accept invites.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.27.0rc2 (2021-02-11)
==============================
Features
--------
- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
Bugfixes
--------
- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
Improved Documentation
----------------------
- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
|
| | | |
|
| | |
| |
| |
| | |
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
|
| | | |
|
| | |
| |
| |
| | |
This has the side-effect of being able to remove use of `inlineCallbacks`
in the test-cases for cleaner tracebacks.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
(#9003)
Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests.
Fix #8996
|
| | | |
|
| | |
| |
| |
| |
| | |
* Adds type hints to the groups servlet and stringutils code.
* Assert the maximum length of some input values for spec compliance.
|
| | |
| |
| |
| | |
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
|
| |\ \
| | |
| | | |
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
|
| | | | |
|
| | | |
| | |
| | |
| | | |
Signed-off-by: David Teller <davidt@element.io>
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
Signed-off-by: Joseph Arnault <computerdude90042@outlook.com>
Signed-off-by: Dan Callahan <danc@element.io>
Co-authored-by: compu42 <56663749+compu42@users.noreply.github.com>
|
| |\ \ \ |
|
| | | |/
| |/| |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
|
| | | |
| | |
| | | |
This fixes the Jinja2 templates for the mapping provider.
|
| | | |
| | |
| | |
| | | |
* Adds proper dependencies.
* Minor fixes in database layer.
|
| | | | |
|
| |\ \ \ |
|
| | |\| | |
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
* fix validation and don't use built-in validation UI
Co-authored-by: Bruno Windels <brunow@element.io>
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
config: Add detail to auto_join_rooms comment
Signed-off-by: Gary Dykstra <gary@dykstranet.com>
|
| | |/ /
|/| |
| | | |
Also add a few more IP ranges to the default blacklist.
|
| |/ / |
|
| | | |
|
| | |
| |
| |
| |
| | |
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.
This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
|
| | | |
|
| | | |
|
| | | |
|
| |\ \ |
|
| | | | |
|
| | | |
| | |
| | | |
fixes #9171
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
was tweaked to depend on whether there was 1 sender or multiple senders,
not based on if there was 1 room or multiple rooms.
|
| |\| | |
|
| | | |
| | |
| | |
| | |
| | | |
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.
I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Fixes #8966.
* Factor out build_synapse_client_resource_tree
Start a function which will mount resources common to all workers.
* Move sso init into build_synapse_client_resource_tree
... so that we don't have to do it for each worker
* Fix SSO-login-via-a-worker
Expose the SSO login endpoints on workers, like the documentation says.
* Update workers config for new endpoints
Add documentation for endpoints recently added (#8942, #9017, #9262)
* remove submit_token from workers endpoints list
this *doesn't* work on workers (yet).
* changelog
* Add a comment about the odd path for SAML2Resource
|
| | | |
| | |
| | | |
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
|
| | | |
| | |
| | | |
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
Context, Fixes: https://github.com/matrix-org/synapse/issues/9263
In the past to fix an issue with old Riots re-requesting threepid validation tokens, we raised a `LoginError` during UIA instead of `InteractiveAuthIncompleteError`. This is now breaking the way Tchap logs in - which isn't standard, but also isn't disallowed by the spec.
An easy fix is just to remove the 4 year old workaround.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
* synapse.app.base: only call gc.freeze() on CPython
gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.
Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.
Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Adds note about updating dh-virtualenv once we drop support for Xenial.
We can't update now, because it needs debhelper 12, while Xenial only
backports 10.
Signed-off-by: Dan Callahan <danc@element.io>
|
| |\| | |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
We've decided to add a 'brand' field to help clients decide how to style the
buttons.
Also, fix up the allowed characters for idp_id, while I'm in the area.
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | | |
Reading from the config object is *slow*.
|
| | |/
|/|
| | |
This finishes adding type hints to the `synapse.handlers` module.
|
| | | |
|
| | |
| |
| |
| | |
behalf (#9235)
|
| |\| |
|
| | |
| |
| | |
Fixes #8928.
|
| | |
| |
| |
| |
| | |
Co-authored-by: Dan Callahan <danc@element.io>
pip 21.0 stopped supporting Python 3.5.
|
| | |\
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
|
| | | |
| | |
| | |
| | |
| | | |
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
the homeserver config (#9229)
If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead.
This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](https://github.com/matrix-org/synapse/blob/5310808d3bebd17275355ecd474bc013e8c7462d/docs/sample_config.yaml#L1816-L1845) states) then `None` is not passed to the module. An empty dict is passed instead.
This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
|
| | | |
| | |
| | |
| | | |
With this change all handlers except the e2e_* ones have
type hints enabled.
|
| | | |
| | |
| | |
| | | |
The current configuration is handled for backwards compatibility,
but is considered deprecated.
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | | |
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
|
| | | |
| | |
| | |
| | |
| | | |
This new version no longer has the problem of adding/removing a blank line in `.pyi` files, which black disagrees with. This would cause `isort` to slightly modify `.pyi` files, before `black` would subsequently modify back directly afterwards.
Relevant `isort` issue: https://github.com/pycqa/isort/issues/1284
|
| |\ \ \
| | | |
| | | | |
Add forward extremities endpoint to rooms admin API
|
| | |\ \ \
| | | | |
| | | | |
| | | | |
| | | | | |
# Conflicts:
# synapse/rest/admin/__init__.py
|
| | |\ \ \ \
| | | | | |
| | | | | |
| | | | | | |
jaywink/admin-forward-extremities
|
| | |\ \ \ \ \
| | | | | | |
| | | | | | |
| | | | | | | |
jaywink/admin-forward-extremities
|
| | |\ \ \ \ \ \ |
|
| | | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |_|_|_|/ / /
|/| | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
This is done by creating a custom `RedisFactory` subclass that
periodically pings all connections in its pool.
We also ensure that the `replyTimeout` param is non-null, so that we
timeout waiting for the reply to those pings (and thus triggering a
reconnect).
|
| | | | | | | | |
|
| | |_|_|/ / /
|/| | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.
A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.
Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
|
| | |_|/ / /
|/| | | |
| | | | |
| | | | |
| | | | | |
Co-authored-by: Dan Callahan <danc@element.io>
pip 21.0 stopped supporting Python 3.5.
|
| |\ \ \ \ \
| | |_|_|/
| |/| | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
|
| | | | | | |
|
| | | | | | |
|
| | | | | |
| | | | |
| | | | | |
As we use `execute_values` with the `fetch` parameter.
|
| | |_|/ /
|/| | |
| | | |
| | | | |
Treat the content as untrusted and do not assume it is of
the proper form.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
(#9191)
The lists of source directories to lint between `tox.ini` and `lint.sh` became out of sync. This PR tightens them up and adds some comments reminding any future readers to keep the list in sync.
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
We have seen a failure mode here where if there are many in flight
unfinished IDs then marking an ID as finished takes a lot of CPU (as
calling deque.remove iterates over the list)
|
| |\| | | |
|
| | | | | |
|
| |\| | | |
|
