| Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
|
|
|
|
|
|
|
|
* Correctly ratelimit invites when creating a room
Also allow ratelimiting for more than one action at a time.
|
|
|
|
|
|
|
|
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.
This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
|
|
Per changes in MSC2946, the C-S and S-S APIs for spaces summary
should use GET requests.
Until this is stable, the POST endpoints still exist.
This does not switch federation requests to use the GET version yet
since it is newly added and already deployed servers might not support
it. When switching to the stable endpoint we should switch to GET
requests.
|
|
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
|
|
|
|
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
|
|
|
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
|
|
fixes #9960
|
|
* tests for push rule pattern matching
* tests for acl pattern matching
* factor out common `re.escape`
* Factor out common re.compile
* Factor out common anchoring code
* add word_boundary support to `glob_to_regex`
* Use `glob_to_regex` in push rule evaluator
NB that this drops support for character classes. I don't think anyone ever
used them.
* Improve efficiency of globs with multiple wildcards
The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.
* add assertion on regex pattern
* Fix mypy
* Simplify glob_to_regex
* Inline the glob_to_regex helper function
Signed-off-by: Dan Callahan <danc@element.io>
* Moar comments
Signed-off-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
|
|
We were pulling the full auth chain for the room out of the DB each time
we backfilled, which can be *huge* for large rooms and is totally
unnecessary.
|
|
... and test it.
|
|
Fixes regression in send PDU times introduced in #9905.
|
|
|
|
c.f. #9936
|
|
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.
It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
|
|
|
|
This reverts commit e9eb3549d32a6f93d07de8dbd5e1ebe54c8d8278.
|
|
|
|
Fixes #9936
|
|
|
|
This is an update based on changes to MSC2946. The origin_server_ts
of the m.room.create event is copied into the creation_ts field for each
room returned from the spaces summary.
|
|
|
|
Support both the unstable and stable identifiers. A future release
will disable the unstable identifiers.
|
|
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
|
|
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.
This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.
Adds a `gc_min_seconds_between` config option to override the defaults.
Fixes #9890.
|
|
|
|
|
|
* Add healthcheck startup delay by 5secs and reduced interval check to 15s
to reduce waiting time for docker aware edge routers bringing an
instance online
|
|
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
|
|
|
|
|
|
Adds the `port` option with the default value to the sample config file.
|
|
Signed-off-by: Dan Callahan <danc@element.io>
|
|
|
|
The user_may_create_room_alias method on spam checkers
declared the room_alias parameter as a str when in reality it is
passed a RoomAlias object.
|
|
|
|
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.
Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
|
|
Only affects workers. Introduced in #9819.
Fixes #9899.
|
|
This fixes a regression where the logging context for runWithConnection
was reported as runWithConnection instead of the connection name,
e.g. "POST-XYZ".
|
|
|
|
|
|
|
|
This reverts commit 05e8c70c059f8ebb066e029bc3aa3e0cefef1019.
|
|
|
|
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
|
|
(#9868)
This ensures that something like an auth error (403) will be
returned to the requester instead of attempting to try more
servers, which will likely result in the same error, and then
passing back a generic 400 error.
|
|
This should be a lot quicker than asking the state handler.
|
|
* Simplify `start_listening` callpath
* Correctly check the size of uploaded files
|
|
First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation.
Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
|
|
Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574.
As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.
|
|
|
|
|
|
|
|
|
|
|
|
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join rules.
This only applies to an experimental room version, as defined in MSC3083.
|
|
|
|
|
|
|
|
|
|
Fixes #9866.
|
|
This is no longer required, since we have dropped support for Python 3.5.
|
|
|
|
|
|
|
|
|
|
1.32.0 also introduced an incompatibility with Synapse modules that make use of `synapse.logging.context.LoggingContext`, such as [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider).
This PR adds a note to the 1.32.0 changelog and upgrade notes about it.
|
|
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This undoes part of b076bc276e881b262048307b6a226061d96c4a8d.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
deprecation notices to v1.32.0 upgrade notes (#9849)
Fixes https://github.com/matrix-org/synapse/issues/9846.
Adds important removal information from the top of https://github.com/matrix-org/synapse/releases/tag/v1.32.0rc1 into UPGRADE.rst.
|
|
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.
This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
|
|
|
|
|
|
|
|
hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
|
|
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
|
|
|
|
Fixes: #9797.
Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
|
|
There are a couple of points in `persist_events` where we are doing a
query per event in series, which we can replace.
|
|
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
|
|
handler (#9800)
This refactoring allows adding logic that uses the event context
before persisting it.
|
|
room. (#9763)"
This reverts commit cc51aaaa7adb0ec2235e027b5184ebda9b660ec4.
The PR was prematurely merged and not yet approved.
|
|
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join
rules.
This only applies to an experimental room version, as defined in MSC3083.
|
|
This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max).
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
|
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
|
Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
|
|
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
|
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with:
* A Synapse main process.
* Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime.
* A redis for worker communication.
* A nginx for routing traffic.
* A supervisord to start all worker processes and monitor them if any go down.
Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite.
`configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI.
Huge thanks to @MatMaul for helping get this all working :tada: This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62.
Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
|
|
Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894.
I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support.
The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
|
|
|
|
|
|
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
Signed-off-by: Dan Callahan <danc@element.io>
|
|
|
|
* Drop Python 3.5 from Trove classifier metadata.
Signed-off-by: Dan Callahan <danc@element.io>
|
|
Signed-off-by: Dan Callahan <danc@element.io>
|
|
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
|
|
|
|
There's no point logging this twice.
|
|
By providing the additional build tag for `msc2946`.
|
|
Signed-off-by: Dan Callahan <danc@element.io>
|
|
Related: #8334
Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25)
`GET /_synapse/admin/v1/users/<user_id>` has no
- unit tests
- documentation
API in v2 is available (#5925 - 12/2019, v1.7.0).
API is misleading. It expects `user_id` and returns a list of all users.
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
|
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
|
|
|
We pull all destinations requiring catchup from the DB in batches.
However, if all those destinations get filtered out (due to the
federation sender being sharded), then the `last_processed` destination
doesn't get updated, and we keep requesting the same set repeatedly.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Apparently on tox 2.5, `usedevelop` overrides `skip_install`, so we end up
trying to install the full dependencies even for the `-old` environment.
|
|
They don't make any sense on the intermediate builder image. The final
images needs them to be of use for anyone.
Signed-off-by: Johannes Wienke <languitar@semipol.de>
|
|
(#9735)
When joining a room with join rules set to 'restricted', check if the
user is a member of the spaces defined in the 'allow' key of the join rules.
This only applies to an experimental room version, as defined in MSC3083.
|
|
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
|
|
present (#8926)
This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.
|
|
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.
This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.
A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync.
The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:
* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
|
|
|
|
|
|
The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key`
constraint is superceded by
`remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds
`thumbnail_method` to the unique key).
PR #7124 made an attempt to remove the old constraint, but got the name wrong,
so it didn't work. Here we update the bg update and rerun it.
Fixes #8649.
|
|
|
|
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Per MSC3083.
|
|
|
|
The regex should be terminated so that subdomain matches of another
domain are not accepted. Just ensuring that someone doesn't shoot
themselves in the foot by copying our example.
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
|
|
Fixes #9642.
Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
|
|
This PR rewrites the original complement.sh script with a number of improvements:
* We can now use a local checkout of Complement (configurable with `COMPLEMENT_DIR`), though the default behaviour still downloads the master branch.
* You can now specify a regex of test names to run, or just run all tests.
* We now use the Synapse test blacklist tag (so all tests will pass).
|
|
|
|
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
|
|
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.
We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.
Fixes #9663
|
|
|
|
|
|
For it's obvious performance benefits. `dmypy` support landed in #9692.
|
|
|
|
non-pip package (#9074)
Signed-off-by: blakehawkins blake.hawkins.11@gmail.com
|
|
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
|
|
I've reiterated the advice about using `oidc` to migrate, since I've seen a few
people caught by this.
I've also removed a couple of the examples as they are duplicating the OIDC
documentation, and I think they might be leading people astray.
|
|
|
|
|
|
If you have the wrong version of `cryptography` installed, synapse suggests:
```
To install run:
pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6''
```
However, the use of ' inside '...' doesn't work, so when you run this, you get
an error.
|
|
Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing.
If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘
As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`.
Fix https://github.com/matrix-org/synapse/issues/9364
|
|
This warning is somewhat confusing to users, so let's suppress it
|
|
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.
`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
|
|
|
|
|
|
|
|
cryptography (#9697)
|
|
Fixes redirect loop
Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
|
|
using /usr/bin/env (#9689)
On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work.
`/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters.
Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
|
|
Make it clearer in the source install step that the platform specific
prerequisites must be installed first.
Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
|
|
Split off from https://github.com/matrix-org/synapse/pull/9491
Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133
Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
|
|
Broke in #9640
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
|
|
|
|
When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data.
Fixes: #9447
|
|
|
|
This fixes an error ("Cannot determine consistent method resolution order (MRO)")
when running mypy with a cache.
|
|
|
|
|
|
It's legitimate behaviour to try and join a bunch of rooms at once.
|
|
|
|
Builds on the work done in #9643 to add a federation API for space summaries.
There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
|
|
|
|
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70
This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.
Also added some type hints.
|
|
Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys
Signed-off-by: Johannes Wienke <languitar@semipol.de>
|
|
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
|
|
|
|
|
|
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.
Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
|
|
|
|
serialize_event (#9585)
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](https://github.com/matrix-org/synapse/blob/17cd48fe5171d50da4cb59db647b993168e7dfab/synapse/events/third_party_rules.py#L73-L76) - will raise a 500 if you try to a edit a reply to a message.
`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.
To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
|
|
By splitting this to two separate methods the callers know
what methods they can expect on the handler.
|
|
ones (#9634)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
|
|
* Populate `internal_metadata.outlier` based on `events` table
Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.
* Move `outlier` out of InternalMetadata._dict
Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
|
|
|
|
* Adds B00 to ignored checks.
* Fixes remaining issues.
|
|
Allows limiting who can login using OIDC via the claims
made from the IdP.
|
|
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
|
|
|
|
|
|
|
|
|
|
Fixes https://github.com/matrix-org/synapse/issues/9572
When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.
This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
|
|
Co-authored-by: Will Hunt <willh@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
|
|
* Handle an empty cookie as an invalid macaroon.
* Newsfragment
|
|
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
|
|
... and complain if people try to turn it off.
|
|
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
|
|
* Fix Internal Server Error on `GET /saml2/authn_response`
Seems to have been introduced in #8765 (Synapse 1.24.0)
* Fix newsfile
|
|
|
|
|
|
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.
Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
|
|
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.
This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)
The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).
Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
|
|
|
|
If more transactions arrive from an origin while we're still processing the
first one, reject them.
Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
|
|
Put the room id in the logcontext, to make it easier to understand what's going on.
|
|
|
|
|
|
Fixes: #8393
|
|
... because namedtuples suck
Fix up a couple of other annotations to keep mypy happy.
|
|
We either need to pass the auth provider over the replication api, or make sure
we report the auth provider on the worker that received the request. I've gone
with the latter.
|
|
Mention that parse_config must exist and note the
check_media_file_for_spam method.
|
|
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
|
|
|
|
|
|
This is a companion change to apply the fix in #9498 /
922788c6043138165c025c78effeda87de842bab to previously
purged rooms.
|
|
Earlier [I was convinced](https://github.com/matrix-org/synapse/issues/9565) that we didn't have an Admin API for listing media uploaded by a user. Foolishly I was looking under the Media Admin API documentation, instead of the User Admin API documentation.
I thought it'd be helpful to link to the latter so others don't hit the same dead end :)
|
|
Apple had to be special. They want a client secret which is generated from an EC key.
Fixes #9220. Also fixes #9212 while I'm here.
|
|
Fixes #8915
|
|
Type hint fixes due to Twisted 21.2.0 adding type hints.
|
|
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
|
|
The hashes are from commits due to auto-formatting, e.g. running black.
git can be configured to use this automatically by running the following:
git config blame.ignoreRevsFile .git-blame-ignore-revs
|
|
After 0764d0c6e575793ca506cf021aff3c4b9e0a5972
|
