| Commit message (Collapse) | Author | Files | Lines |
|---|
|
|
|
|
|
To be reverted after the Synapse 1.56 release.
Signed-off-by: Sean Quah <seanq@element.io>
|
|
|
|
start (#12323)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Follow-up to https://github.com/matrix-org/synapse/pull/12083
Since we are now using the new `state_event_ids` parameter to do all of the heavy lifting.
We can remove any spots where we plumbed `auth_event_ids` just for MSC2716 things in
https://github.com/matrix-org/synapse/pull/9247/files.
Removing `auth_event_ids` from following functions:
- `create_and_send_nonmember_event`
- `_local_membership_update`
- `update_membership`
- `update_membership_locked`
|
|
When we are processing a `/backfill` request from a remote server, exclude any
outliers from consideration early on. We can't return outliers anyway (since we
don't know the state at the outlier), and filtering them out earlier means that
we won't attempt to calulate the state for them.
|
|
|
|
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
|
|
|
|
|
HAproxy reverse proxy guide update to stop sending IPv4-mapped address to HS.
|
|
|
|
This should speed up push rule calculations for rooms with large numbers of local users when the main push rule cache fails.
Co-authored-by: reivilibre <oliverw@matrix.org>
|
|
It is currently rather hard to see which rooms are causing inbound federation
traffic. Add the room id to the logs.
|
|
`state_event_ids`, which accurately describes the usage with MSC2716 instead of abusing `auth_event_ids` (#12083)
Spawned from https://github.com/matrix-org/synapse/pull/10975#discussion_r813183430
Part of [MSC2716](https://github.com/matrix-org/matrix-spec-proposals/pull/2716)
|
|
Hopefully this fixes #12257.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
|
|
(#12288)
Co-authored-by: Shay <hillerys@element.io>
|
|
|
|
The `MockHttpResource` and `MockKey` objects were unused
since #9396 (74af356baf79de5d719fad6c32b981eedbdd950d).
|
|
|
|
|
|
as 3.1.0 removed the deprecated jinja2.Markup class which we still rely on.
|
|
`markupsafe.Markup`" (#12296)
This reverts commit 8810c93e828a9ed1dcb008d08be8aa9fcb4d28c3.
|
|
|
|
|
|
(#12289)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
Filter the events returned from `/relations` for the requester's ignored users
in a similar way to `/messages` (and `/sync`).
|
|
|
|
|
|
|
|
documentation (#12038)
Resolves #10339
|
|
Signed-off-by: Nicolas Werner <n.werner@famedly.com>
|
|
Co-authored-by: reivilibre <olivier@librepush.net>
|
|
|
|
|
|
|
|
* Tweak copy for sso account details template
* Update sso footer copyright year
* Add newsfragment
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
|
|
|
|
|
|
registration template, in order to comply with SIWA guidelines. (#12260)
* Reword as 'Choose your account name'
* Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
|
* Make it possible to enable compression for the metrics HTTP resource
This can provide significant bandwidth savings pulling metrics from
synapse instances.
* Add changelog file.
* Fix type hint
|
|
* Replace `get_state_for_pdu` with `get_state_ids_for_pdu` and `get_events_as_list`.
* Return a 404 from `/state` and `/state_ids` for an outlier
|
|
* Try to detect borked package installations.
Fixes #12223.
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
|
The get_bundled_aggregations code is fairly high-level and uses
a lot of store methods, we move it into the handler as that seems
like a better fit.
|
|
We fetch the thread summary in two phases:
1. The summary that is shared by all users (count of messages and latest event).
2. Whether the requesting user has participated in the thread.
There's no use in attempting step 2 for events which did not return a summary
from step 1.
|
|
|
|
* inherit `MonthlyActiveUsersStore` from `RegistrationWorkerStore`
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
* Formally type the UserProfile in user searches
* export UserProfile in synapse.module_api
* Update docs
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
|
|
|
|
|
|
|
An error occured if a filter was supplied with `event_fields` which did not include
`unsigned`.
In that case, bundled aggregations are still added as the spec states it is allowed
for servers to add additional fields.
|
|
To handle cancellation, we ensure that `after_callback`s and
`exception_callback`s are always run, since the transaction will
complete on another thread regardless of cancellation.
We also wait until everything is done before releasing the
`CancelledError`, so that logging contexts won't get used after they
have been finished.
Signed-off-by: Sean Quah <seanq@element.io>
|
|
Adds a handler layer between the REST and datastore layers for relations.
|
|
|
|
|
|
* Moves the relation pagination tests to a separate class.
* Move the assertion of the response code into the `_send_relation` helper.
* Moves some helpers into the base-class.
|
|
By using urljoin from the standard library and reducing the number
of places URLs are rebased.
|
|
Instead of fetching the raw account data and re-parsing it. The
ignored_users table is a denormalised version of the account data
for quick searching.
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
|
|
|
|
|
|
These decorators mostly support cancellation already. Add cancellation
tests and fix use of finished logging contexts by delaying cancellation,
as suggested by @erikjohnston.
Signed-off-by: Sean Quah <seanq@element.io>
|
|
Also convert `ReadWriteLock` to use async context managers.
Signed-off-by: Sean Quah <seanq@element.io>
|
|
(#12215)
Signed-off-by: Sean Quah <seanq@element.io>
|
|
`delay_cancellation` behaves like `stop_cancellation`, except it
delays `CancelledError`s until the original `Deferred` resolves.
This is handy for unifying cleanup paths and ensuring that uncancelled
coroutines don't use finished logcontexts.
Signed-off-by: Sean Quah <seanq@element.io>
|
|
configuration flag. (#12200)
|
|
|
|
|
|
|
|
And rename the field used for caching from _id to _cache_key.
|
|
Fixes https://github.com/matrix-org/synapse/issues/12205
|
|
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
|
|
|
|
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
This test was relying on poking events which weren't in the database into
filter_events_for_client.
|
|
This includes the `typing`, `to_device`, `account_data`, `receipts`, and `presence`
streams (really anything except the `events` stream).
|
|
|
|
compatibility. Breaks compatibility with Mjolnir v1.3.1 and earlier. (#11700)
|
|
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
|
|
Some stuff that came up while we were talking about #12173.
|
|
This is allowed per MSC2675, although the original implementation did
not allow for it and would return an empty chunk / not bundle aggregations.
The main thing to improve is that the various caches get cleared properly
when an event is redacted, and that edits must not leak if the original
event is redacted (as that would presumably leak something similar to
the original event content).
|
|
Since the object it returns is a ReplicationCommandHandler.
This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
deactivate a user (#12028)
* Add check_can_deactivate_user
* Add check_can_shutdown_rooms
* Documentation
* callbacks, not functions
* Various suggested tweaks
* Add tests for test_check_can_shutdown_room and test_check_can_deactivate_user
* Update check_can_deactivate_user to not take a Requester
* Fix check_can_shutdown_room docs
* Renegade and use `by_admin` instead of `admin_user_id`
* fix lint
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Update docs/modules/third_party_rules_callbacks.md
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
|
* `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key.
* Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity.
* Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
|
|
|
|
This allows for the target process to be down for around a minute
which provides time for restarts during synapse upgrades/config updates.
Closes: #12178
Signed off by Nick Mills-Barrett nick@beeper.com
|
|
We don't *have* the state at a backwards-extremity, so this is never going to
do anything useful.
|
|
* Rewrites the demo documentation to be clearer, accurate, and moves it to our documentation tree.
* Improvements to the demo scripts:
* `clean.sh` now runs `stop.sh` first to avoid zombie processes.
* Uses more modern Synapse configuration (and removes some obsolete configuration).
* Consistently use the HTTP ports for server name, etc.
* Remove the `demo/etc` directory and place everything into the `demo/808x` directories.
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
(#12131)
|
|
|
|
|
|
|
|
This field is only to be used in the Server-Server API, and not the
Client-Server API, but was being leaked when a federation response
was used in the /hierarchy API.
|
|
Some properties were marked as RedisProtocol instead of ConnectionHandler,
which wraps RedisProtocol instance(s).
|
|
|
|
|
|
start if pre-release versions of dependencies were installed. (#12177)
* Add failing test to characterise the regression #12176
* Permit pre-release versions of specified packages
* Newsfile (bugfix)
Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
|
|
using the default batch size (#12157)
|
|
The caches for the target of the relation must be cleared
so that the bundled aggregations are re-calculated after
the redaction is processed.
|
|
I've factored it out for easier use in other workflows.
|
|
It’s just occurred to me that #12088 pulled in the “packaging” package (~=21.3). I pulled in the newest version I had at the time.
I only use it for packaging.requirements.Requirements. Which was added in packaging 16.1: https://github.com/pypa/packaging/releases/tag/16.1
https://pkgs.org/download/python3-packaging suggests that the oldest version we care about is 17.1 in Ubuntu Bionic. So I think with this bound we're hunky dory.
|
|
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
|
|
|
|
|
|
|
|
Avoid trying to get the state for outliers, which isn't a sensible thing to do.
|
|
We make multiple passes over this, so a regular iterable won't do.
|
|
If we locally generate a rejection for an invite received over federation, it
is stored as an outlier (because we probably don't have the state for the
room). However, currently we still generate a state group for it (even though
the state in that state group will be nonsense).
By setting the `outlier` param on `create_event`, we avoid the nonsensical
state.
|
|
|
|
I've argued in #11537 that poetry and tox don't cooperate well at the
moment. (See also #12119.) Therefore I'm pruning away bits of tox to make the transition to poetry easier. This change removes the commands for coverage.
We don't use coverage in anger at the moment. It shouldn't be too hard to add coverage as a dev-dependency and reintroduce this if we really want it.
|
|
* Add type hints to `tests/rest`
* newsfile
* change import from `SigningKey`
|
|
This ensures that the `latest_event` field of the bundled aggregation
for threads uses the same format as the other events in the response.
|
|
* Fix incorrect argument in test case
* Add copyright header
* Docstring and __all__
* Exclude dev depenencies
* Use changelog from #12088
* Include version in error messages
This will hopefully distinguish between the version of the source code
and the version of the distribution package that is installed.
* Linter script is your friend
|
|
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
|
|
There's no reason to let people use long keys.
|
|
The Complement tests for MSC3030 are now merged, https://github.com/matrix-org/complement/pull/178
Synapse implmentation: https://github.com/matrix-org/synapse/pull/9445
|
|
|
|
* Move the `snapcraft` configuration to `contrib`.
We're happy for people to package this as a snap image if it's useful,
but we don't support or maintain it. I'd like to move the config to
`contrib` to reflect this state of affairs.
* Changelog
|
|
* Rename scripts-dev to have suffices
* Update references to `scripts-dev`
* Changelog
* These scripts don't pass mypy
|
|
* Remove unused mocks from `test_typing`
It's not clear what these do. `get_user_by_access_token` has the wrong
signature, including the return type. Tests all pass without these. I
think we should nuke them.
* Changelog
* Fixup imports
|
|
* Add type hints to `tests/rest/client`
* newsfile
* fix imports
* add `test_account.py`
* Remove one type hint in `test_report_event.py`
* change `on_create_room` to `async`
* update new functions in `test_third_party_rules.py`
* Add `test_filter.py`
* add `test_rooms.py`
* change to `assertEquals` to `assertEqual`
* lint
|
|
|
|
As we want to include the previous version in the "No new changes..."
string.
|
|
As a str is a sequence, the checks were not granular
enough and would allow lists or strings, when only
lists were valid.
|
|
(#12118)
* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
|
|
|
|
* Modernizes code (f-strings, etc.)
* Fixes incorrect comments.
* Splits the test case into two.
* Factors out some duplicated code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
* Don't use `tox` for `check-sampleconfig`
* Don't use `tox` for check-newsfragment
|
|
* Pull runtime dep checks into their own module
* Reimplement `check_requirements` using `importlib`
I've tried to make this clearer. We start by working out which of
Synapse's requirements we need to be installed here and now. I was
surprised that there wasn't an easier way to see which packages were
installed by a given extra.
I've pulled out the error messages into functions that deal with "is
this for an extra or not". And I've rearranged the loop over two
different sets of requirements into one loop with a "must be instaled"
flag.
I hope you agree that this is clearer.
* Test cases
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
profile update (#12062)
|
|
|
|
|
|
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
When we get a partial_state response from send_join, store information in the
database about it:
* store a record about the room as a whole having partial state, and stash the
list of member servers too.
* flag the join event itself as having partial state
* also, for any new events whose prev-events are partial-stated, note that
they will *also* be partial-stated.
We don't yet make any attempt to interpret this data, so API calls (and a bunch
of other things) are just going to get incorrect data.
|
|
|
|
* Fix 'Unhandled error in Deferred'
Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when
a function wrapped with `@cachedList` failed
* Minor optimisation to cachedListDescriptor
we can avoid re-using `missing`, which saves looking up entries in
`deferreds_map`, and means we don't need to copy it.
* Improve type annotation on CachedListDescriptor
|
|
* fix incorrect unwrapFirstError import
this was being imported from the wrong place
* Refactor `concurrently_execute` to use `yieldable_gather_results`
* Improve exception handling in `yieldable_gather_results`
Try to avoid swallowing so many stack traces.
* mark unwrapFirstError deprecated
* changelog
|
|
* Add type hints to `tests/rest/client`
* update `mypy.ini`
* newsfile
* add `test_register.py`
|
|
...and various code supporting it.
The /spaces endpoint was from an old version of MSC2946 and included
both a Client-Server and Server-Server API. Note that the unstable
/hierarchy endpoint (from the final version of MSC2946) is not yet
removed.
|
|
Instead of being part of /capabilities, this matches a change to
MSC3440 to properly use these endpoints.
|
|
|
|
* Fix `PushRuleEvaluator` to work on frozendicts
frozendicts do not (necessarily) inherit from dict, so this needs to handle
them correctly.
* Fix event filtering for frozen events
Looks like this one was introduced by #11194.
|
|
transactions (#12098)
|
|
Before this fix, a legitimate 404 from a federation endpoint (e.g. due
to an unknown room) would be treated as an unknown endpoint. This
could cause unnecessary federation traffic.
|
|
|
|
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
Signed-off-by: Sean Quah <seanq@element.io>
|
|
|
|
states to Application Services. (#11617)
Co-authored-by: Erik Johnston <erik@matrix.org>
|
|
Don't attempt to add non-string `value`s to `event_search` and add a
background update to clear out bad rows from `event_search` when
using sqlite.
Signed-off-by: Sean Quah <seanq@element.io>
|
|
And clean-up the endpoints which should be routed to workers.
|
|
|
|
|
|
These have snuck in since #12031 was started.
Also a couple of other cleanups while we're in the area.
|
|
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.
Part of #11733
|
|
The complement.sh script relies on the name of the ref matching the name
of the unpacked folder. The branch redirect from renaming the default
branch breaks that assumption.
Signed-off-by: Nicolas Werner <n.werner@famedly.com>
|
|
|
|
|
|
See matrix-org/matrix-doc#3720
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
|
|
* Remove `trial` section from setup.cfg
This was added in the initial commit from 2014. I can't see that it does
anything. Maybe it's there so that you can run `trial` without any extra
args, but if I do that then I just get the `--help` message.
* Move flake8's config to its own file
|
|
|
|
|
|
... to ensure it gets a proper log context, mostly.
|
|
use. The slowness existed since the initial implementation of refresh tokens. (#12056)
|
|
This is an endpoint that we have server-side support for, but no client-side support. It's going to be useful for resyncing partial-stated rooms, so let's introduce it.
|
|
By including "bot" in the User-Agent, which some sites use
to decide whether to include additional Open Graph information.
|
|
|
|
|
|
|
|
* Don't build distribution pkgs in tests.yml
* Run `release-artifacts` on release branches
* Use backend-meta workflow for packaging
|
|
|
|
|
|
|
|
A follow-up to #12005, in which I apparently missed that there are a bunch of other places that assume the create event is in the auth chain.
|
|
These started failing in
https://github.com/matrix-org/synapse/pull/12031... I'm a bit mystified by how
they ever worked.
|
|
|
|
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
|
|
Since it is reasonable to give a future or past auth provider,
which might not be in the current configuration.
|
|
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
|
|
|
|
|
|
Keeping backwards compatibility with the unstable flag for now.
|
|
`/timestamp_to_event` endpoint (#12024)
|
|
|
|
Fixes #11962
|
|
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
Part of my work on #11249: add code to handle the new fields added in MSC3706.
|
|
As the comment says, virtualenv is a pile of fail.
|
|
|
|
msc3706 proposes changing the `/send_join` response:
> Any events returned within `state` can be omitted from `auth_chain`.
Currently, we rely on `m.room.create` being returned in `auth_chain`, but since
the `m.room.create` event must necessarily be part of the state, the above
change will break this.
In short, let's look for `m.room.create` in `state` rather than `auth_chain`.
|
|
|
|
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
|
|
|
Initially introduced in matrix-org-hotfixes by e5537cf (and tweaked by later commits).
Fixes #11995
See also #4826
|
|
For users with large accounts it is inefficient to calculate the set of
users they share a room with (and takes a lot of space in the cache).
Instead we can look at users whose devices have changed since the last
sync and check if they share a room with the syncing user.
|
|
We already have two copies of this code, in 2/3 of the callers of
`_auth_and_persist_outliers`. Before I add a third, let's push it down.
|
|
Currently we only track evictions due to size or time constraints.
|
|
When the server leaves a room the `get_rooms_for_user` cache is not
correctly invalidated for the remote users in the room. This means that
subsequent calls to `get_rooms_for_user` for the remote users would
incorrectly include the room (it shouldn't be included because the
server no longer knows anything about the room).
|
|
The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway.
I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes.
That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data.
So... this wasn't doing anything, except annoying Complement?
|
|
Splits the search code into a few logical functions instead of a single
unreadable function.
There are also a few additional changes for readability.
After refactoring it was clear to see there were some unused and
unnecessary variables, which were simplified.
|
|
If the latest event in a thread was edited than the original
event content was included in bundled aggregation for
threads instead of the edited event content.
|
|
(#11972)
|
|
|
|
|
|
This reverts commit 2bf31f7807c7a0c229170803c97090d612dc16f9.
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
|
|
(#11988)
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
|
|
|
|
|
|
To allow for easier backwards/forwards compatibility by making
it easier to add/remove callbacks.
|
|
* Require latest matrix-common
* Use the common function
|
|
Co-authored-by: Patrick |
