| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
table (#10947)
* add test
* add function to remove user from monthly active table in deactivate code
* add function to remove user from monthly active table
* add changelog entry
* update changelog number
* requested changes
* update docstring on new function
* fix lint error
* Update synapse/storage/databases/main/monthly_active_users.py
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
| |
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.44.0rc3 (2021-10-04)
==============================
Bugfixes
--------
- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933))
- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Introduce `should_include_local_users_in_dir`
We exclude three kinds of local users from the user_directory tables. At
present we don't consistently exclude all three in the same places. This
commit introduces a new function to gather those exclusion conditions
together. Because we have to handle local and remote users in different
ways, I've made that function only consider the case of remote users.
It's the caller's responsibility to make the local versus remote
distinction clear and correct.
A test fixup is required. The test now hits a path which makes db
queries against the users table. The expected rows were missing, because
we were using a dummy user that hadn't actually been registered.
We also add new test cases to covert the exclusion logic.
----
By my reading this makes these changes:
* When an app service user registers or changes their profile, they will
_not_ be added to the user directory. (Previously only support and
deactivated users were excluded). This is consistent with the logic that
rebuilds the user directory. See also [the discussion
here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548).
* When rebuilding the directory, exclude support and disabled users from
room sharing tables. Previously only appservice users were excluded.
* Exclude all three categories of local users when rebuilding the
directory. Previously `_populate_user_directory_process_users` didn't do
any exclusion.
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | | |
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
it. (#10933)
This fixes a "Event not signed by authorising server" error when
transition room member from join -> join, e.g. when updating a
display name or avatar URL for restricted rooms.
|
| | |
| |
| |
| |
| | |
* type-hint `HomeserverTestcase.setup_test_homeserver`
For better IDE completion. A small drive-by.
|
| | | |
|
| | |
| |
| |
| | |
Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent
to properly configure auto-join rooms.
|
| | |
| |
| |
| |
| |
| |
| | |
it. (#10933)
This fixes a "Event not signed by authorising server" error when
transition room member from join -> join, e.g. when updating a
display name or avatar URL for restricted rooms.
|
| | |
| |
| |
| |
| | |
Fix some harmless errors from background processes (mostly
due to awaiting Mock objects) that occurred in the Synapse
logs during unit tests.
|
| |\|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.44.0rc2 (2021-09-30)
==============================
Bugfixes
--------
- Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error. ([\#10938](https://github.com/matrix-org/synapse/issues/10938))
- Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944))
Improved Documentation
----------------------
- Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Fix event context for outlier causing failures in all of the MSC2716
Complement tests.
The `EventContext.for_outlier` refactor happened in
https://github.com/matrix-org/synapse/pull/10883
and this spot was left out.
|
| | |
| |
| |
| | |
annotations) (#10944)
|
| | |
| |
| | |
Various updates to the install docs.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
`ttl` must be an integer according to the OpenAPI spec:
https://github.com/matrix-org/matrix-doc/blob/old_master/data/api/client-server/voip.yaml#L70
True division (`/`) returns a float instead (`"ttl": 7200.0`).
Floor division (`//`) returns an integer, so the response is spec compliant.
Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
|
| | |
| |
| |
| |
| |
| |
| | |
This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job.
Until that change is in a twisted release, we'll have to ignore the type
of the `host` argument. I've raised #10899 to remind us to review the
issue in a few months' time.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Pull out GetUserDirectoryTables helper
* Don't rebuild the dir in tests that don't need it
In #10796 I changed registering a user to add directory entries under.
This means we don't have to force a directory regbuild in to tests of
the user directory search.
* Move test_initial to tests/storage
* Add type hints to both test_user_directory files
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Broadly, the existing `event_auth.check` function has two parts:
* a validation section: checks that the event isn't too big, that it has the rught signatures, etc.
This bit is independent of the rest of the state in the room, and so need only be done once
for each event.
* an auth section: ensures that the event is allowed, given the rest of the state in the room.
This gets done multiple times, against various sets of room state, because it forms part of
the state res algorithm.
Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think
that makes everything hard to follow. Instead, we split the function in two and call each part
separately where it is needed.
|
| |\|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.44.0rc1 (2021-09-29)
==============================
Features
--------
- Only allow the
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)
`/batch_send?chunk_id=xxx` endpoint to connect to an already existing
insertion event.
([\#10776](https://github.com/matrix-org/synapse/issues/10776))
- Improve oEmbed URL previews by processing the author name, photo, and
video information.
([\#10814](https://github.com/matrix-org/synapse/issues/10814),
[\#10819](https://github.com/matrix-org/synapse/issues/10819))
- Speed up responding with large JSON objects to requests.
([\#10868](https://github.com/matrix-org/synapse/issues/10868),
[\#10905](https://github.com/matrix-org/synapse/issues/10905))
- Add a `user_may_create_room_with_invites` spam checker callback to
allow modules to allow or deny a room creation request based on the
invites and/or 3PID invites it includes.
([\#10898](https://github.com/matrix-org/synapse/issues/10898))
Bugfixes
--------
- Fix a long-standing bug that caused an `AssertionError` when purging
history in certain rooms. Contributed by @Kokokokoka.
([\#10690](https://github.com/matrix-org/synapse/issues/10690))
- Fix a long-standing bug which caused deactivated users that were later
reactivated to be missing from the user directory.
([\#10782](https://github.com/matrix-org/synapse/issues/10782))
- Fix a long-standing bug that caused unbanning a user by sending a
membership event to fail. Contributed by @aaronraimist.
([\#10807](https://github.com/matrix-org/synapse/issues/10807))
- Fix a long-standing bug where logging contexts would go missing when
federation requests time out.
([\#10810](https://github.com/matrix-org/synapse/issues/10810))
- Fix a long-standing bug causing an error in the deprecated
`/initialSync` endpoint when using the undocumented `from` and `to`
parameters.
([\#10827](https://github.com/matrix-org/synapse/issues/10827))
- Fix a bug causing the `remove_stale_pushers` background job to
repeatedly fail and log errors. This bug affected Synapse servers that
had been upgraded from version 1.28 or older and are using SQLite.
([\#10843](https://github.com/matrix-org/synapse/issues/10843))
- Fix a long-standing bug in Unicode support of the room search admin
API breaking search for rooms with non-ASCII characters.
([\#10859](https://github.com/matrix-org/synapse/issues/10859))
- Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership
events which we sent to remote servers to be incorrectly stored in the
local database.
([\#10873](https://github.com/matrix-org/synapse/issues/10873))
- Fix invalidating one-time key count cache after claiming keys. The bug
was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper.
([\#10875](https://github.com/matrix-org/synapse/issues/10875))
- Fix a long-standing bug causing application service users to be
subject to MAU blocking if the MAU limit had been reached, even if
configured not to be blocked.
([\#10881](https://github.com/matrix-org/synapse/issues/10881))
- Fix a long-standing bug which could cause events pulled over
federation to be incorrectly rejected.
([\#10907](https://github.com/matrix-org/synapse/issues/10907))
- Fix a long-standing bug causing URL cache files to be stored in
storage providers. Server admins may safely delete the `url_cache/` and
`url_cache_thumbnails/` directories from any configured storage
providers to reclaim space.
([\#10911](https://github.com/matrix-org/synapse/issues/10911))
- Fix a long-standing bug leading to race conditions when creating media
store and config directories.
([\#10913](https://github.com/matrix-org/synapse/issues/10913))
Improved Documentation
----------------------
- Fix some crashes in the Module API example code, by adding JSON
encoding/decoding.
([\#10845](https://github.com/matrix-org/synapse/issues/10845))
- Add developer documentation about experimental configuration flags.
([\#10865](https://github.com/matrix-org/synapse/issues/10865))
- Properly remove deleted files from GitHub pages when generating the
documentation.
([\#10869](https://github.com/matrix-org/synapse/issues/10869))
Internal Changes
----------------
- Fix GitHub Actions config so we can run sytest on synapse from
parallel branches.
([\#10659](https://github.com/matrix-org/synapse/issues/10659))
- Split out
[MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) meta
events to their own fields in the `/batch_send` response.
([\#10777](https://github.com/matrix-org/synapse/issues/10777))
- Add missing type hints to REST servlets.
([\#10785](https://github.com/matrix-org/synapse/issues/10785),
[\#10817](https://github.com/matrix-org/synapse/issues/10817))
- Simplify the internal logic which maintains the user directory
database tables.
([\#10796](https://github.com/matrix-org/synapse/issues/10796))
- Use direct references to config flags.
([\#10812](https://github.com/matrix-org/synapse/issues/10812),
[\#10885](https://github.com/matrix-org/synapse/issues/10885),
[\#10893](https://github.com/matrix-org/synapse/issues/10893),
[\#10897](https://github.com/matrix-org/synapse/issues/10897))
- Specify the type of token in generic "Invalid token" error messages.
([\#10815](https://github.com/matrix-org/synapse/issues/10815))
- Make `StateFilter` frozen so it is hashable.
([\#10816](https://github.com/matrix-org/synapse/issues/10816))
- Fix a long-standing bug where an `m.room.message` event containing a
null byte would cause an internal server error.
([\#10820](https://github.com/matrix-org/synapse/issues/10820))
- Add type hints to the state database.
([\#10823](https://github.com/matrix-org/synapse/issues/10823))
- Opt out of cache expiry for `get_users_who_share_room_with_user`, to
hopefully improve `/sync` performance when you
haven't synced recently.
([\#10826](https://github.com/matrix-org/synapse/issues/10826))
- Track cache eviction rates more finely in Prometheus's monitoring.
([\#10829](https://github.com/matrix-org/synapse/issues/10829))
- Add missing type hints to `synapse.handlers`.
([\#10831](https://github.com/matrix-org/synapse/issues/10831),
[\#10856](https://github.com/matrix-org/synapse/issues/10856))
- Extend the Module API to let plug-ins check whether an ID is local and
to access IP + User Agent data.
([\#10833](https://github.com/matrix-org/synapse/issues/10833))
- Factor out PNG image data to a constant to be used in several tests.
([\#10834](https://github.com/matrix-org/synapse/issues/10834))
- Add a test to ensure state events sent by modules get persisted
correctly.
([\#10835](https://github.com/matrix-org/synapse/issues/10835))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)
fields and event types from `chunk` to `batch` to match the
`/batch_send` endpoint.
([\#10838](https://github.com/matrix-org/synapse/issues/10838))
- Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)
`/batch_send` query parameter from `?prev_event` to more obvious usage
with `?prev_event_id`.
([\#10839](https://github.com/matrix-org/synapse/issues/10839))
- Add type hints to `synapse.http.site`.
([\#10867](https://github.com/matrix-org/synapse/issues/10867))
- Include outlier status when we log V2 or V3 events.
([\#10879](https://github.com/matrix-org/synapse/issues/10879))
- Break down Grafana's cache expiry time series based on reason for
eviction, c.f.
[\#10829](https://github.com/matrix-org/synapse/issues/10829).
([\#10880](https://github.com/matrix-org/synapse/issues/10880))
- Clean up some of the federation event authentication code for clarity.
([\#10883](https://github.com/matrix-org/synapse/issues/10883),
[\#10884](https://github.com/matrix-org/synapse/issues/10884),
[\#10896](https://github.com/matrix-org/synapse/issues/10896),
[\#10901](https://github.com/matrix-org/synapse/issues/10901))
- Allow the `.` and `~` characters when creating registration tokens as
per the change to
[MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
([\#10887](https://github.com/matrix-org/synapse/issues/10887))
- Clean up some unnecessary parentheses in places around the codebase.
([\#10889](https://github.com/matrix-org/synapse/issues/10889))
- Improve type hinting in the user directory code.
([\#10891](https://github.com/matrix-org/synapse/issues/10891))
- Update development testing script `test_postgresql.sh` to use a
supported Python version and make re-runs quicker.
([\#10906](https://github.com/matrix-org/synapse/issues/10906))
- Document and summarize changes in schema version `61` – `64`.
([\#10917](https://github.com/matrix-org/synapse/issues/10917))
- Update release script to sign the newly created git tags.
([\#10925](https://github.com/matrix-org/synapse/issues/10925))
- Fix Debian builds due to `dh-virtualenv` no longer being able to build
their docs.
([\#10931](https://github.com/matrix-org/synapse/issues/10931))
F124520CEEE062448FE1C8442D2EFA2F32FBE047
<olivier@librepush.net>" [ultimate]
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| | |
Introduced in #10905
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| | |
Various refactors to use `RoomVersion` objects instead of room version identifiers.
|
| | |
| |
| |
| | |
(#10924)
|
| | |
| |
| |
| |
| |
| |
| |
| | |
between batches (MSC2716) (#10877)
Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)
Part of https://github.com/matrix-org/synapse/issues/10737
|
| | |
| |
| |
| |
| |
| |
| | |
* update _handle_frozendict to work with c implementations of frozen dict
* add changelog
* add clarifying comment to _handle_frozendict
|
| | |
| |
| |
| |
| | |
Before Synapse 1.31 (#9411), we relied on `outlier` being stored in the
`internal_metadata` column. We can now assume nobody will roll back their
deployment that far and drop the legacy support.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Inline `_check_event_auth` for outliers
When we are persisting an outlier, most of `_check_event_auth` is redundant:
* `_update_auth_events_and_context_for_auth` does nothing, because the
`input_auth_events` are (now) exactly the event's auth_events,
which means that `missing_auth` is empty.
* we don't care about soft-fail, kicking guest users or `send_on_behalf_of`
for outliers
... so the only thing that matters is the auth itself, so let's just do that.
* `_auth_and_persist_fetched_events_inner`: de-async `prep`
`prep` no longer calls any `async` methods, so let's make it synchronous.
* Simplify `_check_event_auth`
We no longer need to support outliers here, which makes things rather simpler.
* changelog
* lint
|
| | |
| |
| |
| |
| |
| | |
By using attrs instead of dicts to store configuration.
Also updates some of the attrs classes to use proper type
hints and auto_attribs.
|
| |/
|
|
| |
Otherwise the presence of a (bogus, unused) field could cause
auth checks to fail.
|
| |
|
|
| |
This was due to dh-virtualenv builds being broken due to Shpinx removing
deprecated APIs.
|
| |
|
|
|
|
|
|
|
|
| |
Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library.
Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this:
1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or
2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types.
I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
|
| | |
|
| |
|
|
|
| |
URL cache files are short-lived and it does not make sense to offload
them (eg. to the cloud) or back them up.
|
| | |
|
| |
|
|
|
| |
As pointed out by @richvdh, https://github.com/matrix-org/synapse/pull/10838#discussion_r715424244
Retroactively summarize `61` - `64`
|
| |
|
|
|
|
|
| |
(#10898)
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.
This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Use sytest:bionic. Sytest:latest is two years old (do we want
CI to push out latest at all?) and comes with Python 3.5, which we
explictly no longer support. The script now runs under PostgreSQL 10
as a result.
- Advertise script in the docs
- Move pg testing script to scripts-dev directory
- Write to host as the script's exector, not root
A few changes to make it speedier to re-run the tests:
- Create blank DB in the container, not the script, so we don't have to
`initdb` each time
- Use a named volume to persist the tox environment, so we don't have to
fetch and install a bunch of packages from PyPI each time
Co-authored-by: reivilibre <olivier@librepush.net>
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Factor more stuff out of `_get_events_and_persist`
It turns out that the event-sorting algorithm in `_get_events_and_persist` is
also useful in other circumstances. Here we move the current
`_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`,
and then factor the sorting part out to `_auth_and_persist_fetched_events`.
* `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment
`get_event_auth` returns events with the outlier flag already set, so this is
redundant (though we need to update a test where `get_event_auth` is mocked).
* `_get_remote_auth_chain_for_event`: move existing-event tests earlier
Move a couple of tests outside the loop. This is a bit inefficient for now, but
a future commit will make it better. It should be functionally identical.
* `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events`
We can use the same codepath for persisting the events fetched as part of an
auth chain as for those fetched individually by `_get_events_and_persist` for
building the state at a backwards extremity.
* `_get_remote_auth_chain_for_event`: use a dict for efficiency
`_auth_and_persist_fetched_events` sorts the events itself, so we no longer
need to care about maintaining the ordering from `get_event_auth` (and no
longer need to sort by depth in `get_event_auth`).
That means that we can use a map, making it easier to filter out events we
already have, etc.
* changelog
* `_auth_and_persist_fetched_events`: improve docstring
|
| |
|
|
|
| |
Combine the two loops over the list of events, and hence get rid of
`_NewEventInfo`. Also pass the event back alongside the context, so that it's
easier to process the result.
|
| | |
|
| |
|
|
|
| |
If the MAU count had been reached, Synapse incorrectly blocked appservice users even though they've been explicitly configured not to be tracked (the default). This was due to bypassing the relevant if as it was chained behind another earlier hit if as an elif.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| |
|
|
|
|
|
|
|
|
|
| |
* Improve typing in user_directory files
This makes the user_directory.py in storage pass most of mypy's
checks (including `no-untyped-defs`). Unfortunately that file is in the
tangled web of Store class inheritance so doesn't pass mypy at the moment.
The handlers directory has already been mypyed.
Co-authored-by: reivilibre <olivier@librepush.net>
|
| |
|
|
|
|
| |
accessing rows (#10690)
This change adds a check for row existence before accessing row element, this should fix issue #10669
Signed-off-by: Vasya Boytsov vasiliy.boytsov@phystech.edu
|
| |
|
|
| |
Per updates to MSC3231 in order to use the same grammar
as other identifiers.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`_update_auth_events_and_context_for_auth` (#10884)
* Reload auth events from db after fetching and persisting
In `_update_auth_events_and_context_for_auth`, when we fetch the remote auth
tree and persist the returned events: load the missing events from the database
rather than using the copies we got from the remote server.
This is mostly in preparation for additional refactors, but does have an
advantage in that if we later get around to checking the rejected status, we'll
be able to make use of it.
* Factor out `_get_remote_auth_chain_for_event` from `_update_auth_events_and_context_for_auth`
* changelog
|
| | |
|
| |
|
| |
A follow-up to #10829
|
| |
|
|
|
| |
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| |
|
|
|
|
|
|
| |
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.
It also improves type hints since mypy can now resolve the
types of the configuration variables.
|
| | |
|
| |
|
|
|
|
| |
Constructing an EventContext for an outlier is actually really simple, and
there's no sense in going via an `async` method in the `StateHandler`.
This also means that we can resolve a bunch of FIXMEs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
indexing) (#10820)
* add test to check if null code points are being inserted
* add logic to detect and replace null code points before insertion into db
* lints
* add license to test
* change approach to null substitution
* add type hint for SearchEntry
* Add changelog entry
Signed-off-by: H.Shay <shaysquared@gmail.com>
* updated changelog
* update chanelog message
* remove duplicate changelog
* Update synapse/storage/databases/main/events.py remove extra space
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* rename and move test file, update tests, delete old test file
* fix typo in comments
* update _find_highlights_in_postgres to replace null byte with space
* replace null byte in sqlite search insertion
* beef up and reorganize test for this pr
* update changelog
* add type hints and update docstring
* check db engine directly vs using env variable
* refactor tests to be less repetetive
* move rplace logic into seperate function
* requested changes
* Fix typo.
* Update synapse/storage/databases/main/search.py
Co-authored-by: reivilibre <olivier@librepush.net>
* Update changelog.d/10820.misc
Co-authored-by: Aaron Raimist <aaron@raim.ist>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Aaron Raimist <aaron@raim.ist>
|
| |
|
|
|
|
|
| |
The invalidation was missing in `_claim_e2e_one_time_key_returning`,
which is used on SQLite 3.24+ and Postgres. This could break e2ee if
nothing else happened to invalidate the caches before the keys ran out.
Signed-off-by: Tulir Asokan <tulir@beeper.com>
|
| | |
|
| |
|
|
|
|
|
| |
* Improved titles (fall back to the author name if there's not title) and include the site name.
* Handle photo/video payloads.
* Include the original URL in the Open Graph response.
* Fix the expiration time (by properly converting from seconds to milliseconds).
|
| |
|
|
|
|
|
|
| |
The deprecated /initialSync endpoint maintains a cache of responses,
using parameter values as part of the cache key. When a `from` or `to`
parameter is specified, it gets converted into a `StreamToken`, which
contains a `RoomStreamToken` and forms part of the cache key.
`RoomStreamToken`s need to be made hashable for this to work.
|
| | |
|
| |
|
|
| |
* Allow LruCaches to opt out of time-based expiry
* Don't expire `get_users_who_share_room` & friends
|
| |
|
|
|
|
|
|
| |
resolves #10832 (#10833)
Extend ModuleApi with the methods we'll need to reject spam based on IP - resolves #10832
Signed-off-by: David Teller <davidt@element.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I meant to do this before, in #10591, but because I'm stupid I forgot to do it
for V2 and V3 events.
I've factored the common code out to `EventBase` to save us having two copies
of it.
This means that for `FrozenEvent` we replace `self.get("event_id", None)` with
`self.event_id`, which I think is safe. `get()` is an alias for
`self._dict.get()`, whereas `event_id()` is an `@property` method which looks
up `self._event_id`, which is populated during construction from the same
dict. We don't seem to rely on the fallback, because if the `event_id` key is
absent from the dict then construction of the `EventBase` object will
fail.
Long story short, the only way this could change behaviour is if
`event_dict["event_id"]` is changed *after* the `EventBase` object is
constructed without updating the `_event_id` field, or vice versa - either of
which would be very problematic anyway and the behavior of `str(event)` is the
least of our worries.
|
| |
|
| |
So we can see distinguish between "evicting because the cache is too big" and "evicting because the cache entries haven't been recently used".
|
| |
|
|
|
|
|
|
| |
endpoint (#10838)
See https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684574497
Dropping support for older MSC2716 room versions so we don't have to worry about
supporting both chunk and batch events.
|
| | |
|
| | |
|
| |
|
| |
This should fix stale deleted files being still accessible.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The major change is moving the decision of whether to use oEmbed
further up the call-stack. This reverts the _download_url method to
being a "dumb" functionwhich takes a single URL and downloads it
(as it was before #7920).
This also makes more minor refactorings:
* Renames internal variables for clarity.
* Factors out shared code between the HTML and rich oEmbed
previews.
* Fixes tests to preview an oEmbed image.
|
| |
|
|
|
|
|
| |
(#10835)
Test for #10830
Ideally the test would also make sure the new state event comes down sync, but this is probably good enough.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* add tests for checking if room search works with non-ascii char
* change encoding on parse_string to UTF-8
* lints
* properly encode search term
* lints
* add changelog file
* update changelog number
* set changelog entry filetype to .bugfix
* Revert "set changelog entry filetype to .bugfix"
This reverts commit be8e5a314251438ec4ec7dbc59ba32162c93e550.
* update changelog message and file type
* change parse_string default encoding back to ascii and update room search admin api calll to parse string
* refactor tests
* Update tests/rest/admin/test_room.py
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| |
|
|
|
|
|
| |
usage with `?prev_event_id` (MSC2716) (#10839)
As mentioned in https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r705872887
and https://github.com/matrix-org/synapse/issues/10737
|
| |\ |
|
| | | |
|
| | |
| |
| | |
one day I'll learn how to spell hierarchy
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It's a simplification, but one that'll help make the user directory logic easier
to follow with the other changes upcoming. It's not strictly required for those
changes, but this will help simplify the resulting logic that listens for
`m.room.member` events and generally make the logic easier to follow.
This means the config option `search_all_users` ends up controlling the
search query only, and not the data we store. The cost of doing so is an
extra row in the `user_directory` and `user_directory_search` tables for
each local user which
- belongs to no public rooms
- belongs to no private rooms of size ≥ 2
I think the cost of this will be marginal (since they'll already have entries
in `users` and `profiles` anyway).
As a small upside, a homeserver whose directory was built with this
change can toggle `search_all_users` without having to rebuild their
directory.
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
| |
| |
| |
| | |
* Allow membership event to unban user
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| | |
| |
| | |
This should ensure GHA runs synapse against the same-named sytest branch
|
| | |
| |
| |
| |
| |
| |
| | |
Adds missing type hints to methods in the synapse.handlers
module and requires all methods to have type hints there.
This also removes the unused construct_auth_difference method
from the FederationHandler.
|
| | | |
|
| | |
| |
| |
| | |
(#10845)
|
| |\| |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
We added a bunch of spans in #10704, but this ended up adding a lot of
redundant spans for rooms where nothing changed, so instead we only
start the span if there might be something interesting going on.
|
| | |
| |
| | |
To avoid duplicating it between a few tests.
|
| | | |
|
| | |
| |
| |
| | |
Including admin, consent, key, synapse, and media. All REST servlets
(the synapse.rest module) now require typed method definitions.
|
| | | |
|
| | |
| |
| |
| | |
(MSC2716) (#10776)
|
| | | |
|
| | |
| |
| | |
Also enables Mypy for related tests.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
In `MatrixFederationHttpClient._send_request()`, we make a HTTP request
using an `Agent`, wrap that request in a timeout and await the resulting
`Deferred`. On its own, the `Agent` performing the HTTP request
correctly stashes and restores the logging context while waiting.
The addition of the timeout introduces a path where the logging context
is not restored when execution resumes.
To address this, we wrap the timeout `Deferred` in a
`make_deferred_yieldable()` to stash the logging context and restore it
on completion of the `await`. However this is not sufficient, since by
the time we construct the timeout `Deferred`, the `Agent` has already
stashed and cleared the logging context when using
`make_deferred_yieldable()` to produce its `Deferred` for the request.
Hence, we wrap the `Agent` request in a `run_in_background()` to "fork"
and preserve the logging context so that we can stash and restore it
when `await`ing the timeout `Deferred`.
This approach is similar to the one used with `defer.gatherResults`.
Note that the code is still not fully correct. When a timeout occurs,
the request remains running in the background (existing behavior which
is nothing to do with the new call to `run_in_background`) and may
re-start the logging context after it has finished.
|
| |\|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.43.0rc1 (2021-09-14)
This release drops support for the deprecated, unstable API for [MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), as well as the undocumented `experimental.msc2858_enabled` config option. Client authors should update their clients to use the stable API, available since Synapse 1.30.
Features
--------
- Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) in existing room versions. ([\#10566](https://github.com/matrix-org/synapse/issues/10566))
- Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643))
- Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712))
- Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714), [\#10759](https://github.com/matrix-org/synapse/issues/10759))
- Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244) API. ([\#10772](https://github.com/matrix-org/synapse/issues/10772))
Bugfixes
--------
- Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658))
- Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings. ([\#10730](https://github.com/matrix-org/synapse/issues/10730))
- Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated). ([\#10743](https://github.com/matrix-org/synapse/issues/10743))
- Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757))
- Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr. ([\#10733](https://github.com/matrix-org/synapse/issues/10733))
- Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10760](https://github.com/matrix-org/synapse/issues/10760))
- Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774))
- Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image. ([\#10783](https://github.com/matrix-org/synapse/issues/10783))
Improved Documentation
----------------------
- Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556))
- Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648))
- Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735))
- Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758))
- Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html). ([\#10795](https://github.com/matrix-org/synapse/issues/10795))
- Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804))
Deprecations and Removals
-------------------------
- Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693))
Internal Changes
----------------
- Add OpenTracing logging to help debug stuck messages (as described by issue [#9424](https://github.com/matrix-org/synapse/issues/9424)). ([\#10704](https://github.com/matrix-org/synapse/issues/10704))
- Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601))
- Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) usage later. ([\#10697](https://github.com/matrix-org/synapse/issues/10697))
- Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707), [\#10728](https://github.com/matrix-org/synapse/issues/10728), [\#10736](https://github.com/matrix-org/synapse/issues/10736))
- Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727))
- Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738))
- Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744), [\#10745](https://github.com/matrix-org/synapse/issues/10745), [\#10746](https://github.com/matrix-org/synapse/issues/10746), [\#10771](https://github.com/matrix-org/synapse/issues/10771), [\#10773](https://github.com/matrix-org/synapse/issues/10773), [\#10781](https://github.com/matrix-org/synapse/issues/10781))
- Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748))
- Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted. ([\#10750](https://github.com/matrix-org/synapse/issues/10750))
- Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752))
- Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753))
- Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754), [\#10755](https://github.com/matrix-org/synapse/issues/10755), [\#10756](https://github.com/matrix-org/synapse/issues/10756), [\#10780](https://github.com/matrix-org/synapse/issues/10780), [\#10784](https://github.com/matrix-org/synapse/issues/10784))
- Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775))
- Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778))
- Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779))
- Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788))
- Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789))
- Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798))
- Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799))
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
This is mostly an internal change, but improves type hints in the
media code.
|
| |/
|
|
|
|
| |
I had one of these error messages yesterday and assumed it was an
invalid auth token (because that was an HTTP query parameter in the
test) I was working on. In fact, it was an invalid next batch token for
syncing.
|
| |
|
|
| |
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
|
| | |
|
| |
|
|
| |
This fixes mypy errors with jaeger-client >= 4.7.0 and should be a no-op
for versions before that.
|
| | |
|
| | |
|
| |
|
|
|
| |
By copying the `room_type` field of the create event and
migrating any non-empty `m.space.child` events to the
new room that is created.
|
| |
|
| |
No functional changes here. This came out as I was working to tackle #5677
|
| |
|
| |
We want to blacklist only known, consistent failures. We should deflake tests rather than ignoring them.
|
| |
|
|
|
|
| |
Fixes #10732: consent flow skipped during SSO user registration if username is left at default
Signed-off-by: Andrew Ferrazzutti fair@miscworks.net
|
| |
|
|
|
| |
We've already batched up the events previously, and assume in other
places in the events.py file that we have. Removing this makes it easier
to adjust the batch sizes in one place.
|
| |
|
|
|
| |
option (#10693)
Signed-off-by: Sean Quah <seanq@element.io>
|
| |
|
|
|
| |
Hint to clients via the room capabilities API (MSC3244) that
room version 9 should be preferred for creating a room with
restricted join rules (instead of room version 8).
|
| |
|
| |
This is only called in two places, and the code seems much clearer without it.
|
| |
|
|
|
|
| |
* Split up the documentation in several files rather than one huge one
* Add examples for each callback category
* Other niceties like fixing https://github.com/matrix-org/synapse/issues/10632
* Add titles to callbacks so they're easier to find in the navigation panels and link to
|
| | |
|
| |
|
| |
When releasing 1.42.0 with @Azrenbeth and talking with @clokep yesterday I realised doing the dch incantations related to releasing Synapse wasn't trivial on eg a macOS system, so this is a script to run in a Debian container to make things a bit easier.
|
| |
|
| |
Also refactors some of the registration of endpoints on workers.
|
| |
|
|
| |
Outlier events don't ever have push actions associated with them, so we
can skip some expensive queries during event persistence.
|
| | |
|
| |
|
| |
This used to be a comma and got accidentally changed to a period in #9654, but a pipe character is more easier to parse visually.
|
| |
|
|
|
|
|
|
| |
This adds the format to the request arguments / URL to
ensure that JSON data is returned (which is all that
Synapse supports).
This also adds additional error checking / filtering to the
configuration file to ignore XML-only providers.
|
| |
|
|
| |
I think I have finally teased apart the codepaths which handle outliers, and those that handle non-outliers.
Let's add some assertions to demonstrate my newfound knowledge.
|
| |
|
|
|
|
|
| |
If we're persisting an event E which has auth_events A1, A2, then we ought to make sure that we correctly auth
and persist A1 and A2, before we blindly accept E.
This PR does part of that - it persists the auth events first - but it does not fully solve the problem, because we
still don't check that the auth events weren't rejected.
|
| |\ |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
The full event content cannot be trusted from this API (as no auth
chain, etc.) is processed over federation. Returning the full event
content was a bug as MSC2946 specifies that only the stripped
state should be returned.
This also avoids calculating aggregations / annotations which go
unused.
|
| | | |
|
| |\|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.42.0rc2 (2021-09-06)
==============================
This version of Synapse removes deprecated room-management admin APIs, removes out-of-date
email pushers, and improves error handling for fallback templates for user-interactive
authentication. For more information on these points, server administrators are
encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
Features
--------
- Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375). ([\#10747](https://github.com/matrix-org/synapse/issues/10747))
Internal Changes
----------------
- Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768))
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| | |
The deprecation itself happened in #10596 which shipped with Synapse v1.41.0. However, it doesn't seem fair to suddenly drop support for these settings in ~4-6w without being more vocal about said deprecation.
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
It's now only used in a couple of places, so we can drop it altogether.
|
| | |
| |
| |
| | |
workers (#10757)
|
| | | |
|
| | | |
|
| | |
| |
| | |
This is part of my ongoing war against BaseHandler. I've moved kick_guest_users into RoomMemberHandler (since it calls out to that handler anyway), and split maybe_kick_guest_users into the two places it is called.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
(#10566)
* Allow room creator to send MSC2716 related events in existing room versions
Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869
Restoring `get_create_event_for_room_txn` from,
https://github.com/matrix-org/synapse/pull/10245/commits/44bb3f0cf5cb365ef9281554daceeecfb17cc94d
* Add changelog
* Stop people from trying to redact MSC2716 events in unsupported room versions
* Populate rooms.creator column for easy lookup
> From some [out of band discussion](https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`).
>
> https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642
* Remove and switch away from get_create_event_for_room_txn
* Fix no create event being found because no state events persisted yet
* Fix and add tests for rooms creator bg update
* Populate rooms.creator field for easy lookup
Part of https://github.com/matrix-org/synapse/pull/10566
- Fill in creator whenever we insert into the rooms table
- Add background update to backfill any missing creator values
* Add changelog
* Fix usage
* Remove extra delta already included in #10697
* Don't worry about setting creator for invite
* Only iterate over rows missing the creator
See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898
* Use constant to fetch room creator field
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029
* More protection from other random types
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853
* Move new background update to end of list
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181
* Fix query casing
* Fix ambiguity iterating over cursor instead of list
Fix `psycopg2.ProgrammingError: no results to fetch` error
when tests run with Postgres.
```
SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room
```
---
We use `txn.fetchall` because it will return the results as a
list or an empty list when there are no results.
Docs:
> `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used:
>
> https://www.psycopg.org/docs/cursor.html#cursor-iterable
And I'm guessing iterating over a raw cursor does something weird when there are no results.
---
Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530
```
tests.test_visibility.FilterEventsForServerTestCase.test_large_room
===============================================================================
[FAIL]
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column
self.get_success(
File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success
return self.successResultOf(d)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf
self.fail(
twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead:
Traceback (most recent call last):
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback
self._startRunCallbacks(fail)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
self._runCallbacks()
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult
current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
result = current_context.run(
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update
await self._do_background_update(desired_duration_ms)
File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update
items_updated = await update_handler(progress, batch_size)
File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column
end = await self.db_pool.runInteraction(
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction
result = await self.runWithConnection(
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection
return await make_deferred_yieldable(
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda>
d.addCallback(lambda x: function(*args, **kwargs))
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
return function(*args, **kwargs)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
raise exception.with_traceback(traceback)
File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
result = func(conn, *args, **kw)
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func
return func(db_conn, *args, **kwargs)
File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction
r = func(cursor, *args, **kwargs)
File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn
for room_id, event_json in txn:
psycopg2.ProgrammingError: no results to fetch
```
* Move code not under the MSC2716 room version underneath an experimental config option
See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909
* Add ordering to rooms creator background update
See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277
* Add comment to better document constant
See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458
* Use constant field
|
| | | |
|
| | |
| |
| | |
Signed-off-by: Sean Quah <seanq@element.io>
|
| | |
| |
| | |
`execute_values` is a faster version of `execute_batch`.
|
| | |
| |
| |
| | |
Raises a 400 error instead of a 500 if an unknown preset is passed
from a client to create a room.
|
| | |
| |
| |
| | |
Additionally this enforce type hints on all function signatures inside
of the synapse.rest.client package.
|
| | |
| |
| |
| | |
This will only happen when a server has multiple out of band membership
events in a single room.
|
| | | |
|
| | |
| |
| |
| | |
This avoids breaking the entire endpoint if a room with
an unsupported room version is encountered.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
rooms. (#10730)
This updates the ordering of the returned events from the spaces
summary API to that defined in MSC2946 (which updates MSC1772).
Previously a step was skipped causing ordering to be inconsistent with
clients.
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
Part of https://github.com/matrix-org/synapse/pull/10566
- Fill in creator whenever we insert into the rooms table
- Add background update to backfill any missing creator values
|
| |\|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.42.0rc1 (2021-09-01)
==============================
Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
Features
--------
- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142))
- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524))
- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725))
Bugfixes
--------
- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713))
Improved Documentation
----------------------
- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711))
Deprecations and Removals
-------------------------
- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830))
Internal Changes
----------------
- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608))
- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621))
- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627))
- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645))
- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647))
- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662))
- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664))
- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692))
- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))
|
| | | |
|
| | |
| |
| |
| | |
Expand OIDC to OpenID Connect.
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Judging by the template, this was intended ages ago, but we never
actually passed an avatar URL to the template. So let's provide one.
Closes #1546.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Use `gc.freeze()` on exit to exclude all existing objects from the final GC.
In testing, this sped up shutdown by up to a few seconds.
`gc.freeze()` runs in constant time, so there is little chance of performance
regression.
Signed-off-by: Sean Quah <seanq@element.io>
|
| | |
| |
| |
| |
| | |
This adds configuration options (under an `oembed` section) to
configure which URLs are matched to use oEmbed for URL
previews.
|
| |/ |
|
| |
|
| |
This was erroneously put under schema version 62 instead of 63.
|
| |\ |
|
| | | |
|
| | |
| |
| | |
These were broken in an incorrect merge of GHSA-jj53-8fmw-f2w2 (cb35df9)
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Add some tests to characterise the problem
Some failing. Current states:
RoomsMemberListTestCase
test_get_member_list ...
[OK]
test_get_member_list_mixed_memberships ...
[OK]
test_get_member_list_no_permission ...
[OK]
test_get_member_list_no_permission_former_member ...
[OK]
test_get_member_list_no_permission_former_member_with_at_token ...
[FAIL]
test_get_member_list_no_room ...
[OK]
test_get_member_list_no_permission_with_at_token ...
[FAIL]
* Correct the tests
* Check user is/was member before divulging room membership
* Pull out only the 1 membership event we want.
* Update tests/rest/client/v1/test_rooms.py
Co-authored-by: Erik Johnston <erik@matrix.org>
* Fixup tests (following apply review suggestion)
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| | |
| |
| |
| |
| | |
Point to the book where possible, and use hyperlinks to github to refer to files not included in the book.
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
|
| |\| |
|
| | |
| |
| |
| |
| |
| |
| | |
Turns out that the functionality added in #10546 to skip TLS was incompatible
with older Twisted versions, so we need to be a bit more inventive.
Also, add a test to (hopefully) not break this in future. Sadly, testing TLS is
really hard.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
pages (#10711)
- Removed page summaries from CONTRIBUTING and installation pages as
this information was already in the table of contents on the right hand side
- Fixed some broken links in CONTRIBUTING
- Added margin-right tag for when table of contents is being shown
(otherwise the text in the page sometimes overlaps with it)
|
| | |
| |
| |
| |
| | |
* Update reverse_proxy.md
* Create 10708.doc
|
| | | |
|
| | |
| |
| | |
Fixes: #9544
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The code to deduplicate repeated fetches of the same set of events was
N^2 (over the number of events requested), which could lead to a process
being completely wedged.
The main fix is to deduplicate the returned deferreds so we only await
on a deferred once rather than many times. Seperately, when handling the
returned events from the defrered we only add the events we care about
to the event map to be returned (so that we don't pay the price of
inserting extraneous events into the dict).
|
| | |
| |
| | |
The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.
|
| | |
| |
| | |
Given that backfill and get_missing_events are basically the same thing, it's somewhat crazy that we have entirely separate code paths for them. This makes backfill use the existing get_missing_events code, and then clears up all the unused code.
|
| | |
| |
| |
| | |
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| | |
| |
| |
| |
| | |
When a user deletes an email from their account it will
now also remove all pushers for that email and that user
(even if these pushers were created by a different client)
|
| | |
| |
| | |
Applies the changes from #10665 to additional modules.
|
| | | |
|
| | |
| |
| | |
I found this easy to miss (and evidently, it looks like it was missed for schema version 62).
|
| | |
| |
| |
| |
| |
| | |
workers. (#10686)
We now always rebuild the matrixdotorg/synapse image, then
build the matrixdotorg/synapse-workers image on top of it.
|
| | |
| |
| | |
Fixes #10318
|
| |\| |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
Ubuntu 20.10 was not an LTS release
Signed-off-by: John-Scott Atlakson 24574+jsma@users.noreply.github.com
|
| | | |
|
| | |
| |
| |
| |
| | |
Fix a bug where the prometheus metrics for SSO logins wouldn't be initialised
until the first user logged in with a given auth provider.
|
| | |
| |
| |
| | |
Signed-off-by: Hugo Delval <hugo.delval@gmail.com>
|
| | |
| |
| | |
To match the maximum lengths allowed for profile data.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Fix the titles in the OIDC documentation
Having them as links broke the table-of-contents rendering in mdbook.
Plus there's no reason for only some of the provider titles to be links.
* Changelog
* Add link to google idp docs
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Setting `update_existing: true` in the `create-an-issue` GitHub Action
will avoid opening duplicate issues if an open issue already exists with
an identical title.
If no open issues match the title, then a new issue will be created.
This helps avoid spamming our issue tracker should there be a failure
when testing against Twisted's trunk.
This PR also pins the SHA of the `create-an-issue` action to mitigate
the risk of a malicious actor gaining access to JasonEtco's account.
See GitHub's page on security hardening third party actions for more:
https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions
Signed-off-by: Dan Callahan <danc@element.io>
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
federation destinations. (#10662)
|
| | |
| |
| | |
This creates a GHA workflow which runs at 8am every day, and runs mypy, trial and sytest against Twisted's current trunk. If any of the jobs fail, it opens an issue.
|
| | |
| |
| |
| |
| | |
Signed-off-by: Callum Brown <callum@calcuode.com>
This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
|
| | | |
|
| | |
| |
| |
| | |
By replacing duplicated code with parameterized tests and
avoiding unnecessary dumping of JSON data.
|
| | |
| |
| |
| | |
Merged before approval; these comments from @clokep on that PR.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
* Validate device_keys for C-S /keys/query requests
Closes #10354
A small, not particularly critical fix. I'm interested in seeing if we
can find a more systematic approach though. #8445 is the place for any discussion.
|
| | |
| |
| | |
Here we split on_receive_pdu into two functions (on_receive_pdu and process_pulled_event), rather than having both cases in the same method. There's a tiny bit of overlap, but not that much.
|
| | |
| |
| | |
This is a follow-up to #10615: it takes the code that constructs the state at a backwards extremity, and extracts it to a separate method.
|
| | |
| |
| |
| | |
A user will still see this room if it is in a local cache, but it will
not reappear if clearing the cache and reloading.
|
| | | |
|
| | | |
|
| | | |
|
| |\| |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
only (#10628)
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
|
| | | |
|
| | |
| |
| |
| | |
By default the calls only ran on the worker configured to run background
tasks.
|
| | |
| |
| | |
Fixes https://github.com/matrix-org/synapse/issues/7867
|
| | |
| |
| | |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | | |
|
| | |
| |
| | |
Several configuration sections are using separate settings for custom template directories, which can be confusing. This PR adds a new top-level configuration for a custom template directory which is then used for every module. The only exception is the consent templates, since the consent template directory require a specific hierarchy, so it's probably better that it stays separate from everything else.
|
| | |
| |
| |
| |
| |
| |
| | |
If the new /hierarchy API does not exist on all destinations,
fallback to querying the /spaces API and translating the results.
This is a backwards compatibility hack since not all of the
federated homeservers will update at the same time.
|
| | | |
|
| | |
| |
| | |
Follow-up to #10629 which set it to true, not false.
|
| | |
| |
| |
| | |
Instead of using namedtuples. This helps with asserting type hints
and code completion.
|
| | | |
|
