| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
| |
Setting the value will help PostgreSQL free up memory by recycling
the connections in the connection pool.
Signed-off-by: Toni Spets <toni.spets@iki.fi>
|
| |
|
|
|
| |
Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to
follow suit.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
scrollback history (MSC2716) (#10245)
* Make historical messages available to federated servers
Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
Follow-up to https://github.com/matrix-org/synapse/pull/9247
* Debug message not available on federation
* Add base starting insertion point when no chunk ID is provided
* Fix messages from multiple senders in historical chunk
Follow-up to https://github.com/matrix-org/synapse/pull/9247
Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
---
Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.
* Remove debug lines
* Messing with selecting insertion event extremeties
* Move db schema change to new version
* Add more better comments
* Make a fake requester with just what we need
See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080
* Store insertion events in table
* Make base insertion event float off on its own
See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889
Conflicts:
synapse/rest/client/v1/room.py
* Validate that the app service can actually control the given user
See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455
Conflicts:
synapse/rest/client/v1/room.py
* Add some better comments on what we're trying to check for
* Continue debugging
* Share validation logic
* Add inserted historical messages to /backfill response
* Remove debug sql queries
* Some marker event implemntation trials
* Clean up PR
* Rename insertion_event_id to just event_id
* Add some better sql comments
* More accurate description
* Add changelog
* Make it clear what MSC the change is part of
* Add more detail on which insertion event came through
* Address review and improve sql queries
* Only use event_id as unique constraint
* Fix test case where insertion event is already in the normal DAG
* Remove debug changes
* Switch to chunk events so we can auth via power_levels
Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.
So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.
* Switch to chunk events for federation
* Add unstable room version to support new historical PL
* Fix federated events being rejected for no state_groups
Add fix from https://github.com/matrix-org/synapse/pull/10439
until it merges.
* Only connect base insertion event to prev_event_ids
Per discussion with @erikjohnston,
https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org
* Make it possible to get the room_version with txn
* Allow but ignore historical events in unsupported room version
See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489
We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either.
Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`.
* Move to unique index syntax
See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509
* High-level document how the insertion->chunk lookup works
* Remove create_event fallback for room_versions
See https://github.com/matrix-org/synapse/pull/10245/files#r677641879
* Use updated method name
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.39.0rc3 (2021-07-28)
==============================
Bugfixes
--------
- Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477))
- Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485))
- Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486))
Internal Changes
----------------
- Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461))
- Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
|
| | |
| |
| | |
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
|
| | |
| |
| |
| | |
If the staging area was empty we'd report an age of 51 years, which is
not true or helpful.
|
| | | |
|
| | |
| |
| | |
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Now that we have `simple_upsert` that should be used in preference to
trying to insert and looking for an exception. The main benefit is that
we ERROR message don't get written to postgres logs.
We also have tidy up the return value on `simple_upsert`, rather than
having a tri-state of inserted/not-inserted/unknown.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
Improves type hints for:
* parse_{boolean,integer}
* parse_{boolean,integer}_from_args
* parse_json_{value,object}_from_request
And fixes any incorrect calls that resulted from unknown types.
|
| |/
|
|
| |
And set the required attribute in a few places which will error if
a parameter is not provided.
|
| | |
|
| |
|
|
|
| |
impression of retention given by the old R30 metric (#10332)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
|
| |
|
|
|
|
|
|
|
| |
This PR is tantamount to running
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```
Part of #9744
|
| |
|
|
|
| |
It looks like it was first used and introduced in https://github.com/matrix-org/synapse/commit/5130d80d79fe1f95ce03b8f1cfd4fbf0a32f5ac8#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305 but the
But the usage was removed in https://github.com/matrix-org/synapse/commit/4c6a31cd6efa25be4c9f1b357e8f92065fac63eb#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8
|
| | |
|
| |
|
|
|
|
|
|
| |
The postgres statistics collector sometimes massively underestimates the
number of distinct state groups are in the `state_groups_state`, which
can cause postgres to use table scans for queries for multiple state
groups.
We fix this by manually setting `n_distinct` on the column.
|
| |
|
|
|
|
|
|
|
|
| |
Previously only world-readable rooms were shown. This means that
rooms which are public, knockable, or invite-only with a pending invitation,
are included in a space summary. It also applies the same logic to
the experimental room version from MSC3083 -- if a user has access
to the proper allowed rooms then it is shown in the spaces summary.
This change is made per MSC3173 allowing stripped state of a room to
be shown to any potential room joiner.
|
| |
|
| |
We should be reporting the age rather than absolute timestamp.
|
| |
|
|
|
| |
while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth
with a BIGINT.
|
| |
|
|
|
| |
knocking and room statistics are in use. (#10344)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Synapse 1.38.0rc2 (2021-07-09)
==============================
Bugfixes
--------
- Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336))
Improved Documentation
----------------------
- Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287))
- Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Upsert redactions in case they already exists
Occasionally, in combination with retention, redactions aren't deleted
from the database whenever they are due for deletion. The server will
eventually try to backfill the deleted events and trip over the already
existing redaction events.
Switching to an UPSERT for those events allows us to recover from there
situations. The retention code still needs fixing but that is outside of
my current comfort zone on this code base.
This is related to #8707 where the error was discussed already.
Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
* Also purge redactions when purging events
Previously redacints where left behind leading to backfilling issues
when the server stumbled across the already existing yet to be
backfilled redactions.
This issues has been discussed in #8707.
Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
|
| | |
| |
| | |
Fixes #9602
|
| |/
|
|
|
|
|
| |
* Use fake time in tests in _get_start_of_day.
* Change the inequality of last_seen in user_daily_visits
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
| |
Fixes #10325
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
We might have events in the staging area if the service was restarted while there were unhandled events in the staging area.
Fixes #10295
|
| |
|
|
|
| |
this was a typo introduced in #10282. We don't want to end up doing the
`replace_stream_ordering_column` update after anything that comes up in
migration 60/03.
|
| | |
|
| | |
|
| |
|
|
| |
We need to rebuild *all* of the indexes that use the current `stream_ordering`
column.
|
| |
|
| |
This broke in #10272
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes #9490
This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous.
Two drawbacks:
Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately.
We now only process one event per room at a time. This can be fixed up further down the line.
|
| | |
| |
| | |
This adds a simple best effort locking mechanism that works cross workers.
|
| | |
| |
| |
| | |
It was possible for us not to find any rows in a batch, and hence conclude that
we had finished. Let's not do that.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Move background update names out to a separate class
`EventsBackgroundUpdatesStore` gets inherited and we don't really want to
further pollute the namespace.
* Migrate stream_ordering to a bigint
* changelog
|
| |/
|
|
|
|
|
|
|
|
| |
This implements refresh tokens, as defined by MSC2918
This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235
The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.
Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
|
| |
|
| |
Introduced in #6739
|
| |
|
| |
Work on https://github.com/matrix-org/matrix-doc/pull/2716
|
| |
|
|
|
| |
We were repeatedly looking up a config option in a loop (using the
unclassed config style), which is expensive enough that it can cause
large CPU usage.
|
| |
|
|
|
|
|
|
|
| |
Fixes https://github.com/matrix-org/synapse/issues/10030.
We were expecting milliseconds where we should have provided a value in seconds.
The impact of this bug isn't too bad. The code is intended to count the number of remote servers that the homeserver can see and report that as a metric. This metric is supposed to run initially 1 second after server startup, and every 60s as well. Instead, it ran 1,000 seconds after server startup, and every 60s after startup.
This fix allows for the correct metrics to be collected immediately, as well as preventing a random collection 1,000s in the future after startup.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Trace event persistence
When we persist a batch of events, set the parent opentracing span to the that
from the request, so that we can trace all the way in.
* changelog
* When we force tracing, set a baggage item
... so that we can check again later.
* Link in both directions between persist_events spans
|
| |
|
| |
some cleanup, pulled out of #10134.
|
| |
|
| |
This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
|
| |
|
| |
Hopefully fixes #10027.
|
| |
|
|
|
|
|
|
|
|
| |
This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403.
Specifically, it:
* Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory
* Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room
Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72
|
| |
|
|
|
|
| |
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403
Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
|
| |
|
|
| |
Fixes #10140
|
| |
|
| |
Add a couple of extra logs/spans, to give a bit of a better idea.
|
| |
|
|
|
|
|
|
|
|
| |
This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels:
* emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions.
* emit a span while we hold a database connection open
* emit a span for each database transaction - actual actual transaction.
* emit a span for each database query.
I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
|
| |
|
|
|
| |
Related to: #6681, #5956, #10040
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
| |
|
| |
Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
|
| |
|
|
|
|
| |
Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414
See #8551
Signed-off-by: Callum Brown <callum@calcuode.com>
|
| |
|
|
|
|
|
|
|
|
| |
* Make `invalidate` and `invalidate_many` do the same thing
... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.
* Kill off `invalidate_many`
* changelog
|
| |
|
| |
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
| | |
|
| |
|
|
|
|
|
| |
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.
The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).
Commits should be independently reviewable.
|
| |
|
|
|
| |
actually does (#10050)
It looks like a typo copy/paste from `get_state_for_event` above.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Fixes: https://github.com/matrix-org/synapse/issues/9962
This is a fix for above problem.
I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts.
Signed-off-by: Marek Matys <themarcq@gmail.com>
|
| |
|
|
| |
to them, instead of something in-memory (#9823)
|
| |
|
|
|
|
| |
- use a tuple rather than a list for the iterable that is passed into the
wrapped function, for performance
- test that we can pass an iterable and that keys are correctly deduped.
|
| |
|
|
|
|
|
|
| |
It's not obvious that instances of SQLBaseStore each need their own
instances of random.SystemRandom(); let's just use random directly.
Introduced by 52839886d664576831462e033b88e5aba4c019e3
Signed-off-by: Dan Callahan <danc@element.io>
|
| |
|
|
|
| |
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.
Signed-off-by: Aaron Raimist <aaron@raim.ist>
|
| |
|
|
|
| |
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
|
| |
|
|
|
| |
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.
It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
|
| | |
|
| |
|
|
|
| |
This fixes a regression where the logging context for runWithConnection
was reported as runWithConnection instead of the connection name,
e.g. "POST-XYZ".
|
| |
|
|
| |
This reverts commit 05e8c70c059f8ebb066e029bc3aa3e0cefef1019.
|
| |
|
| |
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
|
| | |
|
| | |
|
| |
|
| |
Fixes #9866.
|
| |
|
| |
This is no longer required, since we have dropped support for Python 3.5.
|
| | |
|
| |
|
|
|
| |
hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
|
| |
|
|
|
| |
Fixes: #9797.
Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
|
| |
|
|
|
| |
There are a couple of points in `persist_events` where we are doing a
query per event in series, which we can replace.
|
| |
|
|
|
| |
This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max).
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
| |
|
|
|
|
|
| |
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
|
| |
|
| |
Signed-off-by: Dan Callahan <danc@element.io>
|
| | |
|
| |
|
|
|
|
|
| |
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key`
constraint is superceded by
`remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds
`thumbnail_method` to the unique key).
PR #7124 made an attempt to remove the old constraint, but got the name wrong,
so it didn't work. Here we update the bg update and rerun it.
Fixes #8649.
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
| |
Fixes #9642.
Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
|
| |
|
|
|
|
|
| |
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
|
| | |
|
| |
|
|
|
|
|
| |
Split off from https://github.com/matrix-org/synapse/pull/9491
Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133
Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* Populate `internal_metadata.outlier` based on `events` table
Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.
* Move `outlier` out of InternalMetadata._dict
Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
|
| |
|
|
|
|
|
|
|
|
| |
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.
Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.
This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)
The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).
Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
|
| |
|
|
|
|
| |
... because namedtuples suck
Fix up a couple of other annotations to keep mypy happy.
|
| |
|
|
| |
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
|
| | |
|
| |
|
|
|
| |
This is a companion change to apply the fix in #9498 /
922788c6043138165c025c78effeda87de842bab to previously
purged rooms.
|
| |
|
|
| |
Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
It landed in schema version 58 after 59 had been created, causing some
servers to not run it. The main effect of was that not all rooms had
their chain cover calculated correctly. After the BG updates complete
the chain covers will get fixed when a new state event in the affected
rooms is received.
|
| |
|
|
| |
Fixes #9504
|
| | |
|
| | |
|
| |
|
| |
The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
|
| |
|
| |
And ensure the consistency of `event_auth_chain_id`.
|
| | |
|
| |
|
|
|
|
|
| |
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.
The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.
Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
|
| |\ |
|
| | | |
|
| |/
|
|
|
|
|
|
|
| |
results (#9383)
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.
Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.
This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
|
| |
|
|
|
|
| |
(#9428)
As the comment says, this guard was there for when the
initial user directory update has yet to happen.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
- Update `code_style.md` docs around installing black to use the correct version
|
| | |
|
| |
|
|
|
|
|
| |
(#9003)
Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests.
Fix #8996
|
| | |
|
| |
|
|
| |
* Adds proper dependencies.
* Minor fixes in database layer.
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Fixes #8966.
* Factor out build_synapse_client_resource_tree
Start a function which will mount resources common to all workers.
* Move sso init into build_synapse_client_resource_tree
... so that we don't have to do it for each worker
* Fix SSO-login-via-a-worker
Expose the SSO login endpoints on workers, like the documentation says.
* Update workers config for new endpoints
Add documentation for endpoints recently added (#8942, #9017, #9262)
* remove submit_token from workers endpoints list
this *doesn't* work on workers (yet).
* changelog
* Add a comment about the odd path for SAML2Resource
|
| | |\
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
|
| | | |
| | |
| | | |
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
|
| | | |
| | |
| | | |
This finishes adding type hints to the `synapse.handlers` module.
|
| | | |
| | |
| | |
| | | |
With this change all handlers except the e2e_* ones have
type hints enabled.
|
| | | |
| | |
| | |
| | | |
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| |\ \ \
| | | |
| | | |
| | | |
| | | | |
# Conflicts:
# synapse/rest/admin/__init__.py
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.
A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.
Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
|
| | | | |
| | | |
| | | | |
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |\| | | |
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
We have seen a failure mode here where if there are many in flight
unfinished IDs then marking an ID as finished takes a lot of CPU (as
calling deque.remove iterates over the list)
|
| | |\ \ \
| | | |/
| | |/| |
|
| | | | | |
|
| | |\| | |
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
Introduced in #9104
This wasn't picked up by the tests as this is all fine the first time you run Synapse (after upgrading), but then when you restart the wrong value is pulled from `stream_positions`.
|
| | | |/
| |/|
| | |
| | |
| | | |
* Use execute_batch in more places
* Newsfile
|
| | |/
| |
| | |
`execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| | |
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
|
| | |
| |
| |
| |
| | |
I don't think there's any need to use canonicaljson here.
Fixes: #4475.
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
(#9115)
|
| | |
| |
| |
| | |
t was doing a sequential scan on `destination_rooms`, which took
minutes.
|
| | | |
|
| | | |
|
| | |
| |
| | |
This only applies if the user's data is to be erased.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
These tables are unused, and can be dropped now the schema version has been bumped.
|
| | |
| |
| | |
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
| |
| |
| |
| | |
* Dropped last_used column from access_tokens
Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
|
| | |
| |
| |
| |
| |
| | |
Also add a warning on the admin API documentation.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| | |
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| |
| |
| | |
As per feedback.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* docs updates
* prettify SQL
* add missing copyright
* cursor_to_dict
* update touched files copyright years
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| | |
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| |\| |
|
| | |
| |
| |
| |
| | |
You can't continue using a transaction once an exception has been
raised, so catching and dropping the error here is pointless and just
causes more errors.
|
| | |
| |
| |
| |
| |
| | |
Also run linter.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
| |
| |
| | |
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| |/
|
|
|
|
| |
GET /_synapse/admin/v1/rooms/<identifier>/forward_extremities now gets forward extremities for a room, returning count and the list of extremities.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
|
| | |
|
| | |
|
| |
|
|
|
|
| |
This allows for efficiently finding which users ignore a particular
user.
Co-authored-by: Erik Johnston <erik@matrix.org>
|
| |
|
|
| |
This can happen when using a split out state database and we've upgraded
the schema version without there being any changes in the state schema.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
| |
This table has been unused since Synapse v1.17.0.
|
| | |
|
| |
|
|
| |
If we see stale extremities while persisting events, and notice that
they don't change the result of state resolution, we drop them.
|
| |
|
|
|
| |
* Use the simple dictionary in fts for the user directory
* Clarify naming
|
| | |
|
| |
|
| |
This improves type hinting and should use less memory.
|
| | |
|
| |
|
|
| |
Removes faulty assertions and fixes the logic to ensure the max
stream token is always set.
|
| | |
|
| |
|
|
|
| |
This is so that we can choose which algorithm to use based on the room ID.
|
| |
|
|
|
|
|
| |
During user-interactive auth, do not offer password auth to users with no
password, nor SSO auth to users with no SSO.
Fixes #7559.
|
| |
|
|
|
|
|
| |
It's important that we make sure our background updates happen in a defined
order, to avoid disasters like #6923.
Add an ordering to all of the background updates that have landed since #7190.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This is another PR that grew out of #6739.
The existing code for checking whether a user is currently invited to a room when they want to leave the room looks like the following:
https://github.com/matrix-org/synapse/blob/f737368a26bb9eea401fcc3a5bdd7e0b59e91f09/synapse/handlers/room_member.py#L518-L540
It calls `get_invite_for_local_user_in_room`, which will actually query *all* rooms the user has been invited to, before iterating over them and matching via the room ID. It will then return a tuple of a lot of information which we pull the event ID out of.
I need to do a similar check for knocking, but this code wasn't very efficient. I then tried to write a different implementation using `StateHandler.get_current_state` but this actually didn't work as we haven't *joined* the room yet - we've only been invited to it. That means that only certain tables in Synapse have our desired `invite` membership state. One of those tables is `local_current_membership`.
So I wrote a store method that just queries that table instead
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
no stream_id is stored. (#8744)
* Make this line debug (it's noisy)
* Don't include from_key for presence if we are at 0
* Limit read receipts for all rooms to 100
* changelog.d/8744.bugfix
* Allow from_key to be None
* Update 8744.bugfix
* The from_key is superflous
* Update comment
|
| | |
|
| |
|
|
|
|
|
|
|
| |
There's a handy function called maybe_store_room_on_invite which allows us to create an entry in the rooms table for a room and its version for which we aren't joined to yet, but we can reference when ingesting events about.
This is currently used for invites where we receive some stripped state about the room and pass it down via /sync to the client, without us being in the room yet.
There is a similar requirement for knocking, where we will eventually do the same thing, and need an entry in the rooms table as well. Thus, reusing this function works, however its name needs to be generalised a bit.
Separated out from #6739.
|
| |
|
|
|
| |
This should hopefully speed up `get_auth_chain_difference` a bit in the case of repeated state res on the same rooms.
`get_auth_chain_difference` does a breadth first walk of the auth graphs by repeatedly looking up events' auth events. Different state resolutions on the same room will end up doing a lot of the same event to auth events lookups, so by caching them we should speed things up in cases of repeated state resolutions on the same room.
|
| |
|
|
|
|
| |
`adbapi.ConnectionPool` let's you turn on auto reconnect of DB connections. This is off by default.
As far as I can tell if its not enabled dead connections never get removed from the pool.
Maybe helps #8574
|
| |
|
|
|
|
|
|
| |
Add `GET /_synapse/admin/v1/statistics/users/media` to get statisics about local media usage by users.
Related to #6094
It is the first API for statistics.
Goal is to avoid/reduce usage of sql queries like [Wiki analyzing Synapse](https://github.com/matrix-org/synapse/wiki/SQL-for-analyzing-Synapse-PostgreSQL-database-stats)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
| |
|
| |
We do a `SELECT MAX(stream_id) FROM e2e_cross_signing_keys` on startup.
|
| | |
|
| |
|
| |
Fixes #6755
|
| |
|
| |
Missed in #8671.
|
| |
|
|
|
|
|
|
|
|
| |
another user. (#8616)
We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't).
A future PR will add an API for creating such a token.
When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
|
| |\
| |
| | |
Fix serialisation errors when using third-party event rules.
|
| | |
| |
| |
| |
| |
| | |
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
|
| | | |
|
| | |
| |
| |
| |
| | |
I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | |
| |
| |
| |
| |
| |
| |
| | |
This allows trailing commas in multi-line arg lists.
Minor, but we might as well keep our formatting current with regard to
our minimum supported Python version.
Signed-off-by: Dan Callahan <danc@element.io>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is a requirement for [knocking](https://github.com/matrix-org/synapse/pull/6739), and is abstracting some code that was originally used by the invite flow. I'm separating it out into this PR as it's a fairly contained change.
For a bit of context: when you invite a user to a room, you send them [stripped state events](https://matrix.org/docs/spec/server_server/unstable#put-matrix-federation-v2-invite-roomid-eventid) as part of `invite_room_state`. This is so that their client can display useful information such as the room name and avatar. The same requirement applies to knocking, as it would be nice for clients to be able to display a list of rooms you've knocked on - room name and avatar included.
The reason we're sending membership events down as well is in the case that you are invited to a room that does not have an avatar or name set. In that case, the client should use the displayname/avatar of the inviter. That information is located in the inviter's membership event.
This is optional as knocks don't really have any user in the room to link up to. When you knock on a room, your knock is sent by you and inserted into the room. It wouldn't *really* make sense to show the avatar of a random user - plus it'd be a data leak. So I've opted not to send membership events to the client here. The UX on the client for when you knock on a room without a name/avatar is a separate problem.
In essence this is just moving some inline code to a reusable store method.
|
| | |
| |
| | |
Add admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information of users' uploaded files.
|
| | |
| |
| | |
This also fixes a bug by fixing handling of an account which doesn't expire.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Split admin API for reported events in detail und list view.
API was introduced with #8217 in synapse v.1.21.0.
It makes the list (`GET /_synapse/admin/v1/event_reports`) less complex and provides a better overview.
The details can be queried with: `GET /_synapse/admin/v1/event_reports/<report_id>`.
It is similar to room and users API.
It is a kind of regression in `GET /_synapse/admin/v1/event_reports`. `event_json` was removed. But the api was introduced one version before and it is an admin API (not under spec).
Signed-off-by: Dirk Klimpel dirk@klimpel.org
|
| | |
| |
| |
| |
| |
| | |
Related to: #6459, #3479
Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete
a single file from server.
|
| |\| |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
* Fix user_daily_visits to not have duplicate rows for UA.
Fixes #8641.
* Newsfile
* Fix typo.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* Move schema file
* Add a .
* Add matching changelog entry
* Fix sqlite
|
| |
|
| |
Signed-off-by: Vasilis Gerakaris <vasilis.gerakaris@navarino.gr>
|
| |
|
|
|
|
|
|
|
|
|
| |
* Add `DeferredCache.get_immediate` method
A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.
* Remove unused 'default' parameter to DeferredCache.get()
* another get_immediate instance
|
| |
|
| |
Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
|
| |
|
|
|
| |
(#8577)
Fix the Connection protocol according to typeshed's assertions about sqlite3.Connection
|
| | |
|
| |
|
| |
Optionally sends typing, presence, and read receipt information to appservices.
|
| |\
| |
| | |
Rename Cache to DeferredCache, and related changes
|
| | | |
|
| | | |
|
| | |
| |
| | |
Include user agent in user daily visits table.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Autocommit means that we don't wrap the functions in transactions, and instead get executed directly. Introduced in #8456. This will help:
1. reduce the number of `could not serialize access due to concurrent delete` errors that we see (though there are a few functions that often cause serialization errors that we don't fix here);
2. improve the DB performance, as it no longer needs to deal with the overhead of `REPEATABLE READ` isolation levels; and
3. improve wall clock speed of these functions, as we no longer need to send `BEGIN` and `COMMIT` to the DB.
Some notes about the differences between autocommit mode and our default `REPEATABLE READ` transactions:
1. Currently `autocommit` only applies when using PostgreSQL, and is ignored when using SQLite (due to silliness with [Twisted DB classes](https://twistedmatrix.com/trac/ticket/9998)).
2. Autocommit functions may get retried on error, which means they can get applied *twice* (or more) to the DB (since they are not in a transaction the previous call would not get rolled back). This means that the functions need to be idempotent (or otherwise not care about being called multiple times). Read queries, simple deletes, and updates/upserts that replace rows (rather than generating new values from existing rows) are all idempotent.
3. Autocommit functions no longer get executed in [`REPEATABLE READ`](https://www.postgresql.org/docs/current/transaction-iso.html) isolation level, and so data can change queries, which is fine for single statement queries.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
We asserted that the IDs returned by postgres sequence was greater than
any we had seen, however this is technically racey as we may update the
current positions out of order.
We now assert that the sequences are correct on startup, so the
assertion is no longer really required, so we remove them.
|
| |/
|
|
|
| |
* Make sure a retention policy is a state event
* Changelog
|
| | |
|
