summary refs log tree commit diff
path: root/synapse/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Make `has_completed_background_updates` asyncRichard van der Hoff2020-03-311-4/+3
| | | | (Almost) everywhere that uses it is happy with an awaitable.
* Remove unused `start_background_update`Richard van der Hoff2020-03-311-21/+0
| | | | This was only used in a unit test, so let's just inline it in the test.
* Fill in the 'default' field for user-defined push rules (#6639)Karlinde2020-03-311-0/+1
| | | Signed-off-by: Karl Linderhed <git@karlinde.se>
* Only setdefault for signatures if device has key_json (#7177)Andrew Morgan2020-03-311-10/+14
|
* Fix use of async/await in media code (#7184)Patrick Cloke2020-03-311-2/+2
|
* Rewrite prune_old_outbound_device_pokes for efficiency (#7159)Richard van der Hoff2020-03-301-13/+58
| | | | make sure we clear out all but one update for the user
* Transfer alias mappings when joining an upgraded room (#6946)Andrew Morgan2020-03-301-3/+23
|
* Ensure is_verified on /_matrix/client/r0/room_keys/keys is a boolean (#7150)Andrew Morgan2020-03-271-1/+2
|
* Only import sqlite3 when type checking (#7155)David Vo2020-03-271-2/+5
| | | | Fixes: #7127 Signed-off-by: David Vo <david@vovo.id.au>
* Move catchup of replication streams to worker. (#7024)Erik Johnston2020-03-255-199/+201
| | | This changes the replication protocol so that the server does not send down `RDATA` for rows that happened before the client connected. Instead, the server will send a `POSITION` and clients then query the database (or master out of band) to get up to date.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-242-8/+7
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Remove concept of a non-limited stream. (#7011)Erik Johnston2020-03-203-16/+31
|
* Change device list streams to have one row per ID (#7010)Erik Johnston2020-03-192-65/+72
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add 'device_lists_outbound_pokes' as extra table. This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change. * Change device lists stream to have one row per id. This will make it possible to process the streams more incrementally, avoiding having to process large chunks at once. * Change device list replication to match new semantics. Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host). * Newsfile * Remove handling of multiple rows per ID * Fix worker handling * Comments from review
| * Comments from reviewErik Johnston2020-03-181-8/+19
| |
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2020-03-021-1/+3
| |\ | | | | | | | | | erikj/fixup_devices_stream
| * | Remove handling of multiple rows per IDErik Johnston2020-02-281-34/+1
| | |
| * | Change device list replication to match new semantics.Erik Johnston2020-02-281-6/+9
| | | | | | | | | | | | | | | Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host).
| * | Change device lists stream to have one row per id.Erik Johnston2020-02-281-18/+41
| | | | | | | | | | | | | | | This will make it possible to process the streams more incrementally, avoiding having to process large chunks at once.
| * | Add 'device_lists_outbound_pokes' as extra table.Erik Johnston2020-02-281-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change.
* | | Improve get auth chain difference algorithm. (#7095)Erik Johnston2020-03-181-1/+149
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It was originally implemented by pulling the full auth chain of all state sets out of the database and doing set comparison. However, that can take a lot work if the state and auth chains are large. Instead, lets try and fetch the auth chains at the same time and calculate the difference on the fly, allowing us to bail early if all the auth chains converge. Assuming that the auth chains do converge more often than not, this should improve performance. Hopefully.
* | | Populate the room version from state events (#7070)Brendan Abolivier2020-03-162-0/+62
| | | | | | | | | | | | | | | Fixes #7065 This is basically the same as https://github.com/matrix-org/synapse/pull/6847 except it tries to populate events from `state_events` rather than `current_state_events`, since the latter might have been cleared from the state of some rooms too early, leaving them with a `NULL` room version.
* | | Hopefully mypy is happy nowBrendan Abolivier2020-03-101-2/+8
| | |
* | | Merge pull request #7055 from ↵Brendan Abolivier2020-03-091-17/+17
|\ \ \ | | | | | | | | | | | | | | | | matrix-org/babolivier/get_time_of_last_push_action_before Move get_time_of_last_push_action_before to the EventPushActionsWorkerStore
| * | | Move `get_time_of_last_push_action_before` to the `EventPushActionsWorkerStore`Brendan Abolivier2020-03-091-17/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #7054 I also had a look at the rest of the functions in `EventPushActionsStore` and in the push notifications send code and it looks to me like there shouldn't be any other method with this issue in this part of the codebase.
* | | | Remove special auth and redaction rules for aliases events in experimental ↵Patrick Cloke2020-03-091-2/+8
| | | | | | | | | | | | | | | | room ver. (#7037)
* | | | Break down monthly active users by appservice_id (#7030)Neil Johnson2020-03-061-1/+31
|/ / / | | | | | | | | | | | | * Break down monthly active users by appservice_id and emit via prometheus. Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | / Read the room version from database when fetching events (#6874)Richard van der Hoff2020-03-041-16/+68
| |/ |/| | | | | | | This is a precursor to giving EventBase objects the knowledge of which room version they belong to.
* | Always return a deferred from `get_current_state_deltas`. (#7019)Erik Johnston2020-03-021-1/+3
|/ | | | | This currently causes presence notify code to log exceptions when there is no state changes to process. This doesn't actually cause any problems as we'd simply do nothing anyway.
* Fixed set a user as an admin with the new API (#6928)Dirk Klimpel2020-02-281-6/+10
| | | Fix #6910
* Add some type annotations in `synapse.storage` (#6987)Richard van der Hoff2020-02-276-82/+266
| | | | | I cracked, and added some type definitions in synapse.storage.
* Store room version on invite (#6983)Richard van der Hoff2020-02-261-0/+20
| | | | | When we get an invite over federation, store the room version in the rooms table. The general idea here is that, when we pull the invite out again, we'll want to know what room_version it belongs to (so that we can later redact it if need be). So we need to store it somewhere...
* Updated warning for incorrect database collation/ctype (#6985)Uday Bansal2020-02-261-3/+7
| | | Signed-off-by: Uday Bansal <43824981+udaybansal19@users.noreply.github.com>
* Sanity-check database before running upgrades (#6982)Richard van der Hoff2020-02-252-17/+32
| | | | | | Some of the database deltas rely on `config.server_name` being set correctly, so we should check that it is before running the deltas. Fixes #6870.
* Merge worker apps into one. (#6964)Erik Johnston2020-02-251-78/+78
|
* Upsert room version when we join over federation (#6968)Richard van der Hoff2020-02-241-0/+17
| | | | | | | | This is intended as a precursor to storing room versions when we receive an invite over federation, but has the happy side-effect of fixing #3374 at last. In short: change the store_room with try/except to a proper upsert which updates the right columns.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-2119-55/+56
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* don't insert into the device table for remote cross-signing keys (#6956)Hubert Chathi2020-02-201-15/+18
|
* Add some clarifications to README.md in the database schema directory. (#6615)Ruben Barkow-Kuder2020-02-201-8/+16
| | | Signed-off-by: Ruben Barkow-Kuder <github@r.z11.de>
* Minor perf fixes to `get_auth_chain_ids`.Erik Johnston2020-02-192-7/+5
|
* Reduce auth chains fetched during v2 state res. (#6952)Erik Johnston2020-02-191-5/+23
| | | | | | The state res v2 algorithm only cares about the difference between auth chains, so we can pass in the known common state to the `get_auth_chain` storage function so that it can ignore those events.
* Revert `get_auth_chain_ids` changes (#6951)Erik Johnston2020-02-191-28/+0
|
* Increase DB/CPU perf of `_is_server_still_joined` check. (#6936)Erik Johnston2020-02-192-15/+59
| | | | | | | | | | | | | | | | | | | | | | | | * Increase DB/CPU perf of `_is_server_still_joined` check. For rooms with large amount of state a single user leaving could cause us to go and load a lot of membership events and then pull out membership state in a large number of batches. * Newsfile * Update synapse/storage/persist_events.py Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> * Fix adding if too soon * Update docstring * Review comments * Woops typo Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Limit size of get_auth_chain_ids query (#6947)Erik Johnston2020-02-191-18/+23
|
* Increase perf of `get_auth_chain_ids` used in state res v2. (#6937)Erik Johnston2020-02-181-0/+23
| | | We do this by moving the recursive query to be fully in the DB.
* wait for current_state_events_membership before ↵Richard van der Hoff2020-02-171-2/+5
| | | | delete_old_current_state_events (#6924)
* Remove unused `get_room_stats_state` method. (#6869)Richard van der Hoff2020-02-071-25/+0
|
* Allow moving group read APIs to workers (#6866)Erik Johnston2020-02-071-359/+361
|
* Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-062-3/+3
| | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* Merge tag 'v1.10.0rc2' into developErik Johnston2020-02-061-2/+2
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.10.0rc2 (2020-02-06) ============================== Bugfixes -------- - Fix an issue with cross-signing where device signatures were not sent to remote servers. ([\#6844](https://github.com/matrix-org/synapse/issues/6844)) - Fix to the unknown remote device detection which was introduced in 1.10.rc1. ([\#6848](https://github.com/matrix-org/synapse/issues/6848)) Internal Changes ---------------- - Detect unexpected sender keys on remote encrypted events and resync device lists. ([\#6850](https://github.com/matrix-org/synapse/issues/6850))
| * Merge pull request #6844 from matrix-org/uhoreg/cross_signing_fix_device_fedHubert Chathi2020-02-051-0/+10
| | | | | | add device signatures to device key query results
| * Fix detecting unknown devices from remote encrypted events. (#6848)Erik Johnston2020-02-041-2/+2
| | | | | | | | | | | | | | | | We were looking at the wrong event type (`m.room.encryption` vs `m.room.encrypted`). Also fixup the duplicate `EvenTypes` entries. Introduced in #6776.
* | Reduce performance logging to DEBUG (#6833)Michael Kaye2020-02-051-1/+1
| | | | | | | | | | * Reduce tnx performance logging to DEBUG * Changelog.d
* | Database updates to populate rooms.room_version (#6847)Richard van der Hoff2020-02-042-0/+57
| | | | | | We're going to need this so that we can figure out how to handle redactions when fetching events from the database.
* | add device signatures to device key query resultsHubert Chathi2020-02-041-0/+10
|/
* Fix deleting of stale marker for device lists (#6819)Erik Johnston2020-01-311-7/+7
| | | | We were in fact only deleting stale marker when we got an incremental update, rather than when we did a full resync.
* Add `get_room_version` methodRichard van der Hoff2020-01-311-1/+24
| | | | So that we can start factoring out some of this boilerplatey boilerplate.
* s/get_room_version/get_room_version_id/Richard van der Hoff2020-01-312-2/+2
| | | | | ... to make way for a forthcoming get_room_version which returns a RoomVersion object.
* Log when we delete room in bg update (#6816)Erik Johnston2020-01-301-0/+2
|
* Backgroud update to clean out rooms from current state (#6802)Erik Johnston2020-01-302-2/+125
|
* When server leaves room check for stale device lists. (#6801)Erik Johnston2020-01-302-6/+82
| | | | | | | | | When a server leaves a room it may stop sharing a room with remote users, and thus not get any updates to their device lists. So we need to check for this case and delete those device lists from the cache. We don't need to do this if we stop sharing a room because the remote user leaves the room, because we track that case via looking at membership changes.
* When a client asks for remote keys check if should resync. (#6797)Erik Johnston2020-01-301-3/+29
| | | | | | If we detect that the remote users' keys may have changed then we should attempt to resync against the remote server rather than using the (potentially) stale local cache.
* Delete current state when server leaves a room (#6792)Erik Johnston2020-01-292-75/+197
| | | | | | Otherwise its just stale data, which may get deleted later anyway so can't be relied on. It's also a bit of a shotgun if we're trying to get the current state of a room we're not in.
* Detect unknown remote devices and mark cache as stale (#6776)Erik Johnston2020-01-282-5/+49
| | | | We just mark the fact that the cache may be stale in the database for now.
* Warn if postgres database has non-C locale. (#6734)Erik Johnston2020-01-283-0/+52
| | | As using non-C locale can cause issues on upgrading OS.
* Merge branch 'master' into developErik Johnston2020-01-281-1/+7
|\
| * Fix setting `mau_limit_reserved_threepids` config (#6793)Erik Johnston2020-01-281-1/+7
| | | | | | | | | | Calling the invalidation function during initialisation of the data stores introduces a circular dependency, causing Synapse to fail to start.
* | Add `rooms.room_version` column (#6729)Erik Johnston2020-01-273-18/+134
| | | | | | This is so that we don't have to rely on pulling it out from `current_state_events` table.
* | Propagate cache invalidates from workers to other workers. (#6748)Erik Johnston2020-01-271-1/+21
| | | | | | Currently if a worker invalidates a cache it will be streamed to master, which then didn't forward those to other workers.
* | Make 'event.redacts' never raise. (#6771)Erik Johnston2020-01-232-2/+2
| | | | | | | | | | | | There are quite a few places that we assume that a redaction event has a corresponding `redacts` key, which is not always the case. So lets cheekily make it so that event.redacts just returns None instead.
* | Remove unnecessary abstractions in admin handler (#6751)Andrew Morgan2020-01-221-1/+1
|/
* Admin API to list, filter and sort rooms (#6720)Andrew Morgan2020-01-221-1/+124
|
* Allow monthly active user limiting support for worker mode, fixes #4639. (#6742)Neil Johnson2020-01-221-82/+83
|
* Allow streaming cache invalidate all to workers. (#6749)Erik Johnston2020-01-222-9/+36
|
* Add a DeltaState to track changes to be made to current state (#6716)Erik Johnston2020-01-202-99/+111
|
* Fix syntax error in run_upgrade for schema 57 (#6728)Satsuki Yanagi2020-01-171-3/+4
| | | | | | | Fix #6727 Related #6655 Co-authored-by: Erik Johnston <erikj@jki.re>
* Merge pull request #6714 from matrix-org/babolivier/retention_select_eventBrendan Abolivier2020-01-171-5/+5
|\ | | | | Fix instantiation of message retention purge jobs
| * Fixup diffBrendan Abolivier2020-01-161-6/+7
| |
| * Remove get_room_event_after_stream_ordering entirelyBrendan Abolivier2020-01-161-57/+12
| |
| * LintBrendan Abolivier2020-01-161-6/+12
| |
| * Correctly order when selecting before stream orderingBrendan Abolivier2020-01-151-2/+5
| |
| * Fix typoBrendan Abolivier2020-01-151-1/+1
| |
| * Fix instantiation of message retention purge jobsBrendan Abolivier2020-01-151-12/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When figuring out which topological token to start a purge job at, we need to do the following: 1. Figure out a timestamp before which events will be purged 2. Select the first stream ordering after that timestamp 3. Select info about the first event after that stream ordering 4. Build a topological token from that info In some situations (e.g. quiet rooms with a short max_lifetime), there might not be an event after the stream ordering at step 3, therefore we abort the purge with the error `No event found`. To mitigate that, this patch fetches the first event _before_ the stream ordering, instead of after.
* | move batch_iter to a separate moduleRichard van der Hoff2020-01-166-6/+6
| |
* | Add StateMap type alias (#6715)Erik Johnston2020-01-163-43/+55
| |
* | Fix purge_room admin API (#6711)Erik Johnston2020-01-151-1/+1
| |
* | Add `local_current_membership` table (#6655)Erik Johnston2020-01-154-80/+238
|/ | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Quarantine media by ID or user ID (#6681)Andrew Morgan2020-01-131-1/+115
|
* Refuse to start if sqlite is older than 3.11.0Richard van der Hoff2020-01-092-4/+7
|
* Check postgres version in check_databaseRichard van der Hoff2020-01-091-13/+12
| | | | | this saves doing it on each connection, and will allow us to pass extra options in.
* Modify check_database to take a connection rather than a cursorRichard van der Hoff2020-01-093-10/+11
| | | | We might not need the cursor at all.
* Allow admin users to create or modify users without a shared secret (#6495)Manuel Stahl2020-01-091-0/+2
| | | Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
* Merge pull request #6664 from matrix-org/erikj/media_admin_apisErik Johnston2020-01-081-120/+128
|\ | | | | Fix media repo admin APIs when using a media worker.
| * Shuffle the codeErik Johnston2020-01-081-25/+16
| |
| * CommentsErik Johnston2020-01-081-0/+3
| |
| * Do not rely on streaming events, as media repo doesn'tErik Johnston2020-01-081-12/+26
| |
| * Move media admin store functions to worker storeErik Johnston2020-01-081-120/+120
| |
* | Fix GET request on /_synapse/admin/v2/users endpoint (#6563)Manuel Stahl2020-01-081-2/+2
|/ | | | Fixes #6552
* Merge pull request #6652 from matrix-org/babolivier/depth_missing_eventsBrendan Abolivier2020-01-071-1/+1
|\ | | | | Fix conditions failing if min_depth = 0
| * Fix conditions failing if min_depth = 0Brendan Abolivier2020-01-071-1/+1
| | | | | | | | This could result in Synapse not fetching prev_events for new events in the room if it has missed some events.
* | Add a background update to clear tombstoned rooms from the directory (#6648)Richard van der Hoff2020-01-074-1/+100
| | | | | | | | | | | | * Add a background update to clear tombstoned rooms from the directory * use the ABC metaclass
* | Async/await for background updates (#6647)Richard van der Hoff2020-01-071-16/+20
| | | | | | | | so that bg update routines can be async
* | Merge pull request #6645 from matrix-org/rav/fix_synchrotron_errorRichard van der Hoff2020-01-071-9/+35
|\ \ | | | | | | Fix exceptions in the synchrotron worker log when events are rejected.
| * | Clarify documentation on get_event* methodsRichard van der Hoff2020-01-061-9/+35
| | | | | | | | | | | | Make it clearer how they behave in the face of rejected and/or missing events.
* | | Merge pull request #6629 from matrix-org/rav/kill_event_reference_hashesRichard van der Hoff2020-01-061-53/+17
|\ \ \ | |/ / |/| | Remove a bunch of unused code from event creation
| * | Remove unused get_latest_event_ids_and_hashes_in_roomRichard van der Hoff2020-01-061-42/+0
| | |
| * | Remove unused get_prev_events_and_hashes_for_roomRichard van der Hoff2020-01-061-30/+0
| | |
| * | replace get_prev_events_and_hashes_for_room with get_prev_events_for_room in ↵Richard van der Hoff2020-01-061-0/+35
| | | | | | | | | | | | create_new_client_event
| * | rename get_prev_events_for_room to get_prev_events_and_hashes_for_roomRichard van der Hoff2020-01-061-2/+3
| |/ | | | | | | ... to make way for a new method which just returns the event ids
* | Fix some test failures when frozen_dicts are enabled (#6642)Richard van der Hoff2020-01-061-2/+2
| | | | | | | | Fixes #4026
* | Add experimental 'databases' config (#6580)Erik Johnston2020-01-061-0/+21
|/
* Split state groups into a separate data store (#6296)Erik Johnston2019-12-2023-1157/+1147
|
* Change EventContext to use the Storage class (#6564)Erik Johnston2019-12-202-2/+2
|
* Explode on duplicate delta file names. (#6565)Erik Johnston2019-12-191-0/+18
|
* Improve diagnostics on database upgrade failure (#6570)Richard van der Hoff2019-12-191-1/+4
| | | | `Failed to upgrade database` is not helpful, and it's unlikely that UPGRADE.rst has anything useful.
* Clean up startup for the pusher (#6558)Richard van der Hoff2019-12-181-17/+8
| | | | | | | | | | | | | | | | | | | | | | | * Remove redundant python2 support code `str.decode()` doesn't exist on python3, so presumably this code was doing nothing * Filter out pushers with corrupt data When we get a row with unparsable json, drop the row, rather than returning a row with null `data`, which will then cause an explosion later on. * Improve logging when we can't start a pusher Log the ID to help us understand the problem * Make email pusher setup more robust We know we'll have a `data` member, since that comes from the database. What we *don't* know is if that is a dict, and if that has a `brand` member, and if that member is a string.
* Add delta file to fix missing default table data (#6555)Andrew Morgan2019-12-183-15/+23
|
* Add database config class (#6513)Erik Johnston2019-12-186-19/+93
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Automatically delete empty groups/communities (#6453)Werner Sembach2019-12-161-0/+29
| | | Signed-off-by: Werner Sembach <werner.sembach@fau.de>
* look up cross-signing keys from the DB in bulk (#6486)Hubert Chathi2019-12-121-4/+213
|
* Fix redacted events being returned in search results ordered by "recent" (#6522)Andrew Morgan2019-12-121-3/+8
|
* Merge tag 'v1.7.0rc2' into developErik Johnston2019-12-111-0/+5
|\ | | | | | | | | | | | | | | | | | | | | | | Synapse 1.7.0rc2 (2019-12-11) ============================= Bugfixes -------- - Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](https://github.com/matrix-org/synapse/issues/6497)) - Fix support for SQLite 3.7. ([\#6499](https://github.com/matrix-org/synapse/issues/6499)) - Fix regression where sending email push would not work when using a pusher worker. ([\#6507](https://github.com/matrix-org/synapse/issues/6507), [\#6509](https://github.com/matrix-org/synapse/issues/6509))
| * Merge branch 'erikj/fix_sqlite_7' of github.com:matrix-org/synapse into ↵Erik Johnston2019-12-115-11/+41
| |\ | | | | | | | | | release-v1.7.0
| * | Give the server config to the RoomWorkerStoreBrendan Abolivier2019-12-101-0/+5
| | |
* | | Prevent redacted events from appearing in message search (#6377)Andrew Morgan2019-12-112-38/+67
| | |
* | | Add dev script to generate full SQL schema files (#6394)Andrew Morgan2019-12-112-19/+13
| | |
* | | Prevent message search in upgraded rooms we're not in (#6385)Andrew Morgan2019-12-111-5/+13
| | |
* | | Fix race which caused deleted devices to reappear (#6514)Richard van der Hoff2019-12-101-3/+5
| |/ |/| | | | | Stop the `update_client_ips` background job from recreating deleted devices.
* | Drop unused indexErik Johnston2019-12-104-13/+18
| |
* | Convert _censor_redactions to async since it awaits on coroutinesErik Johnston2019-12-101-11/+9
| |
* | Only start censor background job after indices are createdErik Johnston2019-12-102-2/+9
| |
* | Fix support for SQLite 3.7.Erik Johnston2019-12-093-2/+22
|/ | | | | Partial indices support was added in 3.8.0, so we need to use the background updates that handles this correctly.
* Merge pull request #6487 from matrix-org/erikj/pass_in_dbErik Johnston2019-12-0929-122/+162
|\ | | | | Pass in Database object to data stores.
| * Fix commentErik Johnston2019-12-091-1/+1
| | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * Fix port db scriptErik Johnston2019-12-061-1/+1
| |
| * Move start up DB checks to main data store.Erik Johnston2019-12-061-0/+7
| |
| * Pass Database into the data storeErik Johnston2019-12-064-26/+23
| |
| * Move are_all_users_on_domain checks to main data store.Erik Johnston2019-12-062-13/+23
| |
| * Change DataStores to accept 'database' param.Erik Johnston2019-12-0625-82/+108
| |
* | Merge pull request #6484 from matrix-org/erikj/port_sync_handlerErik Johnston2019-12-092-3/+3
|\ \ | |/ |/| Port SyncHandler to async/await
| * Fixup functions to consistently return deferredsErik Johnston2019-12-062-3/+3
| |
* | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-12-063-47/+71
|\ \ | | | | | | | | | erikj/make_database_class
| * | Replace /admin/v1/users_paginate endpoint with /admin/v2/users (#5925)Manuel Stahl2019-12-053-44/+71
| |/
* | Remove unused varErik Johnston2019-12-061-2/+0
| |
* | Move background update handling out of storeErik Johnston2019-12-0517-146/+163
| |
* | CommentsErik Johnston2019-12-053-5/+16
| |
* | Move DB pool and helper functions into dedicated Database classErik Johnston2019-12-0544-2192/+2271
|/
* Merge pull request #6464 from matrix-org/erikj/make_public_sql_baseErik Johnston2019-12-0543-507/+508
|\ | | | | Clean up SQLBaseStore private function usage
| * Remove underscore from SQLBaseStore functionsErik Johnston2019-12-0443-500/+492
| |
| * Don't call SQLBaseStore methods from outside storesErik Johnston2019-12-042-8/+17
| |
* | Merge pull request #6470 from matrix-org/babolivier/port_db_ci_failureBrendan Abolivier2019-12-041-2/+10
|\ \ | | | | | | Make synapse_port_db exit with a non-0 code if something failed
| * | Fix background updates for synapse_port_dbBrendan Abolivier2019-12-041-2/+10
| |/
* / get rid of (most of) have_events from ↵Richard van der Hoff2019-12-041-34/+0
|/ | | | | | | | | | | | | _update_auth_events_and_context_for_auth (#6468) have_events was a map from event_id to rejection reason (or None) for events which are in our local database. It was used as filter on the list of event_ids being passed into get_events_as_list. However, since get_events_as_list will ignore any event_ids that are unknown or rejected, we can equivalently just leave it to get_events_as_list to do the filtering. That means that we don't have to keep `have_events` up-to-date, and can use `have_seen_events` instead of `get_seen_events_with_rejection` in the one place we do need it.
* Merge pull request #6454 from matrix-org/erikj/clean_base_StoreErik Johnston2019-12-047-190/+218
|\ | | | | Move things out of SQLBaseStore
| * Revert "Move get_user_count_txn out of base store"Erik Johnston2019-12-042-12/+12
| | | | | | | | | | | | This reverts commit 00f0d67566cdfe8eae44aeae1c982c42a255cfcd. Its going to get removed soon, so lets not make merge conflicts.
| * _CURRENT_STATE_CACHE_NAME is publicErik Johnston2019-12-041-3/+3
| |
| * Move get_user_count_txn out of base storeErik Johnston2019-12-042-12/+12
| |
| * Move cache invalidation to main data storeErik Johnston2019-12-045-112/+141
| |
| * Move event fetch vars to EventWorkStoreErik Johnston2019-12-044-14/+15
| |
| * Move account validity bg updates to registration storeErik Johnston2019-12-042-66/+64
| |
* | Merge pull request #6329 from matrix-org/babolivier/context_filtersBrendan Abolivier2019-12-041-0/+3
|\ \ | |/ |/| Filter state, events_before and events_after in /context requests
| * Merge branch 'develop' into babolivier/context_filtersBrendan Abolivier2019-12-0422-305/+1024
| |\
| * | Update copyrightsBrendan Abolivier2019-11-051-0/+3
| | |
* | | Merge pull request #6451 from matrix-org/uhoreg/cross_signing_signatures_indexErik Johnston2019-12-042-1/+24
|\ \ \ | | | | | | | | make cross signing signature index non-unique
| * | | apply changes from reviewHubert Chathi2019-12-031-1/+2
| | | |
| * | | make cross signing signature index non-uniqueHubert Chathi2019-12-022-1/+23
| | |/ | |/|
* | | Fix exception when a cross-signed device is deleted (#6462)Richard van der Hoff2019-12-041-4/+19
| | | | | | | | | | | | | | | (hopefully) ... and deobfuscate the relevant bit of code.
* | | Add ephemeral messages support (MSC2228) (#6409)Brendan Abolivier2019-12-032-6/+141
|/ / | | | | | | | | | | | | | | Implement part [MSC2228](https://github.com/matrix-org/matrix-doc/pull/2228). The parts that differ are: * the feature is hidden behind a configuration flag (`enable_ephemeral_messages`) * self-destruction doesn't happen for state events * only implement support for the `m.self_destruct_after` field (not the `m.self_destruct` one) * doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database
* | Remove local threepids on account deactivation (#6426)Andrew Morgan2019-11-281-0/+13
| |
* | add etag and count to key backup endpoints (#5858)Hubert Chathi2019-11-272-66/+177
| |
* | Merge branch 'develop' into babolivier/message_retentionBrendan Abolivier2019-11-2618-239/+547
|\ \
| * | Don't construct a setErik Johnston2019-11-261-6/+3
| | |
| * | Fix find_next_generated_user_id_localpartErik Johnston2019-11-261-10/+7
| | |
| * | Prevent account_data content from being sent over TCP replication (#6333)Andrew Morgan2019-11-261-3/+3
| |\ \
| | * | Remove content from being sent for account data rdata streamAndrew Morgan2019-11-081-3/+3
| | | |
| * | | Make sure that we close cursors before returning from a query (#6408)Richard van der Hoff2019-11-252-10/+43
| | | | | | | | | | | | | | | | | | | | | | | | There are lots of words in the comment as to why this is a good idea. Fixes #6403.
| * | | Merge tag 'v1.6.0rc2' into developRichard van der Hoff2019-11-251-18/+25
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.6.0rc2 (2019-11-25) ============================= Bugfixes -------- - Fix a bug which could cause the background database update hander for event labels to get stuck in a loop raising exceptions. ([\#6407](https://github.com/matrix-org/synapse/issues/6407))
| | * | | Fix exceptions from background database update for event labels. (#6407)Richard van der Hoff2019-11-251-18/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add some exception handling here so that events whose json cannot be parsed are ignored rather than getting us stuck in a loop. Fixes #6404.
| * | | | Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-2110-22/+16
| |/ / /
| * | | Replace UPDATE with UPSERT on device_max_stream_id table (#6363)Andrew Morgan2019-11-151-2/+15
| | | |
| * | | Fix guest -> real account upgrade with account validity enabled (#6359)Andrew Morgan2019-11-141-6/+3
| |/ /
| * | Merge pull request #6340 from matrix-org/babolivier/pagination_queryBrendan Abolivier2019-11-081-8/+32
| |\ \ | | | | | | | | Fix the SQL SELECT query in _paginate_room_events_txn
| | * | Incorporate reviewBrendan Abolivier2019-11-081-12/+12
| | | |
| | * | LintBrendan Abolivier2019-11-071-2/+2
| | | |
| | * | Only join on event_labels if we're filtering on labelsBrendan Abolivier2019-11-071-9/+24
| | | |
| | * | Handle lack of filterBrendan Abolivier2019-11-071-5/+3
| | | |
| | * | Fix the SQL SELECT query in _paginate_room_events_txnBrendan Abolivier2019-11-071-2/+13
| | | | | | | | | | | | | | | | Doing a SELECT DISTINCT when paginating is quite expensive, because it requires the engine to do sorting on the entire events table. However, we only need to run it if we're filtering on 2+ labels, so this PR is changing the request so that DISTINCT is only used then.
| * | | Merge pull request #6295 from matrix-org/erikj/split_purge_historyErik Johnston2019-11-084-172/+319
| |\ \ \ | | | | | | | | | | Split purge API into events vs state and add PurgeEventsStorage
| | * | | Move type annotation into docstringErik Johnston2019-11-081-3/+3
| | | | |
| | * | | Fix deleting state groups during room purge.Erik Johnston2019-11-061-13/+14
| | | | | | | | | | | | | | | | | | | | And fix the tests to actually test that things got deleted.
| | * | | Use correct type annotationErik Johnston2019-11-061-2/+2
| | | | |
| | * | | Change to not require a state_groups.room_id index.Erik Johnston2019-11-044-53/+45
| | | | | | | | | | | | | | | | | | | | | | | | | This does mean that we won't clean up orphaned state groups (i.e. state groups that were persisted but the associated event wasn't).
| | * | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-11-0418-54/+259
| | |\ \ \ | | | | |/ | | | |/| | | | | | erikj/split_purge_history
| | * | | Fix up commentErik Johnston2019-11-011-2/+1
| | | | |
| | * | | Update log line to lie a little lessErik Johnston2019-10-311-1/+1
| | | | |
| | * | | Add state_groups.room_id indexErik Johnston2019-10-312-0/+24
| | | | |
| | * | | DocstringsErik Johnston2019-10-311-3/+11
| | | | |
| | * | | Fix purge room APIErik Johnston2019-10-311-5/+8
| | | | |
| | * | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-317-12/+243
| | |\ \ \ | | | | | | | | | | | | | | | | | | erikj/split_purge_history
| | * | | | Split purge API into events vs stateErik Johnston2019-10-304-175/+295
| | | | | |
| * | | | | Merge pull request #6310 from matrix-org/babolivier/msc2326_bg_updateBrendan Abolivier2019-11-072-0/+80
| |\ \ \ \ \ | | | | | | | | | | | | | | MSC2326: Add background update to take previous events into account
| | * | | | | Update synapse/storage/data_stores/main/events_bg_updates.pyBrendan Abolivier2019-11-071-1/+1
| | | | | | | | | | | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | * | | | | Update synapse/storage/data_stores/main/events_bg_updates.pyBrendan Abolivier2019-11-071-1/+1
| | | | | | | | | | | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | * | | | | Copy resultsBrendan Abolivier2019-11-071-1/+3
| | | | | | |
| | * | | | | Revert "Back to using cursor_to_dict"Brendan Abolivier2019-11-071-9/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 1186612d6cd728e6b7ed7806579db3cea7410b54.
| | * | | | | Back to using cursor_to_dictBrendan Abolivier2019-11-071-9/+9
| | | | | | |
| | * | | | | Initialise value before loopingBrendan Abolivier2019-11-071-4/+8
| | | | | | |
| | * | | | | Incorporate reviewBrendan Abolivier2019-11-071-14/+9
| | | | | | |
| | * | | | | Fix field nameBrendan Abolivier2019-11-061-1/+1
| | | | | | |
| | * | | | | Update insertBrendan Abolivier2019-11-061-1/+6
| | | | | | |
| | * | | | | Print out the actual number of affected rowsBrendan Abolivier2019-11-041-4/+4
| | | | | | |
| | * | | | | Correctly order resultsBrendan Abolivier2019-11-041-5/+5
| | | | | | |
| | * | | | | TODOBrendan Abolivier2019-11-041-0/+2
| | | | | | |
| | * | | | | Fix exit conditionBrendan Abolivier2019-11-041-1/+3
| | | | | | |
| | * | | | | LintBrendan Abolivier2019-11-041-6/+3
| | | | | | |
| | * | | | | Don't try to process events we already have a label forBrendan Abolivier2019-11-041-1/+2
| | | | | | |
| | * | | | | Use a sensible default value for labelsBrendan Abolivier2019-11-041-1/+1
| | | | | | |
| | * | | | | Use the right format for rowsBrendan Abolivier2019-11-041-1/+1
| | | | | | |
| | * | | | | MSC2326: Add background update to take previous events into accountBrendan Abolivier2019-11-042-0/+72
| | | |_|/ / | | |/| | |
| * | | | | Merge pull request #6235 from matrix-org/anoa/room_upgrade_groupsAndrew Morgan2019-11-072-1/+20
| |\ \ \ \ \
| | * | | | | Re-add docstring, with caveats detailedAndrew Morgan2019-11-042-2/+6
| | | | | | |
| | * | | | | Transfer upgraded rooms on groupsAndrew Morgan2019-11-041-0/+15
| | |/ / / /
| * | | | / Numeric ID checker now checks @0, don't ratelimit on checkingAndrew Morgan2019-11-061-4/+4
| | |_|_|/ | |/| | |
| * | | | Fix bug which caused rejected events to be stored with the wrong room state ↵Richard van der Hoff2019-11-061-1/+1
| |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#6320) Fixes a bug where rejected events were persisted with the wrong state group. Also fixes an occasional internal-server-error when receiving events over federation which are rejected and (possibly because they are backwards-extremities) have no prev_group. Fixes #6289.
* | | | Fix worker modeBrendan Abolivier2019-11-191-56/+56
| | | |
* | | | LintBrendan Abolivier2019-11-191-25/+24
| | | |
* | | | Implement per-room message retention policiesBrendan Abolivier2019-11-043-0/+288
|/ / /
* | | Merge pull request #6301 from matrix-org/babolivier/msc2326Brendan Abolivier2019-11-013-2/+75
|\ \ \ | | | | | | | | Implement MSC2326 (label based filtering)
| * | | Incorporate reviewBrendan Abolivier2019-11-012-1/+7
| | | |
| * | | Update synapse/storage/data_stores/main/schema/delta/56/event_labels.sqlBrendan Abolivier2019-11-011-1/+1
| | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * | | Add more data to the event_labels table and fix the indexesBrendan Abolivier2019-11-013-5/+21
| | | |
| * | | LintBrendan Abolivier2019-11-011-1/+1
| | | |
| * | | Incorporate reviewBrendan Abolivier2019-11-011-2/+10
| | | |
| * | | LintBrendan Abolivier2019-10-301-7/+1
| | | |
| * | | Add more integration testingBrendan Abolivier2019-10-301-1/+1
| | | |
| * | | Add index on labelBrendan Abolivier2019-10-301-1/+3
| | | |
| * | | Implement filteringBrendan Abolivier2019-10-301-0/+9
| | | |
| * | | Store labels for new eventsBrendan Abolivier2019-10-301-1/+19
| | | |
| * | | Add database table for keeping track of labels on eventsBrendan Abolivier2019-10-291-0/+20
| | | |
* | | | Merge pull request #6313 from matrix-org/uhoreg/cross_signing_fix_sqlite_schemaHubert Chathi2019-11-011-0/+42
|\ \ \ \ | | | | | | | | | | fix hidden field in devices table for older sqlite
| * | | | fix hidden field in devices table for older sqliteHubert Chathi2019-10-311-0/+42
| | | | |
* | | | | Update synapse/storage/data_stores/main/devices.pyHubert Chathi2019-10-311-1/+1
| | | | | | | | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | | | | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-3119-40/+300
|\| | | |
| * | | | Merge branch 'develop' into uhoreg/cross_signing_fix_workers_notifyHubert Chathi2019-10-3118-39/+272
| |\ \ \ \
| | * | | | Update black to 19.10b0 (#6304)Amber Brown2019-11-0113-27/+29
| | | |_|/ | | |/| | | | | | | * update version of black and also fix the mypy config being overridden
| | * | | Merge pull request #6294 from matrix-org/erikj/add_state_storageErik Johnston2019-10-313-1/+236
| | |\ \ \ | | | | | | | | | | | | Add StateGroupStorage interface
| | | * | | Apply suggestions from code review Erik Johnston2019-10-311-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix docstring Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | | * | | Add StateGroupStorage interfaceErik Johnston2019-10-303-1/+235
| | | | |/ | | | |/|
| | * | | Fix /purge_room API.Erik Johnston2019-10-311-1/+0
| | | | | | | | | | | | | | | | | | | | It fails trying to clean the `topic` table which was recently removed.
| | * | | Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-313-5/+5
| | | | | | | | | | | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
| | * | | fix delete_existing for _persist_events (#6300)Richard van der Hoff2019-10-302-5/+2
| | |/ / | | | | | | | | this is part of _retry_on_integrity_error, so should only be on _persist_events_and_state_updates
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-09 02:04:55 +0000