summary refs log tree commit diff
path: root/tests/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-293-16/+97
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Track ongoing event fetches correctly (again) (#11376)Sean Quah2021-11-261-1/+138
| | | | | | | | | | | | | The previous fix for the ongoing event fetches counter (8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and incorrect. When the database is unreachable, `_do_fetch` never gets run and so `_event_fetch_ongoing` is never decremented. The previous fix also moved the `_event_fetch_ongoing` decrement outside of the `_event_fetch_lock` which allowed race conditions to corrupt the counter.
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-251-2/+2
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-241-4/+4
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix incorrect return value in tests. (#11359)Patrick Cloke2021-11-161-1/+1
|
* Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-161-1/+51
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-1/+51
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Database storage profile passes mypy (#11342)David Robertson2021-11-151-3/+6
| | | | | | | | | | | | It already seems to pass mypy. I wonder what changed, given that it was on the exclusion list. So this commit consists of me ensuring `--disallow-untyped-defs` passes and a minor fixup to a function that returned either `True` or `None`.
* | Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-121-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-0/+207
|/ | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-051-0/+69
| | | Fixes #11252
* Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-021-0/+74
|
* Test that `ClientIpStore` combines database and in-memory data correctly ↵Sean Quah2021-11-011-0/+206
| | | | (#11179)
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-271-0/+90
| | | Fixes: #9346
* Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-151-44/+33
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-133-3/+570
| | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-062-2/+2
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-066-60/+10
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)David Robertson2021-10-051-16/+78
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two steps to rebuilding the user directory: 1. a scan over rooms, followed by 2. a scan over local users. The former reads avatars and display names from the `room_memberships` table and therefore contains potentially private avatars and display names. The latter reads from the the `profiles` table which only contains public data; moreover it will overwrite any private profiles that the rooms scan may have written to the user directory. This means that the rebuild could leak private user while the rebuild was in progress, only to later cover up the leaks once the rebuild had completed. This change skips over local users when writing user_directory rows when scanning rooms. Doing so means that it'll take longer for a rebuild to make local users searchable, which is unfortunate. I think a future PR can improve this by swapping the order of the two steps above. (And indeed there's more to do here, e.g. copying from `profiles` without going via Python.) Small tidy-ups while I'm here: * Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`. * Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`. * Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Merge tag 'v1.44.0rc3' into developBrendan Abolivier2021-10-041-0/+34
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc3 (2021-10-04) ============================== Bugfixes -------- - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933)) - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
| * Fix error in `get_user_ip_and_agents` when fetching from the database (#10968)Sean Quah2021-10-011-0/+34
| |
* | Consistently exclude from user_directory (#10960)David Robertson2021-10-041-2/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Refactor user directory tests (#10935)David Robertson2021-09-301-4/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-7/+7
|/
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+0
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-232-10/+6
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Treat "\u0000" as "\u0020" for the purposes of message search (message ↵Hillery Shay2021-09-221-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | indexing) (#10820) * add test to check if null code points are being inserted * add logic to detect and replace null code points before insertion into db * lints * add license to test * change approach to null substitution * add type hint for SearchEntry * Add changelog entry Signed-off-by: H.Shay <shaysquared@gmail.com> * updated changelog * update chanelog message * remove duplicate changelog * Update synapse/storage/databases/main/events.py remove extra space Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * rename and move test file, update tests, delete old test file * fix typo in comments * update _find_highlights_in_postgres to replace null byte with space * replace null byte in sqlite search insertion * beef up and reorganize test for this pr * update changelog * add type hints and update docstring * check db engine directly vs using env variable * refactor tests to be less repetetive * move rplace logic into seperate function * requested changes * Fix typo. * Update synapse/storage/databases/main/search.py Co-authored-by: reivilibre <olivier@librepush.net> * Update changelog.d/10820.misc Co-authored-by: Aaron Raimist <aaron@raim.ist> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Aaron Raimist <aaron@raim.ist>
* Make StateFilter frozen so we can hash it (#10816)reivilibre2021-09-141-17/+29
| | | Also enables Mypy for related tests.
* Skip handling of push actions for outlier events (#10780)Erik Johnston2021-09-081-0/+1
| | | | Outlier events don't ever have push actions associated with them, so we can skip some expensive queries during event persistence.
* Populate `rooms.creator` field for easy lookup (#10697)Eric Eastwood2021-09-011-0/+98
| | | | | | Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-177-7/+7
|
* Improve event caching code (#10119)Erik Johnston2021-08-041-0/+50
| | | | Ensure we only load an event from the DB once when the same event is requested multiple times at once.
* Fix codestyle CI from #10440 (#10511)reivilibre2021-08-021-1/+1
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Prune inbound federation queues if they get too long (#10390)Erik Johnston2021-08-021-0/+57
|
* Allow setting transaction limit for db connections (#10440)Toni Spets2021-08-021-0/+36
| | | | | | | Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>
* Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke2021-07-261-3/+3
| | | | (#10254)
* Fix dropping locks on shut down (#10433)Erik Johnston2021-07-201-0/+13
|
* Use inline type hints in `tests/` (#10350)Jonathan de Jong2021-07-132-6/+4
| | | | | | | | This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)
* [pyupgrade] `tests/` (#10347)Jonathan de Jong2021-07-134-13/+5
|
* Add a distributed lock (#10269)Erik Johnston2021-06-291-0/+100
| | | This adds a simple best effort locking mechanism that works cross workers.
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-221-2/+11
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* update black to 21.6b0 (#10197)Marcus2021-06-171-1/+1
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* add a cache to have_seen_event (#9953)Richard van der Hoff2021-06-013-0/+122
| | | Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-2/+6
|
* Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-071-3/+1
| | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-1/+2
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-2/+2
|
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1429-29/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-097-11/+7
|
* Remove old admin API `GET /_synapse/admin/v1/users/<user_id>` (#9401)Dirk Klimpel2021-04-091-2/+2
| | | | | | | | | | | | | Related: #8334 Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25) `GET /_synapse/admin/v1/users/<user_id>` has no - unit tests - documentation API in v2 is available (#5925 - 12/2019, v1.7.0). API is misleading. It expects `user_id` and returns a list of all users. Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-082-8/+16
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Merge remote-tracking branch 'origin/develop' into rav/drop_py35Richard van der Hoff2021-04-0810-499/+264
|\
| * Convert storage test cases to HomeserverTestCase. (#9736)Patrick Cloke2021-04-0610-499/+264
| |
* | remove unused param on `make_tuple_comparison_clause`Richard van der Hoff2021-04-081-2/+1
| |
* | Drop support for sqlite<3.22 as wellRichard van der Hoff2021-04-081-11/+1
|/
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-14/+8
|
* Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-3/+73
| | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* Add a basic test for purging rooms. (#9541)Patrick Cloke2021-03-081-26/+45
| | | | | Unfortunately this doesn't test re-joining the room since that requires having another homeserver to query over federation, which isn't easily doable in unit tests.
* Add the shadow-banning status to the display user admin API. (#9400)Dirk Klimpel2021-02-171-0/+1
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1612-47/+89
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-181-6/+106
|
* Make chain cover index bg update go faster (#9124)Erik Johnston2021-01-151-31/+186
| | | | | | We do this by allowing a single iteration to process multiple rooms at a time, as there are often a lot of really tiny rooms, which can massively slow things down.
* Add background update for add chain cover index (#9029)Erik Johnston2021-01-141-0/+114
|
* Remove user's avatar URL and displayname when deactivated. (#8932)Dirk Klimpel2021-01-121-0/+26
| | | This only applies if the user's data is to be erased.
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-112-24/+697
|
* Improve the performance of calculating ignored users in large rooms (#9024)Patrick Cloke2021-01-071-0/+120
| | | | | | This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>
* Try and drop stale extremities. (#8929)Erik Johnston2020-12-181-0/+334
| | | | If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.
* Use the simple dictionary in full text search for the user directory (#8959)Brendan Abolivier2020-12-171-0/+23
| | | | | * Use the simple dictionary in fts for the user directory * Clarify naming
* Make search statement in List Room and User Admin API case-insensitive (#8931)Dirk Klimpel2020-12-171-0/+7
|
* Add number of local devices to Room Details Admin API (#8886)Dirk Klimpel2020-12-111-0/+26
|
* Pass room_id to get_auth_chain_difference (#8879)Erik Johnston2020-12-041-8/+10
| | | | | This is so that we can choose which algorithm to use based on the room ID.
* Remove redundant mockingRichard van der Hoff2020-12-022-16/+3
|
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-024-4/+4
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Correctly handle unpersisted events when calculating auth chain difference. ↵Erik Johnston2020-12-021-0/+5
| | | | | | | | | (#8827) We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as: 1. the next time a state event happens in the room the current state should correct itself; 2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and 3. we mostly use the state at an event to do important operations, which isn't affected by this.
* Remove deprecated `/_matrix/client/*/admin` endpoints (#8785)Dirk Klimpel2020-11-251-1/+1
| | | These are now only available via `/_synapse/admin/v1`.
* Merge pull request #8761 from matrix-org/rav/test_request_renderingRichard van der Hoff2020-11-171-2/+1
|\ | | | | Make `make_request` actually render the request
| * Remove redundant calls to `render()`Richard van der Hoff2020-11-161-2/+1
| |
* | Add admin API for logging in as a user (#8617)Erik Johnston2020-11-171-30/+0
|/
* fixup testRichard van der Hoff2020-11-161-0/+1
|
* Add a `custom_headers` param to `make_request` (#8760)Richard van der Hoff2020-11-161-6/+7
| | | | Some tests want to set some custom HTTP request headers, so provide a way to do that before calling requestReceived().
* Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-6/+4
| | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
| | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt2020-10-261-0/+56
|
* Don't instansiate Requester directly (#8614)Erik Johnston2020-10-223-7/+7
|
* Combine the two sets of tests for CacheDescriptorRichard van der Hoff2020-10-211-228/+0
|
* Send some ephemeral events to appservices (#8437)Will Hunt2020-10-151-4/+4
| | | Optionally sends typing, presence, and read receipt information to appservices.
* Merge pull request #8548 from matrix-org/rav/deferred_cacheRichard van der Hoff2020-10-151-72/+1
|\ | | | | Rename Cache to DeferredCache, and related changes
| * Combine the two sets of DeferredCache testsRichard van der Hoff2020-10-141-72/+0
| |
| * move DeferredCache into its own moduleRichard van der Hoff2020-10-141-1/+2
| |
| * Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-5/+5
| |
* | Simplify `_locally_reject_invite`Richard van der Hoff2020-10-131-2/+2
|/ | | | | Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.
* Make event persisters periodically announce position over replication. (#8499)Erik Johnston2020-10-121-9/+16
| | | | | Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress. This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
* Reduce serialization errors in MultiWriterIdGen (#8456)Erik Johnston2020-10-071-0/+1
| | | | | | We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.
* Add logging on startup/shutdown (#8448)Erik Johnston2020-10-021-5/+9
| | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* Make token serializing/deserializing async (#8427)Erik Johnston2020-09-301-3/+6
| | | The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
* Rewrite BucketCollectorRichard van der Hoff2020-09-301-8/+11
| | | | | | | | | | | | This was a bit unweildy for what I wanted: in particular, I wanted to assign each measurement straight into a bucket, rather than storing an intermediate Counter which didn't do any bucketing at all. I've replaced it with something that is hopefully a bit easier to use. (I'm not entirely sure what the difference between a HistogramMetricFamily and a GaugeHistogramMetricFamily is, but given our counters can go down as well as up the latter *sounds* more accurate?)
* Various clean ups to room stream tokens. (#8423)Erik Johnston2020-09-291-6/+4
|
* Don't table scan events on worker startup (#8419)Erik Johnston2020-09-291-0/+18
| | | | | | | | | | | | | | | | | | | | * Fix table scan of events on worker startup. This happened because we assumed "new" writers had an initial stream position of 0, so the replication code tried to fetch all events written by the instance between 0 and the current position. Instead, set the initial position of new writers to the current persisted up to position, on the assumption that new writers won't have written anything before that point. * Consider old writers coming back as "new". Otherwise we'd try and fetch entries between the old stale token and the current position, even though it won't have written any rows. Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Filter out appservices from mau count (#8404)Will Hunt2020-09-291-1/+16
| | | This is an attempt to fix #8403.
* Add checks for postgres sequence consistency (#8402)Erik Johnston2020-09-281-3/+19
|
* Fix MultiWriteIdGenerator's handling of restarts. (#8374)Erik Johnston2020-09-241-8/+111
| | | | | | | | | | | | | | | | | | | On startup `MultiWriteIdGenerator` fetches the maximum stream ID for each instance from the table and uses that as its initial "current position" for each writer. This is problematic as a) it involves either a scan of events table or an index (neither of which is ideal), and b) if rows are being persisted out of order elsewhere while the process restarts then using the maximum stream ID is not correct. This could theoretically lead to race conditions where e.g. events that are persisted out of order are not sent down sync streams. We fix this by creating a new table that tracks the current positions of each writer to the stream, and update it each time we finish persisting a new entry. This is a relatively small overhead when persisting events. However for the cache invalidation stream this is a much bigger relative overhead, so instead we note that for invalidation we don't actually care about reliability over restarts (as there's no caches to invalidate) and simply don't bother reading and writing to the new table in that particular case.
* Use `async with` for ID gens (#8383)Erik Johnston2020-09-231-25/+41
| | | This will allow us to hit the DB after we've finished using the generated stream ID.
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-2/+2
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Fix `MultiWriterIdGenerator.current_position`. (#8257)Erik Johnston2020-09-081-0/+50
| | | | | It did not correctly handle IDs finishing being persisted out of order, resulting in the `current_position` lagging until new IDs are persisted.
* Allow for make_awaitable's return value to be re-used. (#8261)Patrick Cloke2020-09-082-13/+5
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-041-9/+9
|
* Re-implement unread counts (again) (#8059)Brendan Abolivier2020-09-021-2/+6
|
* Make MultiWriterIDGenerator work for streams that use negative stream IDs ↵Erik Johnston2020-09-011-0/+105
| | | | | (#8203) This is so that we can use it for the backfill events stream.
* Rename `get_e2e_device_keys` to better reflect its purpose (#8205)Richard van der Hoff2020-08-291-3/+5
| | | | | | | | | | | ... and to show that it does something slightly different to `_get_e2e_device_keys_txn`. `include_all_devices` and `include_deleted_devices` were never used (and `include_deleted_devices` was broken, since that would cause `None`s in the result which were not handled in the loop below. Add some typing too.
* Do not yield on awaitables in tests. (#8193)Patrick Cloke2020-08-277-68/+143
|
* Convert stats and related calls to async/await (#8192)Patrick Cloke2020-08-271-3/+2
|
* Convert simple_delete to async/await. (#8191)Patrick Cloke2020-08-271-2/+4
|
* Fix missing _add_persisted_position (#8179)Erik Johnston2020-08-271-3/+49
| | | This was forgotten in #8164.
* Convert additional database methods to async (select list, search, ↵Patrick Cloke2020-08-272-5/+6
| | | | insert_many, delete_*) (#8168)
* Convert simple_update* and simple_select* to async (#8173)Patrick Cloke2020-08-273-12/+24
|
* Reduce run-times of tests by advancing the reactor less (#7757)Andrew Morgan2020-08-272-4/+4
|
* Convert simple_select_one and simple_select_one_onecol to async (#8162)Patrick Cloke2020-08-265-25/+56
|
* Add functions to `MultiWriterIdGen` used by events stream (#8164)Erik Johnston2020-08-251-0/+36
|
* Don't fail /submit_token requests on incorrect session ID if ↵Brendan Abolivier2020-08-241-0/+31
| | | | | | | | | | | | | | | | | | | request_token_inhibit_3pid_errors is turned on (#7991) * Don't raise session_id errors on submit_token if request_token_inhibit_3pid_errors is set * Changelog * Also wait some time before responding to /requestToken * Incorporate review * Update synapse/storage/databases/main/registration.py Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> * Incorporate review Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Switch the JSON byte producer from a pull to a push producer. (#8116)Patrick Cloke2020-08-191-1/+2
|
* Separate `get_current_token` into two. (#8113)Erik Johnston2020-08-191-8/+8
| | | | | | | | | | | | The function is used for two purposes: 1) for subscribers of streams to get a token they can use to get further updates with, and 2) for replication to track position of the writers of the stream. For streams with a single writer the two scenarios produce the same result, however the situation becomes complicated for streams with multiple writers. The current `MultiWriterIdGenerator` does not correctly handle the first case (which is not an issue as its only used for the `caches` stream which nothing subscribes to outside of replication).
* Convert events worker database to async/await. (#8071)Patrick Cloke2020-08-181-1/+2
|
* Convert some of the general database methods to async (#8100)Patrick Cloke2020-08-175-27/+41
|
* Convert stream database to async/await. (#8074)Patrick Cloke2020-08-171-33/+16
|
* Add a shadow-banned flag to users. (#8092)Patrick Cloke2020-08-143-4/+4
|
* Convert devices database to async/await. (#8069)Patrick Cloke2020-08-122-22/+38
|
* Convert appservice, group server, profile and more databases to async (#8066)Patrick Cloke2020-08-121-12/+12
|
* Converts event_federation and registration databases to async/await (#8061)Patrick Cloke2020-08-112-8/+18
|
* Add an assertion on prev_events in create_new_client_event (#8041)Richard van der Hoff2020-08-101-0/+4
| | | | | | I think this would have caught all the cases in https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes more sense here than a 403
* Convert directory, e2e_room_keys, end_to_end_keys, monthly_active_users ↵Patrick Cloke2020-08-073-17/+44
| | | | database to async (#8042)
* Convert additional database stores to async/await (#8045)Patrick Cloke2020-08-071-2/+2
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-0512-75/+79
|
* Convert some of the data store to async. (#7976)Patrick Cloke2020-07-303-19/+29
|
* Convert storage layer to async/await. (#7963)Patrick Cloke2020-07-283-29/+49
|
* Convert a synapse.events to async/await. (#7949)Patrick Cloke2020-07-271-1/+3
|
* Convert push to async/await. (#7948)Patrick Cloke2020-07-271-2/+4
|
* Convert state resolution to async/await (#7942)Patrick Cloke2020-07-241-2/+6
|
* Convert the message handler to async/await. (#7884)Patrick Cloke2020-07-222-26/+34
|
* Add delete room admin endpoint (#7613)Dirk Klimpel2020-07-141-0/+8
| | | | | | | | | | | | | | | | | | The Delete Room admin API allows server admins to remove rooms from server and block these rooms. `DELETE /_synapse/admin/v1/rooms/<room_id>` It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API. Fixes: #6425 It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`. It should return `None` if the room is unknown. But it returns an `IndexError`. https://github.com/matrix-org/synapse/blob/901b1fa561e3cc661d78aa96d59802cf2078cb0d/synapse/storage/data_stores/main/room.py#L99-L105 Related to: - #5575 - https://github.com/Awesome-Technologies/synapse-admin/issues/17 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-5/+2
| | | ... instead of duplicating `config.signing_key[0]` everywhere
* Back out MSC2625 implementation (#7761)Brendan Abolivier2020-07-011-29/+16
|
* Remove debug loggingBrendan Abolivier2020-06-121-2/+0
|
* Test that a mark_unread action updates the right counterBrendan Abolivier2020-06-121-5/+16
|
* Fix testsBrendan Abolivier2020-06-101-14/+18
|
* Replace instances of reactor pumping with get_success. (#7619)Andrew Morgan2020-06-031-118/+149
| | | | | Calls `self.get_success` on all deferred methods instead of abusing `self.pump()`. This has the benefit of working with coroutines, as well as checking that method execution completed successfully. There are also a few small cleanups that I made in the process.
* Optimise some references to hs.config (#7546)Richard van der Hoff2020-05-222-58/+65
| | | These are surprisingly expensive, and we only really need to do them at startup.
* Add ability to wait for replication streams (#7542)Erik Johnston2020-05-222-3/+3
| | | | | | | The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room). Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on. People probably want to look at this commit by commit.
* Merge tag 'v1.13.0rc2' into developRichard van der Hoff2020-05-141-1/+49
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.13.0rc2 (2020-05-14) ============================== Bugfixes -------- - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376)) - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483)) Internal Changes ---------------- - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
| * Stop `get_joined_users` corruption from custom statuses (#7376)Richard van der Hoff2020-05-141-1/+49
| | | | | | | | | | | | | | | | Fix a bug where the `get_joined_users` cache could be corrupted by custom status events (or other state events with a state_key matching the user ID). The bug was introduced by #2229, but has largely gone unnoticed since then. Fixes #7099, #7373.
* | Shuffle persist event data store functions. (#7440)Erik Johnston2020-05-131-1/+2
| | | | | | | | | | | | | | | | | | | | The aim here is to get to a stage where we have a `PersistEventStore` that holds all the write methods used during event persistence, so that we can take that class out of the `DataStore` mixin and instansiate it separately. This will allow us to instansiate it on processes other than master, while also ensuring it is only available on processes that are configured to write to events stream. This is a bit of an architectural change, where we end up with multiple classes per data store (rather than one per data store we have now). We end up having: 1. Storage classes that provide high level APIs that can talk to multiple data stores. 2. Data store modules that consist of classes that must point at the same database instance. 3. Classes in a data store that can be instantiated on processes depending on config.
* | Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-113-10/+11
| | | | | | | | variables (#6391)
* | Add room details admin endpoint (#7317)Manuel Stahl2020-05-071-0/+11
| |
* | Add MultiWriterIdGenerator. (#7281)Erik Johnston2020-05-041-0/+184
|/ | | | | | This will be used to coordinate stream IDs across multiple writers. Functions as the equivalent of both `StreamIdGenerator` and `SlavedIdTracker`.
* Return total number of users and profile attributes in admin users endpoint ↵Manuel Stahl2020-04-281-0/+46
| | | | | (#6881) Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
* bg update to clear out duplicate outbound_device_list_pokes (#7193)Richard van der Hoff2020-04-071-0/+52
| | | | We seem to have some duplicates, which could do with being cleared out.
* Make do_next_background_update return a boolRichard van der Hoff2020-03-311-3/+3
| | | | returning a None or an int that we don't use is confusing.
* Make `has_completed_background_updates` asyncRichard van der Hoff2020-03-311-1/+3
| | | | (Almost) everywhere that uses it is happy with an awaitable.
* Remove unused `start_background_update`Richard van der Hoff2020-03-311-5/+9
| | | | This was only used in a unit test, so let's just inline it in the test.
* Change device list streams to have one row per ID (#7010)Erik Johnston2020-03-191-45/+0
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add 'device_lists_outbound_pokes' as extra table. This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change. * Change device lists stream to have one row per id. This will make it possible to process the streams more incrementally, avoiding having to process large chunks at once. * Change device list replication to match new semantics. Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host). * Newsfile * Remove handling of multiple rows per ID * Fix worker handling * Comments from review
| * Remove handling of multiple rows per IDErik Johnston2020-02-281-45/+0
| |
* | Improve get auth chain difference algorithm. (#7095)Erik Johnston2020-03-181-17/+140
| | | | | | | | | | | | | | | | | | | | | | It was originally implemented by pulling the full auth chain of all state sets out of the database and doing set comparison. However, that can take a lot work if the state and auth chains are large. Instead, lets try and fetch the auth chains at the same time and calculate the difference on the fly, allowing us to bail early if all the auth chains converge. Assuming that the auth chains do converge more often than not, this should improve performance. Hopefully.
* | Break down monthly active users by appservice_id (#7030)Neil Johnson2020-03-061-0/+42
|/ | | | | * Break down monthly active users by appservice_id and emit via prometheus. Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-215-46/+42
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Replace _event_dict_property with DictPropertyRichard van der Hoff2020-02-141-1/+1
| | | | | this amounts to the same thing, but replaces `_event_dict` with `_dict`, and removes some of the function layers generated by `property`.
* Add typing to SyncHandler (#6821)Erik Johnston2020-02-031-1/+4
| | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Type defintions for use in refactoring for redaction changes (#6803)Richard van der Hoff2020-01-301-4/+11
| | | | | | | | | | * Bump signedjson to 1.1 ... so that we can use the type definitions * Fix breakage caused by upgrade to signedjson 1.1 Thanks, @illicitonion...
* Add `rooms.room_version` column (#6729)Erik Johnston2020-01-272-2/+10
| | | This is so that we don't have to rely on pulling it out from `current_state_events` table.
* Make 'event.redacts' never raise. (#6771)Erik Johnston2020-01-231-0/+35
| | | | | | There are quite a few places that we assume that a redaction event has a corresponding `redacts` key, which is not always the case. So lets cheekily make it so that event.redacts just returns None instead.
* Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-1/+1
| | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* Allow admin users to create or modify users without a shared secret (#6495)Manuel Stahl2020-01-091-0/+2
| | | Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
* port BackgroundUpdateTestCase to HomeserverTestCase (#6653)Richard van der Hoff2020-01-071-35/+37
|
* Remove unused get_prev_events_and_hashes_for_roomRichard van der Hoff2020-01-061-13/+6
|
* rename get_prev_events_for_room to get_prev_events_and_hashes_for_roomRichard van der Hoff2020-01-061-2/+2
| | | | ... to make way for a new method which just returns the event ids
* Split state groups into a separate data store (#6296)Erik Johnston2019-12-201-1/+1
|
* Add database config class (#6513)Erik Johnston2019-12-183-18/+34
| | | | | This encapsulates config for a given database and is the way to get new connections.
* Fix race which caused deleted devices to reappear (#6514)Richard van der Hoff2019-12-101-20/+29
| | | | Stop the `update_client_ips` background job from recreating deleted devices.
* Fixup testsErik Johnston2019-12-064-11/+11
|
* Change DataStores to accept 'database' param.Erik Johnston2019-12-061-2/+3
|
* Move background update handling out of storeErik Johnston2019-12-054-25/+59
|
* Move DB pool and helper functions into dedicated Database classErik Johnston2019-12-0510-41/+43
|
* Remove underscore from SQLBaseStore functionsErik Johnston2019-12-046-24/+24
|
* Implementation of MSC2314 (#6176)Amber Brown2019-11-281-25/+1
|
* add etag and count to key backup endpoints (#5858)Hubert Chathi2019-11-271-4/+4
|
* Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-11-041-7/+9
|\ | | | | | | erikj/split_purge_history
| * Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-311-48/+102
| |\
| * | rename get_devices_by_remote to get_device_updates_by_remoteHubert Chathi2019-10-301-6/+6
| | |
| * | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-306-9/+88
| |\ \
| * \ \ Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-244-3/+5
| |\ \ \
| * | | | fix unit testHubert Chathi2019-10-221-1/+3
| | | | |
* | | | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-311-48/+102
|\ \ \ \ \ | | |_|_|/ | |/| | | | | | | | erikj/split_purge_history
| * | | | Port to use state storageErik Johnston2019-10-301-48/+102
| | |_|/ | |/| |
* / | | Split purge API into events vs stateErik Johnston2019-10-301-7/+8
|/ / /
* | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-302-1/+76
|\ \ \ | | | | | | | | | | | | erikj/split_out_persistence_store
| * | | Make ObservableDeferred.observe() always return deferred.Erik Johnston2019-10-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.
| * | | remove unneeded importsHubert Chathi2019-10-251-3/+1
| | | |
| * | | switch to using HomeserverTestCaseHubert Chathi2019-10-251-19/+25
| | | |
| * | | remove some unnecessary linesHubert Chathi2019-10-241-5/+0
| | | |
| * | | delete keys when deleting backupsHubert Chathi2019-10-241-0/+76
| | |/ | |/|
* / | Use new EventPersistenceStoreErik Johnston2019-10-234-8/+12
|/ /
* / Move storage classes into a main "data store".Erik Johnston2019-10-214-3/+5
|/ | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storageHubert Chathi2019-10-111-6/+6
|\ | | | | make storage layer in charge of interpreting the device key data
| * make storage layer in charge of interpreting the device key dataHubert Chathi2019-10-101-6/+6
| |
* | Fix MAU reaping where reserved users are specified. (#6168)Neil Johnson2019-10-111-9/+49
| |
* | Fix postgres unit testsErik Johnston2019-10-101-1/+1
|/
* Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timingsErik Johnston2019-10-021-0/+11
|\ | | | | Fix errors storing large retry intervals.
| * Fix errors storing large retry intervals.Erik Johnston2019-10-021-0/+11
| | | | | | | | | | | | | | | | | | We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).
* | Fix fetching censored redactions from DBErik Johnston2019-10-021-0/+36
|/ | | | | | Fetching a censored redactions caused an exception due to the code expecting redactions to have a `redact` key, which redacted redactions don't have.
* Fix dummy event insertion consent bug (#6053)Neil Johnson2019-09-262-9/+178
| | | Fixes #5905
* Test that pruning of old user IPs worksErik Johnston2019-09-241-0/+71
|
* Test background updateErik Johnston2019-09-231-0/+79
|
* Query devices table for last seen info.Erik Johnston2019-09-231-1/+0
| | | | | | | | This is a) simpler than querying user_ips directly and b) means we can purge older entries from user_ips without losing the required info. The storage functions now no longer return the access_token, since it was unused.
* Add 'failure_ts' column to 'destinations' table (#6016)Richard van der Hoff2019-09-171-3/+5
| | | | Track the time that a server started failing at, for general analysis purposes.
* Merge pull request #5934 from matrix-org/erikj/censor_redactionsErik Johnston2019-09-091-1/+76
|\ | | | | Censor redactions in DB after a month
| * Fix testErik Johnston2019-09-051-1/+3
| |
| * Make redaction retention period configurableErik Johnston2019-09-051-1/+3
| |
| * Censor redactions in DB after a monthErik Johnston2019-08-301-0/+71
| |
* | Servers-known-about statistic (#5981)Amber Brown2019-09-071-37/+89
|/
* Fix registration testHalf-Shot2019-08-231-0/+1
|
* Merge pull request #5788 from matrix-org/rav/metaredactionsRichard van der Hoff2019-08-121-0/+70
|\ | | | | Fix handling of redactions of redactions
| * Handle loops in redaction eventsRichard van der Hoff2019-07-301-0/+70
| |
* | Merge pull request #5746 from matrix-org/erikj/test_bg_update_currnet_stateErik Johnston2019-07-301-1/+36
|\ \ | |/ |/| Add unit test for current state membership bg update
| * Add unit test for current state membership bg updateErik Johnston2019-07-231-1/+36
| |
* | Convert RedactionTestCase to modern test style (#5768)Richard van der Hoff2019-07-261-36/+38
| |
* | Replace returnValue with return (#5736)Amber Brown2019-07-234-6/+6
|/
* Support Prometheus_client 0.4.0+ (#5636)Amber Brown2019-07-181-3/+1
|
* Implement access token expiry (#5660)Richard van der Hoff2019-07-121-3/+5
| | | | Record how long an access token is valid for, and raise a soft-logout once it expires.
* Remove access-token support from RegistrationStore.register (#5642)Richard van der Hoff2019-07-103-41/+17
| | | | | The 'token' param is no longer used anywhere except the tests, so let's kill that off too.
* Run Black. (#5482)Amber Brown2019-06-2011-72/+74
|
* Run as background process and fix commentsErik Johnston2019-06-191-1/+1
|
* Add experimental option to reduce extremities.Erik Johnston2019-06-181-0/+41
| | | | | | | Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.
* Prometheus histograms are cumalativeErik Johnston2019-06-141-10/+10
|
* fix prometheus rendering errorAmber H. Brown2019-06-141-38/+23
|
* Expose statistics on extrems to prometheus (#5384)Amber Brown2019-06-132-79/+146
|
* Prevent multiple device list updates from breaking a batch send (#5156)Andrew Morgan2019-06-061-0/+69
| | | fixes #5153
* Set default room version to v4. (#5379)Neil Johnson2019-06-061-0/+6
| | | | Set default room version to v4.
* Add testErik Johnston2019-05-291-0/+248
|
* Store key validity time in the storage layerRichard van der Hoff2019-05-231-14/+30
| | | | | | | | This is a first step to checking that the key is valid at the required moment. The idea here is that, rather than passing VerifyKey objects in and out of the storage layer, we instead pass FetchKeyResult objects, which simply wrap the VerifyKey and add a valid_until_ts field.
* Rewrite store_server_verify_key to store several keys at once (#5234)Richard van der Hoff2019-05-231-14/+30
| | | | | Storing server keys hammered the database a bit. This replaces the implementation which stored a single key, with one which can do many updates at once.
* Expose DataStore._get_events as get_events_as_listErik Johnston2019-05-141-1/+1
| | | | This is in preparation for reaction work which requires it.
* Run Black on the tests again (#5170)Amber Brown2019-05-109-80/+44
|
* Add admin api for sending server_notices (#5121)Richard van der Hoff2019-05-021-1/+4
|
* Move admin api impl to its own packageRichard van der Hoff2019-05-011-2/+3
| | | | It doesn't really belong under rest/client/v1 any more.
* Rewrite Datastore.get_server_verify_keysRichard van der Hoff2019-04-091-3/+50
| | | | Rewrite this so that it doesn't hammer the database.
* Rewrite test_keys as a HomeserverTestCaseRichard van der Hoff2019-04-081-19/+15
|
* Remove presence lists (#4989)Neil Johnson2019-04-031-118/+0
| | | Remove presence list support as per MSC 1819
* Collect room-version variations into one place (#4969)Richard van der Hoff2019-04-013-8/+11
| | | | Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.
* Migrate the user directory initial population to a background task (#4864)Amber Brown2019-03-191-8/+3
|
* fixupAmber Brown2019-03-121-2/+2
|
* Rewrite userdir to be faster (#4537)Amber Brown2019-03-071-2/+0
|
* UPSERT many functionality (#4644)Amber Brown2019-02-201-0/+88
|