summary refs log tree commit diff
path: root/tests/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Merge commit 'a7a379006' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-3/+73
|\
| * Use the chain cover index in get_auth_chain_ids. (#9576)Patrick Cloke2021-03-101-3/+73
| | | | | | | | This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.
* | Merge commit '22db45bd4' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-26/+45
|\|
| * Add a basic test for purging rooms. (#9541)Patrick Cloke2021-03-081-26/+45
| | | | | | | | | | Unfortunately this doesn't test re-joining the room since that requires having another homeserver to query over federation, which isn't easily doable in unit tests.
* | Merge commit '8ec221710' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-231-0/+1
|\|
| * Add the shadow-banning status to the display user admin API. (#9400)Dirk Klimpel2021-02-171-0/+1
| |
* | Merge commit 'd2f0ec12d' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-2312-47/+89
|\|
| * Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1612-47/+89
| | | | | | | | | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* | Merge commit '73b03722f' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-6/+106
|\|
| * Allow moving account data and receipts streams off master (#9104)Erik Johnston2021-01-181-6/+106
| |
* | Merge commit '2b467d0b6' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-31/+186
|\|
| * Make chain cover index bg update go faster (#9124)Erik Johnston2021-01-151-31/+186
| | | | | | | | | | | | We do this by allowing a single iteration to process multiple rooms at a time, as there are often a lot of really tiny rooms, which can massively slow things down.
* | Merge commit '631dd06f2' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-221-0/+114
|\|
| * Add background update for add chain cover index (#9029)Erik Johnston2021-01-141-0/+114
| |
* | Merge commit '98a64b7f7' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-223-24/+723
|\|
| * Remove user's avatar URL and displayname when deactivated. (#8932)Dirk Klimpel2021-01-121-0/+26
| | | | | | This only applies if the user's data is to be erased.
| * Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-112-24/+697
| |
* | Merge commit '23d701864' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-211-0/+120
|\|
| * Improve the performance of calculating ignored users in large rooms (#9024)Patrick Cloke2021-01-071-0/+120
| | | | | | | | | | | | This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>
* | Merge commit 'a7a913918' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-201-0/+334
|\|
| * Try and drop stale extremities. (#8929)Erik Johnston2020-12-181-0/+334
| | | | | | | | If we see stale extremities while persisting events, and notice that they don't change the result of state resolution, we drop them.
* | Merge commit 'c9c1c9d82' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-202-0/+30
|\|
| * Use the simple dictionary in full text search for the user directory (#8959)Brendan Abolivier2020-12-171-0/+23
| | | | | | | | | | * Use the simple dictionary in fts for the user directory * Clarify naming
| * Make search statement in List Room and User Admin API case-insensitive (#8931)Dirk Klimpel2020-12-171-0/+7
| |
* | Merge commit 'f14428b25' into anoa/dinsic_release_1_31_0Andrew Morgan2021-04-161-0/+26
|\|
| * Add number of local devices to Room Details Admin API (#8886)Dirk Klimpel2020-12-111-0/+26
| |
* | Merge commit 'cf7d3c90d' into dinsicAndrew Morgan2021-04-163-23/+17
|\|
| * Pass room_id to get_auth_chain_difference (#8879)Erik Johnston2020-12-041-8/+10
| | | | | | | | | | This is so that we can choose which algorithm to use based on the room ID.
| * Remove redundant mockingRichard van der Hoff2020-12-022-16/+3
| |
| * Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-024-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
| * Correctly handle unpersisted events when calculating auth chain difference. ↵Erik Johnston2020-12-021-0/+5
| | | | | | | | | | | | | | | | | | (#8827) We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as: 1. the next time a state event happens in the room the current state should correct itself; 2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and 3. we mostly use the state at an event to do important operations, which isn't affected by this.
* | Port "Add support for no_proxy and case insensitive env variables" from ↵Andrew Morgan2021-03-224-4/+4
| | | | | | | | | | | | | | mainline to dinsic (#93) This PR is simply porting https://github.com/matrix-org/synapse/pull/9372 to dinsic. I also had to bring in https://github.com/matrix-org/synapse/pull/8821 and https://github.com/matrix-org/synapse/pull/9084 for this code to work properly - a sign that we should merge mainline into dinsic again soon.
* | Merge commit '7c4344747' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-311-1/+1
|\|
| * Remove deprecated `/_matrix/client/*/admin` endpoints (#8785)Dirk Klimpel2020-11-251-1/+1
| | | | | | These are now only available via `/_synapse/admin/v1`.
| * Merge pull request #8761 from matrix-org/rav/test_request_renderingRichard van der Hoff2020-11-171-2/+1
| |\ | | | | | | Make `make_request` actually render the request
| * | Add admin API for logging in as a user (#8617)Erik Johnston2020-11-171-30/+0
| | |
* | | Merged commit 'deff8f628' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-311-30/+0
| | |
* | | Merge commit 'acfe3b306' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-311-2/+1
|\ \ \ | | |/ | |/|
| * | Remove redundant calls to `render()`Richard van der Hoff2020-11-161-2/+1
| |/
| * fixup testRichard van der Hoff2020-11-161-0/+1
| |
* | Merge commit 'f12589547' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-311-6/+8
|\|
| * Add a `custom_headers` param to `make_request` (#8760)Richard van der Hoff2020-11-161-6/+7
| | | | | | | | Some tests want to set some custom HTTP request headers, so provide a way to do that before calling requestReceived().
* | Merge commit 'ef2d62701' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-311-6/+4
|\|
| * Add ability for access tokens to belong to one user but grant access to ↵Erik Johnston2020-10-291-6/+4
| | | | | | | | | | | | | | | | | | | | another user. (#8616) We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't). A future PR will add an API for creating such a token. When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
* | Merge commit 'c97da1e45' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-311-1/+1
|\|
| * Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
| | | | | | | | | | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
| * Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt2020-10-261-0/+56
| |
* | Merge commit '7a3adbd7a' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-311-0/+56
|\ \
| * | Fix get|set_type_stream_id_for_appservice store functions (#8648)Will Hunt2020-10-261-0/+56
| | |
* | | Merge commit 'b19b63e6b' into anoa/dinsic_release_1_23_1Andrew Morgan2020-12-313-7/+7
|\ \ \ | | |/ | |/|
| * | Don't instansiate Requester directly (#8614)Erik Johnston2020-10-223-7/+7
| |/
* | Merge commit '9146a8a69' into dinsicAndrew Morgan2020-12-311-228/+0
|\|
| * Combine the two sets of tests for CacheDescriptorRichard van der Hoff2020-10-211-228/+0
| |
* | Merge commit '3ee17585c' into dinsicAndrew Morgan2020-12-311-4/+4
|\|
| * Send some ephemeral events to appservices (#8437)Will Hunt2020-10-151-4/+4
| | | | | | Optionally sends typing, presence, and read receipt information to appservices.
* | Merge commit '74976a8e4' into dinsicAndrew Morgan2020-12-314-88/+28
|\|
| * Merge pull request #8548 from matrix-org/rav/deferred_cacheRichard van der Hoff2020-10-151-72/+1
| |\ | | | | | | Rename Cache to DeferredCache, and related changes
| | * Combine the two sets of DeferredCache testsRichard van der Hoff2020-10-141-72/+0
| | |
| | * move DeferredCache into its own moduleRichard van der Hoff2020-10-141-1/+2
| | |
| | * Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-5/+5
| | |
| * | Simplify `_locally_reject_invite`Richard van der Hoff2020-10-131-2/+2
| |/ | | | | | | | | Update `EventCreationHandler.create_event` to accept an auth_events param, and use it in `_locally_reject_invite` instead of reinventing the wheel.
| * Make event persisters periodically announce position over replication. (#8499)Erik Johnston2020-10-121-9/+16
| | | | | | | | | | Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress. This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
| * Reduce serialization errors in MultiWriterIdGen (#8456)Erik Johnston2020-10-071-0/+1
| | | | | | | | | | | | We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.
| * Add logging on startup/shutdown (#8448)Erik Johnston2020-10-021-5/+9
| | | | | | | | | | This is so we can tell what is going on when things are taking a while to start up. The main change here is to ensure that transactions that are created during startup get correctly logged like normal transactions.
* | Merge commit 'f76194a02' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-211-0/+1
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'f76194a02': 1.21.0 Update change log 1.21.0rc3 Reduce serialization errors in MultiWriterIdGen (#8456) Add Ubuntu 20.10 (Groovy Gorilla) to build scripts. (#8475) move #8444 to 'feature' linkify changelog
| * | Reduce serialization errors in MultiWriterIdGen (#8456)Erik Johnston2020-10-071-0/+1
| |/ | | | | | | | | | | | | We call `_update_stream_positions_table_txn` a lot, which is an UPSERT that can conflict in `REPEATABLE READ` isolation level. Instead of doing a transaction consisting of a single query we may as well run it outside of a transaction.
* | Merge commit 'cc40a59b4' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-212-14/+18
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'cc40a59b4': 1.21.0 Add prometheus metrics to track federation delays (#8430) Make token serializing/deserializing async (#8427) Allow additional SSO properties to be passed to the client (#8413) changelog Add an improved "forward extremities" metric Rewrite BucketCollector Fix _exposition.py to stop stripping samples Drop support for ancient prometheus_client (#8426) Various clean ups to room stream tokens. (#8423) changelog Report state res metrics to Prometheus and log Move Measure calls into `resolve_events_with_store` Expose a `get_resource_usage` method in `Measure` Move `resolve_events_with_store` into StateResolutionHandler
| * Make token serializing/deserializing async (#8427)Erik Johnston2020-09-301-3/+6
| | | | | | The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
| * Rewrite BucketCollectorRichard van der Hoff2020-09-301-8/+11
| | | | | | | | | | | | | | | | | | | | | | | | This was a bit unweildy for what I wanted: in particular, I wanted to assign each measurement straight into a bucket, rather than storing an intermediate Counter which didn't do any bucketing at all. I've replaced it with something that is hopefully a bit easier to use. (I'm not entirely sure what the difference between a HistogramMetricFamily and a GaugeHistogramMetricFamily is, but given our counters can go down as well as up the latter *sounds* more accurate?)
| * Various clean ups to room stream tokens. (#8423)Erik Johnston2020-09-291-6/+4
| |
* | Merge commit '8238b55e0' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-211-0/+18
|\| | | | | | | | | | | | | | | * commit '8238b55e0': Update description of server_name config option (#8415) Discard an empty upload_name before persisting an uploaded file (#7905) Don't table scan events on worker startup (#8419) Mypy fixes for `synapse.handlers.federation` (#8422)
| * Don't table scan events on worker startup (#8419)Erik Johnston2020-09-291-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix table scan of events on worker startup. This happened because we assumed "new" writers had an initial stream position of 0, so the replication code tried to fetch all events written by the instance between 0 and the current position. Instead, set the initial position of new writers to the current persisted up to position, on the assumption that new writers won't have written anything before that point. * Consider old writers coming back as "new". Otherwise we'd try and fetch entries between the old stale token and the current position, even though it won't have written any rows. Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Merge commit 'f43c66d23' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-212-4/+35
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'f43c66d23': Add support for running Complement against the local checkout (#8317) Filter out appservices from mau count (#8404) Only assert valid next_link params when provided (#8417) Add metrics to track success/otherwise of replication requests (#8406) Fix handling of connection timeouts in outgoing http requests (#8400) Changelog Don't check whether a 3pid is allowed to register during password reset Add checks for postgres sequence consistency (#8402) Create a mechanism for marking tests "logcontext clean" (#8399) Add `ui_auth_sessions_ips` table to `synapse_port_db` ignore list (#8410) A pair of tiny cleanups in the federation request code. (#8401) typo
| * Filter out appservices from mau count (#8404)Will Hunt2020-09-291-1/+16
| | | | | | This is an attempt to fix #8403.
| * Add checks for postgres sequence consistency (#8402)Erik Johnston2020-09-281-3/+19
| |
* | Merge commit '31acc5c30' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-211-8/+111
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit '31acc5c30': Escape the error description on the sso_error template. (#8405) Fix occasional "Re-starting finished log context" from keyring (#8398) Allow existing users to login via OpenID Connect. (#8345) Fix schema delta for servers that have not backfilled (#8396) Fix MultiWriteIdGenerator's handling of restarts. (#8374) s/URLs/variables in changelog s/accidentally/incorrectly in changelog Update changelog wording Add type annotations to SimpleHttpClient (#8372) Add new sequences to port DB script (#8387) Add EventStreamPosition type (#8388) Mark the shadow_banned column as boolean in synapse_port_db. (#8386)
| * Fix MultiWriteIdGenerator's handling of restarts. (#8374)Erik Johnston2020-09-241-8/+111
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On startup `MultiWriteIdGenerator` fetches the maximum stream ID for each instance from the table and uses that as its initial "current position" for each writer. This is problematic as a) it involves either a scan of events table or an index (neither of which is ideal), and b) if rows are being persisted out of order elsewhere while the process restarts then using the maximum stream ID is not correct. This could theoretically lead to race conditions where e.g. events that are persisted out of order are not sent down sync streams. We fix this by creating a new table that tracks the current positions of each writer to the stream, and update it each time we finish persisting a new entry. This is a relatively small overhead when persisting events. However for the cache invalidation stream this is a much bigger relative overhead, so instead we note that for invalidation we don't actually care about reliability over restarts (as there's no caches to invalidate) and simply don't bother reading and writing to the new table in that particular case.
* | Merge commit '2983049a7' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-211-25/+41
|\| | | | | | | | | | | | | | | | | | | | | | | * commit '2983049a7': Factor out `_send_dummy_event_for_room` (#8370) Improve logging of state resolution (#8371) Fix bug which caused failure on join with malformed membership events (#8385) Use `async with` for ID gens (#8383) Don't push if an user account has expired (#8353) Do not check lint/test dependencies at runtime. (#8377) Add note to reverse_proxy.md about disabling Apache's mod_security2 (#8375) Changelog
| * Use `async with` for ID gens (#8383)Erik Johnston2020-09-231-25/+41
| | | | | | This will allow us to hit the DB after we've finished using the generated stream ID.
* | Merge commit '4325be1a5' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-212-2/+2
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit '4325be1a5': Fix missing null character check on guest_access room state Fixed a bug with reactivating users with the admin API (#8362) Admin API for reported events (#8217) Fix wording of deprecation notice in changelog Deprecation warning for synapse admin api being accessible under /_matrix Create function to check for long names in devices (#8364) Add a comment re #1691 Fix a bad merge from release-v1.20.0. (#8354) Admin API for querying rooms where a user is a member (#8306) Catch-up after Federation Outage (bonus): Catch-up on Synapse Startup (#8322) Simplify super() calls to Python 3 syntax. (#8344) Allow appservice users to /login (#8320) Update test logging to be able to accept braces (#8335) Move lint dependencies to extras_require (#8330)
| * Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-2/+2
| | | | | | | | | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* | Merge commit 'e45b83411' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-201-0/+50
|\| | | | | | | | | | | | | | | | | | | | | | | * commit 'e45b83411': Add types to async_helpers (#8260) Fix mypy error on develop (#8282) Include method in thumbnail media name (#7124) Add types to StreamToken and RoomStreamToken (#8279) Add a config option for validating 'next_link' parameters against a domain whitelist (#8275) Clean up types for PaginationConfig (#8250) Use the right constructor for log records (#8278) Fix `MultiWriterIdGenerator.current_position`. (#8257)
| * Fix `MultiWriterIdGenerator.current_position`. (#8257)Erik Johnston2020-09-081-0/+50
| | | | | | | | | | It did not correctly handle IDs finishing being persisted out of order, resulting in the `current_position` lagging until new IDs are persisted.
* | Merge commit 'cca03dbec' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-202-13/+5
|\| | | | | | | | | | | | | | | | | * commit 'cca03dbec': fix typo s/fixes/fix/ Directly import json from the standard library. (#8259) Allow for make_awaitable's return value to be re-used. (#8261) Rename 'populate_stats_process_rooms_2' background job back to 'populate_stats_process_rooms' again (#8243)
| * Allow for make_awaitable's return value to be re-used. (#8261)Patrick Cloke2020-09-082-13/+5
| |
* | Merge commit '17fa4c7ca' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-201-9/+9
|\| | | | | | | | | | | | | | | * commit '17fa4c7ca': Catch up after Federation Outage (split, 2): Track last successful stream ordering after transmission (#8247) Catch-up after Federation Outage (split, 1) (#8230) Fix type signature in simple_select_one_onecol and friends (#8241) Stop sub-classing object (#8249)
| * Stop sub-classing object (#8249)Patrick Cloke2020-09-041-9/+9
| |
* | Merge commit '112266eaf' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-201-2/+6
|\| | | | | | | | | | | * commit '112266eaf': Add StreamStore to mypy (#8232) Re-implement unread counts (again) (#8059)
| * Re-implement unread counts (again) (#8059)Brendan Abolivier2020-09-021-2/+6
| |
* | Merge commit '5bf8e5f55' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-202-3/+110
|\| | | | | | | | | | | | | | | | | | | | | * commit '5bf8e5f55': Convert the well known resolver to async (#8214) Convert additional databases to async/await part 2 (#8200) Make MultiWriterIDGenerator work for streams that use negative stream IDs (#8203) Do not install setuptools 50.0. (#8212) Move and rename `get_devices_with_keys_by_user` (#8204) Rename `get_e2e_device_keys` to better reflect its purpose (#8205) Add a comment about _LimitedHostnameResolver
| * Make MultiWriterIDGenerator work for streams that use negative stream IDs ↵Erik Johnston2020-09-011-0/+105
| | | | | | | | | | (#8203) This is so that we can use it for the backfill events stream.
| * Rename `get_e2e_device_keys` to better reflect its purpose (#8205)Richard van der Hoff2020-08-291-3/+5
| | | | | | | | | | | | | | | | | | | | | | ... and to show that it does something slightly different to `_get_e2e_device_keys_txn`. `include_all_devices` and `include_deleted_devices` were never used (and `include_deleted_devices` was broken, since that would cause `None`s in the result which were not handled in the loop below. Add some typing too.
* | Merge commit 'e00816ad9' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-207-67/+140
|\| | | | | | | | | * commit 'e00816ad9': Do not yield on awaitables in tests. (#8193)
| * Do not yield on awaitables in tests. (#8193)Patrick Cloke2020-08-277-68/+143
| |
* | Merge commit 'b49a5b930' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-201-3/+2
|\| | | | | | | | | * commit 'b49a5b930': Convert stats and related calls to async/await (#8192)
| * Convert stats and related calls to async/await (#8192)Patrick Cloke2020-08-271-3/+2
| |
* | Merge commit 'b71d4a094' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-201-2/+4
|\| | | | | | | | | * commit 'b71d4a094': Convert simple_delete to async/await. (#8191)
| * Convert simple_delete to async/await. (#8191)Patrick Cloke2020-08-271-2/+4
| |
* | Merge commit '9b7ac03af' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-201-3/+49
|\| | | | | | | | | | | | | * commit '9b7ac03af': Convert calls of async database methods to async (#8166) simple_search_list_txn should return None, not 0. (#8187) Fix missing _add_persisted_position (#8179)
| * Fix missing _add_persisted_position (#8179)Erik Johnston2020-08-271-3/+49
| | | | | | This was forgotten in #8164.
* | Merge commit '30426c706' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-202-5/+6
|\| | | | | | | | | * commit '30426c706': Convert additional database methods to async (select list, search, insert_many, delete_*) (#8168)
| * Convert additional database methods to async (select list, search, ↵Patrick Cloke2020-08-272-5/+6
| | | | | | | | insert_many, delete_*) (#8168)
* | Merge commit '4a739c73b' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-202-11/+21
|\| | | | | | | | | * commit '4a739c73b': Convert simple_update* and simple_select* to async (#8173)
| * Convert simple_update* and simple_select* to async (#8173)Patrick Cloke2020-08-273-12/+24
| |
* | Merge commit 'a466b6797' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-207-26/+53
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'a466b6797': Reduce run-times of tests by advancing the reactor less (#7757) Update debian systemd service to use Type=notify (#8169) Remove remaining is_guest argument uses from get_room_data calls (#8181) Do not propagate typing notifications from shadow-banned users. (#8176) Remove unused parameter from, and add safeguard in, get_room_data (#8174) Add required Debian dependencies to allow docker builds on the arm platform (#8144) Allow running mypy directly. (#8175) Update the test federation client to handle streaming responses (#8130) Do not propagate profile changes of shadow-banned users into rooms. (#8157) Make SlavedIdTracker.advance have same interface as MultiWriterIDGenerator (#8171) Convert simple_select_one and simple_select_one_onecol to async (#8162)
| * Reduce run-times of tests by advancing the reactor less (#7757)Andrew Morgan2020-08-272-4/+4
| |
| * Convert simple_select_one and simple_select_one_onecol to async (#8162)Patrick Cloke2020-08-265-25/+56
| |
* | Merge commit '56efa9ec7' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-202-0/+67
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit '56efa9ec7': (22 commits) Fix rate limiting unit tests. (#8167) Add functions to `MultiWriterIdGen` used by events stream (#8164) Do not allow send_nonmember_event to be called with shadow-banned users. (#8158) Changelog fixes Make StreamIdGen `get_next` and `get_next_mult` async (#8161) Wording fixes to 'name' user admin api filter (#8163) Fix missing double-backtick in RST document Search in columns 'name' and 'displayname' in the admin users endpoint (#7377) Add type hints for state. (#8140) Stop shadow-banned users from sending non-member events. (#8142) Allow capping a room's retention policy (#8104) Add healthcheck for default localhost 8008 port on /health endpoint. (#8147) Fix flaky shadow-ban tests. (#8152) Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on (#7991) Do not apply ratelimiting on joins to appservices (#8139) Micro-optimisations to get_auth_chain_ids (#8132) Allow denying or shadow banning registrations via the spam checker (#8034) Stop shadow-banned users from sending invites. (#8095) Be more tolerant of membership events in unknown rooms (#8110) Improve the error code when trying to register using a name reserved for guests. (#8135) ...
| * Add functions to `MultiWriterIdGen` used by events stream (#8164)Erik Johnston2020-08-251-0/+36
| |
| * Don't fail /submit_token requests on incorrect session ID if ↵Brendan Abolivier2020-08-241-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | request_token_inhibit_3pid_errors is turned on (#7991) * Don't raise session_id errors on submit_token if request_token_inhibit_3pid_errors is set * Changelog * Also wait some time before responding to /requestToken * Incorporate review * Update synapse/storage/databases/main/registration.py Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> * Incorporate review Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Merge commit 'c9c544cda' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-194-13/+17
|\| | | | | | | | | | | | | | | | | | | | | | | | | * commit 'c9c544cda': Remove `ChainedIdGenerator`. (#8123) Switch the JSON byte producer from a pull to a push producer. (#8116) Updated docs: Added note about missing 308 redirect support. (#8120) Be stricter about JSON that is accepted by Synapse (#8106) Convert runWithConnection to async. (#8121) Remove the unused inlineCallbacks code-paths in the caching code (#8119) Separate `get_current_token` into two. (#8113) Convert events worker database to async/await. (#8071) Add a link to the matrix-synapse-rest-password-provider. (#8111)
| * Switch the JSON byte producer from a pull to a push producer. (#8116)Patrick Cloke2020-08-191-1/+2
| |
| * Separate `get_current_token` into two. (#8113)Erik Johnston2020-08-191-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | The function is used for two purposes: 1) for subscribers of streams to get a token they can use to get further updates with, and 2) for replication to track position of the writers of the stream. For streams with a single writer the two scenarios produce the same result, however the situation becomes complicated for streams with multiple writers. The current `MultiWriterIdGenerator` does not correctly handle the first case (which is not an issue as its only used for the `caches` stream which nothing subscribes to outside of replication).
| * Convert events worker database to async/await. (#8071)Patrick Cloke2020-08-181-1/+2
| |
* | Merge commit '050e20e7c' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-195-28/+50
|\| | | | | | | | | * commit '050e20e7c': Convert some of the general database methods to async (#8100)
| * Convert some of the general database methods to async (#8100)Patrick Cloke2020-08-175-27/+41
| |
* | Merge commit 'e04e465b4' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-194-37/+20
|\| | | | | | | | | | | | | | | | | | | * commit 'e04e465b4': Use the default templates when a custom template file cannot be found (#8037) Changelog changes Convert stream database to async/await. (#8074) Add a shadow-banned flag to users. (#8092) Convert pusher databases to async/await. (#8075) Convert receipts and events databases to async/await. (#8076)
| * Convert stream database to async/await. (#8074)Patrick Cloke2020-08-171-33/+16
| |
| * Add a shadow-banned flag to users. (#8092)Patrick Cloke2020-08-143-4/+4
| |
* | Merge commit '5ecc8b582' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-192-22/+38
|\| | | | | | | | | * commit '5ecc8b582': Convert devices database to async/await. (#8069)
| * Convert devices database to async/await. (#8069)Patrick Cloke2020-08-122-22/+38
| |
* | Merge commit 'a3a59bab7' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-191-12/+12
|\| | | | | | | | | * commit 'a3a59bab7': Convert appservice, group server, profile and more databases to async (#8066)
| * Convert appservice, group server, profile and more databases to async (#8066)Patrick Cloke2020-08-121-12/+12
| |
* | Merge commit 'a0acdfa9e' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-192-8/+18
|\| | | | | | | | | * commit 'a0acdfa9e': Converts event_federation and registration databases to async/await (#8061)
| * Converts event_federation and registration databases to async/await (#8061)Patrick Cloke2020-08-112-8/+18
| |
* | Merge commit 'db131b6b2' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-191-0/+4
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'db131b6b2': Change the default log config to reduce disk I/O and storage (#8040) Implement login blocking based on SAML attributes (#8052) Add an assertion on prev_events in create_new_client_event (#8041) Typo Lint why mypy why Lint Incorporate review Incorporate review Fix PUT /pushrules to use the right rule IDs Back out the database hack and replace it with a temporary config setting Fix cache name Fix cache invalidation calls Lint Changelog Implement new experimental push rules with a database hack to enable them
| * Add an assertion on prev_events in create_new_client_event (#8041)Richard van der Hoff2020-08-101-0/+4
| | | | | | | | | | | | I think this would have caught all the cases in https://github.com/matrix-org/synapse/issues/7642 - and I think a 500 makes more sense here than a 403
* | Merge commit '7f837959e' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-194-19/+46
|\| | | | | | | | | | | * commit '7f837959e': Convert directory, e2e_room_keys, end_to_end_keys, monthly_active_users database to async (#8042) Convert additional database stores to async/await (#8045)
| * Convert directory, e2e_room_keys, end_to_end_keys, monthly_active_users ↵Patrick Cloke2020-08-073-17/+44
| | | | | | | | database to async (#8042)
| * Convert additional database stores to async/await (#8045)Patrick Cloke2020-08-071-2/+2
| |
* | Merge commit 'a7bdf98d0' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-1612-75/+79
|\| | | | | | | | | * commit 'a7bdf98d0': Rename database classes to make some sense (#8033)
| * Rename database classes to make some sense (#8033)Erik Johnston2020-08-0512-75/+79
| |
* | Merge commit 'b3a97d6da' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-163-19/+29
|\| | | | | | | | | * commit 'b3a97d6da': Convert some of the data store to async. (#7976)
| * Convert some of the data store to async. (#7976)Patrick Cloke2020-07-303-19/+29
| |
* | Merge commit '3950ae51e' into anoa/dinsic_release_1_21_xAndrew Morgan2020-10-165-32/+56
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit '3950ae51e': Ensure that remove_pusher is always async (#7981) Ensure the msg property of HttpResponseException is a string. (#7979) Remove from the event_relations table when purging historical events. (#7978) Add additional logging for SAML sessions. (#7971) Add MSC reference to changelog for #7736 Re-implement unread counts (#7736) Various improvements to the docs (#7899) Convert storage layer to async/await. (#7963) Add an option to disable purge in delete room admin API (#7964) Move some log lines from default logger to sql/transaction loggers (#7952) Use the JSON module from the std library instead of simplejson. (#7936) Fix exit code for `check_line_terminators.sh` (#7970) Option to allow server admins to join complex rooms (#7902) Fix typo in metrics docs (#7966) Add script for finding files with unix line terminators (#7965) Convert the remaining media repo code to async / await. (#7947) Convert a synapse.events to async/await. (#7949) Convert groups and visibility code to async / await. (#7951) Convert push to async/await. (#7948)
| * Convert storage layer to async/await. (#7963)Patrick Cloke2020-07-283-29/+49
| |
| * Convert a synapse.events to async/await. (#7949)Patrick Cloke2020-07-271-1/+3
| |
| * Convert push to async/await. (#7948)Patrick Cloke2020-07-271-2/+4
| |
* | Merge commit 'f88c48f3b' into anoa/dinsic_release_1_18_xAndrew Morgan2020-08-031-2/+6
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'f88c48f3b': 1.18.0rc1 Fix error reporting when using `opentracing.trace` (#7961) Fix typing replication not being handled on master (#7959) Remove hacky error handling for inlineDeferreds. (#7950) Convert tests/rest/admin/test_room.py to unix file endings (#7953) Support oEmbed for media previews. (#7920) Convert state resolution to async/await (#7942) Fix up types and comments that refer to Deferreds. (#7945) Do not convert async functions to Deferreds in the interactive_auth_handler (#7944) Convert more of the media code to async/await (#7873) Return an empty body for OPTIONS requests. (#7886) Downgrade warning on client disconnect to INFO (#7928) Convert presence handler helpers to async/await. (#7939) Update the auth providers to be async. (#7935) Put a cache on `/state_ids` (#7931)
| * Convert state resolution to async/await (#7942)Patrick Cloke2020-07-241-2/+6
| |
* | Merge commit 'cc9bb3dc3' into anoa/dinsic_release_1_18_xAndrew Morgan2020-08-032-26/+34
|\| | | | | | | | | * commit 'cc9bb3dc3': Convert the message handler to async/await. (#7884)
| * Convert the message handler to async/await. (#7884)Patrick Cloke2020-07-222-26/+34
| |
* | Merge commit '491f0dab1' into anoa/dinsic_release_1_18_xAndrew Morgan2020-08-031-0/+8
|\| | | | | | | | | * commit '491f0dab1': Add delete room admin endpoint (#7613)
| * Add delete room admin endpoint (#7613)Dirk Klimpel2020-07-141-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Delete Room admin API allows server admins to remove rooms from server and block these rooms. `DELETE /_synapse/admin/v1/rooms/<room_id>` It is a combination and improvement of "[Shutdown room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/shutdown_room.md)" and "[Purge room](https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/purge_room.md)" API. Fixes: #6425 It also fixes a bug in [synapse/storage/data_stores/main/room.py](synapse/storage/data_stores/main/room.py) in ` get_room_with_stats`. It should return `None` if the room is unknown. But it returns an `IndexError`. https://github.com/matrix-org/synapse/blob/901b1fa561e3cc661d78aa96d59802cf2078cb0d/synapse/storage/data_stores/main/room.py#L99-L105 Related to: - #5575 - https://github.com/Awesome-Technologies/synapse-admin/issues/17 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Merge commit '43726783e' into dinsicAndrew Morgan2020-08-031-5/+2
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit '43726783e': (22 commits) 1.17.0rc1 Fix some spelling mistakes / typos. (#7811) `update_membership` declaration: now always returns an event id. (#7809) Improve stacktraces from exceptions in background processes (#7808) Fix `can only concatenate list (not "tuple") to list` exception (#7810) Pass original request headers from workers to the main process. (#7797) Generate real events when we reject invites (#7804) Add `HomeServer.signing_key` property (#7805) Revert "Update the installation docs on apt-transport-https (#7801)" Do not use simplejson in Synapse. (#7800) Stop passing bytes when dumping JSON (#7799) Update the installation docs on apt-transport-https (#7801) shuffle changelog slightly Change Caddy links (old is deprecated) (#7789) Stop populating unused table `local_invites`. (#7793) Refactor getting replication updates from database v2. (#7740) Add libwebp dependency to Dockerfile (#7791) Add documentation for JWT login type and improve sample config. (#7776) Convert the appservice handler to async/await. (#7775) Don't ignore `set_tweak` actions with no explicit `value`. (#7766) ...
| * Add `HomeServer.signing_key` property (#7805)Richard van der Hoff2020-07-081-5/+2
| | | | | | ... instead of duplicating `config.signing_key[0]` everywhere
* | Merge commit 'dc80a0762' into dinsicAndrew Morgan2020-08-031-29/+16
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'dc80a0762': 1.16.0rc1 Back out MSC2625 implementation (#7761) Additional configuration options for auto-join rooms (#7763) Add some metrics for inbound and outbound federation processing times (#7755) Explain the purpose of the "tests" conditional dependency requirement (#7751) Add another yield point to state res v2 (#7746) Move flake8 to end. Don't exit script on failure (#7738) Make tox actions work on Debian 10 (#7703) Yield during large v2 state res. (#7735) add org.matrix.login.jwt so that m.login.jwt can be deprecated (#7675) Set Content-Length for Metrics requests (#7730) Sync ignored table names in synapse_port_db to current database schema (#7717) Allow local media to be marked as safe from being quarantined. (#7718) Convert directory handler to async/await (#7727) Speed up state res v2 across large state differences. (#7725)
| * Back out MSC2625 implementation (#7761)Brendan Abolivier2020-07-011-29/+16
| |
* | Merge commit '46613aaf7' into dinsicAndrew Morgan2020-08-031-16/+29
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit '46613aaf7': (27 commits) Incorporate review Lint Incorporate review bits Pre-populate the unread_count column Don't update the schema version Use attr instead of a dict Lint Test that a mark_unread action updates the right counter when using a slave store Remove debug logging Test that a mark_unread action updates the right counter Fix summary rotation Log for invalid values of notif Fix SQL Fix schema update Lint Save the count of unread messages to event_push_summary Actually act on mark_unread Appease mypy Lint Use temporary prefixes as per the MSC ...
| * Remove debug loggingBrendan Abolivier2020-06-121-2/+0
| |
| * Test that a mark_unread action updates the right counterBrendan Abolivier2020-06-121-5/+16
| |
| * Fix testsBrendan Abolivier2020-06-101-14/+18
| |
* | Merge branch 'release-v1.15.0' of github.com:matrix-org/synapse into ↵Andrew Morgan2020-06-241-118/+149
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dinsic-release-v1.15.x * 'release-v1.15.0' of github.com:matrix-org/synapse: (55 commits) 1.15.0 Fix some attributions Update CHANGES.md 1.15.0rc1 Revert "1.15.0rc1" 1.15.0rc1 Fix bug in account data replication stream. (#7656) Convert the registration handler to async/await. (#7649) Accept device information at the login fallback endpoint. (#7629) Convert user directory handler and related classes to async/await. (#7640) Add an option to disable autojoin for guest accounts (#6637) Clarifications to the admin api documentation (#7647) Update to the stable SSO prefix for UI Auth. (#7630) Fix type information on `assert_*_is_admin` methods (#7645) Remove some unused constants. (#7644) Typo fixes. Allow new users to be registered via the admin API even if the monthly active user limit has been reached (#7263) Add device management to admin API (#7481) Attempt to fix PhoneHomeStatsTestCase.test_performance_100 being flaky. (#7634) Support CS API v0.6.0 (#6585) ...
| * Replace instances of reactor pumping with get_success. (#7619)Andrew Morgan2020-06-031-118/+149
| | | | | | | | | | Calls `self.get_success` on all deferred methods instead of abusing `self.pump()`. This has the benefit of working with coroutines, as well as checking that method execution completed successfully. There are also a few small cleanups that I made in the process.
* | Merge branch 'release-v1.14.0' of github.com:matrix-org/synapse into ↵Andrew Morgan2020-06-1011-73/+279
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dinsic-release-v1.14.x * 'release-v1.14.0' of github.com:matrix-org/synapse: (108 commits) Fix typo in PR link Update debian changelog 1.14.0 Improve changelog wording 1.14.0rc2 Fix sample config docs error (#7581) Fix up comments Fix specifying cache factors via env vars with * in name. (#7580) Don't apply cache factor to event cache. (#7578) Ensure ReplicationStreamer is always started when replication enabled. (#7579) Remove the changes to the debian changelog Not full release yet, this is rc1 Merge event persistence move changelog entries More changelog fix Changelog fixes 1.14.0 Replace device_27_unique_idx bg update with a fg one (#7562) Fix incorrect exception handling in KeyUploadServlet.on_POST (#7563) Fix recording of federation stream token (#7564) Simplify reap_monthly_active_users (#7558) ...
| * Optimise some references to hs.config (#7546)Richard van der Hoff2020-05-222-58/+65
| | | | | | These are surprisingly expensive, and we only really need to do them at startup.
| * Add ability to wait for replication streams (#7542)Erik Johnston2020-05-222-3/+3
| | | | | | | | | | | | | | The idea here is that if an instance persists an event via the replication HTTP API it can return before we receive that event over replication, which can lead to races where code assumes that persisting an event immediately updates various caches (e.g. current state of the room). Most of Synapse doesn't hit such races, so we don't do the waiting automagically, instead we do so where necessary to avoid unnecessary delays. We may decide to change our minds here if it turns out there are a lot of subtle races going on. People probably want to look at this commit by commit.
| * Merge tag 'v1.13.0rc2' into developRichard van der Hoff2020-05-141-1/+49
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.13.0rc2 (2020-05-14) ============================== Bugfixes -------- - Fix a long-standing bug which could cause messages not to be sent over federation, when state events with state keys matching user IDs (such as custom user statuses) were received. ([\#7376](https://github.com/matrix-org/synapse/issues/7376)) - Restore compatibility with non-compliant clients during the user interactive authentication process, fixing a problem introduced in v1.13.0rc1. ([\#7483](https://github.com/matrix-org/synapse/issues/7483)) Internal Changes ---------------- - Fix linting errors in new version of Flake8. ([\#7470](https://github.com/matrix-org/synapse/issues/7470))
| * | Shuffle persist event data store functions. (#7440)Erik Johnston2020-05-131-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The aim here is to get to a stage where we have a `PersistEventStore` that holds all the write methods used during event persistence, so that we can take that class out of the `DataStore` mixin and instansiate it separately. This will allow us to instansiate it on processes other than master, while also ensuring it is only available on processes that are configured to write to events stream. This is a bit of an architectural change, where we end up with multiple classes per data store (rather than one per data store we have now). We end up having: 1. Storage classes that provide high level APIs that can talk to multiple data stores. 2. Data store modules that consist of classes that must point at the same database instance. 3. Classes in a data store that can be instantiated on processes depending on config.
| * | Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-113-10/+11
| | | | | | | | | | | | variables (#6391)
| * | Add room details admin endpoint (#7317)Manuel Stahl2020-05-071-0/+11
| | |
| * | Add MultiWriterIdGenerator. (#7281)Erik Johnston2020-05-041-0/+184
| | | | | | | | | | | | | | | | | | This will be used to coordinate stream IDs across multiple writers. Functions as the equivalent of both `StreamIdGenerator` and `SlavedIdTracker`.
* | | Merge branch 'release-v1.13.0' of github.com:matrix-org/synapse into ↵Andrew Morgan2020-06-105-55/+162
|\ \ \ | | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dinsic-release-v1.14.x * 'release-v1.13.0' of github.com:matrix-org/synapse: (257 commits) Update changelog based on feedback. Move warnings in the changelog and re-iterate changes to branches. 1.13.0 update dh-virtualenv (#7526) 1.13.0rc3 Hash passwords earlier in the registration process (#7523) 1.13.0rc2 1.13.0rc2 Stop `get_joined_users` corruption from custom statuses (#7376) Do not validate that the client dict is stable during UI Auth. (#7483) Fix new flake8 errors (#7489) Don't UPGRADE database rows RST indenting Put rollback instructions in upgrade notes Fix changelog typo Oh yeah, RST Absolute URL it is then Fix upgrade notes link Provide summary of upgrade issues in changelog. Fix ) Move next version notes from changelog to upgrade notes ...
| * | Stop `get_joined_users` corruption from custom statuses (#7376)Richard van der Hoff2020-05-141-1/+49
| |/ | | | | | | | | | | | | | | Fix a bug where the `get_joined_users` cache could be corrupted by custom status events (or other state events with a state_key matching the user ID). The bug was introduced by #2229, but has largely gone unnoticed since then. Fixes #7099, #7373.
| * Return total number of users and profile attributes in admin users endpoint ↵Manuel Stahl2020-04-281-0/+46
| | | | | | | | | | (#6881) Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
| * bg update to clear out duplicate outbound_device_list_pokes (#7193)Richard van der Hoff2020-04-071-0/+52
| | | | | | | | We seem to have some duplicates, which could do with being cleared out.
| * Make do_next_background_update return a boolRichard van der Hoff2020-03-311-3/+3
| | | | | | | | returning a None or an int that we don't use is confusing.
| * Make `has_completed_background_updates` asyncRichard van der Hoff2020-03-311-1/+3
| | | | | | | | (Almost) everywhere that uses it is happy with an awaitable.
| * Remove unused `start_background_update`Richard van der Hoff2020-03-311-5/+9
| | | | | | | | This was only used in a unit test, so let's just inline it in the test.
| * Change device list streams to have one row per ID (#7010)Erik Johnston2020-03-191-45/+0
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add 'device_lists_outbound_pokes' as extra table. This makes sure we check all the relevant tables to get the current max stream ID. Currently not doing so isn't problematic as the max stream ID in `device_lists_outbound_pokes` is the same as in `device_lists_stream`, however that will change. * Change device lists stream to have one row per id. This will make it possible to process the streams more incrementally, avoiding having to process large chunks at once. * Change device list replication to match new semantics. Instead of sending down batches of user ID/host tuples, send down a row per entity (user ID or host). * Newsfile * Remove handling of multiple rows per ID * Fix worker handling * Comments from review
| | * Remove handling of multiple rows per IDErik Johnston2020-02-281-45/+0
| | |
* | | Improve get auth chain difference algorithm. (#7095)Andrew Morgan2020-03-241-17/+140
|\| | | | | | | | | | | | | | * commit '4a17a647a': Improve get auth chain difference algorithm. (#7095)
| * | Improve get auth chain difference algorithm. (#7095)Erik Johnston2020-03-181-17/+140
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It was originally implemented by pulling the full auth chain of all state sets out of the database and doing set comparison. However, that can take a lot work if the state and auth chains are large. Instead, lets try and fetch the auth chains at the same time and calculate the difference on the fly, allowing us to bail early if all the auth chains converge. Assuming that the auth chains do converge more often than not, this should improve performance. Hopefully.
* | | Break down monthly active users by appservice_id (#7030)Andrew Morgan2020-03-241-0/+42
|\| | | | | | | | | | | | | | * commit '1d66dce83': Break down monthly active users by appservice_id (#7030)
| * | Break down monthly active users by appservice_id (#7030)Neil Johnson2020-03-061-0/+42
| |/ | | | | | | | | * Break down monthly active users by appservice_id and emit via prometheus. Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Andrew Morgan2020-03-245-46/+42
|\| | | | | | | | | | | | | * commit '509e381af': Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) changelog Increase MAX_EVENTS_BEHIND for replication clients
| * Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-215-46/+42
| | | | | | | | Ensure good comprehension hygiene using flake8-comprehensions.
* | Merge pull request #6872 from matrix-org/rav/dictpropertyAndrew Morgan2020-03-241-1/+1
|\| | | | | | | | | | | | | * commit '3f1cd1479': changelog Rewrite _EventInternalMetadata to back it with a _dict Replace _event_dict_property with DictProperty
| * Replace _event_dict_property with DictPropertyRichard van der Hoff2020-02-141-1/+1
| | | | | | | | | | this amounts to the same thing, but replaces `_event_dict` with `_dict`, and removes some of the function layers generated by `property`.
* | Add typing to SyncHandler (#6821)Andrew Morgan2020-03-231-1/+4
|\| | | | | | | | | * commit 'b9391c957': Add typing to SyncHandler (#6821)
| * Add typing to SyncHandler (#6821)Erik Johnston2020-02-031-1/+4
| | | | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * Type defintions for use in refactoring for redaction changes (#6803)Richard van der Hoff2020-01-301-4/+11
| | | | | | | | | | | | | | | | | | | | * Bump signedjson to 1.1 ... so that we can use the type definitions * Fix breakage caused by upgrade to signedjson 1.1 Thanks, @illicitonion...
* | Add `rooms.room_version` column (#6729)Andrew Morgan2020-03-232-2/+10
|\| | | | | | | | | * commit '8df862e45': Add `rooms.room_version` column (#6729)
| * Add `rooms.room_version` column (#6729)Erik Johnston2020-01-272-2/+10
| | | | | | This is so that we don't have to rely on pulling it out from `current_state_events` table.
* | Make 'event.redacts' never raise. (#6771)Andrew Morgan2020-03-231-0/+35
|\| | | | | | | | | | | | | | | * commit 'fa4d609e2': Make 'event.redacts' never raise. (#6771) Fixup changelog Fixup changelog Fixup changelog
| * Make 'event.redacts' never raise. (#6771)Erik Johnston2020-01-231-0/+35
| | | | | | | | | | | | There are quite a few places that we assume that a redaction event has a corresponding `redacts` key, which is not always the case. So lets cheekily make it so that event.redacts just returns None instead.
* | Add `local_current_membership` table (#6655)Andrew Morgan2020-03-231-1/+1
|\| | | | | | | | | * commit '28c98e51f': Add `local_current_membership` table (#6655)
| * Add `local_current_membership` table (#6655)Erik Johnston2020-01-151-1/+1
| | | | | | | | | | | | | | Currently we rely on `current_state_events` to figure out what rooms a user was in and their last membership event in there. However, if the server leaves the room then the table may be cleaned up and that information is lost. So lets add a table that separately holds that information.
* | Allow admin users to create or modify users without a shared secret (#6495)Andrew Morgan2020-03-231-0/+2
|\| | | | | | | | | | | * commit 'd2906fe66': Allow admin users to create or modify users without a shared secret (#6495) Fixup changelog
| * Allow admin users to create or modify users without a shared secret (#6495)Manuel Stahl2020-01-091-0/+2
| | | | | | Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
* | port BackgroundUpdateTestCase to HomeserverTestCase (#6653)Andrew Morgan2020-03-201-35/+37
|\| | | | | | | | | | | | | | | | | * commit 'd20c34654': port BackgroundUpdateTestCase to HomeserverTestCase (#6653) changelog Fix exceptions in log when rejected event is replicated async/await for SyncReplicationHandler.process_and_notify Clarify documentation on get_event* methods
| * port BackgroundUpdateTestCase to HomeserverTestCase (#6653)Richard van der Hoff2020-01-071-35/+37
| |
* | Merge pull request #6629 from matrix-org/rav/kill_event_reference_hashesAndrew Morgan2020-03-201-11/+4
|\| | | | | | | | | | | | | | | | | | | | | | | * commit '1807db5e7': changelog Remove unused get_latest_event_ids_and_hashes_in_room Remove unused get_prev_events_and_hashes_for_room Remove unused hashes and depths from _update_membership params Remove unused hashes and depths from create_event params Remove unused hashes and depths from create_new_client_event params replace get_prev_events_and_hashes_for_room with get_prev_events_for_room in create_new_client_event rename get_prev_events_for_room to get_prev_events_and_hashes_for_room
| * Remove unused get_prev_events_and_hashes_for_roomRichard van der Hoff2020-01-061-13/+6
| |
| * rename get_prev_events_for_room to get_prev_events_and_hashes_for_roomRichard van der Hoff2020-01-061-2/+2
| | | | | | | | ... to make way for a new method which just returns the event ids
* | Split state groups into a separate data store (#6296)Andrew Morgan2020-03-201-1/+1
|\| | | | | | | | | * commit '75d8f26ac': Split state groups into a separate data store (#6296)
| * Split state groups into a separate data store (#6296)Erik Johnston2019-12-201-1/+1
| |
* | Add database config class (#6513)Andrew Morgan2020-03-203-18/+34
|\| | | | | | | | | | | * commit '2284eb3a5': Add database config class (#6513) too many parens
| * Add database config class (#6513)Erik Johnston2019-12-183-18/+34
| | | | | | | | | | This encapsulates config for a given database and is the way to get new connections.
* | Fix race which caused deleted devices to reappear (#6514)Andrew Morgan2020-03-191-20/+29
|\| | | | | | | | | * commit '40eda8493': Fix race which caused deleted devices to reappear (#6514)
| * Fix race which caused deleted devices to reappear (#6514)Richard van der Hoff2019-12-101-20/+29
| | | | | | | | Stop the `update_client_ips` background job from recreating deleted devices.
* | Back out perf regression from get_cross_signing_keys_from_cache. (#6494)Andrew Morgan2020-03-194-13/+14
|\| | | | | | | | | | | | | | | | | | | | | | | | | * commit 'adfdd82b2': Back out perf regression from get_cross_signing_keys_from_cache. (#6494) Fix comment Fix port db script Newsfile Fixup tests Move start up DB checks to main data store. Pass Database into the data store Move are_all_users_on_domain checks to main data store. Change DataStores to accept 'database' param.
| * Fixup testsErik Johnston2019-12-064-11/+11
| |
| * Change DataStores to accept 'database' param.Erik Johnston2019-12-061-2/+3
| |
* | Merge pull request #6469 from matrix-org/erikj/make_database_classAndrew Morgan2020-03-1910-66/+102
|\| | | | | | | | | | | | | | | | | | | * commit 'f3ea2f5a0': Remove unused var Fix DB scripts Newsfile Move background update handling out of store Comments Move DB pool and helper functions into dedicated Database class
| * Move background update handling out of storeErik Johnston2019-12-054-25/+59
| |
| * Move DB pool and helper functions into dedicated Database classErik Johnston2019-12-0510-41/+43
| |
* | Merge pull request #6464 from matrix-org/erikj/make_public_sql_baseAndrew Morgan2020-03-196-24/+24
|\| | | | | | | | | | | | | * commit 'ddbbfc951': Newsfile Remove underscore from SQLBaseStore functions Don't call SQLBaseStore methods from outside stores
| * Remove underscore from SQLBaseStore functionsErik Johnston2019-12-046-24/+24
| |
* | Implementation of MSC2314 (#6176)Andrew Morgan2020-03-181-25/+1
|\| | | | | | | | | * commit '0f87b912a': Implementation of MSC2314 (#6176)
| * Implementation of MSC2314 (#6176)Amber Brown2019-11-281-25/+1
| |
* | add etag and count to key backup endpoints (#5858)Andrew Morgan2020-03-181-4/+4
|\| | | | | | | | | * commit '0d27aba90': add etag and count to key backup endpoints (#5858)
| * add etag and count to key backup endpoints (#5858)Hubert Chathi2019-11-271-4/+4
| |
* | Merge pull request #6295 from matrix-org/erikj/split_purge_historyAndrew Morgan2020-03-181-7/+8
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'f713c01e2': Move type annotation into docstring Fix deleting state groups during room purge. Use correct type annotation Change to not require a state_groups.room_id index. Fix up comment Update log line to lie a little less Add state_groups.room_id index Docstrings Fix purge room API Newsfile Split purge API into events vs state
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-11-041-7/+9
| |\ | | | | | | | | | erikj/split_purge_history
| * \ Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-311-48/+102
| |\ \ | | | | | | | | | | | | erikj/split_purge_history
| * | | Split purge API into events vs stateErik Johnston2019-10-301-7/+8
| | | |
* | | | Merge pull request #5727 from matrix-org/uhoreg/e2e_cross-signing2-part3Andrew Morgan2020-03-161-7/+9
|\ \ \ \ | | |_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit '53d7680e3': Update synapse/storage/data_stores/main/devices.py rename get_devices_by_remote to get_device_updates_by_remote black apply changes as a result of PR review don't error if federation query doesn't have cross-signing keys move get_e2e_cross_signing_key to EndToEndKeyWorkerStore so it works with workers black vendor-prefix the EDU name until MSC1756 is merged into the spec fix unit test add news file update to work with newer code, and fix formatting add missing param make black happy don't crash if the user doesn't have cross-signing keys implement federation parts of cross-signing
| * | | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-311-48/+102
| |\ \ \ | | | |/ | | |/|
| * | | rename get_devices_by_remote to get_device_updates_by_remoteHubert Chathi2019-10-301-6/+6
| | | |
| * | | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-306-9/+88
| |\ \ \ | | | |/ | | |/|
| * | | Merge branch 'develop' into cross-signing_federationHubert Chathi2019-10-244-3/+5
| |\ \ \
| * | | | fix unit testHubert Chathi2019-10-221-1/+3
| | | | |
* | | | | Merge pull request #6294 from matrix-org/erikj/add_state_storageAndrew Morgan2020-03-161-48/+102
|\ \ \ \ \ | | |_|_|/ | |/| | |
| * | | | Port to use state storageErik Johnston2019-10-301-48/+102
| | |_|/ | |/| |
* | | | Merge pull request #6240 from matrix-org/erikj/split_out_persistence_storeAndrew Morgan2020-03-164-8/+12
|\| | |
| * | | Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2019-10-302-1/+76
| |\ \ \ | | | | | | | | | | | | | | | erikj/split_out_persistence_store
| * | | | Use new EventPersistenceStoreErik Johnston2019-10-234-8/+12
| | |_|/ | |/| |
* | | | Merge pull request #6291 from matrix-org/erikj/fix_cache_descriptorAndrew Morgan2020-03-161-1/+1
|\ \ \ \ | | |/ / | |/| |
| * | | Make ObservableDeferred.observe() always return deferred.Erik Johnston2019-10-301-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.
* | | | Merge pull request #6253 from matrix-org/uhoreg/e2e_backup_delete_keysAndrew Morgan2020-03-161-0/+75
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * commit 'da78f6177': remove unneeded imports switch to using HomeserverTestCase remove some unnecessary lines add changelog delete keys when deleting backups
| * | | remove unneeded importsHubert Chathi2019-10-251-3/+1
| | | |
| * | | switch to using HomeserverTestCaseHubert Chathi2019-10-251-19/+25
| | | |
| * | | remove some unnecessary linesHubert Chathi2019-10-241-5/+0
| | | |
| * | | delete keys when deleting backupsHubert Chathi2019-10-241-0/+76
| |/ /
* | | Merge pull request #6231 from matrix-org/erikj/refactor_storesAndrew Morgan2020-03-164-3/+5
|\| |
| * | Move storage classes into a main "data store".Erik Johnston2019-10-214-3/+5
| |/ | | | | | | | | This is in preparation for having multiple data stores that offer different functionality, e.g. splitting out state or event storage.
* | Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storageAndrew Morgan2020-02-261-6/+6
|\|
| * Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storageHubert Chathi2019-10-111-6/+6
| |\ | | | | | | make storage layer in charge of interpreting the device key data
| | * make storage layer in charge of interpreting the device key dataHubert Chathi2019-10-101-6/+6
| | |
* | | Fix MAU reaping where reserved users are specified. (#6168)Andrew Morgan2020-02-261-9/+49
|\| |
| * | Fix MAU reaping where reserved users are specified. (#6168)Neil Johnson2019-10-111-9/+49
| | |
* | | Merge pull request #6186 from matrix-org/erikj/disable_sql_bytesAndrew Morgan2020-02-261-1/+1
|\| |
| * | Fix postgres unit testsErik Johnston2019-10-101-1/+1
| |/
* | Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timingsAndrew Morgan2020-02-261-0/+11
|\|
| * Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timingsErik Johnston2019-10-021-0/+11
| |\ | | | | | | Fix errors storing large retry intervals.
| | * Fix errors storing large retry intervals.Erik Johnston2019-10-021-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).
* | | Merge pull request #6145 from matrix-org/erikj/fix_censored_redactionsAndrew Morgan2020-02-261-0/+36
|\| |
| * | Fix fetching censored redactions from DBErik Johnston2019-10-021-0/+36
| |/ | | | | | | | | | | Fetching a censored redactions caused an exception due to the code expecting redactions to have a `redact` key, which redacted redactions don't have.
* | Fix dummy event insertion consent bug (#6053)Andrew Morgan2020-02-252-9/+178
|\|
generated by cgit-magenta 1.5.1 (git 2.48.1) at 2025-08-10 13:17:38 +0000