summary refs log tree commit diff
path: root/synapse/util (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add logging to tasks managed by the task scheduler, showing CPU and database ↵reivilibre2024-05-221-2/+67
| | | | | | | | | | | | | | | | | | | usage. (#17219) The log format is the same as the request log format, except: - fields that are specific to HTTP requests have been removed - the task's params are included at the end of the log line. These log lines are emitted: - when the task function finishes — both completion and failure (and I suppose it is possible for a task to become schedulable again?) - every 5 minutes whilst it is running Closes #17217. --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
* Add optimisation to `StreamChangeCache` (#17130)Erik Johnston2024-05-061-1/+19
| | | | | | | When there have been lots of changes compared with the number of entities, we can do a fast(er) path. Locally I ran some benchmarking, and the comparison seems to give the best determination of which method we use.
* Fix bug where `StreamChangeCache` would not respect cache factors (#17152)Erik Johnston2024-05-031-1/+1
| | | Annoyingly mypy didn't pick up this typo.
* Ensure that pending to-device events are sent over federation at startup ↵Richard van der Hoff2024-03-221-3/+41
| | | | | | | | | | | | | | (#16925) Fixes https://github.com/element-hq/synapse/issues/16680, as well as a related bug, where servers which we had *never* successfully sent an event to would not be retried. In order to fix the case of pending to-device messages, we hook into the existing `wake_destinations_needing_catchup` process, by extending it to look for destinations that have pending to-device messages. The federation transmission loop then attempts to send the pending to-device messages as normal.
* Bump mypy from 1.5.1 to 1.8.0 (#16901)dependabot[bot]2024-03-133-12/+4
|
* Bump black from 23.10.1 to 24.2.0 (#16936)dependabot[bot]2024-03-135-37/+20
|
* Don't invalidate the entire event cache when we purge history (#16905)Erik Johnston2024-02-131-0/+59
| | | | | We do this by adding support to the LRU cache for "extra indices" based on the cached value. This allows us to efficiently map from room ID to the cached events and only invalidate those.
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-2337-0/+43
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Update license headersPatrick Cloke2023-11-2141-420/+655
|
* Speed up how quickly we launch new tasks (#16660)Erik Johnston2023-11-171-1/+1
| | | Now that we're reducing concurrency (#16656), this is more important.
* Reduce task concurrency (#16656)Erik Johnston2023-11-171-1/+1
|
* Speed up deleting device messages (#16643)Erik Johnston2023-11-161-1/+1
| | | Keeping track of a lower bound of stream ID where we've deleted everything below makes the queries much faster. Otherwise, every time we scan for rows to delete we'd re-scan across all the rows that have previously deleted (until the next table VACUUM).
* Speed up persisting large number of outliers (#16649)Erik Johnston2023-11-161-0/+51
| | | Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-151-1/+2
|
* Collect information for PushRuleEvaluator in parallel. (#16590)Patrick Cloke2023-11-061-0/+14
| | | | | | | | Fetch information needed for push rule evaluation in parallel. Ideally this would use query pipelining, but this is not available in psycopg2. Due to the database thread pool this may result in little to no parallelization.
* Bump twisted from 23.8.0 to 23.10.0 (#16588)dependabot[bot]2023-11-011-1/+1
|
* Fix type hint errors from Twisted trunk (#16526)Patrick Cloke2023-10-231-5/+11
|
* Fix a bug where servers could be marked as up when they were failing (#16506)Patrick Cloke2023-10-171-13/+17
| | | | After this change a server will only be reported as back online if they were previously having requests fail.
* mypy plugin to check `@cached` return types (#14911)David Robertson2023-10-021-9/+55
| | | | | | | | Co-authored-by: David Robertson <davidr@element.io> Co-authored-by: Patrick Cloke <patrickc@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org> Assert that the return type of callables wrapped in @cached and @cachedList are cachable (aka immutable).
* Only use literal strings for process names (#16315)Erik Johnston2023-09-151-3/+1
|
* Some minor performance fixes for task schedular (#16313)Erik Johnston2023-09-141-30/+70
|
* Improve type hints for attrs classes (#16276)David Robertson2023-09-084-33/+32
|
* Fix bug with new task scheduler using lots of CPU. (#16278)Erik Johnston2023-09-081-21/+22
| | | | | Using the new `TaskScheduler` meant that we'ed create lots of new metrics (due to adding task ID to the desc of background process), resulting in requests for metrics taking an increasing amount of CPU.
* Merge remote-tracking branch 'origin/release-v1.92' into developPatrick Cloke2023-09-061-22/+0
|\
| * Merge remote-tracking branch 'origin/release-v1.91' into release-v1.92Patrick Cloke2023-09-061-22/+0
| |\
| | * Revert MSC3861 introspection cache, admin impersonation and account lock ↵Quentin Gliech2023-09-061-22/+0
| | | | | | | | | | | | (#16258)
* | | Delete device messages asynchronously and in staged batches (#16240)Mathieu Velten2023-09-061-10/+7
| | |
* | | Bump twisted from 22.10.0 to 23.8.0 (#16235)dependabot[bot]2023-09-051-1/+1
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Bump twisted from 22.10.0 to 23.8.0 Bumps [twisted](https://github.com/twisted/twisted) from 22.10.0 to 23.8.0. - [Release notes](https://github.com/twisted/twisted/releases) - [Changelog](https://github.com/twisted/twisted/blob/trunk/NEWS.rst) - [Commits](https://github.com/twisted/twisted/compare/twisted-22.10.0...twisted-23.8.0) --- updated-dependencies: - dependency-name: twisted dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Fix types * Fix lint * Newsfile --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* | Don't wake up destination transaction queue if they're not due for retry. ↵Erik Johnston2023-09-041-0/+25
| | | | | | | | (#16223)
* | Don't reset retry timers on "valid" error codes (#16221)Erik Johnston2023-09-041-2/+16
| |
* | Describe which rate limiter was hit in logs (#16135)David Robertson2023-08-301-1/+2
| |
* | Bump mypy-zope & mypy. (#16188)Patrick Cloke2023-08-291-3/+3
| |
* | Task scheduler: add replication notify for new task to launch ASAP (#16184)Mathieu Velten2023-08-281-49/+43
| |
* | Fix typechecking with twisted trunk (#16121)David Robertson2023-08-241-1/+1
|/
* Task scheduler: mark task as active if we are scheduling ASAP (#16165)Mathieu Velten2023-08-231-1/+3
|
* Add an admin endpoint to allow authorizing server to signal token ↵Shay2023-08-221-0/+22
| | | | revocations (#16125)
* Implements a task scheduler for resumable potentially long running tasks ↵Mathieu Velten2023-08-211-0/+364
| | | | (#15891)
* Attempt to fix twisted trunk (#16115)David Robertson2023-08-151-1/+3
|
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-153-3/+4
|
* Allow config of the backoff algorithm for the federation client. (#15754)Mathieu Velten2023-08-031-13/+16
| | | | | | | | | | | Adds three new configuration variables: * destination_min_retry_interval is identical to before (10mn). * destination_retry_multiplier is now 2 instead of 5, the maximum value will be reached slower. * destination_max_retry_interval is one day instead of (essentially) infinity. Capping this will cause destinations to continue to be retried sometimes instead of being lost forever. The previous value was 2 ^ 62 milliseconds.
* Add metrics tracking for eviction to ResponseCache (#16028)Jason Little2023-08-011-2/+8
| | | | Track whether the ResponseCache is evicting due to invalidation or due to time.
* Remove support for Python 3.7 (#15851)Eric Eastwood2023-07-051-4/+1
| | | Fix https://github.com/matrix-org/synapse/issues/15836
* Prefill events after invalidate not before when persisting events (#15758)Jason Little2023-06-141-1/+7
| | | | | Fixes #15757
* Document `looping_call()` functionality that will wait for the given ↵Eric Eastwood2023-06-131-0/+5
| | | | | | | function to finish before scheduling another (#15772) Thanks to @erikjohnston for clarifying, https://github.com/matrix-org/synapse/pull/15743#discussion_r1226544457 We don't have to worry about calls stacking up if the given function takes longer than the scheduled time.
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-081-1/+1
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Add stricter mypy options (#15694)Patrick Cloke2023-05-313-7/+5
| | | | Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any, and disallow_incomplete_defs.
* Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-241-0/+27
| | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* Improve type hints for cached decorator. (#15658)Patrick Cloke2023-05-241-2/+4
| | | | | The cached decorators always return a Deferred, which was not properly propagated. It was close enough when wrapping coroutines, but failed if a bare function was wrapped.
* Fix `HomeServer`s leaking during `trial` test runs (#15630)Sean Quah2023-05-191-1/+5
| | | | | | | | | | | | | | | | | | | This change fixes two memory leaks during `trial` test runs. Garbage collection is disabled during each test case and a gen-0 GC is run at the end of each test. However, when the gen-0 GC is run, the `TestCase` object usually still holds references to the `HomeServer` used during the test. As a result, the `HomeServer` gets promoted to gen-1 and then never garbage collected. Fix this by periodically running full GCs. Additionally, fix `HomeServer`s leaking after tests that touch inbound federation due to `FederationRateLimiter`s adding themselves to a global set, by turning the set into a `WeakSet`. Resolves #15622. Signed-off-by: Sean Quah <seanq@matrix.org>
* Re-type config paths in `ConfigError`s to be `StrSequence`s (#15615)Sean Quah2023-05-181-15/+9
| | | | | | Part of #14809. Signed-off-by: Sean Quah <seanq@matrix.org>
* Clean up and clarify "Create or modify Account" Admin API documentation (#15544)Andrew Morgan2023-05-051-1/+5
|
* Use immutabledict instead of frozendict (#15113)David Robertson2023-03-222-15/+11
| | | | | | | | | | Additionally: * Consistently use `freeze()` in test --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: 6543 <6543@obermui.de>
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-223-3/+1
|
* Remove unnecessary reactor reference from `_PerHostRatelimiter` (#14842)Sean Quah2023-01-161-8/+2
| | | | | Fix up #14812 to avoid introducing a reference to the reactor. Signed-off-by: Sean Quah <seanq@matrix.org>
* Fix stack overflow in `_PerHostRatelimiter` due to synchronous requests (#14812)Sean Quah2023-01-131-9/+25
| | | | | | | | | | | | | | | | | | When there are many synchronous requests waiting on a `_PerHostRatelimiter`, each request will be started recursively just after the previous request has completed. Under the right conditions, this leads to stack exhaustion. A common way for requests to become synchronous is when the remote client disconnects early, because the homeserver is overloaded and slow to respond. Avoid stack exhaustion under these conditions by deferring subsequent requests until the next reactor tick. Fixes #14480. Signed-off-by: Sean Quah <seanq@matrix.org>
* Batch up replication requests to request the resyncing of remote users's ↵reivilibre2023-01-101-4/+51
| | | | devices. (#14716)
* Support RFC7636 PKCE in the OAuth 2.0 flow. (#14750)Patrick Cloke2023-01-041-0/+7
| | | | | | | PKCE can protect against certain attacks and is enabled by default. Support can be controlled manually by setting the pkce_method of each oidc_providers entry to 'auto' (default), 'always', or 'never'. This is required by Twitter OAuth 2.0 support.
* Add missing type hints to tests.config. (#14681)Patrick Cloke2022-12-161-7/+2
|
* Improve type annotations for the helper methods on a `CachedFunction`. (#14685)reivilibre2022-12-161-3/+3
|
* Make `handle_new_client_event` throws `PartialStateConflictError` (#14665)Mathieu Velten2022-12-151-4/+10
| | | | | | | Then adapts calling code to retry when needed so it doesn't 500 to clients. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Respond with proper error responses on unknown paths. (#14621)Patrick Cloke2022-12-081-2/+4
| | | | Returns a proper 404 with an errcode of M_RECOGNIZED for unknown endpoints per MSC3743.
* Check the stream position before checking if the cache is empty. (#14639)Patrick Cloke2022-12-081-4/+5
| | | | | | An empty cache does not mean the entity has no changed, if it is earlier than the earliest known stream position return that the entity *has* changed since the cache cannot accurately answer that query.
* Better return type for `get_all_entities_changed` (#14604)Erik Johnston2022-12-051-15/+37
| | | | Help callers from using the return value incorrectly by ensuring that callers explicitly check if there was a cache hit or not.
* Compare to the earliest known stream pos in the stream change cache. (#14435)Patrick Cloke2022-12-051-26/+116
| | | | | | The internal methods of the StreamChangeCache were inconsistently treating the earliest known stream position as valid. It is now treated as invalid, meaning the cache cannot determine if an entity at the earliest known stream position has changed or not.
* Batch fetch bundled annotations (#14491)Patrick Cloke2022-11-221-1/+1
| | | | | | | | Avoid an n+1 query problem and fetch the bundled aggregations for m.annotation relations in a single query instead of a query per event. This applies similar logic for as was previously done for edits in 8b309adb436c162510ed1402f33b8741d71fc058 (#11660) and threads in b65acead428653b988351ae8d7b22127a22039cd (#11752).
* Remove redundant types from comments. (#14412)Patrick Cloke2022-11-169-16/+18
| | | | | | | Remove type hints from comments which have been added as Python type hints. This helps avoid drift between comments and reality, as well as removing redundant information. Also adds some missing type hints which were simple to fill in.
* Remove duplicated code to evict entries. (#14410)Patrick Cloke2022-11-101-9/+2
| | | | | | | | This code was factored out to a method, but also left in-place. Calling this twice in a row makes no sense: the first call will reduce the size appropriately, but the loop will immediately exit since the cache size was already reduced.
* Refactor MSC3030 `/timestamp_to_event` to move away from our snowflake pull ↵Eric Eastwood2022-10-261-1/+1
| | | | | | | | | from `destination` pattern (#14096) 1. `federation_client.timestamp_to_event(...)` now handles all `destination` looping and uses our generic `_try_destination_list(...)` helper. 2. Consistently handling `NotRetryingDestination` and `FederationDeniedError` across `get_pdu` , backfill, and the generic `_try_destination_list` which is used for many places we use this pattern. 3. `get_pdu(...)` now returns `PulledPduInfo` so we know which `destination` we ended up pulling the PDU from
* Save login tokens in database (#13844)Quentin Gliech2022-10-261-86/+1
| | | | | | | | | | | * Save login tokens in database Signed-off-by: Quentin Gliech <quenting@element.io> * Add upgrade notes * Track login token reuse in a Prometheus metric Signed-off-by: Quentin Gliech <quenting@element.io>
* Remove unused `@lru_cache` decorator (#13595)Nick Mills-Barrett2022-10-251-104/+0
| | | | | | | * Remove unused `@lru_cache` decorator Spotted this working on something else. Co-authored-by: David Robertson <davidr@element.io>
* Bump flake8-bugbear from 21.3.2 to 22.9.23 (#14042)dependabot[bot]2022-10-192-3/+3
| | | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Making parse_server_name more consistent (#14007)Abdullah Osama2022-10-111-2/+2
| | | Fixes #12122
* Always close _all_ `ijson` coroutines, even if doing so raises Exceptions ↵David Robertson2022-10-061-1/+13
| | | | (#14065)
* Don't require `setuptools_rust` at runtime (#13952)David Robertson2022-09-291-1/+16
|
* Fix `have_seen_event` cache not being invalidated (#13863)Eric Eastwood2022-09-271-0/+6
| | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/13856 Fix https://github.com/matrix-org/synapse/issues/13865 > Discovered while trying to make Synapse fast enough for [this MSC2716 test for importing many batches](https://github.com/matrix-org/complement/pull/214#discussion_r741678240). As an example, disabling the `have_seen_event` cache saves 10 seconds for each `/messages` request in that MSC2716 Complement test because we're not making as many federation requests for `/state` (speeding up `have_seen_event` itself is related to https://github.com/matrix-org/synapse/issues/13625) > > But this will also make `/messages` faster in general so we can include it in the [faster `/messages` milestone](https://github.com/matrix-org/synapse/milestone/11). > > *-- https://github.com/matrix-org/synapse/issues/13856* ### The problem `_invalidate_caches_for_event` doesn't run in monolith mode which means we never even tried to clear the `have_seen_event` and other caches. And even in worker mode, it only runs on the workers, not the master (AFAICT). Additionally there was bug with the key being wrong so `_invalidate_caches_for_event` never invalidates the `have_seen_event` cache even when it does run. Because we were using the `@cachedList` wrong, it was putting items in the cache under keys like `((room_id, event_id),)` with a `set` in a `set` (ex. `(('!TnCIJPKzdQdUlIyXdQ:test', '$Iu0eqEBN7qcyF1S9B3oNB3I91v2o5YOgRNPwi_78s-k'),)`) and we we're trying to invalidate with just `(room_id, event_id)` which did nothing.
* Add cache invalidation across workers to module API (#13667)Mathieu Velten2022-09-211-7/+7
| | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* Fix a memory leak when running the unit tests. (#13798)reivilibre2022-09-142-6/+7
|
* Check if Rust lib needs rebuilding. (#13759)Erik Johnston2022-09-121-0/+84
| | | This protects against the common mistake of failing to remember to rebuild Rust code after making changes.
* Fix cache metrics not being updated when not using the legacy exposition ↵reivilibre2022-09-082-14/+80
| | | | module. (#13717)
* Update Grafana dashboard to not use legacy metric names. (#13714)reivilibre2022-09-061-2/+2
|
* Generalise the `@cancellable` annotation so it can be used on functions ↵reivilibre2022-08-311-0/+56
| | | | other than just servlet methods. (#13662)
* Merge branch 'release-v1.66' into developDavid Robertson2022-08-301-31/+124
|\
| * Fix rate limit metrics registering twice and misreporting (#13649)Eric Eastwood2022-08-301-31/+124
| | | | | | | | | | | | | | | | | | | | | | * Fix rate limit metrics registering twice and misreporting Fix https://github.com/matrix-org/synapse/issues/13641 * Fix lints * Add changelog * Document `metrics_name=None`.
* | Add experimental configuration option to allow disabling legacy Prometheus ↵reivilibre2022-08-241-8/+8
| | | | | | | | | | metric names. (#13540) Co-authored-by: David Robertson <davidr@element.io>
* | Speed up `@cachedList` (#13591)Erik Johnston2022-08-233-141/+297
| | | | | | | | | | | | | | | | | | This speeds things up by ~2x. The vast majority of the time is now spent in `LruCache` moving things around the linked lists. We do this via two things: 1. Don't create a deferred per-key during bulk set operations in `DeferredCache`. Instead, only create them if a subsequent caller asks for the key. 2. Add a bulk lookup API to `DeferredCache` rather than use a loop.
* | Cache user IDs instead of profile objects (#13573)Nick Mills-Barrett2022-08-231-7/+19
|/ | | The profile objects are never used and increase cache size significantly.
* Fix incorrect juggling of logging contexts in `_PerHostRatelimiter` (#13554)Sean Quah2022-08-181-10/+7
| | | | | | Signed-off-by: Sean Quah <seanq@matrix.org> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Track number of hosts affected by the rate limiter (#13541)Eric Eastwood2022-08-181-4/+39
| | | | | | | Track number of hosts affected by the rate limiter so we can differentiate one really noisy homeserver from a general ratelimit tuning problem across the federation. Follow-up to https://github.com/matrix-org/synapse/pull/13534 Part of https://github.com/matrix-org/synapse/issues/13356
* Add metrics to track how the rate limiter is affecting requests ↵Eric Eastwood2022-08-171-8/+29
| | | | | | | (sleep/reject) (#13534) Related to https://github.com/matrix-org/synapse/pull/13499 Part of https://github.com/matrix-org/synapse/issues/13356
* Add metrics to track rate limiter queue timing (#13544)Eric Eastwood2022-08-171-0/+30
|
* Instrument `FederationStateIdsServlet` - `/state_ids` (#13499)Eric Eastwood2022-08-151-0/+4
| | | Instrument FederationStateIdsServlet - `/state_ids` so it's easier to follow what's going on in Jaeger when viewing a trace.
* Optimise async get event lookups (#13435)Nick Mills-Barrett2022-08-041-0/+17
| | | | | | Still maintains local in memory lookup optimisation, but does any external lookup as part of the deferred that prevents duplicate lookups for the same event at once. This makes the assumption that fetching from an external cache is a non-zero load operation.
* Rename `RateLimitConfig` to `RatelimitSettings` (#13442)Dirk Klimpel2022-08-031-3/+3
|
* Make DictionaryCache have better expiry properties (#13292)Erik Johnston2022-07-213-33/+313
|
* Async get event cache prep (#13242)Nick Mills-Barrett2022-07-151-0/+38
| | | | | Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)
* Type `tests.utils` (#13028)David Robertson2022-07-052-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Cast to postgres types when handling postgres db * Remove unused method * Easy annotations * Annotate create_room * Use `ParamSpec` to annotate looping_call * Annotate `default_config` * Track `now` as a float `time_ms` returns an int like the proper Synapse `Clock` * Introduce a `Timer` dataclass * Introduce a Looper type * Suppress checking of a mock * tests.utils is typed * Changelog * Whoops, import ParamSpec from typing_extensions * ditch the psycopg2 casts
* Move the "email unsubscribe" resource, refactor the macaroon generator & ↵Quentin Gliech2022-06-141-0/+308
| | | | | | | | | | | | | | | | | | | | | simplify the access token verification logic. (#12986) This simplifies the access token verification logic by removing the `rights` parameter which was only ever used for the unsubscribe link in email notifications. The latter has been moved under the `/_synapse` namespace, since it is not a standard API. This also makes the email verification link more secure, by embedding the app_id and pushkey in the macaroon and verifying it. This prevents the user from tampering the query parameters of that unsubscribe link. Macaroon generation is refactored: - Centralised all macaroon generation and verification logic to the `MacaroonGenerator` - Moved to `synapse.utils` - Changed the constructor to require only a `Clock`, hostname, and a secret key (instead of a full `Homeserver`). - Added tests for all methods.
* Fix Synapse git info missing in version strings (#12973)David Robertson2022-06-071-0/+6
|
* Fix caching behavior for relations push rules. (#12859)Patrick Cloke2022-05-251-7/+8
| | | | | By always returning all requested values from the function wrapped by cachedList. Otherwise implicit None values get added into the cache, which are unexpected.
* Fix `RetryDestinationLimiter` re-starting finished log contexts (#12803)Sean Quah2022-05-191-2/+2
| | | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Discard null-containing strings before updating the user directory (#12762)David Robertson2022-05-181-1/+9
|
* Add config flags to allow for cache auto-tuning (#12701)Shay2022-05-131-10/+69
|
* Immediately retry any requests that have backed off when a server comes back ↵Erik Johnston2022-05-102-1/+80
| | | | | online. (#12500) Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
* Use `ParamSpec` in a few places (#12667)David Robertson2022-05-094-29/+72
|
* Prevent memory leak from reoccurring when presence is disabled. (#12656)Erik Johnston2022-05-061-13/+26
|
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-1/+1
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Consistently use collections.abc.Mapping to check frozendict. (#12564)Patrick Cloke2022-04-271-1/+2
|
* Update `delay_cancellation` to accept any awaitable (#12468)Sean Quah2022-04-221-10/+42
| | | | | | | | This will mainly be useful when dealing with module callbacks, which are all typed as returning `Awaitable`s instead of coroutines or `Deferred`s. Signed-off-by: Sean Quah <seanq@element.io>
* Fix edge case where a `Linearizer` could get stuck (#12358)Sean Quah2022-04-051-1/+5
| | | | | | | Just after a task acquires a contended `Linearizer` lock, it sleeps. If the task is cancelled during this sleep, we need to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Refactor and convert `Linearizer` to async (#12357)Sean Quah2022-04-051-77/+67
| | | | | | | | | | | Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Add more type hints to the main state store. (#12267)Patrick Cloke2022-03-311-2/+4
|
* Bump `black` and `click` versions (#12320)David Robertson2022-03-292-8/+11
|
* Try to detect borked package installations. (#12244)David Robertson2022-03-181-1/+23
| | | | | | | * Try to detect borked package installations. Fixes #12223. Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Add cancellation support to `@cached` and `@cachedList` decorators (#12183)Sean Quah2022-03-141-0/+11
| | | | | | | | These decorators mostly support cancellation already. Add cancellation tests and fix use of finished logging contexts by delaying cancellation, as suggested by @erikjohnston. Signed-off-by: Sean Quah <seanq@element.io>
* Add cancellation support to `ReadWriteLock` (#12120)Sean Quah2022-03-141-31/+40
| | | | | | Also convert `ReadWriteLock` to use async context managers. Signed-off-by: Sean Quah <seanq@element.io>
* Add `delay_cancellation` utility function (#12180)Sean Quah2022-03-141-6/+42
| | | | | | | | | `delay_cancellation` behaves like `stop_cancellation`, except it delays `CancelledError`s until the original `Deferred` resolves. This is handy for unifying cleanup paths and ensuring that uncancelled coroutines don't use finished logcontexts. Signed-off-by: Sean Quah <seanq@element.io>
* Implement a Jinja2 filter to extract localparts from email addresses (#12212)Brendan Abolivier2022-03-111-0/+5
|
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-2/+0
|
* Remove workaround introduced in Synapse v1.50.0rc1 for Mjolnir ↵reivilibre2022-03-101-7/+0
| | | | compatibility. Breaks compatibility with Mjolnir v1.3.1 and earlier. (#11700)
* Allow for ignoring some arguments when caching. (#12189)Patrick Cloke2022-03-091-16/+58
| | | | | * `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key. * Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity. * Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2022-03-081-1/+2
|\
| * Fix a bug introduced in 1.54.0rc1 which meant that Synapse would refuse to ↵reivilibre2022-03-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | start if pre-release versions of dependencies were installed. (#12177) * Add failing test to characterise the regression #12176 * Permit pre-release versions of specified packages * Newsfile (bugfix) Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Merge remote-tracking branch 'origin/release-v1.54' into developRichard van der Hoff2022-03-041-7/+54
|\|
| * Don't impose version checks on dev extras at runtime (#12129)David Robertson2022-03-031-7/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix incorrect argument in test case * Add copyright header * Docstring and __all__ * Exclude dev depenencies * Use changelog from #12088 * Include version in error messages This will hopefully distinguish between the version of the source code and the version of the distribution package that is installed. * Linter script is your friend
* | Add type hints for `ObservableDeferred` attributes (#12159)Sean Quah2022-03-041-3/+11
|/ | | Signed-off-by: Sean Quah <seanq@element.io>
* Use importlib.metadata to read requirements (#12088)David Robertson2022-03-011-0/+127
| | | | | | | | | | | | | | | | | | * Pull runtime dep checks into their own module * Reimplement `check_requirements` using `importlib` I've tried to make this clearer. We start by working out which of Synapse's requirements we need to be installed here and now. I was surprised that there wasn't an easier way to see which packages were installed by a given extra. I've pulled out the error messages into functions that deal with "is this for an extra or not". And I've rearranged the loop over two different sets of requirements into one loop with a "must be instaled" flag. I hope you agree that this is clearer. * Test cases
* Fix rare error in `ReadWriteLock` when writers complete immediately (#12105)Sean Quah2022-03-011-1/+4
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Add `stop_cancellation` utility function (#12106)Sean Quah2022-03-011-0/+19
|
* Fix 'Unhandled error in Deferred' (#12089)Richard van der Hoff2022-03-011-32/+32
| | | | | | | | | | | | | * Fix 'Unhandled error in Deferred' Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when a function wrapped with `@cachedList` failed * Minor optimisation to cachedListDescriptor we can avoid re-using `missing`, which saves looking up entries in `deferreds_map`, and means we don't need to copy it. * Improve type annotation on CachedListDescriptor
* Improve exception handling for concurrent execution (#12109)Richard van der Hoff2022-03-012-23/+35
| | | | | | | | | | | | | | | * fix incorrect unwrapFirstError import this was being imported from the wrong place * Refactor `concurrently_execute` to use `yieldable_gather_results` * Improve exception handling in `yieldable_gather_results` Try to avoid swallowing so many stack traces. * mark unwrapFirstError deprecated * changelog
* Upgrade mypy to version 0.931 (#12030)Sean Quah2022-02-182-5/+9
| | | Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
* Track cache invalidations (#12000)Erik Johnston2022-02-153-1/+9
| | | Currently we only track evictions due to size or time constraints.
* Use version string helper from matrix-common (#11979)David Robertson2022-02-141-85/+0
| | | | * Require latest matrix-common * Use the common function
* Add missing type hints to synapse.replication. (#11938)Patrick Cloke2022-02-081-3/+2
|
* Add a callback to allow modules to deny 3PID (#11854)Brendan Abolivier2022-02-081-1/+12
| | | | | Part of the Tchap Synapse mainlining. This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
* Describe `prune_unread_entries` in docstrings (#11876)David Robertson2022-02-023-2/+17
| | | Should have been caught in #10826.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-132-9/+8
|
* Work around Mjolnir compatibility issue by adding an import for ↵reivilibre2022-01-061-0/+7
| | | | `glob_to_regex` in `synapse.util`, where it moved from. (#11696)
* Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre2022-01-051-58/+1
| | | | | #11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
* Update to the current version of Black and run it on Synapse codebase (#11596)Shay2021-12-231-1/+0
| | | | | | | | | * update black version * run updated version of black on code * newsfragment * enumerate python versions
* Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff2021-12-202-36/+113
| | | This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
* Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-143-1/+58
|
* Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah2021-12-071-1/+58
| | | | | | `matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
* Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)Sean Quah2021-12-061-58/+1
|
* Clean-up get_version_string (#11468)Patrick Cloke2021-12-011-57/+25
|
* Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)Sean Quah2021-11-301-1/+4
| | | | | | | | | | | When all entries in an `LruCache` have a size of 0 according to the provided `size_callback`, and `drop_from_cache` is called on a cache node, the node would be unlinked from the LRU linked list but remain in the cache dictionary. An assertion would be later be tripped due to the inconsistency. Avoid unintentionally calling `__len__` and use a strict `is None` check instead when unwrapping the weak reference.
* Eliminate a few `Any`s in `LruCache` type hints (#11453)Sean Quah2021-11-303-19/+31
|
* Merge branch 'master' into developSean Quah2021-11-231-7/+14
|\
| * Prevent the media store from writing outside of the configured directorySean Quah2021-11-191-7/+14
| | | | | | | | | | Also tighten validation of server names by forbidding invalid characters in IPv6 addresses and empty domain labels.
* | Add type annotations to `synapse.metrics` (#10847)Sean Quah2021-11-172-8/+9
| |
* | Add most missing type hints to synapse.util (#11328)Patrick Cloke2021-11-168-81/+157
| |
* | Add more type hints to synapse.util. (#11321)Patrick Cloke2021-11-121-12/+20
| |
* | Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-103-9/+9
|/
* ObservableDeferred: run observers in order (#11229)Richard van der Hoff2021-11-021-16/+18
|
* Move DNS lookups into separate thread pool (#11177)Erik Johnston2021-10-261-0/+136
| | | | | | This is to stop large bursts of lookups starving out other users of the thread pools. Fixes #11049.
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-081-1/+4
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix long-standing bug where `ReadWriteLock` could drop logging contexts (#10993)Sean Quah2021-10-081-2/+4
| | | | | | | | | | | Use `PreserveLoggingContext()` to ensure that logging contexts are not lost when exiting a read/write lock. When exiting a read/write lock, callbacks on a `Deferred` are triggered as a signal to any waiting coroutines. Any waiting coroutine that becomes runnable is likely to follow the Synapse logging context rules and will restore its own logging context, then either run to completion or await another `Deferred`, resetting the logging context in the process.
* More types for synapse.util, part 1 (#10888)David Robertson2021-10-0610-73/+109
| | | | | | | | | | | | | | The following modules now pass `disallow_untyped_defs`: * synapse.util.caches.cached_call * synapse.util.caches.lrucache * synapse.util.caches.response_cache * synapse.util.caches.stream_change_cache * synapse.util.caches.ttlcache pass * synapse.util.daemonize * synapse.util.patch_inline_callbacks pass `no-untyped-defs` * synapse.util.versionstring Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
* Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-041-2/+2
|
* Update utility code to handle C implementations of frozendict (#10902)Hillery Shay2021-09-281-1/+7
| | | | | | | * update _handle_frozendict to work with c implementations of frozen dict * add changelog * add clarifying comment to _handle_frozendict
* Encode JSON responses on a thread in C, mk2 (#10905)Erik Johnston2021-09-281-2/+17
| | | | | | | | | | Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library. Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this: 1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or 2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types. I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
* Opt out of cache expiry for `get_users_who_share_room_with_user` (#10826)David Robertson2021-09-223-3/+20
| | | | * Allow LruCaches to opt out of time-based expiry * Don't expire `get_users_who_share_room` & friends
* Track why we're evicting from caches (#10829)David Robertson2021-09-223-15/+30
| | | So we can see distinguish between "evicting because the cache is too big" and "evicting because the cache entries haven't been recently used".
* Add type hints to state database module. (#10823)Patrick Cloke2021-09-151-2/+2
|
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-2/+2
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Add types to synapse.util. (#10601)reivilibre2021-09-1025-208/+281
|
* Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-061-3/+12
|
* Manhole: wrap coroutines in `defer.ensureDeferred` automatically (#10602)reivilibre2021-08-161-0/+14
|
* Convert Transaction and Edu object to attrs (#10542)Patrick Cloke2021-08-061-102/+0
| | | | | Instead of wrapping the JSON into an object, this creates concrete instances for Transaction and Edu. This allows for improved type hints and simplified code.
* Remove shebang line from module files (#10415)V024602021-07-291-1/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* Generics for `ObservableDeferred` (#10491)Richard van der Hoff2021-07-281-6/+8
| | | | | Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to follow suit.
* Fix up type hints for Twisted 21.7 (#10490)Richard van der Hoff2021-07-283-12/+21
| | | Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
* improve typing annotations in CachedCall (#10450)Richard van der Hoff2021-07-281-10/+17
| | | | | tighten up some of the typing in CachedCall, which is going to be needed when Twisted 21.7 brings better typing on Deferred.
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-193-8/+6
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-1516-59/+57
|
* Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-052-35/+352
|
* Standardise the module interface (#10062)Brendan Abolivier2021-06-181-15/+20
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Add debug logging when we enter/exit Measure block (#10183)Erik Johnston2021-06-161-0/+5
| | | It can be helpful to know when trying to track down slow requests.
* Extend `ResponseCache` to pass a context object into the callback (#10157)Richard van der Hoff2021-06-141-26/+73
| | | | | This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
* When joining a remote room limit the number of events we concurrently check ↵Erik Johnston2021-06-081-5/+16
| | | | | signatures/hashes for (#10117) If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.
* Fix up `BatchingQueue` (#10078)Erik Johnston2021-05-271-22/+48
| | | Fixes #10068
* Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-274-35/+36
| | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* Add missing type hints to synapse.util (#9982)Patrick Cloke2021-05-244-21/+19
|
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-244-47/+70
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-5/+3
|
* Add a batching queue implementation. (#10017)Erik Johnston2021-05-211-0/+153
|
* Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-141-6/+8
| | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* Simplify exception handling in is_ascii. (#9985)Dan Callahan2021-05-141-3/+1
| | | | | | | | | | | | | | | We can get away with just catching UnicodeError here. ⋮ +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError ⋮ https://docs.python.org/3/library/exceptions.html#exception-hierarchy Signed-off-by: Dan Callahan <danc@element.io>
* Use Python's secrets module instead of random (#9984)Dan Callahan2021-05-141-8/+11
| | | | | | | | | | | | | | | | Functionally identical, but more obviously cryptographically secure. ...Explicit is better than implicit? Avoids needing to know that SystemRandom() implies a CSPRNG, and complies with the big scary red box on the documentation for random: > Warning: > The pseudo-random generators of this module should not be used for > security purposes. For security or cryptographic uses, see the > secrets module. https://docs.python.org/3/library/random.html Signed-off-by: Dan Callahan <danc@element.io>
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-121-0/+20
|
* Merge branch 'master' into developBrendan Abolivier2021-05-111-14/+47
|\
| * Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff2021-05-111-14/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* | Optionally track memory usage of each LruCache (#9881)Erik Johnston2021-05-052-1/+78
| | | | | | | | | | This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
* | Reduce memory footprint of caches (#9886)Erik Johnston2021-04-281-18/+59
|/
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-1/+1
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Limit length of accepted email addresses (#9855)Erik Johnston2021-04-221-0/+30
|
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-222-4/+2
| | | This is no longer required, since we have dropped support for Python 3.5.
* Merge branch 'master' into developAndrew Morgan2021-04-201-5/+9
|\
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-201-5/+9
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Sanity check identity server passed to bind/unbind. (#9802)Denis Kasak2021-04-191-0/+32
| | | | | | | | Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1432-33/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-6/+8
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-061-32/+51
|
* Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-1/+3
|
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-292-44/+73
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-2/+2
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-242-2/+2
|
* Fixed code misc. quality issues (#9649)Ankit Dobhal2021-03-221-1/+1
| | | | | - Merge 'isinstance' calls. - Remove unnecessary dict call outside of comprehension. - Use 'sys.exit()' calls.
* Add logging to ObservableDeferred callbacks (#9523)Jonathan de Jong2021-03-091-8/+18
|
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-6/+4
|
* Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff2021-03-041-0/+89
| | | This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
* Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"Patrick Cloke2021-03-021-32/+2
| | | | | | | This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46. This is being backed out due to a regression (#9507) and additional review feedback being provided.
* Fix #8518 (sync requests being cached wrongly on timeout) (#9358)Jonathan de Jong2021-02-241-2/+32
| | | | | | | This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1611-48/+45
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-161-0/+129
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Fix some typos.Patrick Cloke2021-02-121-1/+1
|
* Merge tag 'v1.27.0rc2' into developPatrick Cloke2021-02-111-3/+12
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.27.0rc2 (2021-02-11) ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
| * Backout changes for automatically calculating the public baseurl. (#9313)Patrick Cloke2021-02-111-3/+12
| | | | | | | | This breaks some people's configurations (if their Client-Server API is not accessed via port 443).
* | Type hints and validation improvements. (#9321)Patrick Cloke2021-02-081-15/+18
|/ | | | | * Adds type hints to the groups servlet and stringutils code. * Assert the maximum length of some input values for spec compliance.
* Merge branch 'social_login' into developRichard van der Hoff2021-02-011-0/+106
|\
| * Replace username picker with a template (#9275)Richard van der Hoff2021-02-011-0/+106
| | | | | | | | | | There's some prelimiary work here to pull out the construction of a jinja environment to a separate function. I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
* | Pass a dict, instead of None, to modules if a None config is specified in ↵Andrew Morgan2021-01-271-1/+2
|/ | | | | | | | | the homeserver config (#9229) If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead. This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](https://github.com/matrix-org/synapse/blob/5310808d3bebd17275355ecd474bc013e8c7462d/docs/sample_config.yaml#L1816-L1845) states) then `None` is not passed to the module. An empty dict is passed instead. This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
* Fix chain cover update to handle events with duplicate auth events (#9210)Erik Johnston2021-01-221-1/+1
|
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-0/+92
|
* Remote dependency on distutils (#9125)Richard van der Hoff2021-01-151-0/+19
| | | | | | | | | `distutils` is pretty much deprecated these days, and replaced with `setuptools`. It's also annoying because it's you can't `pip install` it, and it's hard to figure out which debian package we should depend on to make sure it's there. Since we only use it for a tiny function anyway, let's just vendor said function into our codebase.
* Fix event chain bg update. (#9118)Erik Johnston2021-01-141-1/+1
| | | | We passed in a graph to `sorted_topologically` which didn't have an entry for each node (as we dropped nodes with no edges).
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-111-1/+52
|
* Fix "Starting metrics collection from sentinel context" errors (#9053)Erik Johnston2021-01-081-1/+2
|
* Empty iterables should count towards cache usage. (#9028)Patrick Cloke2021-01-061-1/+1
|
* Add type hints to the logging context code. (#8939)Patrick Cloke2021-01-051-1/+9
|
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-112-8/+7
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Better formatting for config errors from modules (#8874)Richard van der Hoff2020-12-081-6/+58
| | | | | | | | | | The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError, and it will be magically turned into a legible error message. There's a few components to it: * Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it into an absolute path. * Generally improving the way ConfigErrors get printed. * Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
* Implement and use an @lru_cache decorator (#8595)Richard van der Hoff2020-10-301-47/+188
| | | We don't always need the full power of a DeferredCache.
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-282-25/+21
|\ | | | | Fix serialisation errors when using third-party event rules.
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-282-25/+21
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
|/ | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Optimise CacheDescriptor (#8594) github/release-v1.21.3 release-v1.21.3Richard van der Hoff2020-10-211-5/+7
| | | don't bother constricting a CacheContext unless we need one.
* fix failure caseRichard van der Hoff2020-10-211-1/+4
|
* optimise DeferredCache.setRichard van der Hoff2020-10-211-3/+12
|
* Push some deferred wrangling down into DeferredCacheRichard van der Hoff2020-10-212-34/+55
|
* Fix 'LruCache' object has no attribute '_on_resize' (#8591)Richard van der Hoff2020-10-191-1/+9
| | | We need to make sure we are readu for the `set_cache_factor` callback.
* Add `DeferredCache.get_immediate` method (#8568)Richard van der Hoff2020-10-191-10/+25
| | | | | | | | | | | * Add `DeferredCache.get_immediate` method A bunch of things that are currently calling `DeferredCache.get` are only really interested in the result if it's completed. We can optimise and simplify this case. * Remove unused 'default' parameter to DeferredCache.get() * another get_immediate instance
* Replace DeferredCache with LruCache where possible (#8563)Richard van der Hoff2020-10-191-0/+3
| | | Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
* review commentsRichard van der Hoff2020-10-161-3/+8
|
* Apply suggestions from code reviewRichard van der Hoff2020-10-161-1/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* type annotations for LruCacheRichard van der Hoff2020-10-163-23/+77
|
* Make LruCache register its own metrics (#8561)Richard van der Hoff2020-10-164-54/+57
| | | | | rather than have everything that instantiates an LruCache manage metrics separately, have LruCache do it itself.
* Enable mypy for synapse.util.caches (#8547)Richard van der Hoff2020-10-151-1/+1
| | | This seemed to entail dragging in a type stub for SortedList.
* move DeferredCache into its own moduleRichard van der Hoff2020-10-142-282/+294
|
* Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-7/+12
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-02 09:32:13 +0000