summary refs log tree commit diff
path: root/synapse/util (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix `HomeServer`s leaking during `trial` test runs (#15630)Sean Quah2023-05-191-1/+5
| | | | | | | | | | | | | | | | | | | This change fixes two memory leaks during `trial` test runs. Garbage collection is disabled during each test case and a gen-0 GC is run at the end of each test. However, when the gen-0 GC is run, the `TestCase` object usually still holds references to the `HomeServer` used during the test. As a result, the `HomeServer` gets promoted to gen-1 and then never garbage collected. Fix this by periodically running full GCs. Additionally, fix `HomeServer`s leaking after tests that touch inbound federation due to `FederationRateLimiter`s adding themselves to a global set, by turning the set into a `WeakSet`. Resolves #15622. Signed-off-by: Sean Quah <seanq@matrix.org>
* Re-type config paths in `ConfigError`s to be `StrSequence`s (#15615)Sean Quah2023-05-181-15/+9
| | | | | | Part of #14809. Signed-off-by: Sean Quah <seanq@matrix.org>
* Clean up and clarify "Create or modify Account" Admin API documentation (#15544)Andrew Morgan2023-05-051-1/+5
|
* Use immutabledict instead of frozendict (#15113)David Robertson2023-03-222-15/+11
| | | | | | | | | | Additionally: * Consistently use `freeze()` in test --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: 6543 <6543@obermui.de>
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-223-3/+1
|
* Remove unnecessary reactor reference from `_PerHostRatelimiter` (#14842)Sean Quah2023-01-161-8/+2
| | | | | Fix up #14812 to avoid introducing a reference to the reactor. Signed-off-by: Sean Quah <seanq@matrix.org>
* Fix stack overflow in `_PerHostRatelimiter` due to synchronous requests (#14812)Sean Quah2023-01-131-9/+25
| | | | | | | | | | | | | | | | | | When there are many synchronous requests waiting on a `_PerHostRatelimiter`, each request will be started recursively just after the previous request has completed. Under the right conditions, this leads to stack exhaustion. A common way for requests to become synchronous is when the remote client disconnects early, because the homeserver is overloaded and slow to respond. Avoid stack exhaustion under these conditions by deferring subsequent requests until the next reactor tick. Fixes #14480. Signed-off-by: Sean Quah <seanq@matrix.org>
* Batch up replication requests to request the resyncing of remote users's ↵reivilibre2023-01-101-4/+51
| | | | devices. (#14716)
* Support RFC7636 PKCE in the OAuth 2.0 flow. (#14750)Patrick Cloke2023-01-041-0/+7
| | | | | | | PKCE can protect against certain attacks and is enabled by default. Support can be controlled manually by setting the pkce_method of each oidc_providers entry to 'auto' (default), 'always', or 'never'. This is required by Twitter OAuth 2.0 support.
* Add missing type hints to tests.config. (#14681)Patrick Cloke2022-12-161-7/+2
|
* Improve type annotations for the helper methods on a `CachedFunction`. (#14685)reivilibre2022-12-161-3/+3
|
* Make `handle_new_client_event` throws `PartialStateConflictError` (#14665)Mathieu Velten2022-12-151-4/+10
| | | | | | | Then adapts calling code to retry when needed so it doesn't 500 to clients. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Respond with proper error responses on unknown paths. (#14621)Patrick Cloke2022-12-081-2/+4
| | | | Returns a proper 404 with an errcode of M_RECOGNIZED for unknown endpoints per MSC3743.
* Check the stream position before checking if the cache is empty. (#14639)Patrick Cloke2022-12-081-4/+5
| | | | | | An empty cache does not mean the entity has no changed, if it is earlier than the earliest known stream position return that the entity *has* changed since the cache cannot accurately answer that query.
* Better return type for `get_all_entities_changed` (#14604)Erik Johnston2022-12-051-15/+37
| | | | Help callers from using the return value incorrectly by ensuring that callers explicitly check if there was a cache hit or not.
* Compare to the earliest known stream pos in the stream change cache. (#14435)Patrick Cloke2022-12-051-26/+116
| | | | | | The internal methods of the StreamChangeCache were inconsistently treating the earliest known stream position as valid. It is now treated as invalid, meaning the cache cannot determine if an entity at the earliest known stream position has changed or not.
* Batch fetch bundled annotations (#14491)Patrick Cloke2022-11-221-1/+1
| | | | | | | | Avoid an n+1 query problem and fetch the bundled aggregations for m.annotation relations in a single query instead of a query per event. This applies similar logic for as was previously done for edits in 8b309adb436c162510ed1402f33b8741d71fc058 (#11660) and threads in b65acead428653b988351ae8d7b22127a22039cd (#11752).
* Remove redundant types from comments. (#14412)Patrick Cloke2022-11-169-16/+18
| | | | | | | Remove type hints from comments which have been added as Python type hints. This helps avoid drift between comments and reality, as well as removing redundant information. Also adds some missing type hints which were simple to fill in.
* Remove duplicated code to evict entries. (#14410)Patrick Cloke2022-11-101-9/+2
| | | | | | | | This code was factored out to a method, but also left in-place. Calling this twice in a row makes no sense: the first call will reduce the size appropriately, but the loop will immediately exit since the cache size was already reduced.
* Refactor MSC3030 `/timestamp_to_event` to move away from our snowflake pull ↵Eric Eastwood2022-10-261-1/+1
| | | | | | | | | from `destination` pattern (#14096) 1. `federation_client.timestamp_to_event(...)` now handles all `destination` looping and uses our generic `_try_destination_list(...)` helper. 2. Consistently handling `NotRetryingDestination` and `FederationDeniedError` across `get_pdu` , backfill, and the generic `_try_destination_list` which is used for many places we use this pattern. 3. `get_pdu(...)` now returns `PulledPduInfo` so we know which `destination` we ended up pulling the PDU from
* Save login tokens in database (#13844)Quentin Gliech2022-10-261-86/+1
| | | | | | | | | | | * Save login tokens in database Signed-off-by: Quentin Gliech <quenting@element.io> * Add upgrade notes * Track login token reuse in a Prometheus metric Signed-off-by: Quentin Gliech <quenting@element.io>
* Remove unused `@lru_cache` decorator (#13595)Nick Mills-Barrett2022-10-251-104/+0
| | | | | | | * Remove unused `@lru_cache` decorator Spotted this working on something else. Co-authored-by: David Robertson <davidr@element.io>
* Bump flake8-bugbear from 21.3.2 to 22.9.23 (#14042)dependabot[bot]2022-10-192-3/+3
| | | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Making parse_server_name more consistent (#14007)Abdullah Osama2022-10-111-2/+2
| | | Fixes #12122
* Always close _all_ `ijson` coroutines, even if doing so raises Exceptions ↵David Robertson2022-10-061-1/+13
| | | | (#14065)
* Don't require `setuptools_rust` at runtime (#13952)David Robertson2022-09-291-1/+16
|
* Fix `have_seen_event` cache not being invalidated (#13863)Eric Eastwood2022-09-271-0/+6
| | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/13856 Fix https://github.com/matrix-org/synapse/issues/13865 > Discovered while trying to make Synapse fast enough for [this MSC2716 test for importing many batches](https://github.com/matrix-org/complement/pull/214#discussion_r741678240). As an example, disabling the `have_seen_event` cache saves 10 seconds for each `/messages` request in that MSC2716 Complement test because we're not making as many federation requests for `/state` (speeding up `have_seen_event` itself is related to https://github.com/matrix-org/synapse/issues/13625) > > But this will also make `/messages` faster in general so we can include it in the [faster `/messages` milestone](https://github.com/matrix-org/synapse/milestone/11). > > *-- https://github.com/matrix-org/synapse/issues/13856* ### The problem `_invalidate_caches_for_event` doesn't run in monolith mode which means we never even tried to clear the `have_seen_event` and other caches. And even in worker mode, it only runs on the workers, not the master (AFAICT). Additionally there was bug with the key being wrong so `_invalidate_caches_for_event` never invalidates the `have_seen_event` cache even when it does run. Because we were using the `@cachedList` wrong, it was putting items in the cache under keys like `((room_id, event_id),)` with a `set` in a `set` (ex. `(('!TnCIJPKzdQdUlIyXdQ:test', '$Iu0eqEBN7qcyF1S9B3oNB3I91v2o5YOgRNPwi_78s-k'),)`) and we we're trying to invalidate with just `(room_id, event_id)` which did nothing.
* Add cache invalidation across workers to module API (#13667)Mathieu Velten2022-09-211-7/+7
| | | Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
* Fix a memory leak when running the unit tests. (#13798)reivilibre2022-09-142-6/+7
|
* Check if Rust lib needs rebuilding. (#13759)Erik Johnston2022-09-121-0/+84
| | | This protects against the common mistake of failing to remember to rebuild Rust code after making changes.
* Fix cache metrics not being updated when not using the legacy exposition ↵reivilibre2022-09-082-14/+80
| | | | module. (#13717)
* Update Grafana dashboard to not use legacy metric names. (#13714)reivilibre2022-09-061-2/+2
|
* Generalise the `@cancellable` annotation so it can be used on functions ↵reivilibre2022-08-311-0/+56
| | | | other than just servlet methods. (#13662)
* Merge branch 'release-v1.66' into developDavid Robertson2022-08-301-31/+124
|\
| * Fix rate limit metrics registering twice and misreporting (#13649)Eric Eastwood2022-08-301-31/+124
| | | | | | | | | | | | | | | | | | | | | | * Fix rate limit metrics registering twice and misreporting Fix https://github.com/matrix-org/synapse/issues/13641 * Fix lints * Add changelog * Document `metrics_name=None`.
* | Add experimental configuration option to allow disabling legacy Prometheus ↵reivilibre2022-08-241-8/+8
| | | | | | | | | | metric names. (#13540) Co-authored-by: David Robertson <davidr@element.io>
* | Speed up `@cachedList` (#13591)Erik Johnston2022-08-233-141/+297
| | | | | | | | | | | | | | | | | | This speeds things up by ~2x. The vast majority of the time is now spent in `LruCache` moving things around the linked lists. We do this via two things: 1. Don't create a deferred per-key during bulk set operations in `DeferredCache`. Instead, only create them if a subsequent caller asks for the key. 2. Add a bulk lookup API to `DeferredCache` rather than use a loop.
* | Cache user IDs instead of profile objects (#13573)Nick Mills-Barrett2022-08-231-7/+19
|/ | | The profile objects are never used and increase cache size significantly.
* Fix incorrect juggling of logging contexts in `_PerHostRatelimiter` (#13554)Sean Quah2022-08-181-10/+7
| | | | | | Signed-off-by: Sean Quah <seanq@matrix.org> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Track number of hosts affected by the rate limiter (#13541)Eric Eastwood2022-08-181-4/+39
| | | | | | | Track number of hosts affected by the rate limiter so we can differentiate one really noisy homeserver from a general ratelimit tuning problem across the federation. Follow-up to https://github.com/matrix-org/synapse/pull/13534 Part of https://github.com/matrix-org/synapse/issues/13356
* Add metrics to track how the rate limiter is affecting requests ↵Eric Eastwood2022-08-171-8/+29
| | | | | | | (sleep/reject) (#13534) Related to https://github.com/matrix-org/synapse/pull/13499 Part of https://github.com/matrix-org/synapse/issues/13356
* Add metrics to track rate limiter queue timing (#13544)Eric Eastwood2022-08-171-0/+30
|
* Instrument `FederationStateIdsServlet` - `/state_ids` (#13499)Eric Eastwood2022-08-151-0/+4
| | | Instrument FederationStateIdsServlet - `/state_ids` so it's easier to follow what's going on in Jaeger when viewing a trace.
* Optimise async get event lookups (#13435)Nick Mills-Barrett2022-08-041-0/+17
| | | | | | Still maintains local in memory lookup optimisation, but does any external lookup as part of the deferred that prevents duplicate lookups for the same event at once. This makes the assumption that fetching from an external cache is a non-zero load operation.
* Rename `RateLimitConfig` to `RatelimitSettings` (#13442)Dirk Klimpel2022-08-031-3/+3
|
* Make DictionaryCache have better expiry properties (#13292)Erik Johnston2022-07-213-33/+313
|
* Async get event cache prep (#13242)Nick Mills-Barrett2022-07-151-0/+38
| | | | | Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)
* Type `tests.utils` (#13028)David Robertson2022-07-052-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Cast to postgres types when handling postgres db * Remove unused method * Easy annotations * Annotate create_room * Use `ParamSpec` to annotate looping_call * Annotate `default_config` * Track `now` as a float `time_ms` returns an int like the proper Synapse `Clock` * Introduce a `Timer` dataclass * Introduce a Looper type * Suppress checking of a mock * tests.utils is typed * Changelog * Whoops, import ParamSpec from typing_extensions * ditch the psycopg2 casts
* Move the "email unsubscribe" resource, refactor the macaroon generator & ↵Quentin Gliech2022-06-141-0/+308
| | | | | | | | | | | | | | | | | | | | | simplify the access token verification logic. (#12986) This simplifies the access token verification logic by removing the `rights` parameter which was only ever used for the unsubscribe link in email notifications. The latter has been moved under the `/_synapse` namespace, since it is not a standard API. This also makes the email verification link more secure, by embedding the app_id and pushkey in the macaroon and verifying it. This prevents the user from tampering the query parameters of that unsubscribe link. Macaroon generation is refactored: - Centralised all macaroon generation and verification logic to the `MacaroonGenerator` - Moved to `synapse.utils` - Changed the constructor to require only a `Clock`, hostname, and a secret key (instead of a full `Homeserver`). - Added tests for all methods.
* Fix Synapse git info missing in version strings (#12973)David Robertson2022-06-071-0/+6
|
* Fix caching behavior for relations push rules. (#12859)Patrick Cloke2022-05-251-7/+8
| | | | | By always returning all requested values from the function wrapped by cachedList. Otherwise implicit None values get added into the cache, which are unexpected.
* Fix `RetryDestinationLimiter` re-starting finished log contexts (#12803)Sean Quah2022-05-191-2/+2
| | | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Discard null-containing strings before updating the user directory (#12762)David Robertson2022-05-181-1/+9
|
* Add config flags to allow for cache auto-tuning (#12701)Shay2022-05-131-10/+69
|
* Immediately retry any requests that have backed off when a server comes back ↵Erik Johnston2022-05-102-1/+80
| | | | | online. (#12500) Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
* Use `ParamSpec` in a few places (#12667)David Robertson2022-05-094-29/+72
|
* Prevent memory leak from reoccurring when presence is disabled. (#12656)Erik Johnston2022-05-061-13/+26
|
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-1/+1
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Consistently use collections.abc.Mapping to check frozendict. (#12564)Patrick Cloke2022-04-271-1/+2
|
* Update `delay_cancellation` to accept any awaitable (#12468)Sean Quah2022-04-221-10/+42
| | | | | | | | This will mainly be useful when dealing with module callbacks, which are all typed as returning `Awaitable`s instead of coroutines or `Deferred`s. Signed-off-by: Sean Quah <seanq@element.io>
* Fix edge case where a `Linearizer` could get stuck (#12358)Sean Quah2022-04-051-1/+5
| | | | | | | Just after a task acquires a contended `Linearizer` lock, it sleeps. If the task is cancelled during this sleep, we need to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Refactor and convert `Linearizer` to async (#12357)Sean Quah2022-04-051-77/+67
| | | | | | | | | | | Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Add more type hints to the main state store. (#12267)Patrick Cloke2022-03-311-2/+4
|
* Bump `black` and `click` versions (#12320)David Robertson2022-03-292-8/+11
|
* Try to detect borked package installations. (#12244)David Robertson2022-03-181-1/+23
| | | | | | | * Try to detect borked package installations. Fixes #12223. Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Add cancellation support to `@cached` and `@cachedList` decorators (#12183)Sean Quah2022-03-141-0/+11
| | | | | | | | These decorators mostly support cancellation already. Add cancellation tests and fix use of finished logging contexts by delaying cancellation, as suggested by @erikjohnston. Signed-off-by: Sean Quah <seanq@element.io>
* Add cancellation support to `ReadWriteLock` (#12120)Sean Quah2022-03-141-31/+40
| | | | | | Also convert `ReadWriteLock` to use async context managers. Signed-off-by: Sean Quah <seanq@element.io>
* Add `delay_cancellation` utility function (#12180)Sean Quah2022-03-141-6/+42
| | | | | | | | | `delay_cancellation` behaves like `stop_cancellation`, except it delays `CancelledError`s until the original `Deferred` resolves. This is handy for unifying cleanup paths and ensuring that uncancelled coroutines don't use finished logcontexts. Signed-off-by: Sean Quah <seanq@element.io>
* Implement a Jinja2 filter to extract localparts from email addresses (#12212)Brendan Abolivier2022-03-111-0/+5
|
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-2/+0
|
* Remove workaround introduced in Synapse v1.50.0rc1 for Mjolnir ↵reivilibre2022-03-101-7/+0
| | | | compatibility. Breaks compatibility with Mjolnir v1.3.1 and earlier. (#11700)
* Allow for ignoring some arguments when caching. (#12189)Patrick Cloke2022-03-091-16/+58
| | | | | * `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key. * Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity. * Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2022-03-081-1/+2
|\
| * Fix a bug introduced in 1.54.0rc1 which meant that Synapse would refuse to ↵reivilibre2022-03-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | start if pre-release versions of dependencies were installed. (#12177) * Add failing test to characterise the regression #12176 * Permit pre-release versions of specified packages * Newsfile (bugfix) Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Merge remote-tracking branch 'origin/release-v1.54' into developRichard van der Hoff2022-03-041-7/+54
|\|
| * Don't impose version checks on dev extras at runtime (#12129)David Robertson2022-03-031-7/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix incorrect argument in test case * Add copyright header * Docstring and __all__ * Exclude dev depenencies * Use changelog from #12088 * Include version in error messages This will hopefully distinguish between the version of the source code and the version of the distribution package that is installed. * Linter script is your friend
* | Add type hints for `ObservableDeferred` attributes (#12159)Sean Quah2022-03-041-3/+11
|/ | | Signed-off-by: Sean Quah <seanq@element.io>
* Use importlib.metadata to read requirements (#12088)David Robertson2022-03-011-0/+127
| | | | | | | | | | | | | | | | | | * Pull runtime dep checks into their own module * Reimplement `check_requirements` using `importlib` I've tried to make this clearer. We start by working out which of Synapse's requirements we need to be installed here and now. I was surprised that there wasn't an easier way to see which packages were installed by a given extra. I've pulled out the error messages into functions that deal with "is this for an extra or not". And I've rearranged the loop over two different sets of requirements into one loop with a "must be instaled" flag. I hope you agree that this is clearer. * Test cases
* Fix rare error in `ReadWriteLock` when writers complete immediately (#12105)Sean Quah2022-03-011-1/+4
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Add `stop_cancellation` utility function (#12106)Sean Quah2022-03-011-0/+19
|
* Fix 'Unhandled error in Deferred' (#12089)Richard van der Hoff2022-03-011-32/+32
| | | | | | | | | | | | | * Fix 'Unhandled error in Deferred' Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when a function wrapped with `@cachedList` failed * Minor optimisation to cachedListDescriptor we can avoid re-using `missing`, which saves looking up entries in `deferreds_map`, and means we don't need to copy it. * Improve type annotation on CachedListDescriptor
* Improve exception handling for concurrent execution (#12109)Richard van der Hoff2022-03-012-23/+35
| | | | | | | | | | | | | | | * fix incorrect unwrapFirstError import this was being imported from the wrong place * Refactor `concurrently_execute` to use `yieldable_gather_results` * Improve exception handling in `yieldable_gather_results` Try to avoid swallowing so many stack traces. * mark unwrapFirstError deprecated * changelog
* Upgrade mypy to version 0.931 (#12030)Sean Quah2022-02-182-5/+9
| | | Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
* Track cache invalidations (#12000)Erik Johnston2022-02-153-1/+9
| | | Currently we only track evictions due to size or time constraints.
* Use version string helper from matrix-common (#11979)David Robertson2022-02-141-85/+0
| | | | * Require latest matrix-common * Use the common function
* Add missing type hints to synapse.replication. (#11938)Patrick Cloke2022-02-081-3/+2
|
* Add a callback to allow modules to deny 3PID (#11854)Brendan Abolivier2022-02-081-1/+12
| | | | | Part of the Tchap Synapse mainlining. This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
* Describe `prune_unread_entries` in docstrings (#11876)David Robertson2022-02-023-2/+17
| | | Should have been caught in #10826.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-132-9/+8
|
* Work around Mjolnir compatibility issue by adding an import for ↵reivilibre2022-01-061-0/+7
| | | | `glob_to_regex` in `synapse.util`, where it moved from. (#11696)
* Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre2022-01-051-58/+1
| | | | | #11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
* Update to the current version of Black and run it on Synapse codebase (#11596)Shay2021-12-231-1/+0
| | | | | | | | | * update black version * run updated version of black on code * newsfragment * enumerate python versions
* Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff2021-12-202-36/+113
| | | This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
* Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-143-1/+58
|
* Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah2021-12-071-1/+58
| | | | | | `matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
* Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)Sean Quah2021-12-061-58/+1
|
* Clean-up get_version_string (#11468)Patrick Cloke2021-12-011-57/+25
|
* Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)Sean Quah2021-11-301-1/+4
| | | | | | | | | | | When all entries in an `LruCache` have a size of 0 according to the provided `size_callback`, and `drop_from_cache` is called on a cache node, the node would be unlinked from the LRU linked list but remain in the cache dictionary. An assertion would be later be tripped due to the inconsistency. Avoid unintentionally calling `__len__` and use a strict `is None` check instead when unwrapping the weak reference.
* Eliminate a few `Any`s in `LruCache` type hints (#11453)Sean Quah2021-11-303-19/+31
|
* Merge branch 'master' into developSean Quah2021-11-231-7/+14
|\
| * Prevent the media store from writing outside of the configured directorySean Quah2021-11-191-7/+14
| | | | | | | | | | Also tighten validation of server names by forbidding invalid characters in IPv6 addresses and empty domain labels.
* | Add type annotations to `synapse.metrics` (#10847)Sean Quah2021-11-172-8/+9
| |
* | Add most missing type hints to synapse.util (#11328)Patrick Cloke2021-11-168-81/+157
| |
* | Add more type hints to synapse.util. (#11321)Patrick Cloke2021-11-121-12/+20
| |
* | Add missing type hints to `synapse.app`. (#11287)Patrick Cloke2021-11-103-9/+9
|/
* ObservableDeferred: run observers in order (#11229)Richard van der Hoff2021-11-021-16/+18
|
* Move DNS lookups into separate thread pool (#11177)Erik Johnston2021-10-261-0/+136
| | | | | | This is to stop large bursts of lookups starving out other users of the thread pools. Fixes #11049.
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-081-1/+4
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix long-standing bug where `ReadWriteLock` could drop logging contexts (#10993)Sean Quah2021-10-081-2/+4
| | | | | | | | | | | Use `PreserveLoggingContext()` to ensure that logging contexts are not lost when exiting a read/write lock. When exiting a read/write lock, callbacks on a `Deferred` are triggered as a signal to any waiting coroutines. Any waiting coroutine that becomes runnable is likely to follow the Synapse logging context rules and will restore its own logging context, then either run to completion or await another `Deferred`, resetting the logging context in the process.
* More types for synapse.util, part 1 (#10888)David Robertson2021-10-0610-73/+109
| | | | | | | | | | | | | | The following modules now pass `disallow_untyped_defs`: * synapse.util.caches.cached_call * synapse.util.caches.lrucache * synapse.util.caches.response_cache * synapse.util.caches.stream_change_cache * synapse.util.caches.ttlcache pass * synapse.util.daemonize * synapse.util.patch_inline_callbacks pass `no-untyped-defs` * synapse.util.versionstring Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
* Use direct references for configuration variables (part 7). (#10959)Patrick Cloke2021-10-041-2/+2
|
* Update utility code to handle C implementations of frozendict (#10902)Hillery Shay2021-09-281-1/+7
| | | | | | | * update _handle_frozendict to work with c implementations of frozen dict * add changelog * add clarifying comment to _handle_frozendict
* Encode JSON responses on a thread in C, mk2 (#10905)Erik Johnston2021-09-281-2/+17
| | | | | | | | | | Currently we use `JsonEncoder.iterencode` to write JSON responses, which ensures that we don't block the main reactor thread when encoding huge objects. The downside to this is that `iterencode` falls back to using a pure Python encoder that is *much* less efficient and can easily burn a lot of CPU for huge responses. To fix this, while still ensuring we don't block the reactor loop, we encode the JSON on a threadpool using the standard `JsonEncoder.encode` functions, which is backed by a C library. Doing so, however, requires `respond_with_json` to have access to the reactor, which it previously didn't. There are two ways of doing this: 1. threading through the reactor object, which is a bit fiddly as e.g. `DirectServeJsonResource` doesn't currently take a reactor, but is exposed to modules and so is a PITA to change; or 2. expose the reactor in `SynapseRequest`, which requires updating a bunch of servlet types. I went with the latter as that is just a mechanical change, and I think makes sense as a request already has a reactor associated with it (via its http channel).
* Opt out of cache expiry for `get_users_who_share_room_with_user` (#10826)David Robertson2021-09-223-3/+20
| | | | * Allow LruCaches to opt out of time-based expiry * Don't expire `get_users_who_share_room` & friends
* Track why we're evicting from caches (#10829)David Robertson2021-09-223-15/+30
| | | So we can see distinguish between "evicting because the cache is too big" and "evicting because the cache entries haven't been recently used".
* Add type hints to state database module. (#10823)Patrick Cloke2021-09-151-2/+2
|
* Use direct references for some configuration variables (#10798)Patrick Cloke2021-09-131-2/+2
| | | | Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).
* Add types to synapse.util. (#10601)reivilibre2021-09-1025-208/+281
|
* Add config option to use non-default manhole password and keys (#10643)Azrenbeth2021-09-061-3/+12
|
* Manhole: wrap coroutines in `defer.ensureDeferred` automatically (#10602)reivilibre2021-08-161-0/+14
|
* Convert Transaction and Edu object to attrs (#10542)Patrick Cloke2021-08-061-102/+0
| | | | | Instead of wrapping the JSON into an object, this creates concrete instances for Transaction and Edu. This allows for improved type hints and simplified code.
* Remove shebang line from module files (#10415)V024602021-07-291-1/+0
| | | Signed-off-by: Kai A. Hiller <V02460@gmail.com>
* Generics for `ObservableDeferred` (#10491)Richard van der Hoff2021-07-281-6/+8
| | | | | Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to follow suit.
* Fix up type hints for Twisted 21.7 (#10490)Richard van der Hoff2021-07-283-12/+21
| | | Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
* improve typing annotations in CachedCall (#10450)Richard van der Hoff2021-07-281-10/+17
| | | | | tighten up some of the typing in CachedCall, which is going to be needed when Twisted 21.7 brings better typing on Deferred.
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-193-8/+6
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-1516-59/+57
|
* Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-052-35/+352
|
* Standardise the module interface (#10062)Brendan Abolivier2021-06-181-15/+20
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Add debug logging when we enter/exit Measure block (#10183)Erik Johnston2021-06-161-0/+5
| | | It can be helpful to know when trying to track down slow requests.
* Extend `ResponseCache` to pass a context object into the callback (#10157)Richard van der Hoff2021-06-141-26/+73
| | | | | This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
* When joining a remote room limit the number of events we concurrently check ↵Erik Johnston2021-06-081-5/+16
| | | | | signatures/hashes for (#10117) If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.
* Fix up `BatchingQueue` (#10078)Erik Johnston2021-05-271-22/+48
| | | Fixes #10068
* Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-274-35/+36
| | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* Add missing type hints to synapse.util (#9982)Patrick Cloke2021-05-244-21/+19
|
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-244-47/+70
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-5/+3
|
* Add a batching queue implementation. (#10017)Erik Johnston2021-05-211-0/+153
|
* Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-141-6/+8
| | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* Simplify exception handling in is_ascii. (#9985)Dan Callahan2021-05-141-3/+1
| | | | | | | | | | | | | | | We can get away with just catching UnicodeError here. ⋮ +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError ⋮ https://docs.python.org/3/library/exceptions.html#exception-hierarchy Signed-off-by: Dan Callahan <danc@element.io>
* Use Python's secrets module instead of random (#9984)Dan Callahan2021-05-141-8/+11
| | | | | | | | | | | | | | | | Functionally identical, but more obviously cryptographically secure. ...Explicit is better than implicit? Avoids needing to know that SystemRandom() implies a CSPRNG, and complies with the big scary red box on the documentation for random: > Warning: > The pseudo-random generators of this module should not be used for > security purposes. For security or cryptographic uses, see the > secrets module. https://docs.python.org/3/library/random.html Signed-off-by: Dan Callahan <danc@element.io>
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-121-0/+20
|
* Merge branch 'master' into developBrendan Abolivier2021-05-111-14/+47
|\
| * Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff2021-05-111-14/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* | Optionally track memory usage of each LruCache (#9881)Erik Johnston2021-05-052-1/+78
| | | | | | | | | | This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
* | Reduce memory footprint of caches (#9886)Erik Johnston2021-04-281-18/+59
|/
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-1/+1
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Limit length of accepted email addresses (#9855)Erik Johnston2021-04-221-0/+30
|
* Remove `synapse.types.Collection` (#9856)Richard van der Hoff2021-04-222-4/+2
| | | This is no longer required, since we have dropped support for Python 3.5.
* Merge branch 'master' into developAndrew Morgan2021-04-201-5/+9
|\
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-201-5/+9
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Sanity check identity server passed to bind/unbind. (#9802)Denis Kasak2021-04-191-0/+32
| | | | | | | | Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1432-33/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-6/+8
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-061-32/+51
|
* Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong2021-04-051-1/+3
|
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-292-44/+73
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-2/+2
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-242-2/+2
|
* Fixed code misc. quality issues (#9649)Ankit Dobhal2021-03-221-1/+1
| | | | | - Merge 'isinstance' calls. - Remove unnecessary dict call outside of comprehension. - Use 'sys.exit()' calls.
* Add logging to ObservableDeferred callbacks (#9523)Jonathan de Jong2021-03-091-8/+18
|
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-6/+4
|
* Record the SSO Auth Provider in the login token (#9510)Richard van der Hoff2021-03-041-0/+89
| | | This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
* Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)"Patrick Cloke2021-03-021-32/+2
| | | | | | | This reverts commit f5c93fc9931e4029bbd8000f398b6f39d67a8c46. This is being backed out due to a regression (#9507) and additional review feedback being provided.
* Fix #8518 (sync requests being cached wrongly on timeout) (#9358)Jonathan de Jong2021-02-241-2/+32
| | | | | | | This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-1611-48/+45
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-161-0/+129
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Fix some typos.Patrick Cloke2021-02-121-1/+1
|
* Merge tag 'v1.27.0rc2' into developPatrick Cloke2021-02-111-3/+12
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.27.0rc2 (2021-02-11) ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
| * Backout changes for automatically calculating the public baseurl. (#9313)Patrick Cloke2021-02-111-3/+12
| | | | | | | | This breaks some people's configurations (if their Client-Server API is not accessed via port 443).
* | Type hints and validation improvements. (#9321)Patrick Cloke2021-02-081-15/+18
|/ | | | | * Adds type hints to the groups servlet and stringutils code. * Assert the maximum length of some input values for spec compliance.
* Merge branch 'social_login' into developRichard van der Hoff2021-02-011-0/+106
|\
| * Replace username picker with a template (#9275)Richard van der Hoff2021-02-011-0/+106
| | | | | | | | | | There's some prelimiary work here to pull out the construction of a jinja environment to a separate function. I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
* | Pass a dict, instead of None, to modules if a None config is specified in ↵Andrew Morgan2021-01-271-1/+2
|/ | | | | | | | | the homeserver config (#9229) If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead. This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](https://github.com/matrix-org/synapse/blob/5310808d3bebd17275355ecd474bc013e8c7462d/docs/sample_config.yaml#L1816-L1845) states) then `None` is not passed to the module. An empty dict is passed instead. This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
* Fix chain cover update to handle events with duplicate auth events (#9210)Erik Johnston2021-01-221-1/+1
|
* Support icons for Identity Providers (#9154)Richard van der Hoff2021-01-201-0/+92
|
* Remote dependency on distutils (#9125)Richard van der Hoff2021-01-151-0/+19
| | | | | | | | | `distutils` is pretty much deprecated these days, and replaced with `setuptools`. It's also annoying because it's you can't `pip install` it, and it's hard to figure out which debian package we should depend on to make sure it's there. Since we only use it for a tiny function anyway, let's just vendor said function into our codebase.
* Fix event chain bg update. (#9118)Erik Johnston2021-01-141-1/+1
| | | | We passed in a graph to `sorted_topologically` which didn't have an entry for each node (as we dropped nodes with no edges).
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-111-1/+52
|
* Fix "Starting metrics collection from sentinel context" errors (#9053)Erik Johnston2021-01-081-1/+2
|
* Empty iterables should count towards cache usage. (#9028)Patrick Cloke2021-01-061-1/+1
|
* Add type hints to the logging context code. (#8939)Patrick Cloke2021-01-051-1/+9
|
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-112-8/+7
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Better formatting for config errors from modules (#8874)Richard van der Hoff2020-12-081-6/+58
| | | | | | | | | | The idea is that the parse_config method of extension modules can raise either a ConfigError or a JsonValidationError, and it will be magically turned into a legible error message. There's a few components to it: * Separating the "path" and the "message" parts of a ConfigError, so that we can fiddle with the path bit to turn it into an absolute path. * Generally improving the way ConfigErrors get printed. * Passing in the config path to load_module so that it can wrap any exceptions that get caught appropriately.
* Implement and use an @lru_cache decorator (#8595)Richard van der Hoff2020-10-301-47/+188
| | | We don't always need the full power of a DeferredCache.
* Merge pull request #8678 from matrix-org/rav/fix_frozen_eventsRichard van der Hoff2020-10-282-25/+21
|\ | | | | Fix serialisation errors when using third-party event rules.
| * Remove frozendict_json_encoder and support frozendicts everywhereRichard van der Hoff2020-10-282-25/+21
| | | | | | | | | | | | Not being able to serialise `frozendicts` is fragile, and it's annoying to have to think about which serialiser you want. There's no real downside to supporting frozendicts, so let's just have one json encoder.
* | Tell Black to format code for Python 3.5 (#8664)Dan Callahan2020-10-271-1/+1
|/ | | | | | | | This allows trailing commas in multi-line arg lists. Minor, but we might as well keep our formatting current with regard to our minimum supported Python version. Signed-off-by: Dan Callahan <danc@element.io>
* Optimise CacheDescriptor (#8594) github/release-v1.21.3 release-v1.21.3Richard van der Hoff2020-10-211-5/+7
| | | don't bother constricting a CacheContext unless we need one.
* fix failure caseRichard van der Hoff2020-10-211-1/+4
|
* optimise DeferredCache.setRichard van der Hoff2020-10-211-3/+12
|
* Push some deferred wrangling down into DeferredCacheRichard van der Hoff2020-10-212-34/+55
|
* Fix 'LruCache' object has no attribute '_on_resize' (#8591)Richard van der Hoff2020-10-191-1/+9
| | | We need to make sure we are readu for the `set_cache_factor` callback.
* Add `DeferredCache.get_immediate` method (#8568)Richard van der Hoff2020-10-191-10/+25
| | | | | | | | | | | * Add `DeferredCache.get_immediate` method A bunch of things that are currently calling `DeferredCache.get` are only really interested in the result if it's completed. We can optimise and simplify this case. * Remove unused 'default' parameter to DeferredCache.get() * another get_immediate instance
* Replace DeferredCache with LruCache where possible (#8563)Richard van der Hoff2020-10-191-0/+3
| | | Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
* review commentsRichard van der Hoff2020-10-161-3/+8
|
* Apply suggestions from code reviewRichard van der Hoff2020-10-161-1/+1
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* type annotations for LruCacheRichard van der Hoff2020-10-163-23/+77
|
* Make LruCache register its own metrics (#8561)Richard van der Hoff2020-10-164-54/+57
| | | | | rather than have everything that instantiates an LruCache manage metrics separately, have LruCache do it itself.
* Enable mypy for synapse.util.caches (#8547)Richard van der Hoff2020-10-151-1/+1
| | | This seemed to entail dragging in a type stub for SortedList.
* move DeferredCache into its own moduleRichard van der Hoff2020-10-142-282/+294
|
* Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-7/+12
|
* Add some more type annotations to CacheRichard van der Hoff2020-10-142-23/+61
|
* Add type hints to response cache. (#8507)Patrick Cloke2020-10-091-22/+28
|
* Clarify error message when plugin config parsers raise an error (#8492)Richard van der Hoff2020-10-081-1/+1
| | | | | | | | | This turns: Failed to parse config for 'myplugin': Exception('error message') into: Failed to parse config for 'myplugin': error message.
* Expose a `get_resource_usage` method in `Measure`Richard van der Hoff2020-09-291-10/+21
|
* Fix handling of connection timeouts in outgoing http requests (#8400)Richard van der Hoff2020-09-291-29/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove `on_timeout_cancel` from `timeout_deferred` The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a timeout (in particular if the canceller raised an exception), so it was unreliable. It was also only used in one place, and to be honest it's easier to do what it does a different way. * Fix handling of connection timeouts in outgoing http requests Turns out that if we get a timeout during connection, then a different exception is raised, which wasn't always handled correctly. To fix it, catch the exception in SimpleHttpClient and turn it into a RequestTimedOutError (which is already a documented exception). Also add a description to RequestTimedOutError so that we can see which stage it failed at. * Fix incorrect handling of timeouts reading federation responses This was trapping the wrong sort of TimeoutError, so was never being hit. The effect was relatively minor, but we should fix this so that it does the expected thing. * Fix inconsistent handling of `timeout` param between methods `get_json`, `put_json` and `delete_json` were applying a different timeout to the response body to `post_json`; bring them in line and test. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-2/+2
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Remove obsolete __future__ imports (#8337)Jonathan de Jong2020-09-171-2/+0
|
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-142-2/+2
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Merge tag 'v1.20.0rc3' into developPatrick Cloke2020-09-111-1/+1
|\ | | | | | | | | | | | | | | | | | | Synapse 1.20.0rc3 (2020-09-11) ============================== Bugfixes -------- - Fix a bug introduced in v1.20.0rc1 where the wrong exception was raised when invalid JSON data is encountered. ([\#8291](https://github.com/matrix-org/synapse/issues/8291))
| * Fix the exception that is raised when invalid JSON is encountered. (#8291)Patrick Cloke2020-09-101-1/+1
| |
* | Remove some unused distributor signals (#8216)Patrick Cloke2020-09-091-42/+8
| | | | | | | | | | Removes the `user_joined_room` and stops calling it since there are no observers. Also cleans-up some other unused signals and related code.
* | Add types to async_helpers (#8260)Patrick Cloke2020-09-081-50/+85
| |
* | Directly import json from the standard library. (#8259)Patrick Cloke2020-09-082-3/+4
|/ | | | | By importing from canonicaljson the simplejson module was still being used in some situations. After this change the std lib json is consistenty used throughout Synapse.
* Stop sub-classing object (#8249)Patrick Cloke2020-09-0417-31/+31
|
* Fix typing for `@cached` wrapped functions (#8240)Erik Johnston2020-09-031-14/+28
| | | This requires adding a mypy plugin to fiddle with the type signatures a bit.
* Convert ReadWriteLock to async/await. (#8202)Patrick Cloke2020-08-281-8/+8
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-191-2/+12
|
* Remove the unused inlineCallbacks code-paths in the caching code (#8119)Patrick Cloke2020-08-191-46/+8
|
* Remove : from allowed client_secret chars (#8101)Andrew Morgan2020-08-181-3/+1
| | | | | | | Closes: https://github.com/matrix-org/synapse/issues/6766 Equivalent Sydent PR: https://github.com/matrix-org/sydent/pull/309 I believe it's now time to remove the extra allowed `:` from `client_secret` parameters.
* Fix typing for notifier (#8064)Erik Johnston2020-08-121-3/+6
|
* Remove unnecessary maybeDeferred calls (#8044)Patrick Cloke2020-08-071-1/+1
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-072-2/+9
|
* Convert some util functions to async (#8035)Patrick Cloke2020-08-062-28/+27
|
* Stop the parent process flushing the logs on exit (#8012)Richard van der Hoff2020-08-051-2/+8
| | | This solves the problem that the first few lines are logged twice on matrix.org. Hopefully the comments explain it.
* re-implement daemonize (#8011)Richard van der Hoff2020-08-041-0/+131
| | | | | | | | | | | | | This has long been something I've wanted to do. Basically the `Daemonize` code is both too flexible and not flexible enough, in that it offers a bunch of features that we don't use (changing UID, closing FDs in the child, logging to syslog) and doesn't offer a bunch that we could do with (redirecting stdout/err to a file instead of /dev/null; having the parent not exit until the child is running). As a first step, I've lifted the Daemonize code and removed the bits we don't use. This should be a non-functional change. Fixing everything else will come later.
* Fix deprecation warning: import ABC from collections.abc (#7892)Karthikeyan Singaravelan2020-07-201-1/+1
|
* Convert device handler to async/await (#7871)Patrick Cloke2020-07-171-2/+26
|
* Fix some spelling mistakes / typos. (#7811)Patrick Cloke2020-07-096-7/+7
|
* Fix inconsistent handling of upper and lower cases of email addresses. (#7021)Dirk Klimpel2020-07-031-0/+23
| | | fixes #7016
* Fix "argument of type 'ObservableDeferred' is not iterable" error (#7708)Patrick Cloke2020-06-161-1/+1
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-165-12/+4
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-153-10/+4
|
* Performance improvements and refactor of Ratelimiter (#7595)Andrew Morgan2020-06-051-1/+1
| | | | | | | | | | While working on https://github.com/matrix-org/synapse/issues/5665 I found myself digging into the `Ratelimiter` class and seeing that it was both: * Rather undocumented, and * causing a *lot* of config checks This PR attempts to refactor and comment the `Ratelimiter` class, as well as encourage config file accesses to only be done at instantiation. Best to be reviewed commit-by-commit.
* Speed up processing of federation stream RDATA rows.Erik Johnston2020-05-271-0/+12
| | | | | | Instead of storing and sending an ACK for every single row we send synchronously, we instead do it asynchronously while batching up updates.
* Don't apply cache factor to event cache. (#7578)Erik Johnston2020-05-271-0/+4
| | | | This is already correctly done when we instansiate the cache, but wasn't when it got reloaded (which always happens at least once on startup).
* Fix stacktrace mangling in `patch_inline_callbacks` (#7554)Richard van der Hoff2020-05-221-2/+7
| | | `Failure()` is more cunning than `Failure(e)`.
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-152-27/+8
|
* remove to_asciiRichard van der Hoff2020-05-151-19/+1
| | | | this is a no-op on python 3.
* Remove `exception_to_unicode`Richard van der Hoff2020-05-151-36/+0
| | | | this is a no-op on python 3.
* Strictly enforce canonicaljson requirements in a new room version (#7381)Patrick Cloke2020-05-141-1/+1
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-117-92/+206
| | | | variables (#6391)
* Speed up fetching device lists changes in sync.Erik Johnston2020-05-051-4/+15
| | | | | Currently we copy `users_who_share_room` needlessly about three times, which is expensive when the set is large (which it can easily be).
* Extend StreamChangeCache to support multiple entities per stream ID (#7303)Richard van der Hoff2020-04-221-46/+71
| | | | | | | | | | | | | | | | | | | First some background: StreamChangeCache is used to keep track of what "entities" have changed since a given stream ID. So for example, we might use it to keep track of when the last to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2]. Now, it turns out that StreamChangeCache didn't support more than one thing being changed at a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send to-device messages to more than one user at a time. As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped ok with having multiple things changing on the same stream ID, and it seems we never actually use the methods which don't work on the stream change caches where we allow multiple changes at the same stream ID. But that feels horribly fragile, hence: let's update StreamChangeCache to properly support this, and add some typing and some more tests while we're at it. [1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301 [2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51
* On catchup, process each row with its own stream id (#7286)Richard van der Hoff2020-04-201-0/+3
| | | | | | Other parts of the code (such as the StreamChangeCache) assume that there will not be multiple changes with the same stream id. This code was introduced in #7024, and I hope this fixes #7206.
* Rewrite prune_old_outbound_device_pokes for efficiency (#7159)Richard van der Hoff2020-03-301-1/+20
| | | | make sure we clear out all but one update for the user
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-242-20/+20
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-1/+1
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-1/+1
| | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-16 03:35:38 +0000