summary refs log tree commit diff
path: root/tests/util (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Allow config of the backoff algorithm for the federation client. (#15754)Mathieu Velten2023-08-031-11/+11
| | | | | | | | | | | Adds three new configuration variables: * destination_min_retry_interval is identical to before (10mn). * destination_retry_multiplier is now 2 instead of 5, the maximum value will be reached slower. * destination_max_retry_interval is one day instead of (essentially) infinity. Capping this will cause destinations to continue to be retried sometimes instead of being lost forever. The previous value was 2 ^ 62 milliseconds.
* Add type hints to test_descriptors. (#15659)Patrick Cloke2023-05-241-93/+104
| | | Require type hints in test_descriptors and add missing ones.
* Add final type hint to tests.unittest. (#15072)Patrick Cloke2023-02-141-0/+2
| | | | Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.
* Remove unnecessary reactor reference from `_PerHostRatelimiter` (#14842)Sean Quah2023-01-161-4/+4
| | | | | Fix up #14812 to avoid introducing a reference to the reactor. Signed-off-by: Sean Quah <seanq@matrix.org>
* Fix stack overflow in `_PerHostRatelimiter` due to synchronous requests (#14812)Sean Quah2023-01-131-3/+42
| | | | | | | | | | | | | | | | | | When there are many synchronous requests waiting on a `_PerHostRatelimiter`, each request will be started recursively just after the previous request has completed. Under the right conditions, this leads to stack exhaustion. A common way for requests to become synchronous is when the remote client disconnects early, because the homeserver is overloaded and slow to respond. Avoid stack exhaustion under these conditions by deferring subsequent requests until the next reactor tick. Fixes #14480. Signed-off-by: Sean Quah <seanq@matrix.org>
* Support RFC7636 PKCE in the OAuth 2.0 flow. (#14750)Patrick Cloke2023-01-041-0/+1
| | | | | | | PKCE can protect against certain attacks and is enabled by default. Support can be controlled manually by setting the pkce_method of each oidc_providers entry to 'auto' (default), 'always', or 'never'. This is required by Twitter OAuth 2.0 support.
* Check the stream position before checking if the cache is empty. (#14639)Patrick Cloke2022-12-081-3/+4
| | | | | | An empty cache does not mean the entity has no changed, if it is earlier than the earliest known stream position return that the entity *has* changed since the cache cannot accurately answer that query.
* Better return type for `get_all_entities_changed` (#14604)Erik Johnston2022-12-051-8/+12
| | | | Help callers from using the return value incorrectly by ensuring that callers explicitly check if there was a cache hit or not.
* Compare to the earliest known stream pos in the stream change cache. (#14435)Patrick Cloke2022-12-051-25/+13
| | | | | | The internal methods of the StreamChangeCache were inconsistently treating the earliest known stream position as valid. It is now treated as invalid, meaning the cache cannot determine if an entity at the earliest known stream position has changed or not.
* Add missing types to tests.util. (#14597)Patrick Cloke2022-12-0219-266/+357
| | | | Removes files under tests.util from the ignored by list, then fully types all tests/util/*.py files.
* Add missing type hints to test.util.caches (#14529)Patrick Cloke2022-11-225-61/+69
|
* Save login tokens in database (#13844)Quentin Gliech2022-10-261-28/+0
| | | | | | | | | | | * Save login tokens in database Signed-off-by: Quentin Gliech <quenting@element.io> * Add upgrade notes * Track login token reuse in a Prometheus metric Signed-off-by: Quentin Gliech <quenting@element.io>
* Remove unused `@lru_cache` decorator (#13595)Nick Mills-Barrett2022-10-251-36/+4
| | | | | | | * Remove unused `@lru_cache` decorator Spotted this working on something else. Co-authored-by: David Robertson <davidr@element.io>
* Bump flake8-bugbear from 21.3.2 to 22.9.23 (#14042)dependabot[bot]2022-10-191-1/+1
| | | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Don't require `setuptools_rust` at runtime (#13952)David Robertson2022-09-291-2/+18
|
* Fix `have_seen_event` cache not being invalidated (#13863)Eric Eastwood2022-09-271-1/+32
| | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/13856 Fix https://github.com/matrix-org/synapse/issues/13865 > Discovered while trying to make Synapse fast enough for [this MSC2716 test for importing many batches](https://github.com/matrix-org/complement/pull/214#discussion_r741678240). As an example, disabling the `have_seen_event` cache saves 10 seconds for each `/messages` request in that MSC2716 Complement test because we're not making as many federation requests for `/state` (speeding up `have_seen_event` itself is related to https://github.com/matrix-org/synapse/issues/13625) > > But this will also make `/messages` faster in general so we can include it in the [faster `/messages` milestone](https://github.com/matrix-org/synapse/milestone/11). > > *-- https://github.com/matrix-org/synapse/issues/13856* ### The problem `_invalidate_caches_for_event` doesn't run in monolith mode which means we never even tried to clear the `have_seen_event` and other caches. And even in worker mode, it only runs on the workers, not the master (AFAICT). Additionally there was bug with the key being wrong so `_invalidate_caches_for_event` never invalidates the `have_seen_event` cache even when it does run. Because we were using the `@cachedList` wrong, it was putting items in the cache under keys like `((room_id, event_id),)` with a `set` in a `set` (ex. `(('!TnCIJPKzdQdUlIyXdQ:test', '$Iu0eqEBN7qcyF1S9B3oNB3I91v2o5YOgRNPwi_78s-k'),)`) and we we're trying to invalidate with just `(room_id, event_id)` which did nothing.
* Make DictionaryCache have better expiry properties (#13292)Erik Johnston2022-07-211-4/+31
|
* Move the "email unsubscribe" resource, refactor the macaroon generator & ↵Quentin Gliech2022-06-141-0/+146
| | | | | | | | | | | | | | | | | | | | | simplify the access token verification logic. (#12986) This simplifies the access token verification logic by removing the `rights` parameter which was only ever used for the unsubscribe link in email notifications. The latter has been moved under the `/_synapse` namespace, since it is not a standard API. This also makes the email verification link more secure, by embedding the app_id and pushkey in the macaroon and verifying it. This prevents the user from tampering the query parameters of that unsubscribe link. Macaroon generation is refactored: - Centralised all macaroon generation and verification logic to the `MacaroonGenerator` - Moved to `synapse.utils` - Changed the constructor to require only a `Clock`, hostname, and a secret key (instead of a full `Homeserver`). - Added tests for all methods.
* Add config flags to allow for cache auto-tuning (#12701)Shay2022-05-131-1/+57
|
* Immediately retry any requests that have backed off when a server comes back ↵Erik Johnston2022-05-101-0/+80
| | | | | online. (#12500) Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
* Update `delay_cancellation` to accept any awaitable (#12468)Sean Quah2022-04-221-2/+31
| | | | | | | | This will mainly be useful when dealing with module callbacks, which are all typed as returning `Awaitable`s instead of coroutines or `Deferred`s. Signed-off-by: Sean Quah <seanq@element.io>
* Fix edge case where a `Linearizer` could get stuck (#12358)Sean Quah2022-04-051-4/+47
| | | | | | | Just after a task acquires a contended `Linearizer` lock, it sleeps. If the task is cancelled during this sleep, we need to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Update docstrings for `ReadWriteLock` tests (#12354)Sean Quah2022-04-051-6/+6
| | | Signed-off-by: Sean Quah <seanq@element.io>
* Refactor and convert `Linearizer` to async (#12357)Sean Quah2022-04-051-2/+2
| | | | | | | | | | | Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Convert `Linearizer` tests from `inlineCallbacks` to async (#12353)Sean Quah2022-04-051-91/+133
| | | Signed-off-by: Sean Quah <seanq@element.io>
* Try to detect borked package installations. (#12244)David Robertson2022-03-181-1/+14
| | | | | | | * Try to detect borked package installations. Fixes #12223. Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Add cancellation support to `@cached` and `@cachedList` decorators (#12183)Sean Quah2022-03-141-2/+145
| | | | | | | | These decorators mostly support cancellation already. Add cancellation tests and fix use of finished logging contexts by delaying cancellation, as suggested by @erikjohnston. Signed-off-by: Sean Quah <seanq@element.io>
* Add cancellation support to `ReadWriteLock` (#12120)Sean Quah2022-03-141-58/+337
| | | | | | Also convert `ReadWriteLock` to use async context managers. Signed-off-by: Sean Quah <seanq@element.io>
* Add `delay_cancellation` utility function (#12180)Sean Quah2022-03-141-6/+118
| | | | | | | | | `delay_cancellation` behaves like `stop_cancellation`, except it delays `CancelledError`s until the original `Deferred` resolves. This is handy for unifying cleanup paths and ensuring that uncancelled coroutines don't use finished logcontexts. Signed-off-by: Sean Quah <seanq@element.io>
* Allow for ignoring some arguments when caching. (#12189)Patrick Cloke2022-03-091-3/+81
| | | | | * `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key. * Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity. * Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2022-03-081-0/+19
|\
| * Fix a bug introduced in 1.54.0rc1 which meant that Synapse would refuse to ↵reivilibre2022-03-081-0/+19
| | | | | | | | | | | | | | | | | | | | | | start if pre-release versions of dependencies were installed. (#12177) * Add failing test to characterise the regression #12176 * Permit pre-release versions of specified packages * Newsfile (bugfix) Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Merge remote-tracking branch 'origin/release-v1.54' into developRichard van der Hoff2022-03-041-2/+19
|\|
| * Don't impose version checks on dev extras at runtime (#12129)David Robertson2022-03-031-2/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix incorrect argument in test case * Add copyright header * Docstring and __all__ * Exclude dev depenencies * Use changelog from #12088 * Include version in error messages This will hopefully distinguish between the version of the source code and the version of the distribution package that is installed. * Linter script is your friend
* | Add test for `ObservableDeferred`'s cancellation behaviour (#12149)Sean Quah2022-03-041-0/+28
|/ | | Signed-off-by: Sean Quah <seanq@element.io>
* Use importlib.metadata to read requirements (#12088)David Robertson2022-03-011-0/+95
| | | | | | | | | | | | | | | | | | * Pull runtime dep checks into their own module * Reimplement `check_requirements` using `importlib` I've tried to make this clearer. We start by working out which of Synapse's requirements we need to be installed here and now. I was surprised that there wasn't an easier way to see which packages were installed by a given extra. I've pulled out the error messages into functions that deal with "is this for an extra or not". And I've rearranged the loop over two different sets of requirements into one loop with a "must be instaled" flag. I hope you agree that this is clearer. * Test cases
* Fix rare error in `ReadWriteLock` when writers complete immediately (#12105)Sean Quah2022-03-011-0/+30
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Add `stop_cancellation` utility function (#12106)Sean Quah2022-03-011-0/+45
|
* Fix 'Unhandled error in Deferred' (#12089)Richard van der Hoff2022-03-011-5/+5
| | | | | | | | | | | | | * Fix 'Unhandled error in Deferred' Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when a function wrapped with `@cachedList` failed * Minor optimisation to cachedListDescriptor we can avoid re-using `missing`, which saves looking up entries in `deferreds_map`, and means we don't need to copy it. * Improve type annotation on CachedListDescriptor
* Improve exception handling for concurrent execution (#12109)Richard van der Hoff2022-03-011-2/+113
| | | | | | | | | | | | | | | * fix incorrect unwrapFirstError import this was being imported from the wrong place * Refactor `concurrently_execute` to use `yieldable_gather_results` * Improve exception handling in `yieldable_gather_results` Try to avoid swallowing so many stack traces. * mark unwrapFirstError deprecated * changelog
* Replace assertEquals and friends with non-deprecated versions. (#12092)Patrick Cloke2022-02-286-151/+151
|
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-231-2/+2
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common ↵reivilibre2022-01-051-59/+0
| | | | | #11505 (#11687) Co-authored-by: Sean Quah <seanq@element.io>
* Improve opentracing support for `ResponseCache` (#11607)Richard van der Hoff2021-12-201-11/+34
| | | This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
* Add missing type hints to `synapse.logging.context` (#11556)Sean Quah2021-12-141-35/+0
|
* Revert "Move `glob_to_regex` and `re_word_boundary` to ↵Sean Quah2021-12-071-0/+59
| | | | | | `matrix-python-common` (#11505) (#11527) This reverts commit a77c36989785c0d5565ab9a1169f4f88e512ce8a.
* Move `glob_to_regex` and `re_word_boundary` to `matrix-python-common` (#11505)Sean Quah2021-12-061-59/+0
|
* Fix `LruCache` corruption bug with a `size_callback` that can return 0 (#11454)Sean Quah2021-11-301-0/+12
| | | | | | | | | | | When all entries in an `LruCache` have a size of 0 according to the provided `size_callback`, and `drop_from_cache` is called on a cache node, the node would be unlinked from the LRU linked list but remain in the cache dictionary. An assertion would be later be tripped due to the inconsistency. Avoid unintentionally calling `__len__` and use a strict `is None` check instead when unwrapping the weak reference.
* Additional test for `cachedList` (#11246)Richard van der Hoff2021-11-041-0/+43
| | | | | I was trying to understand how `cachedList` works, and ended up writing this extra test. I figure we may as well keep it.
* ObservableDeferred: run observers in order (#11229)Richard van der Hoff2021-11-022-4/+69
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+1
|
* Use inline type hints in `tests/` (#10350)Jonathan de Jong2021-07-132-10/+10
| | | | | | | | This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)
* Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-051-1/+45
|
* Extend `ResponseCache` to pass a context object into the callback (#10157)Richard van der Hoff2021-06-141-7/+68
| | | | | This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
* Rewrite the KeyRing (#10035)Erik Johnston2021-06-021-21/+16
|
* Fix up `BatchingQueue` (#10078)Erik Johnston2021-05-271-2/+76
| | | Fixes #10068
* Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-271-3/+3
| | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* Add missing type hints to synapse.util (#9982)Patrick Cloke2021-05-241-2/+2
|
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-242-4/+6
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-211-7/+11
|
* Add a batching queue implementation. (#10017)Erik Johnston2021-05-211-0/+169
|
* Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-141-3/+14
| | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-121-1/+7
|
* Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff2021-05-111-0/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* Merge branch 'master' into developAndrew Morgan2021-04-201-4/+2
|\
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-201-4/+2
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1421-21/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-094-6/+4
|
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-2/+4
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Record more information into structured logs. (#9654)Patrick Cloke2021-04-082-28/+14
| | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-1/+3
|
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-0/+131
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-164-15/+23
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-161-0/+161
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Fix chain cover update to handle events with duplicate auth events (#9210)Erik Johnston2021-01-221-0/+12
|
* Fix event chain bg update. (#9118)Erik Johnston2021-01-141-0/+8
| | | | We passed in a graph to `sorted_topologically` which didn't have an entry for each node (as we dropped nodes with no edges).
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-111-1/+40
|
* Empty iterables should count towards cache usage. (#9028)Patrick Cloke2021-01-061-23/+50
|
* Implement and use an @lru_cache decorator (#8595)Richard van der Hoff2020-10-301-1/+59
| | | We don't always need the full power of a DeferredCache.
* Add some more testsRichard van der Hoff2020-10-212-0/+147
|
* Push some deferred wrangling down into DeferredCacheRichard van der Hoff2020-10-212-12/+11
|
* Combine the two sets of tests for CacheDescriptorRichard van der Hoff2020-10-211-0/+230
|
* Fix 'LruCache' object has no attribute '_on_resize' (#8591)Richard van der Hoff2020-10-191-1/+7
| | | We need to make sure we are readu for the `set_cache_factor` callback.
* Add `DeferredCache.get_immediate` method (#8568)Richard van der Hoff2020-10-191-4/+23
| | | | | | | | | | | * Add `DeferredCache.get_immediate` method A bunch of things that are currently calling `DeferredCache.get` are only really interested in the result if it's completed. We can optimise and simplify this case. * Remove unused 'default' parameter to DeferredCache.get() * another get_immediate instance
* Make LruCache register its own metrics (#8561)Richard van der Hoff2020-10-161-2/+2
| | | | | rather than have everything that instantiates an LruCache manage metrics separately, have LruCache do it itself.
* Combine the two sets of DeferredCache testsRichard van der Hoff2020-10-141-2/+75
|
* move DeferredCache into its own moduleRichard van der Hoff2020-10-142-44/+64
|
* Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-2/+2
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-042-12/+12
|
* Convert ReadWriteLock to async/await. (#8202)Patrick Cloke2020-08-281-2/+4
|
* Reduce run-times of tests by advancing the reactor less (#7757)Andrew Morgan2020-08-271-1/+1
|
* Remove the unused inlineCallbacks code-paths in the caching code (#8119)Patrick Cloke2020-08-191-6/+6
|
* Remove : from allowed client_secret chars (#8101)Andrew Morgan2020-08-181-3/+0
| | | | | | | Closes: https://github.com/matrix-org/synapse/issues/6766 Equivalent Sydent PR: https://github.com/matrix-org/sydent/pull/309 I believe it's now time to remove the extra allowed `:` from `client_secret` parameters.
* Convert some util functions to async (#8035)Patrick Cloke2020-08-061-33/+11
|
* Fix some spelling mistakes / typos. (#7811)Patrick Cloke2020-07-091-2/+2
|
* Fix inconsistent handling of upper and lower cases of email addresses. (#7021)Dirk Klimpel2020-07-031-0/+49
| | | fixes #7016
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-162-3/+1
|
* Add test for Linearizer.is_queued(..)Erik Johnston2020-05-271-0/+32
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-113-8/+5
| | | | variables (#6391)
* Extend StreamChangeCache to support multiple entities per stream ID (#7303)Richard van der Hoff2020-04-221-9/+60
| | | | | | | | | | | | | | | | | | | First some background: StreamChangeCache is used to keep track of what "entities" have changed since a given stream ID. So for example, we might use it to keep track of when the last to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2]. Now, it turns out that StreamChangeCache didn't support more than one thing being changed at a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send to-device messages to more than one user at a time. As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped ok with having multiple things changing on the same stream ID, and it seems we never actually use the methods which don't work on the stream change caches where we allow multiple changes at the same stream ID. But that feels horribly fragile, hence: let's update StreamChangeCache to properly support this, and add some typing and some more tests while we're at it. [1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301 [2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-244-28/+37
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-13/+5
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Validate client_secret parameter (#6767)Andrew Morgan2020-01-241-0/+51
|
* Log saml assertions rather than the whole responseRichard van der Hoff2020-01-161-0/+47
| | | | | | ... since the whole response is huge. We even need to break up the assertions, since kibana otherwise truncates them.
* Merge pull request #6505 from matrix-org/erikj/make_deferred_yiedableErik Johnston2019-12-101-0/+24
|\ | | | | Fix `make_deferred_yieldable` to work with coroutines
| * Fix make_deferred_yieldable to work with coroutinesErik Johnston2019-12-101-0/+24
| |
* | Remove SnapshotCache in favour of ResponseCacheErik Johnston2019-12-091-63/+0
|/
* Make ObservableDeferred.observe() always return deferred.Erik Johnston2019-10-301-2/+2
| | | | | | | This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.
* Quick fix to ensure cache descriptors always return deferredsErik Johnston2019-10-281-2/+2
|
* Add 'failure_ts' column to 'destinations' table (#6016)Richard van der Hoff2019-09-171-0/+127
| | | | Track the time that a server started failing at, for general analysis purposes.
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-2/+2
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Fix some error cases in the caching layer. (#5749)Richard van der Hoff2019-07-251-3/+87
| | | | | | | There was some inconsistent behaviour in the caching layer around how exceptions were handled - particularly synchronously-thrown ones. This seems to be most easily handled by pushing the creation of ObservableDeferreds down from CacheDescriptor to the Cache.
* Replace returnValue with return (#5736)Amber Brown2019-07-231-4/+4
|
* Fixes to the federation rate limiter (#5621)Richard van der Hoff2019-07-051-0/+97
| | | | | | | - Put the default window_size back to 1000ms (broken by #5181) - Make the `rc_federation` config actually do something - fix an off-by-one error in the 'concurrent' limit - Avoid creating an unused `_PerHostRatelimiter` object for every single incoming request
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-045-47/+44
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-12/+21
|
* Run Black. (#5482)Amber Brown2019-06-202-53/+53
|
* Run Black on the tests again (#5170)Amber Brown2019-05-101-12/+11
|
* Remove periods from copyright headers (#5046)Andrew Morgan2019-04-111-1/+1
|
* Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-0/+83
|
* Fix incorrect logcontexts after a Deferred was cancelled (#4407)Richard van der Hoff2019-01-171-0/+104
|
* Include eventid in log lines when processing incoming federation ↵Richard van der Hoff2018-09-271-0/+5
| | | | | | | | | | | transactions (#3959) when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.
* Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* Run black.black2018-08-1010-76/+41
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-102-2/+2
|
* fix invalidationRichard van der Hoff2018-07-271-1/+41
|
* Rewrite cache list decoratorRichard van der Hoff2018-07-271-0/+61
| | | | | Because it was complicated and annoyed me. I suspect this will be more efficient too.
* Test and fix support for cancellation in LinearizerRichard van der Hoff2018-07-201-0/+31
|
* Combine Limiter and LinearizerRichard van der Hoff2018-07-202-70/+47
| | | | | Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.
* Add a sleep to the Limiter to fix stack overflows.Richard van der Hoff2018-07-201-4/+4
| | | | Fixes #3570
* Add unit testErik Johnston2018-07-171-0/+16
|
* Fixup unit testErik Johnston2018-07-131-3/+3
|
* run isortAmber Brown2018-07-0914-30/+38
|
* Reinstate lost run_on_reactor in unit testRichard van der Hoff2018-07-041-2/+15
| | | | | | | | a61738b removed a call to run_on_reactor from a unit test, but that call was doing something useful, in making the function in question asynchronous. Reinstate the call and add a check that we are testing what we wanted to be testing.
* Disable partial state group caching for wildcard lookupsRichard van der Hoff2018-06-221-6/+6
| | | | | | | When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-223-11/+13
|
* Remove run_on_reactor (#3395)Amber Brown2018-06-141-2/+0
|
* Port to sortedcontainers (with tests!) (#3332)Amber Brown2018-06-061-0/+198
|
* fix another dumb typoMatthew Hodgson2018-05-291-2/+2
|
* Merge branch 'develop' into rav/more_logcontext_leaksRichard van der Hoff2018-05-021-0/+38
|\
| * Fix incorrect reference to StringIORichard van der Hoff2018-05-021-0/+38
| | | | | | | | This was introduced in 4f2f5171
* | Remove spurious unittest.DEBUGRichard van der Hoff2018-05-021-1/+0
| |
* | Fix a class of logcontext leaksRichard van der Hoff2018-05-021-11/+56
|/ | | | | | | | | | | | | | | | So, it turns out that if you have a first `Deferred` `D1`, you can add a callback which returns another `Deferred` `D2`, and `D2` must then complete before any further callbacks on `D1` will execute (and later callbacks on `D1` get the *result* of `D2` rather than `D2` itself). So, `D1` might have `called=True` (as in, it has started running its callbacks), but any new callbacks added to `D1` won't get run until `D2` completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield` will 'block'. In conclusion: some of our assumptions in `logcontext` were invalid. We need to make sure that we don't optimise out the logcontext juggling when this situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
* Merge branch 'develop' into rav/deferred_timeoutRichard van der Hoff2018-04-272-2/+3
|\
| * Make tests py3 compatibleAdrian Tschira2018-04-162-2/+3
| | | | | | | | | | | | | | | | | | | | | | This is a mixed commit that fixes various small issues * print parentheses * 01 is invalid syntax (it was octal in py2) * [x for i in 1, 2] is invalid syntax * six moves Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Use deferred.addTimeout instead of time_bound_deferredRichard van der Hoff2018-04-231-33/+0
|/ | | | This doesn't feel like a wheel we need to reinvent.
* Fix overzealous cache invalidationRichard van der Hoff2018-04-051-0/+46
| | | | | Fixes an issue where a cache invalidation would invalidate *all* pending entries, rather than just the entry that we intended to invalidate.
* Merge pull request #2804 from matrix-org/erikj/file_consumerErik Johnston2018-01-181-0/+176
|\ | | | | Add decent impl of a FileConsumer
| * Move test stuff to testsErik Johnston2018-01-181-8/+46
| |
| * Add decent impl of a FileConsumerErik Johnston2018-01-171-0/+138
| | | | | | | | | | | | | | | | | | | | Twisted core doesn't have a general purpose one, so we need to write one ourselves. Features: - All writing happens in background thread - Supports both push and pull producers - Push producers get paused if the consumer falls behind
* | Optimise LoggingContext creation and copyingRichard van der Hoff2018-01-161-8/+8
|/ | | | | | | | It turns out that the only thing we use the __dict__ of LoggingContext for is `request`, and given we create lots of LoggingContexts and then copy them every time we do a db transaction or log line, using the __dict__ seems a bit redundant. Let's try to optimise things by making the request attribute explicit.
* Fix name of test_logcontextRichard van der Hoff2017-10-171-0/+0
| | | | The file under test is logcontext.py, not log_context.py
* Add some tests for make_deferred_yieldableRichard van der Hoff2017-10-171-0/+38
|
* Fix stackoverflow and logcontexts from linearizerRichard van der Hoff2017-10-111-4/+24
| | | | | | | 1. make it not blow out the stack when there are more than 50 things waiting for a lock. Fixes https://github.com/matrix-org/synapse/issues/2505. 2. Make it not mess up the log contexts.
* Make get_state_groups_from_groups faster.Erik Johnston2017-05-171-1/+1
| | | | | | | | | Most of the time was spent copying a dict to filter out sentinel values that indicated that keys did not exist in the dict. The sentinel values were added to ensure that we cached the non-existence of keys. By updating DictionaryCache to keep track of which keys were known to not exist itself we can remove a dictionary copy.
* Fix up testsErik Johnston2017-03-302-1/+41
|
* Fix the logcontext handling in the cache wrappers (#2077)Richard van der Hoff2017-03-301-0/+91
| | | | | | | The cache wrappers had a habit of leaking the logcontext into the reactor while the lookup function was running, and then not restoring it correctly when the lookup function had completed. It's all the fault of `preserve_context_over_{fn,deferred}` which are basically a bit broken.
* Merge pull request #2052 from matrix-org/rav/time_bound_deferredRichard van der Hoff2017-03-231-0/+33
|\ | | | | Fix time_bound_deferred to throw the right exception
| * Fix time_bound_deferred to throw the right exceptionRichard van der Hoff2017-03-231-0/+33
| | | | | | | | | | | | Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred would throw a CancelledError when the deferred timed out, which was rather confusing.
* | Fix caching of remote servers' signature keysRichard van der Hoff2017-03-222-0/+100
|/ | | | | | | | | The `@cached` decorator on `KeyStore._get_server_verify_key` was missing its `num_args` parameter, which meant that it was returning the wrong key for any server which had more than one recorded key. By way of a fix, change the default for `num_args` to be *all* arguments. To implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
* Stop preserve_fn leaking context into the reactorRichard van der Hoff2017-03-181-0/+61
| | | | | | | | Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the reactor, and add a test for it. Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do the same thing but had its own, different, set of bugs.
* Tidy up testErik Johnston2017-01-171-12/+12
|
* Speed up cache size calculationErik Johnston2017-01-171-15/+15
| | | | | | | | | | | Instead of calculating the size of the cache repeatedly, which can take a long time now that it can use a callback, instead cache the size and update that on insertion and deletion. This requires changing the cache descriptors to have two caches, one for pending deferreds and the other for the actual values. There's no reason to evict from the pending deferreds as they won't take up any more memory.
* Add ExpiringCache testsErik Johnston2017-01-161-0/+84
|
* Optionally measure size of cache by sum of length of valuesErik Johnston2017-01-131-0/+25
|
* Add Limiter: limit concurrent access to resourceErik Johnston2016-11-101-0/+70
|
* Ensure invalidation list does not grow unboundedlyErik Johnston2016-08-191-0/+40
|
* Add concept of cache contextsErik Johnston2016-08-191-0/+113
|
* Add ReadWriteLockErik Johnston2016-07-051-0/+85
|
* Deduplicate joinsErik Johnston2016-04-071-0/+44
|
* Add a test for TreeCache.__contains__Mark Haines2016-02-221-0/+6
|
* Fix flake8 warnings for testsMark Haines2016-02-194-2/+2
|
* Add wheeltimer implErik Johnston2016-02-171-0/+74
|
* Fix testErik Johnston2016-02-091-9/+1
|
* Add testsErik Johnston2016-01-292-0/+19
|
* Revert all the bits changing keys of eeverything that used LRUCaches to tuplesDavid Baker2016-01-221-22/+22
|
* Make LRU cache not default to treecache & add options to use itDavid Baker2016-01-221-1/+2
|
* Test treecache directlyDavid Baker2016-01-221-0/+19
|
* Add tests for treecache directly and test del_multi at the LruCache level too.David Baker2016-01-221-0/+66
|
* Change LRUCache to be tree-based so we can delete subtrees.David Baker2016-01-211-22/+22
|
* copyrightsMatthew Hodgson2016-01-074-4/+4
|
* Add a unit test for the snapshot cacheMark Haines2015-12-231-0/+60
|
* Remove the LockManager class because it wasn't being usedMark Haines2015-11-041-108/+0
|
* Move all the caches into their own package, synapse.util.cachesErik Johnston2015-08-112-4/+2
|
* Move DictionaryCacheErik Johnston2015-08-041-1/+1
|
* Add basic dictionary cacheErik Johnston2015-08-041-0/+101
|
* Add a lru cache classMark Haines2015-02-111-0/+56
|
* Add a request-id to each log lineMark Haines2014-10-301-0/+43
|
* Have all unit tests import from our own subclass of trial's unittest ↵Paul "LeoNerd" Evans2014-09-121-2/+2
| | | | TestCase; set up logging in ONE PLACE ONLY
* fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org ↵Matthew Hodgson2014-09-032-2/+2
| | | | hasn't been incorporated in time for launch.
* add in copyrights to everything, not just the synapse subdir, and add a ↵Matthew Hodgson2014-08-132-0/+28
| | | | copyrighter.pl whilst we're at it
* Reference Matrix Home Servermatrix.org2014-08-122-0/+95