summary refs log tree commit diff
path: root/tests/util (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-141-3/+14
| | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-121-1/+7
|
* Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff2021-05-111-0/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* Merge branch 'master' into developAndrew Morgan2021-04-201-4/+2
|\
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-201-4/+2
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1421-21/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Use mock from the stdlib. (#9772)Patrick Cloke2021-04-094-6/+4
|
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-2/+4
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Record more information into structured logs. (#9654)Patrick Cloke2021-04-082-28/+14
| | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
* Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke2021-03-291-1/+3
|
* Add ResponseCache tests. (#9458)Jonathan de Jong2021-03-081-0/+131
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-164-15/+23
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Clean up caching/locking of OIDC metadata load (#9362)Richard van der Hoff2021-02-161-0/+161
| | | | Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.
* Fix chain cover update to handle events with duplicate auth events (#9210)Erik Johnston2021-01-221-0/+12
|
* Fix event chain bg update. (#9118)Erik Johnston2021-01-141-0/+8
| | | | We passed in a graph to `sorted_topologically` which didn't have an entry for each node (as we dropped nodes with no edges).
* Use a chain cover index to efficiently calculate auth chain difference (#8868)Erik Johnston2021-01-111-1/+40
|
* Empty iterables should count towards cache usage. (#9028)Patrick Cloke2021-01-061-23/+50
|
* Implement and use an @lru_cache decorator (#8595)Richard van der Hoff2020-10-301-1/+59
| | | We don't always need the full power of a DeferredCache.
* Add some more testsRichard van der Hoff2020-10-212-0/+147
|
* Push some deferred wrangling down into DeferredCacheRichard van der Hoff2020-10-212-12/+11
|
* Combine the two sets of tests for CacheDescriptorRichard van der Hoff2020-10-211-0/+230
|
* Fix 'LruCache' object has no attribute '_on_resize' (#8591)Richard van der Hoff2020-10-191-1/+7
| | | We need to make sure we are readu for the `set_cache_factor` callback.
* Add `DeferredCache.get_immediate` method (#8568)Richard van der Hoff2020-10-191-4/+23
| | | | | | | | | | | * Add `DeferredCache.get_immediate` method A bunch of things that are currently calling `DeferredCache.get` are only really interested in the result if it's completed. We can optimise and simplify this case. * Remove unused 'default' parameter to DeferredCache.get() * another get_immediate instance
* Make LruCache register its own metrics (#8561)Richard van der Hoff2020-10-161-2/+2
| | | | | rather than have everything that instantiates an LruCache manage metrics separately, have LruCache do it itself.
* Combine the two sets of DeferredCache testsRichard van der Hoff2020-10-141-2/+75
|
* move DeferredCache into its own moduleRichard van der Hoff2020-10-142-44/+64
|
* Rename Cache->DeferredCacheRichard van der Hoff2020-10-141-2/+2
|
* Stop sub-classing object (#8249)Patrick Cloke2020-09-042-12/+12
|
* Convert ReadWriteLock to async/await. (#8202)Patrick Cloke2020-08-281-2/+4
|
* Reduce run-times of tests by advancing the reactor less (#7757)Andrew Morgan2020-08-271-1/+1
|
* Remove the unused inlineCallbacks code-paths in the caching code (#8119)Patrick Cloke2020-08-191-6/+6
|
* Remove : from allowed client_secret chars (#8101)Andrew Morgan2020-08-181-3/+0
| | | | | | | Closes: https://github.com/matrix-org/synapse/issues/6766 Equivalent Sydent PR: https://github.com/matrix-org/sydent/pull/309 I believe it's now time to remove the extra allowed `:` from `client_secret` parameters.
* Convert some util functions to async (#8035)Patrick Cloke2020-08-061-33/+11
|
* Fix some spelling mistakes / typos. (#7811)Patrick Cloke2020-07-091-2/+2
|
* Fix inconsistent handling of upper and lower cases of email addresses. (#7021)Dirk Klimpel2020-07-031-0/+49
| | | fixes #7016
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-162-3/+1
|
* Add test for Linearizer.is_queued(..)Erik Johnston2020-05-271-0/+32
|
* Allow configuration of Synapse's cache without using synctl or environment ↵Amber Brown2020-05-113-8/+5
| | | | variables (#6391)
* Extend StreamChangeCache to support multiple entities per stream ID (#7303)Richard van der Hoff2020-04-221-9/+60
| | | | | | | | | | | | | | | | | | | First some background: StreamChangeCache is used to keep track of what "entities" have changed since a given stream ID. So for example, we might use it to keep track of when the last to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2]. Now, it turns out that StreamChangeCache didn't support more than one thing being changed at a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send to-device messages to more than one user at a time. As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped ok with having multiple things changing on the same stream ID, and it seems we never actually use the methods which don't work on the stream change caches where we allow multiple changes at the same stream ID. But that feels horribly fragile, hence: let's update StreamChangeCache to properly support this, and add some typing and some more tests while we're at it. [1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301 [2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-244-28/+37
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-13/+5
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* Validate client_secret parameter (#6767)Andrew Morgan2020-01-241-0/+51
|
* Log saml assertions rather than the whole responseRichard van der Hoff2020-01-161-0/+47
| | | | | | ... since the whole response is huge. We even need to break up the assertions, since kibana otherwise truncates them.
* Merge pull request #6505 from matrix-org/erikj/make_deferred_yiedableErik Johnston2019-12-101-0/+24
|\ | | | | Fix `make_deferred_yieldable` to work with coroutines
| * Fix make_deferred_yieldable to work with coroutinesErik Johnston2019-12-101-0/+24
| |
* | Remove SnapshotCache in favour of ResponseCacheErik Johnston2019-12-091-63/+0
|/
* Make ObservableDeferred.observe() always return deferred.Erik Johnston2019-10-301-2/+2
| | | | | | | This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.
* Quick fix to ensure cache descriptors always return deferredsErik Johnston2019-10-281-2/+2
|
* Add 'failure_ts' column to 'destinations' table (#6016)Richard van der Hoff2019-09-171-0/+127
| | | | Track the time that a server started failing at, for general analysis purposes.
* Retry well-known lookup before expiry.Erik Johnston2019-08-131-2/+2
| | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
* Fix some error cases in the caching layer. (#5749)Richard van der Hoff2019-07-251-3/+87
| | | | | | | There was some inconsistent behaviour in the caching layer around how exceptions were handled - particularly synchronously-thrown ones. This seems to be most easily handled by pushing the creation of ObservableDeferreds down from CacheDescriptor to the Cache.
* Replace returnValue with return (#5736)Amber Brown2019-07-231-4/+4
|
* Fixes to the federation rate limiter (#5621)Richard van der Hoff2019-07-051-0/+97
| | | | | | | - Put the default window_size back to 1000ms (broken by #5181) - Make the `rc_federation` config actually do something - fix an off-by-one error in the 'concurrent' limit - Avoid creating an unused `_PerHostRatelimiter` object for every single incoming request
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-045-47/+44
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-12/+21
|
* Run Black. (#5482)Amber Brown2019-06-202-53/+53
|
* Run Black on the tests again (#5170)Amber Brown2019-05-101-12/+11
|
* Remove periods from copyright headers (#5046)Andrew Morgan2019-04-111-1/+1
|
* Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-0/+83
|
* Fix incorrect logcontexts after a Deferred was cancelled (#4407)Richard van der Hoff2019-01-171-0/+104
|
* Include eventid in log lines when processing incoming federation ↵Richard van der Hoff2018-09-271-0/+5
| | | | | | | | | | | transactions (#3959) when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.
* Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* Run black.black2018-08-1010-76/+41
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-102-2/+2
|
* fix invalidationRichard van der Hoff2018-07-271-1/+41
|
* Rewrite cache list decoratorRichard van der Hoff2018-07-271-0/+61
| | | | | Because it was complicated and annoyed me. I suspect this will be more efficient too.
* Test and fix support for cancellation in LinearizerRichard van der Hoff2018-07-201-0/+31
|
* Combine Limiter and LinearizerRichard van der Hoff2018-07-202-70/+47
| | | | | Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.
* Add a sleep to the Limiter to fix stack overflows.Richard van der Hoff2018-07-201-4/+4
| | | | Fixes #3570
* Add unit testErik Johnston2018-07-171-0/+16
|
* Fixup unit testErik Johnston2018-07-131-3/+3
|
* run isortAmber Brown2018-07-0914-30/+38
|
* Reinstate lost run_on_reactor in unit testRichard van der Hoff2018-07-041-2/+15
| | | | | | | | a61738b removed a call to run_on_reactor from a unit test, but that call was doing something useful, in making the function in question asynchronous. Reinstate the call and add a check that we are testing what we wanted to be testing.
* Disable partial state group caching for wildcard lookupsRichard van der Hoff2018-06-221-6/+6
| | | | | | | When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-223-11/+13
|
* Remove run_on_reactor (#3395)Amber Brown2018-06-141-2/+0
|
* Port to sortedcontainers (with tests!) (#3332)Amber Brown2018-06-061-0/+198
|
* fix another dumb typoMatthew Hodgson2018-05-291-2/+2
|
* Merge branch 'develop' into rav/more_logcontext_leaksRichard van der Hoff2018-05-021-0/+38
|\
| * Fix incorrect reference to StringIORichard van der Hoff2018-05-021-0/+38
| | | | | | | | This was introduced in 4f2f5171
* | Remove spurious unittest.DEBUGRichard van der Hoff2018-05-021-1/+0
| |
* | Fix a class of logcontext leaksRichard van der Hoff2018-05-021-11/+56
|/ | | | | | | | | | | | | | | | So, it turns out that if you have a first `Deferred` `D1`, you can add a callback which returns another `Deferred` `D2`, and `D2` must then complete before any further callbacks on `D1` will execute (and later callbacks on `D1` get the *result* of `D2` rather than `D2` itself). So, `D1` might have `called=True` (as in, it has started running its callbacks), but any new callbacks added to `D1` won't get run until `D2` completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield` will 'block'. In conclusion: some of our assumptions in `logcontext` were invalid. We need to make sure that we don't optimise out the logcontext juggling when this situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
* Merge branch 'develop' into rav/deferred_timeoutRichard van der Hoff2018-04-272-2/+3
|\
| * Make tests py3 compatibleAdrian Tschira2018-04-162-2/+3
| | | | | | | | | | | | | | | | | | | | | | This is a mixed commit that fixes various small issues * print parentheses * 01 is invalid syntax (it was octal in py2) * [x for i in 1, 2] is invalid syntax * six moves Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Use deferred.addTimeout instead of time_bound_deferredRichard van der Hoff2018-04-231-33/+0
|/ | | | This doesn't feel like a wheel we need to reinvent.
* Fix overzealous cache invalidationRichard van der Hoff2018-04-051-0/+46
| | | | | Fixes an issue where a cache invalidation would invalidate *all* pending entries, rather than just the entry that we intended to invalidate.
* Merge pull request #2804 from matrix-org/erikj/file_consumerErik Johnston2018-01-181-0/+176
|\ | | | | Add decent impl of a FileConsumer
| * Move test stuff to testsErik Johnston2018-01-181-8/+46
| |
| * Add decent impl of a FileConsumerErik Johnston2018-01-171-0/+138
| | | | | | | | | | | | | | | | | | | | Twisted core doesn't have a general purpose one, so we need to write one ourselves. Features: - All writing happens in background thread - Supports both push and pull producers - Push producers get paused if the consumer falls behind
* | Optimise LoggingContext creation and copyingRichard van der Hoff2018-01-161-8/+8
|/ | | | | | | | It turns out that the only thing we use the __dict__ of LoggingContext for is `request`, and given we create lots of LoggingContexts and then copy them every time we do a db transaction or log line, using the __dict__ seems a bit redundant. Let's try to optimise things by making the request attribute explicit.
* Fix name of test_logcontextRichard van der Hoff2017-10-171-0/+0
| | | | The file under test is logcontext.py, not log_context.py
* Add some tests for make_deferred_yieldableRichard van der Hoff2017-10-171-0/+38
|
* Fix stackoverflow and logcontexts from linearizerRichard van der Hoff2017-10-111-4/+24
| | | | | | | 1. make it not blow out the stack when there are more than 50 things waiting for a lock. Fixes https://github.com/matrix-org/synapse/issues/2505. 2. Make it not mess up the log contexts.
* Make get_state_groups_from_groups faster.Erik Johnston2017-05-171-1/+1
| | | | | | | | | Most of the time was spent copying a dict to filter out sentinel values that indicated that keys did not exist in the dict. The sentinel values were added to ensure that we cached the non-existence of keys. By updating DictionaryCache to keep track of which keys were known to not exist itself we can remove a dictionary copy.
* Fix up testsErik Johnston2017-03-302-1/+41
|
* Fix the logcontext handling in the cache wrappers (#2077)Richard van der Hoff2017-03-301-0/+91
| | | | | | | The cache wrappers had a habit of leaking the logcontext into the reactor while the lookup function was running, and then not restoring it correctly when the lookup function had completed. It's all the fault of `preserve_context_over_{fn,deferred}` which are basically a bit broken.
* Merge pull request #2052 from matrix-org/rav/time_bound_deferredRichard van der Hoff2017-03-231-0/+33
|\ | | | | Fix time_bound_deferred to throw the right exception
| * Fix time_bound_deferred to throw the right exceptionRichard van der Hoff2017-03-231-0/+33
| | | | | | | | | | | | Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred would throw a CancelledError when the deferred timed out, which was rather confusing.
* | Fix caching of remote servers' signature keysRichard van der Hoff2017-03-222-0/+100
|/ | | | | | | | | The `@cached` decorator on `KeyStore._get_server_verify_key` was missing its `num_args` parameter, which meant that it was returning the wrong key for any server which had more than one recorded key. By way of a fix, change the default for `num_args` to be *all* arguments. To implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
* Stop preserve_fn leaking context into the reactorRichard van der Hoff2017-03-181-0/+61
| | | | | | | | Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the reactor, and add a test for it. Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do the same thing but had its own, different, set of bugs.
* Tidy up testErik Johnston2017-01-171-12/+12
|
* Speed up cache size calculationErik Johnston2017-01-171-15/+15
| | | | | | | | | | | Instead of calculating the size of the cache repeatedly, which can take a long time now that it can use a callback, instead cache the size and update that on insertion and deletion. This requires changing the cache descriptors to have two caches, one for pending deferreds and the other for the actual values. There's no reason to evict from the pending deferreds as they won't take up any more memory.
* Add ExpiringCache testsErik Johnston2017-01-161-0/+84
|
* Optionally measure size of cache by sum of length of valuesErik Johnston2017-01-131-0/+25
|
* Add Limiter: limit concurrent access to resourceErik Johnston2016-11-101-0/+70
|
* Ensure invalidation list does not grow unboundedlyErik Johnston2016-08-191-0/+40
|
* Add concept of cache contextsErik Johnston2016-08-191-0/+113
|
* Add ReadWriteLockErik Johnston2016-07-051-0/+85
|
* Deduplicate joinsErik Johnston2016-04-071-0/+44
|
* Add a test for TreeCache.__contains__Mark Haines2016-02-221-0/+6
|
* Fix flake8 warnings for testsMark Haines2016-02-194-2/+2
|
* Add wheeltimer implErik Johnston2016-02-171-0/+74
|
* Fix testErik Johnston2016-02-091-9/+1
|
* Add testsErik Johnston2016-01-292-0/+19
|
* Revert all the bits changing keys of eeverything that used LRUCaches to tuplesDavid Baker2016-01-221-22/+22
|
* Make LRU cache not default to treecache & add options to use itDavid Baker2016-01-221-1/+2
|
* Test treecache directlyDavid Baker2016-01-221-0/+19
|
* Add tests for treecache directly and test del_multi at the LruCache level too.David Baker2016-01-221-0/+66
|
* Change LRUCache to be tree-based so we can delete subtrees.David Baker2016-01-211-22/+22
|
* copyrightsMatthew Hodgson2016-01-074-4/+4
|
* Add a unit test for the snapshot cacheMark Haines2015-12-231-0/+60
|
* Remove the LockManager class because it wasn't being usedMark Haines2015-11-041-108/+0
|
* Move all the caches into their own package, synapse.util.cachesErik Johnston2015-08-112-4/+2
|
* Move DictionaryCacheErik Johnston2015-08-041-1/+1
|
* Add basic dictionary cacheErik Johnston2015-08-041-0/+101
|
* Add a lru cache classMark Haines2015-02-111-0/+56
|
* Add a request-id to each log lineMark Haines2014-10-301-0/+43
|
* Have all unit tests import from our own subclass of trial's unittest ↵Paul "LeoNerd" Evans2014-09-121-2/+2
| | | | TestCase; set up logging in ONE PLACE ONLY
* fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org ↵Matthew Hodgson2014-09-032-2/+2
| | | | hasn't been incorporated in time for launch.
* add in copyrights to everything, not just the synapse subdir, and add a ↵Matthew Hodgson2014-08-132-0/+28
| | | | copyrighter.pl whilst we're at it
* Reference Matrix Home Servermatrix.org2014-08-122-0/+95