summary refs log tree commit diff
path: root/synapse/util (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Reduce amount of logging at INFO level. (#6862)Andrew Morgan2020-03-231-1/+1
|\ | | | | | | | | * commit 'ed630ea17': Reduce amount of logging at INFO level. (#6862)
| * Reduce amount of logging at INFO level. (#6862)Erik Johnston2020-02-061-1/+1
| | | | | | | | | | | | | | | | A lot of the things we log at INFO are now a bit superfluous, so lets make them DEBUG logs to reduce the amount we log by default. Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>
* | Fix stacktraces when using ObservableDeferred and async/await (#6836)Andrew Morgan2020-03-231-0/+4
|\| | | | | | | | | * commit 'ae5b3104f': Fix stacktraces when using ObservableDeferred and async/await (#6836)
| * Fix stacktraces when using ObservableDeferred and async/await (#6836)Erik Johnston2020-02-031-0/+4
| |
* | Validate client_secret parameter (#6767)Andrew Morgan2020-03-231-2/+5
|\| | | | | | | | | * commit '9f7aaf90b': Validate client_secret parameter (#6767)
| * Validate client_secret parameter (#6767)Andrew Morgan2020-01-241-0/+17
| |
* | Merge pull request #6724 from matrix-org/rav/log_saml_attributesAndrew Morgan2020-03-232-17/+48
|\| | | | | | | | | | | | | * commit '59dc87c61': changelog Log saml assertions rather than the whole response move batch_iter to a separate module
| * Log saml assertions rather than the whole responseRichard van der Hoff2020-01-161-0/+13
| | | | | | | | | | | | ... since the whole response is huge. We even need to break up the assertions, since kibana otherwise truncates them.
| * move batch_iter to a separate moduleRichard van der Hoff2020-01-162-17/+35
| |
* | Merge pull request #6688 from matrix-org/rav/module_api_extensionsAndrew Morgan2020-03-231-1/+1
|\| | | | | | | | | | | | | * commit '608bf7d74': changelog Handle `config` not being set for synapse plugin modules Cleanups and additions to the module API
| * Handle `config` not being set for synapse plugin modulesRichard van der Hoff2020-01-121-1/+1
| | | | | | | | | | Some modules don't need any config, so having to define a `config` property just to keep the loader happy is a bit annoying.
* | Persist auth/state events at backwards extremities when we fetch them (#6526)Andrew Morgan2020-03-191-2/+2
|\| | | | | | | | | * commit 'bc7de8765': Persist auth/state events at backwards extremities when we fetch them (#6526)
| * Persist auth/state events at backwards extremities when we fetch them (#6526)Richard van der Hoff2019-12-161-2/+2
| | | | | | The main point here is to make sure that the state returned by _get_state_in_room has been authed before we try to use it as state in the room.
* | look up cross-signing keys from the DB in bulk (#6486)Andrew Morgan2020-03-191-1/+1
|\| | | | | | | | | * commit 'cb2db1799': look up cross-signing keys from the DB in bulk (#6486)
| * look up cross-signing keys from the DB in bulk (#6486)Hubert Chathi2019-12-121-1/+1
| |
* | Merge pull request #6506 from matrix-org/erikj/remove_snapshot_cacheAndrew Morgan2020-03-191-94/+0
|\| | | | | | | | | | | * commit 'e3f528c54': Newsfile Remove SnapshotCache in favour of ResponseCache
| * Remove SnapshotCache in favour of ResponseCacheErik Johnston2019-12-091-94/+0
| |
* | Fix inaccurate per-block metrics (#6491)Andrew Morgan2020-03-191-42/+18
|\| | | | | | | | | * commit '18660a34d': Fix inaccurate per-block metrics (#6491)
| * Fix inaccurate per-block metrics (#6491)Richard van der Hoff2019-12-091-42/+18
| | | | | | | | | | `Measure` incorrectly assumed that it was the only thing being done by the parent `LoggingContext`. For instance, during a "renew group attestations" operation, hundreds of `outbound_request` calls could take place in parallel, all using the same `LoggingContext`. This would mean that any resources used during *any* of those calls would be reported against *all* of them, producing wildly inaccurate results. Instead, we now give each `Measure` block its own `LoggingContext` (using the parent `LoggingContext` mechanism to ensure that the log lines look correct and that the metrics are ultimately propogated to the top level for reporting against requests/backgrond tasks).
* | Merge pull request #6484 from matrix-org/erikj/port_sync_handlerAndrew Morgan2020-03-191-6/+17
|\| | | | | | | | | | | | | * commit 'a9b393340': Fixup functions to consistently return deferreds Newsfile Port SyncHandler to async/await
| * Port SyncHandler to async/awaitErik Johnston2019-12-051-6/+17
| |
* | Replace instance variations of homeserver with correct case/spacing (#6357)Andrew Morgan2020-03-181-1/+1
|\| | | | | | | | | | | | | | | * commit '6e1b40dc2': sample config Add changelog A couple more instances Replace instance variations of homeserver with correct case/spacing
| * Replace instance variations of homeserver with correct case/spacingAndrew Morgan2019-11-121-1/+1
| |
* | Fix LruCache callback deduplication (#6213)Andrew Morgan2020-03-181-11/+37
|\| | | | | | | | | * commit 'affcc2cc3': Fix LruCache callback deduplication (#6213)
| * Fix LruCache callback deduplication (#6213)V024602019-11-071-11/+37
| |
* | Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2020-03-164-6/+6
|\|
| * Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-314-6/+6
| | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* | Merge pull request #6291 from matrix-org/erikj/fix_cache_descriptorAndrew Morgan2020-03-161-4/+5
|\|
| * Clarify docstringErik Johnston2019-10-301-0/+4
| |
| * Make ObservableDeferred.observe() always return deferred.Erik Johnston2019-10-301-5/+2
| | | | | | | | | | | | | | This makes it easier to use in an async/await world. Also fixes a bug where cache descriptors would occaisonally return a raw value rather than a deferred.
* | Handle FileNotFound error in checking git repository version (#6284)Andrew Morgan2020-03-161-4/+6
|\| | | | | | | | | * commit 'b39ca49db': Handle FileNotFound error in checking git repository version (#6284)
| * Handle FileNotFound error in checking git repository version (#6284)Andrew Morgan2019-10-301-4/+6
| |
* | Merge pull request #6279 from matrix-org/erikj/federation_server_async_awaitAndrew Morgan2020-03-161-4/+3
|\| | | | | | | | | | | | | | | * commit 'b4465564c': Don't return coroutines Make concurrently_execute work with async/await Newsfile Port federation_server to async/await
| * Make concurrently_execute work with async/awaitErik Johnston2019-10-291-4/+3
| |
* | Merge pull request #6263 from matrix-org/erikj/caches_return_deferredsAndrew Morgan2020-03-161-5/+4
|\| | | | | | | | | | | | | * commit '561133c3c': Update docstring Newsfile Quick fix to ensure cache descriptors always return deferreds
| * Update docstringErik Johnston2019-10-291-3/+2
| |
| * Quick fix to ensure cache descriptors always return deferredsErik Johnston2019-10-281-2/+2
| |
* | Merge branch 'master' into developAndrew Morgan2020-02-261-0/+29
|\|
| * Add maybe_awaitable and fix __init__ bugsErik Johnston2019-10-111-0/+29
| |
* | Merge pull request #6127 from matrix-org/erikj/patch_innerAndrew Morgan2020-02-261-0/+219
|\|
| * Fixup commentsErik Johnston2019-10-101-3/+3
| | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| * Fix py3.5Erik Johnston2019-10-101-1/+1
| |
| * Fix py3.5Erik Johnston2019-10-101-1/+1
| |
| * sortErik Johnston2019-10-101-3/+1
| |
| * Appease mypyErik Johnston2019-10-101-13/+18
| |
| * Add comentsErik Johnston2019-10-101-5/+25
| |
| * Log correct contextErik Johnston2019-10-101-6/+6
| |
| * Test for sentinel commitErik Johnston2019-10-101-2/+19
| |
| * Move patch_inline_callbacks into synapse/Erik Johnston2019-10-101-0/+179
| |
* | add some metrics on the federation sender (#6160)Andrew Morgan2020-02-261-2/+4
|\|
| * add some metrics on the federation sender (#6160)Richard van der Hoff2019-10-031-2/+4
| |
* | Fix up some typechecking (#6150)Andrew Morgan2020-02-265-8/+33
|\|
| * Fix up some typechecking (#6150)Amber Brown2019-10-025-8/+33
| | | | | | | | | | | | * type checking fixes * changelog
* | Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timingsAndrew Morgan2020-02-261-1/+1
|\|
| * Fix errors storing large retry intervals.Erik Johnston2019-10-021-1/+1
| | | | | | | | | | | | | | | | | | We have set the max retry interval to a value larger than a postgres or sqlite int can hold, which caused exceptions when updating the destinations table. To fix postgres we need to change the column to a bigint, and for sqlite we lower the max interval to 2**62 (which is still incredibly long).
* | Merge pull request #6069 from matrix-org/rav/fix_attribute_mappingAndrew Morgan2020-02-251-1/+19
|\|
| * Merge branch 'develop' into rav/fix_attribute_mappingRichard van der Hoff2019-09-193-21/+68
| |\
| * | Fix a bug with saml attribute maps.Richard van der Hoff2019-09-191-1/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes a bug where the default attribute maps were prioritised over user-specified ones, resulting in incorrect mappings. The problem is that if you call SPConfig.load() multiple times, it adds new attribute mappers to a list. So by calling it with the default config first, and then the user-specified config, we would always get the default mappers before the user-specified mappers. To solve this, let's merge the config dicts first, and then pass them to SPConfig.
* | | Add 'failure_ts' column to 'destinations' table (#6016)Andrew Morgan2020-02-251-1/+15
|\ \ \ | | |/ | |/|
| * | Add 'failure_ts' column to 'destinations' table (#6016)Richard van der Hoff2019-09-171-1/+15
| | | | | | | | | | | | Track the time that a server started failing at, for general analysis purposes.
* | | Remove the cap on federation retry interval. (#6026)Andrew Morgan2020-02-251-2/+2
|\| |
| * | Remove the cap on federation retry interval. (#6026)Richard van der Hoff2019-09-121-2/+2
| | | | | | | | | | | | | | | | | | Essentially the intention here is to end up blacklisting servers which never respond to federation requests. Fixes https://github.com/matrix-org/synapse/issues/5113.
* | | Fix bug in calculating the federation retry backoff period (#6025)Andrew Morgan2020-02-251-2/+3
|\| |
| * | Fix bug in calculating the federation retry backoff period (#6025)Richard van der Hoff2019-09-121-2/+3
| | | | | | | | | | | | This was intended to introduce an element of jitter; instead it gave you a 30/60 chance of resetting to zero.
* | | Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976)Andrew Morgan2020-02-251-0/+33
|\| |
| * | Use the v2 Identity Service API for lookups (MSC2134 + MSC2140) (#5976)Andrew Morgan2019-09-111-0/+33
| | | | | | | | | | | | | | | | | | | | | This is a redo of https://github.com/matrix-org/synapse/pull/5897 but with `id_access_token` accepted. Implements [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134) plus Identity Service v2 authentication ala [MSC2140](https://github.com/matrix-org/matrix-doc/pull/2140). Identity lookup-related functions were also moved from `RoomMemberHandler` to `IdentityHandler`.
* | | Clean up some code in the retry logic (#6017)Andrew Morgan2020-02-251-16/+13
|\| |
| * | Clean up some code in the retry logic (#6017)Richard van der Hoff2019-09-111-16/+13
| | | | | | | | | | | | * remove some unused code * make things which were constants into constants for efficiency and clarity
| * | Revert "Use the v2 lookup API for 3PID invites (#5897)" (#5937)Andrew Morgan2019-08-301-33/+0
| | | | | | | | | | | | | | | This reverts commit 71fc04069a5770a204c3514e0237d7374df257a8. This broke 3PID invites as #5892 was required for it to work correctly.
* | | Revert "Use the v2 lookup API for 3PID invites (#5897)"Andrew Morgan2020-02-241-33/+0
| | | | | | | | | | | | | | | This reverts commit 978f263e7c5d1eb440efaf07abc5009408ade25d, reversing changes made to 4f6ee99818d9c338944a10585d0aea4c7349d456.
* | | Use the v2 lookup API for 3PID invites (#5897)Andrew Morgan2020-02-241-0/+33
|\| |
| * | Use the v2 lookup API for 3PID invites (#5897)Andrew Morgan2019-08-281-0/+33
| | | | | | | | | | | | | | | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/5861 Adds support for the v2 lookup API as defined in [MSC2134](https://github.com/matrix-org/matrix-doc/pull/2134). Currently this is only used for 3PID invites. Sytest PR: https://github.com/matrix-org/sytest/pull/679
* | | Merge pull request #5844 from matrix-org/erikj/retry_well_known_lookupAndrew Morgan2020-02-201-3/+5
|\| |
| * | Retry well-known lookup before expiry.Erik Johnston2019-08-131-3/+5
| |/ | | | | | | | | | | | | | | | | This gives a bit of a grace period where we can attempt to refetch a remote `well-known`, while still using the cached result if that fails. Hopefully this will make the well-known resolution a bit more torelant of failures, rather than it immediately treating failures as "no result" and caching that for an hour.
| * Add kwargs and docBrendan Abolivier2019-07-291-2/+4
| |
| * Add ability to pass arguments to looping callsBrendan Abolivier2019-07-291-2/+2
| |
* | Fix some error cases in the caching layer. (#5749)Andrew Morgan2020-02-191-32/+42
|\|
| * Fix some error cases in the caching layer. (#5749)Richard van der Hoff2019-07-251-32/+42
| | | | | | | | | | | | | | There was some inconsistent behaviour in the caching layer around how exceptions were handled - particularly synchronously-thrown ones. This seems to be most easily handled by pushing the creation of ObservableDeferreds down from CacheDescriptor to the Cache.
* | Add a prometheus metric for active cache lookups. (#5750)Andrew Morgan2020-02-192-2/+33
|\|
| * Add a prometheus metric for active cache lookups. (#5750)Richard van der Hoff2019-07-242-2/+33
| | | | | | | | | | | | * Add a prometheus metric for active cache lookups. * changelog
* | Replace returnValue with return (#5736)Andrew Morgan2020-02-196-15/+13
|\|
| * Replace returnValue with return (#5736)Amber Brown2019-07-236-15/+13
| |
* | Cache get_version_string.Andrew Morgan2020-02-191-2/+21
|\|
| * Cache get_version_string.Erik Johnston2019-07-221-2/+21
| | | | | | | | | | | | | | | | | | The version of a module isn't going to change over the lifetime of the process (assuming no funky hot reloading is going on, which it isn't), so let's just cache the result to avoid spawning lots of git subprocesses. Fixes #5672.
* | Fixes to the federation rate limiter (#5621)Andrew Morgan2020-02-171-8/+8
|\|
| * Fixes to the federation rate limiter (#5621)Richard van der Hoff2019-07-051-8/+8
| | | | | | | | | | | | | | - Put the default window_size back to 1000ms (broken by #5181) - Make the `rc_federation` config actually do something - fix an off-by-one error in the 'concurrent' limit - Avoid creating an unused `_PerHostRatelimiter` object for every single incoming request
* | Improve the backwards compatibility re-exports of synapse.logging.context ↵Andrew Morgan2020-02-173-5/+61
|\| | | | | | | (#5617)
| * Improve the backwards compatibility re-exports of synapse.logging.context ↵Amber Brown2019-07-053-5/+61
| | | | | | | | | | | | | | | | (#5617) * Improve the backwards compatibility re-exports of synapse.logging.context. * reexport logformatter too
* | Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Andrew Morgan2020-02-1412-962/+26
|\|
| * Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-0412-962/+26
| |
* | Fix 'utime went backwards' errors on daemonization. (#5609)Andrew Morgan2020-02-141-4/+13
|\|
| * Fix 'utime went backwards' errors on daemonization. (#5609)Richard van der Hoff2019-07-031-4/+13
| | | | | | | | | | | | | | | | * Fix 'utime went backwards' errors on daemonization. Fixes #5608 * remove spurious debug
* | Fix a number of "Starting txn from sentinel context" warnings (#5605)Andrew Morgan2020-02-141-1/+7
|\|
| * Fix a number of "Starting txn from sentinel context" warnings (#5605)Richard van der Hoff2019-07-031-1/+7
| | | | | | | | Fixes #5602, #5603
* | Fix media repo breaking (#5593)Andrew Morgan2020-02-141-2/+7
|\|
| * Fix media repo breaking (#5593)Amber Brown2019-07-021-2/+7
| |
* | Prevent multiple upgrades on the same room at once (#5051)Andrew Morgan2020-02-141-1/+1
|\|
| * Prevent multiple upgrades on the same room at once (#5051)Andrew Morgan2019-06-251-1/+1
| | | | | | | | | | | | | | Closes #4583 Does slightly less than #5045, which prevented a room from being upgraded multiple times, one after another. This PR still allows that, but just prevents two from happening at the same time. Mostly just to mitigate the fact that servers are slow and it can take a moment for the room upgrade to actually complete. We don't want people sending another request to upgrade the room when really they just thought the first didn't go through.
* | Merge pull request #5499 from matrix-org/rav/cleanup_metricsAndrew Morgan2020-02-141-7/+39
|\|
| * Avoid raising exceptions in metricsRichard van der Hoff2019-06-241-8/+14
| | | | | | | | | | Sentry will catch the errors if they happen, so that should be good enough, and woun't make things explode if we hit the error condition.
| * Merge branch 'develop' into rav/cleanup_metricsRichard van der Hoff2019-06-2427-314/+317
| |\
| * | Sanity-checking for metrics updatesRichard van der Hoff2019-06-191-7/+33
| | | | | | | | | | | | Check that our clocks go forward.
* | | Run Black. (#5482)Andrew Morgan2020-02-133-4/+4
|\ \ \ | | |/ | |/|
| * | Run Black. (#5482)Amber Brown2019-06-2027-314/+317
| |/
* | Dinsic Blacking with black==18.6b2Andrew Morgan2020-02-1127-317/+321
| |
* | Validate client_secret parameter according to spec (#20)Andrew Morgan2020-01-221-0/+14
| |
* | Add kwargs and docBrendan Abolivier2019-07-291-2/+4
| |
* | Add ability to pass arguments to looping callsBrendan Abolivier2019-07-291-2/+2
| |
* | Merge branch 'master' into dinsicBrendan Abolivier2019-06-122-27/+50
|\|
| * Call RetryLimiter correctly (#5340)Richard van der Hoff2019-06-041-1/+6
| | | | | | Fixes a regression introduced in #5335.
| * Avoid rapidly backing-off a server if we ignore the retry intervalRichard van der Hoff2019-06-031-23/+37
| |
| * Improve logging for logcontext leaks. (#5288)Richard van der Hoff2019-05-291-9/+13
| |
* | Merge branch 'babolivier/per_room_profiles' into dinsic dinsic_2019-05-17Brendan Abolivier2019-05-173-32/+35
|\|
| * Make all the rate limiting options more consistent (#5181)Amber Brown2019-05-151-32/+15
| |
| * Merge pull request #5183 from matrix-org/erikj/async_serialize_eventErik Johnston2019-05-151-0/+19
| |\ | | | | | | Allow client event serialization to be async
| | * Update docstring with correct return typeErik Johnston2019-05-151-1/+1
| | | | | | | | | Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
| | * Allow client event serialization to be asyncErik Johnston2019-05-141-0/+19
| | |
| * | comment about user_joined_roomRichard van der Hoff2019-05-141-0/+1
| |/
* | Merge branch 'release-v0.99.4' into dinsic dinsic_2019-05-14Brendan Abolivier2019-05-141-2/+7
|\|
| * Merge branch 'master' into developRichard van der Hoff2019-05-031-2/+7
| |\
| | * Use SystemRandom for token generationRichard van der Hoff2019-05-031-2/+7
| | |
* | | Merge branch 'develop' into dinsicAndrew Morgan2019-04-152-3/+58
|\| |
| * | Remove periods from copyright headers (#5046)Andrew Morgan2019-04-111-1/+1
| | |
| * | Fix disappearing exceptions in manhole. (#5035)Richard van der Hoff2019-04-101-2/+57
| |/ | | | | Avoid sending syntax errors from the manhole to sentry.
* | Prevent exception when hs or shadow_hs missing (#5012) dinsic_2019-04-04_2Michael Kaye2019-04-041-2/+2
| | | | | | | | * Prevent exception when hs or shadow_hs missing
* | FixupsErik Johnston2019-03-151-1/+4
| |
* | Merge branch 'master' of github.com:matrix-org/synapse into ↵Erik Johnston2019-03-154-8/+211
|\| | | | | | | erikj/dinsic-merged-master
| * Add a caching layer to .well-known responses (#4516)Richard van der Hoff2019-01-301-0/+161
| |
| * Merge pull request #4486 from xperimental/workaround-4216Richard van der Hoff2019-01-301-1/+4
| |\ | | | | | | Implement workaround for login error.
| | * Implement workaround for login error.Robert Jacob2019-01-301-1/+4
| | | | | | | | | | | | Signed-off-by: Robert Jacob <xperimental@solidproject.de>
| * | Make linearizer more quiet (#4507)Amber Brown2019-01-291-5/+5
| |/
| * Fix incorrect logcontexts after a Deferred was cancelled (#4407)Richard van der Hoff2019-01-171-1/+3
| |
| * Fix UnicodeDecodeError when postgres is not configured in english (#4253)Richard van der Hoff2018-12-041-1/+38
| | | | | | | | This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.
* | Merge branch 'dinsic_anoa/info_split' of github.com:matrix-org/synapse into ↵Andrew Morgan2019-02-271-1/+1
|\ \ | | | | | | | | | dinsic_anoa/info_split
| * | Update synapse/util/threepids.pyRichard van der Hoff2019-02-251-1/+1
| | | | | | | | | Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>
* | | Check shadow_hs as well as hs during 3pid regAndrew Morgan2019-02-271-6/+8
|/ /
* | Use internal-info for identity server. Block reg on fields dinsic_2019-02-19-beta11Andrew Morgan2019-02-191-2/+10
| |
* | merge master into dinsic, again...Matthew Hodgson2018-11-039-120/+306
|\|
| * Merge branch 'develop' of github.com:matrix-org/synapse into ↵Erik Johnston2018-10-253-54/+76
| |\ | | | | | | | | | erikj/alias_disallow_list
| | * Correctly account for cpu usage by background threads (#4074)Richard van der Hoff2018-10-231-51/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064
| | * Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase ↵Amber Brown2018-10-201-1/+3
| | | | | | | | | | | | on py3) (#4068)
| | * Fix manhole on py3 (pt 2) (#4067)Amber Brown2018-10-191-0/+2
| | |
| | * make a bytestringAmber Brown2018-10-191-2/+2
| | |
| * | Anchor returned regex to start and end of stringErik Johnston2018-10-191-2/+6
| | |
| * | Add config option to control alias creationErik Johnston2018-10-191-0/+21
| |/
| * Remove unnecessary extra function call layerErik Johnston2018-10-081-16/+13
| |
| * Use errback pattern and catch async failuresErik Johnston2018-10-081-14/+29
| |
| * Log looping call exceptionsErik Johnston2018-10-051-1/+18
| | | | | | | | | | | | | | | | If a looping call function errors, then it kills the loop entirely. Currently it throws away the exception logs, so we should make it actually log them. Fixes #3929
| * Correctly match 'dict.pop' apiErik Johnston2018-10-011-3/+11
| |
| * Don't update eviction metrics on explicit removalErik Johnston2018-10-011-5/+0
| |
| * Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cacheRichard van der Hoff2018-09-281-4/+37
| |\
| | * Include eventid in log lines when processing incoming federation ↵Richard van der Hoff2018-09-271-4/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | transactions (#3959) when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.
| * | Merge branch 'rav/fix_expiring_cache_len' into erikj/destination_retry_cacheRichard van der Hoff2018-09-261-10/+17
| |\|
| | * Log which cache is throwing exceptionsRichard van der Hoff2018-09-261-10/+17
| | |
| | * Fix ExpiringCache.__len__ to be accurateErik Johnston2018-09-261-10/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch. (This appears to have been a longstanding bug, but one which has been made more of a problem by #3932 and #3933). (This was originally done by Erik as part of #3933. I'm cherry-picking it because really it's a fix in its own right)
| * | Fix ExpiringCache.__len__ to be accurateErik Johnston2018-09-211-12/+9
| | | | | | | | | | | | | | | | | | It used to try and produce an estimate, which was sometimes negative. This caused metrics to be sad, so lets always just calculate it from scratch.
| * | Add a five minute cache to get_destination_retry_timingsErik Johnston2018-09-211-0/+13
| | | | | | | | | | | | Hopefully helps with #3931
| * | Make ExpiringCache slightly more performantErik Johnston2018-09-211-1/+5
| |/
| * Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | | | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
| * Improve the logging when handling a federation transaction (#3904)Richard van der Hoff2018-09-191-1/+1
| | | | | | | | | | | | | | | | | | | | Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.
| * Replace custom DeferredTimeoutError with defer.TimeoutErrorErik Johnston2018-09-191-9/+3
| |
| * Run canceller first to allow it to generate correct errorErik Johnston2018-09-191-2/+5
| |
| * Update to use new timeout function everywhere.Erik Johnston2018-09-191-54/+19
| | | | | | | | | | | | | | The existing deferred timeout helper function (and the one into twisted) suffer from a bug when a deferred's canceller throws an exception, #3842. The new helper function doesn't suffer from this problem.
| * Fix timeout functionErik Johnston2018-09-151-1/+2
| | | | | | | | | | Turns out deferred.cancel sometimes throws, so we do that last to ensure that we always do resolve the new deferred.
| * Add an awful secondary timeout to fix wedged requestsErik Johnston2018-09-141-0/+51
| | | | | | | | This is an attempt to mitigate #3842 by adding yet-another-timeout
| * Add in flight real time metrics for Measure blocksErik Johnston2018-09-141-0/+22
| |
| * Change the manhole SSH key to have more bitsErik Johnston2018-09-111-13/+31
| | | | | | | | | | Newer versions of openssh client refuse to connect to the old key due to its length.
* | merge in masterMatthew Hodgson2018-10-2212-219/+249
|\|
| * Fix exceptions when a connection is closed before we read the headersRichard van der Hoff2018-08-201-1/+3
| | | | | | | | | | This fixes bugs introduced in #3700, by making sure that we behave sanely when an incoming connection is closed before the headers are read.
| * Robustness fix for logcontext filterRichard van der Hoff2018-08-201-1/+7
| | | | | | | | | | Make the logcontext filter not explode if it somehow ends up with a logcontext of None, since that infinite-loops the whole logging system.
| * Port over enough to get some sytests running on Python 3 (#3668)Amber Brown2018-08-203-8/+29
| |
| * Merge branch 'rav/fix_linearizer_cancellation' into developRichard van der Hoff2018-08-101-43/+68
| |\
| | * Fix linearizer cancellation on twisted < 18.7Richard van der Hoff2018-08-101-43/+68
| | | | | | | | | | | | | | | | | | Turns out that cancellation of inlineDeferreds didn't really work properly until Twisted 18.7. This commit refactors Linearizer.queue to avoid inlineCallbacks.
| * | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-105-4/+4
| |/
| * Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-021-3/+3
| |
| * fix invalidationRichard van der Hoff2018-07-271-1/+1
| |
| * Rewrite cache list decoratorRichard van der Hoff2018-07-271-67/+64
| | | | | | | | | | Because it was complicated and annoyed me. I suspect this will be more efficient too.
| * Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-261-1/+1
| | | | | | | | | | | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
| * Test and fix support for cancellation in LinearizerRichard van der Hoff2018-07-201-6/+22
| |
| * Combine Limiter and LinearizerRichard van der Hoff2018-07-201-89/+10
| | | | | | | | | | Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.
| * Improvements to the LimiterRichard van der Hoff2018-07-201-13/+20
| | | | | | | | | | * give them names, to improve logging * use a deque rather than a list for efficiency
| * Add a sleep to the Limiter to fix stack overflows.Richard van der Hoff2018-07-201-3/+20
| | | | | | | | Fixes #3570
| * Don't spew errors because we can't save metrics (#3563)Amber Brown2018-07-192-6/+24
| |
| * Make Distributor run its processes as a background processRichard van der Hoff2018-07-181-26/+18
| | | | | | | | | | | | | | | | | | | | | | This is more involved than it might otherwise be, because the current implementation just drops its logcontexts and runs everything in the sentinel context. It turns out that we aren't actually using a bunch of the functionality here (notably suppress_failures and the fact that Distributor.fire returns a deferred), so the easiest way to fix this is actually by simplifying a bunch of code.
| * Run things as background processesRichard van der Hoff2018-07-182-1/+9
| | | | | | | | | | | | | | | | This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).
* | Merge remote-tracking branch 'origin/master' into dinsicDavid Baker2018-07-1922-141/+241
|\|
| * Use efficient .intersectionErik Johnston2018-07-171-4/+1
| |
| * Fix perf regression in PR #3530Erik Johnston2018-07-171-1/+6
| | | | | | | | | | | | | | | | The get_entities_changed function was changed to return all changed entities since the given stream position, rather than only those changed from a given list of entities. This resulted in the function incorrectly returning large numbers of entities that, for example, caused large increases in database usage.
| * Merge pull request #3530 from matrix-org/erikj/stream_cacheAmber Brown2018-07-171-8/+1
| |\ | | | | | | Don't return unknown entities in get_entities_changed
| | * Don't return unknown entities in get_entities_changedErik Johnston2018-07-131-8/+1
| | | | | | | | | | | | | | | | | | | | | | | | The stream cache keeps track of all entities that have changed since a particular stream position, so get_entities_changed does not need to return unknown entites when given a larger stream position. This makes it consistent with the behaviour of has_entity_changed.
| * | Make FederationRateLimiter queue requests properlyRichard van der Hoff2018-07-131-10/+23
| |/ | | | | | | | | | | | | | | popitem removes the *most recent* item by default [1]. We want the oldest. Fixes #3524 [1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem
| * Reduce set building in get_entities_changedRichard van der Hoff2018-07-121-8/+12
| | | | | | | | | | | | | | | | | | | | | | This line shows up as about 5% of cpu time on a synchrotron: not_known_entities = set(entities) - set(self._entity_to_key) Presumably the problem here is that _entity_to_key can be largeish, and building a set for its keys every time this function is called is slow. Here we rewrite the logic to avoid building so many sets.
| * Attempt to include db threads in cpu usage stats (#3496)Richard van der Hoff2018-07-101-2/+21
| | | | | | | | | | Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.
| * Refactor logcontext resource usage tracking (#3501)Richard van der Hoff2018-07-102-49/+120
| | | | | | | | | | Factor out the resource usage tracking out to a separate object, which can be passed around and copied independently of the logcontext itself.
| * run isortAmber Brown2018-07-0922-73/+71
| |
* | Merge remote-tracking branch 'origin/master' into dinsicDavid Baker2018-07-1710-82/+118
|\|
| * Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-1/+1
| |
| * Revert "Revert "Try to not use as much CPU in the StreamChangeCache"" (#3454)Amber Brown2018-06-281-2/+4
| |
| * Revert "Try to not use as much CPU in the StreamChangeCache"Matthew Hodgson2018-06-261-4/+2
| |
| * fixesAmber Brown2018-06-261-2/+2
| |
| * fixesAmber Brown2018-06-261-2/+2
| |
| * try and make loading items from the cache fasterAmber Brown2018-06-261-2/+4
| |
| * Remove all global reactor imports & pass it around explicitly (#3424)Amber Brown2018-06-251-0/+3
| |
| * Disable partial state group caching for wildcard lookupsRichard van der Hoff2018-06-221-13/+12
| | | | | | | | | | | | | | When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.
| * Merge pull request #3419 from matrix-org/rav/events_per_requestRichard van der Hoff2018-06-221-0/+15
| |\ | | | | | | Log number of events fetched from DB
| | * Indirect evt_count updates via method callRichard van der Hoff2018-06-221-0/+11
| | | | | | | | | | | | so that we can stub it for the sentinel and not have a billion failing UTs
| | * Log number of events fetched from DBRichard van der Hoff2018-06-211-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]
| * | Pass around the reactor explicitly (#3385)Amber Brown2018-06-224-33/+43
| |/
| * Remove run_on_reactor (#3395)Amber Brown2018-06-141-9/+1
| |
| * Port to sortedcontainers (with tests!) (#3332)Amber Brown2018-06-061-26/+31
| |
| * Add hacky cache factor override systemErik Johnston2018-06-042-2/+12
| |
* | Merge tag 'v0.31.2' into dinsicMichael Kaye2018-06-2220-242/+347
|\|
| * Consistently use six's iteritems and wrap lazy keys/values in list() if ↵Amber Brown2018-05-312-3/+5
| | | | | | | | they're not meant to be lazy (#3307)
| * Merge pull request #3281 from NotAFile/py3-six-isinstanceAmber Brown2018-05-302-11/+15
| |\ | | | | | | remaining isintance fixes
| | * pep8Adrian Tschira2018-05-291-0/+1
| | |
| | * fix recursion errorAdrian Tschira2018-05-241-7/+5
| | |
| | * remaining isintance fixesAdrian Tschira2018-05-242-6/+11
| | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | fix up testsAmber Brown2018-05-281-3/+3
| | |
| * | update to more consistently use seconds in any metrics or loggingAmber Brown2018-05-283-19/+19
| | |
| * | add comment about why unregAmber Brown2018-05-281-0/+2
| | |
| * | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-282-1/+24
| |\|
| | * Merge pull request #3247 from NotAFile/py3-miscAmber Brown2018-05-241-1/+6
| | |\ | | | | | | | | Misc Python3 fixes
| | | * fix py3 intern and remove unnecessary py3 encodeAdrian Tschira2018-05-191-1/+6
| | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| | * | Merge pull request #3245 from NotAFile/batch-iterAmber Brown2018-05-241-0/+18
| | |\ \ | | | | | | | | | | Add batch_iter to utils
| | | * | Add batch_iter to utilsAdrian Tschira2018-05-191-0/+18
| | | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There's a frequent idiom I noticed where an iterable is split up into a number of chunks/batches. Unfortunately that method does not work with iterators like dict.keys() in python3. This implementation works with iterators. Signed-off-by: Adrian Tschira <nota@notafile.com>
| * | | cleanupAmber Brown2018-05-221-5/+10
| | | |
| * | | cleanup pep8 errorsAmber Brown2018-05-221-2/+5
| | | |
| * | | fixesAmber Brown2018-05-222-12/+30
| | | |
| * | | Merge remote-tracking branch 'origin/develop' into 3218-official-promAmber Brown2018-05-221-11/+27
| |\| |
| | * | CommentErik Johnston2018-05-221-1/+1
| | | |
| | * | Fix logcontext resource usage trackingErik Johnston2018-05-221-11/+27
| | |/
| * / replacing portionsAmber Brown2018-05-217-98/+71
| |/
| * Merge remote-tracking branch 'origin/develop' into rav/warn_on_logcontext_failRichard van der Hoff2018-05-0315-136/+342
| |\
| | * Fix logcontext leaks in rate limiterRichard van der Hoff2018-05-031-3/+14
| | |
| | * Merge branch 'develop' into rav/more_logcontext_leaksRichard van der Hoff2018-05-021-1/+1
| | |\
| | | * Fix incorrect reference to StringIORichard van der Hoff2018-05-021-1/+1
| | | | | | | | | | | | | | | | This was introduced in 4f2f5171
| | * | Fix a class of logcontext leaksRichard van der Hoff2018-05-021-22/+38
| | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | So, it turns out that if you have a first `Deferred` `D1`, you can add a callback which returns another `Deferred` `D2`, and `D2` must then complete before any further callbacks on `D1` will execute (and later callbacks on `D1` get the *result* of `D2` rather than `D2` itself). So, `D1` might have `called=True` (as in, it has started running its callbacks), but any new callbacks added to `D1` won't get run until `D2` completes - so if you `yield D1` in an `inlineCallbacks` function, your `yield` will 'block'. In conclusion: some of our assumptions in `logcontext` were invalid. We need to make sure that we don't optimise out the logcontext juggling when this situation happens. Fortunately, it is easy to detect by checking `D1.paused`.
| | * Merge pull request #3144 from ↵Richard van der Hoff2018-04-301-1/+7
| | |\ | | | | | | | | | | | | | | | | matrix-org/rav/run_in_background_exception_handling Trap exceptions thrown within run_in_background
| | | * Trap exceptions thrown within run_in_backgroundRichard van der Hoff2018-04-271-1/+7
| | | | | | | | | | | | | | | | | | | | Turn any exceptions that get thrown synchronously within run_in_background into Failures instead.
| | * | Merge branch 'develop' into py3-xrange-1Richard van der Hoff2018-04-307-12/+17
| | |\ \
| | | * \ Merge pull request #3154 from NotAFile/py3-stringioRichard van der Hoff2018-04-301-1/+1
| | | |\ \ | | | | | | | | | | | | Replace stringIO imports with six
| | | | * | replace stringIO importsAdrian Tschira2018-04-281-1/+1
| | | | | |
| | | * | | Merge pull request #3155 from NotAFile/py3-bytes-1Richard van der Hoff2018-04-301-2/+5
| | | |\ \ \ | | | | | | | | | | | | | | more bytes strings
| | | | * | | more bytes stringsAdrian Tschira2018-04-291-2/+5
| | | | |/ / | | | | | | | | | | | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
| | | * | | Merge pull request #3140 from matrix-org/rav/use_run_in_backgroundRichard van der Hoff2018-04-305-9/+11
| | | |\ \ \ | | | | |/ / | | | |/| | Use run_in_background in preference to preserve_fn
| | | | * | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-271-1/+6
| | | | |\ \
| | | | * | | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-275-9/+11
| | | | | |/ | | | | |/| | | | | | | | | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
| | * | / | Move more xrange to sixAdrian Tschira2018-04-283-5/+10
| | |/ / / | | | | | | | | | | | | | | | | | | | | | | | | | plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>
| | * | | Merge remote-tracking branch 'origin/develop' into rav/deferred_timeoutRichard van der Hoff2018-04-271-1/+6
| | |\ \ \ | | | | |/ | | | |/|
| | | * | Improve exception handling for background processesRichard van der Hoff2018-04-271-1/+6
| | | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
generated by cgit-magenta 1.5.1 (git 2.48.1) at 2025-08-08 03:23:21 +0000