summary refs log tree commit diff
path: root/synapse/logging (follow)
Commit message (Collapse)AuthorAgeFilesLines
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-1/+1
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-154-21/+21
|
* Expose opentracing trace id in response headers (#10199)Richard van der Hoff2021-06-181-0/+21
| | | Fixes: #9480
* Improve comments in the structured logging code. (#10188)Patrick Cloke2021-06-161-4/+5
|
* Make opentracing trace into event persistence (#10134)Richard van der Hoff2021-06-161-2/+55
| | | | | | | | | | | | | | * Trace event persistence When we persist a batch of events, set the parent opentracing span to the that from the request, so that we can trace all the way in. * changelog * When we force tracing, set a baggage item ... so that we can check again later. * Link in both directions between persist_events spans
* Clean up the interface for injecting opentracing over HTTP (#10143)Richard van der Hoff2021-06-091-83/+19
| | | | | | | * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog
* opentracing: use a consistent name for background processes (#10135)Richard van der Hoff2021-06-071-0/+1
| | | | ... otherwise we tend to get a namespace clash between the bg process and the functions that it calls.
* Enable Prometheus metrics for the jaeger client library (#10112)Richard van der Hoff2021-06-041-0/+3
|
* Add OpenTracing for database activity. (#10113)Richard van der Hoff2021-06-031-0/+6
| | | | | | | | | | This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
* Fix opentracing inject to use the SpanContext, not the Span (#10074)Brad Murray2021-06-011-5/+5
| | | Signed-off-by: Brad Murray brad@beeper.com
* Set opentracing priority before setting other tags (#10092)Richard van der Hoff2021-05-281-4/+17
| | | ... because tags on spans which aren't being sampled get thrown away.
* Add debug logging for issue #9533 (#9959)Richard van der Hoff2021-05-111-1/+6
| | | | | Hopefully this will help us track down where to-device messages are getting lost/delayed.
* Merge branch 'master' into developAndrew Morgan2021-04-221-3/+12
|\
| * Make LoggingContext's name optional (#9857)Richard van der Hoff2021-04-211-3/+12
| | | | | | | | Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
* | Merge branch 'master' into developAndrew Morgan2021-04-201-10/+4
|\|
| * Always use the name as the log ID. (#9829)Patrick Cloke2021-04-201-10/+4
| | | | | | | | | | As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
* | Fix (final) Bugbear violations (#9838)Jonathan de Jong2021-04-201-2/+2
| |
* | Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-149-9/+0
|/ | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong2021-04-081-1/+2
| | | | | | | Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
* Record more information into structured logs. (#9654)Patrick Cloke2021-04-081-11/+59
| | | | Records additional request information into the structured logs, e.g. the requester, IP address, etc.
* Improve tracing for to device messages (#9686)Erik Johnston2021-04-011-0/+8
|
* Fix `re.Pattern` mypy error on 3.6 (#9703)Jonathan de Jong2021-03-291-2/+2
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-1/+1
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong2021-03-241-1/+1
|
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-121-8/+15
|
* Fix additional type hints. (#9543)Patrick Cloke2021-03-091-2/+4
| | | Type hint fixes due to Twisted 21.2.0 adding type hints.
* Fix additional type hints from Twisted upgrade. (#9518)Patrick Cloke2021-03-031-2/+4
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-165-13/+23
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Add type hints to E2E handler. (#9232)Patrick Cloke2021-01-281-1/+1
| | | This finishes adding type hints to the `synapse.handlers` module.
* Add type hints to the logging context code. (#8939)Patrick Cloke2021-01-051-18/+32
|
* Various clean-ups to the logging context code (#8935)Patrick Cloke2020-12-141-19/+5
|
* Gracefully handle a pending logging connection during shutdown. (#8685)Patrick Cloke2020-10-291-9/+18
|
* Support generating structured logs in addition to standard logs. (#8607)Patrick Cloke2020-10-295-453/+218
| | | | | | | This modifies the configuration of structured logging to be usable from the standard Python logging configuration. This also separates the formatting of logs from the transport allowing JSON logs to files or standard logs to sockets.
* Start fewer opentracing spans (#8640)Erik Johnston2020-10-261-5/+5
| | | | | | | #8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously. This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary. I don't really know how much this will help.
* Separate the TCP and terse JSON formatting code. (#8587)Patrick Cloke2020-10-212-186/+238
| | | | This should (theoretically) allow for using the TCP code with a different output type and make it easier to use the JSON code with files / console.
* Enable mypy checking for unreachable code and fix instances. (#8432)Patrick Cloke2020-10-011-9/+1
|
* Create a mechanism for marking tests "logcontext clean" (#8399)Richard van der Hoff2020-09-281-20/+23
|
* Simplify super() calls to Python 3 syntax. (#8344)Patrick Cloke2020-09-182-4/+4
| | | | | | | This converts calls like super(Foo, self) -> super(). Generated with: sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-3/+1
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix typos in comments.Patrick Cloke2020-09-141-1/+1
|
* Use the right constructor for log records (#8278)Richard van der Hoff2020-09-081-3/+3
| | | | | | Update `log_function` to use the right factory to create log records, to make sure that they have `request` attributes. Fixes: #8267.
* Stop sub-classing object (#8249)Patrick Cloke2020-09-044-9/+9
|
* Use the JSON encoder without whitespace in more places. (#8124)Patrick Cloke2020-08-201-3/+2
|
* Be stricter about JSON that is accepted by Synapse (#8106)Patrick Cloke2020-08-191-2/+5
|
* Fix error reporting when using `opentracing.trace` (#7961)Erik Johnston2020-07-272-12/+3
|
* Clean up PreserveLoggingContext (#7877)Richard van der Hoff2020-07-221-16/+13
| | | This had some dead code and some just plain wrong docstrings.
* Remove unused code from synapse.logging.utils. (#7897)Karthikeyan Singaravelan2020-07-201-126/+0
|
* Fix the trace function for async functions. (#7872)Patrick Cloke2020-07-171-23/+40
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* isort 5 compatibility (#7786)Will Hunt2020-07-051-2/+2
| | | The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-031-37/+31
|
* Add support for using rust-python-jaeger-reporter (#7697)Erik Johnston2020-06-171-3/+36
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-161-2/+1
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-8/+2
|
* Improve per-block CPU and DB usage metrics (#7426)Patrick Cloke2020-05-061-12/+26
|
* Add backwards compatibility codepath to LoggingContext. (#7408)Patrick Cloke2020-05-051-0/+41
|
* Add instance name to RDATA/POSITION commands (#7364)Erik Johnston2020-04-291-13/+10
| | | | | This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.
* Reduce the number of calls to `resource.getrusage` (#7183)Richard van der Hoff2020-04-031-39/+63
| | | Let's just call `getrusage` once on each logcontext change, rather than twice.
* Fix "'NoneType' has no attribute start|stop" logcontext errors (#7181)Richard van der Hoff2020-03-311-0/+5
| | | | Fixes #7179.
* Clean up some LoggingContext stuff (#7120)Richard van der Hoff2020-03-243-126/+125
| | | | | | | | | | | | | | | | | | | | | | | * Pull Sentinel out of LoggingContext ... and drop a few unnecessary references to it * Factor out LoggingContext.current_context move `current_context` and `set_context` out to top-level functions. Mostly this means that I can more easily trace what's actually referring to LoggingContext, but I think it's generally neater. * move copy-to-parent into `stop` this really just makes `start` and `stop` more symetric. It also means that it behaves correctly if you manually `set_log_context` rather than using the context manager. * Replace `LoggingContext.alive` with `finished` Turn `alive` into `finished` and make it a bit better defined.
* Hopefully mypy is happy nowBrendan Abolivier2020-03-101-1/+2
|
* Attempt at appeasing the gods of mypyBrendan Abolivier2020-03-101-1/+1
|
* Add type hints to logging/context.py (#6309)Neil Pilgrim2020-03-071-47/+74
| | | | | | * Add type hints to logging/context.py Signed-off-by: neiljp (Neil Pilgrim) <github@kepier.clara.net>
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-1/+1
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* comment for run_in_backgroundRichard van der Hoff2020-01-121-0/+3
|
* Bump version of mypyErik Johnston2019-12-122-1/+4
|
* Update commentErik Johnston2019-12-101-2/+4
|
* Fix make_deferred_yieldable to work with coroutinesErik Johnston2019-12-101-1/+8
|
* Add benchmarks for structured logging performance (#6266)Amber Brown2019-12-031-0/+1
|
* Improve the performance of structured logging (#6322)Amber Brown2019-11-262-30/+90
|
* A couple more instancesAndrew Morgan2019-11-121-1/+1
|
* Update black to 19.10b0 (#6304)Amber Brown2019-11-011-1/+1
| | | * update version of black and also fix the mypy config being overridden
* Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-311-1/+1
| | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
* Trace non-JSON APIs, /media, /key etcErik Johnston2019-10-111-4/+10
|
* Fix up some typechecking (#6150)Amber Brown2019-10-022-8/+21
| | | | | | * type checking fixes * changelog
* Fix logcontext spam on non-Linux platforms (#6059)J. Ryan Stinnett2019-09-181-2/+11
| | | | | | | This checks whether the current platform supports thread resource usage tracking before logging a warning to avoid log spam. Fixes https://github.com/matrix-org/synapse/issues/6055
* Fix for structured logging tests stomping on logs (#6023)Amber Brown2019-09-133-9/+11
|
* Don't assume there is a 'self' arg in @trace decoratorErik Johnston2019-09-061-5/+5
|
* Gracefully handle log context slips and missing opentracing import errors. ↵Jorik Schellekens2019-09-051-24/+58
| | | | (#5988)
* Add opentracing to all client servlets (#5983)Jorik Schellekens2019-09-051-1/+1
|
* Fix opentracing contexts missing from outbound replication requests (#5982)Jorik Schellekens2019-09-051-4/+16
|
* Opentrace device lists (#5853)Jorik Schellekens2019-09-031-53/+17
| | | Trace device list changes.
* Implement a structured logging output system. (#5680)Amber Brown2019-08-283-1/+665
|
* Propagate opentracing contexts through EDUs (#5852)Jorik Schellekens2019-08-221-0/+26
| | | | | Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Opentracing across workers (#5771)Jorik Schellekens2019-08-221-65/+79
| | | | | | | | | | | | | | Propagate opentracing contexts across workers Also includes some Convenience modifications to opentracing for servlets, notably: - Add boolean to skip the whitelisting check on inject extract methods. - useful when injecting into carriers locally. Otherwise we'd always have to include our own servername and whitelist our servername - start_active_span_from_request instead of header - Add boolean to decide whether to extract context from a request to a servlet
* Opentracing doc update (#5776)Jorik Schellekens2019-08-201-27/+40
| | | Update opentracing docs to use the unified 'trace' method
* Make Jaeger fully configurable (#5694)Jorik Schellekens2019-07-231-4/+7
| | | | | | * Allow Jaeger to be configured * Update sample config
* Replace returnValue with return (#5736)Amber Brown2019-07-231-3/+3
|
* Opentracing Utils (#5722)Jorik Schellekens2019-07-232-101/+356
| | | | | | | | | | | | | | | | | | | | | | | | | | * Add decerators for tracing functions * Use the new clean contexts * Context and edu utils * Move opentracing setters * Move whitelisting * Sectioning comments * Better args wrapper * Docstrings Co-Authored-By: Erik Johnston <erik@matrix.org> * Remove unused methods. * Don't use global * One tracing decorator to rule them all.
* Merge branch 'release-v1.2.0' into developJorik Schellekens2019-07-221-0/+125
|\
| * Opentracing Documentation (#5703)Jorik Schellekens2019-07-221-0/+125
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Opentracing survival guide * Update decorator names in doc * Doc cleanup These are all alterations as a result of comments in #5703, it includes mostly typos and clarifications. The most interesting changes are: - Split developer and user docs into two sections - Add a high level description of OpenTracing * newsfile * Move contributer specific info to docstring. * Sample config. * Trailing whitespace. * Update 5703.misc * Apply suggestions from code review Mostly just rewording parts of the docs for clarity. Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Don't accept opentracing data from clients. (#5715)Jorik Schellekens2019-07-191-2/+1
|/ | | | | | * Don't accept opentracing data from clients. * newsfile
* Clean up opentracing configuration options (#5712)Richard van der Hoff2019-07-182-26/+20
| | | | | | | | | | | | | | | | | Clean up config settings and dead code. This is mostly about cleaning up the config format, to bring it into line with our conventions. In particular: * There should be a blank line after `## Section ##' headings * There should be a blank line between each config setting * There should be a `#`-only line between a comment and the setting it describes * We don't really do the `# #` style commenting-out of whole sections if we can help it * rename `tracer_enabled` to `enabled` While we're here, do more config parsing upfront, which makes it easier to use later on. Also removes redundant code from LogContextScopeManager. Also changes the changelog fragment to a `feature` - it's exciting!
* Add basic opentracing support (#5544)Jorik Schellekens2019-07-113-2/+508
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Configure and initialise tracer Includes config options for the tracer and sets up JaegerClient. * Scope manager using LogContexts We piggy-back our tracer scopes by using log context. The current log context gives us the current scope. If new scope is created we create a stack of scopes in the context. * jaeger is a dependency now * Carrier inject and extraction for Twisted Headers * Trace federation requests on the way in and out. The span is created in _started_processing and closed in _finished_processing because we need a meaningful log context. * Create logcontext for new scope. Instead of having a stack of scopes in a logcontext we create a new context for a new scope if the current logcontext already has a scope. * Remove scope from logcontext if logcontext is top level * Disable tracer if not configured * typo * Remove dependence on jaeger internals * bools * Set service name * :Explicitely state that the tracer is disabled * Black is the new black * Newsfile * Code style * Use the new config setup. * Generate config. * Copyright * Rename config to opentracing * Remove user whitelisting * Empty whitelist by default * User ConfigError instead of RuntimeError * Use isinstance * Use tag constants for opentracing. * Remove debug comment and no need to explicitely record error * Two errors a "s(c)entry" * Docstrings! * Remove debugging brainslip * Homeserver Whitlisting * Better opentracing config comment * linting * Inclue worker name in service_name * Make opentracing an optional dependency * Neater config retreival * Clean up dummy tags * Instantiate tracing as object instead of global class * Inlcude opentracing as a homeserver member. * Thread opentracing to the request level * Reference opetnracing through hs * Instantiate dummy opentracin g for tests. * About to revert, just keeping the unfinished changes just in case * Revert back to global state, commit number: 9ce4a3d9067bf9889b86c360c05ac88618b85c4f * Use class level methods in tracerutils * Start and stop requests spans in a place where we have access to the authenticated entity * Seen it, isort it * Make sure to close the active span. * I'm getting black and blue from this. * Logger formatting Co-Authored-By: Erik Johnston <erik@matrix.org> * Outdated comment * Import opentracing at the top * Return a contextmanager * Start tracing client requests from the servlet * Return noop context manager if not tracing * Explicitely say that these are federation requests * Include servlet name in client requests * Use context manager * Move opentracing to logging/ * Seen it, isort it again! * Ignore twisted return exceptions on context exit * Escape the scope * Scopes should be entered to make them useful. * Nicer decorator names * Just one init, init? * Don't need to close something that isn't open * Docs make you smarter
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-044-0/+940