summary refs log tree commit diff
path: root/synapse/rest/media (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Asynchronous Uploads (#15503)Sumner Evans2023-11-155-42/+215
| | | Support asynchronous uploads as defined in MSC2246.
* Return attrs for more media repo APIs. (#16611)Patrick Cloke2023-11-091-8/+8
|
* Return ThumbnailInfo in more places (#16438)Patrick Cloke2023-10-061-56/+42
| | | | Improves type hints by using concrete types instead of dictionaries.
* Register media servlets via regex. (#16419)Patrick Cloke2023-10-066-72/+89
| | | | | This converts the media servlet URLs in the same way as (most) of the rest of Synapse. This will give more flexibility in the versions each endpoint exists under.
* Remove some unused `server_name` fields (#15723)Sean Quah2023-06-061-1/+0
| | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Add config option to prevent media downloads from listed domains. (#15197)Travis Ralston2023-05-091-0/+9
| | | | | | | This stops media (and thumbnails) from being accessed from the listed domains. It does not delete any already locally cached media, but will prevent accessing it. Note that admin APIs are unaffected by this change.
* Factor out an `is_mine_server_name` method (#15542)Sean Quah2023-05-052-4/+4
| | | | | | | | | | | | Add an `is_mine_server_name` method, similar to `is_mine_id`. Ideally we would use this consistently, instead of sometimes comparing against `hs.hostname` and other times reaching into `hs.config.server.server_name`. Also fix a bug in the tests where `hs.hostname` would sometimes differ from `hs.config.server.server_name`. Signed-off-by: Sean Quah <seanq@matrix.org>
* Separate HTTP preview code and URL previewer. (#15269)Patrick Cloke2023-03-201-792/+4
| | | Separates REST layer code from the actual URL previewing.
* Refactor media modules. (#15146)Patrick Cloke2023-02-2714-3520/+117
| | | | | | | * Removes the `v1` directory from `test.rest.media.v1`. * Moves the non-REST code from `synapse.rest.media.v1` to `synapse.media`. * Flatten the `v1` directory from `synapse.rest.media`, but leave compatiblity with 3rd party media repositories and spam checkers.
* Do not fail completely if oEmbed autodiscovery fails. (#15092)Patrick Cloke2023-02-231-10/+23
| | | | | | Previously if an autodiscovered oEmbed request failed (e.g. the oEmbed endpoint is down or does not exist) then the entire URL preview would fail. Instead we now return everything we can, even if this additional request fails.
* Stop parsing the unspecced type parameter on thumbnail requests. (#15137)Patrick Cloke2023-02-231-1/+2
| | | | | | | | Ideally we would replace this with parsing of the Accept header or something else, but for now just make Synapse spec compliant by ignoring the unspecced parameter. It does not seem that this is ever sent by a client, and even if it is there's a reasonable fallback.
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-222-2/+0
|
* Use mypy 1.0 (#15052)David Robertson2023-02-161-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Typecheck tests.rest.media.v1.test_media_storage (#15008)David Robertson2023-02-071-4/+3
| | | | | | | | | | | * Fix MediaStorage type hint * Typecheck tests.rest.media.v1.test_media_storage * Changelog * Remove assert and make the comment succinct * Fix syntax for olddeps
* Prefer `type(x) is int` to `isinstance(x, int)` (#14945)David Robertson2023-01-312-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Perfer `type(x) is int` to `isinstance(x, int)` This covered all additional instances I could see where `x` was user-controlled. The remaining cases are ``` $ rg -s 'isinstance.*[^_]int' tests/replication/_base.py 576: if isinstance(obj, int): synapse/util/caches/stream_change_cache.py 136: assert isinstance(stream_pos, int) 214: assert isinstance(stream_pos, int) 246: assert isinstance(stream_pos, int) 267: assert isinstance(stream_pos, int) synapse/replication/tcp/external_cache.py 133: if isinstance(result, int): synapse/metrics/__init__.py 100: if isinstance(calls, (int, float)): synapse/handlers/appservice.py 262: assert isinstance(new_token, int) synapse/config/_util.py 62: if isinstance(p, int): ``` which cover metrics, logic related to `jsonschema`, and replication and data streams. AFAICS these are all internal to Synapse * Changelog
* Unescape HTML entities in oEmbed titles. (#14781)Jeyachandran Rathnam2023-01-091-6/+9
| | | | | | | It doesn't seem valid that HTML entities should appear in the title field of oEmbed responses, but a popular WordPress plug-in seems to do it. There should not be harm in unescaping these.
* Respond with proper error responses on unknown paths. (#14621)Patrick Cloke2022-12-081-2/+2
| | | | Returns a proper 404 with an errcode of M_RECOGNIZED for unknown endpoints per MSC3743.
* Remove redundant types from comments. (#14412)Patrick Cloke2022-11-162-4/+4
| | | | | | | Remove type hints from comments which have been added as Python type hints. This helps avoid drift between comments and reality, as well as removing redundant information. Also adds some missing type hints which were simple to fill in.
* Be more lenient in the oEmbed response parsing. (#14089)Patrick Cloke2022-10-071-50/+57
| | | | | | Attempt to parse any valid information from an oEmbed response (instead of bailing at the first unexpected data). This should allow for more partial oEmbed data to be returned, resulting in better / more URL previews, even if those URL previews are only partial.
* Add a `MXCUri` class to make working with mxc uri's easier. (#13162)Andrew Morgan2022-09-152-4/+8
|
* Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without ↵Jacek Kuśnierz2022-08-311-1/+0
| | | | | | | an `id_access_token` (#13241) Fixes #13206 Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
* Fix bug where we wedge media plugins if clients disconnect early (#13660)Erik Johnston2022-08-301-19/+21
| | | | | | | | We incorrectly didn't use the returned `Responder` if the client had disconnected, which meant that the resource used by the Responder wasn't correctly released. In particular, this exhausted the thread pools so that *all* requests timed out.
* Do not wait for background updates to complete do expire URL cache. (#13657)Patrick Cloke2022-08-301-4/+0
| | | | | | | | | | | | | | Media downloaded as part of a URL preview is normally deleted after two days. However, while a background database migration is running, the process is stopped. A long-running database migration can therefore cause the media store to fill up with old preview files. This logic was added in #2697 to make sure that we didn't try to run the expiry without an index on `local_media_repository.created_ts`; the original logic that needs that index was added in #2478 (in `get_url_cache_media_before`, as amended by 93247a424a5068b088567fa98b6990e47608b7cb), and is still present. Given that the background update was added before Synapse v1.0.0, just drop this check and assume the index exists.
* Provide more info why we don't have any thumbnails to serve (#13038)Eric Eastwood2022-07-151-2/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/13016 ## New error code and status ### Before Previously, we returned a `404` for `/thumbnail` which isn't even in the spec. ```json { "errcode": "M_NOT_FOUND", "error": "Not found [b'hs1', b'tefQeZhmVxoiBfuFQUKRzJxc']" } ``` ### After What does the spec say? > 400: The request does not make sense to the server, or the server cannot thumbnail the content. For example, the client requested non-integer dimensions or asked for negatively-sized images. > > *-- https://spec.matrix.org/v1.1/client-server-api/#get_matrixmediav3thumbnailservernamemediaid* Now with this PR, we respond with a `400` when we don't have thumbnails to serve and we explain why we might not have any thumbnails. ```json { "errcode": "M_UNKNOWN", "error": "Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.)", } ``` > Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.) --- We still respond with a 404 in many other places. But we can iterate on those later and maybe keep some in some specific places after spec updates/clarification: https://github.com/matrix-org/matrix-spec/issues/1122 We can also iterate on the bugs where Synapse doesn't thumbnail when it should in other issues/PRs.
* Inline URL preview documentation. (#13261)Patrick Cloke2022-07-121-4/+58
| | | Inline URL preview documentation near the implementation.
* Uniformize spam-checker API, part 5: expand other spam-checker callbacks to ↵David Teller2022-07-111-1/+3
| | | | | | return `Tuple[Codes, dict]` (#13044) Signed-off-by: David Teller <davidt@element.io> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* fix linting error from the 1.61.1 main -> develop mergeAndrew Morgan2022-06-281-1/+2
|
* Merge branch 'master' into developAndrew Morgan2022-06-281-24/+39
|\
| * Merge pull request from GHSA-22p3-qrh9-cx32reivilibre2022-06-281-24/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make _iterate_over_text easier to read by using simple data structures * Prefer a set of tags to ignore In my tests, it's 4x faster to check for containment in a set of this size * Add a stack size limit to _iterate_over_text * Continue accepting the case where there is no body element * Use an early return instead for None Co-authored-by: Richard van der Hoff <richard@matrix.org>
* | Add Cross-Origin-Resource-Policy header to thumbnail and download media ↵Robert Long2022-06-272-2/+12
| | | | | | | | endpoints (#12944)
* | Improve URL previews for sites with only Twitter card information. (#13056)Patrick Cloke2022-06-161-17/+95
| | | | | | | | | | | | Pull out `twitter:` meta tags when generating a preview and use it to augment any `og:` meta tags. Prefers Open Graph information over Twitter card information.
* | Uniformize spam-checker API, part 4: port other spam-checker callbacks to ↵David Teller2022-06-131-3/+4
|/ | | | | return `Union[Allow, Codes]`. (#12857) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Prevent local quarantined media from being claimed by media retention (#12972)Andrew Morgan2022-06-071-6/+16
|
* Do not break URL previews if an image is unreachable. (#12950)Patrick Cloke2022-06-061-6/+17
| | | | Avoid breaking a URL preview completely if the chosen image 404s or is unreachable for some other reason (e.g. DNS).
* Improve URL previews for some pages (#12951)Patrick Cloke2022-06-031-17/+35
| | | | | * Skip `og` and `meta` tags where the value is empty. * Fallback to the favicon if there are no other images. * Ignore tags meant for navigation.
* Fix potential thumbnail memory leaks. (#12932)Erik Johnston2022-06-012-135/+201
|
* Add config options for media retention (#12732)Andrew Morgan2022-05-311-1/+70
|
* Merge tag 'v1.60.0rc2' into developSean Quah2022-05-271-1/+9
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.60.0rc2 (2022-05-27) ============================== This release of Synapse adds a unique index to the `state_group_edges` table, in order to prevent accidentally introducing duplicate information (for example, because a database backup was restored multiple times). If your Synapse database already has duplicate rows in this table, this could fail with an error and require manual remediation. Additionally, the signature of the `check_event_for_spam` module callback has changed. The previous signature has been deprecated and remains working for now. Module authors should update their modules to use the new signature where possible. See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1600) for more details. Features -------- - Add an option allowing users to use their password to reauthenticate for privileged actions even though password login is disabled. ([\#12883](https://github.com/matrix-org/synapse/issues/12883)) Bugfixes -------- - Explicitly close `ijson` coroutines once we are done with them, instead of leaving the garbage collector to close them. ([\#12875](https://github.com/matrix-org/synapse/issues/12875)) Internal Changes ---------------- - Improve URL previews by not including the content of media tags in the generated description. ([\#12887](https://github.com/matrix-org/synapse/issues/12887))
| * Improve URL previews by not including the content of media tags in the ↵reivilibre2022-05-261-1/+9
| | | | | | | | generated description. (#12887)
* | Correct annotation of `_iterate_over_text` (#12860)David Robertson2022-05-241-1/+1
|/
* URL preview cache expiry logs: INFO -> DEBUG, text clarifications (#12720)Andrew Morgan2022-05-121-9/+21
|
* Fix mypy against latest pillow stubs (#12671)David Robertson2022-05-091-3/+3
|
* Refactor and convert `Linearizer` to async (#12357)Sean Quah2022-04-051-3/+3
| | | | | | | | | | | Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>
* Ensure the type of URL attributes is always str when matching against ↵Brendan Abolivier2022-03-311-2/+7
| | | | preview blacklist (#12333)
* Bump `black` and `click` versions (#12320)David Robertson2022-03-292-3/+3
|
* Clean-up logic for rebasing URLs during URL preview. (#12219)Patrick Cloke2022-03-162-48/+14
| | | | By using urljoin from the standard library and reducing the number of places URLs are rebased.
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-1/+0
|
* Use `ParamSpec` in type hints for `synapse.logging.context` (#12150)Sean Quah2022-03-081-2/+7
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-234-4/+4
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Fetch images when previewing Twitter URLs. (#11985)AndrewRyanChama2022-02-221-1/+9
| | | | By including "bot" in the User-Agent, which some sites use to decide whether to include additional Open Graph information.
* Implement a content type allow list for URL previews (#11936)Denis Kasak2022-02-101-0/+8
| | | | | | | This implements an allow list for content types for which Synapse will attempt URL preview. If a URL resolves to a resource with a content type which isn't in the list, the download will terminate immediately. This makes sense given that Synapse would never successfully generate a URL preview for such files in the first place, and helps prevent issues with streaming media servers, such as #8302. Signed-off-by: Denis Kasak dkasak@termina.org.uk
* Pass the proper type when uploading files. (#11927)Patrick Cloke2022-02-071-4/+9
| | | | The Content-Length header should be treated as an int, not a string. This shouldn't have any user-facing change.
* Support rendering previews with data: URLs in them (#11767)Patrick Cloke2022-01-242-64/+191
| | | | | Images which are data URLs will no longer break URL previews and will properly be "downloaded" and thumbnailed.
* Fix preview of imgur and Tenor URLs. (#11669)Philippe Daouadi2022-01-182-13/+32
| | | | | | By scraping Open Graph information from the HTML even when an autodiscovery endpoint is found. The results are then combined to capture as much information as possible from the page.
* Use auto_attribs/native type hints for attrs classes. (#11692)Patrick Cloke2022-01-131-3/+3
|
* Convert all namedtuples to attrs. (#11665)Patrick Cloke2021-12-301-6/+13
| | | To improve type hints throughout the code.
* Move HTML parsing to a separate file for URL previews. (#11566)Patrick Cloke2021-12-133-378/+407
| | | | | | | * Splits the logic for parsing HTML from the resource handling code. * Fix a circular import in the oEmbed code (which uses the HTML parsing code). * Renames some of the HTML parsing methods to: * Make it clear which methods are "internal" to the module. * Clarify what the methods do.
* Fix media repository failing when media store path contains symlinks (#11446)Sean Quah2021-12-021-44/+71
|
* Merge branch 'master' into developSean Quah2021-11-232-43/+216
|\
| * Prevent the media store from writing outside of the configured directorySean Quah2021-11-192-43/+216
| | | | | | | | | | Also tighten validation of server names by forbidding invalid characters in IPv6 addresses and empty domain labels.
* | Add type hints to media repository storage module (#11311)Patrick Cloke2021-11-121-4/+4
| |
* | Fix error in thumbnail generation (#11288)Neeeflix2021-11-101-4/+4
|/ | | Signed-off-by: Jonas Zeunert <jonas@zeunert.org>
* Merge remote-tracking branch 'origin/master' into developErik Johnston2021-11-021-3/+6
|\
| * Fix URL preview errors when previewing XML documents. (#11196)Patrick Cloke2021-10-271-3/+6
| |
* | Handle missing Content-Type header when accessing remote media (#11200)Shay2021-11-012-2/+12
|/ | | | | | | | | | | | | | | | | | | | | * add code to handle missing content-type header and a test to verify that it works * add handling for missing content-type in the /upload endpoint as well * slightly refactor test code to put private method in approriate place * handle possible null value for content-type when pulling from the local db * add changelog * refactor test and add code to handle missing content-type in cached remote media * requested changes * Update changelog.d/11200.bugfix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Ensure each charset is attempted only once during media preview. (#11089)Patrick Cloke2021-10-141-6/+28
| | | | There's no point in trying more than once since it is guaranteed to continually fail.
* Attempt different character encodings when previewing a URL. (#11077)Patrick Cloke2021-10-141-41/+39
| | | | This follows similar logic to BeautifulSoup where we attempt different character encodings until we find one which works.
* Remove dead code from `MediaFilePaths` (#11056)Sean Quah2021-10-131-17/+0
|
* Be more lenient when parsing the version for oEmbed responses. (#11065)Patrick Cloke2021-10-132-8/+7
|
* Merge remote-tracking branch 'origin/release-v1.45' into developPatrick Cloke2021-10-121-1/+1
|\
| * Fix formatting string when oEmbed errors occur. (#11061)Patrick Cloke2021-10-121-1/+1
| |
* | Update `_wrap_in_base_path` type hints to preserve function arguments (#11055)Sean Quah2021-10-121-3/+6
|/
* Autodiscover oEmbed endpoint from returned HTML (#10822)Patrick Cloke2021-10-082-38/+100
| | | | | Searches the returned HTML for an oEmbed endpoint using the autodiscovery mechanism (`<link rel=...>`), and will request it to generate the preview.
* Relax `ignore-missing-imports` for modules that have stubs now and update ↵David Robertson2021-10-082-29/+30
| | | | | | | | | | | | mypy (#11006) Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details. We therefore pull in stub packages in setup.py Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them. The rest of this change consists of fixups to make the newer mypy + stubs pass CI. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix empty `url_cache_thumbnails/yyyy-mm-dd/` directories being left behind ↵Sean Quah2021-09-291-31/+43
| | | | (#10924)
* Avoid storing URL cache files in storage providers (#10911)Sean Quah2021-09-273-6/+16
| | | | | URL cache files are short-lived and it does not make sense to offload them (eg. to the cloud) or back them up.
* Fix race conditions when creating media store and config directories (#10913)Sean Quah2021-09-272-6/+3
|
* Use direct references for configuration variables (part 5). (#10897)Patrick Cloke2021-09-246-17/+21
|
* Add reactor to `SynapseRequest` and fix up types. (#10868)Erik Johnston2021-09-247-25/+23
|
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+3
|
* Include more information in oEmbed previews. (#10819)Patrick Cloke2021-09-222-5/+46
| | | | | | | * Improved titles (fall back to the author name if there's not title) and include the site name. * Handle photo/video payloads. * Include the original URL in the Open Graph response. * Fix the expiration time (by properly converting from seconds to milliseconds).
* Refactor oEmbed previews (#10814)Patrick Cloke2021-09-212-199/+272
| | | | | | | | | | | | | The major change is moving the decision of whether to use oEmbed further up the call-stack. This reverts the _download_url method to being a "dumb" functionwhich takes a single URL and downloads it (as it was before #7920). This also makes more minor refactorings: * Renames internal variables for clarity. * Factors out shared code between the HTML and rich oEmbed previews. * Fixes tests to preview an oEmbed image.
* Add missing type hints to non-client REST servlets. (#10817)Patrick Cloke2021-09-157-25/+56
| | | | Including admin, consent, key, synapse, and media. All REST servlets (the synapse.rest module) now require typed method definitions.
* Convert media repo's FileInfo to attrs. (#10785)Patrick Cloke2021-09-144-108/+139
| | | | This is mostly an internal change, but improves type hints in the media code.
* Request JSON for oEmbed requests (and ignore XML only providers). (#10759)Patrick Cloke2021-09-081-3/+23
| | | | | | | | This adds the format to the request arguments / URL to ensure that JSON data is returned (which is all that Synapse supports). This also adds additional error checking / filtering to the configuration file to ignore XML-only providers.
* Use attrs internally for the URL preview code & add documentation. (#10753)Patrick Cloke2021-09-071-42/+79
|
* Allow configuration of the oEmbed URLs. (#10714)Patrick Cloke2021-08-312-142/+140
| | | | | This adds configuration options (under an `oembed` section) to configure which URLs are matched to use oEmbed for URL previews.
* Fix error when selecting between thumbnails with the same quality (#10684)Sean2021-08-251-9/+17
| | | Fixes #10318
* Admin API to delete media for a specific user (#10558)Dirk Klimpel2021-08-111-2/+4
|
* Support underscores (in addition to hyphens) for charset detection. (#10410)sri-vidyut2021-07-271-2/+4
|
* Mitigate media repo XSSs on IE11. (#10468)Denis Kasak2021-07-271-0/+2
| | | | | | | | IE11 doesn't support Content-Security-Policy but it has support for a non-standard X-Content-Security-Policy header, which only supports the sandbox directive. This prevents script execution, so it at least offers some protection against media repo-based attacks. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
* Add a return type to parse_string. (#10438)Patrick Cloke2021-07-211-7/+3
| | | | And set the required attribute in a few places which will error if a parameter is not provided.
* [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-191-2/+2
| | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-165-15/+15
|
* Correct type hints for parse_string(s)_from_args. (#10137)Patrick Cloke2021-06-081-5/+6
|
* Fix /upload 500'ing when presented a very large image (#10029)Michael Telatynski2021-05-212-0/+11
| | | | | | | | | | | | * Fix /upload 500'ing when presented a very large image Catch DecompressionBombError and re-raise as ThumbnailErrors * Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml to get it to bomb out quicker, to load less into memory in the case of super large images * Add changelog entry for 10029
* Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan2021-04-271-1/+1
| | | I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
* Improved validation for received requests (#9817)Richard van der Hoff2021-04-231-2/+0
| | | | | | * Simplify `start_listening` callpath * Correctly check the size of uploaded files
* remove `HomeServer.get_config` (#9815)Richard van der Hoff2021-04-141-1/+1
| | | | Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.
* More robust handling of the Content-Type header for thumbnail generation (#9788)rkfg2021-04-141-0/+3
| | | | Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
* Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong2021-04-1412-12/+0
| | | | | | | Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
* Add type hints to expiring cache. (#9730)Patrick Cloke2021-04-061-1/+1
|
* Make it possible to use dmypy (#9692)Erik Johnston2021-03-261-0/+2
| | | | | | | | | Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
* Import HomeServer from the proper module. (#9665)Patrick Cloke2021-03-237-7/+7
|
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-123-3/+6
|
* Convert Requester to attrs (#9586)Richard van der Hoff2021-03-101-1/+2
| | | | | | ... because namedtuples suck Fix up a couple of other annotations to keep mypy happy.
* Handle image transparency better when thumbnailing. (#9473)Patrick Cloke2021-03-091-3/+8
| | | | Properly uses RGBA mode for 1- and 8-bit images with transparency (instead of RBG mode).
* Use the proper Request in type hints. (#9515)Patrick Cloke2021-03-017-17/+22
| | | | This also pins the Twisted version in the mypy job for CI until proper type hints are fixed throughout Synapse.
* Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-261-2/+1
| | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
* Return a 404 if we don't have the original fileErik Johnston2021-02-192-1/+8
|
* Regenerate exact thumbnails if missingErik Johnston2021-02-182-2/+49
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-164-12/+26
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix some typos.Patrick Cloke2021-02-122-5/+5
|
* Handle additional errors when previewing URLs. (#9333)Patrick Cloke2021-02-081-19/+52
| | | | * Handle the case of lxml not finding a document tree. * Parse the document encoding from the XML tag.
* Add check_media_file_for_spam spam checker hookErik Johnston2021-02-042-6/+65
|
* Try to recover from unknown encodings when previewing media. (#9164)Patrick Cloke2021-01-261-10/+34
| | | | | | Treat unknown encodings (according to lxml) as UTF-8 when generating a preview for HTML documents. This isn't fully accurate, but will hopefully give a reasonable title and summary.
* Return a 404 if no valid thumbnail is found. (#9163)Patrick Cloke2021-01-212-81/+158
| | | | | | If no thumbnail of the requested type exists, return a 404 instead of erroring. This doesn't quite match the spec (which does not define what happens if no thumbnail can be found), but is consistent with what Synapse already does.
* Add type hints to media rest resources. (#9093)Patrick Cloke2021-01-1511-164/+283
|
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-111-10/+6
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Add X-Robots-Tag header to stop crawlers from indexing media (#8887)Aaron Raimist2020-12-081-0/+5
| | | | | | | Fixes / related to: https://github.com/matrix-org/synapse/issues/6533 This should do essentially the same thing as a robots.txt file telling robots to not index the media repo. https://developers.google.com/search/reference/robots_meta_tag Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Do not raise a 500 exception when previewing empty media. (#8883)Patrick Cloke2020-12-071-1/+5
|
* Do not 500 if the content-length is not provided when uploading media. (#8862)Patrick Cloke2020-12-041-1/+1
| | | Instead return the proper 400 error.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-1/+1
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Fix race for concurrent downloads of remote media. (#8682)Erik Johnston2020-10-302-70/+125
| | | Fixes #6755
* Add an admin api to delete local media. (#8519)Dirk Klimpel2020-10-262-1/+88
| | | | | | Related to: #6459, #3479 Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server.
* Don't bother responding to client requests that have already disconnected ↵Andrew Morgan2020-10-061-0/+6
| | | | | | | | | | (#8465) This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so. This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003. Fixes https://github.com/matrix-org/synapse/issues/6700 Fixes https://github.com/matrix-org/synapse/issues/5304
* Fix bug in remote thumbnail search (#8438)Richard van der Hoff2020-10-021-20/+23
| | | | | #7124 changed the behaviour of remote thumbnails so that the thumbnailing method was included in the filename of the thumbnail. To support existing files, it included a fallback so that we would check the old filename if the new filename didn't exist. Unfortunately, it didn't apply this logic to storage providers, so any thumbnails stored on such a storage provider was broken.
* fix a logging error in thumbnailer (#8435)Richard van der Hoff2020-10-011-1/+1
| | | Introduced in #8236
* Discard an empty upload_name before persisting an uploaded file (#7905)Will Hunt2020-09-292-3/+8
|
* Add type annotations to SimpleHttpClient (#8372)Richard van der Hoff2020-09-241-4/+10
|
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-1/+1
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix typos in comments.Patrick Cloke2020-09-141-2/+2
|
* Do not error when thumbnailing invalid files (#8236)Patrick Cloke2020-09-093-12/+76
| | | | If a file cannot be thumbnailed for some reason (e.g. the file is empty), then catch the exception and convert it to a reasonable error message for the client.
* Include method in thumbnail media name (#7124)DeepBlueV7.X2020-09-082-1/+46
| | | | | | | | | | | | | | This fixes an issue where different methods (crop/scale) overwrite each other. This first tries the new path. If that fails and we are looking for a remote thumbnail, it tries the old path. If that still isn't found, it continues as normal. This should probably be removed in the future, after some of the newer thumbnails were generated with the new path on most deployments. Then the overhead should be minimal if the other thumbnails need to be regenerated. Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
* Stop sub-classing object (#8249)Patrick Cloke2020-09-045-6/+6
|
* Remove unnecessary maybeDeferred calls (#8044)Patrick Cloke2020-08-071-2/+1
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-071-2/+2
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-1/+1
|
* Fix async/await calls for broken media providers. (#8027)Patrick Cloke2020-08-042-22/+20
|
* Convert the remaining media repo code to async / await. (#7947)Patrick Cloke2020-07-275-107/+130
|
* Support oEmbed for media previews. (#7920)Patrick Cloke2020-07-271-45/+220
| | | Fixes previews of Twitter URLs by using their oEmbed endpoint to grab content.
* Convert more of the media code to async/await (#7873)Patrick Cloke2020-07-242-33/+42
|
* isort 5 compatibility (#7786)Will Hunt2020-07-051-2/+1
| | | The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-035-46/+14
|
* Support running multiple media repos. (#7706)Erik Johnston2020-06-171-3/+15
| | | | | This requires a new config option to specify which media repo should be responsible for running background jobs to e.g. clear out expired URL preview caches.
* Fetch from the r0 media path instead of the unspecced v1. (#7714)Patrick Cloke2020-06-171-4/+4
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-163-13/+5
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-3/+1
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-18/+9
|
* Reduce logging verbosity of URL cache cleanup. (#7295)Michael Kaye2020-04-221-2/+7
|
* Allow specifying the value of Accept-Language header for URL previews (#7265)Andrew Morgan2020-04-151-2/+6
|
* Set Referrer-Policy to no-referrer for media (#7009)Dionysis Grigoropoulos2020-03-231-0/+3
|
* Convert some of the media REST code to async/await (#7110)Patrick Cloke2020-03-203-111/+90
|
* Set charset to utf-8 when adding headers for certain text content types (#7044)The Stranjer2020-03-171-1/+24
| | | | Fixes #7043
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-21/+19
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* a bit of debugging for media storage providers (#6757)Richard van der Hoff2020-01-232-0/+7
| | | | | | * a bit of debugging for media storage providers * changelog
* Fix typo in _select_thumbnailBrendan Abolivier2020-01-221-1/+1
|
* Fixup synapse.rest to pass mypy (#6732)Erik Johnston2020-01-203-11/+13
|
* Remove unused, undocumented "content repo" resource (#6628)Richard van der Hoff2020-01-032-103/+0
| | | | | | This looks like it got half-killed back in #888. Fixes #6567.
* Move background update handling out of storeErik Johnston2019-12-051-1/+1
|
* Fix: Pillow error when uploading RGBA image (#3325) (#6241)Filip Štědronský2019-12-021-1/+4
| | | Signed-Off-By: Filip Štědronský <g@regnarg.cz>
* Fix startup error when http proxy is defined. (#6421)Richard van der Hoff2019-11-261-2/+2
| | | | | Guess I only tested this on python 2 :/ Fixes #6419.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2Richard van der Hoff2019-11-051-1/+2
|\ | | | | Fix exception when OpenGraph tag values are ints
| * Fix exception when OpenGraph tag values are intsRichard van der Hoff2019-11-051-1/+2
| |
* | Merge branch 'develop' into rav/url_preview_limit_titleRichard van der Hoff2019-11-053-20/+26
|\ \ | |/ |/|
| * Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-011-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
| * Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-313-15/+17
| | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
| * Address codestyle concernsMichael Kaye2019-10-241-1/+3
| |
| * Delay printf until logging is required.Michael Kaye2019-10-241-6/+6
| | | | | | | | | | Using % will cause the string to be generated even if debugging is off.
* | Apply suggestions from code reviewRichard van der Hoff2019-11-051-1/+1
| | | | | | | | Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-Authored-By: Erik Johnston <erik@matrix.org>
* | Strip overlong OpenGraph data from url previewRichard van der Hoff2019-11-051-1/+19
|/ | | | ... to stop people causing DoSes with malicious web pages
* Fix store_url_cache using bytesErik Johnston2019-10-101-2/+2
|
* add M_TOO_LARGE error code for uploading a too large file (#6151)Anshul Angaria2019-10-081-2/+6
| | | Fixes #6109
* Log responder we are using. (#6139)Michael Kaye2019-10-071-1/+1
| | | This prevents us logging "Responding to media request with responder %s".
* media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142)Robert Swain2019-10-041-3/+11
| | | | | | Pillow will use nearest neighbour as the resampling algorithm if the source image is either 1-bit or a color palette using 8 bits. If we convert to RGB before scaling, we'll probably get a better result.
* Remove double return statements (#5962)Andrew Morgan2019-09-031-1/+0
| | | | | | | | | | Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.
* fix thumbnail storage location (#5915)L0ric02019-09-021-1/+1
| | | | | | | | | | | | | | * fix thumbnail storage location Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Add changelog file. Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Update Changelog Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-302-4/+4
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Don't load the media repo when configured to use an external media repo (#5754)Amber Brown2019-08-131-1/+5
|
* Replace returnValue with return (#5736)Amber Brown2019-07-233-33/+31
|
* fix typo: backgroud -> backgroundAndrew Morgan2019-07-121-1/+1
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-045-17/+13
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-0/+1
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-295-68/+47
|
* Run Black. (#5482)Amber Brown2019-06-2012-408/+425
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-2/+4
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Fix error when downloading thumbnail with width/height param missing (#5258)Aaron Raimist2019-05-291-2/+2
| | | | | | | Fix error when downloading thumbnail with width/height param missing Fixes #2748 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Fix image orientation when generating thumbnail (#5039)PauRE2019-05-162-0/+44
|
* Migrate all tests to use the dict-based config format instead of hanging ↵Amber Brown2019-05-131-0/+1
| | | | items off HomeserverConfig (#5171)
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-0/+10
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Prevent "producer not unregistered" message (#5009)Amber Brown2019-04-241-0/+4
|
* Remove periods from copyright headers (#5046)Andrew Morgan2019-04-111-1/+1
|
* fix incorrect encoding of filenames with spaces in (#2090)Matthew Hodgson2019-03-111-3/+51
| | | | fixes https://github.com/vector-im/riot-web/issues/3155
* Fix parsing of Content-Disposition headers (#4763)Richard van der Hoff2019-02-271-20/+65
| | | | | | | | | | | * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings
* Don't log stack trace when client has gone away during media download (#4738)Erik Johnston2019-02-251-2/+9
| | | | | | | | * Don't log stack trace when client has gone away during media download * Newsfile * Fixup newsfile
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-3/+4
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Fix IP URL previews on Python 3 (#4215)Amber Brown2018-12-221-2/+12
|
* Add 'sandbox' to CSP for media repo (#4284)David Baker2018-12-111-1/+2
| | | | | | * Add 'sandbox' to the CSP for media repo * Changelog
* Move imports to one lineWill Hunt2018-12-101-4/+1
|
* Use `send_cors`Will Hunt2018-12-101-4/+1
|
* changelog & isortWill Hunt2018-12-091-1/+5
|
* Make /config more CORS-yWill Hunt2018-12-091-1/+3
|
* Fix Content-Disposition in media repository (#4176)Amber Brown2018-11-153-90/+110
|
* Use <meta> tags to discover the per-page encoding of html previews (#4183)Amber Brown2018-11-151-9/+22
|
* Fix URL preview bugs (type error when loading cache from db, content-type ↵Amber Brown2018-11-081-6/+16
| | | | including quotes) (#4157)
* Delete the disused & unspecced identicon functionality (#4106)Amber Brown2018-10-292-70/+0
|
* Fix a number of flake8 errorsRichard van der Hoff2018-10-241-1/+1
| | | | | | | | | | | | Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.
* Correctly account for cpu usage by background threads (#4074)Richard van der Hoff2018-10-233-15/+23
| | | | | | | | | | Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064
* Fix bug where we raised StopIteration in a generatorErik Johnston2018-10-171-2/+5
| | | | This made python 3.7 unhappy
* Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_cachesRichard van der Hoff2018-09-251-1/+0
|\ | | | | Fix some instances of ExpiringCache not expiring cache items
| * Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | | | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* | added "media-src: 'self'" to CSP for resources (#3578)Jérémy Farnaud2018-09-251-0/+1
|/ | | | | | | | | | | | | | | Synapse doesn’t allow for media resources to be played directly from Chrome. It is a problem for users on other networks (e.g. IRC) communicating with Matrix users through a gateway. The gateway sends them the raw URL for the resource when a Matrix user uploads a video and the video cannot be played directly in Chrome using that URL. Chrome argues it is not authorized to play the video because of the Content Security Policy. Chrome checks for the "media-src" policy which is missing, and defauts to the "default-src" policy which is "none". As Synapse already sends "object-src: 'self'" I thought it wouldn’t be a problem to add "media-src: 'self'" to the CSP to fix this problem.
* Port rest/ to Python 3 (#3823)Amber Brown2018-09-125-40/+45
|
* Port over enough to get some sytests running on Python 3 (#3668)Amber Brown2018-08-201-5/+5
|
* Add GET media/v1/config (#3184)Will Hunt2018-08-162-1/+50
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-102-2/+2
|
* Refactor code for turning HttpResponseException into SynapseErrorRichard van der Hoff2018-08-011-1/+1
| | | | | | | | This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call.
* Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-021-1/+1
|
* Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-262-2/+2
| | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
* Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-252-2/+14
| | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
* check isort by travisKrombel2018-07-161-2/+2
|
* Use parse_{int,str} and assert from http.servletKrombel2018-07-133-6/+10
| | | | | | | parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable
* run isortAmber Brown2018-07-0912-92/+79
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-282-2/+4
|
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-222-3/+7
|
* Merge pull request #3244 from NotAFile/py3-six-4Amber Brown2018-05-241-1/+2
|\ | | | | replace some iteritems with six
| * replace some iteritems with sixAdrian Tschira2018-05-191-1/+2
| | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | py3-ize url previewAdrian Tschira2018-05-191-3/+5
|/
* Set Server header in SynapseRequestRichard van der Hoff2018-05-104-5/+1
| | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.
* Remove redundant request_handler decoratorRichard van der Hoff2018-05-104-26/+33
| | | | | | This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler.
* Fix logcontext leak in media repoRichard van der Hoff2018-05-022-1/+4
| | | | Make FileResponder.write_to_consumer uphold the logcontext contract
* Merge pull request #3155 from NotAFile/py3-bytes-1Richard van der Hoff2018-04-301-3/+3
|\ | | | | more bytes strings
| * more bytes stringsAdrian Tschira2018-04-291-3/+3
| | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Merge pull request #3140 from matrix-org/rav/use_run_in_backgroundRichard van der Hoff2018-04-301-2/+3
|\ \ | | | | | | Use run_in_background in preference to preserve_fn
| * | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-271-2/+7
| |\|
| * | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-271-2/+3
| | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* | | Merge pull request #3108 from NotAFile/py3-six-urlparseRichard van der Hoff2018-04-302-2/+2
|\ \ \ | |_|/ |/| | Use six.moves.urlparse
| * | Use six.moves.urlparseAdrian Tschira2018-04-152-2/+2
| |/ | | | | | | | | | | The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>
* / Improve exception handling for background processesRichard van der Hoff2018-04-271-2/+7
|/ | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* Replace old-style raise with six.reraiseAdrian Tschira2018-04-061-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com>
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-191-1/+1
|\
| * Replace ujson with simplejsonErik Johnston2018-03-151-1/+1
| |
* | Make store_file use store_into_fileErik Johnston2018-02-141-26/+11
| |
* | Tell storage providers about new file so they can uploadErik Johnston2018-02-071-0/+6
| |
* | Merge pull request #2791 from matrix-org/erikj/media_storage_refactorErik Johnston2018-02-053-12/+47
|\ \ | | | | | | Ensure media is in local cache before thumbnailing
| * | Use better file consumerErik Johnston2018-01-181-5/+10
| | |
| * | Ensure media is in local cache before thumbnailingErik Johnston2018-01-183-10/+40
| | |
* | | Handle url_previews with no content-typeRichard van der Hoff2018-02-021-21/+34
| | | | | | | | | | | | | | | | | | | | | avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.
* | | Add federation_domain_whitelist option (#2820)Matthew Hodgson2018-01-221-2/+17
| | | | | | | | | | | | | | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-18 14:30:57 +0000