summary refs log tree commit diff
path: root/synapse/rest/media (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix additional type hints from Twisted 21.2.0. (#9591)Patrick Cloke2021-03-123-3/+6
|
* Convert Requester to attrs (#9586)Richard van der Hoff2021-03-101-1/+2
| | | | | | ... because namedtuples suck Fix up a couple of other annotations to keep mypy happy.
* Handle image transparency better when thumbnailing. (#9473)Patrick Cloke2021-03-091-3/+8
| | | | Properly uses RGBA mode for 1- and 8-bit images with transparency (instead of RBG mode).
* Use the proper Request in type hints. (#9515)Patrick Cloke2021-03-017-17/+22
| | | | This also pins the Twisted version in the mypy job for CI until proper type hints are fixed throughout Synapse.
* Add support for no_proxy and case insensitive env variables (#9372)Tim Leung2021-02-261-2/+1
| | | | | | | | | | | | | | | ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2519) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](https://github.com/python/cpython/blob/bdb941be423bde8b02a5695ccf51c303d6204bed/Lib/urllib/request.py#L2488) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk
* Return a 404 if we don't have the original fileErik Johnston2021-02-192-1/+8
|
* Regenerate exact thumbnails if missingErik Johnston2021-02-182-2/+49
|
* Update black, and run auto formatting over the codebase (#9381)Eric Eastwood2021-02-164-12/+26
| | | | | | | - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version
* Fix some typos.Patrick Cloke2021-02-122-5/+5
|
* Handle additional errors when previewing URLs. (#9333)Patrick Cloke2021-02-081-19/+52
| | | | * Handle the case of lxml not finding a document tree. * Parse the document encoding from the XML tag.
* Add check_media_file_for_spam spam checker hookErik Johnston2021-02-042-6/+65
|
* Try to recover from unknown encodings when previewing media. (#9164)Patrick Cloke2021-01-261-10/+34
| | | | | | Treat unknown encodings (according to lxml) as UTF-8 when generating a preview for HTML documents. This isn't fully accurate, but will hopefully give a reasonable title and summary.
* Return a 404 if no valid thumbnail is found. (#9163)Patrick Cloke2021-01-212-81/+158
| | | | | | If no thumbnail of the requested type exists, return a 404 instead of erroring. This doesn't quite match the spec (which does not define what happens if no thumbnail can be found), but is consistent with what Synapse already does.
* Add type hints to media rest resources. (#9093)Patrick Cloke2021-01-1511-164/+283
|
* Allow spam-checker modules to be provide async methods. (#8890)David Teller2020-12-111-10/+6
| | | | Spam checker modules can now provide async methods. This is implemented in a backwards-compatible manner.
* Add X-Robots-Tag header to stop crawlers from indexing media (#8887)Aaron Raimist2020-12-081-0/+5
| | | | | | | Fixes / related to: https://github.com/matrix-org/synapse/issues/6533 This should do essentially the same thing as a robots.txt file telling robots to not index the media repo. https://developers.google.com/search/reference/robots_meta_tag Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Do not raise a 500 exception when previewing empty media. (#8883)Patrick Cloke2020-12-071-1/+5
|
* Do not 500 if the content-length is not provided when uploading media. (#8862)Patrick Cloke2020-12-041-1/+1
| | | Instead return the proper 400 error.
* Apply an IP range blacklist to push and key revocation requests. (#8821)Patrick Cloke2020-12-021-1/+1
| | | | | | | | | | | | Replaces the `federation_ip_range_blacklist` configuration setting with an `ip_range_blacklist` setting with wider scope. It now applies to: * Federation * Identity servers * Push notifications * Checking key validitity for third-party invite events The old `federation_ip_range_blacklist` setting is still honored if present, but with reduced scope (it only applies to federation and identity servers).
* Fix race for concurrent downloads of remote media. (#8682)Erik Johnston2020-10-302-70/+125
| | | Fixes #6755
* Add an admin api to delete local media. (#8519)Dirk Klimpel2020-10-262-1/+88
| | | | | | Related to: #6459, #3479 Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server.
* Don't bother responding to client requests that have already disconnected ↵Andrew Morgan2020-10-061-0/+6
| | | | | | | | | | (#8465) This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so. This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003. Fixes https://github.com/matrix-org/synapse/issues/6700 Fixes https://github.com/matrix-org/synapse/issues/5304
* Fix bug in remote thumbnail search (#8438)Richard van der Hoff2020-10-021-20/+23
| | | | | #7124 changed the behaviour of remote thumbnails so that the thumbnailing method was included in the filename of the thumbnail. To support existing files, it included a fallback so that we would check the old filename if the new filename didn't exist. Unfortunately, it didn't apply this logic to storage providers, so any thumbnails stored on such a storage provider was broken.
* fix a logging error in thumbnailer (#8435)Richard van der Hoff2020-10-011-1/+1
| | | Introduced in #8236
* Discard an empty upload_name before persisting an uploaded file (#7905)Will Hunt2020-09-292-3/+8
|
* Add type annotations to SimpleHttpClient (#8372)Richard van der Hoff2020-09-241-4/+10
|
* Use slots in attrs classes where possible (#8296)Patrick Cloke2020-09-141-1/+1
| | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time.
* Fix typos in comments.Patrick Cloke2020-09-141-2/+2
|
* Do not error when thumbnailing invalid files (#8236)Patrick Cloke2020-09-093-12/+76
| | | | If a file cannot be thumbnailed for some reason (e.g. the file is empty), then catch the exception and convert it to a reasonable error message for the client.
* Include method in thumbnail media name (#7124)DeepBlueV7.X2020-09-082-1/+46
| | | | | | | | | | | | | | This fixes an issue where different methods (crop/scale) overwrite each other. This first tries the new path. If that fails and we are looking for a remote thumbnail, it tries the old path. If that still isn't found, it continues as normal. This should probably be removed in the future, after some of the newer thumbnails were generated with the new path on most deployments. Then the overhead should be minimal if the other thumbnails need to be regenerated. Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
* Stop sub-classing object (#8249)Patrick Cloke2020-09-045-6/+6
|
* Remove unnecessary maybeDeferred calls (#8044)Patrick Cloke2020-08-071-2/+1
|
* Reduce unnecessary whitespace in JSON. (#7372)David Vo2020-08-071-2/+2
|
* Rename database classes to make some sense (#8033)Erik Johnston2020-08-051-1/+1
|
* Fix async/await calls for broken media providers. (#8027)Patrick Cloke2020-08-042-22/+20
|
* Convert the remaining media repo code to async / await. (#7947)Patrick Cloke2020-07-275-107/+130
|
* Support oEmbed for media previews. (#7920)Patrick Cloke2020-07-271-45/+220
| | | Fixes previews of Twitter URLs by using their oEmbed endpoint to grab content.
* Convert more of the media code to async/await (#7873)Patrick Cloke2020-07-242-33/+42
|
* isort 5 compatibility (#7786)Will Hunt2020-07-051-2/+1
| | | The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse.
* Merge different Resource implementation classes (#7732)Erik Johnston2020-07-035-46/+14
|
* Support running multiple media repos. (#7706)Erik Johnston2020-06-171-3/+15
| | | | | This requires a new config option to specify which media repo should be responsible for running background jobs to e.g. clear out expired URL preview caches.
* Fetch from the r0 media path instead of the unspecced v1. (#7714)Patrick Cloke2020-06-171-4/+4
|
* Replace all remaining six usage with native Python 3 equivalents (#7704)Dagfinn Ilmari Mannsåker2020-06-163-13/+5
|
* Replace iteritems/itervalues/iterkeys with native versions. (#7692)Patrick Cloke2020-06-151-3/+1
|
* remove miscellaneous PY2 codeRichard van der Hoff2020-05-151-18/+9
|
* Reduce logging verbosity of URL cache cleanup. (#7295)Michael Kaye2020-04-221-2/+7
|
* Allow specifying the value of Accept-Language header for URL previews (#7265)Andrew Morgan2020-04-151-2/+6
|
* Set Referrer-Policy to no-referrer for media (#7009)Dionysis Grigoropoulos2020-03-231-0/+3
|
* Convert some of the media REST code to async/await (#7110)Patrick Cloke2020-03-203-111/+90
|
* Set charset to utf-8 when adding headers for certain text content types (#7044)The Stranjer2020-03-171-1/+24
| | | | Fixes #7043
* Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)Patrick Cloke2020-02-211-21/+19
| | | | Ensure good comprehension hygiene using flake8-comprehensions.
* a bit of debugging for media storage providers (#6757)Richard van der Hoff2020-01-232-0/+7
| | | | | | * a bit of debugging for media storage providers * changelog
* Fix typo in _select_thumbnailBrendan Abolivier2020-01-221-1/+1
|
* Fixup synapse.rest to pass mypy (#6732)Erik Johnston2020-01-203-11/+13
|
* Remove unused, undocumented "content repo" resource (#6628)Richard van der Hoff2020-01-032-103/+0
| | | | | | This looks like it got half-killed back in #888. Fixes #6567.
* Move background update handling out of storeErik Johnston2019-12-051-1/+1
|
* Fix: Pillow error when uploading RGBA image (#3325) (#6241)Filip Štědronský2019-12-021-1/+4
| | | Signed-Off-By: Filip Štědronský <g@regnarg.cz>
* Fix startup error when http proxy is defined. (#6421)Richard van der Hoff2019-11-261-2/+2
| | | | | Guess I only tested this on python 2 :/ Fixes #6419.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2Richard van der Hoff2019-11-051-1/+2
|\ | | | | Fix exception when OpenGraph tag values are ints
| * Fix exception when OpenGraph tag values are intsRichard van der Hoff2019-11-051-1/+2
| |
* | Merge branch 'develop' into rav/url_preview_limit_titleRichard van der Hoff2019-11-053-20/+26
|\ \ | |/ |/|
| * Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-011-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
| * Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-313-15/+17
| | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
| * Address codestyle concernsMichael Kaye2019-10-241-1/+3
| |
| * Delay printf until logging is required.Michael Kaye2019-10-241-6/+6
| | | | | | | | | | Using % will cause the string to be generated even if debugging is off.
* | Apply suggestions from code reviewRichard van der Hoff2019-11-051-1/+1
| | | | | | | | Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-Authored-By: Erik Johnston <erik@matrix.org>
* | Strip overlong OpenGraph data from url previewRichard van der Hoff2019-11-051-1/+19
|/ | | | ... to stop people causing DoSes with malicious web pages
* Fix store_url_cache using bytesErik Johnston2019-10-101-2/+2
|
* add M_TOO_LARGE error code for uploading a too large file (#6151)Anshul Angaria2019-10-081-2/+6
| | | Fixes #6109
* Log responder we are using. (#6139)Michael Kaye2019-10-071-1/+1
| | | This prevents us logging "Responding to media request with responder %s".
* media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142)Robert Swain2019-10-041-3/+11
| | | | | | Pillow will use nearest neighbour as the resampling algorithm if the source image is either 1-bit or a color palette using 8 bits. If we convert to RGB before scaling, we'll probably get a better result.
* Remove double return statements (#5962)Andrew Morgan2019-09-031-1/+0
| | | | | | | | | | Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.
* fix thumbnail storage location (#5915)L0ric02019-09-021-1/+1
| | | | | | | | | | | | | | * fix thumbnail storage location Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Add changelog file. Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Update Changelog Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-302-4/+4
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Don't load the media repo when configured to use an external media repo (#5754)Amber Brown2019-08-131-1/+5
|
* Replace returnValue with return (#5736)Amber Brown2019-07-233-33/+31
|
* fix typo: backgroud -> backgroundAndrew Morgan2019-07-121-1/+1
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-045-17/+13
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-0/+1
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-295-68/+47
|
* Run Black. (#5482)Amber Brown2019-06-2012-408/+425
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-2/+4
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Fix error when downloading thumbnail with width/height param missing (#5258)Aaron Raimist2019-05-291-2/+2
| | | | | | | Fix error when downloading thumbnail with width/height param missing Fixes #2748 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Fix image orientation when generating thumbnail (#5039)PauRE2019-05-162-0/+44
|
* Migrate all tests to use the dict-based config format instead of hanging ↵Amber Brown2019-05-131-0/+1
| | | | items off HomeserverConfig (#5171)
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-0/+10
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Prevent "producer not unregistered" message (#5009)Amber Brown2019-04-241-0/+4
|
* Remove periods from copyright headers (#5046)Andrew Morgan2019-04-111-1/+1
|
* fix incorrect encoding of filenames with spaces in (#2090)Matthew Hodgson2019-03-111-3/+51
| | | | fixes https://github.com/vector-im/riot-web/issues/3155
* Fix parsing of Content-Disposition headers (#4763)Richard van der Hoff2019-02-271-20/+65
| | | | | | | | | | | * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings
* Don't log stack trace when client has gone away during media download (#4738)Erik Johnston2019-02-251-2/+9
| | | | | | | | * Don't log stack trace when client has gone away during media download * Newsfile * Fixup newsfile
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-3/+4
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Fix IP URL previews on Python 3 (#4215)Amber Brown2018-12-221-2/+12
|
* Add 'sandbox' to CSP for media repo (#4284)David Baker2018-12-111-1/+2
| | | | | | * Add 'sandbox' to the CSP for media repo * Changelog
* Move imports to one lineWill Hunt2018-12-101-4/+1
|
* Use `send_cors`Will Hunt2018-12-101-4/+1
|
* changelog & isortWill Hunt2018-12-091-1/+5
|
* Make /config more CORS-yWill Hunt2018-12-091-1/+3
|
* Fix Content-Disposition in media repository (#4176)Amber Brown2018-11-153-90/+110
|
* Use <meta> tags to discover the per-page encoding of html previews (#4183)Amber Brown2018-11-151-9/+22
|
* Fix URL preview bugs (type error when loading cache from db, content-type ↵Amber Brown2018-11-081-6/+16
| | | | including quotes) (#4157)
* Delete the disused & unspecced identicon functionality (#4106)Amber Brown2018-10-292-70/+0
|
* Fix a number of flake8 errorsRichard van der Hoff2018-10-241-1/+1
| | | | | | | | | | | | Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.
* Correctly account for cpu usage by background threads (#4074)Richard van der Hoff2018-10-233-15/+23
| | | | | | | | | | Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064
* Fix bug where we raised StopIteration in a generatorErik Johnston2018-10-171-2/+5
| | | | This made python 3.7 unhappy
* Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_cachesRichard van der Hoff2018-09-251-1/+0
|\ | | | | Fix some instances of ExpiringCache not expiring cache items
| * Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | | | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* | added "media-src: 'self'" to CSP for resources (#3578)Jérémy Farnaud2018-09-251-0/+1
|/ | | | | | | | | | | | | | | Synapse doesn’t allow for media resources to be played directly from Chrome. It is a problem for users on other networks (e.g. IRC) communicating with Matrix users through a gateway. The gateway sends them the raw URL for the resource when a Matrix user uploads a video and the video cannot be played directly in Chrome using that URL. Chrome argues it is not authorized to play the video because of the Content Security Policy. Chrome checks for the "media-src" policy which is missing, and defauts to the "default-src" policy which is "none". As Synapse already sends "object-src: 'self'" I thought it wouldn’t be a problem to add "media-src: 'self'" to the CSP to fix this problem.
* Port rest/ to Python 3 (#3823)Amber Brown2018-09-125-40/+45
|
* Port over enough to get some sytests running on Python 3 (#3668)Amber Brown2018-08-201-5/+5
|
* Add GET media/v1/config (#3184)Will Hunt2018-08-162-1/+50
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-102-2/+2
|
* Refactor code for turning HttpResponseException into SynapseErrorRichard van der Hoff2018-08-011-1/+1
| | | | | | | | This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call.
* Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-021-1/+1
|
* Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-262-2/+2
| | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
* Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-252-2/+14
| | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
* check isort by travisKrombel2018-07-161-2/+2
|
* Use parse_{int,str} and assert from http.servletKrombel2018-07-133-6/+10
| | | | | | | parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable
* run isortAmber Brown2018-07-0912-92/+79
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-282-2/+4
|
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-222-3/+7
|
* Merge pull request #3244 from NotAFile/py3-six-4Amber Brown2018-05-241-1/+2
|\ | | | | replace some iteritems with six
| * replace some iteritems with sixAdrian Tschira2018-05-191-1/+2
| | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | py3-ize url previewAdrian Tschira2018-05-191-3/+5
|/
* Set Server header in SynapseRequestRichard van der Hoff2018-05-104-5/+1
| | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.
* Remove redundant request_handler decoratorRichard van der Hoff2018-05-104-26/+33
| | | | | | This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler.
* Fix logcontext leak in media repoRichard van der Hoff2018-05-022-1/+4
| | | | Make FileResponder.write_to_consumer uphold the logcontext contract
* Merge pull request #3155 from NotAFile/py3-bytes-1Richard van der Hoff2018-04-301-3/+3
|\ | | | | more bytes strings
| * more bytes stringsAdrian Tschira2018-04-291-3/+3
| | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Merge pull request #3140 from matrix-org/rav/use_run_in_backgroundRichard van der Hoff2018-04-301-2/+3
|\ \ | | | | | | Use run_in_background in preference to preserve_fn
| * | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-271-2/+7
| |\|
| * | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-271-2/+3
| | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* | | Merge pull request #3108 from NotAFile/py3-six-urlparseRichard van der Hoff2018-04-302-2/+2
|\ \ \ | |_|/ |/| | Use six.moves.urlparse
| * | Use six.moves.urlparseAdrian Tschira2018-04-152-2/+2
| |/ | | | | | | | | | | The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>
* / Improve exception handling for background processesRichard van der Hoff2018-04-271-2/+7
|/ | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* Replace old-style raise with six.reraiseAdrian Tschira2018-04-061-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com>
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-191-1/+1
|\
| * Replace ujson with simplejsonErik Johnston2018-03-151-1/+1
| |
* | Make store_file use store_into_fileErik Johnston2018-02-141-26/+11
| |
* | Tell storage providers about new file so they can uploadErik Johnston2018-02-071-0/+6
| |
* | Merge pull request #2791 from matrix-org/erikj/media_storage_refactorErik Johnston2018-02-053-12/+47
|\ \ | | | | | | Ensure media is in local cache before thumbnailing
| * | Use better file consumerErik Johnston2018-01-181-5/+10
| | |
| * | Ensure media is in local cache before thumbnailingErik Johnston2018-01-183-10/+40
| | |
* | | Handle url_previews with no content-typeRichard van der Hoff2018-02-021-21/+34
| | | | | | | | | | | | | | | | | | | | | avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.
* | | Add federation_domain_whitelist option (#2820)Matthew Hodgson2018-01-221-2/+17
| | | | | | | | | | | | | | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
* | | Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbsRichard van der Hoff2018-01-191-0/+8
|\ \ \ | | | | | | | | Use the right path for url_preview thumbnails
| * | | Use the right path for url_preview thumbnailsRichard van der Hoff2018-01-191-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This was introduced by #2627: we were overwriting the original media for url previews with the thumbnails :/ (fixes https://github.com/vector-im/riot-web/issues/6012, hopefully)
* | | | Fix storage provider bug introduced when renamed to store_localErik Johnston2018-01-181-1/+1
| | | |
* | | | Merge pull request #2812 from matrix-org/erikj/media_storage_provider_configErik Johnston2018-01-182-20/+27
|\ \ \ \ | | | | | | | | | | Make storage providers configurable
| * | | | Missing staticmethodErik Johnston2018-01-181-0/+1
| | | | |
| * | | | Fixup commentsErik Johnston2018-01-181-2/+2
| | | | |
| * | | | Make storage providers more configurableErik Johnston2018-01-182-20/+26
| | |/ / | |/| |
* | | | Fix typo in thumbnail resource causing access times to be incorrectErik Johnston2018-01-181-2/+2
| |/ / |/| |
* | | Update last access time when thumbnails are viewedErik Johnston2018-01-171-0/+2
| | |
* | | Keep track of last access time for local mediaErik Johnston2018-01-171-7/+25
|/ /
* | Add docstringErik Johnston2018-01-171-1/+1
| |
* | Use local varsErik Johnston2018-01-161-6/+2
| |
* | Change _generate_thumbnails to take media_typeErik Johnston2018-01-162-7/+10
| |
* | Move setting of file_id up to callerErik Johnston2018-01-162-13/+14
| |
* | Fix typoErik Johnston2018-01-161-7/+10
| |
* | Fix up log linesErik Johnston2018-01-162-7/+10
| |
* | Correctly use server_name/file_id when generating/fetching remote thumbnailsErik Johnston2018-01-162-5/+8
| |
* | Log when we respond with 404Erik Johnston2018-01-162-1/+8
| |
* | Fix thumbnailing remote filesErik Johnston2018-01-162-2/+30
| |
* | Merge pull request #2767 from matrix-org/erikj/media_storage_refactorErik Johnston2018-01-167-346/+843
|\ \ | | | | | | Refactor MediaRepository to separate out storage
| * | Fix up commentsErik Johnston2018-01-122-7/+16
| | |
| * | Correctly reraise exceptionErik Johnston2018-01-121-2/+4
| | |
| * | Make Responder a context managerErik Johnston2018-01-122-9/+10
| | |
| * | Add missing class varErik Johnston2018-01-121-0/+3
| | |
| * | Remove unnecessary conditionErik Johnston2018-01-121-4/+1
| | |
| * | Remove unused variablesErik Johnston2018-01-121-3/+0
| | |
| * | Make class var localErik Johnston2018-01-121-3/+3
| | |
| * | CommentsErik Johnston2018-01-123-15/+65
| | |
| * | Add StorageProvider conceptErik Johnston2018-01-093-9/+162
| | |
| * | Make PreviewUrlResource use MediaStorageErik Johnston2018-01-092-8/+14
| | |
| * | Make ThumbnailResource use MediaStorageErik Johnston2018-01-092-48/+68
| | |
| * | Use MediaStorage for remote mediaErik Johnston2018-01-092-136/+156
| | |
| * | Use MediaStorage for local filesErik Johnston2018-01-092-119/+73
| | |
| * | Add MediaStorage classErik Johnston2018-01-091-0/+198
| | |
| * | Add some helper classesErik Johnston2018-01-091-0/+73
| | |
| * | Split out add_file_headersErik Johnston2018-01-091-28/+42
| | |
* | | Reinstate media download on thumbnail requestRichard van der Hoff2018-01-121-0/+5
|/ / | | | | | | | | We need to actually download the remote media when we get a request for a thumbnail.
* | Remove dead TODOErik Johnston2018-01-091-2/+0
| |
* | Remove dead code related to default thumbnailsErik Johnston2018-01-091-73/+3
|/
* Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_errorRichard van der Hoff2017-11-271-3/+7
|\ | | | | Fix error on sqlite 3.7
| * Check database in has_completed_background_updatesRichard van der Hoff2017-11-221-1/+1
| | | | | | | | so that the right thing happens on workers.
| * Fix error on sqlite 3.7Richard van der Hoff2017-11-211-3/+7
| | | | | | | | | | | | | | | | | | | | | | Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.
* | Fix OPTIONS on preview_urlRichard van der Hoff2017-11-231-1/+5
|/ | | | Fixes #2706
* Bit more loggingRichard van der Hoff2017-11-101-0/+2
|
* Cache failures in url_preview handlerRichard van der Hoff2017-11-101-41/+45
| | | | | Reshuffle the caching logic in the url_preview handler so that failures are cached (and to generally simplify things and fix the logcontext leaks).
* Ignore noscript tags when generating URL previewsMaxime Vaillancourt2017-10-251-1/+8
|
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-233-5/+5
| | | | what could possibly go wrong
* Fix a logcontext leak in the media repoRichard van der Hoff2017-10-231-1/+4
|
* Fix typo in thumbnail generationErik Johnston2017-10-191-2/+5
|
* fix thumbnailing (#2548)Krombel2017-10-171-2/+2
| | | | | | | in commit 0e28281a the code for thumbnailing got refactored and the renaming of this variables was not done correctly. Signed-Off-by: Matthias Kesler <krombel@krombel.de>
* Add try/finally block to close t_byte_sourceErik Johnston2017-10-131-24/+41
|
* Remove old commentErik Johnston2017-10-131-1/+0
|
* Don't reuse source but instead copy from primary media store to backupErik Johnston2017-10-132-22/+9
|
* PEP8Erik Johnston2017-10-131-1/+1
|
* Move type into keyErik Johnston2017-10-131-3/+3
|
* CommentErik Johnston2017-10-131-1/+1
|
* Remove spurious additionErik Johnston2017-10-131-4/+0
|
* Don't needlessly rename variableErik Johnston2017-10-131-13/+13
|
* Add paths back in...Erik Johnston2017-10-131-4/+4
|
* Remove unnecessary diffErik Johnston2017-10-131-3/+1
|
* Make things be absolute paths againErik Johnston2017-10-131-7/+7
|
* s/write_to_file/write_to_file_and_backup/Erik Johnston2017-10-131-5/+5
|
* Fix upErik Johnston2017-10-132-43/+32
|
* Fix up thumbnailing functionErik Johnston2017-10-133-75/+73
|
* Fix typoErik Johnston2017-10-131-1/+1
|
* Fix up commentsErik Johnston2017-10-131-2/+26
|
* Move makedirs to threadErik Johnston2017-10-131-2/+1
|
* Use make_deferred_yieldableErik Johnston2017-10-131-7/+7
|
* Fix commentsErik Johnston2017-10-122-1/+4
|
* Don't close prematurely..Erik Johnston2017-10-121-2/+5
|
* getvalue closes bufferErik Johnston2017-10-121-11/+10
|
* Get len before closeErik Johnston2017-10-121-6/+11
|
* Don't close file prematurelyErik Johnston2017-10-122-8/+18
|
* Copy everything to backupErik Johnston2017-10-124-76/+148
|
* TypoErik Johnston2017-10-121-1/+1
|
* Remove dead codeErik Johnston2017-10-121-3/+0
|
* Fix typosErik Johnston2017-10-121-20/+26
|
* Fix typoErik Johnston2017-10-121-1/+1
|
* Basic implementation of backup media storeErik Johnston2017-10-123-126/+113
|
* Only log if we've removed mediaErik Johnston2017-09-281-2/+4
|
* Clear up expired url cache every 10sErik Johnston2017-09-281-1/+1
|
* Actually assign result to variableErik Johnston2017-09-281-1/+1
|
* Fix typo in url cache expiry timerErik Johnston2017-09-281-1/+1
|
* Support new and old style media id formatsErik Johnston2017-09-281-31/+81
|
* Change expires column to expires_tsErik Johnston2017-09-281-2/+2
|
* Delete expired url cache dataErik Johnston2017-09-282-4/+129
|
* Store URL cache preview downloads seperatelyErik Johnston2017-06-235-18/+72
| | | | This makes it easier to clear old media out at a later date
* Add API to quarantine mediaErik Johnston2017-06-193-3/+5
|
* Throw exception when not retrying when downloading mediaErik Johnston2017-06-131-0/+1
|
* actually trim oversize og:description metaMatthew Hodgson2017-05-221-0/+2
|
* Don't log exceptions for NotRetryingDestinationErik Johnston2017-05-151-1/+3
|
* Address review commentsRichard van der Hoff2017-03-141-15/+17
| | | | | | | - don't blindly proxy all HTTPRequestExceptions - log unexpected exceptions at error - avoid `isinstance` - improve docs on `from_http_response_exception`
* Fix routing loop when fetching remote mediaRichard van der Hoff2017-03-132-4/+38
| | | | | | | | | | | When we proxy a media request to a remote server, add a query-param, which will tell the remote server to 404 if it doesn't recognise the server_name. This should fix a routing loop where the server keeps forwarding back to itself. Also improves the error handling on remote media fetches, so that we don't always return a rather obscure 502.
* Fix dynamic thumbnails aspectJurek2017-02-241-0/+3
|
* Log which files we saved attachments to in the media_repositoryMark Haines2017-01-103-0/+11
|
* Name linearizer's for better logsErik Johnston2017-01-091-1/+1
|
* Fix crash in url preview when html tag has no textMarcin Bachry2016-12-141-1/+4
| | | | Signed-off-by: Marcin Bachry <hegel666@gmail.com>
* preview_url_resource: Ellipsis must be in unicode stringJohannes Löthberg2016-12-011-1/+1
| | | | Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
* Set CORs headers on responses from the media repoMark Haines2016-11-022-2/+4
|
* Add quotes and be explicity about script-srcErik Johnston2016-09-051-1/+2
|
* Allow PDF to be rendered from media repoErik Johnston2016-09-051-1/+7
|
* Set `Content-Security-Policy` on media repoErik Johnston2016-08-171-0/+1
| | | | | This is to inform browsers that they should sandbox the returned media. This is particularly cruical for javascript/HTML files.
* Add None check to _iterate_over_textErik Johnston2016-08-171-1/+1
|
* Flake8Erik Johnston2016-08-161-4/+1
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-05 18:52:27 +0000