Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Don't bother responding to client requests that have already disconnected ↵ | Andrew Morgan | 2020-10-06 | 1 | -0/+6 |
| | | | | | | | | | | (#8465) This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so. This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003. Fixes https://github.com/matrix-org/synapse/issues/6700 Fixes https://github.com/matrix-org/synapse/issues/5304 | ||||
* | Fix bug in remote thumbnail search (#8438) | Richard van der Hoff | 2020-10-02 | 1 | -20/+23 |
| | | | | | #7124 changed the behaviour of remote thumbnails so that the thumbnailing method was included in the filename of the thumbnail. To support existing files, it included a fallback so that we would check the old filename if the new filename didn't exist. Unfortunately, it didn't apply this logic to storage providers, so any thumbnails stored on such a storage provider was broken. | ||||
* | fix a logging error in thumbnailer (#8435) | Richard van der Hoff | 2020-10-01 | 1 | -1/+1 |
| | | | Introduced in #8236 | ||||
* | Discard an empty upload_name before persisting an uploaded file (#7905) | Will Hunt | 2020-09-29 | 2 | -3/+8 |
| | |||||
* | Add type annotations to SimpleHttpClient (#8372) | Richard van der Hoff | 2020-09-24 | 1 | -4/+10 |
| | |||||
* | Use slots in attrs classes where possible (#8296) | Patrick Cloke | 2020-09-14 | 1 | -1/+1 |
| | | | | | slots use less memory (and attribute access is faster) while slightly limiting the flexibility of the class attributes. This focuses on objects which are instantiated "often" and for short periods of time. | ||||
* | Fix typos in comments. | Patrick Cloke | 2020-09-14 | 1 | -2/+2 |
| | |||||
* | Do not error when thumbnailing invalid files (#8236) | Patrick Cloke | 2020-09-09 | 3 | -12/+76 |
| | | | | If a file cannot be thumbnailed for some reason (e.g. the file is empty), then catch the exception and convert it to a reasonable error message for the client. | ||||
* | Include method in thumbnail media name (#7124) | DeepBlueV7.X | 2020-09-08 | 2 | -1/+46 |
| | | | | | | | | | | | | | | This fixes an issue where different methods (crop/scale) overwrite each other. This first tries the new path. If that fails and we are looking for a remote thumbnail, it tries the old path. If that still isn't found, it continues as normal. This should probably be removed in the future, after some of the newer thumbnails were generated with the new path on most deployments. Then the overhead should be minimal if the other thumbnails need to be regenerated. Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de> | ||||
* | Stop sub-classing object (#8249) | Patrick Cloke | 2020-09-04 | 5 | -6/+6 |
| | |||||
* | Remove unnecessary maybeDeferred calls (#8044) | Patrick Cloke | 2020-08-07 | 1 | -2/+1 |
| | |||||
* | Reduce unnecessary whitespace in JSON. (#7372) | David Vo | 2020-08-07 | 1 | -2/+2 |
| | |||||
* | Rename database classes to make some sense (#8033) | Erik Johnston | 2020-08-05 | 1 | -1/+1 |
| | |||||
* | Fix async/await calls for broken media providers. (#8027) | Patrick Cloke | 2020-08-04 | 2 | -22/+20 |
| | |||||
* | Convert the remaining media repo code to async / await. (#7947) | Patrick Cloke | 2020-07-27 | 5 | -107/+130 |
| | |||||
* | Support oEmbed for media previews. (#7920) | Patrick Cloke | 2020-07-27 | 1 | -45/+220 |
| | | | Fixes previews of Twitter URLs by using their oEmbed endpoint to grab content. | ||||
* | Convert more of the media code to async/await (#7873) | Patrick Cloke | 2020-07-24 | 2 | -33/+42 |
| | |||||
* | isort 5 compatibility (#7786) | Will Hunt | 2020-07-05 | 1 | -2/+1 |
| | | | The CI appears to use the latest version of isort, which is a problem when isort gets a major version bump. Rather than try to pin the version, I've done the necessary to make isort5 happy with synapse. | ||||
* | Merge different Resource implementation classes (#7732) | Erik Johnston | 2020-07-03 | 5 | -46/+14 |
| | |||||
* | Support running multiple media repos. (#7706) | Erik Johnston | 2020-06-17 | 1 | -3/+15 |
| | | | | | This requires a new config option to specify which media repo should be responsible for running background jobs to e.g. clear out expired URL preview caches. | ||||
* | Fetch from the r0 media path instead of the unspecced v1. (#7714) | Patrick Cloke | 2020-06-17 | 1 | -4/+4 |
| | |||||
* | Replace all remaining six usage with native Python 3 equivalents (#7704) | Dagfinn Ilmari Mannsåker | 2020-06-16 | 3 | -13/+5 |
| | |||||
* | Replace iteritems/itervalues/iterkeys with native versions. (#7692) | Patrick Cloke | 2020-06-15 | 1 | -3/+1 |
| | |||||
* | remove miscellaneous PY2 code | Richard van der Hoff | 2020-05-15 | 1 | -18/+9 |
| | |||||
* | Reduce logging verbosity of URL cache cleanup. (#7295) | Michael Kaye | 2020-04-22 | 1 | -2/+7 |
| | |||||
* | Allow specifying the value of Accept-Language header for URL previews (#7265) | Andrew Morgan | 2020-04-15 | 1 | -2/+6 |
| | |||||
* | Set Referrer-Policy to no-referrer for media (#7009) | Dionysis Grigoropoulos | 2020-03-23 | 1 | -0/+3 |
| | |||||
* | Convert some of the media REST code to async/await (#7110) | Patrick Cloke | 2020-03-20 | 3 | -111/+90 |
| | |||||
* | Set charset to utf-8 when adding headers for certain text content types (#7044) | The Stranjer | 2020-03-17 | 1 | -1/+24 |
| | | | | Fixes #7043 | ||||
* | Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) | Patrick Cloke | 2020-02-21 | 1 | -21/+19 |
| | | | | Ensure good comprehension hygiene using flake8-comprehensions. | ||||
* | a bit of debugging for media storage providers (#6757) | Richard van der Hoff | 2020-01-23 | 2 | -0/+7 |
| | | | | | | * a bit of debugging for media storage providers * changelog | ||||
* | Fix typo in _select_thumbnail | Brendan Abolivier | 2020-01-22 | 1 | -1/+1 |
| | |||||
* | Fixup synapse.rest to pass mypy (#6732) | Erik Johnston | 2020-01-20 | 3 | -11/+13 |
| | |||||
* | Remove unused, undocumented "content repo" resource (#6628) | Richard van der Hoff | 2020-01-03 | 2 | -103/+0 |
| | | | | | | This looks like it got half-killed back in #888. Fixes #6567. | ||||
* | Move background update handling out of store | Erik Johnston | 2019-12-05 | 1 | -1/+1 |
| | |||||
* | Fix: Pillow error when uploading RGBA image (#3325) (#6241) | Filip Štědronský | 2019-12-02 | 1 | -1/+4 |
| | | | Signed-Off-By: Filip Štědronský <g@regnarg.cz> | ||||
* | Fix startup error when http proxy is defined. (#6421) | Richard van der Hoff | 2019-11-26 | 1 | -2/+2 |
| | | | | | Guess I only tested this on python 2 :/ Fixes #6419. | ||||
* | Clean up newline quote marks around the codebase (#6362) | Andrew Morgan | 2019-11-21 | 1 | -1/+1 |
| | |||||
* | Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2 | Richard van der Hoff | 2019-11-05 | 1 | -1/+2 |
|\ | | | | | Fix exception when OpenGraph tag values are ints | ||||
| * | Fix exception when OpenGraph tag values are ints | Richard van der Hoff | 2019-11-05 | 1 | -1/+2 |
| | | |||||
* | | Merge branch 'develop' into rav/url_preview_limit_title | Richard van der Hoff | 2019-11-05 | 3 | -20/+26 |
|\ \ | |/ |/| | |||||
| * | Support for routing outbound HTTP requests via a proxy (#6239) | Richard van der Hoff | 2019-11-01 | 1 | -0/+2 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198. | ||||
| * | Remove usage of deprecated logger.warn method from codebase (#6271) | Andrew Morgan | 2019-10-31 | 3 | -15/+17 |
| | | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated. | ||||
| * | Address codestyle concerns | Michael Kaye | 2019-10-24 | 1 | -1/+3 |
| | | |||||
| * | Delay printf until logging is required. | Michael Kaye | 2019-10-24 | 1 | -6/+6 |
| | | | | | | | | | | Using % will cause the string to be generated even if debugging is off. | ||||
* | | Apply suggestions from code review | Richard van der Hoff | 2019-11-05 | 1 | -1/+1 |
| | | | | | | | | Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-Authored-By: Erik Johnston <erik@matrix.org> | ||||
* | | Strip overlong OpenGraph data from url preview | Richard van der Hoff | 2019-11-05 | 1 | -1/+19 |
|/ | | | | ... to stop people causing DoSes with malicious web pages | ||||
* | Fix store_url_cache using bytes | Erik Johnston | 2019-10-10 | 1 | -2/+2 |
| | |||||
* | add M_TOO_LARGE error code for uploading a too large file (#6151) | Anshul Angaria | 2019-10-08 | 1 | -2/+6 |
| | | | Fixes #6109 | ||||
* | Log responder we are using. (#6139) | Michael Kaye | 2019-10-07 | 1 | -1/+1 |
| | | | This prevents us logging "Responding to media request with responder %s". | ||||
* | media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142) | Robert Swain | 2019-10-04 | 1 | -3/+11 |
| | | | | | | Pillow will use nearest neighbour as the resampling algorithm if the source image is either 1-bit or a color palette using 8 bits. If we convert to RGB before scaling, we'll probably get a better result. | ||||
* | Remove double return statements (#5962) | Andrew Morgan | 2019-09-03 | 1 | -1/+0 |
| | | | | | | | | | | Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully. | ||||
* | fix thumbnail storage location (#5915) | L0ric0 | 2019-09-02 | 1 | -1/+1 |
| | | | | | | | | | | | | | | * fix thumbnail storage location Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Add changelog file. Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Update Changelog Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> | ||||
* | Remove unnecessary parentheses around return statements (#5931) | Andrew Morgan | 2019-08-30 | 2 | -4/+4 |
| | | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :) | ||||
* | Don't load the media repo when configured to use an external media repo (#5754) | Amber Brown | 2019-08-13 | 1 | -1/+5 |
| | |||||
* | Replace returnValue with return (#5736) | Amber Brown | 2019-07-23 | 3 | -33/+31 |
| | |||||
* | fix typo: backgroud -> background | Andrew Morgan | 2019-07-12 | 1 | -1/+1 |
| | |||||
* | Move logging utilities out of the side drawer of util/ and into logging/ (#5606) | Amber Brown | 2019-07-04 | 5 | -17/+13 |
| | |||||
* | Fix media repo breaking (#5593) | Amber Brown | 2019-07-02 | 1 | -0/+1 |
| | |||||
* | Make the http server handle coroutine-making REST servlets (#5475) | Amber Brown | 2019-06-29 | 5 | -68/+47 |
| | |||||
* | Run Black. (#5482) | Amber Brown | 2019-06-20 | 12 | -408/+425 |
| | |||||
* | Don't log exception when failing to fetch remote content. | Erik Johnston | 2019-06-07 | 1 | -2/+4 |
| | | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large. | ||||
* | Fix error when downloading thumbnail with width/height param missing (#5258) | Aaron Raimist | 2019-05-29 | 1 | -2/+2 |
| | | | | | | | Fix error when downloading thumbnail with width/height param missing Fixes #2748 Signed-off-by: Aaron Raimist <aaron@raim.ist> | ||||
* | Fix image orientation when generating thumbnail (#5039) | PauRE | 2019-05-16 | 2 | -0/+44 |
| | |||||
* | Migrate all tests to use the dict-based config format instead of hanging ↵ | Amber Brown | 2019-05-13 | 1 | -0/+1 |
| | | | | items off HomeserverConfig (#5171) | ||||
* | URL preview blacklisting fixes (#5155) | Andrew Morgan | 2019-05-10 | 1 | -0/+10 |
| | | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases. | ||||
* | Prevent "producer not unregistered" message (#5009) | Amber Brown | 2019-04-24 | 1 | -0/+4 |
| | |||||
* | Remove periods from copyright headers (#5046) | Andrew Morgan | 2019-04-11 | 1 | -1/+1 |
| | |||||
* | fix incorrect encoding of filenames with spaces in (#2090) | Matthew Hodgson | 2019-03-11 | 1 | -3/+51 |
| | | | | fixes https://github.com/vector-im/riot-web/issues/3155 | ||||
* | Fix parsing of Content-Disposition headers (#4763) | Richard van der Hoff | 2019-02-27 | 1 | -20/+65 |
| | | | | | | | | | | | * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings | ||||
* | Don't log stack trace when client has gone away during media download (#4738) | Erik Johnston | 2019-02-25 | 1 | -2/+9 |
| | | | | | | | | * Don't log stack trace when client has gone away during media download * Newsfile * Fixup newsfile | ||||
* | Refactor request sending to have better excpetions (#4358) | Erik Johnston | 2019-01-08 | 1 | -3/+4 |
| | | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions. | ||||
* | Fix IP URL previews on Python 3 (#4215) | Amber Brown | 2018-12-22 | 1 | -2/+12 |
| | |||||
* | Add 'sandbox' to CSP for media repo (#4284) | David Baker | 2018-12-11 | 1 | -1/+2 |
| | | | | | | * Add 'sandbox' to the CSP for media repo * Changelog | ||||
* | Move imports to one line | Will Hunt | 2018-12-10 | 1 | -4/+1 |
| | |||||
* | Use `send_cors` | Will Hunt | 2018-12-10 | 1 | -4/+1 |
| | |||||
* | changelog & isort | Will Hunt | 2018-12-09 | 1 | -1/+5 |
| | |||||
* | Make /config more CORS-y | Will Hunt | 2018-12-09 | 1 | -1/+3 |
| | |||||
* | Fix Content-Disposition in media repository (#4176) | Amber Brown | 2018-11-15 | 3 | -90/+110 |
| | |||||
* | Use <meta> tags to discover the per-page encoding of html previews (#4183) | Amber Brown | 2018-11-15 | 1 | -9/+22 |
| | |||||
* | Fix URL preview bugs (type error when loading cache from db, content-type ↵ | Amber Brown | 2018-11-08 | 1 | -6/+16 |
| | | | | including quotes) (#4157) | ||||
* | Delete the disused & unspecced identicon functionality (#4106) | Amber Brown | 2018-10-29 | 2 | -70/+0 |
| | |||||
* | Fix a number of flake8 errors | Richard van der Hoff | 2018-10-24 | 1 | -1/+1 |
| | | | | | | | | | | | | Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there. | ||||
* | Correctly account for cpu usage by background threads (#4074) | Richard van der Hoff | 2018-10-23 | 3 | -15/+23 |
| | | | | | | | | | | Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064 | ||||
* | Fix bug where we raised StopIteration in a generator | Erik Johnston | 2018-10-17 | 1 | -2/+5 |
| | | | | This made python 3.7 unhappy | ||||
* | Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches | Richard van der Hoff | 2018-09-25 | 1 | -1/+0 |
|\ | | | | | Fix some instances of ExpiringCache not expiring cache items | ||||
| * | Fix some instances of ExpiringCache not expiring cache items | Erik Johnston | 2018-09-21 | 1 | -1/+0 |
| | | | | | | | | | | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead. | ||||
* | | added "media-src: 'self'" to CSP for resources (#3578) | Jérémy Farnaud | 2018-09-25 | 1 | -0/+1 |
|/ | | | | | | | | | | | | | | | Synapse doesn’t allow for media resources to be played directly from Chrome. It is a problem for users on other networks (e.g. IRC) communicating with Matrix users through a gateway. The gateway sends them the raw URL for the resource when a Matrix user uploads a video and the video cannot be played directly in Chrome using that URL. Chrome argues it is not authorized to play the video because of the Content Security Policy. Chrome checks for the "media-src" policy which is missing, and defauts to the "default-src" policy which is "none". As Synapse already sends "object-src: 'self'" I thought it wouldn’t be a problem to add "media-src: 'self'" to the CSP to fix this problem. | ||||
* | Port rest/ to Python 3 (#3823) | Amber Brown | 2018-09-12 | 5 | -40/+45 |
| | |||||
* | Port over enough to get some sytests running on Python 3 (#3668) | Amber Brown | 2018-08-20 | 1 | -5/+5 |
| | |||||
* | Add GET media/v1/config (#3184) | Will Hunt | 2018-08-16 | 2 | -1/+50 |
| | |||||
* | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) | Amber Brown | 2018-08-10 | 2 | -2/+2 |
| | |||||
* | Refactor code for turning HttpResponseException into SynapseError | Richard van der Hoff | 2018-08-01 | 1 | -1/+1 |
| | | | | | | | | This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call. | ||||
* | Python 3: Convert some unicode/bytes uses (#3569) | Amber Brown | 2018-08-02 | 1 | -1/+1 |
| | |||||
* | Fix some looping_call calls which were broken in #3604 | Richard van der Hoff | 2018-07-26 | 2 | -2/+2 |
| | | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call. | ||||
* | Wrap a number of things that run in the background | Richard van der Hoff | 2018-07-25 | 2 | -2/+14 |
| | | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics. | ||||
* | check isort by travis | Krombel | 2018-07-16 | 1 | -2/+2 |
| | |||||
* | Use parse_{int,str} and assert from http.servlet | Krombel | 2018-07-13 | 3 | -6/+10 |
| | | | | | | | parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable | ||||
* | run isort | Amber Brown | 2018-07-09 | 12 | -92/+79 |
| | |||||
* | Attempt to be more performant on PyPy (#3462) | Amber Brown | 2018-06-28 | 2 | -2/+4 |
| | |||||
* | Pass around the reactor explicitly (#3385) | Amber Brown | 2018-06-22 | 2 | -3/+7 |
| | |||||
* | Merge pull request #3244 from NotAFile/py3-six-4 | Amber Brown | 2018-05-24 | 1 | -1/+2 |
|\ | | | | | replace some iteritems with six | ||||
| * | replace some iteritems with six | Adrian Tschira | 2018-05-19 | 1 | -1/+2 |
| | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | | py3-ize url preview | Adrian Tschira | 2018-05-19 | 1 | -3/+5 |
|/ | |||||
* | Set Server header in SynapseRequest | Richard van der Hoff | 2018-05-10 | 4 | -5/+1 |
| | | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really. | ||||
* | Remove redundant request_handler decorator | Richard van der Hoff | 2018-05-10 | 4 | -26/+33 |
| | | | | | | This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler. | ||||
* | Fix logcontext leak in media repo | Richard van der Hoff | 2018-05-02 | 2 | -1/+4 |
| | | | | Make FileResponder.write_to_consumer uphold the logcontext contract | ||||
* | Merge pull request #3155 from NotAFile/py3-bytes-1 | Richard van der Hoff | 2018-04-30 | 1 | -3/+3 |
|\ | | | | | more bytes strings | ||||
| * | more bytes strings | Adrian Tschira | 2018-04-29 | 1 | -3/+3 |
| | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | | Merge pull request #3140 from matrix-org/rav/use_run_in_background | Richard van der Hoff | 2018-04-30 | 1 | -2/+3 |
|\ \ | | | | | | | Use run_in_background in preference to preserve_fn | ||||
| * | | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background | Richard van der Hoff | 2018-04-27 | 1 | -2/+7 |
| |\| | |||||
| * | | Use run_in_background in preference to preserve_fn | Richard van der Hoff | 2018-04-27 | 1 | -2/+3 |
| | | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object. | ||||
* | | | Merge pull request #3108 from NotAFile/py3-six-urlparse | Richard van der Hoff | 2018-04-30 | 2 | -2/+2 |
|\ \ \ | |_|/ |/| | | Use six.moves.urlparse | ||||
| * | | Use six.moves.urlparse | Adrian Tschira | 2018-04-15 | 2 | -2/+2 |
| |/ | | | | | | | | | | | The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* / | Improve exception handling for background processes | Richard van der Hoff | 2018-04-27 | 1 | -2/+7 |
|/ | | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background. | ||||
* | Replace old-style raise with six.reraise | Adrian Tschira | 2018-04-06 | 1 | -1/+3 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | Merge branch 'master' of github.com:matrix-org/synapse into develop | Erik Johnston | 2018-03-19 | 1 | -1/+1 |
|\ | |||||
| * | Replace ujson with simplejson | Erik Johnston | 2018-03-15 | 1 | -1/+1 |
| | | |||||
* | | Make store_file use store_into_file | Erik Johnston | 2018-02-14 | 1 | -26/+11 |
| | | |||||
* | | Tell storage providers about new file so they can upload | Erik Johnston | 2018-02-07 | 1 | -0/+6 |
| | | |||||
* | | Merge pull request #2791 from matrix-org/erikj/media_storage_refactor | Erik Johnston | 2018-02-05 | 3 | -12/+47 |
|\ \ | | | | | | | Ensure media is in local cache before thumbnailing | ||||
| * | | Use better file consumer | Erik Johnston | 2018-01-18 | 1 | -5/+10 |
| | | | |||||
| * | | Ensure media is in local cache before thumbnailing | Erik Johnston | 2018-01-18 | 3 | -10/+40 |
| | | | |||||
* | | | Handle url_previews with no content-type | Richard van der Hoff | 2018-02-02 | 1 | -21/+34 |
| | | | | | | | | | | | | | | | | | | | | | avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit. | ||||
* | | | Add federation_domain_whitelist option (#2820) | Matthew Hodgson | 2018-01-22 | 1 | -2/+17 |
| | | | | | | | | | | | | | | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network | ||||
* | | | Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs | Richard van der Hoff | 2018-01-19 | 1 | -0/+8 |
|\ \ \ | | | | | | | | | Use the right path for url_preview thumbnails | ||||
| * | | | Use the right path for url_preview thumbnails | Richard van der Hoff | 2018-01-19 | 1 | -0/+8 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was introduced by #2627: we were overwriting the original media for url previews with the thumbnails :/ (fixes https://github.com/vector-im/riot-web/issues/6012, hopefully) | ||||
* | | | | Fix storage provider bug introduced when renamed to store_local | Erik Johnston | 2018-01-18 | 1 | -1/+1 |
| | | | | |||||
* | | | | Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config | Erik Johnston | 2018-01-18 | 2 | -20/+27 |
|\ \ \ \ | | | | | | | | | | | Make storage providers configurable | ||||
| * | | | | Missing staticmethod | Erik Johnston | 2018-01-18 | 1 | -0/+1 |
| | | | | | |||||
| * | | | | Fixup comments | Erik Johnston | 2018-01-18 | 1 | -2/+2 |
| | | | | | |||||
| * | | | | Make storage providers more configurable | Erik Johnston | 2018-01-18 | 2 | -20/+26 |
| | |/ / | |/| | | |||||
* | | | | Fix typo in thumbnail resource causing access times to be incorrect | Erik Johnston | 2018-01-18 | 1 | -2/+2 |
| |/ / |/| | | |||||
* | | | Update last access time when thumbnails are viewed | Erik Johnston | 2018-01-17 | 1 | -0/+2 |
| | | | |||||
* | | | Keep track of last access time for local media | Erik Johnston | 2018-01-17 | 1 | -7/+25 |
|/ / | |||||
* | | Add docstring | Erik Johnston | 2018-01-17 | 1 | -1/+1 |
| | | |||||
* | | Use local vars | Erik Johnston | 2018-01-16 | 1 | -6/+2 |
| | | |||||
* | | Change _generate_thumbnails to take media_type | Erik Johnston | 2018-01-16 | 2 | -7/+10 |
| | | |||||
* | | Move setting of file_id up to caller | Erik Johnston | 2018-01-16 | 2 | -13/+14 |
| | | |||||
* | | Fix typo | Erik Johnston | 2018-01-16 | 1 | -7/+10 |
| | | |||||
* | | Fix up log lines | Erik Johnston | 2018-01-16 | 2 | -7/+10 |
| | | |||||
* | | Correctly use server_name/file_id when generating/fetching remote thumbnails | Erik Johnston | 2018-01-16 | 2 | -5/+8 |
| | | |||||
* | | Log when we respond with 404 | Erik Johnston | 2018-01-16 | 2 | -1/+8 |
| | | |||||
* | | Fix thumbnailing remote files | Erik Johnston | 2018-01-16 | 2 | -2/+30 |
| | | |||||
* | | Merge pull request #2767 from matrix-org/erikj/media_storage_refactor | Erik Johnston | 2018-01-16 | 7 | -346/+843 |
|\ \ | | | | | | | Refactor MediaRepository to separate out storage | ||||
| * | | Fix up comments | Erik Johnston | 2018-01-12 | 2 | -7/+16 |
| | | | |||||
| * | | Correctly reraise exception | Erik Johnston | 2018-01-12 | 1 | -2/+4 |
| | | | |||||
| * | | Make Responder a context manager | Erik Johnston | 2018-01-12 | 2 | -9/+10 |
| | | | |||||
| * | | Add missing class var | Erik Johnston | 2018-01-12 | 1 | -0/+3 |
| | | | |||||
| * | | Remove unnecessary condition | Erik Johnston | 2018-01-12 | 1 | -4/+1 |
| | | | |||||
| * | | Remove unused variables | Erik Johnston | 2018-01-12 | 1 | -3/+0 |
| | | | |||||
| * | | Make class var local | Erik Johnston | 2018-01-12 | 1 | -3/+3 |
| | | | |||||
| * | | Comments | Erik Johnston | 2018-01-12 | 3 | -15/+65 |
| | | | |||||
| * | | Add StorageProvider concept | Erik Johnston | 2018-01-09 | 3 | -9/+162 |
| | | | |||||
| * | | Make PreviewUrlResource use MediaStorage | Erik Johnston | 2018-01-09 | 2 | -8/+14 |
| | | | |||||
| * | | Make ThumbnailResource use MediaStorage | Erik Johnston | 2018-01-09 | 2 | -48/+68 |
| | | | |||||
| * | | Use MediaStorage for remote media | Erik Johnston | 2018-01-09 | 2 | -136/+156 |
| | | | |||||
| * | | Use MediaStorage for local files | Erik Johnston | 2018-01-09 | 2 | -119/+73 |
| | | | |||||
| * | | Add MediaStorage class | Erik Johnston | 2018-01-09 | 1 | -0/+198 |
| | | | |||||
| * | | Add some helper classes | Erik Johnston | 2018-01-09 | 1 | -0/+73 |
| | | | |||||
| * | | Split out add_file_headers | Erik Johnston | 2018-01-09 | 1 | -28/+42 |
| | | | |||||
* | | | Reinstate media download on thumbnail request | Richard van der Hoff | 2018-01-12 | 1 | -0/+5 |
|/ / | | | | | | | | | We need to actually download the remote media when we get a request for a thumbnail. | ||||
* | | Remove dead TODO | Erik Johnston | 2018-01-09 | 1 | -2/+0 |
| | | |||||
* | | Remove dead code related to default thumbnails | Erik Johnston | 2018-01-09 | 1 | -73/+3 |
|/ | |||||
* | Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error | Richard van der Hoff | 2017-11-27 | 1 | -3/+7 |
|\ | | | | | Fix error on sqlite 3.7 | ||||
| * | Check database in has_completed_background_updates | Richard van der Hoff | 2017-11-22 | 1 | -1/+1 |
| | | | | | | | | so that the right thing happens on workers. | ||||
| * | Fix error on sqlite 3.7 | Richard van der Hoff | 2017-11-21 | 1 | -3/+7 |
| | | | | | | | | | | | | | | | | | | | | | | Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572. | ||||
* | | Fix OPTIONS on preview_url | Richard van der Hoff | 2017-11-23 | 1 | -1/+5 |
|/ | | | | Fixes #2706 | ||||
* | Bit more logging | Richard van der Hoff | 2017-11-10 | 1 | -0/+2 |
| | |||||
* | Cache failures in url_preview handler | Richard van der Hoff | 2017-11-10 | 1 | -41/+45 |
| | | | | | Reshuffle the caching logic in the url_preview handler so that failures are cached (and to generally simplify things and fix the logcontext leaks). | ||||
* | Ignore noscript tags when generating URL previews | Maxime Vaillancourt | 2017-10-25 | 1 | -1/+8 |
| | |||||
* | replace 'except:' with 'except Exception:' | Richard van der Hoff | 2017-10-23 | 3 | -5/+5 |
| | | | | what could possibly go wrong | ||||
* | Fix a logcontext leak in the media repo | Richard van der Hoff | 2017-10-23 | 1 | -1/+4 |
| | |||||
* | Fix typo in thumbnail generation | Erik Johnston | 2017-10-19 | 1 | -2/+5 |
| | |||||
* | fix thumbnailing (#2548) | Krombel | 2017-10-17 | 1 | -2/+2 |
| | | | | | | | in commit 0e28281a the code for thumbnailing got refactored and the renaming of this variables was not done correctly. Signed-Off-by: Matthias Kesler <krombel@krombel.de> | ||||
* | Add try/finally block to close t_byte_source | Erik Johnston | 2017-10-13 | 1 | -24/+41 |
| | |||||
* | Remove old comment | Erik Johnston | 2017-10-13 | 1 | -1/+0 |
| | |||||
* | Don't reuse source but instead copy from primary media store to backup | Erik Johnston | 2017-10-13 | 2 | -22/+9 |
| | |||||
* | PEP8 | Erik Johnston | 2017-10-13 | 1 | -1/+1 |
| | |||||
* | Move type into key | Erik Johnston | 2017-10-13 | 1 | -3/+3 |
| | |||||
* | Comment | Erik Johnston | 2017-10-13 | 1 | -1/+1 |
| | |||||
* | Remove spurious addition | Erik Johnston | 2017-10-13 | 1 | -4/+0 |
| | |||||
* | Don't needlessly rename variable | Erik Johnston | 2017-10-13 | 1 | -13/+13 |
| | |||||
* | Add paths back in... | Erik Johnston | 2017-10-13 | 1 | -4/+4 |
| | |||||
* | Remove unnecessary diff | Erik Johnston | 2017-10-13 | 1 | -3/+1 |
| | |||||
* | Make things be absolute paths again | Erik Johnston | 2017-10-13 | 1 | -7/+7 |
| | |||||
* | s/write_to_file/write_to_file_and_backup/ | Erik Johnston | 2017-10-13 | 1 | -5/+5 |
| | |||||
* | Fix up | Erik Johnston | 2017-10-13 | 2 | -43/+32 |
| | |||||
* | Fix up thumbnailing function | Erik Johnston | 2017-10-13 | 3 | -75/+73 |
| | |||||
* | Fix typo | Erik Johnston | 2017-10-13 | 1 | -1/+1 |
| | |||||
* | Fix up comments | Erik Johnston | 2017-10-13 | 1 | -2/+26 |
| | |||||
* | Move makedirs to thread | Erik Johnston | 2017-10-13 | 1 | -2/+1 |
| | |||||
* | Use make_deferred_yieldable | Erik Johnston | 2017-10-13 | 1 | -7/+7 |
| | |||||
* | Fix comments | Erik Johnston | 2017-10-12 | 2 | -1/+4 |
| | |||||
* | Don't close prematurely.. | Erik Johnston | 2017-10-12 | 1 | -2/+5 |
| | |||||
* | getvalue closes buffer | Erik Johnston | 2017-10-12 | 1 | -11/+10 |
| | |||||
* | Get len before close | Erik Johnston | 2017-10-12 | 1 | -6/+11 |
| | |||||
* | Don't close file prematurely | Erik Johnston | 2017-10-12 | 2 | -8/+18 |
| | |||||
* | Copy everything to backup | Erik Johnston | 2017-10-12 | 4 | -76/+148 |
| | |||||
* | Typo | Erik Johnston | 2017-10-12 | 1 | -1/+1 |
| | |||||
* | Remove dead code | Erik Johnston | 2017-10-12 | 1 | -3/+0 |
| | |||||
* | Fix typos | Erik Johnston | 2017-10-12 | 1 | -20/+26 |
| | |||||
* | Fix typo | Erik Johnston | 2017-10-12 | 1 | -1/+1 |
| | |||||
* | Basic implementation of backup media store | Erik Johnston | 2017-10-12 | 3 | -126/+113 |
| | |||||
* | Only log if we've removed media | Erik Johnston | 2017-09-28 | 1 | -2/+4 |
| | |||||
* | Clear up expired url cache every 10s | Erik Johnston | 2017-09-28 | 1 | -1/+1 |
| | |||||
* | Actually assign result to variable | Erik Johnston | 2017-09-28 | 1 | -1/+1 |
| | |||||
* | Fix typo in url cache expiry timer | Erik Johnston | 2017-09-28 | 1 | -1/+1 |
| | |||||
* | Support new and old style media id formats | Erik Johnston | 2017-09-28 | 1 | -31/+81 |
| | |||||
* | Change expires column to expires_ts | Erik Johnston | 2017-09-28 | 1 | -2/+2 |
| | |||||
* | Delete expired url cache data | Erik Johnston | 2017-09-28 | 2 | -4/+129 |
| | |||||
* | Store URL cache preview downloads seperately | Erik Johnston | 2017-06-23 | 5 | -18/+72 |
| | | | | This makes it easier to clear old media out at a later date | ||||
* | Add API to quarantine media | Erik Johnston | 2017-06-19 | 3 | -3/+5 |
| | |||||
* | Throw exception when not retrying when downloading media | Erik Johnston | 2017-06-13 | 1 | -0/+1 |
| | |||||
* | actually trim oversize og:description meta | Matthew Hodgson | 2017-05-22 | 1 | -0/+2 |
| | |||||
* | Don't log exceptions for NotRetryingDestination | Erik Johnston | 2017-05-15 | 1 | -1/+3 |
| | |||||
* | Address review comments | Richard van der Hoff | 2017-03-14 | 1 | -15/+17 |
| | | | | | | | - don't blindly proxy all HTTPRequestExceptions - log unexpected exceptions at error - avoid `isinstance` - improve docs on `from_http_response_exception` | ||||
* | Fix routing loop when fetching remote media | Richard van der Hoff | 2017-03-13 | 2 | -4/+38 |
| | | | | | | | | | | | When we proxy a media request to a remote server, add a query-param, which will tell the remote server to 404 if it doesn't recognise the server_name. This should fix a routing loop where the server keeps forwarding back to itself. Also improves the error handling on remote media fetches, so that we don't always return a rather obscure 502. | ||||
* | Fix dynamic thumbnails aspect | Jurek | 2017-02-24 | 1 | -0/+3 |
| | |||||
* | Log which files we saved attachments to in the media_repository | Mark Haines | 2017-01-10 | 3 | -0/+11 |
| | |||||
* | Name linearizer's for better logs | Erik Johnston | 2017-01-09 | 1 | -1/+1 |
| | |||||
* | Fix crash in url preview when html tag has no text | Marcin Bachry | 2016-12-14 | 1 | -1/+4 |
| | | | | Signed-off-by: Marcin Bachry <hegel666@gmail.com> | ||||
* | preview_url_resource: Ellipsis must be in unicode string | Johannes Löthberg | 2016-12-01 | 1 | -1/+1 |
| | | | | Signed-off-by: Johannes Löthberg <johannes@kyriasis.com> | ||||
* | Set CORs headers on responses from the media repo | Mark Haines | 2016-11-02 | 2 | -2/+4 |
| | |||||
* | Add quotes and be explicity about script-src | Erik Johnston | 2016-09-05 | 1 | -1/+2 |
| | |||||
* | Allow PDF to be rendered from media repo | Erik Johnston | 2016-09-05 | 1 | -1/+7 |
| | |||||
* | Set `Content-Security-Policy` on media repo | Erik Johnston | 2016-08-17 | 1 | -0/+1 |
| | | | | | This is to inform browsers that they should sandbox the returned media. This is particularly cruical for javascript/HTML files. | ||||
* | Add None check to _iterate_over_text | Erik Johnston | 2016-08-17 | 1 | -1/+1 |
| | |||||
* | Flake8 | Erik Johnston | 2016-08-16 | 1 | -4/+1 |
| | |||||
* | Fix up preview URL API. Add tests. | Erik Johnston | 2016-08-16 | 1 | -157/+196 |
| | | | | | | | | | This includes: - Splitting out methods of a class into stand alone functions, to make them easier to test. - Adding unit tests to split out functions, testing HTML -> preview. - Handle the fact that elements in lxml may have tail text. | ||||
* | Don't include html comments in description | Erik Johnston | 2016-08-05 | 1 | -1/+2 |
| | |||||
* | Typo | Erik Johnston | 2016-08-04 | 1 | -1/+1 |
| | |||||
* | Test summarization | Erik Johnston | 2016-08-04 | 1 | -52/+54 |
| | |||||
* | Don't infer paragrahs from newlines | Erik Johnston | 2016-08-02 | 1 | -2/+1 |
| | |||||
* | Comment on why we clone | Erik Johnston | 2016-08-02 | 1 | -0/+1 |
| | |||||
* | Spelling. | Erik Johnston | 2016-08-02 | 1 | -3/+3 |
| | |||||
* | Make it actually compile | Erik Johnston | 2016-08-02 | 1 | -1/+1 |
| | |||||
* | Change the way we summarize URLs | Erik Johnston | 2016-08-02 | 1 | -11/+67 |
| | | | | | | | | Using XPath is slow on some machines (for unknown reasons), so use a different approach to get a list of text nodes. Try to generate a summary that respect paragraph and then word boundaries, adding ellipses when appropriate. | ||||
* | Remove race | Erik Johnston | 2016-06-29 | 1 | -11/+0 |
| | |||||
* | Implement purge_media_cache admin API | Erik Johnston | 2016-06-29 | 2 | -20/+64 |
| | |||||
* | Track approximate last access time for remote media | Erik Johnston | 2016-06-29 | 1 | -0/+24 |
| | |||||
* | Remove the legacy v0 content upload API. | Mark Haines | 2016-06-21 | 1 | -110/+2 |
| | | | | | | The existing content can still be downloaded. The last upload to the matrix.org server was in January 2015, so it is probably safe to remove the upload API. | ||||
* | Line lengths | Erik Johnston | 2016-06-15 | 1 | -3/+3 |
| | |||||
* | Handle floats as img widths | Erik Johnston | 2016-06-15 | 1 | -1/+1 |
| | |||||
* | Handle og props with not content | Erik Johnston | 2016-06-15 | 1 | -1/+2 |
| | |||||
* | 502 on /thumbnail when can't contact remote server | Erik Johnston | 2016-06-09 | 1 | -4/+10 |
| | |||||
* | Clean up the blacklist/whitelist handling. | Mark Haines | 2016-05-16 | 1 | -33/+30 |
| | | | | | | | Always set the config key with an empty list, even if a list isn't specified. This means that the codepaths are the same for both the empty list and for a missing key. Since the behaviour is the same for both cases this makes the code somewhat easier to reason about. | ||||
* | Report per request metrics for all of the things using request_handler | Mark Haines | 2016-04-28 | 4 | -4/+7 |
| | |||||
* | Add self.media_repo to PreviewUrlResource | Erik Johnston | 2016-04-19 | 1 | -0/+1 |
| | |||||
* | _make_dirs was moved to MediaRepository | Erik Johnston | 2016-04-19 | 1 | -1/+1 |
| |