summary refs log tree commit diff
path: root/synapse/rest/media/v1 (follow)
Commit message (Collapse)AuthorAgeFilesLines
* a bit of debugging for media storage providers (#6757)Richard van der Hoff2020-01-232-0/+7
| | | | | | * a bit of debugging for media storage providers * changelog
* Fix typo in _select_thumbnailBrendan Abolivier2020-01-221-1/+1
|
* Fixup synapse.rest to pass mypy (#6732)Erik Johnston2020-01-203-11/+13
|
* Move background update handling out of storeErik Johnston2019-12-051-1/+1
|
* Fix: Pillow error when uploading RGBA image (#3325) (#6241)Filip Štědronský2019-12-021-1/+4
| | | Signed-Off-By: Filip Štědronský <g@regnarg.cz>
* Fix startup error when http proxy is defined. (#6421)Richard van der Hoff2019-11-261-2/+2
| | | | | Guess I only tested this on python 2 :/ Fixes #6419.
* Clean up newline quote marks around the codebase (#6362)Andrew Morgan2019-11-211-1/+1
|
* Merge pull request #6334 from matrix-org/rav/url_preview_limit_title_2Richard van der Hoff2019-11-051-1/+2
|\ | | | | Fix exception when OpenGraph tag values are ints
| * Fix exception when OpenGraph tag values are intsRichard van der Hoff2019-11-051-1/+2
| |
* | Merge branch 'develop' into rav/url_preview_limit_titleRichard van der Hoff2019-11-053-20/+26
|\ \ | |/ |/|
| * Support for routing outbound HTTP requests via a proxy (#6239)Richard van der Hoff2019-11-011-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `http_proxy` and `HTTPS_PROXY` env vars can be set to a `host[:port]` value which should point to a proxy. The address of the proxy should be excluded from IP blacklists such as the `url_preview_ip_range_blacklist`. The proxy will then be used for * push * url previews * phone-home stats * recaptcha validation * CAS auth validation It will *not* be used for: * Application Services * Identity servers * Outbound federation * In worker configurations, connections from workers to masters Fixes #4198.
| * Remove usage of deprecated logger.warn method from codebase (#6271)Andrew Morgan2019-10-313-15/+17
| | | | | | Replace every instance of `logger.warn` with `logger.warning` as the former is deprecated.
| * Address codestyle concernsMichael Kaye2019-10-241-1/+3
| |
| * Delay printf until logging is required.Michael Kaye2019-10-241-6/+6
| | | | | | | | | | Using % will cause the string to be generated even if debugging is off.
* | Apply suggestions from code reviewRichard van der Hoff2019-11-051-1/+1
| | | | | | | | Co-Authored-By: Brendan Abolivier <babolivier@matrix.org> Co-Authored-By: Erik Johnston <erik@matrix.org>
* | Strip overlong OpenGraph data from url previewRichard van der Hoff2019-11-051-1/+19
|/ | | | ... to stop people causing DoSes with malicious web pages
* Fix store_url_cache using bytesErik Johnston2019-10-101-2/+2
|
* add M_TOO_LARGE error code for uploading a too large file (#6151)Anshul Angaria2019-10-081-2/+6
| | | Fixes #6109
* Log responder we are using. (#6139)Michael Kaye2019-10-071-1/+1
| | | This prevents us logging "Responding to media request with responder %s".
* media/thumbnailer: Better quality for 1-bit / 8-bit color palette images (#2142)Robert Swain2019-10-041-3/+11
| | | | | | Pillow will use nearest neighbour as the resampling algorithm if the source image is either 1-bit or a color palette using 8 bits. If we convert to RGB before scaling, we'll probably get a better result.
* Remove double return statements (#5962)Andrew Morgan2019-09-031-1/+0
| | | | | | | | | | Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.
* fix thumbnail storage location (#5915)L0ric02019-09-021-1/+1
| | | | | | | | | | | | | | * fix thumbnail storage location Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Add changelog file. Signed-off-by: Lorenz Steinert <lorenz@steinerts.de> * Update Changelog Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
* Remove unnecessary parentheses around return statements (#5931)Andrew Morgan2019-08-302-4/+4
| | | | | Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)
* Don't load the media repo when configured to use an external media repo (#5754)Amber Brown2019-08-131-1/+5
|
* Replace returnValue with return (#5736)Amber Brown2019-07-233-33/+31
|
* fix typo: backgroud -> backgroundAndrew Morgan2019-07-121-1/+1
|
* Move logging utilities out of the side drawer of util/ and into logging/ (#5606)Amber Brown2019-07-045-17/+13
|
* Fix media repo breaking (#5593)Amber Brown2019-07-021-0/+1
|
* Make the http server handle coroutine-making REST servlets (#5475)Amber Brown2019-06-295-68/+47
|
* Run Black. (#5482)Amber Brown2019-06-2011-401/+417
|
* Don't log exception when failing to fetch remote content.Erik Johnston2019-06-071-2/+4
| | | | | In particular, let's not log stack traces when we stop processing becuase the response body was too large.
* Fix error when downloading thumbnail with width/height param missing (#5258)Aaron Raimist2019-05-291-2/+2
| | | | | | | Fix error when downloading thumbnail with width/height param missing Fixes #2748 Signed-off-by: Aaron Raimist <aaron@raim.ist>
* Fix image orientation when generating thumbnail (#5039)PauRE2019-05-162-0/+44
|
* Migrate all tests to use the dict-based config format instead of hanging ↵Amber Brown2019-05-131-0/+1
| | | | items off HomeserverConfig (#5171)
* URL preview blacklisting fixes (#5155)Andrew Morgan2019-05-101-0/+10
| | | Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
* Prevent "producer not unregistered" message (#5009)Amber Brown2019-04-241-0/+4
|
* Remove periods from copyright headers (#5046)Andrew Morgan2019-04-111-1/+1
|
* fix incorrect encoding of filenames with spaces in (#2090)Matthew Hodgson2019-03-111-3/+51
| | | | fixes https://github.com/vector-im/riot-web/issues/3155
* Fix parsing of Content-Disposition headers (#4763)Richard van der Hoff2019-02-271-20/+65
| | | | | | | | | | | * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings
* Don't log stack trace when client has gone away during media download (#4738)Erik Johnston2019-02-251-2/+9
| | | | | | | | * Don't log stack trace when client has gone away during media download * Newsfile * Fixup newsfile
* Refactor request sending to have better excpetions (#4358)Erik Johnston2019-01-081-3/+4
| | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions.
* Fix IP URL previews on Python 3 (#4215)Amber Brown2018-12-221-2/+12
|
* Add 'sandbox' to CSP for media repo (#4284)David Baker2018-12-111-1/+2
| | | | | | * Add 'sandbox' to the CSP for media repo * Changelog
* Move imports to one lineWill Hunt2018-12-101-4/+1
|
* Use `send_cors`Will Hunt2018-12-101-4/+1
|
* changelog & isortWill Hunt2018-12-091-1/+5
|
* Make /config more CORS-yWill Hunt2018-12-091-1/+3
|
* Fix Content-Disposition in media repository (#4176)Amber Brown2018-11-153-90/+110
|
* Use <meta> tags to discover the per-page encoding of html previews (#4183)Amber Brown2018-11-151-9/+22
|
* Fix URL preview bugs (type error when loading cache from db, content-type ↵Amber Brown2018-11-081-6/+16
| | | | including quotes) (#4157)
* Delete the disused & unspecced identicon functionality (#4106)Amber Brown2018-10-292-70/+0
|
* Fix a number of flake8 errorsRichard van der Hoff2018-10-241-1/+1
| | | | | | | | | | | | Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.
* Correctly account for cpu usage by background threads (#4074)Richard van der Hoff2018-10-233-15/+23
| | | | | | | | | | Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064
* Fix bug where we raised StopIteration in a generatorErik Johnston2018-10-171-2/+5
| | | | This made python 3.7 unhappy
* Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_cachesRichard van der Hoff2018-09-251-1/+0
|\ | | | | Fix some instances of ExpiringCache not expiring cache items
| * Fix some instances of ExpiringCache not expiring cache itemsErik Johnston2018-09-211-1/+0
| | | | | | | | | | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead.
* | added "media-src: 'self'" to CSP for resources (#3578)Jérémy Farnaud2018-09-251-0/+1
|/ | | | | | | | | | | | | | | Synapse doesn’t allow for media resources to be played directly from Chrome. It is a problem for users on other networks (e.g. IRC) communicating with Matrix users through a gateway. The gateway sends them the raw URL for the resource when a Matrix user uploads a video and the video cannot be played directly in Chrome using that URL. Chrome argues it is not authorized to play the video because of the Content Security Policy. Chrome checks for the "media-src" policy which is missing, and defauts to the "default-src" policy which is "none". As Synapse already sends "object-src: 'self'" I thought it wouldn’t be a problem to add "media-src: 'self'" to the CSP to fix this problem.
* Port rest/ to Python 3 (#3823)Amber Brown2018-09-124-38/+43
|
* Port over enough to get some sytests running on Python 3 (#3668)Amber Brown2018-08-201-5/+5
|
* Add GET media/v1/config (#3184)Will Hunt2018-08-162-1/+50
|
* Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678)Amber Brown2018-08-102-2/+2
|
* Refactor code for turning HttpResponseException into SynapseErrorRichard van der Hoff2018-08-011-1/+1
| | | | | | | | This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call.
* Python 3: Convert some unicode/bytes uses (#3569)Amber Brown2018-08-021-1/+1
|
* Fix some looping_call calls which were broken in #3604Richard van der Hoff2018-07-262-2/+2
| | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.
* Wrap a number of things that run in the backgroundRichard van der Hoff2018-07-252-2/+14
| | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.
* check isort by travisKrombel2018-07-161-2/+2
|
* Use parse_{int,str} and assert from http.servletKrombel2018-07-133-6/+10
| | | | | | | parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable
* run isortAmber Brown2018-07-0911-81/+71
|
* Attempt to be more performant on PyPy (#3462)Amber Brown2018-06-281-1/+2
|
* Pass around the reactor explicitly (#3385)Amber Brown2018-06-222-3/+7
|
* Merge pull request #3244 from NotAFile/py3-six-4Amber Brown2018-05-241-1/+2
|\ | | | | replace some iteritems with six
| * replace some iteritems with sixAdrian Tschira2018-05-191-1/+2
| | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | py3-ize url previewAdrian Tschira2018-05-191-3/+5
|/
* Set Server header in SynapseRequestRichard van der Hoff2018-05-104-5/+1
| | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really.
* Remove redundant request_handler decoratorRichard van der Hoff2018-05-104-26/+33
| | | | | | This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler.
* Fix logcontext leak in media repoRichard van der Hoff2018-05-022-1/+4
| | | | Make FileResponder.write_to_consumer uphold the logcontext contract
* Merge pull request #3155 from NotAFile/py3-bytes-1Richard van der Hoff2018-04-301-3/+3
|\ | | | | more bytes strings
| * more bytes stringsAdrian Tschira2018-04-291-3/+3
| | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com>
* | Merge pull request #3140 from matrix-org/rav/use_run_in_backgroundRichard van der Hoff2018-04-301-2/+3
|\ \ | | | | | | Use run_in_background in preference to preserve_fn
| * | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_backgroundRichard van der Hoff2018-04-271-2/+7
| |\|
| * | Use run_in_background in preference to preserve_fnRichard van der Hoff2018-04-271-2/+3
| | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.
* | | Merge pull request #3108 from NotAFile/py3-six-urlparseRichard van der Hoff2018-04-302-2/+2
|\ \ \ | |_|/ |/| | Use six.moves.urlparse
| * | Use six.moves.urlparseAdrian Tschira2018-04-152-2/+2
| |/ | | | | | | | | | | The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com>
* / Improve exception handling for background processesRichard van der Hoff2018-04-271-2/+7
|/ | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.
* Replace old-style raise with six.reraiseAdrian Tschira2018-04-061-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com>
* Merge branch 'master' of github.com:matrix-org/synapse into developErik Johnston2018-03-191-1/+1
|\
| * Replace ujson with simplejsonErik Johnston2018-03-151-1/+1
| |
* | Make store_file use store_into_fileErik Johnston2018-02-141-26/+11
| |
* | Tell storage providers about new file so they can uploadErik Johnston2018-02-071-0/+6
| |
* | Merge pull request #2791 from matrix-org/erikj/media_storage_refactorErik Johnston2018-02-053-12/+47
|\ \ | | | | | | Ensure media is in local cache before thumbnailing
| * | Use better file consumerErik Johnston2018-01-181-5/+10
| | |
| * | Ensure media is in local cache before thumbnailingErik Johnston2018-01-183-10/+40
| | |
* | | Handle url_previews with no content-typeRichard van der Hoff2018-02-021-21/+34
| | | | | | | | | | | | | | | | | | | | | avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.
* | | Add federation_domain_whitelist option (#2820)Matthew Hodgson2018-01-221-2/+17
| | | | | | | | | | | | | | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
* | | Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbsRichard van der Hoff2018-01-191-0/+8
|\ \ \ | | | | | | | | Use the right path for url_preview thumbnails
| * | | Use the right path for url_preview thumbnailsRichard van der Hoff2018-01-191-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This was introduced by #2627: we were overwriting the original media for url previews with the thumbnails :/ (fixes https://github.com/vector-im/riot-web/issues/6012, hopefully)
* | | | Fix storage provider bug introduced when renamed to store_localErik Johnston2018-01-181-1/+1
| | | |
* | | | Merge pull request #2812 from matrix-org/erikj/media_storage_provider_configErik Johnston2018-01-182-20/+27
|\ \ \ \ | | | | | | | | | | Make storage providers configurable
| * | | | Missing staticmethodErik Johnston2018-01-181-0/+1
| | | | |
| * | | | Fixup commentsErik Johnston2018-01-181-2/+2
| | | | |
| * | | | Make storage providers more configurableErik Johnston2018-01-182-20/+26
| | |/ / | |/| |
* | | | Fix typo in thumbnail resource causing access times to be incorrectErik Johnston2018-01-181-2/+2
| |/ / |/| |
* | | Update last access time when thumbnails are viewedErik Johnston2018-01-171-0/+2
| | |
* | | Keep track of last access time for local mediaErik Johnston2018-01-171-7/+25
|/ /
* | Add docstringErik Johnston2018-01-171-1/+1
| |
* | Use local varsErik Johnston2018-01-161-6/+2
| |
* | Change _generate_thumbnails to take media_typeErik Johnston2018-01-162-7/+10
| |
* | Move setting of file_id up to callerErik Johnston2018-01-162-13/+14
| |
* | Fix typoErik Johnston2018-01-161-7/+10
| |
* | Fix up log linesErik Johnston2018-01-162-7/+10
| |
* | Correctly use server_name/file_id when generating/fetching remote thumbnailsErik Johnston2018-01-162-5/+8
| |
* | Log when we respond with 404Erik Johnston2018-01-162-1/+8
| |
* | Fix thumbnailing remote filesErik Johnston2018-01-162-2/+30
| |
* | Merge pull request #2767 from matrix-org/erikj/media_storage_refactorErik Johnston2018-01-167-346/+843
|\ \ | | | | | | Refactor MediaRepository to separate out storage
| * | Fix up commentsErik Johnston2018-01-122-7/+16
| | |
| * | Correctly reraise exceptionErik Johnston2018-01-121-2/+4
| | |
| * | Make Responder a context managerErik Johnston2018-01-122-9/+10
| | |
| * | Add missing class varErik Johnston2018-01-121-0/+3
| | |
| * | Remove unnecessary conditionErik Johnston2018-01-121-4/+1
| | |
| * | Remove unused variablesErik Johnston2018-01-121-3/+0
| | |
| * | Make class var localErik Johnston2018-01-121-3/+3
| | |
| * | CommentsErik Johnston2018-01-123-15/+65
| | |
| * | Add StorageProvider conceptErik Johnston2018-01-093-9/+162
| | |
| * | Make PreviewUrlResource use MediaStorageErik Johnston2018-01-092-8/+14
| | |
| * | Make ThumbnailResource use MediaStorageErik Johnston2018-01-092-48/+68
| | |
| * | Use MediaStorage for remote mediaErik Johnston2018-01-092-136/+156
| | |
| * | Use MediaStorage for local filesErik Johnston2018-01-092-119/+73
| | |
| * | Add MediaStorage classErik Johnston2018-01-091-0/+198
| | |
| * | Add some helper classesErik Johnston2018-01-091-0/+73
| | |
| * | Split out add_file_headersErik Johnston2018-01-091-28/+42
| | |
* | | Reinstate media download on thumbnail requestRichard van der Hoff2018-01-121-0/+5
|/ / | | | | | | | | We need to actually download the remote media when we get a request for a thumbnail.
* | Remove dead TODOErik Johnston2018-01-091-2/+0
| |
* | Remove dead code related to default thumbnailsErik Johnston2018-01-091-73/+3
|/
* Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_errorRichard van der Hoff2017-11-271-3/+7
|\ | | | | Fix error on sqlite 3.7
| * Check database in has_completed_background_updatesRichard van der Hoff2017-11-221-1/+1
| | | | | | | | so that the right thing happens on workers.
| * Fix error on sqlite 3.7Richard van der Hoff2017-11-211-3/+7
| | | | | | | | | | | | | | | | | | | | | | Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.
* | Fix OPTIONS on preview_urlRichard van der Hoff2017-11-231-1/+5
|/ | | | Fixes #2706
* Bit more loggingRichard van der Hoff2017-11-101-0/+2
|
* Cache failures in url_preview handlerRichard van der Hoff2017-11-101-41/+45
| | | | | Reshuffle the caching logic in the url_preview handler so that failures are cached (and to generally simplify things and fix the logcontext leaks).
* Ignore noscript tags when generating URL previewsMaxime Vaillancourt2017-10-251-1/+8
|
* replace 'except:' with 'except Exception:'Richard van der Hoff2017-10-233-5/+5
| | | | what could possibly go wrong
* Fix a logcontext leak in the media repoRichard van der Hoff2017-10-231-1/+4
|
* Fix typo in thumbnail generationErik Johnston2017-10-191-2/+5
|
* fix thumbnailing (#2548)Krombel2017-10-171-2/+2
| | | | | | | in commit 0e28281a the code for thumbnailing got refactored and the renaming of this variables was not done correctly. Signed-Off-by: Matthias Kesler <krombel@krombel.de>
* Add try/finally block to close t_byte_sourceErik Johnston2017-10-131-24/+41
|
* Remove old commentErik Johnston2017-10-131-1/+0
|
* Don't reuse source but instead copy from primary media store to backupErik Johnston2017-10-132-22/+9
|
* PEP8Erik Johnston2017-10-131-1/+1
|
* Move type into keyErik Johnston2017-10-131-3/+3
|
* CommentErik Johnston2017-10-131-1/+1
|
* Remove spurious additionErik Johnston2017-10-131-4/+0
|
* Don't needlessly rename variableErik Johnston2017-10-131-13/+13
|
* Add paths back in...Erik Johnston2017-10-131-4/+4
|
* Remove unnecessary diffErik Johnston2017-10-131-3/+1
|
* Make things be absolute paths againErik Johnston2017-10-131-7/+7
|
* s/write_to_file/write_to_file_and_backup/Erik Johnston2017-10-131-5/+5
|
* Fix upErik Johnston2017-10-132-43/+32
|
* Fix up thumbnailing functionErik Johnston2017-10-133-75/+73
|
* Fix typoErik Johnston2017-10-131-1/+1
|
* Fix up commentsErik Johnston2017-10-131-2/+26
|
* Move makedirs to threadErik Johnston2017-10-131-2/+1
|
* Use make_deferred_yieldableErik Johnston2017-10-131-7/+7
|
* Fix commentsErik Johnston2017-10-122-1/+4
|
* Don't close prematurely..Erik Johnston2017-10-121-2/+5
|
* getvalue closes bufferErik Johnston2017-10-121-11/+10
|
* Get len before closeErik Johnston2017-10-121-6/+11
|
* Don't close file prematurelyErik Johnston2017-10-122-8/+18
|
* Copy everything to backupErik Johnston2017-10-124-76/+148
|
* TypoErik Johnston2017-10-121-1/+1
|
* Remove dead codeErik Johnston2017-10-121-3/+0
|
* Fix typosErik Johnston2017-10-121-20/+26
|
* Fix typoErik Johnston2017-10-121-1/+1
|
* Basic implementation of backup media storeErik Johnston2017-10-123-126/+113
|
* Only log if we've removed mediaErik Johnston2017-09-281-2/+4
|
* Clear up expired url cache every 10sErik Johnston2017-09-281-1/+1
|
* Actually assign result to variableErik Johnston2017-09-281-1/+1
|
* Fix typo in url cache expiry timerErik Johnston2017-09-281-1/+1
|
* Support new and old style media id formatsErik Johnston2017-09-281-31/+81
|
* Change expires column to expires_tsErik Johnston2017-09-281-2/+2
|
* Delete expired url cache dataErik Johnston2017-09-282-4/+129
|
* Store URL cache preview downloads seperatelyErik Johnston2017-06-235-18/+72
| | | | This makes it easier to clear old media out at a later date
* Add API to quarantine mediaErik Johnston2017-06-193-3/+5
|
* Throw exception when not retrying when downloading mediaErik Johnston2017-06-131-0/+1
|
* actually trim oversize og:description metaMatthew Hodgson2017-05-221-0/+2
|
* Don't log exceptions for NotRetryingDestinationErik Johnston2017-05-151-1/+3
|
* Address review commentsRichard van der Hoff2017-03-141-15/+17
| | | | | | | - don't blindly proxy all HTTPRequestExceptions - log unexpected exceptions at error - avoid `isinstance` - improve docs on `from_http_response_exception`
* Fix routing loop when fetching remote mediaRichard van der Hoff2017-03-132-4/+38
| | | | | | | | | | | When we proxy a media request to a remote server, add a query-param, which will tell the remote server to 404 if it doesn't recognise the server_name. This should fix a routing loop where the server keeps forwarding back to itself. Also improves the error handling on remote media fetches, so that we don't always return a rather obscure 502.
* Fix dynamic thumbnails aspectJurek2017-02-241-0/+3
|
* Log which files we saved attachments to in the media_repositoryMark Haines2017-01-103-0/+11
|
* Name linearizer's for better logsErik Johnston2017-01-091-1/+1
|
* Fix crash in url preview when html tag has no textMarcin Bachry2016-12-141-1/+4
| | | | Signed-off-by: Marcin Bachry <hegel666@gmail.com>
* preview_url_resource: Ellipsis must be in unicode stringJohannes Löthberg2016-12-011-1/+1
| | | | Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
* Set CORs headers on responses from the media repoMark Haines2016-11-022-2/+4
|
* Add quotes and be explicity about script-srcErik Johnston2016-09-051-1/+2
|
* Allow PDF to be rendered from media repoErik Johnston2016-09-051-1/+7
|
* Set `Content-Security-Policy` on media repoErik Johnston2016-08-171-0/+1
| | | | | This is to inform browsers that they should sandbox the returned media. This is particularly cruical for javascript/HTML files.
* Add None check to _iterate_over_textErik Johnston2016-08-171-1/+1
|
* Flake8Erik Johnston2016-08-161-4/+1
|
* Fix up preview URL API. Add tests.Erik Johnston2016-08-161-157/+196
| | | | | | | | | This includes: - Splitting out methods of a class into stand alone functions, to make them easier to test. - Adding unit tests to split out functions, testing HTML -> preview. - Handle the fact that elements in lxml may have tail text.
* Don't include html comments in descriptionErik Johnston2016-08-051-1/+2
|
* TypoErik Johnston2016-08-041-1/+1
|
* Test summarizationErik Johnston2016-08-041-52/+54
|
* Don't infer paragrahs from newlinesErik Johnston2016-08-021-2/+1
|
* Comment on why we cloneErik Johnston2016-08-021-0/+1
|
* Spelling.Erik Johnston2016-08-021-3/+3
|
* Make it actually compileErik Johnston2016-08-021-1/+1
|
* Change the way we summarize URLsErik Johnston2016-08-021-11/+67
| | | | | | | | Using XPath is slow on some machines (for unknown reasons), so use a different approach to get a list of text nodes. Try to generate a summary that respect paragraph and then word boundaries, adding ellipses when appropriate.
* Remove raceErik Johnston2016-06-291-11/+0
|
* Implement purge_media_cache admin APIErik Johnston2016-06-292-20/+64
|
* Track approximate last access time for remote mediaErik Johnston2016-06-291-0/+24
|
* Line lengthsErik Johnston2016-06-151-3/+3
|
* Handle floats as img widthsErik Johnston2016-06-151-1/+1
|
* Handle og props with not contentErik Johnston2016-06-151-1/+2
|
* 502 on /thumbnail when can't contact remote serverErik Johnston2016-06-091-4/+10
|
* Clean up the blacklist/whitelist handling.Mark Haines2016-05-161-33/+30
| | | | | | | Always set the config key with an empty list, even if a list isn't specified. This means that the codepaths are the same for both the empty list and for a missing key. Since the behaviour is the same for both cases this makes the code somewhat easier to reason about.
* Report per request metrics for all of the things using request_handlerMark Haines2016-04-284-4/+7
|
* Add self.media_repo to PreviewUrlResourceErik Johnston2016-04-191-0/+1
|
* _make_dirs was moved to MediaRepositoryErik Johnston2016-04-191-1/+1
|
* Add store to PreviewUrlResourceErik Johnston2016-04-191-0/+1
|
* Reorder importsErik Johnston2016-04-191-7/+8
|
* Move MediaRepository to media_repository moduleErik Johnston2016-04-195-489/+496
|
* Split out BaseMediaResource into MediaRepositoryErik Johnston2016-04-196-138/+180
| | | | | | | | | This is so that a single MediaRepository can be shared across all resources, rather than having a "copy" per resource. In particular this allows us to guard against both the thumbnail and download resource triggering a download of remote content at the same time.
* explicitly pass in the charset from Content-Type to lxml to fix cyrillic ↵Matthew Hodgson2016-04-151-6/+18
| | | | woes better
* fix cyrillic URL previews by hardcoding all page decoding to UTF-8 for now, ↵Matthew Hodgson2016-04-151-16/+8
| | | | rather than relying on lxml's heuristics which seem to get it wrong
* fix urlparse import thinko breaking tiny URLsMatthew Hodgson2016-04-141-5/+5
|
* comment out 2c838f6459db35ad9812a83184d85a06ca5d940a due to risk of ↵Matthew Hodgson2016-04-141-16/+16
| | | | https://en.wikipedia.org/wiki/Billion_laughs attacks - thanks @torhve
* Sanitize the optional dependencies for spider APIErik Johnston2016-04-132-27/+3
|
* PEP8Erik Johnston2016-04-111-1/+0
|
* fix typos and needless try/except from PR reviewMatthew Hodgson2016-04-111-143/+140
|
* actually throw meaningful errorsMatthew Hodgson2016-04-081-9/+24
|
* Fix pep8 warningMark Haines2016-04-081-3/+4
|
* more PR feedbackMatthew Hodgson2016-04-081-4/+8
|
* Add url_preview_enabled config option to turn on/off preview_url endpoint. ↵Matthew Hodgson2016-04-082-18/+64
| | | | | | | | | defaults to off. Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered. Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered. Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist Add commentary and generally address PR feedback
* report image size (bytewise) in OG metaMatthew Hodgson2016-04-031-0/+2
|
* char encodingMatthew Hodgson2016-04-031-0/+1
|
* pep8Matthew Hodgson2016-04-032-55/+73
|
* fix etag typing error. fix timestamp typing errorMatthew Hodgson2016-04-031-2/+2
|
* rebase all image URLsMatthew Hodgson2016-04-031-4/+4
|
* remove stale todoMatthew Hodgson2016-04-031-3/+0
|
* Ensure only one download for a given URL is active at a timeMatthew Hodgson2016-04-031-1/+18
|
* add a persistent cache of URL lookups, and fix up the in-memory one to workMatthew Hodgson2016-04-031-10/+54
|
* support gzip compression, and don't pass through error msgsMatthew Hodgson2016-04-021-2/+3
|
* how was _respond_default_thumbnail ever meant to work?Matthew Hodgson2016-04-021-0/+2
|
* pass back SVGs as their own thumbnailsMatthew Hodgson2016-04-021-0/+20
|
* handle missing dimensions without crashingMatthew Hodgson2016-04-021-4/+13
|
* make meta comparisons case insensitiveMatthew Hodgson2016-04-021-2/+2
|
* refactor calc_og; spider image URLs; fix xpath; add a (broken) ↵Matthew Hodgson2016-04-021-81/+121
| | | | expiringcache; loads of other fixes
* fix assorted redirect, unicode and screenscraping bugsMatthew Hodgson2016-04-011-78/+96
|
* handle spidered relative images correctlyMatthew Hodgson2016-03-311-2/+12
|
* prevent choking on invalid utf-8, and handle image thumbnailing smarterMatthew Hodgson2016-03-311-5/+18
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-12-22 05:59:09 +0000