Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Prevent "producer not unregistered" message (#5009) | Amber Brown | 2019-04-24 | 1 | -0/+4 |
| | |||||
* | Remove periods from copyright headers (#5046) | Andrew Morgan | 2019-04-11 | 1 | -1/+1 |
| | |||||
* | fix incorrect encoding of filenames with spaces in (#2090) | Matthew Hodgson | 2019-03-11 | 1 | -3/+51 |
| | | | | fixes https://github.com/vector-im/riot-web/issues/3155 | ||||
* | Fix parsing of Content-Disposition headers (#4763) | Richard van der Hoff | 2019-02-27 | 1 | -20/+65 |
| | | | | | | | | | | | * Fix parsing of Content-Disposition headers TIL: filenames in content-dispostion headers can contain semicolons, and aren't %-encoded. * fix python2 incompatibility * Fix docstrings | ||||
* | Don't log stack trace when client has gone away during media download (#4738) | Erik Johnston | 2019-02-25 | 1 | -2/+9 |
| | | | | | | | | * Don't log stack trace when client has gone away during media download * Newsfile * Fixup newsfile | ||||
* | Refactor request sending to have better excpetions (#4358) | Erik Johnston | 2019-01-08 | 1 | -3/+4 |
| | | | | | | | | | | | | | | * Correctly retry and back off if we get a HTTPerror response * Refactor request sending to have better excpetions MatrixFederationHttpClient blindly reraised exceptions to the caller without differentiating "expected" failures (e.g. connection timeouts etc) versus more severe problems (e.g. programming errors). This commit adds a RequestSendFailed exception that is raised when "expected" failures happen, allowing the TransactionQueue to log them as warnings while allowing us to log other exceptions as actual exceptions. | ||||
* | Fix IP URL previews on Python 3 (#4215) | Amber Brown | 2018-12-22 | 1 | -2/+12 |
| | |||||
* | Add 'sandbox' to CSP for media repo (#4284) | David Baker | 2018-12-11 | 1 | -1/+2 |
| | | | | | | * Add 'sandbox' to the CSP for media repo * Changelog | ||||
* | Move imports to one line | Will Hunt | 2018-12-10 | 1 | -4/+1 |
| | |||||
* | Use `send_cors` | Will Hunt | 2018-12-10 | 1 | -4/+1 |
| | |||||
* | changelog & isort | Will Hunt | 2018-12-09 | 1 | -1/+5 |
| | |||||
* | Make /config more CORS-y | Will Hunt | 2018-12-09 | 1 | -1/+3 |
| | |||||
* | Fix Content-Disposition in media repository (#4176) | Amber Brown | 2018-11-15 | 3 | -90/+110 |
| | |||||
* | Use <meta> tags to discover the per-page encoding of html previews (#4183) | Amber Brown | 2018-11-15 | 1 | -9/+22 |
| | |||||
* | Fix URL preview bugs (type error when loading cache from db, content-type ↵ | Amber Brown | 2018-11-08 | 1 | -6/+16 |
| | | | | including quotes) (#4157) | ||||
* | Delete the disused & unspecced identicon functionality (#4106) | Amber Brown | 2018-10-29 | 2 | -70/+0 |
| | |||||
* | Fix a number of flake8 errors | Richard van der Hoff | 2018-10-24 | 1 | -1/+1 |
| | | | | | | | | | | | | Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there. | ||||
* | Correctly account for cpu usage by background threads (#4074) | Richard van der Hoff | 2018-10-23 | 3 | -15/+23 |
| | | | | | | | | | | Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064 | ||||
* | Fix bug where we raised StopIteration in a generator | Erik Johnston | 2018-10-17 | 1 | -2/+5 |
| | | | | This made python 3.7 unhappy | ||||
* | Merge pull request #3932 from matrix-org/erikj/auto_start_expiring_caches | Richard van der Hoff | 2018-09-25 | 1 | -1/+0 |
|\ | | | | | Fix some instances of ExpiringCache not expiring cache items | ||||
| * | Fix some instances of ExpiringCache not expiring cache items | Erik Johnston | 2018-09-21 | 1 | -1/+0 |
| | | | | | | | | | | | | | | | | ExpiringCache required that `start()` be called before it would actually start expiring entries. A number of places didn't do that. This PR removes `start` from ExpiringCache, and automatically starts backround reaping process on creation instead. | ||||
* | | added "media-src: 'self'" to CSP for resources (#3578) | Jérémy Farnaud | 2018-09-25 | 1 | -0/+1 |
|/ | | | | | | | | | | | | | | | Synapse doesn’t allow for media resources to be played directly from Chrome. It is a problem for users on other networks (e.g. IRC) communicating with Matrix users through a gateway. The gateway sends them the raw URL for the resource when a Matrix user uploads a video and the video cannot be played directly in Chrome using that URL. Chrome argues it is not authorized to play the video because of the Content Security Policy. Chrome checks for the "media-src" policy which is missing, and defauts to the "default-src" policy which is "none". As Synapse already sends "object-src: 'self'" I thought it wouldn’t be a problem to add "media-src: 'self'" to the CSP to fix this problem. | ||||
* | Port rest/ to Python 3 (#3823) | Amber Brown | 2018-09-12 | 5 | -40/+45 |
| | |||||
* | Port over enough to get some sytests running on Python 3 (#3668) | Amber Brown | 2018-08-20 | 1 | -5/+5 |
| | |||||
* | Add GET media/v1/config (#3184) | Will Hunt | 2018-08-16 | 2 | -1/+50 |
| | |||||
* | Rename async to async_helpers because `async` is a keyword on Python 3.7 (#3678) | Amber Brown | 2018-08-10 | 2 | -2/+2 |
| | |||||
* | Refactor code for turning HttpResponseException into SynapseError | Richard van der Hoff | 2018-08-01 | 1 | -1/+1 |
| | | | | | | | | This commit replaces SynapseError.from_http_response_exception with HttpResponseException.to_synapse_error. The new method actually returns a ProxiedRequestError, which allows us to pass through additional metadata from the API call. | ||||
* | Python 3: Convert some unicode/bytes uses (#3569) | Amber Brown | 2018-08-02 | 1 | -1/+1 |
| | |||||
* | Fix some looping_call calls which were broken in #3604 | Richard van der Hoff | 2018-07-26 | 2 | -2/+2 |
| | | | | | | | | | It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call. | ||||
* | Wrap a number of things that run in the background | Richard van der Hoff | 2018-07-25 | 2 | -2/+14 |
| | | | | | This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics. | ||||
* | check isort by travis | Krombel | 2018-07-16 | 1 | -2/+2 |
| | |||||
* | Use parse_{int,str} and assert from http.servlet | Krombel | 2018-07-13 | 3 | -6/+10 |
| | | | | | | | parse_integer and parse_string can take a request and raise errors in case we have wrong or missing params. This PR tries to use them more to deduplicate some code and make it better readable | ||||
* | run isort | Amber Brown | 2018-07-09 | 12 | -92/+79 |
| | |||||
* | Attempt to be more performant on PyPy (#3462) | Amber Brown | 2018-06-28 | 2 | -2/+4 |
| | |||||
* | Pass around the reactor explicitly (#3385) | Amber Brown | 2018-06-22 | 2 | -3/+7 |
| | |||||
* | Merge pull request #3244 from NotAFile/py3-six-4 | Amber Brown | 2018-05-24 | 1 | -1/+2 |
|\ | | | | | replace some iteritems with six | ||||
| * | replace some iteritems with six | Adrian Tschira | 2018-05-19 | 1 | -1/+2 |
| | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | | py3-ize url preview | Adrian Tschira | 2018-05-19 | 1 | -3/+5 |
|/ | |||||
* | Set Server header in SynapseRequest | Richard van der Hoff | 2018-05-10 | 4 | -5/+1 |
| | | | | | | | | | | | | (instead of everywhere that writes a response. Or rather, the subset of places which write responses where we haven't forgotten it). This also means that we don't have to have the mysterious version_string attribute in anything with a request handler. Unfortunately it does mean that we have to pass the version string wherever we instantiate a SynapseSite, which has been c&ped 150 times, but that is code that ought to be cleaned up anyway really. | ||||
* | Remove redundant request_handler decorator | Richard van der Hoff | 2018-05-10 | 4 | -26/+33 |
| | | | | | | This is needless complexity; we might as well use the wrapper directly. Also rename wrap_request_handler->wrap_json_request_handler. | ||||
* | Fix logcontext leak in media repo | Richard van der Hoff | 2018-05-02 | 2 | -1/+4 |
| | | | | Make FileResponder.write_to_consumer uphold the logcontext contract | ||||
* | Merge pull request #3155 from NotAFile/py3-bytes-1 | Richard van der Hoff | 2018-04-30 | 1 | -3/+3 |
|\ | | | | | more bytes strings | ||||
| * | more bytes strings | Adrian Tschira | 2018-04-29 | 1 | -3/+3 |
| | | | | | | | | Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | | Merge pull request #3140 from matrix-org/rav/use_run_in_background | Richard van der Hoff | 2018-04-30 | 1 | -2/+3 |
|\ \ | | | | | | | Use run_in_background in preference to preserve_fn | ||||
| * | | Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background | Richard van der Hoff | 2018-04-27 | 1 | -2/+7 |
| |\| | |||||
| * | | Use run_in_background in preference to preserve_fn | Richard van der Hoff | 2018-04-27 | 1 | -2/+3 |
| | | | | | | | | | | | | | | | | | | While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object. | ||||
* | | | Merge pull request #3108 from NotAFile/py3-six-urlparse | Richard van der Hoff | 2018-04-30 | 2 | -2/+2 |
|\ \ \ | |_|/ |/| | | Use six.moves.urlparse | ||||
| * | | Use six.moves.urlparse | Adrian Tschira | 2018-04-15 | 2 | -2/+2 |
| |/ | | | | | | | | | | | The imports were shuffled around a bunch in py3 Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* / | Improve exception handling for background processes | Richard van der Hoff | 2018-04-27 | 1 | -2/+7 |
|/ | | | | | | | | | | | | | | | | | | There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background. | ||||
* | Replace old-style raise with six.reraise | Adrian Tschira | 2018-04-06 | 1 | -1/+3 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com> | ||||
* | Merge branch 'master' of github.com:matrix-org/synapse into develop | Erik Johnston | 2018-03-19 | 1 | -1/+1 |
|\ | |||||
| * | Replace ujson with simplejson | Erik Johnston | 2018-03-15 | 1 | -1/+1 |
| | | |||||
* | | Make store_file use store_into_file | Erik Johnston | 2018-02-14 | 1 | -26/+11 |
| | | |||||
* | | Tell storage providers about new file so they can upload | Erik Johnston | 2018-02-07 | 1 | -0/+6 |
| | | |||||
* | | Merge pull request #2791 from matrix-org/erikj/media_storage_refactor | Erik Johnston | 2018-02-05 | 3 | -12/+47 |
|\ \ | | | | | | | Ensure media is in local cache before thumbnailing | ||||
| * | | Use better file consumer | Erik Johnston | 2018-01-18 | 1 | -5/+10 |
| | | | |||||
| * | | Ensure media is in local cache before thumbnailing | Erik Johnston | 2018-01-18 | 3 | -10/+40 |
| | | | |||||
* | | | Handle url_previews with no content-type | Richard van der Hoff | 2018-02-02 | 1 | -21/+34 |
| | | | | | | | | | | | | | | | | | | | | | avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit. | ||||
* | | | Add federation_domain_whitelist option (#2820) | Matthew Hodgson | 2018-01-22 | 1 | -2/+17 |
| | | | | | | | | | | | | | | | | | | Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network | ||||
* | | | Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs | Richard van der Hoff | 2018-01-19 | 1 | -0/+8 |
|\ \ \ | | | | | | | | | Use the right path for url_preview thumbnails | ||||
| * | | | Use the right path for url_preview thumbnails | Richard van der Hoff | 2018-01-19 | 1 | -0/+8 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This was introduced by #2627: we were overwriting the original media for url previews with the thumbnails :/ (fixes https://github.com/vector-im/riot-web/issues/6012, hopefully) | ||||
* | | | | Fix storage provider bug introduced when renamed to store_local | Erik Johnston | 2018-01-18 | 1 | -1/+1 |
| | | | | |||||
* | | | | Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config | Erik Johnston | 2018-01-18 | 2 | -20/+27 |
|\ \ \ \ | | | | | | | | | | | Make storage providers configurable | ||||
| * | | | | Missing staticmethod | Erik Johnston | 2018-01-18 | 1 | -0/+1 |
| | | | | | |||||
| * | | | | Fixup comments | Erik Johnston | 2018-01-18 | 1 | -2/+2 |
| | | | | | |||||
| * | | | | Make storage providers more configurable | Erik Johnston | 2018-01-18 | 2 | -20/+26 |
| | |/ / | |/| | | |||||
* | | | | Fix typo in thumbnail resource causing access times to be incorrect | Erik Johnston | 2018-01-18 | 1 | -2/+2 |
| |/ / |/| | | |||||
* | | | Update last access time when thumbnails are viewed | Erik Johnston | 2018-01-17 | 1 | -0/+2 |
| | | | |||||
* | | | Keep track of last access time for local media | Erik Johnston | 2018-01-17 | 1 | -7/+25 |
|/ / | |||||
* | | Add docstring | Erik Johnston | 2018-01-17 | 1 | -1/+1 |
| | | |||||
* | | Use local vars | Erik Johnston | 2018-01-16 | 1 | -6/+2 |
| | | |||||
* | | Change _generate_thumbnails to take media_type | Erik Johnston | 2018-01-16 | 2 | -7/+10 |
| | | |||||
* | | Move setting of file_id up to caller | Erik Johnston | 2018-01-16 | 2 | -13/+14 |
| | | |||||
* | | Fix typo | Erik Johnston | 2018-01-16 | 1 | -7/+10 |
| | | |||||
* | | Fix up log lines | Erik Johnston | 2018-01-16 | 2 | -7/+10 |
| | | |||||
* | | Correctly use server_name/file_id when generating/fetching remote thumbnails | Erik Johnston | 2018-01-16 | 2 | -5/+8 |
| | | |||||
* | | Log when we respond with 404 | Erik Johnston | 2018-01-16 | 2 | -1/+8 |
| | | |||||
* | | Fix thumbnailing remote files | Erik Johnston | 2018-01-16 | 2 | -2/+30 |
| | | |||||
* | | Merge pull request #2767 from matrix-org/erikj/media_storage_refactor | Erik Johnston | 2018-01-16 | 7 | -346/+843 |
|\ \ | | | | | | | Refactor MediaRepository to separate out storage | ||||
| * | | Fix up comments | Erik Johnston | 2018-01-12 | 2 | -7/+16 |
| | | | |||||
| * | | Correctly reraise exception | Erik Johnston | 2018-01-12 | 1 | -2/+4 |
| | | | |||||
| * | | Make Responder a context manager | Erik Johnston | 2018-01-12 | 2 | -9/+10 |
| | | | |||||
| * | | Add missing class var | Erik Johnston | 2018-01-12 | 1 | -0/+3 |
| | | | |||||
| * | | Remove unnecessary condition | Erik Johnston | 2018-01-12 | 1 | -4/+1 |
| | | | |||||
| * | | Remove unused variables | Erik Johnston | 2018-01-12 | 1 | -3/+0 |
| | | | |||||
| * | | Make class var local | Erik Johnston | 2018-01-12 | 1 | -3/+3 |
| | | | |||||
| * | | Comments | Erik Johnston | 2018-01-12 | 3 | -15/+65 |
| | | | |||||
| * | | Add StorageProvider concept | Erik Johnston | 2018-01-09 | 3 | -9/+162 |
| | | | |||||
| * | | Make PreviewUrlResource use MediaStorage | Erik Johnston | 2018-01-09 | 2 | -8/+14 |
| | | | |||||
| * | | Make ThumbnailResource use MediaStorage | Erik Johnston | 2018-01-09 | 2 | -48/+68 |
| | | | |||||
| * | | Use MediaStorage for remote media | Erik Johnston | 2018-01-09 | 2 | -136/+156 |
| | | | |||||
| * | | Use MediaStorage for local files | Erik Johnston | 2018-01-09 | 2 | -119/+73 |
| | | | |||||
| * | | Add MediaStorage class | Erik Johnston | 2018-01-09 | 1 | -0/+198 |
| | | | |||||
| * | | Add some helper classes | Erik Johnston | 2018-01-09 | 1 | -0/+73 |
| | | | |||||
| * | | Split out add_file_headers | Erik Johnston | 2018-01-09 | 1 | -28/+42 |
| | | | |||||
* | | | Reinstate media download on thumbnail request | Richard van der Hoff | 2018-01-12 | 1 | -0/+5 |
|/ / | | | | | | | | | We need to actually download the remote media when we get a request for a thumbnail. | ||||
* | | Remove dead TODO | Erik Johnston | 2018-01-09 | 1 | -2/+0 |
| | | |||||
* | | Remove dead code related to default thumbnails | Erik Johnston | 2018-01-09 | 1 | -73/+3 |
|/ | |||||
* | Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error | Richard van der Hoff | 2017-11-27 | 1 | -3/+7 |
|\ | | | | | Fix error on sqlite 3.7 | ||||
| * | Check database in has_completed_background_updates | Richard van der Hoff | 2017-11-22 | 1 | -1/+1 |
| | | | | | | | | so that the right thing happens on workers. | ||||
| * | Fix error on sqlite 3.7 | Richard van der Hoff | 2017-11-21 | 1 | -3/+7 |
| | | | | | | | | | | | | | | | | | | | | | | Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572. | ||||
* | | Fix OPTIONS on preview_url | Richard van der Hoff | 2017-11-23 | 1 | -1/+5 |
|/ | | | | Fixes #2706 | ||||
* | Bit more logging | Richard van der Hoff | 2017-11-10 | 1 | -0/+2 |
| | |||||
* | Cache failures in url_preview handler | Richard van der Hoff | 2017-11-10 | 1 | -41/+45 |
| | | | | | Reshuffle the caching logic in the url_preview handler so that failures are cached (and to generally simplify things and fix the logcontext leaks). | ||||
* | Ignore noscript tags when generating URL previews | Maxime Vaillancourt | 2017-10-25 | 1 | -1/+8 |
| | |||||
* | replace 'except:' with 'except Exception:' | Richard van der Hoff | 2017-10-23 | 3 | -5/+5 |
| | | | | what could possibly go wrong | ||||
* | Fix a logcontext leak in the media repo | Richard van der Hoff | 2017-10-23 | 1 | -1/+4 |
| | |||||
* | Fix typo in thumbnail generation | Erik Johnston | 2017-10-19 | 1 | -2/+5 |
| | |||||
* | fix thumbnailing (#2548) | Krombel | 2017-10-17 | 1 | -2/+2 |
| | | | | | | | in commit 0e28281a the code for thumbnailing got refactored and the renaming of this variables was not done correctly. Signed-Off-by: Matthias Kesler <krombel@krombel.de> | ||||
* | Add try/finally block to close t_byte_source | Erik Johnston | 2017-10-13 | 1 | -24/+41 |
| | |||||
* | Remove old comment | Erik Johnston | 2017-10-13 | 1 | -1/+0 |
| | |||||
* | Don't reuse source but instead copy from primary media store to backup | Erik Johnston | 2017-10-13 | 2 | -22/+9 |
| | |||||
* | PEP8 | Erik Johnston | 2017-10-13 | 1 | -1/+1 |
| | |||||
* | Move type into key | Erik Johnston | 2017-10-13 | 1 | -3/+3 |
| | |||||
* | Comment | Erik Johnston | 2017-10-13 | 1 | -1/+1 |
| | |||||
* | Remove spurious addition | Erik Johnston | 2017-10-13 | 1 | -4/+0 |
| | |||||
* | Don't needlessly rename variable | Erik Johnston | 2017-10-13 | 1 | -13/+13 |
| | |||||
* | Add paths back in... | Erik Johnston | 2017-10-13 | 1 | -4/+4 |
| | |||||
* | Remove unnecessary diff | Erik Johnston | 2017-10-13 | 1 | -3/+1 |
| | |||||
* | Make things be absolute paths again | Erik Johnston | 2017-10-13 | 1 | -7/+7 |
| | |||||
* | s/write_to_file/write_to_file_and_backup/ | Erik Johnston | 2017-10-13 | 1 | -5/+5 |
| | |||||
* | Fix up | Erik Johnston | 2017-10-13 | 2 | -43/+32 |
| | |||||
* | Fix up thumbnailing function | Erik Johnston | 2017-10-13 | 3 | -75/+73 |
| | |||||
* | Fix typo | Erik Johnston | 2017-10-13 | 1 | -1/+1 |
| | |||||
* | Fix up comments | Erik Johnston | 2017-10-13 | 1 | -2/+26 |
| | |||||
* | Move makedirs to thread | Erik Johnston | 2017-10-13 | 1 | -2/+1 |
| | |||||
* | Use make_deferred_yieldable | Erik Johnston | 2017-10-13 | 1 | -7/+7 |
| | |||||
* | Fix comments | Erik Johnston | 2017-10-12 | 2 | -1/+4 |
| | |||||
* | Don't close prematurely.. | Erik Johnston | 2017-10-12 | 1 | -2/+5 |
| | |||||
* | getvalue closes buffer | Erik Johnston | 2017-10-12 | 1 | -11/+10 |
| | |||||
* | Get len before close | Erik Johnston | 2017-10-12 | 1 | -6/+11 |
| | |||||
* | Don't close file prematurely | Erik Johnston | 2017-10-12 | 2 | -8/+18 |
| | |||||
* | Copy everything to backup | Erik Johnston | 2017-10-12 | 4 | -76/+148 |
| | |||||
* | Typo | Erik Johnston | 2017-10-12 | 1 | -1/+1 |
| | |||||
* | Remove dead code | Erik Johnston | 2017-10-12 | 1 | -3/+0 |
| | |||||
* | Fix typos | Erik Johnston | 2017-10-12 | 1 | -20/+26 |
| | |||||
* | Fix typo | Erik Johnston | 2017-10-12 | 1 | -1/+1 |
| | |||||
* | Basic implementation of backup media store | Erik Johnston | 2017-10-12 | 3 | -126/+113 |
| | |||||
* | Only log if we've removed media | Erik Johnston | 2017-09-28 | 1 | -2/+4 |
| | |||||
* | Clear up expired url cache every 10s | Erik Johnston | 2017-09-28 | 1 | -1/+1 |
| | |||||
* | Actually assign result to variable | Erik Johnston | 2017-09-28 | 1 | -1/+1 |
| | |||||
* | Fix typo in url cache expiry timer | Erik Johnston | 2017-09-28 | 1 | -1/+1 |
| | |||||
* | Support new and old style media id formats | Erik Johnston | 2017-09-28 | 1 | -31/+81 |
| | |||||
* | Change expires column to expires_ts | Erik Johnston | 2017-09-28 | 1 | -2/+2 |
| | |||||
* | Delete expired url cache data | Erik Johnston | 2017-09-28 | 2 | -4/+129 |
| | |||||
* | Store URL cache preview downloads seperately | Erik Johnston | 2017-06-23 | 5 | -18/+72 |
| | | | | This makes it easier to clear old media out at a later date | ||||
* | Add API to quarantine media | Erik Johnston | 2017-06-19 | 3 | -3/+5 |
| | |||||
* | Throw exception when not retrying when downloading media | Erik Johnston | 2017-06-13 | 1 | -0/+1 |
| | |||||
* | actually trim oversize og:description meta | Matthew Hodgson | 2017-05-22 | 1 | -0/+2 |
| | |||||
* | Don't log exceptions for NotRetryingDestination | Erik Johnston | 2017-05-15 | 1 | -1/+3 |
| | |||||
* | Address review comments | Richard van der Hoff | 2017-03-14 | 1 | -15/+17 |
| | | | | | | | - don't blindly proxy all HTTPRequestExceptions - log unexpected exceptions at error - avoid `isinstance` - improve docs on `from_http_response_exception` | ||||
* | Fix routing loop when fetching remote media | Richard van der Hoff | 2017-03-13 | 2 | -4/+38 |
| | | | | | | | | | | | When we proxy a media request to a remote server, add a query-param, which will tell the remote server to 404 if it doesn't recognise the server_name. This should fix a routing loop where the server keeps forwarding back to itself. Also improves the error handling on remote media fetches, so that we don't always return a rather obscure 502. | ||||
* | Fix dynamic thumbnails aspect | Jurek | 2017-02-24 | 1 | -0/+3 |
| | |||||
* | Log which files we saved attachments to in the media_repository | Mark Haines | 2017-01-10 | 3 | -0/+11 |
| | |||||
* | Name linearizer's for better logs | Erik Johnston | 2017-01-09 | 1 | -1/+1 |
| | |||||
* | Fix crash in url preview when html tag has no text | Marcin Bachry | 2016-12-14 | 1 | -1/+4 |
| | | | | Signed-off-by: Marcin Bachry <hegel666@gmail.com> | ||||
* | preview_url_resource: Ellipsis must be in unicode string | Johannes Löthberg | 2016-12-01 | 1 | -1/+1 |
| | | | | Signed-off-by: Johannes Löthberg <johannes@kyriasis.com> | ||||
* | Set CORs headers on responses from the media repo | Mark Haines | 2016-11-02 | 2 | -2/+4 |
| | |||||
* | Add quotes and be explicity about script-src | Erik Johnston | 2016-09-05 | 1 | -1/+2 |
| | |||||
* | Allow PDF to be rendered from media repo | Erik Johnston | 2016-09-05 | 1 | -1/+7 |
| | |||||
* | Set `Content-Security-Policy` on media repo | Erik Johnston | 2016-08-17 | 1 | -0/+1 |
| | | | | | This is to inform browsers that they should sandbox the returned media. This is particularly cruical for javascript/HTML files. | ||||
* | Add None check to _iterate_over_text | Erik Johnston | 2016-08-17 | 1 | -1/+1 |
| | |||||
* | Flake8 | Erik Johnston | 2016-08-16 | 1 | -4/+1 |
| | |||||
* | Fix up preview URL API. Add tests. | Erik Johnston | 2016-08-16 | 1 | -157/+196 |
| | | | | | | | | | This includes: - Splitting out methods of a class into stand alone functions, to make them easier to test. - Adding unit tests to split out functions, testing HTML -> preview. - Handle the fact that elements in lxml may have tail text. | ||||
* | Don't include html comments in description | Erik Johnston | 2016-08-05 | 1 | -1/+2 |
| | |||||
* | Typo | Erik Johnston | 2016-08-04 | 1 | -1/+1 |
| | |||||
* | Test summarization | Erik Johnston | 2016-08-04 | 1 | -52/+54 |
| | |||||
* | Don't infer paragrahs from newlines | Erik Johnston | 2016-08-02 | 1 | -2/+1 |
| | |||||
* | Comment on why we clone | Erik Johnston | 2016-08-02 | 1 | -0/+1 |
| | |||||
* | Spelling. | Erik Johnston | 2016-08-02 | 1 | -3/+3 |
| | |||||
* | Make it actually compile | Erik Johnston | 2016-08-02 | 1 | -1/+1 |
| | |||||
* | Change the way we summarize URLs | Erik Johnston | 2016-08-02 | 1 | -11/+67 |
| | | | | | | | | Using XPath is slow on some machines (for unknown reasons), so use a different approach to get a list of text nodes. Try to generate a summary that respect paragraph and then word boundaries, adding ellipses when appropriate. | ||||
* | Remove race | Erik Johnston | 2016-06-29 | 1 | -11/+0 |
| | |||||
* | Implement purge_media_cache admin API | Erik Johnston | 2016-06-29 | 2 | -20/+64 |
| | |||||
* | Track approximate last access time for remote media | Erik Johnston | 2016-06-29 | 1 | -0/+24 |
| | |||||
* | Remove the legacy v0 content upload API. | Mark Haines | 2016-06-21 | 1 | -110/+2 |
| | | | | | | The existing content can still be downloaded. The last upload to the matrix.org server was in January 2015, so it is probably safe to remove the upload API. | ||||
* | Line lengths | Erik Johnston | 2016-06-15 | 1 | -3/+3 |
| | |||||
* | Handle floats as img widths | Erik Johnston | 2016-06-15 | 1 | -1/+1 |
| | |||||
* | Handle og props with not content | Erik Johnston | 2016-06-15 | 1 | -1/+2 |
| | |||||
* | 502 on /thumbnail when can't contact remote server | Erik Johnston | 2016-06-09 | 1 | -4/+10 |
| | |||||
* | Clean up the blacklist/whitelist handling. | Mark Haines | 2016-05-16 | 1 | -33/+30 |
| | | | | | | | Always set the config key with an empty list, even if a list isn't specified. This means that the codepaths are the same for both the empty list and for a missing key. Since the behaviour is the same for both cases this makes the code somewhat easier to reason about. | ||||
* | Report per request metrics for all of the things using request_handler | Mark Haines | 2016-04-28 | 4 | -4/+7 |
| | |||||
* | Add self.media_repo to PreviewUrlResource | Erik Johnston | 2016-04-19 | 1 | -0/+1 |
| | |||||
* | _make_dirs was moved to MediaRepository | Erik Johnston | 2016-04-19 | 1 | -1/+1 |
| | |||||
* | Add store to PreviewUrlResource | Erik Johnston | 2016-04-19 | 1 | -0/+1 |
| | |||||
* | Reorder imports | Erik Johnston | 2016-04-19 | 1 | -7/+8 |
| | |||||
* | Move MediaRepository to media_repository module | Erik Johnston | 2016-04-19 | 5 | -489/+496 |
| | |||||
* | Split out BaseMediaResource into MediaRepository | Erik Johnston | 2016-04-19 | 6 | -138/+180 |
| | | | | | | | | | This is so that a single MediaRepository can be shared across all resources, rather than having a "copy" per resource. In particular this allows us to guard against both the thumbnail and download resource triggering a download of remote content at the same time. | ||||
* | explicitly pass in the charset from Content-Type to lxml to fix cyrillic ↵ | Matthew Hodgson | 2016-04-15 | 1 | -6/+18 |
| | | | | woes better | ||||
* | fix cyrillic URL previews by hardcoding all page decoding to UTF-8 for now, ↵ | Matthew Hodgson | 2016-04-15 | 1 | -16/+8 |
| | | | | rather than relying on lxml's heuristics which seem to get it wrong | ||||
* | fix urlparse import thinko breaking tiny URLs | Matthew Hodgson | 2016-04-14 | 1 | -5/+5 |
| | |||||
* | comment out 2c838f6459db35ad9812a83184d85a06ca5d940a due to risk of ↵ | Matthew Hodgson | 2016-04-14 | 1 | -16/+16 |
| | | | | https://en.wikipedia.org/wiki/Billion_laughs attacks - thanks @torhve | ||||
* | Sanitize the optional dependencies for spider API | Erik Johnston | 2016-04-13 | 2 | -27/+3 |
| | |||||
* | PEP8 | Erik Johnston | 2016-04-11 | 1 | -1/+0 |
| | |||||
* | fix typos and needless try/except from PR review | Matthew Hodgson | 2016-04-11 | 1 | -143/+140 |
| | |||||
* | actually throw meaningful errors | Matthew Hodgson | 2016-04-08 | 1 | -9/+24 |
| | |||||
* | Fix pep8 warning | Mark Haines | 2016-04-08 | 1 | -3/+4 |
| | |||||
* | more PR feedback | Matthew Hodgson | 2016-04-08 | 1 | -4/+8 |
| | |||||
* | Add url_preview_enabled config option to turn on/off preview_url endpoint. ↵ | Matthew Hodgson | 2016-04-08 | 2 | -18/+64 |
| | | | | | | | | | defaults to off. Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered. Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered. Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist Add commentary and generally address PR feedback | ||||
* | report image size (bytewise) in OG meta | Matthew Hodgson | 2016-04-03 | 1 | -0/+2 |
| | |||||
* | char encoding | Matthew Hodgson | 2016-04-03 | 1 | -0/+1 |
| | |||||
* | pep8 | Matthew Hodgson | 2016-04-03 | 2 | -55/+73 |
| | |||||
* | fix etag typing error. fix timestamp typing error | Matthew Hodgson | 2016-04-03 | 1 | -2/+2 |
| | |||||
* | rebase all image URLs | Matthew Hodgson | 2016-04-03 | 1 | -4/+4 |
| | |||||
* | remove stale todo | Matthew Hodgson | 2016-04-03 | 1 | -3/+0 |
| | |||||
* | Ensure only one download for a given URL is active at a time | Matthew Hodgson | 2016-04-03 | 1 | -1/+18 |
| | |||||
* | add a persistent cache of URL lookups, and fix up the in-memory one to work | Matthew Hodgson | 2016-04-03 | 1 | -10/+54 |
| | |||||
* | support gzip compression, and don't pass through error msgs | Matthew Hodgson | 2016-04-02 | 1 | -2/+3 |
| | |||||
* | how was _respond_default_thumbnail ever meant to work? | Matthew Hodgson | 2016-04-02 | 1 | -0/+2 |
| | |||||
* | pass back SVGs as their own thumbnails | Matthew Hodgson | 2016-04-02 | 1 | -0/+20 |
| | |||||
* | handle missing dimensions without crashing | Matthew Hodgson | 2016-04-02 | 1 | -4/+13 |
| | |||||
* | make meta comparisons case insensitive | Matthew Hodgson | 2016-04-02 | 1 | -2/+2 |
| | |||||
* | refactor calc_og; spider image URLs; fix xpath; add a (broken) ↵ | Matthew Hodgson | 2016-04-02 | 1 | -81/+121 |
| | | | | expiringcache; loads of other fixes | ||||
* | fix assorted redirect, unicode and screenscraping bugs | Matthew Hodgson | 2016-04-01 | 1 | -78/+96 |
| | |||||
* | handle spidered relative images correctly | Matthew Hodgson | 2016-03-31 | 1 | -2/+12 |
| | |||||
* | prevent choking on invalid utf-8, and handle image thumbnailing smarter | Matthew Hodgson | 2016-03-31 | 1 | -5/+18 |
| | |||||
* | synthesise basig OG metadata from pages lacking it | Matthew Hodgson | 2016-03-31 | 1 | -0/+47 |
| | |||||
* | handle requests with missing content-length headers (e.g. YouTube) | Matthew Hodgson | 2016-03-31 | 1 | -2/+2 |
| | |||||
* | fix bugs | Matthew Hodgson | 2016-03-29 | 1 | -5/+6 |
| | |||||
* | debug | Matthew Hodgson | 2016-03-29 | 1 | -3/+2 |
| | |||||
* | make it work | Matthew Hodgson | 2016-03-29 | 2 | -55/+77 |
| | |||||
* | make it build | Matthew Hodgson | 2016-03-29 | 1 | -8/+9 |
| | |||||
* | Merge branch 'develop' into matthew/preview_urls | Matthew Hodgson | 2016-03-27 | 2 | -8/+11 |
|\ | |||||
| * | Catch the exceptions thrown by twisted when you write to a closed connection | Mark Haines | 2016-02-12 | 2 | -4/+4 |
| | | |||||
| * | Preserve log context over when deferring to thread pool in media repo | Erik Johnston | 2016-02-03 | 1 | -4/+7 |
| | | |||||
* | | initial WIP of a tentative preview_url endpoint - incomplete, untested, ↵ | Matthew Hodgson | 2016-01-24 | 2 | -0/+167 |
|/ | | | | experimental, etc. just putting it here for safekeeping for now | ||||
* | Merge pull request #478 from matrix-org/daniel/userobject | Daniel Wagner-Hall | 2016-01-11 | 2 | -5/+5 |
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | Introduce a User object I'm sick of passing around more and more things as tuple items around the whole world, and needing to edit every call site every time there is more information about a user. So pass them around together as an object. This object has incredibly poorly named fields because we have a convention that `user` indicates a UserID object, and `user_id` indicates a string. I tried to clean up the whole repo to fix this, but gave up. So instead, I introduce a second convention. A user_object is a User, and a user_id_object is a UserId. I may have cried a little bit. | ||||
| * | Introduce a Requester object | Daniel Wagner-Hall | 2016-01-11 | 2 | -5/+5 |
| | | | | | | | | | | | | | | | | | | This tracks data about the entity which made the request. This is instead of passing around a tuple, which requires call-site modifications every time a new piece of optional context is passed around. I tried to introduce a User object. I gave up. | ||||
* | | Only use cropped thumbnails when asked for a cropped thumbnail. | Mark Haines | 2016-01-07 | 1 | -1/+1 |
|/ | | | | Even though ones cropped with scale might be technically valid. | ||||
* | copyrights | Matthew Hodgson | 2016-01-07 | 9 | -9/+9 |
| | |||||
* | Allow guests to upgrade their accounts | Daniel Wagner-Hall | 2016-01-05 | 1 | -1/+1 |
| | |||||
* | Use larger thumbnail rather than smaller. | Erik Johnston | 2016-01-05 | 1 | -4/+13 |
| | |||||
* | Allow guests to register and call /events?room_id= | Daniel Wagner-Hall | 2015-11-04 | 2 | -2/+2 |
| | | | | | | | This follows the same flows-based flow as regular registration, but as the only implemented flow has no requirements, it auto-succeeds. In the future, other flows (e.g. captcha) may be required, so clients should treat this like the regular registration flow choices. | ||||
* | Merge branch 'release-v0.10.0' into develop | Mark Haines | 2015-08-28 | 2 | -15/+51 |
|\ | | | | | | | | | | | | | Conflicts: synapse/handlers/auth.py synapse/python_dependencies.py synapse/rest/client/v1/login.py | ||||
| * | Consistency | Erik Johnston | 2015-08-27 | 1 | -3/+3 |
| | | |||||
| * | Check for an internationalised filename first | Erik Johnston | 2015-08-27 | 1 | -8/+14 |
| | | |||||
| * | None check the correct variable | Erik Johnston | 2015-08-27 | 1 | -2/+3 |
| | | |||||
| * | Check for None | Erik Johnston | 2015-08-27 | 1 | -1/+1 |
| | | |||||
| * | Give a sensible error message if the filename is invalid UTF-8 | Mark Haines | 2015-08-26 | 2 | -2/+8 |
| | | |||||
| * | Handle unicode filenames given when downloading or received over federation | Mark Haines | 2015-08-26 | 1 | -5/+21 |
| | | |||||
| * | Allow non-ascii filenames for attachments | Mark Haines | 2015-08-26 | 2 | -8/+15 |
| | | |||||
* | | Remove completely unused concepts from codebase | Daniel Wagner-Hall | 2015-08-25 | 2 | -2/+2 |
|/ | | | | | | | | | | Removes device_id and ClientInfo device_id is never actually written, and the matrix.org DB has no non-null entries for it. Right now, it's just cluttering up code. This doesn't remove the columns from the database, because that's fiddly. | ||||
* | Merge remote-tracking branch 'origin/develop' into ↵ | Mark Haines | 2015-08-13 | 1 | -34/+43 |
|\ | | | | | | | erikj/generate_presice_thumbnails | ||||
| * | Generate local thumbnails on a thread | Erik Johnston | 2015-07-23 | 1 | -34/+43 |
| | | |||||
* | | Add config option for setting the list of thumbnail sizes to precalculate | Mark Haines | 2015-08-12 | 1 | -16/+2 |
| | | |||||
* | | Make a config option for whether to generate new thumbnail sizes dynamically | Mark Haines | 2015-08-12 | 2 | -7/+19 |
| | | |||||
* | | Factor out thumbnail() | Erik Johnston | 2015-07-23 | 1 | -56/+40 |
| | | |||||
* | | Fix remote thumbnailing | Erik Johnston | 2015-07-23 | 2 | -7/+11 |
| | | |||||
* | | Always return a thumbnail of the requested size. | Erik Johnston | 2015-07-23 | 2 | -2/+162 |
|/ | | | | | | Before, we returned a thumbnail that was at least as big (if possible) as the requested size. Now, if we don't have a thumbnail of the given size we generate (and persist) one of that size. | ||||
* | Pick larger than desired thumbnail for 'crop' | Erik Johnston | 2015-07-23 | 1 | -1/+2 |
| |