pvc/api-daemon/pvcapid
Joshua Boniface ffaa4c033f Improve handling of large file uploads
By default, Werkzeug would require the entire file (be it an OVA or
image file) to be uploaded and saved to a temporary, fake file under
`/tmp`, before any further processing could occur. This blocked most of
the execution of these functions until the upload was completed.

This entirely defeated the purpose of what I was trying to do, which was
to save the uploads directly to the temporary blockdev in each case,
thus avoiding any sort of memory or (host) disk usage.

The solution is two-fold:

  1. First, ensure that the `location='args'` value is set in
  RequestParser; without this, the `files` portion would be parsed
  during the argument parsing, which was the original source of this
  blocking behaviour.

  2. Instead of the convoluted request handling that was being done
  originally here, instead entirely defer the parsing of the `files`
  arguments until the point in the code where they are ready to be
  saved. Then, using an override stream_factory that simply opens the
  temporary blockdev, the upload can commence while being written
  directly out to it, rather than using `/tmp` space.

This does alter the error handling slightly; it is impossible to check
if the argument was passed until this point in the code, so it may take
longer to fail if the API consumer does not specify a file as they
should. This is a minor trade-off and I would expect my API consumers to
be sane here.
2020-10-19 01:00:34 -04:00
..
Daemon.py Use monkeypatch to allow multithreaded prod flask 2020-08-28 02:09:31 -04:00
__init__.py Use consistent naming of components 2020-02-08 19:34:07 -05:00
benchmark.py Disable gtod_reduce for benchmarks 2020-08-25 17:02:06 -04:00
flaskapi.py Improve handling of large file uploads 2020-10-19 01:00:34 -04:00
helper.py Improve handling of large file uploads 2020-10-19 01:00:34 -04:00
libvirt_schema.py Complete integration of OVA provisioner 2020-02-18 14:42:45 -05:00
models.py Add storage benchmarking to API 2020-08-25 01:57:21 -04:00
ova.py Improve handling of large file uploads 2020-10-19 01:00:34 -04:00
provisioner.py Correct invalid comparison in template VNI add 2020-08-18 09:48:56 -04:00