pvc/api-daemon
Joshua Boniface ffaa4c033f Improve handling of large file uploads
By default, Werkzeug would require the entire file (be it an OVA or
image file) to be uploaded and saved to a temporary, fake file under
`/tmp`, before any further processing could occur. This blocked most of
the execution of these functions until the upload was completed.

This entirely defeated the purpose of what I was trying to do, which was
to save the uploads directly to the temporary blockdev in each case,
thus avoiding any sort of memory or (host) disk usage.

The solution is two-fold:

  1. First, ensure that the `location='args'` value is set in
  RequestParser; without this, the `files` portion would be parsed
  during the argument parsing, which was the original source of this
  blocking behaviour.

  2. Instead of the convoluted request handling that was being done
  originally here, instead entirely defer the parsing of the `files`
  arguments until the point in the code where they are ready to be
  saved. Then, using an override stream_factory that simply opens the
  temporary blockdev, the upload can commence while being written
  directly out to it, rather than using `/tmp` space.

This does alter the error handling slightly; it is impossible to check
if the argument was passed until this point in the code, so it may take
longer to fail if the API consumer does not specify a file as they
should. This is a minor trade-off and I would expect my API consumers to
be sane here.
2020-10-19 01:00:34 -04:00
..
migrations Add storage benchmarking to API 2020-08-25 01:57:21 -04:00
provisioner/examples Improve handling of root disk in GRUB 2020-03-02 10:02:39 -05:00
pvcapid Improve handling of large file uploads 2020-10-19 01:00:34 -04:00
daemon_lib Rename API and common Debian packages 2020-02-08 18:50:38 -05:00
pvc-api-db-upgrade Ensure database migrations are in source control 2020-02-15 23:16:40 -05:00
pvcapid-manage.py Add database management with SQLAlchemy 2020-02-15 22:51:27 -05:00
pvcapid-worker.service Fix renamed import for worker 2020-02-18 12:20:42 -05:00
pvcapid.py Fix startup of API daemon 2020-02-08 20:27:45 -05:00
pvcapid.sample.yaml Rename sample database for API 2020-08-25 01:59:35 -04:00
pvcapid.service Use consistent naming of components 2020-02-08 19:34:07 -05:00
swagger.html Rename API and common Debian packages 2020-02-08 18:50:38 -05:00
swagger.json Rename API and common Debian packages 2020-02-08 18:50:38 -05:00