Upload

The upload resource represents data in the state of being uploaded. It is nested under the scope of the resource the data belongs to, that is under a clip or a file.

The following steps are necessary to perform an upload:

  1. Acquire the URI of the parent resource
  2. Create the resource (file or clip) you want to upload to
  3. Create the upload, passing the total size.
  4. Upload the first chunk using the update action.
  5. Repeat the last step until the entire data has been transferred

Chunking

Up to a certain file size, splitting the data in chunks is optional. However it is recommended to do so for large files. The chunk size is up to the client application, reasonable chunk sizes are a few megabytes up to 64 MB. The chunk size should not exceed 128 MB and there is a hard limit to the chunk size of 1000 MB. Larger files have to be split into chunks.

All chunks, except the last one, must have the same size. The last chunk must not be larger than the other chunks.

Chunk size is taken from the upload of chunk 1 which must be provided first. Alternatively you can provide an additional parameter total_chunks which allows us to determine the chunk size from any chunk. In that case the order of chunks does not matter and it is allowed to send multiple chunks in parallel.

When all chunks are provided the upload is finished automatically.

Upload Hosts

Chunk upload must not be done to the main webgate server. Instead it needs to go to an upload server. The create call returns a list of valid upload servers, that may be used for uploads. E.g.

{ "status": 201,
  "status_message": "Created",
  "info": "",
  "data": { "flowuploader::upload": {
            "current_size": 0,
            "total_size": 13,
            "hosts": ["keks.webgate.io", "praline.webgate.io", ...],
            "url":  "..." }
          } }

Since this list can change, it is not recommended to hard code one of these servers.

(Note: the create call was restricted to an upload host as well in the past, but that restriction has been removed.)

It is allowed to upload different chunks to different upload hosts.

Uploads to the main webgate server (e.g. https://webgate.io/) are rejected.

Call Details

Attributes
total_size Integer 12345 create Total size of the upload in bytes.
chunk Integer 2 update Number of chunk provided. Defaults to 1
total_chunks Integer 3 update Total number of chunks.
Actions
Create POST {resource_path}/upload necessary data: {"total_size": total upload size in bytes (int not string)}
Update PUT {resource_path}/upload?chunk=13&total_chunks=16 Header must contain: "Content-Type: application/octet-stream"

Both actions respond with data providing the current_size and total_size in the namespace flowuploader::upload E.g.

{ "status": 201,
  "status_message": "Created",
  "info": "",
  "data": { "flowuploader::upload": {
            "current_size": 0,
            "total_size": 13,
            "hosts": ["keks.webgate.io", "praline.webgate.io", ...],
            "url":  "..." }
          } }

Error Handling

When a chunk is processed correctly the webserver will send a response with status 200 (OK).

Errors

Errors, that can occur during uploads, include:

  • 404 in response to: POST/PUT The resource you are uploading to has been deleted. It is suggested to terminate the upload in this case.
  • 422 in response to: POST An upload has been created already.
  • 422 in response to: PUT Unexpected data has been received, e.g. a chunk number larger than the expected chunk numbers or an unexpected amount of data. Or the upload has not been created yet. It does not make sense to repeat the request in this case.
  • In case of other errors, e.g. a generic 500 or errors on the network level, it makes sense to retry the upload of a chunk, probably on a different upload host.

Example

The following requests demonstrate uploading the ASCII data "Hello World!!", nested under the fictional file resource "/api/projects/12/folders/33/files/45" using a chunk size of 4.

Step 1: Create upload

Request:

POST /api/projects/12/folders/33/files/45/upload HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/json

{"upload": {"total_size":13} }

Response:

HTTP/1.1 201 Created

{ "status": 201,
  "status_message": "Created",
  "info": "",
  "data": { "flowuploader::upload": {
            "current_size": 0,
            "total_size": 13,
            "hosts": ["keks.webgate.io", "praline.webgate.io", ...],
            "url":  "..." }
          } }

Step 2: Upload first chunk

Request:

PUT /api/projects/12/folders/33/files/45/upload?chunk=1 HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/octet-stream
Content-Length: 4

Hell

Response:

HTTP/1.1 200 OK

{ "status": 200,
  "status_message": "OK",
  "info": "",
  "data":  { "flowuploader::upload": {
              "current_size": 4,
              "total_size": 13,
              "url":  "..." }
           } }

Step 3: Upload second chunk

Request:

PUT /api/projects/12/folders/33/files/45/upload?chunk=2 HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/octet-stream
Content-Length: 4

o Wo

Response:

HTTP/1.1 200 OK

{ "status": 200,
  "status_message": "OK",
  "info": "",
  "data": { "flowuploader::upload": {
            "current_size": 8,
            "total_size": 13,
            "url":  "..." }
          } }

Step 4: Upload third chunk

Request:

PUT /api/projects/12/folders/33/files/45/upload?chunk=3 HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/octet-stream
Content-Length: 4

rld!

Response:

HTTP/1.1 200 OK

{ "status": 200,
  "status_message": "OK",
  "info": "",
  "data": { "flowuploader::upload": {
            "current_size": 12,
            "total_size": 13,
            "url":  "..." }
          } }

Step 5: Upload last chunk

Request:

PUT /api/projects/12/folders/33/files/45/uploadchunk=4 HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/octet-stream
Content-Length: 1

!

Response:

HTTP/1.1 200 OK

{ "status": 200,
  "status_message": "OK",
  "info": "",
  "data": { "flowuploader::upload": {
            "current_size": 12,
            "total_size": 13,
            "url":  "..." }
          } }

Tutorial - Upload a clip into a new playlist

Step 1: Choose a project for the upload!

Request:

GET /api/projects HTTP/1.1
Authorization: Bearer example-token

Response:

HTTP/1.1 200 OK

{
  "status": 200,
  "status_message": "OK",
  "info": "",
  "data": {"array":[{"id":6,"name":"Philipp's Project","starts_at":"2013-07-17","ends_at":"2015-01-08"},
                    {"id":8,"name":"project 3","starts_at":"2011-07-07","ends_at":"2055-01-01"}]}
}

Step 2: Choose a folder for the upload!

Request:

GET /api/projects/6/folders HTTP/1.1
Authorization: Bearer example-token

Response:

HTTP/1.1 200 OK

{
  "status": 200,
  "status_message": "OK",
  "info": "",
  "data": {"array":[{"id":1,"name":"philipp","project_id":6,"parent_id":null},
                    {"id":7,"name":"test 2","project_id":6,"parent_id":null},
                    {"id":8,"name":"test 3","project_id":6,"parent_id":null},
                    {"id":129,"name":"Dailies","project_id":6,"parent_id":null},
                    {"id":257,"name":"another useless folder","project_id":6,"parent_id":null},
                    {"id":258,"name":"another useless folder","project_id":6,"parent_id":null}]}
}

Step 3: (optional) List child folders of "philipp" (id=1)

Request:

GET /api/projects/6/folders?parent_id=1 HTTP/1.1
Authorization: Bearer example-token

Response:

HTTP/1.1 200 OK

{
  "status": 200,
  "status_message": "OK",
  "info": "",
  "data": {"array":[{"id":157,"name":"metadata","project_id":6,"parent_id":1},
                    {"id":158,"name":"flickr","project_id":6,"parent_id":1},
                    {"id":163,"name":"file_mass","project_id":6,"parent_id":1},
                    {"id":223,"name":"galleries","project_id":6,"parent_id":1},
                    {"id":255,"name":"another useless subfolder","project_id":6,"parent_id":1}]}
}

Step 4: Create a playlist in e.g. "metadata" (id=157)

Request:

POST /api/projects/6/folders/157/playlists HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/json

data: {"name":"how-to-upload"}

Response:

HTTP/1.1 201 Created

{
  "status": 201,
  "status_message": "Created",
  "info": "",
  "data": {"playlist":{"id":61441,"name":"how-to-upload","folder_id":157}}
}

Step 5: Create a playlist item in "how-to-upload" (id=61437)

Request:

POST /api/projects/6/folders/157/playlists/61441/items HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/json

data: {"playlist_id":61441}

Response:

HTTP/1.1 201 Created

{
  "status": 201,
  "status_message": "Created",
  "info": "",
  "data": {"playlistitem":{"id":1371,"playlist_id":61441,"clip_id":1173,"position":1}}
}

Step 6: Create an upload

Note: To upload the clip, the resource path for the upload is /api/projects/6/clips/1173 and NOT /api/projects/6/folders/157/playlists/61441/items/1371

Request:

POST /api/projects/6/clips/1167/upload HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/json

upload: {"total_size":29533258}

Response:

HTTP/1.1 201 Created

{
  "status": 201,
  "status_message": "Created",
  "info": "",
  "data": {"flowuploader::upload":{
            "current_size":0,
            "total_size":29533258,
            "hosts": ["keks.webgate.io", "praline.webgate.io", ...]}}
}

Step 7: Upload the video file

Request:

PUT /api/projects/6/clips/1167/upload HTTP/1.1
Authorization: Bearer example-token
Content-Type: application/octet-stream



cURL example:
curl -X PUT localhost:3000/api/projects/6/clips/1169/upload -H "Authorization: Bearer example-token"
-H "Content-Type: application/octet-stream" -T "/path/to/video.mp4"

Response:

HTTP/1.1 200 OK

{
  "status": 200,
  "status_message": "OK",
  "info": "",
  "data": {"flowuploader::upload":{"current_size":29533258,"total_size":29533258}}
}