Upload

The sync tasks upload service allows you to upload files directly to an uploadable data source via its sync task ID. This is the recommended way to upload files programmatically.

Finding the sync task ID

Before uploading, find the sync task ID for the data source you want to upload to. Use the ?uploadable=true filter to narrow the results:

GET /sync_tasks/?uploadable=true

Example

Request:

curl "https://nexus.stellaspark.com/api/v1/sync_tasks/?uploadable=true&token={web_api_token}"

Response:

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": 123,
            "name": "Dataset export",
            "owner": "Organization Y",
            "task_type": "data_source",
            "uploadable": true,
            ...
        }
    ]
}

The id field is the sync task ID used in the upload URL.

Uploading a file

POST /sync_tasks/{id}/upload/

Upload a new version of the data source. The request must be a multipart form upload with the file in the upload_file field. In case of timeseries measurements, rasters or files (such as photos, documents or videos) the data will be appended to the currently stored data. Any overlapping data will be overwritten with the newest version (this upload). Non-timestamped data (regular spatial datasets) will replace the data currently in Nexus. As such, the upload is considered to contain the 'latest version' of the data source.

Example

Request:

curl -F "upload_file=@C:/path/to/new/dataset.csv" "https://nexus.stellaspark.com/api/v1/sync_tasks/123/upload/?token={web_api_token}"

Response:

{
    "uploader": "Data supplier X",
    "filename": "dataset.csv",
    "upload_date": "2021-01-18T11:55:11Z",
    "download_link": "{temp_download_link}"
}

Check the tutorial Uploading data with Python for a full Python example.

Upload limits

Two limits apply to every upload:

Limit	Applies to	Example
100 MB	A single HTML, Excel or JSON file that you upload directly	A 400 MB JSON file is refused
1 GB	The uploaded file itself, whatever it contains. An archive counts as one file, at its compressed size	A 1.2 GB CSV file is refused; that same CSV zipped down to 200 MB is accepted

HTML, Excel and JSON files have their own limit because Nexus reads those formats in full while processing. All other formats are read in pieces, so those are only bound by the first limit.

Warning

Files inside an archive are not measured while the upload is accepted, but while it is processed. An archive holding a JSON file of 400 MB is therefore accepted first, and then reported as a failed upload in the uploads dialog.

Tip

If your file is too large, you can split the data into smaller files and upload them one after another. Wait for each upload to be processed before you send the next one; as long as the previous upload is still being processed, a new one is refused with a 409 response.

Response codes

Code	Meaning
201	File uploaded successfully and processing has started
400	The sync task has no associated data source, or the data source is not marked as uploadable
403	Authentication is missing or the user does not have the Data Supplier role
404	The sync task does not exist
409	The data source is currently being processed; retry once the previous upload has finished
413	The file exceeds an upload limit; see Upload limits

Upload processing

Nexus processes uploads asynchronously: your file is stored right away, and a task is queued that reads its contents into Nexus. The uploads dialog in the Viewer shows the status of that task.

Nexus also sanitizes the filename for security reasons. Only alphanumeric characters (A-Z, a-z and 0-9), hyphens (-), dots (.) and underscores (_) are retained; any other character will be removed. For example; a filename like evaporation#2020-1-1.tif will become evaporation2020-1-1.tif after upload.

Warning

This is generally not a problem, as only the file contents matter. However, if the filename itself carries special meaning that is to be interpreted by Nexus, then this might get lost in the upload process.

Tip

If you have files with special characters in the filename and you want to retain them, consider zipping the file before uploading to ensure that the filename stays the same, or rename the files before uploading.