Using LucidLink Connect with HTTP Link Files

Introduction

LucidLink version 3.7.8703 introduces support for HTTP file linking. This allows external files to be linked within the filespace by supplying a URL as part of the link request. These are called “HTTP Link Files” to distinguish them from the “Single Object Files” used in conjunction with LucidLink-managed data stores.

Because many providers offer a custom API with an HTTP-based method for accessing files (even those who do not directly expose S3), this feature makes possible a wide assortment of integrations with new tools and storage platforms.

• Google Drive

• OneDrive

• Dropbox

• Box

• Frame.io

• and many more

HTTP file linking also supports providing pre-signed URLs generated by S3-compatible storages as an alternative to LucidLink-managed data stores and external entries.

HTTP file entries are associated with a Data Store, but not the same kind that SingleObjectFile entries use. An S3DataStore holds the credentials LucidLink uses to generate and refresh pre-signed URLs on your behalf. An HttpLinkDataStore holds no credentials and no connection information at all — it exists to group HTTP link entries and to carry the key their URLs are encrypted under. There is still no LucidLink-managed credential store authenticating or refreshing URLs on your behalf.

Whichever format you are on, LucidLink does not take over the URL lifecycle:

• LucidLink does not manage authentication or receive credentials - the user or automation is responsible for authenticating with external providers and supplying the asset URL for linking.

• LucidLink will not automatically renew, re-sign, or otherwise maintain the accessibility of HTTP file URLs (which depending on provider generally expire within a few days maximum).

• If a URL expires or becomes inaccessible, the linked file will be unreadable until updated via PATCH with a valid URL.

• LucidLink cannot see or access the content.

High-level Diagram

HTTP File Linking - API Reference (version 1.4.3 and up)

A new external entry kind, HttpLinkFile, can be used with the existing External Entries endpoints:

On filespace format 3.8 and above, create one HttpLinkDataStore in the filespace before linking any HTTP files, then reuse its ID for every entry. Unlike an S3 data store, the request carries no s3StorageParams — the name and the kind are the whole payload:

{  "name": "http-links",  "kind": "HttpLinkDataStore"}

The response returns the ID to pass as dataStoreId when linking. It is the base64url encoding of the data store name, and no s3StorageParams block is present for this kind:

{  "id": "aHR0cC1saW5rcw",  "name": "http-links",  "kind": "HttpLinkDataStore"}

• The kind must match. Linking an HttpLinkFile against an S3DataStore is rejected with “This data store type is not supported”.

• Credential rotation does not apply to this kind - there are no credentials to rotate, so updating the data store is not supported for it.

• A data store cannot be deleted while entries are still linked to it.

Creating an HTTP file entry:

{  "path": "/reports/dataset.csv",  "kind": "HttpLinkFile",  "dataStoreId": "aHR0cC1saW5rcw",  "httpFileParams": {    "url": "https://example.com/data/dataset.csv"  }}

• path — the absolute path where the file will appear in your filespace.

• kind — set to "HttpLinkFile" (as opposed to "SingleObjectFile" for S3-backed entries).

• dataStoreId — the ID of an HttpLinkDataStore. Required on filespace format 3.8 and above; must be omitted on 3.6 and below.

• httpFileParams.url — the HTTP or HTTPS URL of the file. Must include the protocol (http:// or https://).

• httpFileParams.size — optional. The size of the file in bytes. When provided, LucidLink skips the access check it would otherwise make to determine the size, so the URL is not contacted at link time.

To change the URL backing an existing HTTP file entry, send a PATCH request with the entry ID and the new URL:

{  "httpFileParams": {    "url": "https://example.com/data/dataset-v2.csv"  }}

The entryId is returned in the response when the entry is first created via POST. Only entries with kind HttpLinkFile can be updated this way. No dataStoreId or kind field is required on the update — just the new httpFileParams.url.

This is particularly useful for rotating pre-signed or time-limited URLs before they expire.

The replacement URL must serve content of the same size as the original. If the size differs, the update is rejected with “New URL content size does not match the original file size” — PATCH re-points an entry at the same asset, it does not swap in different content.

• The HTTP server hosting the file must return Content-Length and Range headers.

• The URL should directly serve a static asset (such as a media file or document) and not a webpage that embeds these resources.

  • While any URL that meets the Content-Length and Range header requirements should work, supplying the wrong type of link may result in unintended consequences (such as linking an .html file in the filespace).

  • Users of this feature should consult their providers' API documentation to ensure the correct type of URL is being used.

  • Share links from popular platforms in particular can be expected to direct to a web portal that embeds the content rather than serving it directly.

Security

HTTP Link Files are a great fit for users with stringent security requirements because only a URL needs to be provided to link a file, and that URL is securely encrypted with keys that only the filespace owner and users possess in line with the filespace’s permissions. LucidLink does not receive credentials for your storage or tools and does not control the lifecycle for the links provided - giving the customer complete control over what is shared, with whom, and for how long.

The HttpLinkDataStore required on newer filespace formats does not change this. It stores no credentials and no connection information — only the key material used to encrypt the URLs of the entries linked against it.