LucidLink Connect

What you can do

A data store is the configuration a filespace links objects from — for an S3 data store, the bucket, its region and endpoint, and the access credentials, encrypted and stored per filespace. With the Python SDK, you can:

• Add data stores — register an S3 or S3-compatible bucket, or an HTTP-link data store, with the filespace.

• Link objects — map individual objects or URLs to file paths in the filespace. After a sync, desktop, web, and mobile clients stream them.

• Bulk-link objects — link every object under a bucket prefix in one script, for example to onboard an existing media library.

• Manage data stores — list and count connected files, rotate access credentials, and remove data stores.

Add a data store

The examples assume a linked filespace, set up as in Getting started. Replace <ACCESS_KEY_ID> and <SECRET_ACCESS_KEY> with the bucket's access credentials. connect.data_stores.add() returns a handle to the new data store.

from lucidlink.connect import S3DataStoreConfig
connect = filespace.connect
# AWS S3 uses HTTPS by defaultstore = connect.data_stores.add("my-store", S3DataStoreConfig(    access_key="<ACCESS_KEY_ID>",    secret_key="<SECRET_ACCESS_KEY>",    bucket_name="my-bucket",    region="us-east-1",))

For an S3-compatible service such as Google Cloud Storage interoperability, pass the full URL, including the scheme, in endpoint and use path-style addressing. url_expiration_minutes sets the lifetime of presigned URLs: 4320 to 10080 minutes, with a default of 10080 (7 days).

gcs = connect.data_stores.add("gcs", S3DataStoreConfig(    access_key="<ACCESS_KEY_ID>",    secret_key="<SECRET_ACCESS_KEY>",    bucket_name="datasets",    region="us-central1",    endpoint="https://storage.googleapis.com",    use_virtual_addressing=False,    url_expiration_minutes=10080,))

Link objects as files

store.link() maps one object key to one file path in the filespace. The parent directory must already exist: link() raises FileNotFoundError otherwise.

# link() does not create directories: create the parents firstfilespace.file_system.create_dir("/proj1")filespace.file_system.create_dir("/proj1/dataset1")
store.link("/proj1/dataset1/file1.csv", "dataset1_file1.csv")

For bulk linking, pass each object's size and ETag — S3's ListObjectsV2 returns both — as size and checksum. The SDK then skips the HeadObject call for each file. Pass both or neither, and pass the ETag without its surrounding double quotes. The SDK stores both values as given, so a wrong value makes every read of the file fail.

The example links the objects under dataset1/ into the /proj1/dataset1 directory created above. Keys under deeper prefixes need their directories created first.

import boto3
s3 = boto3.client("s3")pages = s3.get_paginator("list_objects_v2").paginate(    Bucket="my-bucket", Prefix="dataset1/")for page in pages:    for obj in page.get("Contents", []):        store.link(            f"/proj1/{obj['Key']}",            obj["Key"],            size=obj["Size"],            # S3 returns the ETag in double quotes; pass it without them            checksum=obj["ETag"].strip('"'),        )
# Listings and counts come from LucidLink services, so push the new# store and links therefilespace.sync()

Read and list connected files

Read a connected file through filespace.file_system, like any other file. store.iter_files() lists the connected files of one data store, and store.count() counts them.

from lucidlink.connect import DataStoreKind
# A connected file reads like any other filewith filespace.file_system.open("/proj1/dataset1/file1.csv", "rb") as f:    content = f.read()
# Pages are fetched as the iteration advancesfor f in store.iter_files():    print(f.get_path())
# Count connected files without resolving their pathsprint(store.count())
# Get a store by name later; the kind gives the precise handle typestore = connect.data_stores.get("my-store", DataStoreKind.S3)

LucidLink services answer listing and counting, so both show only changes synced with sync(). While a credential rotation is in progress, listing results — paginated ones in particular — can be incomplete or unstable.

Rotate credentials

store.rekey() starts a credential rotation, and LucidLink services move the connected files to the new credentials in the background. store.info().rekey_state reports IN_PROGRESS until they are done. Keep the old credentials valid until then: files not moved yet are still read with them.

Sync the data store before rekeying it. The new credentials are checked only against a connected file that LucidLink services already have, so on an unsynced data store they are not checked.

import time
from lucidlink.connect import DataStoreRekeyState, S3Credentials
store.rekey(S3Credentials(    access_key="<NEW_ACCESS_KEY_ID>",    secret_key="<NEW_SECRET_ACCESS_KEY>",))while store.info().rekey_state is DataStoreRekeyState.IN_PROGRESS:    time.sleep(5)

Unlink files and remove a data store

Delete a connected file to unlink it. A data store can be removed only once it has no connected files and the deletes are synced.

# Unlink the files by deleting them. List them all before deleting anyfor path in [f.get_path() for f in store.iter_files()]:    filespace.file_system.delete(path)
# Remove the data store once it has no connected files and the deletes are syncedfilespace.sync()connect.data_stores.remove("my-store")

Link HTTP URLs

Link publicly reachable HTTP or HTTPS URLs as files. On filespace version 3.8 or later, HTTP links belong to an HTTP-link data store. Change a file's URL through its handle, which link(), store.get_file(path), and store.iter_files() return.

from lucidlink.connect import HttpLinkDataStoreConfig
# link() does not create directories: create the parent firstfilespace.file_system.create_dir("/web")
http_store = connect.data_stores.add("web", HttpLinkDataStoreConfig())page = http_store.link("/web/page.html", "https://example.com/page.html")# The new URL must serve content of the same sizepage.update_url("https://mirror.example.com/page.html")
# Change many URLs: list the store's files after a sync, then update each onefilespace.sync()for f in http_store.iter_files():    f.update_url(f"https://cdn.example.com{f.get_path()}")

On filespace version 3.6, HTTP links are read-only and need no data store: link them through HttpConnect_3_6(filespace), which takes the file path to change a URL.

from lucidlink.connect import HttpConnect_3_6
# link() does not create directories: create the parent firstfilespace.file_system.create_dir("/web")
http = HttpConnect_3_6(filespace)http.link("/web/page.html", "https://example.com/page.html")http.update_url("/web/page.html", "https://mirror.example.com/page.html")

For how HTTP links work, see Using LucidLink Connect with HTTP Link Files.