connect

LucidLink Connect - external file management.

Provides the ability to attach existing S3 datasets to a LucidLink filespace as read-only external files (1:1 file-to-S3-object mapping).

Also supports linking publicly accessible HTTP URLs directly as external files, without the need for an S3 data store.

class

lucidlink.connect.ConnectManager

ConnectManager()

Manages LucidLink Connect data stores and external file linking.

Obtained via filespace.connect after linking to a filespace. Not intended to be constructed directly.

ConnectManager example

connect = filespace.connectconnect.add_data_store("my-store", S3DataStoreConfig(    access_key="AKIA...",    secret_key="...",    bucket_name="my-bucket",    region="us-east-1",))connect.link_file("/data/file.csv", "my-store", "path/to/file.csv")

method

add_data_store(name: `str`, config: `S3DataStoreConfig`) -> `DataStoreInfo`

Add a new S3 data store.

The new data store is registered in this client's local metadata and appears in list_data_stores() immediately. It is not propagated to LucidLink services until the change is synced: call filespace.sync_all() before list_external_files(), count_external_files(), remove_data_store(), or rekey_data_store() — those are served by LucidLink services and fail while the data store has not reached them yet.

Parameters

Parameter	Description
`name`	Unique name for the data store
`config`	S3 data store configuration

name

Description: Unique name for the data store

config

Description: S3 data store configuration

Returns

DataStoreInfo for the newly created data store

Raises

Exception	Condition
`RuntimeError`	If data store creation fails
`ValueError`	If name is empty or config is invalid

RuntimeError

Condition: If data store creation fails

ValueError

Condition: If name is empty or config is invalid

method

add_http_link_data_store(name: `str`) -> `DataStoreInfo`

Add a new HTTP-link data store. Available on filespace format 3.8+

Parameters

Parameter	Description
`name`	Unique name for the data store

name

Description: Unique name for the data store

Returns

DataStoreInfo for the newly created data store

Raises

Exception	Condition
`RuntimeError`	If data store creation fails
`ValueError`	If name is empty or invalid

RuntimeError

Condition: If data store creation fails

ValueError

Condition: If name is empty or invalid

method

are_data_stores_available() -> `bool`

Check if data stores can be managed on this filespace.

Verifies both that the filespace version supports external files (V9+) and that the feature has been explicitly configured.

Returns

True if data stores are available, False otherwise

method

count_external_files(data_store_name: `str`) -> `int`

Count external files linked to a data store (no path resolution).

Much faster than iterating list_external_files() when only the count is needed, as it skips per-file path lookups in the metadata store.

Like list_external_files(), the count is served by LucidLink services: call filespace.sync_all() first so locally made changes are reflected. The same caveat about an in-progress credential rotation applies.

Parameters

Parameter	Description
`data_store_name`	Name of the data store

data_store_name

Description: Name of the data store

Returns

Number of linked files

Raises

Exception	Condition
`KeyError`	If the data store is not known to LucidLink services — including a data store added on this client but not yet synced
`RuntimeError`	If the operation fails

KeyError

Condition: If the data store is not known to LucidLink services — including a data store added on this client but not yet synced

RuntimeError

Condition: If the operation fails

method

get_data_store(name: `str`) -> `Optional[DataStoreInfo]`

Get a data store by name.

Parameters

Parameter	Description
`name`	Name of the data store

name

Description: Name of the data store

Returns

DataStoreInfo if found (includes decrypted secret_key), None otherwise

method

link_file(file_path: `str`, data_store_name: `str`, object_id: `str`, size: `int | None` = None, checksum: `str` = '') -> `None`

Link an S3 object as an external file in the filespace.

Creates a read-only file entry that maps to an S3 object via presigned URL.

The new link is written to this client's local metadata; it is not reflected by list_external_files() / count_external_files() — which are served by LucidLink services — until the change is synced — call filespace.sync_all() before listing.

Parameters

Parameter	Description
`file_path`	Path in the filespace for the new file
`data_store_name`	Name of the data store containing the object
`object_id`	S3 object key
`size`	Optional object size in bytes. When provided together with checksum, skips the S3 HeadObject call (much faster for bulk ops).
`checksum`	Optional object ETag/checksum. When provided together with size, skips the S3 HeadObject call.

file_path

Description: Path in the filespace for the new file

data_store_name

Description: Name of the data store containing the object

object_id

Description: S3 object key

size

Description: Optional object size in bytes. When provided together with checksum, skips the S3 HeadObject call (much faster for bulk ops).

checksum

Description: Optional object ETag/checksum. When provided together with size, skips the S3 HeadObject call.

Raises

Exception	Condition
`RuntimeError`	If data store not found or link fails
`ValueError`	If only one of size/checksum is provided

RuntimeError

Condition: If data store not found or link fails

ValueError

Condition: If only one of size/checksum is provided

method

link_http_file(file_path: `str`, url: `str`, data_store_name: `str | None` = None, size: `int | None` = None) -> `None`

Link an HTTP URL as an external file in the filespace.

Creates a read-only file entry that maps to content at the given URL.

Parameters

Parameter	Description
`file_path`	Path in the filespace for the new file
`url`	HTTP URL of the file content
`data_store_name`	Name of an http link data store Required on filespace format 3.8+, should be None otherwise
`size`	Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

file_path

Description: Path in the filespace for the new file

url

Description: HTTP URL of the file content

data_store_name

Description: Name of an http link data store Required on filespace format 3.8+, should be None otherwise

size

Description: Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

Raises

Exception	Condition
`ValueError`	If size is negative
`RuntimeError`	If the URL is inaccessible or expired, the data store is not found, or the link fails

ValueError

Condition: If size is negative

RuntimeError

Condition: If the URL is inaccessible or expired, the data store is not found, or the link fails

method

list_data_stores() -> `List[DataStoreInfo]`

List all registered data stores.

Served from this client's local metadata: data stores added or removed on this client are visible here immediately, without sync_all().

Returns

List of DataStoreInfo objects (secret_key will be empty)

Raises

Exception	Condition
`RuntimeError`	If operation fails

RuntimeError

Condition: If operation fails

method

list_external_files(data_store_name: `str`, limit: `int` = 100, cursor: `str` = '') -> `LinkedFilesResult`

List external files linked to a data store.

The listing is served by LucidLink services: the index of linked files is maintained there, not in this client's local metadata. Locally made changes — a newly added data store, newly linked or unlinked files — are not reflected until they are synced to LucidLink services. Call filespace.sync_all() after add_data_store(), link_file(), or unlink_file() before listing.

While a credential rotation started by rekey_data_store() is in progress (DataStoreInfo.rekey_state is IN_PROGRESS), results — in particular paginated ones — may be incomplete or unstable, because LucidLink services migrate linked files to the new key in the background. Prefer listing after the rotation completes.

Parameters

Parameter	Description
`data_store_name`	Name of the data store
`limit`	Maximum number of results (default: 100)
`cursor`	Opaque pagination token from a previous result (default: start)

data_store_name

Description: Name of the data store

limit

Description: Maximum number of results (default: 100)

cursor

Description: Opaque pagination token from a previous result (default: start)

Returns

LinkedFilesResult with the linked entries (path, entry ID, inode ID per file), pagination flag, and cursor

Raises

Exception	Condition
`KeyError`	If the data store is not known to LucidLink services — including a data store added on this client but not yet synced
`RuntimeError`	If the operation fails

KeyError

Condition: If the data store is not known to LucidLink services — including a data store added on this client but not yet synced

RuntimeError

Condition: If the operation fails

method

rekey_data_store(name: `str`, credentials: `DataStoreCredentials | None` = None, new_access_key: `str | None` = None, new_secret_key: `str | None` = None) -> `None`

Rotate credentials for a data store.

The rotation is asynchronous: this call initiates it, and LucidLink services then migrate the store's linked files to the new credentials in the background. DataStoreInfo.rekey_state reports IN_PROGRESS until the migration completes; while it is in progress, list_external_files() / count_external_files() results may be incomplete or unstable. The data store must already be synced (filespace.sync_all()) before it can be rekeyed.

Parameters

Parameter	Description
`name`	Name of the data store
`credentials`	Typed credentials object
`new_access_key`	New S3 access key ID
`new_secret_key`	New S3 secret access key

name

Description: Name of the data store

credentials

Description: Typed credentials object

new_access_key

Description: New S3 access key ID

new_secret_key

Description: New S3 secret access key

Raises

Exception	Condition
`RuntimeError`	If data store not found or rekey fails
`ValueError`	If credentials are missing or empty

RuntimeError

Condition: If data store not found or rekey fails

ValueError

Condition: If credentials are missing or empty

method

remove_data_store(name: `str`) -> `None`

Remove a data store by name.

Parameters

Parameter	Description
`name`	Name of the data store to remove

name

Description: Name of the data store to remove

Raises

Exception	Condition
`RuntimeError`	If data store not found or removal fails

RuntimeError

Condition: If data store not found or removal fails

method

unlink_file(file_path: `str`) -> `None`

Remove an external file link from the filespace.

The removal is written to this client's local metadata; list_external_files() / count_external_files() — which are served by LucidLink services — keep reporting the file until the change is synced — call filespace.sync_all() after unlinking.

Parameters

Parameter	Description
`file_path`	Path of the external file to remove

file_path

Description: Path of the external file to remove

Raises

Exception	Condition
`RuntimeError`	If file not found or unlink fails

RuntimeError

Condition: If file not found or unlink fails

method

update_http_file_url(file_path: `str`, new_url: `str`, size: `int | None` = None) -> `None`

Update the URL of an existing HTTP link file.

Parameters

Parameter	Description
`file_path`	Path of the HTTP link file in the filespace
`new_url`	New HTTP URL for the file content
`size`	Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

file_path

Description: Path of the HTTP link file in the filespace

new_url

Description: New HTTP URL for the file content

size

Description: Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

Raises

Exception	Condition
`ValueError`	If size is negative
`RuntimeError`	If the file is not found or is not an HTTP link file

ValueError

Condition: If size is negative

RuntimeError

Condition: If the file is not found or is not an HTTP link file

connect

lucidlink.connect.ConnectManager

add_data_store(name: str, config: S3DataStoreConfig) -> DataStoreInfo

Parameters

Returns

Raises

add_http_link_data_store(name: str) -> DataStoreInfo

Parameters

Returns

Raises

are_data_stores_available() -> bool

Returns

count_external_files(data_store_name: str) -> int

Parameters

Returns

Raises

get_data_store(name: str) -> Optional[DataStoreInfo]

Parameters

Returns

link_file(file_path: str, data_store_name: str, object_id: str, size: int | None = None, checksum: str = '') -> None

Parameters

Raises

link_http_file(file_path: str, url: str, data_store_name: str | None = None, size: int | None = None) -> None

Parameters

Raises

list_data_stores() -> List[DataStoreInfo]

Returns

Raises

list_external_files(data_store_name: str, limit: int = 100, cursor: str = '') -> LinkedFilesResult

Parameters

Returns

Raises

rekey_data_store(name: str, credentials: DataStoreCredentials | None = None, new_access_key: str | None = None, new_secret_key: str | None = None) -> None

Parameters

Raises

remove_data_store(name: str) -> None

Parameters

Raises

unlink_file(file_path: str) -> None

Parameters

Raises

update_http_file_url(file_path: str, new_url: str, size: int | None = None) -> None

Parameters

Raises

add_data_store(name: `str`, config: `S3DataStoreConfig`) -> `DataStoreInfo`

add_http_link_data_store(name: `str`) -> `DataStoreInfo`

are_data_stores_available() -> `bool`

count_external_files(data_store_name: `str`) -> `int`

get_data_store(name: `str`) -> `Optional[DataStoreInfo]`

link_file(file_path: `str`, data_store_name: `str`, object_id: `str`, size: `int | None` = None, checksum: `str` = '') -> `None`

link_http_file(file_path: `str`, url: `str`, data_store_name: `str | None` = None, size: `int | None` = None) -> `None`

list_data_stores() -> `List[DataStoreInfo]`

list_external_files(data_store_name: `str`, limit: `int` = 100, cursor: `str` = '') -> `LinkedFilesResult`

rekey_data_store(name: `str`, credentials: `DataStoreCredentials | None` = None, new_access_key: `str | None` = None, new_secret_key: `str | None` = None) -> `None`

remove_data_store(name: `str`) -> `None`

unlink_file(file_path: `str`) -> `None`

update_http_file_url(file_path: `str`, new_url: `str`, size: `int | None` = None) -> `None`