connect

LucidLink Connect - external file management.

Provides the ability to attach existing S3 datasets to a LucidLink filespace as read-only external files (1:1 file-to-S3-object mapping).

Also supports linking publicly accessible HTTP URLs directly as external files, without the need for an S3 data store.

class

lucidlink.connect.ConnectManager

ConnectManager()

Manages LucidLink Connect data stores and external file linking.

Obtained via filespace.connect after linking to a filespace. Not intended to be constructed directly.

Example

connect = filespace.connectconnect.add_data_store("my-store", S3DataStoreConfig(    access_key="AKIA...",    secret_key="...",    bucket_name="my-bucket",    region="us-east-1",))connect.link_file("/data/file.csv", "my-store", "path/to/file.csv")

method

add_data_store(name: `str`, config: `S3DataStoreConfig`) -> `DataStoreInfo`

Add a new S3 data store.

Parameters

Parameter	Description
`name`	Unique name for the data store
`config`	S3 data store configuration

name

Description: Unique name for the data store

config

Description: S3 data store configuration

Returns

DataStoreInfo for the newly created data store

Raises

Exception	Condition
`RuntimeError`	If data store creation fails
`ValueError`	If name is empty or config is invalid

RuntimeError

Condition: If data store creation fails

ValueError

Condition: If name is empty or config is invalid

method

are_data_stores_available() -> `bool`

Check if data stores can be managed on this filespace.

Verifies both that the filespace version supports external files (V9+) and that the feature has been explicitly configured.

Returns

True if data stores are available, False otherwise

method

count_external_files(data_store_name: `str`) -> `int`

Count external files linked to a data store (no path resolution).

Much faster than iterating list_external_files() when only the count is needed, as it skips per-file path lookups in the metadata store.

Parameters

Parameter	Description
`data_store_name`	Name of the data store

data_store_name

Description: Name of the data store

Returns

Number of linked files

method

get_data_store(name: `str`) -> `Optional[DataStoreInfo]`

Get a data store by name.

Parameters

Parameter	Description
`name`	Name of the data store

name

Description: Name of the data store

Returns

DataStoreInfo if found (includes decrypted secret_key), None otherwise

method

link_file(file_path: `str`, data_store_name: `str`, object_id: `str`, size: `int | None` = None, checksum: `str` = '') -> `None`

Link an S3 object as an external file in the filespace.

Creates a read-only file entry that maps to an S3 object via presigned URL.

Parameters

Parameter	Description
`file_path`	Path in the filespace for the new file
`data_store_name`	Name of the data store containing the object
`object_id`	S3 object key
`size`	Optional object size in bytes. When provided together with checksum, skips the S3 HeadObject call (much faster for bulk ops).
`checksum`	Optional object ETag/checksum. When provided together with size, skips the S3 HeadObject call.

file_path

Description: Path in the filespace for the new file

data_store_name

Description: Name of the data store containing the object

object_id

Description: S3 object key

size

Description: Optional object size in bytes. When provided together with checksum, skips the S3 HeadObject call (much faster for bulk ops).

checksum

Description: Optional object ETag/checksum. When provided together with size, skips the S3 HeadObject call.

Raises

Exception	Condition
`RuntimeError`	If data store not found or link fails
`ValueError`	If only one of size/checksum is provided

RuntimeError

Condition: If data store not found or link fails

ValueError

Condition: If only one of size/checksum is provided

method

link_http_file(file_path: `str`, url: `str`, size: `int | None` = None) -> `None`

Link an HTTP URL as an external file in the filespace.

Creates a read-only file entry that maps to content at the given URL.

Parameters

Parameter	Description
`file_path`	Path in the filespace for the new file
`url`	HTTP URL of the file content
`size`	Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

file_path

Description: Path in the filespace for the new file

url

Description: HTTP URL of the file content

size

Description: Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

Raises

Exception	Condition
`ValueError`	If size is negative
`RuntimeError`	If the URL is inaccessible, expired, or the link fails

ValueError

Condition: If size is negative

RuntimeError

Condition: If the URL is inaccessible, expired, or the link fails

method

list_data_stores() -> `List[DataStoreInfo]`

List all registered data stores.

Returns

List of DataStoreInfo objects (secret_key will be empty)

Raises

Exception	Condition
`RuntimeError`	If operation fails

RuntimeError

Condition: If operation fails

method

list_external_files(data_store_name: `str`, limit: `int` = 100, cursor: `str` = '') -> `LinkedFilesResult`

List external files linked to a data store.

Parameters

Parameter	Description
`data_store_name`	Name of the data store
`limit`	Maximum number of results (default: 100)
`cursor`	Opaque pagination token from a previous result (default: start)

data_store_name

Description: Name of the data store

limit

Description: Maximum number of results (default: 100)

cursor

Description: Opaque pagination token from a previous result (default: start)

Returns

LinkedFilesResult with file paths, pagination flag, and cursor

method

rekey_data_store(name: `str`, credentials: `DataStoreCredentials | None` = None, new_access_key: `str | None` = None, new_secret_key: `str | None` = None) -> `None`

Rotate credentials for a data store.

Parameters

Parameter	Description
`name`	Name of the data store
`credentials`	Typed credentials object
`new_access_key`	New S3 access key ID
`new_secret_key`	New S3 secret access key

name

Description: Name of the data store

credentials

Description: Typed credentials object

new_access_key

Description: New S3 access key ID

new_secret_key

Description: New S3 secret access key

Raises

Exception	Condition
`RuntimeError`	If data store not found or rekey fails
`ValueError`	If credentials are missing or empty

RuntimeError

Condition: If data store not found or rekey fails

ValueError

Condition: If credentials are missing or empty

method

remove_data_store(name: `str`) -> `None`

Remove a data store by name.

Parameters

Parameter	Description
`name`	Name of the data store to remove

name

Description: Name of the data store to remove

Raises

Exception	Condition
`RuntimeError`	If data store not found or removal fails

RuntimeError

Condition: If data store not found or removal fails

method

unlink_file(file_path: `str`) -> `None`

Remove an external file link from the filespace.

Parameters

Parameter	Description
`file_path`	Path of the external file to remove

file_path

Description: Path of the external file to remove

Raises

Exception	Condition
`RuntimeError`	If file not found or unlink fails

RuntimeError

Condition: If file not found or unlink fails

method

update_http_file_url(file_path: `str`, new_url: `str`, size: `int | None` = None) -> `None`

Update the URL of an existing HTTP link file.

Parameters

Parameter	Description
`file_path`	Path of the HTTP link file in the filespace
`new_url`	New HTTP URL for the file content
`size`	Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

file_path

Description: Path of the HTTP link file in the filespace

new_url

Description: New HTTP URL for the file content

size

Description: Optional file size in bytes. When provided, skips the access check and size validation (useful for bulk operations where size is known in advance).

Raises

Exception	Condition
`ValueError`	If size is negative
`RuntimeError`	If the file is not found or is not an HTTP link file

ValueError

Condition: If size is negative

RuntimeError

Condition: If the file is not found or is not an HTTP link file

connect

lucidlink.connect.ConnectManager

add_data_store(name: str, config: S3DataStoreConfig) -> DataStoreInfo

Parameters

Returns

Raises

are_data_stores_available() -> bool

Returns

count_external_files(data_store_name: str) -> int

Parameters

Returns

get_data_store(name: str) -> Optional[DataStoreInfo]

Parameters

Returns

link_file(file_path: str, data_store_name: str, object_id: str, size: int | None = None, checksum: str = '') -> None

Parameters

Raises

link_http_file(file_path: str, url: str, size: int | None = None) -> None

Parameters

Raises

list_data_stores() -> List[DataStoreInfo]

Returns

Raises

list_external_files(data_store_name: str, limit: int = 100, cursor: str = '') -> LinkedFilesResult

Parameters

Returns

rekey_data_store(name: str, credentials: DataStoreCredentials | None = None, new_access_key: str | None = None, new_secret_key: str | None = None) -> None

Parameters

Raises

remove_data_store(name: str) -> None

Parameters

Raises

unlink_file(file_path: str) -> None

Parameters

Raises

update_http_file_url(file_path: str, new_url: str, size: int | None = None) -> None

Parameters

Raises

add_data_store(name: `str`, config: `S3DataStoreConfig`) -> `DataStoreInfo`

are_data_stores_available() -> `bool`

count_external_files(data_store_name: `str`) -> `int`

get_data_store(name: `str`) -> `Optional[DataStoreInfo]`

link_file(file_path: `str`, data_store_name: `str`, object_id: `str`, size: `int | None` = None, checksum: `str` = '') -> `None`

link_http_file(file_path: `str`, url: `str`, size: `int | None` = None) -> `None`

list_data_stores() -> `List[DataStoreInfo]`

list_external_files(data_store_name: `str`, limit: `int` = 100, cursor: `str` = '') -> `LinkedFilesResult`

rekey_data_store(name: `str`, credentials: `DataStoreCredentials | None` = None, new_access_key: `str | None` = None, new_secret_key: `str | None` = None) -> `None`

remove_data_store(name: `str`) -> `None`

unlink_file(file_path: `str`) -> `None`

update_http_file_url(file_path: `str`, new_url: `str`, size: `int | None` = None) -> `None`