connect

LucidLink Connect - external file management.

Links existing content into a filespace as files, without copying it:

S3 objects, through S3 data stores.
HTTP/HTTPS URLs, through HTTP-link data stores (filespace format 3.8 or newer).

On filespace format 3.6, linked files are read-only, and HTTP URLs are linked without a data store via HttpConnect_3_6(filespace). On 3.8 or newer, linked files can be written; writes stay in the filespace and never modify the S3 object or the URL's content.

Obtained via filespace.connect; data stores are managed through filespace.connect.data_stores. As on the Filespace, calls on these objects raise FilespaceNotLinkedError once the filespace is unlinked, FilespaceInternalError for an internal error of the filespace and ClientClosedError once the owning Client is closed.

A name, object key, URL, credential or checksum containing a NUL character raises InvalidArgumentError (a ValueError). Sizes are integers from 0 to 2**63 - 1: one out of that range raises InvalidArgumentError, a non-integer TypeError.

class

lucidlink.connect.ConnectManager

ConnectManager()

Manages LucidLink Connect data stores and external file linking.

Obtained via filespace.connect after linking to a filespace. Not intended to be constructed directly.

Every Connect call raises UnsupportedOperationError when Connect is not available on the filespace and FilespaceNotLinkedError once the filespace is unlinked.

ConnectManager example

from lucidlink.connect import S3DataStoreConfig
connect = filespace.connectstore = connect.data_stores.add("my-store", S3DataStoreConfig(    access_key="AKIA...",    secret_key="...",    bucket_name="my-bucket",    region="us-east-1",))store.link("/data/file.csv", "path/to/file.csv")

property

data_stores: `DataStores`

The filespace's data stores.

class

lucidlink.connect.DataStore

DataStore()

A data store in this filespace, addressed by name.

A handle refers to the data store by its name, so info() always returns the current details.

Obtained from connect.data_stores; not constructed directly.

property

kind: `DataStoreKind`

Data store type.

property

name: `str`

Data store name.

method

count() -> `int`

Count the files linked from this data store.

Faster than iter_files() when only the number is needed. Like iter_files(), the count comes from LucidLink services, so it shows only synced changes: call filespace.sync() first. On an S3 data store, the credential rotation note in S3DataStore.iter_files() applies too.

Returns

Number of linked files.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, the name now refers to a data store of the other kind, or the data store has not been synced yet (a KeyError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, the name now refers to a data store of the other kind, or the data store has not been synced yet (a KeyError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

get_file(path: `str`) -> `ExternalFile`

Get a handle to the external file at path, linked from this data store.

Parameters

Parameter	Description
`path`	Path of the external file in the filespace.

path

Description: Path of the external file in the filespace.

Returns

A handle to the file.

Raises

Exception	Condition
`FsNotFoundError`	If path does not exist (a FileNotFoundError).
`InvalidArgumentError`	If path is not an external file (a directory or symbolic link too), or the file is linked from another data store or from no existing one (the error names it; a ValueError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

FsNotFoundError

Condition: If path does not exist (a FileNotFoundError).

InvalidArgumentError

Condition: If path is not an external file (a directory or symbolic link too), or the file is linked from another data store or from no existing one (the error names it; a ValueError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

info() -> `DataStoreInfo`

Read the data store's current details.

Returns

The data store's details.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, or the name now refers to a data store of the other kind (a KeyError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, or the name now refers to a data store of the other kind (a KeyError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

iter_files() -> `Iterator[ExternalFile]`

Iterate over the files linked from this data store.

The listing comes from LucidLink services, so it shows only synced changes: call filespace.sync() after connect.data_stores.add() or link() before listing (see "Sync changes" in the Getting started guide). Sync after filespace.file_system.delete() too: a listed file that was deleted here but not yet synced makes the listing raise InternalError.

Files are fetched a page at a time as the iterator advances. This call fetches the first page, so an unknown data store raises here rather than on the first next().

Iterating can raise the same errors when it fetches a later page, except the kind error.

Returns

An iterator over handles to the linked files.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, the name now refers to a data store of the other kind, or the data store has not been synced yet (a KeyError).
`InternalError`	If a listed file was deleted here and the delete is not synced yet.
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, the name now refers to a data store of the other kind, or the data store has not been synced yet (a KeyError).

InternalError

Condition: If a listed file was deleted here and the delete is not synced yet.

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

remove() -> `None`

Remove this data store; same as connect.data_stores.remove(name), except that the store must still be of this handle's kind.

Sync first: removing asks LucidLink services whether the store still has linked files, so a data store added with connect.data_stores.add() must be synced with filespace.sync() before it can be removed. An unsynced store raises DataStoreNotFoundError although list() still shows it.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, the name now refers to a data store of the other kind, or the store has not been synced yet (a KeyError).
`DataStoreInUseError`	If the data store still has linked files.
`ConnectError`	If another client holds the data-store lock (retry).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, the name now refers to a data store of the other kind, or the store has not been synced yet (a KeyError).

DataStoreInUseError

Condition: If the data store still has linked files.

ConnectError

Condition: If another client holds the data-store lock (retry).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

dataclass

lucidlink.connect.DataStoreInfo

Information about a registered data store.

Attribute	Type	Description
`name`	`str`	Data store name.
`kind`	`DataStoreKind`	Data store type.

name

Type: str
Description: Data store name.

kind

Type: DataStoreKind
Description: Data store type.

enum

lucidlink.connect.DataStoreKind

Type of external data store.

Member	Value	Description
`S3`	`'s3'`	Amazon S3 or S3-compatible object storage.
`HTTP_LINK`	`'http_link'`	Groups files linked via direct HTTP/HTTPS URLs.

S3

Value: 's3'
Description: Amazon S3 or S3-compatible object storage.

HTTP_LINK

Value: 'http_link'
Description: Groups files linked via direct HTTP/HTTPS URLs.

enum

lucidlink.connect.DataStoreRekeyState

Credential rotation state for a data store.

Member	Value	Description
`NO_REKEY`	`'no_rekey'`	No credential rotation in progress.
`IN_PROGRESS`	`'in_progress'`	Credential rotation is currently in progress.

NO_REKEY

Value: 'no_rekey'
Description: No credential rotation in progress.

IN_PROGRESS

Value: 'in_progress'
Description: Credential rotation is currently in progress.

class

lucidlink.connect.DataStores

DataStores()

The data stores of a filespace: add, get, list and remove.

Supports get() and list() only; not a mapping. Obtained from connect.data_stores; not constructed directly.

method

add(name: `str`, config: `S3DataStoreConfig | HttpLinkDataStoreConfig`) -> `S3DataStore | HttpLinkDataStore`

Add a data store; the config type selects its kind.

Call filespace.sync() before DataStore.iter_files(), DataStore.count() or DataStore.remove() on the new data store: they fail until the data store is synced (see "Sync changes" in the Getting started guide).

The S3 credentials are not checked here: wrong credentials make S3DataStore.link() raise DataStoreCredentialsError.

Parameters

Parameter	Description
`name`	Unique name for the data store.
`config`	S3DataStoreConfig for an S3 data store, or HttpLinkDataStoreConfig() for an HTTP-link data store (filespace format 3.8 or newer).

name

Description: Unique name for the data store.

config

Description: S3DataStoreConfig for an S3 data store, or HttpLinkDataStoreConfig() for an HTTP-link data store (filespace format 3.8 or newer).

Returns

A handle to the new data store.

Raises

Exception	Condition
`InvalidArgumentError`	If the name or config is invalid (a ValueError).
`ConnectError`	If the filespace already has the maximum number of data stores, or another client holds the data-store lock (retry).
`TypeError`	If config is of any other type.
`UnsupportedOperationError`	If Connect is not available on this filespace, or an HTTP-link data store is added on filespace format 3.6.
`DataStoreExistsError`	If a data store with this name already exists.

InvalidArgumentError

Condition: If the name or config is invalid (a ValueError).

ConnectError

Condition: If the filespace already has the maximum number of data stores, or another client holds the data-store lock (retry).

TypeError

Condition: If config is of any other type.

UnsupportedOperationError

Condition: If Connect is not available on this filespace, or an HTTP-link data store is added on filespace format 3.6.

DataStoreExistsError

Condition: If a data store with this name already exists.

method

get(name: `str`, kind: `DataStoreKind | None` = None) -> `S3DataStore | HttpLinkDataStore`

Get a handle to an existing data store.

Parameters

Parameter	Description
`name`	Name of the data store.
`kind`	Expected kind. When given, the returned handle has the matching type (S3DataStore or HttpLinkDataStore).

name

Description: Name of the data store.

kind

Description: Expected kind. When given, the returned handle has the matching type (S3DataStore or HttpLinkDataStore).

Returns

A handle to the data store.

Raises

Exception	Condition
`DataStoreNotFoundError`	If no data store with this name exists (a KeyError; name is the name).
`InvalidArgumentError`	If kind is given and the data store is of the other kind, or kind is not a DataStoreKind value (a ValueError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If no data store with this name exists (a KeyError; name is the name).

InvalidArgumentError

Condition: If kind is given and the data store is of the other kind, or kind is not a DataStoreKind value (a ValueError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

list() -> `list[S3DataStore | HttpLinkDataStore]`

List handles to all data stores.

Returns

One handle per data store; call info() for its details.

Raises

Exception	Condition
`UnsupportedOperationError`	If Connect is not available on this filespace.

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

remove(name: `str`) -> `None`

Remove a data store by name.

Sync first: removing asks LucidLink services whether the store still has linked files, so a data store added with add() must be synced with filespace.sync() before it can be removed. An unsynced store raises DataStoreNotFoundError although list() still shows it.

Parameters

Parameter	Description
`name`	Name of the data store.

name

Description: Name of the data store.

Raises

Exception	Condition
`DataStoreNotFoundError`	If no data store with this name exists (a KeyError).
`DataStoreInUseError`	If the data store still has linked files.
`ConnectError`	If another client holds the data-store lock (retry).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If no data store with this name exists (a KeyError).

DataStoreInUseError

Condition: If the data store still has linked files.

ConnectError

Condition: If another client holds the data-store lock (retry).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

class

lucidlink.connect.ExternalFile

ExternalFile()

An external file linked from a data store.

The handle follows the file when it is renamed or moved; once the file is deleted, operations raise FsNotFoundError. get_path() returns the current path; delete the file with filespace.file_system.delete(file.get_path()).

Obtained from store.iter_files(), store.get_file() or store.link(); not constructed directly.

property

data_store: `DataStore`

The data store the file is linked from.

property

entry_id: `str`

Entry ID. Matches FsEntry.id; hard links to the same file have distinct entry IDs.

property

inode_id: `str`

Inode ID. Matches FsEntry.inode_id; shared by all hard links to the same file.

property

kind: `DataStoreKind`

Type of the data store the file is linked from.

method

get_path() -> `str`

Read the file's current path within the filespace.

Returns

The file's current path, reflecting any rename or move.

Raises

Exception	Condition
`FsNotFoundError`	If the file no longer exists.

FsNotFoundError

Condition: If the file no longer exists.

class

lucidlink.connect.HttpConnect_3_6

HttpConnect_3_6(filespace: Filespace)

Parameter	Description
`filespace`	A linked filespace of format 3.6.

filespace

Description: A linked filespace of format 3.6.

HTTP links on filespace format 3.6, which has no HTTP-link data stores.

Links HTTP URLs as read-only files without a data store, and changes their URLs. Delete a file with filespace.file_system.delete(path) to unlink it.

On format 3.8 or newer, add an HTTP-link data store with connect.data_stores.add(name, HttpLinkDataStoreConfig()) instead. Its link() takes the same arguments, and the handle from store.get_file(path) has an update_url() that takes the same arguments minus the path.

HttpConnect_3_6 example

from lucidlink.connect import HttpConnect_3_6
# link() does not create directories: create the parent firstfilespace.file_system.create_dir("/web")
http = HttpConnect_3_6(filespace)http.link("/web/page.html", "https://example.com/page.html")

method

link(path: `str`, url: `str`, *, size: `int | None` = None) -> `None`

Link an HTTP URL as a read-only file in the filespace.

Parameters

Parameter	Description
`path`	Path in the filespace for the new file.
`url`	HTTP URL of the file content.
`size`	File size in bytes. When given, skips the access check and size validation (useful for bulk linking where sizes are known in advance).

path

Description: Path in the filespace for the new file.

url

Description: HTTP URL of the file content.

size

Description: File size in bytes. When given, skips the access check and size validation (useful for bulk linking where sizes are known in advance).

Raises

Exception	Condition
`InvalidArgumentError`	If size is negative or larger than 2**63 - 1 (a ValueError).
`TypeError`	If size is not an integer.
`FsExistsError`	If path already exists (a FileExistsError).
`FsNotFoundError`	If the parent directory of path does not exist (a FileNotFoundError).
`ExternalStoreError`	If the URL is not an http:// or https:// URL, or is inaccessible or expired (by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError; an OSError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

InvalidArgumentError

Condition: If size is negative or larger than 2**63 - 1 (a ValueError).

TypeError

Condition: If size is not an integer.

FsExistsError

Condition: If path already exists (a FileExistsError).

FsNotFoundError

Condition: If the parent directory of path does not exist (a FileNotFoundError).

ExternalStoreError

Condition: If the URL is not an http:// or https:// URL, or is inaccessible or expired (by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError; an OSError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

update_url(path: `str`, url: `str`, *, size: `int | None` = None) -> `None`

Change the URL of an existing HTTP link file.

Parameters

Parameter	Description
`path`	Path of the HTTP link file in the filespace.
`url`	New HTTP URL for the file content.
`size`	File size in bytes; must equal the size the file was linked with. When given, the new URL is not checked for access, which speeds up bulk updates.

path

Description: Path of the HTTP link file in the filespace.

url

Description: New HTTP URL for the file content.

size

Description: File size in bytes; must equal the size the file was linked with. When given, the new URL is not checked for access, which speeds up bulk updates.

Raises

Exception	Condition
`InvalidArgumentError`	If size is negative or larger than 2**63 - 1, or the file at path is not an HTTP link file (a symbolic link too; a ValueError).
`TypeError`	If size is not an integer.
`FsNotFoundError`	If no file exists at path (a FileNotFoundError).
`ExternalStoreError`	If the new URL is not http(s), is inaccessible or expired, or its size (size when given, otherwise the URL's content size) differs from the size the file was linked with (by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError; an OSError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

InvalidArgumentError

Condition: If size is negative or larger than 2**63 - 1, or the file at path is not an HTTP link file (a symbolic link too; a ValueError).

TypeError

Condition: If size is not an integer.

FsNotFoundError

Condition: If no file exists at path (a FileNotFoundError).

ExternalStoreError

Condition: If the new URL is not http(s), is inaccessible or expired, or its size (size when given, otherwise the URL's content size) differs from the size the file was linked with (by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError; an OSError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

class

lucidlink.connect.HttpLinkDataStore

HttpLinkDataStore()

An HTTP-link data store: groups files linked from HTTP/HTTPS URLs.

Needs filespace format 3.8 or newer. See DataStore for the handle semantics.

method

get_file(path: `str`) -> `HttpLinkExternalFile`

Get a handle to the external file at path, linked from this data store.

See DataStore.get_file() for how the path is checked.

Parameters

Parameter	Description
`path`	Path of the external file in the filespace.

path

Description: Path of the external file in the filespace.

Returns

A handle to the file.

Raises

Exception	Condition
`FsNotFoundError`	If path does not exist (a FileNotFoundError).
`InvalidArgumentError`	If path is not an external file (a directory or symbolic link too), or the file is linked from another data store (the error names it; a ValueError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

FsNotFoundError

Condition: If path does not exist (a FileNotFoundError).

InvalidArgumentError

Condition: If path is not an external file (a directory or symbolic link too), or the file is linked from another data store (the error names it; a ValueError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

iter_files() -> `Iterator[HttpLinkExternalFile]`

Iterate over the files linked from this data store.

See DataStore.iter_files() for when linked files appear in the listing and how it is paged. To change many URLs, update each file through its handle

for f in http_store.iter_files():    f.update_url(new_url_for(f.get_path()))

Returns

An iterator over handles to the linked files.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, the name now refers to an S3 data store, or the data store has not been synced yet (a KeyError).
`InternalError`	If a listed file was deleted here and the delete is not synced yet.
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, the name now refers to an S3 data store, or the data store has not been synced yet (a KeyError).

InternalError

Condition: If a listed file was deleted here and the delete is not synced yet.

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

link(path: `str`, url: `str`, *, size: `int | None` = None) -> `HttpLinkExternalFile`

Link an HTTP URL as a file in the filespace.

Change the URL later with update_url() on the returned handle; delete the file with filespace.file_system.delete(path) to unlink it.

Parameters

Parameter	Description
`path`	Path in the filespace for the new file.
`url`	HTTP URL of the file content.
`size`	File size in bytes. When given, skips the access check and size validation (useful for bulk linking where sizes are known in advance).

path

Description: Path in the filespace for the new file.

url

Description: HTTP URL of the file content.

size

Description: File size in bytes. When given, skips the access check and size validation (useful for bulk linking where sizes are known in advance).

Returns

A handle to the new file.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, the name now refers to an S3 data store, or the data store was added by another client and is not visible here yet (a KeyError).
`InvalidArgumentError`	If size is negative or larger than 2**63 - 1 (a ValueError).
`TypeError`	If size is not an integer.
`FsExistsError`	If path already exists (a FileExistsError).
`ExternalStoreError`	If the URL is inaccessible or expired (an OSError; by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, the name now refers to an S3 data store, or the data store was added by another client and is not visible here yet (a KeyError).

InvalidArgumentError

Condition: If size is negative or larger than 2**63 - 1 (a ValueError).

TypeError

Condition: If size is not an integer.

FsExistsError

Condition: If path already exists (a FileExistsError).

ExternalStoreError

Condition: If the URL is inaccessible or expired (an OSError; by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

dataclass

lucidlink.connect.HttpLinkDataStoreConfig

Selects the HTTP-link overload of data_stores.add().

HTTP-link data stores need filespace format 3.8 or newer.

class

lucidlink.connect.HttpLinkExternalFile

HttpLinkExternalFile()

An external file linked from an HTTP-link data store.

See ExternalFile for the handle semantics.

property

data_store: `HttpLinkDataStore`

The HTTP-link data store the file is linked from.

method

update_url(url: `str`, *, size: `int | None` = None) -> `None`

Change the URL of this HTTP link file.

Every hard link to the file reads from the new URL.

Parameters

Parameter	Description
`url`	New HTTP URL for the file content.
`size`	File size in bytes; must equal the size the file was linked with. When given, the new URL is not checked for access, which speeds up bulk updates.

url

Description: New HTTP URL for the file content.

size

Description: File size in bytes; must equal the size the file was linked with. When given, the new URL is not checked for access, which speeds up bulk updates.

Raises

Exception	Condition
`InvalidArgumentError`	If size is negative or larger than 2**63 - 1, or the file is not an HTTP link file (a ValueError).
`TypeError`	If size is not an integer.
`FsNotFoundError`	If the file no longer exists (a FileNotFoundError).
`ExternalStoreError`	If the URL is not an http:// or https:// URL, is inaccessible or expired (by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError), or its size (size when given, otherwise the URL's content size) differs from the size the file was linked with (an OSError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

InvalidArgumentError

Condition: If size is negative or larger than 2**63 - 1, or the file is not an HTTP link file (a ValueError).

TypeError

Condition: If size is not an integer.

FsNotFoundError

Condition: If the file no longer exists (a FileNotFoundError).

ExternalStoreError

Condition: If the URL is not an http:// or https:// URL, is inaccessible or expired (by the server's status the subclass ExternalObjectNotFoundError or ExternalStoreUnavailableError), or its size (size when given, otherwise the URL's content size) differs from the size the file was linked with (an OSError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

dataclass

lucidlink.connect.S3Credentials

Credentials for S3DataStore.rekey().

Attribute	Type	Description
`access_key`	`str`	—
`secret_key`	`str`	—

access_key

Type: str
Description: —

secret_key

Type: str
Description: —

class

lucidlink.connect.S3DataStore

S3DataStore()

An S3 data store: links S3 objects as files.

See DataStore for the handle semantics.

method

get_file(path: `str`) -> `S3ExternalFile`

Get a handle to the external file at path, linked from this data store.

See DataStore.get_file() for how the path is checked.

Parameters

Parameter	Description
`path`	Path of the external file in the filespace.

path

Description: Path of the external file in the filespace.

Returns

A handle to the file.

Raises

Exception	Condition
`FsNotFoundError`	If path does not exist (a FileNotFoundError).
`InvalidArgumentError`	If path is not an external file (a directory or symbolic link too), or the file is linked from another data store (the error names it; a ValueError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

FsNotFoundError

Condition: If path does not exist (a FileNotFoundError).

InvalidArgumentError

Condition: If path is not an external file (a directory or symbolic link too), or the file is linked from another data store (the error names it; a ValueError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

info() -> `S3DataStoreInfo`

Read the data store's current details.

Returns

The data store's details.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, or the name now refers to an HTTP-link data store (a KeyError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, or the name now refers to an HTTP-link data store (a KeyError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

iter_files() -> `Iterator[S3ExternalFile]`

Iterate over the files linked from this data store.

See DataStore.iter_files() for when linked files appear in the listing and how it is paged.

While a credential rotation started by rekey() is in progress (info().rekey_state is IN_PROGRESS), results, in particular paginated ones, may be incomplete or unstable. List after the rotation completes.

Returns

An iterator over handles to the linked files.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, the name now refers to an HTTP-link data store, or the data store has not been synced yet (a KeyError).
`InternalError`	If a listed file was deleted here and the delete is not synced yet.
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, the name now refers to an HTTP-link data store, or the data store has not been synced yet (a KeyError).

InternalError

Condition: If a listed file was deleted here and the delete is not synced yet.

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

link(path: `str`, object_id: `str`, *, size: `int | None` = None, checksum: `str | None` = None) -> `S3ExternalFile`

Link an S3 object as a file in the filespace.

The file reads its content from the S3 object; on filespace format 3.6 it is read-only. On 3.8 or newer, writes stay in the filespace and never modify the S3 object. The parent directory of path must already exist. Delete the file with filespace.file_system.delete(path) to unlink it.

Parameters

Parameter	Description
`path`	Path in the filespace for the new file.
`object_id`	S3 object key.
`size`	Object size in bytes. Given together with checksum, the object is not looked up in S3, which speeds up bulk linking; both are stored as given, so link() succeeds even for a missing object.
`checksum`	The object's ETag without the surrounding double quotes. Must be given together with size. A wrong value makes every read of the file fail.

path

Description: Path in the filespace for the new file.

object_id

Description: S3 object key.

size

Description: Object size in bytes. Given together with checksum, the object is not looked up in S3, which speeds up bulk linking; both are stored as given, so link() succeeds even for a missing object.

checksum

Description: The object's ETag without the surrounding double quotes. Must be given together with size. A wrong value makes every read of the file fail.

Returns

A handle to the new file.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, the name now refers to an HTTP-link data store, or the data store was added by another client and is not visible here yet (a KeyError).
`FsNotFoundError`	If the parent directory of path does not exist (a FileNotFoundError).
`InvalidArgumentError`	If only one of size / checksum is given, or size is negative or larger than 2**63 - 1 (a ValueError).
`TypeError`	If size is not an integer.
`FsExistsError`	If path already exists (a FileExistsError).
`FsNotFoundError`	If the parent directory of path does not exist (a FileNotFoundError).
`FsNotADirectoryError`	If a component of path is a file (a NotADirectoryError).
`FsPermissionError`	If the service account may not write there (a PermissionError).
`FsNoSpaceError`	If the external-file count or size limit, or a storage limit, is reached.
`ExternalObjectNotFoundError`	If the S3 object does not exist (HTTP 404; an OSError with EIO, never a FileNotFoundError).
`DataStoreCredentialsError`	If the data store's credentials may not read the S3 object (HTTP 401 / 403).
`ExternalStoreUnavailableError`	If S3 throttles, fails or is unreachable (HTTP 429 / 5xx; a ConnectionError; retry).
`ExternalStoreError`	If the S3 object cannot be reached for another reason, or its checksum cannot be determined (an OSError).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, the name now refers to an HTTP-link data store, or the data store was added by another client and is not visible here yet (a KeyError).

FsNotFoundError

Condition: If the parent directory of path does not exist (a FileNotFoundError).

InvalidArgumentError

Condition: If only one of size / checksum is given, or size is negative or larger than 2**63 - 1 (a ValueError).

TypeError

Condition: If size is not an integer.

FsExistsError

Condition: If path already exists (a FileExistsError).

FsNotFoundError

Condition: If the parent directory of path does not exist (a FileNotFoundError).

FsNotADirectoryError

Condition: If a component of path is a file (a NotADirectoryError).

FsPermissionError

Condition: If the service account may not write there (a PermissionError).

FsNoSpaceError

Condition: If the external-file count or size limit, or a storage limit, is reached.

ExternalObjectNotFoundError

Condition: If the S3 object does not exist (HTTP 404; an OSError with EIO, never a FileNotFoundError).

DataStoreCredentialsError

Condition: If the data store's credentials may not read the S3 object (HTTP 401 / 403).

ExternalStoreUnavailableError

Condition: If S3 throttles, fails or is unreachable (HTTP 429 / 5xx; a ConnectionError; retry).

ExternalStoreError

Condition: If the S3 object cannot be reached for another reason, or its checksum cannot be determined (an OSError).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

method

rekey(credentials: `S3Credentials`) -> `None`

Rotate the data store's S3 credentials.

The rotation is asynchronous: this call starts it, and LucidLink services then move the store's linked files to the new credentials in the background. Poll info().rekey_state: it reports IN_PROGRESS until they are done. Keep the old credentials valid until then: files not moved yet are still read with them. Meanwhile, iter_files() and count() results may be incomplete or unstable.

Sync (filespace.sync()) before rekeying: the new credentials are checked only against a linked file that LucidLink services already have, so on an unsynced data store they are not checked.

Parameters

Parameter	Description
`credentials`	The new S3 credentials.

credentials

Description: The new S3 credentials.

Raises

Exception	Condition
`DataStoreNotFoundError`	If the data store no longer exists, or the name now refers to an HTTP-link data store (a KeyError).
`InvalidArgumentError`	If the access key or secret key is empty (a ValueError).
`DataStoreCredentialsError`	If the store rejects the new S3 credentials (checked against a linked file's object).
`ConnectError`	If another client holds the data-store lock (retry).
`UnsupportedOperationError`	If Connect is not available on this filespace.

DataStoreNotFoundError

Condition: If the data store no longer exists, or the name now refers to an HTTP-link data store (a KeyError).

InvalidArgumentError

Condition: If the access key or secret key is empty (a ValueError).

DataStoreCredentialsError

Condition: If the store rejects the new S3 credentials (checked against a linked file's object).

ConnectError

Condition: If another client holds the data-store lock (retry).

UnsupportedOperationError

Condition: If Connect is not available on this filespace.

dataclass

lucidlink.connect.S3DataStoreConfig

Configuration for an S3-compatible data store.

Attribute	Type	Description
`access_key`	`str`	S3 access key ID.
`secret_key`	`str`	S3 secret access key.
`bucket_name`	`str`	S3 bucket name.
`region`	`str`	S3 provider region (e.g., "us-east-1"). Must be non-empty when endpoint is empty (AWS S3); with an endpoint, pass "" if the provider has no region.
`endpoint`	`str`	S3-compatible endpoint URL with http:// or https://, e.g. "http://127.0.0.1:9090". Empty uses the AWS S3 endpoint for region, over HTTPS.
`url_expiration_minutes`	`int`	Presigned URL expiration in minutes: 4320 (3 days) to 10080 (7 days); defaults to 10080.
`use_virtual_addressing`	`bool`	Use virtual-hosted-style addressing.

access_key

Type: str
Description: S3 access key ID.

secret_key

Type: str
Description: S3 secret access key.

bucket_name

Type: str
Description: S3 bucket name.

region

Type: str
Description: S3 provider region (e.g., "us-east-1"). Must be non-empty when endpoint is empty (AWS S3); with an endpoint, pass "" if the provider has no region.

endpoint

Type: str
Description: S3-compatible endpoint URL with http:// or https://, e.g. "http://127.0.0.1:9090". Empty uses the AWS S3 endpoint for region, over HTTPS.

url_expiration_minutes

Type: int
Description: Presigned URL expiration in minutes: 4320 (3 days) to 10080 (7 days); defaults to 10080.

use_virtual_addressing

Type: bool
Description: Use virtual-hosted-style addressing.

dataclass

lucidlink.connect.S3DataStoreInfo

Information about a registered S3 data store.

Attribute	Type	Description
`bucket_name`	`str`	S3 bucket name.
`region`	`str`	S3 provider region; empty if none was given.
`endpoint`	`str`	Full S3 endpoint URL (e.g., "http://127.0.0.1:9090"), empty if using AWS default.
`access_key`	`str`	S3 access key ID.
`url_expiration_minutes`	`int`	Presigned URL expiration in minutes.
`use_virtual_addressing`	`bool`	Whether virtual-hosted-style addressing is used.
`rekey_state`	`DataStoreRekeyState`	Current credential rotation state.

bucket_name

Type: str
Description: S3 bucket name.

region

Type: str
Description: S3 provider region; empty if none was given.

endpoint

Type: str
Description: Full S3 endpoint URL (e.g., "http://127.0.0.1:9090"), empty if using AWS default.

access_key

Type: str
Description: S3 access key ID.

url_expiration_minutes

Type: int
Description: Presigned URL expiration in minutes.

use_virtual_addressing

Type: bool
Description: Whether virtual-hosted-style addressing is used.

rekey_state

Type: DataStoreRekeyState
Description: Current credential rotation state.

class

lucidlink.connect.S3ExternalFile

S3ExternalFile()

An external file linked from an S3 data store.

See ExternalFile for the handle semantics.

property

data_store: `S3DataStore`

The S3 data store the file is linked from.

connect

lucidlink.connect.ConnectManager

data_stores: DataStores

lucidlink.connect.DataStore

kind: DataStoreKind

name: str

count() -> int

Returns

Raises

get_file(path: str) -> ExternalFile

Parameters

Returns

Raises

info() -> DataStoreInfo

Returns

Raises

iter_files() -> Iterator[ExternalFile]

Returns

Raises

remove() -> None

Raises

lucidlink.connect.DataStoreInfo

lucidlink.connect.DataStoreKind

lucidlink.connect.DataStoreRekeyState

lucidlink.connect.DataStores

add(name: str, config: S3DataStoreConfig | HttpLinkDataStoreConfig) -> S3DataStore | HttpLinkDataStore

Parameters

Returns

Raises

get(name: str, kind: DataStoreKind | None = None) -> S3DataStore | HttpLinkDataStore

Parameters

Returns

Raises

list() -> list[S3DataStore | HttpLinkDataStore]

Returns

Raises

remove(name: str) -> None

Parameters

Raises

lucidlink.connect.ExternalFile

data_store: DataStore

entry_id: str

inode_id: str

kind: DataStoreKind

get_path() -> str

Returns

Raises

lucidlink.connect.HttpConnect_3_6

link(path: str, url: str, *, size: int | None = None) -> None

Parameters

Raises

update_url(path: str, url: str, *, size: int | None = None) -> None

Parameters

Raises

lucidlink.connect.HttpLinkDataStore

get_file(path: str) -> HttpLinkExternalFile

Parameters

Returns

Raises

iter_files() -> Iterator[HttpLinkExternalFile]

Returns

Raises

link(path: str, url: str, *, size: int | None = None) -> HttpLinkExternalFile

Parameters

Returns

Raises

lucidlink.connect.HttpLinkDataStoreConfig

lucidlink.connect.HttpLinkExternalFile

data_store: HttpLinkDataStore

update_url(url: str, *, size: int | None = None) -> None

Parameters

Raises

lucidlink.connect.S3Credentials

lucidlink.connect.S3DataStore

get_file(path: str) -> S3ExternalFile

Parameters

Returns

Raises

info() -> S3DataStoreInfo

Returns

data_stores: `DataStores`

kind: `DataStoreKind`

name: `str`

count() -> `int`

get_file(path: `str`) -> `ExternalFile`

info() -> `DataStoreInfo`

iter_files() -> `Iterator[ExternalFile]`

remove() -> `None`

add(name: `str`, config: `S3DataStoreConfig | HttpLinkDataStoreConfig`) -> `S3DataStore | HttpLinkDataStore`

get(name: `str`, kind: `DataStoreKind | None` = None) -> `S3DataStore | HttpLinkDataStore`

list() -> `list[S3DataStore | HttpLinkDataStore]`

remove(name: `str`) -> `None`

data_store: `DataStore`

entry_id: `str`

inode_id: `str`

kind: `DataStoreKind`

get_path() -> `str`

link(path: `str`, url: `str`, *, size: `int | None` = None) -> `None`

update_url(path: `str`, url: `str`, *, size: `int | None` = None) -> `None`

get_file(path: `str`) -> `HttpLinkExternalFile`

iter_files() -> `Iterator[HttpLinkExternalFile]`

link(path: `str`, url: `str`, *, size: `int | None` = None) -> `HttpLinkExternalFile`

data_store: `HttpLinkDataStore`

update_url(url: `str`, *, size: `int | None` = None) -> `None`

get_file(path: `str`) -> `S3ExternalFile`

info() -> `S3DataStoreInfo`

iter_files() -> `Iterator[S3ExternalFile]`

link(path: `str`, object_id: `str`, *, size: `int | None` = None, checksum: `str | None` = None) -> `S3ExternalFile`

rekey(credentials: `S3Credentials`) -> `None`

data_store: `S3DataStore`