fsspec

fsspec integration for LucidLink filesystem.

Provides fsspec.AbstractFileSystem implementation that allows standard Python data libraries (Pandas, Dask, PyArrow, etc.) to access LucidLink filespaces using URL protocols like lucidlink://workspace/filespace/path/to/file.csv

Features

Standard file operations: open, read, write, ls, mkdir, rm
Native move/rename operations (much faster than copy+delete fallback)
Directory operations: mkdir, makedirs, rmdir
Sync control: automatic or manual sync to LucidLink services

class

lucidlink.fsspec.LucidLinkFileSystem

LucidLinkFileSystem(token: Optional[str] = LucidLinkOptions.DEFAULT_TOKEN, sandboxed: bool = LucidLinkOptions.DEFAULT_SANDBOXED, persist_files: bool = LucidLinkOptions.DEFAULT_PERSIST_FILES, root_path: Optional[Union[str, Path]] = LucidLinkOptions.DEFAULT_ROOT_PATH, sync_mode: SyncMode = LucidLinkOptions.DEFAULT_SYNC_MODE, **kwargs)

Parameter	Description
`token`	Service account token (sa_live:...)
`sandboxed`	Use temporary directory (True) or persistent .lucid folder (False)
`persist_files`	Keep files after the filesystem closes (only for physical mode)
`root_path`	Custom root path for storage (optional)
`sync_mode`	Sync mode - SYNC_ALL (default) syncs on close, SYNC_NONE skips sync
`**kwargs`	Additional fsspec parameters

token

Description: Service account token (sa_live:...)

sandboxed

Description: Use temporary directory (True) or persistent .lucid folder (False)

persist_files

Description: Keep files after the filesystem closes (only for physical mode)

root_path

Description: Custom root path for storage (optional)

sync_mode

Description: Sync mode - SYNC_ALL (default) syncs on close, SYNC_NONE skips sync

**kwargs

Description: Additional fsspec parameters

fsspec filesystem implementation for LucidLink.

Enables standard Python data libraries to access LucidLink filespaces using URL protocols like: lucidlink://workspace/filespace/path/to/file.csv

Usage with Pandas

df = pd.read_csv('lucidlink://my-workspace/production-data/data.csv',                 storage_options={'token': 'sa_live:...'})

Usage with Dask

df = dd.read_parquet('lucidlink://workspace/filespace/dataset/*.parquet',                     storage_options={'token': 'sa_live:...'})

Usage with PyArrow

table = pq.read_table('lucidlink://workspace/filespace/data.parquet',                      filesystem=LucidLinkFileSystem(token='sa_live:...'))

Direct usage

fs = LucidLinkFileSystem(token='sa_live:...')with fs.open('lucidlink://workspace/filespace/file.txt', 'r') as f:    content = f.read()

property

options: `LucidLinkOptions`

Read-only access to filesystem configuration options.

method

cat(path: `str`, start: `Optional[int]` = None, end: `Optional[int]` = None, **kwargs) -> `bytes`

Read entire file or byte range.

Parameters

Parameter	Description
`path`	LucidLink URL
`start`	Start byte position (optional)
`end`	End byte position (optional)
`**kwargs`	Additional parameters

path

Description: LucidLink URL

start

Description: Start byte position (optional)

end

Description: End byte position (optional)

**kwargs

Description: Additional parameters

Returns

File contents as bytes

method

cat_file(path: `str`, start: `Optional[int]` = None, end: `Optional[int]` = None, **kwargs) -> `bytes`

Alias for cat() for compatibility.

method

close() -> `None`

Close all connections and release resources.

method

exists(path: `str`, **kwargs) -> `bool`

Check if path exists.

Raises

Exception	Condition
`PermissionError`	If token doesn't have access to the workspace or filespace
`FilespaceError`	If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as "missing"

PermissionError

Condition: If token doesn't have access to the workspace or filespace

FilespaceError

Condition: If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as "missing"

method

get(rpath: `str`, lpath: `str`, recursive: `bool` = False, **kwargs) -> `None`

Download file from LucidLink to local filesystem.

Parameters

Parameter	Description
`rpath`	Remote LucidLink URL
`lpath`	Local filesystem path
`recursive`	Download directory recursively
`**kwargs`	Additional parameters

rpath

Description: Remote LucidLink URL

lpath

Description: Local filesystem path

recursive

Description: Download directory recursively

**kwargs

Description: Additional parameters

method

info(path: `str`, **kwargs) -> `Dict[str, Any]`

Get file or directory information.

Parameters

Parameter	Description
`path`	LucidLink URL
`**kwargs`	Additional parameters

path

Description: LucidLink URL

**kwargs

Description: Additional parameters

Returns

Dictionary with file/directory metadata

method

isdir(path: `str`) -> `bool`

Check if path is a directory.

Raises

Exception	Condition
`PermissionError`	If token doesn't have access to the workspace or filespace
`FilespaceError`	If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as "missing"

PermissionError

Condition: If token doesn't have access to the workspace or filespace

FilespaceError

Condition: If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as "missing"

method

isfile(path: `str`) -> `bool`

Check if path is a file.

Raises

Exception	Condition
`PermissionError`	If token doesn't have access to the workspace or filespace
`FilespaceError`	If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as "missing"

PermissionError

Condition: If token doesn't have access to the workspace or filespace

FilespaceError

Condition: If the filespace link itself failed (e.g. a filespace internal error) — a failed link must not read as "missing"

method

ls(path: `str`, detail: `bool` = True, **kwargs) -> `Union[List[str], List[Dict[str, Any]]]`

List directory contents.

Parameters

Parameter	Description
`path`	LucidLink URL
`detail`	Return detailed info (True) or just names (False)
`**kwargs`	Additional parameters

path

Description: LucidLink URL

detail

Description: Return detailed info (True) or just names (False)

**kwargs

Description: Additional parameters

Returns

List of full paths (detail=False) or detailed info dicts (detail=True)

method

makedirs(path: `str`, exist_ok: `bool` = False) -> `None`

Create directory recursively.

Parameters

Parameter	Description
`path`	LucidLink URL
`exist_ok`	Don't raise error if directory exists

path

Description: LucidLink URL

exist_ok

Description: Don't raise error if directory exists

method

mkdir(path: `str`, create_parents: `bool` = True, **kwargs) -> `None`

Create a directory.

Parameters

Parameter	Description
`path`	LucidLink URL
`create_parents`	Create parent directories if needed
`**kwargs`	Additional parameters

path

Description: LucidLink URL

create_parents

Description: Create parent directories if needed

**kwargs

Description: Additional parameters

method

mv(path1: `str`, path2: `str`, recursive: `bool` = False, maxdepth: `Optional[int]` = None, **kwargs) -> `None`

Move/rename a file or directory.

Uses native rename operation which is much faster than copy+delete.

Parameters

Parameter	Description
`path1`	Source LucidLink URL
`path2`	Destination LucidLink URL
`recursive`	Ignored (move is always recursive for directories)
`maxdepth`	Ignored
`**kwargs`	Additional parameters (ignored)

path1

Description: Source LucidLink URL

path2

Description: Destination LucidLink URL

recursive

Description: Ignored (move is always recursive for directories)

maxdepth

Description: Ignored

**kwargs

Description: Additional parameters (ignored)

Raises

Exception	Condition
`ValueError`	If paths are in different filespaces

ValueError

Condition: If paths are in different filespaces

method

put(lpath: `str`, rpath: `str`, recursive: `bool` = False, **kwargs) -> `None`

Upload file from local filesystem to LucidLink.

Parameters

Parameter	Description
`lpath`	Local filesystem path
`rpath`	Remote LucidLink URL
`recursive`	Upload directory recursively
`**kwargs`	Additional parameters

lpath

Description: Local filesystem path

rpath

Description: Remote LucidLink URL

recursive

Description: Upload directory recursively

**kwargs

Description: Additional parameters

method

rename(path1: `str`, path2: `str`, **kwargs) -> `None`

Rename a file or directory (alias for mv).

Parameters

Parameter	Description
`path1`	Source LucidLink URL
`path2`	Destination LucidLink URL
`**kwargs`	Additional parameters (passed to mv)

path1

Description: Source LucidLink URL

path2

Description: Destination LucidLink URL

**kwargs

Description: Additional parameters (passed to mv)

method

rm(path: `str`, recursive: `bool` = False, maxdepth: `Optional[int]` = None) -> `None`

Remove file or directory.

Parameters

Parameter	Description
`path`	LucidLink URL
`recursive`	Remove directory recursively
`maxdepth`	Maximum recursion depth (ignored, always full recursion)

path

Description: LucidLink URL

recursive

Description: Remove directory recursively

maxdepth

Description: Maximum recursion depth (ignored, always full recursion)

method

rmdir(path: `str`) -> `None`

Remove an empty directory.

Parameters

Parameter	Description
`path`	LucidLink URL

path

Description: LucidLink URL

method

sync_all() -> `None`

Synchronize all pending changes to LucidLink services.

Calls sync_all() on all connected filespaces to ensure all metadata and data changes are propagated to LucidLink services.

Raises

Exception	Condition
`FilespaceInternalError`	If a filespace hit an internal error — its pending changes cannot be synced (they are lost), which callers must not mistake for a successful sync.

FilespaceInternalError

Condition: If a filespace hit an internal error — its pending changes cannot be synced (they are lost), which callers must not mistake for a successful sync.

fsspec

Features

lucidlink.fsspec.LucidLinkFileSystem

options: LucidLinkOptions

cat(path: str, start: Optional[int] = None, end: Optional[int] = None, **kwargs) -> bytes

Parameters

Returns

cat_file(path: str, start: Optional[int] = None, end: Optional[int] = None, **kwargs) -> bytes

close() -> None

exists(path: str, **kwargs) -> bool

Raises

get(rpath: str, lpath: str, recursive: bool = False, **kwargs) -> None

Parameters

info(path: str, **kwargs) -> Dict[str, Any]

Parameters

Returns

isdir(path: str) -> bool

Raises

isfile(path: str) -> bool

Raises

ls(path: str, detail: bool = True, **kwargs) -> Union[List[str], List[Dict[str, Any]]]

Parameters

Returns

makedirs(path: str, exist_ok: bool = False) -> None

Parameters

mkdir(path: str, create_parents: bool = True, **kwargs) -> None

Parameters

mv(path1: str, path2: str, recursive: bool = False, maxdepth: Optional[int] = None, **kwargs) -> None

Parameters

Raises

put(lpath: str, rpath: str, recursive: bool = False, **kwargs) -> None

Parameters

rename(path1: str, path2: str, **kwargs) -> None

Parameters

rm(path: str, recursive: bool = False, maxdepth: Optional[int] = None) -> None

Parameters

rmdir(path: str) -> None

Parameters

sync_all() -> None

Raises

options: `LucidLinkOptions`

cat(path: `str`, start: `Optional[int]` = None, end: `Optional[int]` = None, **kwargs) -> `bytes`

cat_file(path: `str`, start: `Optional[int]` = None, end: `Optional[int]` = None, **kwargs) -> `bytes`

close() -> `None`

exists(path: `str`, **kwargs) -> `bool`

get(rpath: `str`, lpath: `str`, recursive: `bool` = False, **kwargs) -> `None`

info(path: `str`, **kwargs) -> `Dict[str, Any]`

isdir(path: `str`) -> `bool`

isfile(path: `str`) -> `bool`

ls(path: `str`, detail: `bool` = True, **kwargs) -> `Union[List[str], List[Dict[str, Any]]]`

makedirs(path: `str`, exist_ok: `bool` = False) -> `None`

mkdir(path: `str`, create_parents: `bool` = True, **kwargs) -> `None`

mv(path1: `str`, path2: `str`, recursive: `bool` = False, maxdepth: `Optional[int]` = None, **kwargs) -> `None`

put(lpath: `str`, rpath: `str`, recursive: `bool` = False, **kwargs) -> `None`

rename(path1: `str`, path2: `str`, **kwargs) -> `None`

rm(path: `str`, recursive: `bool` = False, maxdepth: `Optional[int]` = None) -> `None`

rmdir(path: `str`) -> `None`

sync_all() -> `None`