Filesystem operations

Supported operations

Work with files and directories

filespace.file_system provides the file system operations. FsUtils adds one-call helpers on top, such as reading or writing a whole file and deleting a directory tree.

from lucidlink import FsUtils
fs_utils = FsUtils(filespace.file_system)
fs_utils.write_bytes("/hello.txt", b"Hello, world!")
content = fs_utils.read_bytes("/hello.txt")print(content)  # b"Hello, world!"

# create_dir() creates one level: create the parents firstfilespace.file_system.create_dir("/projects")filespace.file_system.create_dir("/projects/2024")
# Names onlynames = [e.name for e in filespace.file_system.read_dir("/projects")]print(names)  # ["2024"]
# Full metadatafor entry in filespace.file_system.read_dir("/projects"):    print(f"{entry.name}: {entry.size} bytes")

from lucidlink import FsUtils
filespace.file_system.delete("/hello.txt")filespace.file_system.delete_dir("/projects/2024")  # Directory must be emptyFsUtils(filespace.file_system).delete_tree("/projects")
filespace.file_system.move("/old_name.txt", "/new_name.txt")

from lucidlink import FsUtils
# A missing path raises FileNotFoundErrortry:    data = FsUtils(filespace.file_system).read_bytes("/config.json")except FileNotFoundError:    data = None
try:    entries = filespace.file_system.read_dir("/backups")except FileNotFoundError:    entries = []
info = filespace.file_system.get_entry("/report.pdf")print(f"Size: {info.size}, Type: {info.type}")print(f"Modified (ns since epoch): {info.modified_at}")
# Counters for the whole file systemstats = filespace.file_system.statistics()print(f"Entries: {stats.entries_count}, Storage: {stats.storage_size} bytes")

Work with symbolic links

A symbolic link among the directories of a path is followed: with /latest pointing to /data/2024, /latest/jan.csv is /data/2024/jan.csv. The last component of a path is never followed, so a call that names a link acts on the link itself or refuses it:

• get_entry() describes the link itself, with type EntryType.SYMLINK, and read_dir() lists links the same way; so do fsspec's ls() and info() (type "link").

• read_symlink() returns the target a link stores. create_symlink() stores the target exactly as given; the target is never looked up and doesn't need to exist.

• delete() removes the link itself, and move() renames it; neither touches what the link points to. A link at the destination of move() is an existing entry (FileExistsError). fsspec's rm() and mv() behave the same way.

• open() in every mode and truncate() on a link raise FsError (an OSError) with errno.ELOOP, and so do FsUtils.read_bytes() and FsUtils.write_bytes(). read_dir() and FsUtils.delete_tree() on a link raise NotADirectoryError.

• A link is not a connected file: LucidLink Connect's get_file() raises ValueError for one, and link() onto one raises FileExistsError.

A link among the directories of a path that points nowhere raises FileNotFoundError, and a loop of links raises FsError with errno.ELOOP.

A relative link target is resolved from the root of the filespace, not from the link's own directory, and .. and . in it are not applied: with /projects/latest pointing to 2024, /projects/latest/jan.csv is /2024/jan.csv. Use absolute link targets. A link with a relative target, created by another client for example, might not resolve, or might resolve to a different directory of the same name at the root.

fs = filespace.file_systemfs.create_symlink("/data/2024", "/latest")print(fs.read_symlink("/latest"))  # "/data/2024"print(fs.get_entry("/latest").is_symlink)  # True: the link itself
with fs.open("/latest/jan.csv", "rb") as f:  # Reads /data/2024/jan.csv    data = f.read()
fs.move("/latest", "/current")  # Renames the link; /data/2024 is untouchedfs.delete("/current")  # Removes the link only

Stream file contents

file_system.open() returns a stream compatible with io.RawIOBase. It supports buffered and text modes, seeking, and chunked reads and writes.

# Binary streaming (read)with filespace.file_system.open("/large_file.dat", "rb", buffering=8192) as f:    for chunk in iter(lambda: f.read(4096), b""):        process(chunk)
# Text streaming with an encoding (read)with filespace.file_system.open("/document.txt", "rt", encoding="utf-8") as f:    for line in f:        print(line.strip())
# Byte-range read: 100 bytes from offset 1000with filespace.file_system.open("/data.bin", "rb") as f:    f.seek(1000)    data = f.read(100)
# Streaming writesdata = b"x" * 1024 * 1024with filespace.file_system.open("/output.dat", "wb") as f:    for i in range(10):        f.write(data)

Use fsspec and pandas

The Python SDK registers the lucidlink:// protocol with fsspec. A path has the form lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/path, and libraries that work with fsspec, such as pandas, read and write filespace files with it. PyArrow does not open these URLs: give it a LucidLinkFileSystem as filesystem. fsspec support is an optional extra:

pip install "lucidlink[fsspec]"

from lucidlink.fsspec import LucidLinkFileSystem
fs = LucidLinkFileSystem(token="<SECRET_KEY>")
entries = fs.ls("lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/", detail=True)
# Download and uploadfs.get(    "lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/file.txt",    "local_file.txt",)fs.put(    "local_file.txt",    "lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/uploaded.txt",)
# A native move, not a copy followed by a delete. An existing file at the# destination is replaced; an existing directory receives the source.fs.mv(    "lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/old.txt",    "lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/new.txt",)
# mv takes one path, never a glob or a list: move each match into an# existing directory. A trailing slash does not make mv move into a# directory, so create it first; otherwise every match overwrites the last.root = "lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>"fs.makedirs(f"{root}/archive", exist_ok=True)for path in fs.glob(f"{root}/staging/*.csv"):    fs.mv(path, f"{root}/archive/")
# Modification and creation times as timezone-aware UTC datetimesfs.modified("lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/new.txt")fs.created("lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/new.txt")
fs.mkdir("lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/new_dir")fs.rmdir("lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/empty_dir")
fs.close()

By default, writing into a missing directory through a LucidLinkFileSystem method, such as fs.open(path, "wb"), raises FileNotFoundError. Pass auto_mkdir=True, or storage_options={"auto_mkdir": True}, to create missing parent directories on write. The SDK creates only missing directories inside the filespace, and only on a write: reading a file from a missing directory still raises FileNotFoundError.

Each LucidLinkFileSystem instance is a separate client, and fsspec creates one instance per thread, per process, and per set of storage options, including auto_mkdir. Clients see each other's changes only after a delay, so read with the same storage options you write with. After fs.close(), every call on that instance raises ClientClosedError; create a new instance for further work.

Worker processes started with the fork start method cannot use LucidLink while a client runs in the parent: every call raises ClientError. Use the spawn or forkserver start method instead.

import pandas as pd
df = pd.read_csv(    "lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/data.csv",    storage_options={"token": "<SECRET_KEY>"},)
df.to_parquet(    "lucidlink://<WORKSPACE_NAME>/<FILESPACE_NAME>/output.parquet",    storage_options={"token": "<SECRET_KEY>"},)