Getting started

Prerequisites

1. Python 3.11 or later, on your machine or in your application environment.

2. A LucidLink account on app.lucidlink.com in a workspace that has service accounts.

3. A service account with an active secret key. A service account is a non-human identity that belongs to one workspace; its secret key authenticates every SDK request. Create one on your workspace's Service accounts page.

The SDK writes real data, and every LucidLink client linked to the filespace sees it. No sandbox environment exists, so run your first scripts against a filespace you keep for testing.

Install the SDK

pip install lucidlink

Read and write your first file

The script below logs in, links a filespace, lists its root directory, and writes a file and reads it back. Replace <SECRET_KEY> with your service account's secret key and <FILESPACE_ID> with the ID of the filespace to link.

import lucidlink
# The client keeps its local state in a temporary directory by defaultwith lucidlink.Client() as client:    credentials = lucidlink.ServiceAccountCredentials(token="<SECRET_KEY>")    client.login(credentials)
    # A service account belongs to exactly one workspace    workspace = client.workspace
    # Get the filespace ID from workspace.list_filespaces()    filespace = workspace.link_filespace("<FILESPACE_ID>")
    # List the root directory    for entry in filespace.file_system.read_dir("/"):        print(f"{entry.name}: {entry.size} bytes")
    # Write a file    with filespace.file_system.open("/example.txt", "wb") as f:        f.write(b"Hello from LucidLink!")
    # Read it back    with filespace.file_system.open("/example.txt", "rb") as f:        print(f.read())
    # Save the changes: blocks until they have reached LucidLink services    filespace.sync()
# Leaving the block calls client.close(): it unlinks every filespace (with a# best-effort final sync) and stops the client

Find and link filespaces

workspace.list_filespaces() returns the name and ID of every filespace the service account can access. Pass an ID to link_filespace() to connect to that filespace.

workspace = client.workspace
# Every filespace the service account can accessfilespaces = workspace.list_filespaces()for fs in filespaces:    print(f"{fs.name}: {fs.id}")
filespace = workspace.link_filespace(filespaces[0].id)

Both Client and Filespace are context managers. Leaving a filespace block unlinks the filespace after a best-effort final sync; leaving the client block unlinks any remaining filespaces and stops the client. To save changes, call filespace.sync() yourself before the block ends, as described in Sync changes below.

import lucidlink
with lucidlink.Client() as client:    credentials = lucidlink.ServiceAccountCredentials(token="<SECRET_KEY>")    client.login(credentials)    workspace = client.workspace
    # Leaving this block unlinks the filespace (with a best-effort final sync)    with workspace.link_filespace("<FILESPACE_ID>") as filespace:        fs_utils = lucidlink.FsUtils(filespace.file_system)        fs_utils.write_bytes("/output.txt", b"data")        filespace.sync()  # Call it yourself when the changes must be saved
# Leaving this block calls client.close(): it unlinks any remaining# filespaces and stops the client

Each linked filespace gets its own data cache (on disk) and metadata cache. Their sizes are caps: the caches grow on demand up to them. By default, the SDK caps them at 5 GiB (data) and 1 GiB (metadata). Pass a CacheConfig to link_filespace() to set either size.

from lucidlink import CacheConfig
# A 2 GiB data cache and a 512 MiB metadata cache; a size left out keeps the size# already set, or the defaultfilespace = workspace.link_filespace(    "<FILESPACE_ID>",    cache=CacheConfig(data_size=2 * 1024**3, metadata_size=512 * 1024**2),)

The SDK picks each size on its own; the first that applies wins:

1. The size given in CacheConfig.

2. A size already set on this machine for this filespace. An explicit size given when the filespace was linked before stays set when the same client links it again and, with StorageMode.PERSISTENT, when later clients with the same root_path link it.

3. The filespace-wide value, if the filespace has one.

4. The SDK default: 5 GiB data, 1 GiB metadata.

The SDK default is removed when the filespace is unlinked. While it is in place, a change to the filespace-wide value takes effect the next time the filespace is linked.

Linking an already linked filespace again returns the existing Filespace when every size set in cache equals either the size given when the filespace was linked or its actual size; a size left as None imposes no requirement. A differing size raises FilespaceAlreadyLinkedError, naming the requested and the actual size.

Sync changes

The Python SDK syncs changes to LucidLink services in the background. filespace.sync() is an explicit barrier: it returns only after pending changes have reached LucidLink services. Other clients pick the changes up from there on their own, so they can still miss them for a short time.

Calls on the same client see its own changes immediately, without sync(). The exceptions are listing and counting the connected files of a LucidLink Connect data store (iter_files() and count()): LucidLink services answer them, so they show only synced changes. A data store also has to be synced before remove() or rekey() can act on it.

When changes must be saved, call filespace.sync() yourself before leaving a with block or calling unlink(). sync() raises if the changes can't reach LucidLink services, while the filespace is still linked and your code can still retry or react.

from lucidlink import FsUtils
with workspace.link_filespace("<FILESPACE_ID>") as filespace:    FsUtils(filespace.file_system).write_bytes("/data.txt", b"important data")    filespace.sync()  # Saved: the changes are now on LucidLink services

By default (SyncBarrier.ON_UNLINK), filespace.unlink() and leaving a with block also run a final sync before disconnecting. That final sync is best-effort. If it fails — for example because LucidLink services are unreachable — the filespace is still unlinked, and unlink() raises FilespaceSyncError. The unsynced changes stay in the local cache only; with ephemeral storage, they're lost when the client closes. If the with block is already exiting with an exception, that exception propagates and carries the sync failure as a note.

Files opened through filespace.file_system that are still open are closed before the final sync, also with SyncBarrier.NONE, so the data they buffered is written. Using such a file afterwards raises ValueError, like any closed file. If closing a file fails, the filespace is still unlinked, and unlink() raises FilespaceError with a note naming the close error.

client.close(), and leaving the client's with block, unlinks every filespace still linked the same way. It unlinks every filespace and closes the client even when a step fails, then raises a LucidLinkErrorGroup with one member per failure — for example, a FilespaceSyncError for each filespace whose final sync failed. workspace.stop() reports the failures of unlinking its filespaces the same way.

To skip the barrier when unlinking, set sync_barrier to SyncBarrier.NONE. Your code is then responsible for calling sync():

from lucidlink import SyncBarrier
filespace = workspace.link_filespace("<FILESPACE_ID>")filespace.sync_barrier = SyncBarrier.NONE
# ... write operations ...
filespace.sync()    # Your code owns the barrier nowfilespace.unlink()  # Returns without waiting for a sync

Choose where the client stores data

The client keeps its local state in a temporary directory and deletes it when the client closes.

import lucidlink
client = lucidlink.Client()

The client keeps its local state in a .lucid folder under root_path, and the folder stays after the client closes. Persistent mode isn't available with LucidLinkFileSystem, which always uses ephemeral storage.

from pathlib import Path
from lucidlink import Client, StorageConfig, StorageMode
client = Client(storage=StorageConfig(    mode=StorageMode.PERSISTENT,    root_path=Path.home() / "lucid_data",))

Multiple clients must not share persistent storage: give each client its own root_path. Use ephemeral storage whenever more than one client is involved.

Handle errors

Every error the Python SDK raises on purpose is a LucidLinkError, except where a Python convention applies: TypeError for an argument of the wrong type, ValueError and io.UnsupportedOperation on a file stream (for example, a closed file), LookupError for an unknown encoding, NotImplementedError from the fsspec integration, shutil.SameFileError from cp_file() when both paths name the same file, and ImportError for a missing optional dependency. Where a condition has a built-in Python meaning, the class also inherits the built-in exception — FsNotFoundError is also a FileNotFoundError — so the except clauses you already write keep working.

import lucidlinkfrom lucidlink import FsUtilsfrom lucidlink.exceptions import (    AuthenticationError,    LucidLinkError,    ServiceUnavailableError,)
try:    with lucidlink.Client() as client:        credentials = lucidlink.ServiceAccountCredentials(token="<SECRET_KEY>")        client.login(credentials)        with client.workspace.link_filespace("<FILESPACE_ID>") as filespace:            fs_utils = FsUtils(filespace.file_system)            data = fs_utils.read_bytes("/reports/q3.csv")except FileNotFoundError as e:  # Also an FsNotFoundError: errno, filename    print(f"No such file: {e.filename}")except AuthenticationError as e:  # Get a new secret key    print(f"Auth failed: {e}")except ServiceUnavailableError as e:  # Retry later    print(f"LucidLink services unreachable: {e}")except LucidLinkError as e:  # Any other SDK error    print(f"SDK error: {e}")

After client.close(), every operation on the client, and on every object obtained through it, raises ClientClosedError. A file left open was closed with the client and raises ValueError, like any closed file.

Next steps

File operations covers streaming, directory operations, and the fsspec and pandas integrations.

LucidLink Connect covers linking objects from S3-compatible storage, and files at HTTP URLs, into a filespace.

Your feedback shapes what ships next. To report an issue, request a feature, or describe how you use the SDK, contact your LucidLink account representative or LucidLink support.