Skip to main content

file_system

Files, directories and symbolic links of a linked filespace.

filespace.file_system is the filespace's FileSystem: listing, metadata, links, and FileSystem.open() for streaming reads and writes through file objects (a FileStream when unbuffered). FsUtils adds whole-file reads and writes and the deletion of a directory tree on top of it.

enum

lucidlink.EntryType

Kind of a file system entry, as reported by FsEntry.type.

Member

Value

Description

FILE

'file'

Regular file.

DIR

'dir'

Directory.

SYMLINK

'symlink'

Symbolic link.

FILE

Value

'file'

Description

Regular file.

DIR

Value

'dir'

Description

Directory.

SYMLINK

Value

'symlink'

Description

Symbolic link.

class

lucidlink.FileStream

FileStream()

An unbuffered binary file opened on a filespace.

Get one from filespace.file_system.open(path, mode, buffering=0) with a binary mode. The buffered and text file objects FileSystem.open() returns otherwise wrap one: f.raw, or f.buffer.raw in a text mode.

Errors common to every call, not repeated in each Raises section: every method except close() raises ValueError on a closed stream; a read in a mode that does not read, or a write or truncate() in a mode that does not write, raises io.UnsupportedOperation. Unlinking its Filespace or closing its Client closes the stream, and after a filespace internal error its calls raise the lifecycle errors of Filespace.

FileStream example

# buffering=0 returns the FileStream itselfwith filespace.file_system.open("/output.csv", "wb", buffering=0) as f:    f.write(b"column1,column2\n")    f.write(b"value1,value2\n")
with filespace.file_system.open("/output.csv", "rb", buffering=0) as f:    data = f.read()
property

closed: bool

Whether the stream is closed.

property

mode: str

The mode the file was opened in, in its binary form ('rb' for a file opened in 'r').

method

close() -> None

Close the stream.

If closing fails, the error is raised and the stream is closed anyway.

method

fileno() -> int

Raise io.UnsupportedOperation: a file of a filespace has no file descriptor.

Raises

Exception

Condition

io.UnsupportedOperation

Always.

io.UnsupportedOperation

Condition

Always.

method

isatty() -> bool

Return whether the stream is an interactive terminal.

Returns

Always False.

method

read(size: int | None = -1) -> bytes

Read up to size bytes from the current position.

Parameters

Parameter

Description

size

The maximum number of bytes to read; None or a negative value reads to the end of the file.

size

Description

The maximum number of bytes to read; None or a negative value reads to the end of the file.

Returns

The bytes read; empty at the end of the file.

Raises

Exception

Condition

FsNotFoundError

If the file was deleted after it was opened.

Condition

If the file was deleted after it was opened.

method

readable() -> bool

Return whether the stream can be read.

Returns

True if the stream was opened in a mode that reads.

method

readall() -> bytes

Read from the current position to the end of the file.

Returns

The bytes read; empty at the end of the file.

Raises

Exception

Condition

FsNotFoundError

If the file was deleted after it was opened.

Condition

If the file was deleted after it was opened.

method

readinto(b: Buffer) -> int

Read from the current position into a buffer.

Parameters

Parameter

Description

b

Writable bytes-like object to fill (bytearray, memoryview, array.array, a numpy array, ...); up to its size in bytes is read.

b

Description

Writable bytes-like object to fill (bytearray, memoryview, array.array, a numpy array, ...); up to its size in bytes is read.

Returns

The number of bytes read into b; 0 at the end of the file.

Raises

Exception

Condition

TypeError

If b is not a writable bytes-like object.

FsNotFoundError

If the file was deleted after it was opened.

TypeError

Condition

If b is not a writable bytes-like object.

Condition

If the file was deleted after it was opened.

method

seek(offset: int, whence: int = 0) -> int

Move the stream position.

A position before the start of the file, with whence 1 or 2, becomes 0.

Parameters

Parameter

Description

offset

Position in bytes, relative to whence.

whence

0, the default, for the start of the file, 1 for the current position, 2 for the end of the file.

offset

Description

Position in bytes, relative to whence.

whence

Description

0, the default, for the start of the file, 1 for the current position, 2 for the end of the file.

Returns

The new position, from the start of the file.

Raises

Exception

Condition

InvalidArgumentError

If whence is not 0, 1 or 2, offset is negative with whence 0, or the new position is larger than 2**63 - 1.

FsNotFoundError

If whence is 2 and the file was deleted after it was opened.

Condition

If whence is not 0, 1 or 2, offset is negative with whence 0, or the new position is larger than 2**63 - 1.

Condition

If whence is 2 and the file was deleted after it was opened.

method

seekable() -> bool

Return whether the stream supports random access.

Returns

Always True.

method

tell() -> int

Return the stream position.

Returns

The position, in bytes from the start of the file.

method

truncate(size: int | None = None) -> int

Shorten or extend the file to a given size.

Parameters

Parameter

Description

size

New size of the file in bytes; the current position if None.

size

Description

New size of the file in bytes; the current position if None.

Returns

The new size of the file.

Raises

Exception

Condition

TypeError

If size is not an integer.

InvalidArgumentError

If size is negative or larger than 2**64 - 1.

FsError

If size reaches the filespace's maximum file size (errno.EFBIG).

FsNoSpaceError

If a data or storage limit of the filespace is reached, even when size shortens the file.

TypeError

Condition

If size is not an integer.

Condition

If size is negative or larger than 2**64 - 1.

Condition

If size reaches the filespace's maximum file size (errno.EFBIG).

Condition

If a data or storage limit of the filespace is reached, even when size shortens the file.

method

writable() -> bool

Return whether the stream can be written.

Returns

True if the stream was opened in a mode that writes.

method

write(b: Buffer) -> int

Write b at the current position.

In an append mode every write goes to the end of the file, even after seek().

Parameters

Parameter

Description

b

Bytes-like object to write (bytes, bytearray, memoryview, array.array, a numpy array, ...).

b

Description

Bytes-like object to write (bytes, bytearray, memoryview, array.array, a numpy array, ...).

Returns

The number of bytes written, always all of b.

Raises

Exception

Condition

TypeError

If b is not a bytes-like object.

FsError

If the end of the write reaches the filespace's maximum file size (errno.EFBIG); nothing is written.

FsNoSpaceError

If a data or storage limit of the filespace is reached.

FsNotFoundError

If, in an append mode, the file was deleted after it was opened.

TypeError

Condition

If b is not a bytes-like object.

Condition

If the end of the write reaches the filespace's maximum file size (errno.EFBIG); nothing is written.

Condition

If a data or storage limit of the filespace is reached.

Condition

If, in an append mode, the file was deleted after it was opened.

class

lucidlink.FileSystem

FileSystem()

File, directory and symbolic link operations on a linked filespace.

Get it from filespace.file_system.

A symbolic link among the directories of a path is followed; the last component of a path never is: get_entry(), delete() and move() act on the link itself, read_dir() lists links as links, and open() and truncate() on a link raise FsError with errno.ELOOP. A chain of links that is too long, or a loop, raises FsError with errno.ELOOP too. A relative link target is resolved from the root of the filespace, not from the link's own directory: use absolute link targets.

A path that ends in / names a directory: for an existing entry of another kind, a symbolic link included, a call raises FsNotADirectoryError, except open() in a writing mode (FsIsADirectoryError) and a call that creates the path: create_dir(), create_symlink(), create_hard_link() and move() to it (FsExistsError).

Names and symbolic link targets are stored in Unicode NFD form, and read_dir() and read_symlink() return them so; a path in any normalization form finds the same entry.

Errors common to every call, not repeated in each Raises section: FsPermissionError for a path the service account may not access, the audit and lost+found directories included, and with errno.EROFS for a change on a read-only filespace or to an external (Connect) file on filespace format 3.6; FsError with errno.EBUSY when another client holds a lock on the path; FsNoSpaceError once this client learns that a limit of the filespace is reached: the entry limit refuses the calls that create an entry, the data and storage limits also every change to file data, a truncate() that shortens the file included; and InvalidArgumentError for a path with an embedded NUL byte. Every call also raises the lifecycle errors of its Filespace and Client (FilespaceNotLinkedError, FilespaceInternalError, ClientClosedError).

FileSystem example

filespace = workspace.link_filespace("fs-uuid-12345")entries = filespace.file_system.read_dir("/")filespace.file_system.create_dir("/new-folder")with filespace.file_system.open("/new-folder/file.txt", "wb") as f:    f.write(b"data")
method

create_dir(path: str) -> None

Create a directory.

Only path itself is created: its parent directory must already exist, so create missing parents first, from the top down.

Parameters

Parameter

Description

path

Path of the directory to create.

path

Description

Path of the directory to create.

Raises

Exception

Condition

FsNotFoundError

If the parent directory does not exist, or path is the root.

FsNotADirectoryError

If the parent is a file.

FsExistsError

If path already exists, as a directory, a file or a symbolic link.

Condition

If the parent directory does not exist, or path is the root.

Condition

If the parent is a file.

Condition

If path already exists, as a directory, a file or a symbolic link.

method

delete(path: str) -> None

Delete a file, or a symbolic link itself.

Parameters

Parameter

Description

path

Path of the file or link.

path

Description

Path of the file or link.

Raises

Exception

Condition

FsNotFoundError

If path does not exist.

FsIsADirectoryError

If path is a directory.

Condition

If path does not exist.

Condition

If path is a directory.

method

delete_dir(path: str) -> None

Delete an empty directory.

Parameters

Parameter

Description

path

Path of the directory to delete.

path

Description

Path of the directory to delete.

Raises

Exception

Condition

FsNotFoundError

If path does not exist, or is the root of an empty filespace.

FsNotADirectoryError

If path is not a directory.

FsDirectoryNotEmptyError

If the directory is not empty, the root included.

Condition

If path does not exist, or is the root of an empty filespace.

Condition

If path is not a directory.

Condition

If the directory is not empty, the root included.

method

get_entry(path: str) -> FsEntry

Get the metadata of a file, directory or symbolic link.

Parameters

Parameter

Description

path

Path of the entry.

path

Description

Path of the entry.

Returns

The metadata of the entry.

Raises

Exception

Condition

FsNotFoundError

If path does not exist.

Condition

If path does not exist.

method

move(src: str, dst: str) -> None

Rename a file, directory or symbolic link, never overwriting an existing entry.

Parameters

Parameter

Description

src

Path of the entry to rename.

dst

New path of the entry.

src

Description

Path of the entry to rename.

dst

Description

New path of the entry.

Raises

Exception

Condition

FsNotFoundError

If src, or the directory of dst, does not exist; filename is src and filename2 is dst.

FsExistsError

If dst already exists, a symbolic link included; filename2 is dst.

FsNotADirectoryError

If either path ends in / and src is not a directory, or the directory of dst is not a directory, a symbolic link to a file included.

FsError

If src is a directory and dst is inside it (errno.EINVAL). When dst reaches into src through .. or a symbolic link, the move raises FsNotFoundError instead.

Condition

If src, or the directory of dst, does not exist; filename is src and filename2 is dst.

Condition

If dst already exists, a symbolic link included; filename2 is dst.

Condition

If either path ends in / and src is not a directory, or the directory of dst is not a directory, a symbolic link to a file included.

Condition

If src is a directory and dst is inside it (errno.EINVAL). When dst reaches into src through .. or a symbolic link, the move raises FsNotFoundError instead.

method

open(path: str, mode: str = 'rb', buffering: int = -1, encoding: str | None = None, errors: str | None = None, newline: str | None = None) -> IO[Any] | FileStream

Open a file for streaming reads and writes.

The file object it returns works with the libraries that read from or write to file objects, such as pandas. Its type depends on mode and buffering: io.TextIOWrapper in a text mode; io.BufferedReader, io.BufferedWriter or io.BufferedRandom for reading, writing or both in a binary mode; and FileStream with buffering=0.

Parameters

Parameter

Description

path

Path of the file.

mode

How to open the file, 'rb' by default. 'r' reads; 'w' creates or truncates the file and writes; 'a' appends, creating the file if needed, every write going to the end of the file, even after seek(). A '+' adds reading or writing to these. A 'b' opens the file in binary mode, otherwise it opens in text mode ('t').

buffering

-1, the default, for a buffer of io.DEFAULT_BUFFER_SIZE bytes; 0 for no buffering, in a binary mode only; 1 for line buffering, in a text mode only (a binary mode warns and uses the default); a larger value for a buffer of that many bytes.

encoding

Text encoding in a text mode, 'utf-8' by default; 'locale' selects the encoding of the current locale.

errors

How a text mode handles encoding and decoding errors: 'strict' (the default) raises, 'ignore' skips the character, 'replace' puts a replacement character in its place.

newline

Line endings in a text mode. None, the default, reads any line ending as '\n' and writes '\n' as the platform's line ending; '' reads any line ending as it is and translates nothing on write; '\n', '\r' or '\r\n' is the only line ending recognized on read and what '\n' is written as.

path

Description

Path of the file.

mode

Description

How to open the file, 'rb' by default. 'r' reads; 'w' creates or truncates the file and writes; 'a' appends, creating the file if needed, every write going to the end of the file, even after seek(). A '+' adds reading or writing to these. A 'b' opens the file in binary mode, otherwise it opens in text mode ('t').

buffering

Description

-1, the default, for a buffer of io.DEFAULT_BUFFER_SIZE bytes; 0 for no buffering, in a binary mode only; 1 for line buffering, in a text mode only (a binary mode warns and uses the default); a larger value for a buffer of that many bytes.

encoding

Description

Text encoding in a text mode, 'utf-8' by default; 'locale' selects the encoding of the current locale.

errors

Description

How a text mode handles encoding and decoding errors: 'strict' (the default) raises, 'ignore' skips the character, 'replace' puts a replacement character in its place.

newline

Description

Line endings in a text mode. None, the default, reads any line ending as '\n' and writes '\n' as the platform's line ending; '' reads any line ending as it is and translates nothing on write; '\n', '\r' or '\r\n' is the only line ending recognized on read and what '\n' is written as.

Returns

The open file; closing it, or leaving its with block, closes the file.

Raises

Exception

Condition

FsNotFoundError

If the file does not exist in a reading mode, its parent directory does not exist in a creating mode, or a component of path is a file.

FsIsADirectoryError

If path is a directory, or ends in / in a writing mode.

FsNotADirectoryError

If the parent of a file to be created is a file.

FsError

If path is a symbolic link (errno.ELOOP), in every mode.

InvalidArgumentError

If mode, buffering or newline is invalid.

LookupError

If, in a text mode, encoding is not a known text encoding.

Condition

If the file does not exist in a reading mode, its parent directory does not exist in a creating mode, or a component of path is a file.

Condition

If path is a directory, or ends in / in a writing mode.

Condition

If the parent of a file to be created is a file.

Condition

If path is a symbolic link (errno.ELOOP), in every mode.

Condition

If mode, buffering or newline is invalid.

LookupError

Condition

If, in a text mode, encoding is not a known text encoding.

FileSystem.open example

import pandas as pd
fs = filespace.file_system
# Binary modewith fs.open("/output.txt", "wb") as f:    f.write(b"Hello, LucidLink!\n")with fs.open("/output.txt", "rb") as f:    data = f.read()
# Text modewith fs.open("/output.txt", "rt", encoding="utf-8") as f:    for line in f:        print(line.strip())
# With pandaswith fs.open("/data.csv", "w") as f:    f.write("a,b\n1,2\n")with fs.open("/data.csv", "rb") as f:    df = pd.read_csv(f)
method

read_dir(path: str) -> list[FsEntry]

List the entries of a directory.

Parameters

Parameter

Description

path

Path of the directory.

path

Description

Path of the directory.

Returns

The entries in the directory.

Raises

Exception

Condition

FsNotFoundError

If path does not exist.

FsNotADirectoryError

If path is not a directory, or is a symbolic link.

Condition

If path does not exist.

Condition

If path is not a directory, or is a symbolic link.

method

statistics() -> FileSystemStatistics

Get the entry count and the sizes of the whole file system.

Makes no network call, so changes made moments ago may not be counted yet.

Returns

The counters of the file system, as this client last received them.

method

truncate(path: str, size: int) -> None

Shorten or extend a file to a given size.

Parameters

Parameter

Description

path

Path of the file.

size

New size of the file in bytes.

path

Description

Path of the file.

size

Description

New size of the file in bytes.

Raises

Exception

Condition

TypeError

If size is not an integer.

InvalidArgumentError

If size is negative or larger than 2**64 - 1.

FsNotFoundError

If path does not exist.

FsIsADirectoryError

If path is a directory.

FsError

If path is a symbolic link (errno.ELOOP), or size reaches the filespace's maximum file size (errno.EFBIG).

TypeError

Condition

If size is not an integer.

Condition

If size is negative or larger than 2**64 - 1.

Condition

If path does not exist.

Condition

If path is a directory.

Condition

If path is a symbolic link (errno.ELOOP), or size reaches the filespace's maximum file size (errno.EFBIG).

dataclass

lucidlink.FileSystemStatistics

Size and entry counters of the file system.

Returned by FileSystem.statistics(). LucidLink services maintain the values and this client keeps the last ones it received, so reading them makes no network call. External (Connect) files are counted in entries_count and also in the two external_files_* fields; storage_size does not include them.

Attribute

Type

Description

entries_count

int

Number of entries: files, directories and symbolic links, the directories LucidLink services reserve included and the root not. A file with several hard links counts once.

storage_size

int

Bytes the filespace holds in its object storage. Smaller than the size of the file data when the data compresses, and larger while data kept by snapshots, or of deleted or overwritten files, is not yet reclaimed.

external_files_size

int

Size of external (Connect) files in bytes.

external_files_count

int

Number of external (Connect) files.

entries_count

Type

int

Description

Number of entries: files, directories and symbolic links, the directories LucidLink services reserve included and the root not. A file with several hard links counts once.

storage_size

Type

int

Description

Bytes the filespace holds in its object storage. Smaller than the size of the file data when the data compresses, and larger while data kept by snapshots, or of deleted or overwritten files, is not yet reclaimed.

external_files_size

Type

int

Description

Size of external (Connect) files in bytes.

external_files_count

Type

int

Description

Number of external (Connect) files.

dataclass

lucidlink.FsEntry

A file, directory or symbolic link, as FileSystem.read_dir() and FileSystem.get_entry() return it.

Attribute

Type

Description

id

str

Unique entry identifier. Hard links to the same file have distinct entry IDs.

name

str

The name of the entry, without its directory, in Unicode NFD form.

type

EntryType

The kind of entry: file, directory or symbolic link.

size

int

Size in bytes (0 for directories and symbolic links).

modified_at

int

Last modification time as a Unix timestamp in nanoseconds (0 for the root).

created_at

int

Creation time as a Unix timestamp in nanoseconds (0 for the root).

inode_id

str

Inode identifier: shared by all hard links to the same file.

num_hard_links

int

Hard-link count of the file's inode.

id

Type

str

Description

Unique entry identifier. Hard links to the same file have distinct entry IDs.

name

Type

str

Description

The name of the entry, without its directory, in Unicode NFD form.

type

Description

The kind of entry: file, directory or symbolic link.

size

Type

int

Description

Size in bytes (0 for directories and symbolic links).

modified_at

Type

int

Description

Last modification time as a Unix timestamp in nanoseconds (0 for the root).

created_at

Type

int

Description

Creation time as a Unix timestamp in nanoseconds (0 for the root).

inode_id

Type

str

Description

Inode identifier: shared by all hard links to the same file.

num_hard_links

Type

int

Description

Hard-link count of the file's inode.

property

is_dir: bool

Whether this entry is a directory.

property

is_file: bool

Whether this entry is a regular file.

class

lucidlink.FsUtils

FsUtils(fs: FileSystem)

Parameter

Description

fs

The FileSystem to operate on, usually filespace.file_system.

fs

Description

The FileSystem to operate on, usually filespace.file_system.

Whole-file reads and writes and the deletion of a directory tree on a FileSystem.

Its calls raise the errors common to every FileSystem call, the lifecycle errors of its Filespace and Client included.

Raises

Exception

Condition

TypeError

If fs is not a FileSystem.

TypeError

Condition

If fs is not a FileSystem.

FsUtils example

from lucidlink import FsUtils
utils = FsUtils(filespace.file_system)filespace.file_system.create_dir("/reports")utils.write_bytes("/reports/hello.txt", b"world")print(utils.read_bytes("/reports/hello.txt"))  # b'world'utils.delete_tree("/reports")  # the directory and everything in it
method

delete_tree(path: str) -> None

Delete a directory together with everything below it.

The symbolic links below it are deleted themselves; nothing they point to is touched. Not atomic: when the deletion fails partway, for example on a file another client holds a lock on, the entries deleted so far stay deleted.

Parameters

Parameter

Description

path

Path of the directory to delete.

path

Description

Path of the directory to delete.

Raises

Exception

Condition

FsNotFoundError

If path does not exist.

FsNotADirectoryError

If path is not a directory, a symbolic link included.

Condition

If path does not exist.

Condition

If path is not a directory, a symbolic link included.

method

read_bytes(path: str) -> bytes

Read the whole content of a file.

Shorthand for FileSystem.open(path, "rb") followed by read().

Parameters

Parameter

Description

path

Path of the file.

path

Description

Path of the file.

Returns

The content of the file.

Raises

Exception

Condition

FsNotFoundError

If path does not exist.

FsNotADirectoryError

If path ends in / and is not a directory.

FsIsADirectoryError

If path is a directory.

FsError

If path is a symbolic link (errno.ELOOP).

Condition

If path does not exist.

Condition

If path ends in / and is not a directory.

Condition

If path is a directory.

Condition

If path is a symbolic link (errno.ELOOP).

method

write_bytes(path: str, data: Buffer) -> None

Write data to a file, creating it if needed and replacing any previous content.

Not atomic: the file is emptied first, then written, so a write that fails leaves the file empty or partly written. The parent directory must already exist.

Parameters

Parameter

Description

path

Path of the file.

data

Bytes-like object to write (bytes, bytearray, memoryview, ...).

path

Description

Path of the file.

data

Description

Bytes-like object to write (bytes, bytearray, memoryview, ...).

Raises

Exception

Condition

FsNotFoundError

If the parent directory does not exist.

FsNotADirectoryError

If the parent is a file.

FsIsADirectoryError

If path is a directory, or ends in /.

FsError

If path is a symbolic link (errno.ELOOP).

TypeError

If data is not a bytes-like object, a str for example.

Condition

If the parent directory does not exist.

Condition

If the parent is a file.

Condition

If path is a directory, or ends in /.

Condition

If path is a symbolic link (errno.ELOOP).

TypeError

Condition

If data is not a bytes-like object, a str for example.