file_system
Files, directories and symbolic links of a linked filespace.
filespace.file_system is the filespace's FileSystem: listing, metadata, links, and FileSystem.open() for streaming reads and writes through file objects (a FileStream when unbuffered). FsUtils adds whole-file reads and writes and the deletion of a directory tree on top of it.
lucidlink.EntryType
Kind of a file system entry, as reported by FsEntry.type.
Member | Value | Description |
|---|---|---|
|
| Regular file. |
|
| Directory. |
|
| Symbolic link. |
FILE
Value
'file'Description
Regular file.
DIR
Value
'dir'Description
Directory.
SYMLINK
Value
'symlink'Description
Symbolic link.
lucidlink.FileStream
FileStream()An unbuffered binary file opened on a filespace.
Get one from filespace.file_system.open(path, with a binary mode. The buffered and text file objects FileSystem.open() returns otherwise wrap one: f.raw, or f.buffer.raw in a text mode.
Errors common to every call, not repeated in each Raises section: every method except close() raises ValueError on a closed stream; a read in a mode that does not read, or a write or truncate() in a mode that does not write, raises io.UnsupportedOperation. Unlinking its Filespace or closing its Client closes the stream, and after a filespace internal error its calls raise the lifecycle errors of Filespace.
FileStream example
# buffering=0 returns the FileStream itselfwith filespace.file_system.open("/output.csv", "wb", buffering=0) as f:f.write(b"column1,column2\n")f.write(b"value1,value2\n")with filespace.file_system.open("/output.csv", "rb", buffering=0) as f:data = f.read()
closed: bool
Whether the stream is closed.
mode: str
The mode the file was opened in, in its binary form ('rb' for a file opened in 'r').
close() -> None
Close the stream.
If closing fails, the error is raised and the stream is closed anyway.
fileno() -> int
Raise io.UnsupportedOperation: a file of a filespace has no file descriptor.
Raises
Exception | Condition |
|---|---|
| Always. |
io.UnsupportedOperation
Condition
Always.
isatty() -> bool
Return whether the stream is an interactive terminal.
Returns
Always False.
read(size: int | None = -1) -> bytes
Read up to size bytes from the current position.
Parameters
Parameter | Description |
|---|---|
| The maximum number of bytes to read; None or a negative value reads to the end of the file. |
size
Description
The maximum number of bytes to read; None or a negative value reads to the end of the file.
Returns
The bytes read; empty at the end of the file.
Raises
Exception | Condition |
|---|---|
If the file was deleted after it was opened. |
readable() -> bool
Return whether the stream can be read.
Returns
True if the stream was opened in a mode that reads.
readall() -> bytes
Read from the current position to the end of the file.
Returns
The bytes read; empty at the end of the file.
Raises
Exception | Condition |
|---|---|
If the file was deleted after it was opened. |
readinto(b: Buffer) -> int
Read from the current position into a buffer.
Parameters
Parameter | Description |
|---|---|
| Writable bytes-like object to fill (bytearray, memoryview, array.array, a numpy array, ...); up to its size in bytes is read. |
b
Description
Writable bytes-like object to fill (bytearray, memoryview, array.array, a numpy array, ...); up to its size in bytes is read.
Returns
The number of bytes read into b; 0 at the end of the file.
Raises
Exception | Condition |
|---|---|
| If b is not a writable bytes-like object. |
If the file was deleted after it was opened. |
TypeError
Condition
If b is not a writable bytes-like object.
seek(offset: int, whence: int = 0) -> int
Move the stream position.
A position before the start of the file, with whence 1 or 2, becomes 0.
Parameters
Parameter | Description |
|---|---|
| Position in bytes, relative to whence. |
| 0, the default, for the start of the file, 1 for the current position, 2 for the end of the file. |
offset
Description
Position in bytes, relative to whence.
whence
Description
0, the default, for the start of the file, 1 for the current position, 2 for the end of the file.
Returns
The new position, from the start of the file.
Raises
Exception | Condition |
|---|---|
If whence is not 0, 1 or 2, offset is negative with whence 0, or the new position is larger than 2**63 - 1. | |
If whence is 2 and the file was deleted after it was opened. |
Condition
If whence is not 0, 1 or 2, offset is negative with whence 0, or the new position is larger than 2**63 - 1.
seekable() -> bool
Return whether the stream supports random access.
Returns
Always True.
tell() -> int
Return the stream position.
Returns
The position, in bytes from the start of the file.
truncate(size: int | None = None) -> int
Shorten or extend the file to a given size.
Parameters
Parameter | Description |
|---|---|
| New size of the file in bytes; the current position if None. |
size
Description
New size of the file in bytes; the current position if None.
Returns
The new size of the file.
Raises
Exception | Condition |
|---|---|
| If size is not an integer. |
If size is negative or larger than 2**64 - 1. | |
If size reaches the filespace's maximum file size (errno.EFBIG). | |
If a data or storage limit of the filespace is reached, even when size shortens the file. |
TypeError
Condition
If size is not an integer.
Condition
If a data or storage limit of the filespace is reached, even when size shortens the file.
writable() -> bool
Return whether the stream can be written.
Returns
True if the stream was opened in a mode that writes.
write(b: Buffer) -> int
Write b at the current position.
In an append mode every write goes to the end of the file, even after seek().
Parameters
Parameter | Description |
|---|---|
| Bytes-like object to write (bytes, bytearray, memoryview, array.array, a numpy array, ...). |
b
Description
Bytes-like object to write (bytes, bytearray, memoryview, array.array, a numpy array, ...).
Returns
The number of bytes written, always all of b.
Raises
Exception | Condition |
|---|---|
| If b is not a bytes-like object. |
If the end of the write reaches the filespace's maximum file size (errno.EFBIG); nothing is written. | |
If a data or storage limit of the filespace is reached. | |
If, in an append mode, the file was deleted after it was opened. |
TypeError
Condition
If b is not a bytes-like object.
Condition
If the end of the write reaches the filespace's maximum file size (errno.EFBIG); nothing is written.
lucidlink.FileSystem
FileSystem()File, directory and symbolic link operations on a linked filespace.
Get it from filespace.file_system.
A symbolic link among the directories of a path is followed; the last component of a path never is: get_entry(), delete() and move() act on the link itself, read_dir() lists links as links, and open() and truncate() on a link raise FsError with errno.ELOOP. A chain of links that is too long, or a loop, raises FsError with errno.ELOOP too. A relative link target is resolved from the root of the filespace, not from the link's own directory: use absolute link targets.
A path that ends in / names a directory: for an existing entry of another kind, a symbolic link included, a call raises FsNotADirectoryError, except open() in a writing mode (FsIsADirectoryError) and a call that creates the path: create_dir(), create_symlink(), create_hard_link() and move() to it (FsExistsError).
Names and symbolic link targets are stored in Unicode NFD form, and read_dir() and read_symlink() return them so; a path in any normalization form finds the same entry.
Errors common to every call, not repeated in each Raises section: FsPermissionError for a path the service account may not access, the audit and lost+found directories included, and with errno.EROFS for a change on a read-only filespace or to an external (Connect) file on filespace format 3.6; FsError with errno.EBUSY when another client holds a lock on the path; FsNoSpaceError once this client learns that a limit of the filespace is reached: the entry limit refuses the calls that create an entry, the data and storage limits also every change to file data, a truncate() that shortens the file included; and InvalidArgumentError for a path with an embedded NUL byte. Every call also raises the lifecycle errors of its Filespace and Client (FilespaceNotLinkedError, FilespaceInternalError, ClientClosedError).
FileSystem example
filespace = workspace.link_filespace("fs-uuid-12345")entries = filespace.file_system.read_dir("/")filespace.file_system.create_dir("/new-folder")with filespace.file_system.open("/new-folder/file.txt", "wb") as f:f.write(b"data")
create_dir(path: str) -> None
Create a directory.
Only path itself is created: its parent directory must already exist, so create missing parents first, from the top down.
Parameters
Parameter | Description |
|---|---|
| Path of the directory to create. |
path
Description
Path of the directory to create.
Raises
Exception | Condition |
|---|---|
If the parent directory does not exist, or path is the root. | |
If the parent is a file. | |
If path already exists, as a directory, a file or a symbolic link. |
create_hard_link(target_path: str, link_path: str) -> None
Create a hard link to a file.
Requires filespace format 3.8 or newer.
Parameters
Parameter | Description |
|---|---|
| Path of the existing file to link to. |
| Path of the hard link to create. |
target_path
Description
Path of the existing file to link to.
link_path
Description
Path of the hard link to create.
Raises
Exception | Condition |
|---|---|
If target_path is empty. | |
If target_path, or the parent directory of link_path, does not exist, or link_path ends in / and does not exist. | |
If the parent of link_path is a file. | |
If link_path already exists. | |
If target_path is not a file: a directory or a symbolic link (errno.EPERM). | |
If the filespace format is older than 3.8 (errno.ENOSYS). |
Condition
If target_path, or the parent directory of link_path, does not exist, or link_path ends in / and does not exist.
Condition
If target_path is not a file: a directory or a symbolic link (errno.EPERM).
create_symlink(target_path: str, link_path: str) -> None
Create a symbolic link.
Parameters
Parameter | Description |
|---|---|
| Path the link points to, stored as given but in Unicode NFD form. Use an absolute path: a relative one is resolved from the root of the filespace. |
| Path of the link to create. |
target_path
Description
Path the link points to, stored as given but in Unicode NFD form. Use an absolute path: a relative one is resolved from the root of the filespace.
link_path
Description
Path of the link to create.
Raises
Exception | Condition |
|---|---|
If target_path is empty. | |
If link_path already exists. | |
If the parent directory of link_path does not exist, or link_path ends in / and does not exist. | |
If the parent of link_path is a file. |
Condition
If the parent directory of link_path does not exist, or link_path ends in / and does not exist.
delete(path: str) -> None
Delete a file, or a symbolic link itself.
Parameters
Parameter | Description |
|---|---|
| Path of the file or link. |
path
Description
Path of the file or link.
Raises
Exception | Condition |
|---|---|
If path does not exist. | |
If path is a directory. |
delete_dir(path: str) -> None
Delete an empty directory.
Parameters
Parameter | Description |
|---|---|
| Path of the directory to delete. |
path
Description
Path of the directory to delete.
Raises
Exception | Condition |
|---|---|
If path does not exist, or is the root of an empty filespace. | |
If path is not a directory. | |
If the directory is not empty, the root included. |
get_entry(path: str) -> FsEntry
Get the metadata of a file, directory or symbolic link.
Parameters
Parameter | Description |
|---|---|
| Path of the entry. |
path
Description
Path of the entry.
Returns
The metadata of the entry.
Raises
Exception | Condition |
|---|---|
If path does not exist. |
move(src: str, dst: str) -> None
Rename a file, directory or symbolic link, never overwriting an existing entry.
Parameters
Parameter | Description |
|---|---|
| Path of the entry to rename. |
| New path of the entry. |
src
Description
Path of the entry to rename.
dst
Description
New path of the entry.
Raises
Exception | Condition |
|---|---|
If src, or the directory of dst, does not exist; filename is src and filename2 is dst. | |
If dst already exists, a symbolic link included; filename2 is dst. | |
If either path ends in / and src is not a directory, or the directory of dst is not a directory, a symbolic link to a file included. | |
If src is a directory and dst is inside it (errno.EINVAL). When dst reaches into src through .. or a symbolic link, the move raises FsNotFoundError instead. |
Condition
If src, or the directory of dst, does not exist; filename is src and filename2 is dst.
Condition
If either path ends in / and src is not a directory, or the directory of dst is not a directory, a symbolic link to a file included.
Condition
If src is a directory and dst is inside it (errno.EINVAL). When dst reaches into src through .. or a symbolic link, the move raises FsNotFoundError instead.
open(path: str, mode: str = 'rb', buffering: int = -1, encoding: str | None = None, errors: str | None = None, newline: str | None = None) -> IO[Any] | FileStream
Open a file for streaming reads and writes.
The file object it returns works with the libraries that read from or write to file objects, such as pandas. Its type depends on mode and buffering: io.TextIOWrapper in a text mode; io.BufferedReader, io.BufferedWriter or io.BufferedRandom for reading, writing or both in a binary mode; and FileStream with buffering=0.
Parameters
Parameter | Description |
|---|---|
| Path of the file. |
| How to open the file, 'rb' by default. 'r' reads; 'w' creates or truncates the file and writes; 'a' appends, creating the file if needed, every write going to the end of the file, even after seek(). A '+' adds reading or writing to these. A 'b' opens the file in binary mode, otherwise it opens in text mode ('t'). |
| -1, the default, for a buffer of io.DEFAULT_BUFFER_SIZE bytes; 0 for no buffering, in a binary mode only; 1 for line buffering, in a text mode only (a binary mode warns and uses the default); a larger value for a buffer of that many bytes. |
| Text encoding in a text mode, 'utf-8' by default; 'locale' selects the encoding of the current locale. |
| How a text mode handles encoding and decoding errors: 'strict' (the default) raises, 'ignore' skips the character, 'replace' puts a replacement character in its place. |
| Line endings in a text mode. None, the default, reads any line ending as '\n' and writes '\n' as the platform's line ending; '' reads any line ending as it is and translates nothing on write; '\n', '\r' or '\r\n' is the only line ending recognized on read and what '\n' is written as. |
path
Description
Path of the file.
mode
Description
How to open the file, 'rb' by default. 'r' reads; 'w' creates or truncates the file and writes; 'a' appends, creating the file if needed, every write going to the end of the file, even after seek(). A '+' adds reading or writing to these. A 'b' opens the file in binary mode, otherwise it opens in text mode ('t').
buffering
Description
-1, the default, for a buffer of io.DEFAULT_BUFFER_SIZE bytes; 0 for no buffering, in a binary mode only; 1 for line buffering, in a text mode only (a binary mode warns and uses the default); a larger value for a buffer of that many bytes.
encoding
Description
Text encoding in a text mode, 'utf-8' by default; 'locale' selects the encoding of the current locale.
errors
Description
How a text mode handles encoding and decoding errors: 'strict' (the default) raises, 'ignore' skips the character, 'replace' puts a replacement character in its place.
newline
Description
Line endings in a text mode. None, the default, reads any line ending as '\n' and writes '\n' as the platform's line ending; '' reads any line ending as it is and translates nothing on write; '\n', '\r' or '\r\n' is the only line ending recognized on read and what '\n' is written as.
Returns
The open file; closing it, or leaving its with block, closes the file.
Raises
Exception | Condition |
|---|---|
If the file does not exist in a reading mode, its parent directory does not exist in a creating mode, or a component of path is a file. | |
If path is a directory, or ends in / in a writing mode. | |
If the parent of a file to be created is a file. | |
If path is a symbolic link (errno.ELOOP), in every mode. | |
If mode, buffering or newline is invalid. | |
| If, in a text mode, encoding is not a known text encoding. |
Condition
If the file does not exist in a reading mode, its parent directory does not exist in a creating mode, or a component of path is a file.
LookupError
Condition
If, in a text mode, encoding is not a known text encoding.
FileSystem.open example
import pandas as pdfs = filespace.file_system# Binary modewith fs.open("/output.txt", "wb") as f:f.write(b"Hello, LucidLink!\n")with fs.open("/output.txt", "rb") as f:data = f.read()# Text modewith fs.open("/output.txt", "rt", encoding="utf-8") as f:for line in f:print(line.strip())# With pandaswith fs.open("/data.csv", "w") as f:f.write("a,b\n1,2\n")with fs.open("/data.csv", "rb") as f:df = pd.read_csv(f)
read_dir(path: str) -> list[FsEntry]
List the entries of a directory.
Parameters
Parameter | Description |
|---|---|
| Path of the directory. |
path
Description
Path of the directory.
Returns
The entries in the directory.
Raises
Exception | Condition |
|---|---|
If path does not exist. | |
If path is not a directory, or is a symbolic link. |
read_symlink(link_path: str) -> str
Read the target of a symbolic link.
Parameters
Parameter | Description |
|---|---|
| Path of the link. |
link_path
Description
Path of the link.
Returns
The path the link points to, as stored when the link was created.
Raises
Exception | Condition |
|---|---|
If link_path does not exist, or is not a symbolic link. |
statistics() -> FileSystemStatistics
Get the entry count and the sizes of the whole file system.
Makes no network call, so changes made moments ago may not be counted yet.
Returns
The counters of the file system, as this client last received them.
truncate(path: str, size: int) -> None
Shorten or extend a file to a given size.
Parameters
Parameter | Description |
|---|---|
| Path of the file. |
| New size of the file in bytes. |
path
Description
Path of the file.
size
Description
New size of the file in bytes.
Raises
Exception | Condition |
|---|---|
| If size is not an integer. |
If size is negative or larger than 2**64 - 1. | |
If path does not exist. | |
If path is a directory. | |
If path is a symbolic link (errno.ELOOP), or size reaches the filespace's maximum file size (errno.EFBIG). |
TypeError
Condition
If size is not an integer.
Condition
If path is a symbolic link (errno.ELOOP), or size reaches the filespace's maximum file size (errno.EFBIG).
lucidlink.FileSystemStatistics
Size and entry counters of the file system.
Returned by FileSystem.statistics(). LucidLink services maintain the values and this client keeps the last ones it received, so reading them makes no network call. External (Connect) files are counted in entries_count and also in the two external_files_* fields; storage_size does not include them.
Attribute | Type | Description |
|---|---|---|
|
| Number of entries: files, directories and symbolic links, the directories LucidLink services reserve included and the root not. A file with several hard links counts once. |
|
| Bytes the filespace holds in its object storage. Smaller than the size of the file data when the data compresses, and larger while data kept by snapshots, or of deleted or overwritten files, is not yet reclaimed. |
|
| Size of external (Connect) files in bytes. |
|
| Number of external (Connect) files. |
entries_count
Type
intDescription
Number of entries: files, directories and symbolic links, the directories LucidLink services reserve included and the root not. A file with several hard links counts once.
storage_size
Type
intDescription
Bytes the filespace holds in its object storage. Smaller than the size of the file data when the data compresses, and larger while data kept by snapshots, or of deleted or overwritten files, is not yet reclaimed.
external_files_size
Type
intDescription
Size of external (Connect) files in bytes.
external_files_count
Type
intDescription
Number of external (Connect) files.
lucidlink.FsEntry
A file, directory or symbolic link, as FileSystem.read_dir() and FileSystem.get_entry() return it.
Attribute | Type | Description |
|---|---|---|
|
| Unique entry identifier. Hard links to the same file have distinct entry IDs. |
|
| The name of the entry, without its directory, in Unicode NFD form. |
| The kind of entry: file, directory or symbolic link. | |
|
| Size in bytes (0 for directories and symbolic links). |
|
| Last modification time as a Unix timestamp in nanoseconds (0 for the root). |
|
| Creation time as a Unix timestamp in nanoseconds (0 for the root). |
|
| Inode identifier: shared by all hard links to the same file. |
|
| Hard-link count of the file's inode. |
id
Type
strDescription
Unique entry identifier. Hard links to the same file have distinct entry IDs.
name
Type
strDescription
The name of the entry, without its directory, in Unicode NFD form.
size
Type
intDescription
Size in bytes (0 for directories and symbolic links).
modified_at
Type
intDescription
Last modification time as a Unix timestamp in nanoseconds (0 for the root).
created_at
Type
intDescription
Creation time as a Unix timestamp in nanoseconds (0 for the root).
inode_id
Type
strDescription
Inode identifier: shared by all hard links to the same file.
num_hard_links
Type
intDescription
Hard-link count of the file's inode.
is_dir: bool
Whether this entry is a directory.
is_file: bool
Whether this entry is a regular file.
is_symlink: bool
Whether this entry is a symbolic link.
lucidlink.FsUtils
FsUtils(fs: FileSystem)Parameter | Description |
|---|---|
| The FileSystem to operate on, usually filespace.file_system. |
fs
Description
The FileSystem to operate on, usually filespace.file_system.
Whole-file reads and writes and the deletion of a directory tree on a FileSystem.
Its calls raise the errors common to every FileSystem call, the lifecycle errors of its Filespace and Client included.
Raises
Exception | Condition |
|---|---|
| If fs is not a FileSystem. |
TypeError
Condition
If fs is not a FileSystem.
FsUtils example
from lucidlink import FsUtilsutils = FsUtils(filespace.file_system)filespace.file_system.create_dir("/reports")utils.write_bytes("/reports/hello.txt", b"world")print(utils.read_bytes("/reports/hello.txt")) # b'world'utils.delete_tree("/reports") # the directory and everything in it
delete_tree(path: str) -> None
Delete a directory together with everything below it.
The symbolic links below it are deleted themselves; nothing they point to is touched. Not atomic: when the deletion fails partway, for example on a file another client holds a lock on, the entries deleted so far stay deleted.
Parameters
Parameter | Description |
|---|---|
| Path of the directory to delete. |
path
Description
Path of the directory to delete.
Raises
Exception | Condition |
|---|---|
If path does not exist. | |
If path is not a directory, a symbolic link included. |
read_bytes(path: str) -> bytes
Read the whole content of a file.
Shorthand for FileSystem.open(path, followed by read().
Parameters
Parameter | Description |
|---|---|
| Path of the file. |
path
Description
Path of the file.
Returns
The content of the file.
Raises
Exception | Condition |
|---|---|
If path does not exist. | |
If path ends in / and is not a directory. | |
If path is a directory. | |
If path is a symbolic link (errno.ELOOP). |
write_bytes(path: str, data: Buffer) -> None
Write data to a file, creating it if needed and replacing any previous content.
Not atomic: the file is emptied first, then written, so a write that fails leaves the file empty or partly written. The parent directory must already exist.
Parameters
Parameter | Description |
|---|---|
| Path of the file. |
| Bytes-like object to write (bytes, bytearray, memoryview, ...). |
path
Description
Path of the file.
data
Description
Bytes-like object to write (bytes, bytearray, memoryview, ...).
Raises
Exception | Condition |
|---|---|
If the parent directory does not exist. | |
If the parent is a file. | |
If path is a directory, or ends in /. | |
If path is a symbolic link (errno.ELOOP). | |
| If data is not a bytes-like object, a str for example. |
TypeError
Condition
If data is not a bytes-like object, a str for example.