README
¶
fss
Tinkering with new fs.FS
Documentation
¶
Overview ¶
Naming the object is a problem. It stores an in-memmory copy of a small filesystem but it does not provide the API of io/fs.FS .
It provides tree structure for a collection of fileutils/FSItem's and it records this structure in multiple, redundant ways (enabling a certain level of error checking).
So in a name for the object, FSItem is unnecessary because a name involving Tree implicitly has nodes for directories.
So, not FSItemFS. FSITree? FSIBundle? FileTree? FileMemTree? MemFileTree?
Package fsutils provides a MemFileTree, which is summarised as:
- Created from a filesystem path
- Created using an os/Root, which is saved but unexported
- Each node contains a fileutils/FSItem
- Tree structure is provided for each node by an orderednodes/Nord
- Items can be selectively omitted from the tree at its creation time
- Note that the "next step up" from this is an mcm/ContentityFS
- Note that it is not necessary to implement io/fs.FS (and related), which do not provide or need hierarchical structure
- Note that in principle we _could_ provide funcs & methods associated with io/fs.FS (and related), like `func Open(path string) (File, error)`; however the return value File would have to return the file on-disk rather than anything in-memory, and this would create all kinds of sync problems; Open and other calls could of course be shadowed / overridden), but this would be overambitious, buggy, and unnecessary
A similar technique will be used to create an improved version of mcm/ContentityFS (and maybe rename it to Contentitree).
MemFileTree will provide read-write capability for individual FSItem's, but adding and deleting entire FSItems by name may prove very difficult, depending on how orderednode/Nord implements tree structure.
In the interest of simplicity and composibility, a new MemFileTree can be created stepwise, like so:
- Walk the os.Root to gather a slice of simple strings of filepaths
- Use that slice to build a slice of (ptrs to) fileutils/FSItems's
- Use the information gathered, and input arguments, to filter out entries
- (Optional) Provide user interactivity for filtering out additional entries
- (If needed) Compress the slice, by removing nil entries
- Provide the hierarchical/tree structure, by "weaving" the slice together (linking parents and children, probably using more than one method as implemented by orderednodes/Nord), and provide other means of access, such as a map from filepaths
fileutils/FSItem implements four interfaces:
- io/fs.FileInfo
- io/fs.DirEntry
- [fileutils.Errer] (via an embed)
- [stringutils.Stringser] (Echo, Infos, Debug)
Notes about the usage of os/Root:
- MemFileTree will store provate copies of the os.Root and of the result of `func (r *os.Root) FS() io/fs.FS`
- The instances of os/Root and io/fs.FS will have to be unexported
- Note that altho we will be able to filter out entries, the technique used in FilteringFS will not be helpful here, because we do not want to return the actual on-disk file, because this would create huge problems with content sync, because we will be storing the file (and manipulating it) in-memory
- The embedded io/fs.FS implements the following interfaces, which will be available to MemFileTree itself but not to users of MemFileTree:
- io/fs.StatFS : Stat(name string) (FileInfo, error) // *PathError (if it is a link, returns the file it links to)
- io/fs.ReadFileFS : ReadFile(name string) ([]byte, error)
- io/fs.ReadDirFS : ReadDir(name string) ([]DirEntry, error) // sorted
- io/fs.ReadLinkFS : ReadLink(name string) (string, error) // *PathError
- io/fs.ReadLinkFS : Lstat(name string) (FileInfo, error) // *PathError (if it is a link, returns the link, does not try to follow the link)
https://benjamincongdon.me/blog/2021/01/21/A-Tour-of-Go-116s-iofs-package/
The Go library allows for more complex behavior by providing other file- system interfaces that can be composed on top of the base fs.FS interface, such as ReadDirFS, which allows you to list the contents of a directory:
type ReadDirFS interface {
FS
ReadDir(name string) ([]DirEntry, error)
}
The FS.Open function returns the new fs.File “ReadStatCloser” interface type, which gives you access to some common file functions:
type File interface {
Stat() (FileInfo, error)
Read([]byte) (int, error)
Close() error
}
However, one big caveat: conspicuously absent from the fs.File interface is any ability to write files. The fs package is a R/O interface for filesystems.
https://lobste.rs/s/kixqgi/tour_go_1_16_s_io_fs_package
fstest.TestFS does more than just assert that a few files exist. It walks the entire file tree in the file system you give it, checking that all the various methods it can find are well-behaved and diagnosing a bunch of common mistakes that file system implementers might make. For example it opens every file it can find and checks that Read+Seek and ReadAt give consistent results. And lots more. So if you write your own FS implementation, one good test you should write is a test that constructs an instance of the new FS and then passes it to fstest.TestFS for inspection.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func WalkFSforFilepathSlice ¶
WalkFSforFilepathSlice takes an fs.FS (assumed to be valid) and returns a (filtered!) slice of strings that are filepaths.
About the results:
- in the returned slice, a filepath that is a directory or symlink will always end with a slash (not with os.Separator)
- pass an os.Root(path).FS() to avoid security issues with symlinks
- results of the walk are returned WITH filtering (using defaults for the m5 app), therefore omitting (e.g.) any path element starting with ".git"
TODO: Describe results for abs v rel
Maybe the path argument should be an absolute filepath, because a relative filepath might cause problems. Altho this is the opposite of the advice for lower-level items.
os.Root has not been exercised, but note tho that when using os.DirFS, it appeared to make no difference whether path
- is relative or absolute
- ends with a trailing slash or not
- is a directory or a symlink to a directory
.
Types ¶
type FileNordFSer ¶
type FileNordFSer interface {
NordFSer
// InputFS is undefined for a TagTree.
InputFS() fs.FS
// DirCount returns zero if the FS is a TagTree.
DirCount() int
// FileCount returns zero if the FS is a TagTree.
FileCount() int
// Interface ReadFileFS is a file system providing a custom ReadFile(path).
// ReadFile reads the named file and returns its contents.
// A successful call returns a nil error, not io.EOF.
ReadFile(path string) ([]byte, error)
// Interface StatFS is a file system with a Stat method.
// Stat returns a FileInfo describing the file.
// Any error should be type *PathError.
Stat(path string) (fs.FileInfo, error)
// Interface ReadDirFS is a file system with a custom ReadDir(path).
// ReadDir reads the named directory and returns a
// list of directory entries sorted by filename.
ReadDir(path string) ([]fs.DirEntry, error)
}
FileNordFSer is implemented by all types that assemble a tree of Nords and embed an fs.FS . For working with tags (or anything else that is not actual files and directories), use NordFSer instead. The godoc for this interface describes methods unique to FileNordFSer.
A FileTree NordFSer is a collection of files & directories - either (1) hierarchically gathered & organized (when a directory tree is walked, like in a fs.DirTreeFS), or (b) listed individually (like in a DITA map or a MapFS or a list of materialized paths). The file or directory name is the last path element of either of (but maybe both) the absolute path and/or the relative path.
Note that fs.File is an interface that provides just three methods: Stat() (FileInfo, error) ; Read([]byte) (int, error) ; Close() error
type FileTreeNode ¶
FileTreeNode is not named "..Nord" because we don't need or use the ordering functionality of the embedded Nord.
func NewFileTreeNode ¶
func NewFileTreeNode(absPath string) *FileTreeNode
type MemFileTree ¶
type MemFileTree struct {
*os.Root
RootPaths *FU.Filepaths
RootNode *FileTreeNode // but.. DRY ?!
AsSlice []*FileTreeNode // [0] pts to RootNode
AsMapOfAbsFP map[string]*FileTreeNode
AsMapOfRelFP map[string]*FileTreeNode
*FU.FSItemSummaryStats
}
func NewMemFileTree ¶
func NewMemFileTree(aPath string, okayFilexts []string) (*MemFileTree, error)
NewMemFileTree proceeds as follows:
- Walk the FS of a new os.Root to get a slice of filepath strings
- Use that slice to build a slice of (ptrs to) FileTreeNode (via ptrs to *fileutils/FSItem)
- Provide the hierarchical/tree structure, by "weaving" the slice together (i.e. linking parents and children, probably using more than one method as implemented by orderednodes/Nord), and provide other means of access, such as a map from filepaths
TBD: Maybe the path argument should be an absolute filepath, because a relative filepath might cause problems. Altho this is the opposite of the advice for lower-level items.
It isn't yet clear precisely how to use os.Root. Note tho that when we used os.DirFS, it appeared to make no difference whether path
- is relative or absolute
- ends with a trailing slash or not
- is a directory or a symlink to a directory
The only error returns for this func are:
- a bad path, rejected by func FU.NewFilepaths
- the path is not a directory (altho it can be a symlink to a directory ?)
- filepath-specific errors are in interface [fileutils.Errer] and counted up in [MemFileTree.FileSummaryStats]
MemFileTree does not embed Errer and cannot itself return an error.
TODO: Maybe it needs two boolean arguments:
- One to say whether to be stricter about security using funcs io/fs.ValidPath and path/filepath.IsLocal, and
- One to say whether to follow symlinks (i.e. symlinks that are legal by having targets under the Root)
- These two flags might have some interesting interactions
.
type NordFSer ¶
type NordFSer interface {
// Interface fs.FS is the minimum required of an fs file system.
// The Open(path) method is its only method. It opens the named
// file. An error should be of type *PathError with the Op "open",
// Path set to the path, and Err describing the problem.
// Open should refuse to open a path that fails ValidPath(path),
// returning a *PathError with Err set to ErrInvalid or ErrNotExist.
Open(path string) (fs.File, error)
// Size returns (for a FileTree) the combined number of files and dirs,
// or (for a TagTree) the total number of tags.
Size() int
// RootAbsPath can return the the abs.FP of the markup file
// if the FS is a TagTree.
RootAbsPath() string
// AllPaths can be either all absolute paths or all relative paths.
AllPaths() []string
RootNord() ON.Norder
AsSlice() []ON.Norder
AsMap() map[string]ON.Norder
}
NordFSer is implemented by all types that assemble a tree of Nords. NOTE: This godoc is clearly out of date.
NOTE: For working with actual files and directories, use the superset FileNordFSer instead. The godoc for this interface describes methods common to both.
NordFSer is what an mcfile.Contentity (including directories) should implement: Open, Readfile, Stat, and ReadDir. Then a Contentity can be treated like an fs.File, and the latter three methods can be delegated to by the FS itself.
Tipicly a NordFSer is a tree of tags (e.g. an AST) parsed from an XML file. The tag name is the last tag element of the abs.path and/or the rel.path, but possibly the relative path is just the tag name. Furthermore, calling Open (the only method specified in the fs.FS interface) returns the tag's body (from opening tag to closing tag, inclusive) of that tag only, without the context of the larger tag tree.
Source Files
Directories