mgree/ffs
1
1
Fork 0
mirror of https://github.com/mgree/ffs.git synced 2024-08-17 15:20:43 +03:00
Code Issues Projects Releases Wiki Activity
3857d74d27
ffs/TODO.md
Dan Liu 3857d74d27
pack/unpack (#65) 
* scoping out multi-binary stuff

* todo notes for pack/unpack [ci skip]

* new version of r in gh action

* created lib.rs and moved ffs main.rs bin. added bfs traversal of json structure.

* moved main.rs last commit. forgot to commit the removal

* added file creation for null,bool,num,str

* main.rs > ffs.rs to have ffs as executable

* error condition for already existing root dir and text files

* first cut at BFS with nodes

* unpack supports all 3 types now

* use config object for file name parsing, change from_args to from_ffs_args in ffs.rs

* fixed unpack configs

* added from_pack_args and some code to get the Value structure from a file

* added dep walkdir

* not working pack.rs

* remove dep walkdir

* almost working pack.rs

* pack works, but not too efficient

* change pack to return name and value so recursion can work without queue

* pack.rs similar to as_other_value now and works

* unpack and pack now have their own args parsers

* changed cli options for pack-unpack (unfinished)

* edited config to better support pack and unpack actions

* fixed unpacking into empty dir. only unpacking into unempty dir errors

* checked almost all error statuses

* fix no-xattr option for unpack

* prevent non-object/array from being unpacked

* fixed missing RUST_LOG warnings

* roundtrip tests for formats. since comments and formatting aren't preserved, unpack and pack twice

* fix pack: added --exact

* rm ERR_MSG tmp file in all tests. added all possible format conversion tests. edit to run_tests.sh to support new tests

* fix packunpack tests

* another update to test scripts

* unpack: don't remove dir if value is not map or list. pack: detect file type without xattr. started adapting more scripts to use unpack/pack

* changed user.type xattr for non-lists from 'map' to 'named'. converted 10 scripts

* added more scripts. fixed bug in infer_mount_relative fail()

* more tests, fix: original_name xattr only gets used if name is invalid

* more tests, missing 1 fail message in yaml_output test, changed pack list sorting to use file name instead of parsed integers to match ffs

* fixed tests for unpack/pack

* edit script formatting using quotes around vars, rm readonly for unpack

* pack: added back ignored file for lists and --no-xattr option in cli, added macos_noxattr_cleanup (not exactly the same as ffs test)

* Run `pack/`unpack` in macOS CI; factor out benchmarks (#63)

* added quiet inplace, umask test based on mode.sh

* added fail (un)pack for every call, fix missing n in fail msg for basic_object_exact

* fixed issues with adding fail conditions

* added (un)pack exit status tests

* added symlink support for pack

* symlinks mostly done. cleanup + efficiency checks needed

* added test for packing symlinks, added some comments for pack

* fix test4 for symlink test on linux

* see why test5 not working on linux

* actually print out the xattr

* fix: setting xattr on symlink doesn't work in linux :( making it macOS-specific

* impl --max-depth and --allow-symlink-escape. tests needed

* fix: macos links /var to /private/var so checking if symlinked path starts with mount errors. canonicalizing mount works.

* fix: wrong detected type map instead of named. add: symlink escape and maxdepth test.

* code cleanup, added test for symlink escape and maxdepth together, show warnings in config for pack/unpack not just errors

* simple changes for requests:
pack:122 add loop to error msg
debug! for received config in unpack & pack
f.write(s)? instead of write!
shadowed original_name
verb agreement in cli.rs
remove reserve for BTreeMap and now useless TODOs

* while let instead of queue.empty

* use auto instead of detect and check for auto and is_dir to resolve directory type
use .as_ref instead of .clone for accessing mount
remove non-symlinks from mapping

* resolve directory type always. warn for unknown path_type.

* warn when hitting broken symlinks.

* better warn message for broken symlink

* resolve repeated traversal of broken symlinks
store bool of whether link is broken in symlink mapping
checks symlinks a maximum of two times for broken links

* use struct instead of tuple in symlink map
for better naming and code clarity

* loosen criterion for determining directory type
directories get resolved as list if all files begin with an integer.
if a directory's user.type gets forcibly set to list without obeying
that property, all filenames that don't begin with an integer get put
to the end of the list, but are still sorted alphabetically.

* don't use regex to detect. just check first or first two chars manually.
continue to use regex for sorting because lexicographic sorting for files starting with -
means larger negative numbers go to the right

---------

Co-authored-by: Michael Greenberg <michael.greenberg@stevens.edu>
2023-09-27 14:02:15 +00:00

3.2 KiB
Raw Blame History

binary data gets treated as base64

unpack

JSON, TOML, YAML file -> file system hierarchy

# puts the foo data into the bar directory (making bar if it doesn't exist)
cat foo.json | unpack --into bar

# puts the foo data into the foo directory (making foo if it doesn't exists)
unpack foo.json

# in both of those, it's an error if foo or bar exist and are non-empty

# unpack stdin (coming from baz) into quux, treating input as YAML
cat baz | unpack -s yaml --into quux

src/format.rs describes mappings from these formats into the Nodelike trait

a possible cut through the work:

  • get JSON to work by hand

    write some tests

  • get other formats work using Nodelike

    • wrinkle: YAML has a special notion of anchor that would be cool to treat as a sym- or hardlink problem not actually worth thinking about

      write some more tests

  • implement options

    --debug --exact --no-xattr --quiet --time --unpadded --munge --dirmode, --mode, --gid, --uid -s, --source # rename to -t, --type ? -m, --mount # rename to -i, --into ?

    write tests of unpack write separate tests that compare ffs and unpack's behavior diff -r might do the trick xattr/uid/gid/mtime/etc. stuff is a bit more subtle

things to think about

  • read semi-structured data

    • default to stdin
    • but take a file (many files?!)

    output is... at a default mountpoint, or at a directory based on the filename follow ffs lead here

  • options that matter

  • build the directory tree, write the data, set some xattrs as necessary, that's it

  • test

    follow the general lead of run_tests.sh and tests/*.sh

    how do we ensure that we don't hose the system?

    in docker? in chroot? with pivot_root?

pack

file system hierarchy -> JSON, TOML, YAML file

# save /etc into a JSON file
pack /etc >config.json

pack -o lib.yaml /usr/share/lib

# -t specifying target type
pack -t toml . >bar.toml

  • get it to work for just JSON

    • wrinkle: special file types (devices, FIFOs, etc.) what does tar do? gunzip unzip and one other to see what's standard

    • wrinkle: permissions pack -o everything.json / what does tar etc. do?

    • wrinkle: hard and symlinks

      hardlinks are just files... worst case we copy would be cool in YAML to have them be anchors

      symlinks can cause loops, can go outside of the root specified, etc. cf. cp, tar, find options cp -L to specify following symlinks, --nofollow but also: don't infinite loop PATH_MAX

      i think there are good rust libraries for filesystem traversal

  • get it to work for Nodelike

  • implement options that matter

    --debug --keep-macos-xattr --pretty --time --munge --exact --quite --target --output

testing wrt ffs

ffs and pack/unpack should behave as identically as possible we should explicitly test this on fixed and maybe also random inputs

fuzzing

generate random inputs and run unpack on them

generate random filesystems and run pack on them (or run pack on random points in the FS)

fuzz ffs itself?

performance

  • think about ramdisks

  • compare pack/unpack and ffs in a bunch of ways lol