Summary:
Updating the manifest implementation for remove with the intended API.
When I originally implemented remove I wasn't confident what was the
best way to implement remove. As I've gained more experience, I feel
confident that doing two iteration over the tree is a good approach
for this method. The first iteration should validate that the file
exists then the second iteration will actually traverse down updating
the nodes to mutable ephemerals.
Reviewed By: quark-zju
Differential Revision: D16775353
fbshipit-source-id: 8ebee9ca347efcb694a6d27c1eeae2c149643766
Summary:
`get_link` started as test function that broaden in scope but did not have
it's behavior updated as it started to be used more broadly.
No reason to error out when we request a path that has parent files in the
manifest.
Reviewed By: quark-zju
Differential Revision: D16775356
fbshipit-source-id: a320926100378f16d723ca204746906e79c7752e
Summary: Matching more of the existing API.
Reviewed By: quark-zju
Differential Revision: D16607233
fbshipit-source-id: 7a71f22089067ecfccbfcb2ad072fbf21e360439
Summary:
This allows us to say that the queried path was a directory and simplifies
the API a little bit.
The Python/C++ Manifest API provide inspection of directories through several
functions. We want the Rust manifest to provide the same functionality.
This approach looks to have more consistency as the API evolves.
`FsNode` is a structure that is forward looking. We will want to add an API
that lists a directory and an interator of FsNode fits that well.
I also felt the need for it over the development of the manifest code.
Reviewed By: quark-zju
Differential Revision: D16609253
fbshipit-source-id: f826d7b21e3001f4bef43a35b9d1a9bc5a59eda9
Summary:
Prints out a tree that can make it easy to determine the data
that is in memory.
Reviewed By: quark-zju
Differential Revision: D16571839
fbshipit-source-id: dee8a0c6564853d49a72fa29bd53c6b09a7f3ddf
Summary:
There is no good reason to return a reference for FileMetadata. It is a relatively
small object that implements the `Copy` trait so returning a copy is
approapriate.
Reviewed By: quark-zju
Differential Revision: D16571838
fbshipit-source-id: 0c315c9f405e425832d39da5c67809dd15b4ab5e
Summary:
I wasn't sure how to test this. I implemented `diff` then got it to work
with `hg show` by adding the missing methods.
Reviewed By: quark-zju
Differential Revision: D16497354
fbshipit-source-id: 727979ad8ce4a4615e85ea96c3fe6413aa20b267
Summary:
Diffs can be filtered by a matcher to narrow down the number of files returned.
Adding this capability to the rust tree manifest implementation.
Reviewed By: kulshrax
Differential Revision: D16497355
fbshipit-source-id: fee07112b5bcff63c7c4115f28dade79f41fe6bc
Summary:
Most operations do not work on iterate on all the files of the repository.
Most operations filter the data set in some way. In many cases this filtering
is to a set of files and in some cases using patterns or subdirectories.
The Python code uses `match.py` to represent this filtering. The parallel
data structure in the rust code is `pathmatcher::Matcher`.
This diff adds `files` integration with `pathmatcher::Matcher`.
Reviewed By: quark-zju
Differential Revision: D16352527
fbshipit-source-id: 8b61ac7399f581773bf61ff648634cbc6e1a27b6
Summary:
Title. Small clean up. I think that this makes sense because the code in
file.rs is self contained. It provides a distraction free environment for
adding methods to the structures in there.
Reviewed By: quark-zju
Differential Revision: D16352531
fbshipit-source-id: c23e943198e0a4b50aa00c75e67b13bc4c3ee976
Summary: Bindings so that the rust manifest code can be used in Python.
Reviewed By: quark-zju
Differential Revision: D16352532
fbshipit-source-id: 34d4522f5e084f531f31bcd21770950f15f2fe13
Summary:
This diff sets two Rust lints to warn in fbcode:
```
[rust]
warn_lints = bare_trait_objects, ellipsis_inclusive_range_patterns
```
and fixes occurrences of those warnings within common/rust, hg, and mononoke.
Both of these lints are set to warn by default starting with rustc 1.37. Enabling them early avoids writing even more new code that needs to be fixed when we pull in 1.37 in six weeks.
Upstream tracking issue: https://github.com/rust-lang/rust/issues/54910
Reviewed By: Imxset21
Differential Revision: D16200291
fbshipit-source-id: aca11a7a944e9fa95f94e226b52f6f053b97ec74
Summary: A recent update to the crates in tp2 bumped the `once_cell` crate to version 0.2.0. This broke the build because the `Cargo.toml` for the `manifest` crate specified version 0.1.8. Apparently just changing the crate version to 0.2.0 fixes the build, so we weren't affected by whatever breaking changes were made to the crate.
Reviewed By: DurhamG
Differential Revision: D15492142
fbshipit-source-id: 552b0a751ab7c2aab5f0fbcb1124de4ea427790c
Summary:
The old code did not provide enough information to start debugging
a problem in the serialization code.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D15380388
fbshipit-source-id: 9da51cfe4d735de3961e840bd9cd2a1595131cd9
Summary:
Flush encapsulates the closed abstraction approach.
Finalize addresses the implementations that need to handle
history.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D15290173
fbshipit-source-id: 2cfd7a3f7ca0de1c2566ce3ae680d5b1451e4f91
Summary:
The plan is to have finalize do the complicated history managing stuff and
have flush be a light weight content only hashing approach to storage.
Flush will be used in tests in the short term.
Finalize would be used in Python to write to Revlogs and handle linknodes.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D15290171
fbshipit-source-id: 54ed5c791254507fd9f2e874315284b70bc6b597
Summary:
We want the Manifest share their durable nodes.
The path that I want to go is enable cloning for individual manifests.
What to do with the storage then? Share it in an `Arc`.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D15290170
fbshipit-source-id: 621d5d70700c4372c3b40a246fc45d9446b13ac5
Summary:
This enables more clear exporting so that it can be implemented
externally.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju, xavierd
Differential Revision: D15290172
fbshipit-source-id: 25541c88abab228a0083e5a27e74db31e6819230
Summary:
A tree manifest entry must always end with a line feed. It is somewhat
redundant but that's how the serialization is defined. Sometimes that last
line feed is missing in our data. I don't know why.
Reviewed By: quark-zju
Differential Revision: D15110860
fbshipit-source-id: c4ac5075e22a8b8851f6b246d22af8ab68f42a74
Summary:
This is a quality of life improvement for working with the storage layer.
We probably don't gain a whole lot by statically linking the store and it is
useful to have some flexibility in the storage layer.
Differential Revision: D15110859
fbshipit-source-id: 6102acafa21dd1dbaeed0f8fc3147538a8c301d1
Summary:
Building test objects can be tedious using various of our bottom bytes.
This diff addresses that issue by adding helper functions in a new module
in the types crate.
Handling this case could be improved in rust.
Differential Revision: D14660307
fbshipit-source-id: a866c1f3ede60ba1b87eb17d35817b8a8d7674a4
Summary:
This diff fixes the behavior of `skip_subtree` when called on a Leaf. The bug is
that the path is not correctly handled in this case. The name of the file
continues to stay in the path resulting in incorrect path names for all
subsequent calls to `path()`.
The high level perspective is that `skip_subtree` is a no-op in a Leaf node.
To fix, clarify the behavior and improve readability of the code we ad a new
state that handles poping elements from the path.
Durham noticed this bug when reviewing D14347655.
Reviewed By: quark-zju
Differential Revision: D14654557
fbshipit-source-id: 625278366e492a3048dddc44f9234a06d6928b7e
Summary:
quark-zju noticed in code review that `Cursor` could get into an infinite loop when
it's results would be collected into a Vec<_>. That was the motive that I
needed to update `Cursor` to transition to `State::Done` when the cursor
encounters an error. Previously I felt that users of `Cursor` would only be
empowered by having the ability to retry the failure.
Reviewed By: quark-zju
Differential Revision: D14393590
fbshipit-source-id: b3e0974ac15d62f3f17790229121c0dec3a6149e
Summary:
Follow up from D14178264.
Two changes:
* tree manifest entries must end with a line feed
* `t` is the byte that flags a directory
Reviewed By: DurhamG
Differential Revision: D14368316
fbshipit-source-id: b0b46c876649b8f25bf0ecdb1266527dbeb33796
Summary:
`manifest::tree::diff()` returns an iterator over the differences between two
tree manifests.
I chose a function that takes two parameters over a method on Tree because it
felt more clear to write `left` and `right`. Also because I am not sure how
iterators would be abstracted on a trait.
Differential Revision: D14347656
fbshipit-source-id: 537574070cd18b08c77b3cd1cf4cff38d77fbf81
Summary:
Cursor is a utility for iterating over a manifest tree. In this diff it is used
to implement Files. In the future it will be used to do a diff between two tree
manifests.
I am not sure how to describe an iterator return value in the Manifest trait so
I kept the function on the tree only for now. Looking forward to hearing your
suggestions.
Differential Revision: D14347655
fbshipit-source-id: ffd856443d8abe3ebd0557a096bf7a5ec46312d3
Summary:
`Node` is marked as `Copy`. `FileMetadata` is not much more than `Node` so it
seems pretty clear that it should be marked `Copy`.
Reviewed By: DurhamG
Differential Revision: D14347657
fbshipit-source-id: 939abf88087bc8c6f942047a08d6a4a0d61e053f
Summary:
Cleaning up the `mod.rs` file so that it provides more signal.
`Link` is an internal implementation detail that other internal components may depend on so it is a great candidate to be moved to a dedicated file.
Differential Revision: D14347654
fbshipit-source-id: e5b5a42faf1e9f9c4a0591e5bd94182391ed511f
Summary:
Save, finalize, flush, they mean about the same thing.
The first thing to note is that this implementation is not complete because
the parents are not correctly passed into the hashing function.
The second thing is that store failures make the code a little more complex
than it would have been otherwise.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D14292713
fbshipit-source-id: 807d7a385a62cb5f4948f1781d3146eaa6502ca9
Summary:
This function is a bit on it's own with the removal of the pair conversion.
Since it is used in only one place it makes sense to inline it.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D14292712
fbshipit-source-id: abbf1dc70d61c0ad039f5bc5ed5277d0770e3899
Summary:
Working on the save mechanism I realized that links_to_store_entry is not that
useful because we can avoid the failure states where we would try to serialize
an ephemeral node. I am removing that function and converting the code that was
using that function to using the Entry constructor directly.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D14292714
fbshipit-source-id: 54ef46670319c27d90fc78511a1eb6abf47d3acf
Summary:
This is what Rust is telling us to do. The situation that triggers this update is
writing to the store. Particularly when the store is an in memory hashmap we
need to have a mutable borrow to the hashmap to insert into it. From a general
point of view this means that any sharing of the store between different
instances of a manifest will have to be handled by the struct that implements
the `Store` trait.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D14292716
fbshipit-source-id: 6e789527dbdf3cd3ffe967f4900251bf31f7d6b2
Summary:
Removes a file from the manifest. Nothing special for it.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D14276645
fbshipit-source-id: 85e8ffd6cffee426c73eb627484dfa5a866a364b
Summary:
It is going to be useful in tests to check how certain internal nodes change
so adding an api that allows fetching an internal node.
(Note: this ignores all push blocking failures!)
Differential Revision: D14276642
fbshipit-source-id: 9a3e488be6031f7b4727a8643f64970dcec8c400
Summary:
This removes the need for the local buffer for the parent.
(Note: this ignores all push blocking failures!)
Differential Revision: D14276648
fbshipit-source-id: a9378ea592d502ddf2dcdc35fe6ffa9ba213bc14
Summary:
Using the recently added path utilities so that we don't keep a secondary
parent buffer around.
Updating the file insert logic so that it is readable and intuitive.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D14276649
fbshipit-source-id: 8e7e835814f0039645601abbf1b701e8c1ed3697
Summary:
We currently alias many of the imports from the store submodule. It is nicer
if we just import the submodule name and prefix with the submodule.
Differential Revision: D14276638
fbshipit-source-id: 6661df7f9cb5d976b11153003f653a1f66301c9a
Summary:
The order of imports we're trying to follow is: std, remote crates
(ie: crates.io), local crates (ie: fbcode crates), and crate local. A new line
in between each group can be used to prevent rustfmt from re-ordering them.
Reviewed By: singhsrb
Differential Revision: D14243162
fbshipit-source-id: 6fc2cceb3d6834b602be20b8b8f74e0f61b227e1
Summary:
This diff focuses on addding deserialization. Because the most effective way
of testing deserialization is doing round-trip conversions we also implement
serialization.
`manifest::tree::store::Entry` is the structure that is in charge of perfroming
serialization and deserialization. We update the Store trait to interface with
this new object.
Differential Revision: D14178264
fbshipit-source-id: bb12262c181a518ba4111d40c079d6836ec44301
Summary:
The `once_cell` crate is more flexible than `lazy-init`. It has more types,
a richer api, recent updates and more features.
Differential Revision: D14232727
fbshipit-source-id: 14aeb34a96e094069bb8dc3fb5efcf5b5707ce8c
Summary:
The purpose of `FileMetadata` is to be a data container. Using the
`FileMetadata` implies accessing the underlying values. That means that the
underlying values need to be `public`.
Differential Revision: D14178260
fbshipit-source-id: 8b8e5ead23d47962498f991152c2a5fd94ac3c74
Summary: These implementations are going to used in quickcheck tests.
Differential Revision: D14178259
fbshipit-source-id: 0bded67deab3422b4aad53666c14cf195ea1b0d4
Summary:
`Durable` nodes are inner nodes that come from storage. Their contents can be shared between multiple instances of Tree. They are lazily evaluated. Their children list will be read from storage only when it is accessed.
The inner structure of a durable link poses an interesting question related to handling failures: what do we do when we have a failure when reading from storage?
We can cache the failure or we don't cache it. Caching it is mostly fine if we had an error reading from local storage or when deserializing. It is not the best option if our storage is remote and we hit a network blip. On the other hand we would not want to always retry when there is a failure on remote storage, we'd want to have a least an exponential backoff on retries. Long story short is that caching the failure is a reasonable place to start from.
The lazy-init crate is useful for modeling the lazy initialization model that we have for durable node links. See docs at https://docs.rs/lazy-init/0.3.0/lazy_init/
Reviewed By: quark-zju
Differential Revision: D14142928
fbshipit-source-id: 077f708b38e2ace772f30b3392445326ce17f47c
Summary:
The store abstracts the data that we need from the store for TreeManifest. This is not a long term abstraction, mostly something that we can code against until we write the real parsing logic from the real data store.
Adding a TestStore implementation that we can use in tests and debug.
Reviewed By: quark-zju
Differential Revision: D14142930
fbshipit-source-id: 7f2d4f05a7b7e63758db9247cdbcd51541c88ec0
Summary: Preparing to add the store abstraction for the tree manifest. This store implementations are going to be tied to the tree implementation and should be made private from the rest of the crate. To do this we move the tree to a module.
Reviewed By: quark-zju
Differential Revision: D14142929
fbshipit-source-id: 588d597e1248fc2e632c9efe03f08ba3d491e8cd
Summary: manifest::tree::Link has several convenience functions. In the grand scheme of things this is something that we will want because they are common operations. The problems is that the node structure is currently changing rapidly and the additional layer of abstraction is hindering iteration. I don't know the exact shape that the nodes will have and trying different things out is easier when the Tree functions use the nodes directly.
Reviewed By: quark-zju
Differential Revision: D14142926
fbshipit-source-id: 5e4e8f4d3e1b19fd14dcc290a3dea4271b502d97
Summary: `RepoPath` and `PathComponent` were recently added as path abstractions for source control internals. This diff makes use of these abstractions in the manifest crate.
Reviewed By: quark-zju
Differential Revision: D14142925
fbshipit-source-id: 1e0033f4f2aeb5c1d63899adbce198302086bfda
Summary:
Starting the implementation of tree manifest with the in memory nodes and implementing `get` and `insert`. The in memory nodes are called `Ephemeral` and the stored immutable nodes are going to be called `Durable`.
Using a `BTreeMap` for storing the children because we want to efficiently insert, fetch and remove path components. We also want iteration to be done in ordered fashion so BTreeMap is our collection in this case.
Removing elements from the tree is going to be implemented in a future update.
Reviewed By: DurhamG
Differential Revision: D14016273
fbshipit-source-id: d3bc22e5ddb21b689d07a7d74bd639b8c2b138ce
