Summary:
This test has been extremely flaky in the last few weeks, causing builds to not
being released, or just sandcastle jobs failing randomly. The reason for the
flakyness was due to the mtime stored in the dirstate being -1 instead of the
real mtime of the file.
The reason for the mtime to be -1 is to fix a race where a file is modified in
the same second the dirstate is updated. This can happen when a transaction
spans less than a second. By forcing a transaction to always last at least a
second, we can avoid this issue and make the test stable.
Reviewed By: quark-zju
Differential Revision: D14141924
fbshipit-source-id: ce151cd65d930f8c745633f7575ea006748af9b9
Summary: I am making this a new diff because the I am documenting failures which only make sense when validation was already added.
Differential Revision: D14097490
fbshipit-source-id: 7eb2c22ebbbc4365f8203d68bc29134176f326f6
Summary:
When introducing `Component` we said that it will not have any slashes (`/`). This diff adds enforcement for that along with enforcing that `Component` is not empty.
`Path` and by extension `Component` also have a set of invalid bytes: `\0`, `\1` and `\n`. I got these checks from the mononoke codebase (mononoke-types/path.rs)
Reviewed By: DurhamG
Differential Revision: D14001106
fbshipit-source-id: 5acbdf66214e54f1034fd89d90e88c3e904d7f9b
Summary:
`Component` and `ComponentBuf` are specializations of `RelativePath` and `RelativePathBuf` that do not have any separators: `/`. The main operation that is done on paths is iterating over its components. The components are generally directory names and occasionally file names.
For the path: `foo/bar/baz.txt` we have 3 components: `foo`, `bar` and `baz.txt`.
A lot of algorithms used in source control management operate on directories so having an abstraction for individual components is going to increase readability in the long run. A clear example for where we may want to use `ComponentBuf` is in the treemanifest logic where all indexing is done using components. The index in those cases must be able to own component. Writing it in terms of `RelativePathBuf` would probably be less readable that writing it in terms of `String`.
This diff also adds the `RelativePath::components` which acts as an iterator over `RelativePath` producing `Component`.
The name `Component` is inspired from `std::path::Component`.
Reviewed By: DurhamG
Differential Revision: D14001108
fbshipit-source-id: 30916d2b78fa89537fc5b30420b3b7c12d1f82c7
Summary:
`RelativePath` and `RelativePathBuf` are types for working with paths specialized for source control internals. They are akin to `str` and `String` in high level behavior. `RelativePath` is an unsized type wrapping a `str` so it can't be instantiated directly. `RelativePathBuf` represents the owned version of a `RelativePath` and wraps a `String`.
RelativePaths are going to be used in all contexts for SCM so flexible and efficient abstractions for dealing with common needs is important.
The inspiration for `RelativePath` and `RelativePathBuf` comes from the `std::path` crate however we know that the internal representation of a path is consistently an array of bytes where directories are delimited by `/` thus our types can have a simpler representation. It is because of the same reason that we can't use the abstractions in `std::path` for internal uses where we need to apply the same algorithm for blobs we get from the server across all systems.
This diff adds a new crate called `vfs`. The recommended use of the types in this crate are `use ...::vfs` followed by `vfs::RelativePath` and `vfs::RelativePathBuf`.
Alternatives for representing the path:
* We could use `String` and `&str` directly however these types are inexpressive and have few guarantees. Rust has a strong type system so we can leverage it to provide more safety.
* Mononoke use `PathElement(Vec<u8>)` and `MPath(Vec<PathElement>)`. One issue is that this type requires more allocations. Having read a buffer it is more difficult to provide views into it as &MRelativePath.
`RelativePath` is Dynamically Sized Type (DST). That means that `mem::transmute` is the practical way of constructing it in Rust right now:
* https://doc.rust-lang.org/nomicon/exotic-sizes.html
* https://github.com/rust-lang/rust/pull/39594#issuecomment-279471476
Basing these types on `String` and `&str` means that the paths that we deal with need to have UTF-8 encodings. In the grand scheme of things a lot of things are simplified when we can make this kind of assumption.
The code right now doesn't have much documentation. I'll add that if this direction makes sense to other people.
Reviewed By: DurhamG
Differential Revision: D14001109
fbshipit-source-id: 1ca399827b1284f32cd83219e7e090d8b2487cee
Summary:
A lot of code is duplicated between data stores, and history stores, and one
reason for it is the absence of common trait between these 2. By adding a new
Store trait it will make it easier to write generic code that works accross
data and history store.
Reviewed By: quark-zju
Differential Revision: D14091899
fbshipit-source-id: deef1d43a7d300cb3607c67554ad54f20c870e23
Summary:
Instead of manually implementing DataStore/HistoryStore for Box, Rc, Arc, and
future smart pointers, we can simply implement the trait for all the types that
can be deref into a DataStore/HistoryStore.
Reviewed By: quark-zju
Differential Revision: D14078072
fbshipit-source-id: 47a80ab0179b84aa08836b6e7c5c3c5f9c1a08ff
Summary:
As we are moving to smooth switching to Mononoke and backwards, we can start to deprecate
Mercurial specific optimizations to simplify the code.
Reviewed By: markbt
Differential Revision: D14131594
fbshipit-source-id: fa927011890ecdf0874a3a74b4910412b3c84b70
Summary:
Prepare the code to use 'known' wireprotocol command to check what is backed up
on the server instead of slower 'lookup'. We don't need to reestablish ssh
connection, and we can test all the nodes in one go.
Also test the 'known' request with Mononoke is working (all commands have been
tested with isbackedupnodes => isbackedupnodes2)
Reviewed By: markbt
Differential Revision: D14131383
fbshipit-source-id: 5a150b64d0e84a02357c2367879b2da8493d9167
Summary: Add the actual verification of the server-side repo after `unbundlereplay`
Reviewed By: StanislavGlebik
Differential Revision: D14131577
fbshipit-source-id: 18828479b07a110f26fd8adf108eb94a21d817c6
Summary:
Use the same way to check if the commits have been backed up in Mononoke and Mercurial.
The only common way to check is 'lookup' command because Mercurial doesn't support discovery for commit cloud commits, the command 'known' is also not supported.
Also we have to go one by one because lookup doesn't got any better API.
It is still much faster than backup commits that are already there.
Introduce such check for pushbackup as well.
Remove hacky way to check it from cloud sync.
For commit cloud in Mononoke we will have backfill, so the server side check will be heavily used when you go to Mononoke at the first time.
Unfortunately connection pool module in mercurial is not good enough in detecting closed connections and can easily return a broken connection on the next call.
Reviewed By: markbt
Differential Revision: D14085849
fbshipit-source-id: d76d9a71f9efdbdfec4de3198cd428b6b693418d
Summary:
This is to be used from Mononoke->hg sync.
Currently expects only `pushrebase` bundles, e.g. one head and one book to
move.
Reviewed By: StanislavGlebik
Differential Revision: D14116130
fbshipit-source-id: 959a6e51f51e21da5592c84188e294a33057ffaa
Summary: When comparing the number of files changed, fsmonitor was incorrectly using the length of the result dict (always 4), rather than the number of files watchman returned. Use the right list of files instead.
Reviewed By: markbt
Differential Revision: D14123604
fbshipit-source-id: 94684f1f189d045b2f6a880180b15e52ba9bba8c
Summary:
This subtle bug shows only when the watchman-changed-file-threshold is set to a
value higher than 4.
Reviewed By: markbt
Differential Revision: D14123605
fbshipit-source-id: 51ff3172fef5bc65fb9f68bb34826a9181fd64a9
Summary:
make local / relase are broken on trunk, fix it
in other modules, this code is only for tests. In toml file it is enabled for tests.
Reviewed By: mitrandir77, ikostia
Differential Revision: D14123392
fbshipit-source-id: eceaf17478b8b7e75ee2ba56df69b28ca8374c64
Summary:
this will be used to understand if server supports 'listkeyspatterns'
(Mononoke doesn't support this)
Reviewed By: DurhamG
Differential Revision: D14107742
fbshipit-source-id: c9a42e8516eb5660ab2f498996b211db7086bcb1
Summary: Move the contents of `edenapi-types` into the `types` crate so all of Mercurial's Rust types are in one place.
Reviewed By: quark-zju
Differential Revision: D14114547
fbshipit-source-id: feb8f9c35f102d30bf00b230df81a86a3893a49b
Summary:
In order to move the types in `edenapi-types` (containing types shared between Mercurial and Mononoke) to the `types` crate, we need to move a few types from the `revisionstore` crate into this crate first, because `revisionstore` depends on `types`, which would create a circular dependency since `edenapi-types` uses types from `revisionstore`.
In particular, this diff moves the `Key` and `NodeInfo` types into their own modules in the `types` crate.
Reviewed By: quark-zju
Differential Revision: D14114166
fbshipit-source-id: 8f9e78d610425faec9dc89ecc9e450651d24177a
Summary:
For HTTP data fetching, it will be necessary to have the same Rust types in Mononoke and Mercurial, so that Mononoke can send down the serialized types and Mercurial can deserialize them. These types must live in the Mercurial codebase since Mercurial can't link to code outside of fbcode/scm/hg. As such, this diff adds a new crate to Mercurial that Mononoke can link to, containing these shared types.
Right now the only shared type is a `HistoryEntry`, designed to match the interface of `MutableDatapack::add`. This type will be used as part of the HTTP history fetching API.
In the longer term, it would probably make sense to use something like Thrift for defining the on-the-wire formats used between Mercurial and Mononoke (and eventually for RPC as well). However, given that using Thrift from Mercurial is currently nontrivial (since Mercurial is typically built with Cargo and needs to be compatible with open source tooling), defining the schema in this crate and using `serde` for serialization and HTTP/2 for transport should be sufficient for now.
Reviewed By: quark-zju
Differential Revision: D14079337
fbshipit-source-id: c7880919aeb3fd7e1cf70067a89a17341c1d973f
Summary:
the early exit logic was incorrect, basically if there is at least 1 bookmark
in the repo and the backup state was not empty, it didn't catch that nothing has been changed.
bookmarks are dicts, so it is fine to compare them
if any bookmarks in the repo, everytime `hg pushbackup` established a connection to mercurial
Reviewed By: quark-zju
Differential Revision: D14103938
fbshipit-source-id: 0edc4d9e186199670765fd2362236330e831c7d9
Summary:
There are still users of "hg id -b", namely shell prompt scripts.
However, it seems fine to remove "branch" from the formatted (JSON or other
formats) output.
Reviewed By: ikostia
Differential Revision: D14076296
fbshipit-source-id: a77f4b9e73161d322da3f1bdec993082ac463939
Summary: Without named branch, it is not meaningful to close a branch.
Differential Revision: D14076295
fbshipit-source-id: 527ae457cb14f9415fa7269b7dc123252e7d6867
Summary:
Unfortunately, there are still users of the "hg branch" command so we cannot
remove it right now. But I'm updating internal tools to not call it.
Reviewed By: ikostia
Differential Revision: D14076300
fbshipit-source-id: a5322d9c9ce8148c6ad0800653bd829736556386
Summary:
It seems nobody uses it from dev_commands logging. Search for "allowbranches"
in all internal repos suggests nobody turned it on. The tests were also
migrated to not use the command so let's just remove it.
Reviewed By: ikostia
Differential Revision: D14076298
fbshipit-source-id: a1cb03b2da289c01b15f0580ffe77531941be213
Summary: This would make the test runner run tests with named branch disabled.
Reviewed By: phillco
Differential Revision: D14059835
fbshipit-source-id: 9fbad5688bd8c69d32016b2ec1f931102638da68
Summary:
Migrate away from using the "hg branch" command to "hg commit --extra branch="
instead. In the future the branch namespace would be removed, so we create
local tags instead.
`test-revset2` was split from `test-revset` and has the same header. Do the
same change to it. Besides, de-duplicate tests about the `tag()` revset.
Reviewed By: ikostia
Differential Revision: D14058435
fbshipit-source-id: b59fadc43939d85d14bbf9f81227c523b65557a0
Summary:
My goal is to have `mkscratch` get deployed with the mercurial
release on all platforms. This is currently deployed via packman
as `scratch` on devservers, and that packman build should continue to
succeed because the TARGETS file in here is left unchanged.
That will allow both `scratch` and `mkscratch` to coexist for a while
until we've made it through a few mercurial releases, and then we can
either fixup existing users of `scratch` to use `mkscratch`, or re-do
the `fb-scratch` rpm so that it symlinks to `mkscratch`.
I don't really know if this diff is sufficient to include the binary
in the mercurial package; I'm cargo culting the `scm_daemon` stuff
and that seems to be included in the omnibus, so maybe this is it?
Reviewed By: quark-zju
Differential Revision: D14060864
fbshipit-source-id: 5fd74a877437d84fe7676819c2cefb902fb7002d
Summary:
"st_size" returned by watchman is sometimes 0. We suspect it's incorrect.
It could be the file on disk is actually empty, or the kernel made it
look empty to watchman in the past, or there is something wrong with
serialization, etc. Let's first add a real os.stat to test if the file
is really empty.
Reviewed By: phillco
Differential Revision: D14090851
fbshipit-source-id: bd75cf86bfd117285658ad0dd7fb28d0e8b5585c
Summary: Otherwise mergedrivers become impossible to debug if they don't import correctly.
Reviewed By: ikostia
Differential Revision: D14071200
fbshipit-source-id: 3e677089e6b008d892158290c392e1f9d68a67ae
Summary:
Those types were internally using DataPack/HistoryPack, limiting their use. We
can make them more generic by using the DataStore/HistoryStore traits. The only
drawback is having to implement the new method for each store type.
Ideally, we could have a trait StoreFromPath (or use the experimental TryFrom)
that all the datastore/historystore types would implement.
As a bonus change, I got rid of the *Builder type, these were required as the
new method was already implemented in the AsyncHistoryStore/AsyncDataStore. We
can simply rename the later and use a new method elsewhere.
Reviewed By: DurhamG
Differential Revision: D14060159
fbshipit-source-id: 31fa278f650ba979eecd3df4175cbac30ebb8180
Summary:
Starting the implementation of tree manifest with the in memory nodes and implementing `get` and `insert`. The in memory nodes are called `Ephemeral` and the stored immutable nodes are going to be called `Durable`.
Using a `BTreeMap` for storing the children because we want to efficiently insert, fetch and remove path components. We also want iteration to be done in ordered fashion so BTreeMap is our collection in this case.
Removing elements from the tree is going to be implemented in a future update.
Reviewed By: DurhamG
Differential Revision: D14016273
fbshipit-source-id: d3bc22e5ddb21b689d07a7d74bd639b8c2b138ce
Summary:
The seed for the rust implementation of manifests.
We start with the most primitive API for manifests and maps a paths to a `Node`. At the basic level we need the same operations that a map implements so we start with `insert`, `get` and `remove`. We know that retrieving data for Manifests can fail so we encode that in our interface using `Fallible`.
I let for future iterations requiring iterator or returning manifest flags.
Reviewed By: DurhamG
Differential Revision: D14016274
fbshipit-source-id: 8f1f83610933b9e9a96f8c5ba2c6e50567c76e06
Summary:
`Node` is not friendly with plain old unit tests because constructing them is a bit involved. This diff adds a constructor from u8 purely for test puposes.
I picked an u8 for input because it is the most convenient type. When we move past rust 1.31 it might make sense to use an u32 and use https://doc.rust-lang.org/std/primitive.u32.html#method.to_le_bytes
To note that property testing is best used in addition to plain old unit testing.
Reviewed By: DurhamG
Differential Revision: D14016272
fbshipit-source-id: 5b831ab0011ef2575f7e94d158ab4ddf30d1ac06
Summary:
During repack, the repacked files are deleted without any verification. Since
Adam saw some data loss, it's possible that somehow repack didn't fully repack
a packfile but it was deleted. Let's verify that the entire packfile was
repacked before deleting it.
Since repack is mostly a background operation, we don't have a way to notify
the user, but we can log the error to a scuba table to analyse further.
Reviewed By: DurhamG
Differential Revision: D14069766
fbshipit-source-id: 4358a87deeb9732eec1afdfb742e8d81db41cd87
Summary:
Removing files on Windows is hard. It can fail for many reasons, many of which
involves another process having the file opened in some way. One way to solve
this problem is that renaming the file isn't as restrictive as removing it.
Since hg repack will attempt removing any temporary files it will also try to
remove the packfiles that we failed to remove earlier.
Reviewed By: DurhamG
Differential Revision: D14030445
fbshipit-source-id: 1f3799e021c2e0451943a1d5bd4cd25ed608ffb6
Summary:
Packfiles are named based on their content, so having an on-disk file with the
same name means that they have the same content. If that happens, let's simply
continue without failing.
Reviewed By: DurhamG
Differential Revision: D14030446
fbshipit-source-id: f04c15507c89b2fca19c95a7b41d8e65c88da019
Summary:
Preloading all the pack files on initialization ties the lifetime of the
packfiles to the repo. For normal operations, this is fine, as packfiles are
mostly read. During a repack however, we need to be able to remove them, and
while having an open file handle allows deletion on unix OSes, it prevents it
on Windows.
The Rust repack now succeeds on Windows.
Reviewed By: DurhamG
Differential Revision: D14013786
fbshipit-source-id: 99279d4af67a0dfe8679159e9409186f56a09296
Summary:
Since september (D9840431) rcutil.rccomponents is no longer source of truth
about which config files we load. In fact we use it only in the
`hg config --debug`. This leads to situations where the debug command mentions
loading a specific config file and then ignores it completely which is very
confusing.
Let's remove it.
Reviewed By: suitingtseng
Differential Revision: D14083220
fbshipit-source-id: 362fd9bf574e24639f99a1203206184da42d1e24
Summary:
When using query strings or fragments in an URL we should treat repository paths
as the same repo. This allows servers to use query strings for metadata, without
treating the urls as different servers. By introducing normalization in our grouping,
we will consider the normalized result to be the qualifier for what is the same repo,
rather then the full absolute path - which includes query strings and fragments.
Reviewed By: DurhamG
Differential Revision: D14051479
fbshipit-source-id: c82fe041467f6bd6af210688c0be873e2da93781
Summary: We're using query strings to provide metadata to the server, but they should be stripped before they actually reach the core Mercurial handling. Let's update the tests to simulate this behavior.
Reviewed By: DurhamG
Differential Revision: D14051480
fbshipit-source-id: a9b448fd193f17aeab93c4886efdec3fda533c4d
