Summary:
The mutation store stores entries with a floating-point timestamp. This
pattern was copied from obsmarkers.
However, Mercurial uses integer timestamps in the commit metadata (the
parser supports floats for historical reasons, but only stores integer
timestamps). Mononoke also uses integer timestamps in its `DateTime`
type.
To keep things simple, switch to using integer timestamps for mutation
entries. Existing entries with floating point timestamps are truncated.
Add a new entry format version that encodes the timestamp as an integer.
For now, continue to generate the old version so that old clients can
read entries created by new clients.
Reviewed By: quark-zju
Differential Revision: D20444366
fbshipit-source-id: 4d6d9851aacb314abea19b87c9d0130c47fdf512
Summary:
Tracking the origin of mutation entries did not prove useful, and just creates
an un-necessary overhead. Remove the tracking and repurpose the field as a
version field.
Reviewed By: quark-zju
Differential Revision: D20444365
fbshipit-source-id: 65ff11ee8cfe77d5e67a83d03a510541d58ef69b
Summary: This diff bumps the open call from FUSE to High priority (which is higher than any other blob open request atm). This has shown improvement on the user experience of EdenFS when it's importing many other things from other channels (thrift, etc.)
Reviewed By: chadaustin
Differential Revision: D20287389
fbshipit-source-id: 319bc44ef8be5c904d7cf0db7cc2f8be28b4760a
Summary:
Replacing `HgBackingStore` with `HgQueuedBackingStore`.
This does not really improve anything except that all the caller of `getBlob`/`getTree` on this store is generating a `HgImportRequest` and getting a `SemiFuture` from the Promise associated with the request back.
Reviewed By: chadaustin
Differential Revision: D19184822
fbshipit-source-id: a8aef6b0d7392a6c407d311e8e1982754e736e9f
Summary: This diff implements `HgQueuedBackingStore` that uses `HgImportRequestQueue` to provide SCM data importing with priorities. This can allow us to customize how we import things and batch importing.
Reviewed By: chadaustin
Differential Revision: D19184826
fbshipit-source-id: da579b5bbff0b1449e9689e2c0159d4a3a475a83
Summary: This diff adds `HgImportRequestQueue` that is responsible for managing incoming requests by their priorities. This queue is later used in the `HgQueuedBackingStore` to prioritize works to the workers of the backing store.
Reviewed By: chadaustin
Differential Revision: D20197069
fbshipit-source-id: 246bbc086054a8021226e9ba6ab26d3bf0cfb7a3
Summary:
This class is used to represent an import request that will be used later in the queue implementation.
When the EdenFS needs to import a blob, it creates an instance of this request and send it to the worker. Then it waits for the promise associated with the request.
In the future, we should be able to change the owned `Promise` into a non-owned `SemiFuture` to a `Promise` somewhere else for merging repetitive import requests.
Reviewed By: chadaustin
Differential Revision: D19184824
fbshipit-source-id: 823aabbed1156acf6306b7aefc76580a540d310d
Summary: This diff adds `Priority` added in the previous diff to the `BackingStore` interface with the default value set to `Priority::Normal`.
Reviewed By: chadaustin
Differential Revision: D20197071
fbshipit-source-id: a92f1b49bb82e3478042e5e3b79b047d834755ea
Summary:
This diff introduces a `Priority` type for EdenFS. This type is used to pass along the priority of a request.
The priority class itself contains two parts, `kind` and `offset`. `kind` uses the first 4-bytes and the reset 12-bytes are used to store offset. The idea is that we can roughly assign a priority kind to most of the requests and offset is used to dynamically tweak the priority of some particular requests. For example, when we saw a process is generate millions of requests we can use this to express "normal priority but less important than other process's normal priority".
Reviewed By: chadaustin
Differential Revision: D20287652
fbshipit-source-id: 9a849fb6cc6ba5e443fea978d5b4dc3ab8ca906a
Summary:
D20444137 added a new use of `lfs_threshold`, and D20441264 removed this
variable. These two diffs landed close to the same time without ever being
tested with both diffs together.
Reviewed By: StanislavGlebik
Differential Revision: D20484843
fbshipit-source-id: fd0f0837142cdb641892005a64fd14272da7d2b7
Summary:
We want to delete all the non-treestate dirstate implementations. Let's
start throwing an exception if treestate is not enabled. We temporarily have a
bypass in case we break an important usecase in the process.
This also sets the standard new repo to be created in treestate mode, but adding
treestate to newreporequirements.
This was landed once as D19204621 but was backed out because eden backing repos
were using the odl formats and hadn't been upgraded. We fixed that, and now the
data shows ~10 people still using repos in this condition
(https://fburl.com/scuba/dev_command_timers/zxb5hsg2). Some of them are broken
repos, some are ancient eden repos and a simple eden rm and eden clone should
fix them, some are simply old non-eden repos that no one has run commands in in a while.
Reviewed By: xavierd
Differential Revision: D20472234
fbshipit-source-id: 509b4f22b6ac4741b205ef69decfb26e56aebaf8
Summary: Using ptr.add is shorter and preferred to ptr.offset.
Reviewed By: quark-zju
Differential Revision: D20452752
fbshipit-source-id: 1dc2fdbc392267d2d690673c10dcc161ecd00dfa
Summary:
These warnings are fairly trivial, as it recommends using single quote (char)
for single characters search instead of a double quote (str).
Reviewed By: quark-zju
Differential Revision: D20452408
fbshipit-source-id: b2951e133e57633a8e766536e22969fa9ac0ecee
Summary:
Clippy had 3 sources of warnings in this crate:
- from_str method not in impl FromStr. We still have 2 of them in path.rs, but
this is documented as not supported by the FromStr trait due to returning a
reference. Maybe we can find a different name?
- Use of mem::transmute while casts are sufficient. I find the cast to be
ugly, but they are simply safer as the compiler can do some type checking on
them.
- Unecessary lifetime parameters
Reviewed By: quark-zju
Differential Revision: D20452257
fbshipit-source-id: 94abd8d8cd76ff7af5e0bbfc97c1e106cdd142b0
Summary:
Clippy complains about 3 things:
- Using raw pointers in a public function that is not declared as unsafe. This
happens for C exported ones, this feels like a warning, so I haven't changed
it.
- Using .map(...).unwrap_or(<default value constructed>). The recommendation
is to use .unwrap_or_default().
- Single match instead of if let, the latter makes code much shorter.
Reviewed By: quark-zju
Differential Revision: D20452751
fbshipit-source-id: 8eeff7581c119c651ca41d8117f1f70f15774833
Summary:
Right now the module has one implementation IndexedLogStore. The name could
be more specific in the context of the crate.
The goal will be to add a trait for storage requirements of IdDag and
make IndexedLogStorage one implementation of that trait.
Reviewed By: quark-zju
Differential Revision: D20446042
fbshipit-source-id: 7576e1cc4ad757c1a2c00322936cc884838ff710
Summary:
Update the `getpack` code to calculate how many files (and their total
size) would be served over LFS.
NOTE: The columns have `Possible` in their names as we might not have LFS
enabled, in which case we aren't actually fetching this many blobs from an LFS
server.
Reviewed By: farnz
Differential Revision: D20444137
fbshipit-source-id: 85506d8c468cfdc470684dd216567f1848c43d08
Summary:
In dev mode, the glob benchmark failed inside of
folly::Range::operator[] because asserting null termination
technically violates the bounds check.
Reviewed By: simpkins
Differential Revision: D20268416
fbshipit-source-id: ee9b16a6eb9882e850631aa9d83fffe7b6fb67c3
Summary:
On Centos8, `python` no longer exist, instead `python2` or `python3` must be
used.
The onetime-setup.py will have to be re-worked a bit too for Centos8
differences.
Reviewed By: quark-zju
Differential Revision: D20284986
fbshipit-source-id: bb1d9c8b211f1e50ce379fcb6bd92d677323935b
Summary:
Together with `debugmetalogroots`, this allows some kind of "time travel" to
investigate repo states in the past.
Reviewed By: sfilipco
Differential Revision: D20449409
fbshipit-source-id: ed5c134f9e9ee235b24f45c1aa35867a55a71fe5
Summary:
This makes the content of `bookmarks` stable.
It will make the metalog root IDs stable in tests.
Reviewed By: sfilipco
Differential Revision: D20449410
fbshipit-source-id: 969be63ab231f5865ec62e99398b5318d4257093
Summary:
Purge needs to be able to see what directories the walker traversed, so
it can delete them if they are empty. Instead of having the walker call
match.traversedir (which it seems like a bizarre pattern to use the matcher as a
holder for a non-matching related function), let's have the walker return an
enum and have an option to return directories.
At the python layer we then translate this into match.traversedir calls, but we
can clean that up later.
Reviewed By: quark-zju
Differential Revision: D19543795
fbshipit-source-id: cc51c86c91799d3df2c65d25a7b6cfe810206d0a
Summary:
In preparation for supporting returning directories from the walker (to
support purge), let's rename the result structure to be more generic.
Reviewed By: kulshrax
Differential Revision: D19543791
fbshipit-source-id: 9b71452c879cf397ae92533a4ef4727140ac7369
Summary:
On case insensitive systems we need to normalize file case. I've made a
rust case normalizer, but it requires some more tweaks. In the mean time, let's
handle this at the matching and output stages of the rust walk.
This is probably the pattern we want to follow later anyway, so the walk is
completely decoupled from normalization.
Reviewed By: kulshrax
Differential Revision: D19543797
fbshipit-source-id: 2ef8bdcecb2611a08680441fc030c64c2f4097d1
Summary:
The mercurial tests print errors when they encounter 'fifo' files.
Let's handle that case.
Differential Revision: D19543796
fbshipit-source-id: f87d4b9c3f0ad8b8d8ebe2e6d18e325fc93d0ae9
Summary:
\1 is not a valid character in our new RepoPath's, so let's remove it
from the tests.
Reviewed By: quark-zju
Differential Revision: D19543793
fbshipit-source-id: 383a4b3f5c54b47180fe667f234a43eda22f91b6
Summary:
Another small change moving toward using the new futures. This time we
make get_resume_state() async. This pushes the implicit .compat() we would otherwise have down to where we call ThriftManifoldBlob::get() which still returns an old-style BoxFuture.
Reviewed By: farnz
Differential Revision: D20444381
fbshipit-source-id: a5aab448c5b25e71e4ce5ea031302280d586066f
Summary:
Pushes the `compat` call in main() down one level into
populate_healer_queue(). This ended up being straightforward, just a fn
signature change and a couple of callsite changes.
This only affects main(), going to follow-up with future diffs to push it down
a little farther.
Remaining line deltas are just due to rustfmt.
Reviewed By: farnz
Differential Revision: D20432676
fbshipit-source-id: 98100f1b651648d6e3509fa56e94b80ee075843f
Summary:
Allow to gradually rollout lfs. A lot of the details are covered in D20441254
I won't repeat them here. I'd only mention that in order for fastreplay to
correctly calculate percentages this diff starts to log client_hostname for
fastreplay.
Reviewed By: ikostia
Differential Revision: D20441264
fbshipit-source-id: e272176f68879f6c545784609799d21daedec5eb
Summary:
D20139308 added a FromStr implementation for Identity. The LFS
server uses the same code when reading identities from the command line.
Update `idents_from_values` to use the `FromStr` implementation.
Reviewed By: krallin
Differential Revision: D20441439
fbshipit-source-id: 5809ed54d1971fd778b82cc726739a382312747a
Summary:
Right now, ContextConcurrencyBlobstore is instantiated in make_blobstore, which
makes it a lot more effective (3 times more effective, in fact) than we want it
to be, since a ticket is acquired by 3 blobstores in the chain in order to
complete a put:
- The multiplex
- The two underlying blobstores
This also has the potential to deadlock if all tickets are held by the
multiplex, which results in an eventual timeout after 600s of waiting in the
multiplex (this looks like it might be happening at least once or twice per
hour right now on the experimental tier).
In any case, the intention had always been to have one of those per repo, not
one per sub-blobstore, so let's do that. The more natural place to put this
seems to be the RepoBlobstore instantiation.
Since I anticipate I might not be the only one who gets tripped up by this at
some point, I also added a comment about this. I also updated the blobsync
tests to stop re-implementing `RepoBlobstoreArgs::new()` so that adding new
blobstores in RepoBlobstoreArgs will have minimal friction.
Reviewed By: HarveyHunt
Differential Revision: D20467346
fbshipit-source-id: a6ad2d8f04bff1c6fcaa151e947cb8af919eec07
