Summary:
This way we'll never select more than (no_of_stores * limit) rather than
potentially unbounded output.
NOTE: This diff has to be landed and rolled out **after** D20557702 is rolled out. I'm assuming that after some time since D20557702 rollout all the rows in the production db will have proper `operation_key` value set so we can make queries based on them.
Reviewed By: krallin
Differential Revision: D20557700
fbshipit-source-id: 5a1d4b69949b425915214f5227c5c0dcce374360
Summary: So we're sure that all the quries work not only in sqlite.
Reviewed By: krallin
Differential Revision: D20839958
fbshipit-source-id: 9d05cc175d65396af7495b31f8c6958ac7bd8fb6
Summary:
When we have more entries with a single blobkey we always select all of them
regardles when and how they were added. That's why we need to select basing on
operation_key.
Reviewed By: krallin
Differential Revision: D20557699
fbshipit-source-id: 77ccf992bb24d1a46ea28a13ab0780e6c92935ae
Summary:
Now that we have a proper back-channel to retry failures in Python, the
failures will be retried at a later time, when the anti-virus would have
hopefully release any locks it has on the file.
Reviewed By: DurhamG
Differential Revision: D20847006
fbshipit-source-id: 4fad0e773f69ddff27a23bc86dbbd3ce47bb3b46
Summary:
For our HgExternalSync jobs that pull from git, we don't really use most of the
bells and whistles of hggit. Notably, we don't care about bookmarks: we only
ever pull master, we never update to it, we only ever look at `-r tip`.
However, we do care about things that are actually much harder to fit in a
world where we try to pretend the remote git repository is actually a hg
repository we can pull from.
Notably, we'd like to enforce limits on how many commits we pull (and convert)
at a time, so that if we fall behind a little bit, we don't start falling even
more behind by having to convert bigger and bigger batches of commits. If we're
trying to pretend fetching from git and converting commits is actually a pull,
then that seems harder to pull off (we'd need to somehow rewind the remote head
we're pulling before importing it).
So, this adds a new external-sync command to hggit that basically the bare
minimum that we do need. It lets you specify a git remote and a head you care
about, and import up to N commits from it. That's it — no bookmarks are updated
or anything (but the git-mapfile is, of course). The only thing that changes is
your commits.
If you actually want to interact with your git repository on an ongoing basis
as if it were a remote hg repository, this is completely useless, but that
isn't what we actually do, so that should be OK.
As part of this, I've modified a few other parts of git_handler to remove
places where we called a `uri` `remote_name` (which is a bit confusing), and a
place where we were asking for a `remote_name` parameter that I don't have
here, but which we also didn't actually need (in `import_git_objects`).
Reviewed By: farnz
Differential Revision: D20836601
fbshipit-source-id: 96230e6e8269d0472404414948fd2f02aa98d79c
Summary:
Evolution crashes if you try to create obsmarkers in a repo where obsmarkers
are not enabled. Instead, check before reviving commits with obsmarkers that
it is enabled.
Reviewed By: krallin
Differential Revision: D20868173
fbshipit-source-id: b3a08ef05ddc13e9b78df8ac7c1ca1ef808b7602
Summary: Log errors to scuba regardless of the error_as_data setting, as as finding the logs is much easier from scuba than stderr.
Reviewed By: farnz
Differential Revision: D20838462
fbshipit-source-id: b78e3a3213ed4aee4e4b2feb871ad7e42e25ed00
Summary:
Add methods to check if a process ID is alive, and if it looks like an EdenFS
process.
This also adds an initial version of ProcUtils for Windows, and implements
these two methods on Windows. I have moved parts of the `winproc.py` module
to the new `proc_utils_win.py` module, to help better manage dependencies
between our modules. This keeps all of the Windows-specific `ctypes` code
together in `proc_utils_win.py`. The functionality that is still left on
`winproc.py` depends on `config.py`, and the `proc_utils` code should not
depend on `config.py` to help avoid circular dependencies.
Reviewed By: wez
Differential Revision: D20833245
fbshipit-source-id: 43e9b6dd1b520dcb6b2da7701de885058f0f7ea2
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier changes added collectXSemiFuture and collectXUnsafe as a migration
path. We then migrated collectX callsites to collectXSemiFuture or
collectXUnsafe and switched the implementation of collectX to the SemiFuture
form.
This diff renames uses of collectXSemiFuture to collectX, as the two are
identical.
Reviewed By: simpkins
Differential Revision: D20833464
fbshipit-source-id: c14aa40f8cce121632ff922300ef271721c1d695
Summary:
This renames the `ProcessFinder` class to `ProcUtils`, in preparation for
adding some more generic process management utilities into this class.
This class was originally just used for finding stale processes in
`edenfsctl doctor`. However, it seems like a good place to consolidate all
platform-specific process querying logic. We are currently need
platform-specific querying to tell if specific process IDs are still running
for `edenfsctl status` and `edenfsctl stop`, and we also already have some
Windows-specific spawning logic in `edenfsctl start`. I plan to move all of
that logic into this `proc_utils.py` module.
This renames `ProcessFinder` to `ProcUtils` just to better reflect its
expanded purpose.
Reviewed By: genevievehelsel
Differential Revision: D20833243
fbshipit-source-id: 8d33460b7468d877bc327e855af5c620ac5b80a7
Summary:
Update out platform checks to use `sys.platform` instead of `os.name`
Python type checkers (mypy and Pyre) currently understand checks against
`sys.platform`, and know that code guarded by these checks is platform
specific. They don't understand the `os.name` checks.
This code should behave the same, but allows type checkers to do a better job
checking this code.
Note that there is also `platform.system()`, but this has drawbacks that it
apparently does runtime-checks to determine the platform, whereas `sys.platform`
is baked-in at compile time. Additionally, the typecheckers do not appear to
support checking based on `platform.system()` for now. There is an open
feature request for this for mypy, but it is not implemented yet.
One caveat for `sys.platform` is that the results on Linux are not consistent
across Python versions: older versions of Python used to report `linux2` while
new versions report simply `linux`.
Reviewed By: chadaustin
Differential Revision: D20830149
fbshipit-source-id: d173e41f1ae84951a84b87e2dc05787fe8b01407
Summary:
Update the `edenfsctl.exe` binary to find `edenfs.exe` relative to itself.
This ensures that when you run `edenfsctl.exe` from a development build that it
finds the associated development build of `edenfs.exe`, rather than the current
system-installed version.
Reviewed By: wez
Differential Revision: D20821419
fbshipit-source-id: 29ff0b587800f1b76c0b729512bc19f5aef648a7
Summary:
Let's mandate a 256 color terminal in Windows. We also update less to default
to using VT mode: https://github.com/gwsw/less/pull/29
With this, the colors appears to be working as expected on Windows!
Reviewed By: quark-zju
Differential Revision: D20826922
fbshipit-source-id: 5d2a149e2698b3ccc7bf6388b5cd97da20898eb7
Summary:
I'm not exactly sure why, but my guess is that by writing the py files first,
the pyc might be copied and not generated, causing Python to not read the pyc
at import time, leading to slow startup time.
Reviewed By: quark-zju
Differential Revision: D20829130
fbshipit-source-id: e838ff18b9fbf65d1a9437e0a8ef58d57803d2fa
Summary:
Fix some call sites to use collectAllUnsafe().
It looks like these were missed in D20331206.
Reviewed By: genevievehelsel
Differential Revision: D20833198
fbshipit-source-id: 511f6115a5bfeb5b82f9416d5d54bef25a7209a0
Summary:
Combined with the unbundle resolver stats, we will be able to say which
percentage of pushrebases fails, for example.
Reviewed By: StanislavGlebik
Differential Revision: D20818224
fbshipit-source-id: 70888b1cb90ffae8b11984bb024ec1db0e0542f7
Summary:
We need this to be able to monitor how frequently we get pushes vs
infinitepushes, etc. A furhter diff will add a similar reporting to
`processing.rs`, so that we can compute a percentage of successful pushes to
all pushes, for example.
Reviewed By: StanislavGlebik
Differential Revision: D20818225
fbshipit-source-id: 7945dc285560d1357bdc6aef8e5fe50b61622254
Summary:
For whatever reason, ncurses doesn't interpret the arrow keys properly... Use
whatever is returned from it instead.
Reviewed By: kulshrax
Differential Revision: D20821432
fbshipit-source-id: 019426a1333538a479fb3dd8d147fb9cb4ba6559
Summary:
Our phases caching wasn't great. If you tried to ask for a draft commit then
we'd call mark_reachable_as_public method, bu this method was bypassing
caches.
The reason why we had this problem was because we had caching on a higher level
than necessary - we had SqlPhases struct which was "smarter" (i.e. it has a
logic of traversing ancestors of public heads and marking these ancestors and
public) and SqlPhasesStore which just did sql access. Previously we had our
caching layer on top of SqlPhases, meaning that when SqlPhases calls
`mark_reachable_as_public` it can't use caches anymore.
This diff fixes it by moving caching one layer lower - now we have a cache
right on top of SqlPhasesStore. Because of this change we no longer need
CachingPhases, and they were removed. Also `ephemeral_derive` logic was
simplified a bit
Reviewed By: krallin
Differential Revision: D20834740
fbshipit-source-id: 908b7e17d6588ce85771dedf51fcddcd2fabf00e
Summary:
Very small refactoring to store MemcacheHandler (i.e. an enum which can either
be a real Memcache client or a mock) instead of a memcache client.
It will be used in the next diff to create mock caches
Reviewed By: krallin
Differential Revision: D20834916
fbshipit-source-id: cb1e3e8f0ae0e2c0f7018d3a003ada56725c65c6
Summary: SelectPhases does the same thing - no need to keep two queries
Reviewed By: krallin
Differential Revision: D20817379
fbshipit-source-id: 8cc56ea4a94e81f110a286899a8f5c596566a142
Summary: I'm going to refactor it soon, for now just move it to another file.
Reviewed By: krallin
Differential Revision: D20817293
fbshipit-source-id: 6fb44b4be858ebbd0e8c9dfee160b91806f78285
Summary:
Move the attribute so it won't cause an AttributeError when remotenames is not
enabled.
As we're here, also move it to repo._remotenames from repo. This makes the repo
object a bit clearner and avoids loading the accessed bookmarks if remotenames
are not accessed at all.
Reviewed By: singhsrb
Differential Revision: D20821070
fbshipit-source-id: b5a01016451ee202f778cadb2dd418400406fa6b
Summary:
This diff turns off the support_old_nightly feature of async-trait (https://github.com/dtolnay/async-trait/blob/0.1.24/Cargo.toml#L28-L32) everywhere in fbcode. I am getting ready to remove the feature upstream. It was an alternative implementation of async-trait that produces worse error messages but supports some older toolchains dating back to before stabilization of async/await that the default implementation does not support.
This diff includes updating async-trait from 0.1.24 to 0.1.29 to pull in fixes for some patterns that used to work in the support_old_nightly implementation but not the default implementation.
Differential Revision: D20805832
fbshipit-source-id: cd34ce55b419b5408f4f7efb4377c777209e4a6d
Summary:
This isn't actually a Mercurial extension, so we can't be importing Mercurial
imports here. I'm not sure this is actually being used because we apparently
have 2 copies of this (see D20793452, which is also how I came upon this), but
it's probably worth not breaking it.
Reviewed By: farnz
Differential Revision: D20794695
fbshipit-source-id: daaf0f5822567b55a787669073421fd1ce604e08
Summary:
We've seen a number of recent cases where hg stops using watchman for
status because state-enter or state-leave fail because multiple enters or leaves
are called out of order. The root cause needs to be found, but in the mean time
we should not allow such errors to block of from using watchman entirely.
Let's eat these gracefully and just log that the issue happened.
Reviewed By: quark-zju
Differential Revision: D20807004
fbshipit-source-id: 517d9562febc226ce7db836435ca2ca9db1c68fb
Summary:
On Windows, we have to do a bit of dance for Windows to stop complaining
about ui.ssh.
Reviewed By: DurhamG
Differential Revision: D20817712
fbshipit-source-id: acbda636fe114fd616dee89b2c4d1c9ff26470bf
Summary:
On shared devservers, all users in svnusers should be able to read/write onto
the stores, set the umask to properly reflect this.
Reviewed By: ikostia
Differential Revision: D20817266
fbshipit-source-id: 8a865f499a35313afd3cbeb5f989175862a3549f
Summary: This will allow bash users to use things like `hg commit -l <(echo foo)`.
Reviewed By: krallin
Differential Revision: D20810653
fbshipit-source-id: 42e420e608d41704387a9011cf14a28f92192e5d
Summary:
Add a fingerprint method that returns a subset of the hash.
This will allow us to see compression benefit, or write out a corpus, sampling 1 in N of a group of keys
Reviewed By: krallin
Differential Revision: D20541312
fbshipit-source-id: 93bd44ba9c14285daf50d8cd18eeb4b6dcc38d82
Summary:
Use the new sampling blobstore and sampling key in existing compression-benefit subcommand and check the new vs old reported sizes.
The overall idea for these changes is that the walker uses a CoreContext tagged with a SamplingKey to correlate walker steps for a node to the underlying blobstore reads, this allows us to track overall bytes size (used in scrub stats) or the data itself (used in compression benefit) per node type.
The SamplingVisitor and NodeSamplingHandler cooperate to gather the sampled data into the maps in NodeSamplingHandler, which the output stream from the walk then operates on, e.g. to compress the blobs and report on compression benefit.
The main new logic sits in sampling.rs, it is used from sizing.rs (and later in stack from scrub.rs)
Reviewed By: krallin
Differential Revision: D20534841
fbshipit-source-id: b20e10fcefa5c83559bdb15b86afba209c63119a
Summary:
This updates fastheaddiscovery to use an unfiltered repository when it attempts
to check if the nodes the server sent are present locally.
Reviewed By: quark-zju
Differential Revision: D20792006
fbshipit-source-id: 14ba9605d79ba54f3f4143d6d8ec65357e3d8c07
Summary: Now that everything is using `sql_construct`, we can remove the old `SqlConstructors` trait.
Reviewed By: StanislavGlebik
Differential Revision: D20734881
fbshipit-source-id: af46b41d17b40f6eb0839cdb9e85b00067360fe9
Summary:
Migrate the configuration of sql data managers from the old configuration using `sql_ext::SqlConstructors` to the new configuration using `sql_construct::SqlConstruct`.
In the old configuration, sharded filenodes were included in the configuration of remote databases, even when that made no sense:
```
[storage.db.remote]
db_address = "main_database"
sharded_filenodes = { shard_map = "sharded_database", shard_num = 100 }
[storage.blobstore.multiplexed]
queue_db = { remote = {
db_address = "queue_database",
sharded_filenodes = { shard_map = "valid_config_but_meaningless", shard_num = 100 }
}
```
This change separates out:
* **DatabaseConfig**, which describes a single local or remote connection to a database, used in configuration like the queue database.
* **MetadataDatabaseConfig**, which describes the multiple databases used for repo metadata.
**MetadataDatabaseConfig** is either:
* **Local**, which is a local sqlite database, the same as for **DatabaseConfig**; or
* **Remote**, which contains:
* `primary`, the database used for main metadata.
* `filenodes`, the database used for filenodes, which may be sharded or unsharded.
More fields can be added to **RemoteMetadataDatabaseConfig** when we want to add new databases.
New configuration looks like:
```
[storage.metadata.remote]
primary = { db_address = "main_database" }
filenodes = { sharded = { shard_map = "sharded_database", shard_num = 100 } }
[storage.blobstore.multiplexed]
queue_db = { remote = { db_address = "queue_database" } }
```
The `sql_construct` crate facilitates this by providing the following traits:
* **SqlConstruct** defines the basic rules for construction, and allows construction based on a local sqlite database.
* **SqlShardedConstruct** defines the basic rules for construction based on sharded databases.
* **FbSqlConstruct** and **FbShardedSqlConstruct** allow construction based on unsharded and sharded remote databases on Facebook infra.
* **SqlConstructFromDatabaseConfig** allows construction based on the database defined in **DatabaseConfig**.
* **SqlConstructFromMetadataDatabaseConfig** allows construction based on the appropriate database defined in **MetadataDatabaseConfig**.
* **SqlShardableConstructFromMetadataDatabaseConfig** allows construction based on the appropriate shardable databases defined in **MetadataDatabaseConfig**.
Sql database managers should implement:
* **SqlConstruct** in order to define how to construct an unsharded instance from a single set of `SqlConnections`.
* **SqlShardedConstruct**, if they are shardable, in order to define how to construct a sharded instance.
* If the database is part of the repository metadata database config, either of:
* **SqlConstructFromMetadataDatabaseConfig** if they are not shardable. By default they will use the primary metadata database, but this can be overridden by implementing `remote_database_config`.
* **SqlShardableConstructFromMetadataDatabaseConfig** if they are shardable. They must implement `remote_database_config` to specify where to get the sharded or unsharded configuration from.
Reviewed By: StanislavGlebik
Differential Revision: D20734883
fbshipit-source-id: bb2f4cb3806edad2bbd54a47558a164e3190c5d1
Summary:
Refactor `sql_ext::SqlConstructors` and its related traits into a separate
crate. The new `SqlConstruct` trait is joined by `SqlShardedConstruct` which
allows construction of sharded databases.
The new crate will support a new configuration model where the distinction
between the database configuration for different repository metadata types
will be made clear.
Reviewed By: StanislavGlebik
Differential Revision: D20734882
fbshipit-source-id: b44cf9d1efd014c29df88e2ad933025e440119dc
Summary:
The new API does nothing that cloud sync does not want: bookmarks, obsmarkers,
prefetch, etc. Wrappers to disable features are removed.
This solves a "lagged master" issue where selectivepull adds `-B master` to
pull extra commits but cloud sync cannot hide them without narrow-heads. Now
cloud sync just does not pull the extra commits.
Reviewed By: sfilipco
Differential Revision: D20808884
fbshipit-source-id: 0e60d96f6bbb9d4ce02c04e8851fc6bda442c764
Summary:
With selective pull (now in core) enabled and narrow-heads disabled,
there is another way to cause lagged master.
Reviewed By: sfilipco
Differential Revision: D20808885
fbshipit-source-id: ebaed0e203f2f9f981bf141ce6c82af33710ba53
Summary: This makes the upcoming change easier to verify.
Reviewed By: sfilipco
Differential Revision: D20808883
fbshipit-source-id: 5563601125ef5c961785f7275fd82fc3fefe53ff
Summary:
We have seen one report that the race condition caused lagged remote/master.
The race condition is:
- pull everything: get all commits, but not remotenames.
- expull: get all remotenames. Some point to new commits received by the
server that are unknown to the client. The client ignores them.
This diff adds a double check to pull again if it sees remotenames pointing
to unknown nodes.
This won't be necessary if we migrate everyone to selectivepull, and clean up
the pull command to use the race-free repo.pull API. Because the new code path
will first query the bookmarks, then pull the heads explicitly.
Reviewed By: DurhamG
Differential Revision: D20703268
fbshipit-source-id: 42bd0201c3c4f709f25ad0fea3a191db9991e4fc
Summary:
This makes it possible to do things like `hg up remote/foo` without having to
`hg pull -B foo` first.
Reviewed By: DurhamG
Differential Revision: D20531122
fbshipit-source-id: e95b2f0a49e4b815c136450d1f352a7973cb72ed
Summary:
The new `repo.pull` API aims to be tech-debt free. It does:
- Pull nothing instead of everything if nothing is specified.
- Update remote names that are explicitly specified, not everything the server
has.
- Do not update local bookmarks.
The direct motivation is to implement autopull `remote/foo` behavior while not relying on the pull command.
Reviewed By: DurhamG
Differential Revision: D20531119
fbshipit-source-id: 80c3bdd5556126d81af099a74f1345ecc94904b7
Summary: This will be used by the next diff.
Reviewed By: DurhamG
Differential Revision: D20661056
fbshipit-source-id: 3b4793bc0737bd64196da5b99c2d4da6f5e48efb
Summary: This diff ports TreeInode, FileInode, InodeMap and related classes to Windows. We don't build or test it here, there are more dependcies we need to port. The built script and the test are part of other diffs in this stack.
Reviewed By: simpkins
Differential Revision: D19956266
fbshipit-source-id: 9eb754233bca3d5a336f465c2400512a8593ca4f
Summary: Debugging an issue where watchman returns a state error during histedit.
Reviewed By: quark-zju
Differential Revision: D20785858
fbshipit-source-id: 5baf762d1a5588573df9d01c63a24e751e04a811
Summary:
During the roll-out of commitcloud remotebookmarks sync, some clients may have it enabled, and others may have it disabled.
This can result in some clients having an empty set of remotebookmarks in their sync state for a version which actually
does have remotenames in the cloud workspace.
This becomes a problem when those clients start syncing remotebookmarks, for two reasons:
* They are unable to correctly apply the new cloud remotebookmarks, as they do not have the previous state with which to break ties, and the tie-breaking code doesn't work correctly if the old cloud remotebookmark is `None`.
* When they try to send the remotebookmarks delta to the server, they don't have the old state, and so compute the delta incorrectly. This results in `Duplicate entry for key` errors in the service.
We can handle this as follows:
* When syncing remote bookmarks from the cloud, ensure we correctly apply the remote bookmark updates, even if the old cloud remotebookmark is `None`.
* If the local repo is up-to-date with respect to the workspace, but the last sync state had no remote bookmarks recorded, sync the remote bookmarks again. This may involve pulling some more commits if the cloud remote bookmarks are ahead of the local remote bookmarks. If there truly were no remote bookmarks then this will be a no-op.
Reviewed By: quark-zju
Differential Revision: D20792360
fbshipit-source-id: cd1ed371d8d5b1be2767c8a8d4836ea870a4467b
Summary:
For the initial rollout of lfs on fbsource we want to rollout just for our
team using rollout_smc_tier option. This diff adds a support for that in
Mononoke.
It spawns a future that periodically updates list of enabled hosts in smc tier.
I had a slight concern about listing all the available services and storing
them in memory - what if smc tier have too many services? I decided to go ahead
with that because
1) [Smc antipatterns](https://fburl.com/wiki/ox43ni3a) wiki page doesn't seem
to list it as a concern.
2) We are unlikely to use for large tier - most likely we'll use it just for
hg-dev which contains < 100 hosts.
Reviewed By: krallin
Differential Revision: D20789751
fbshipit-source-id: d35323e49530df6983e159e2ed5bce205cc5666d
Summary:
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Because it is a strong indicator (specifically stronger than the number of imports) for when there is an issue effecting imports, we use it as a warning signal to the user, meaning we display warning colors when the duration of the longest import is long enough to warrant concern.
We have 2 levels of concern to differentiate when there may be something strange happening and when there is almost definitely something strange happening. These correspond to displaying the import metrics in yellow and red respectively. The current threshold for something strange possibly happing is imports taking longer than 10s and something strange definitely happening is imports taking longer than 30s.
Reviewed By: chadaustin
Differential Revision: D20627754
fbshipit-source-id: 7931d2c0ab53952c1291fd1d411dad14d142734b
Summary:
When there is an issue in the import process, there is not an easy way to detect this currently. As a first step towards making it possible to detect when there is an issue here, we expose the number if imports pending to `eden top`.
If there is a problem blocking imports from happening, then they will start to queue up and the number of pending imports will keep growing. When there are a lot of pending imports it indicates that there may be a problem in the import process.
note: there are cases when there are many pending imports under normal operation. It is common to batch fetch a bunch of files at once, and submitting all these prefetches at once will mean a lot of pending imports though there is no problem. Thus number of pending imports only indicates there *may* be an issue in the import process. (Thus we couple it with the duration of the longest import D20627754 for stronger signal)
The other value in showing the number of pending imports is giving users more transparency into eden. When there is a big batch of imports this may leave the user waiting a while, and this gives a way for them to know why. They can see that eden is making progress rather than being stuck.
Reviewed By: fanzeyi
Differential Revision: D20527273
fbshipit-source-id: 59e0fdff6e7b154b409dfab385b4fdb02062b066
Summary:
The primary purpose of exposing counters for the maximum length of duration is to add this to `eden top`.
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Additionally we choose to use these counters to expose this data because we want to log this data. This data can be used to measure performance of the queueing and import process. In the future this data could be used to set up alerting for regressions and allow more proactive fixes.
Reviewed By: chadaustin
Differential Revision: D20611728
fbshipit-source-id: 9307c1ad749ac5fe356ba9eaf868de41b1a8a3a7
Summary: removing duplication to prepare for adding more counters in `EdenServer`
Reviewed By: chadaustin
Differential Revision: D20611713
fbshipit-source-id: 30dd59dff46731b53b08d5681b2add4c94cd049a
Summary:
Here we are tracking the duration of each of the current imports in `HgBackingStore`. This changes is setup to display the duration of the longest outstanding import in `eden top`.
Displaying the duration of the longest import in eden top will help debug where the problem is when eden hangs. These specifically help determine if eden is hanging because of a queueing issue or not being able to fetch an object from the server.
This is a stronger signal to the user that there is an issue in importing over just a count of imports (the previous addition) because many pending imports is not always abnormal. For example a program may prefetch a big batch of files to read, and this could cause a large number of pending imports. A single import should not hang due to a big prefetch and other normal uses.
Thus we want to display this in addition to the number of imports and use this as our warning indicator. (i.e. a high duration of import will mark the metrics as yellow or red in `eden top` to give a warning to the user)
*code changes*:
- adding lists to store watches to time the duration of each of the imports in `HgBackingStore`
- removing the counters previously used to count the number of imports in `HgBackingStore` in favor of using the size of these lists as this is constant time, so there is no benefit to tracking this value ourselves
Reviewed By: chadaustin
Differential Revision: D20611704
fbshipit-source-id: 8bbe0aed8d10688f7bfb7ecfa2fa5ac181945a76
Summary:
git started printing a "+" sign in front of a branch checked out in another worktree:
ab3138146f
and it breaks "git sl"
removing "+" the same way as "*" when parsing "git branch" result to resolve.
Reviewed By: krallin
Differential Revision: D20793452
fbshipit-source-id: 5c0b6be5afbed607b144c652f68a3ab93052f76a
Summary:
This is a followup from D20766465. In D20766465 we've avoided re-traversing
fsnodes for all entries except for copy/move sources. This diff make copy/move
sources fetching more efficient as well.
It does by sending a find_entries() request to prefetch all fsnodes
Reviewed By: mitrandir77
Differential Revision: D20770182
fbshipit-source-id: 7e4a68a2ded20b2895ee4d1c4f8fd897dbe1c850
Summary:
We are currently having problems with streaming clone:
```
$ hg --config 'extensions.fsmonitor=!' clone --shallow -U --config 'ui.ssh=ssh -oControlMaster=no' --configfile /etc/mercurial/repo-specific/fbsource.rc 'ssh://hg.vip.facebook.com//data/scm/fbsource?force_mononoke' "$(pwd)/fbsource-clone-test"
remote: server: https://fburl.com/mononoke
remote: session: vJ3qkiQIm9FT7mCp
connected to twshared11499.02.cln2.facebook.com
streaming all changes
2 files to transfer, 5.42 GB of data
abort: unexpected response from remote server:
'\x00\x01B?AB\x00\x00\x00\x00\x02U\x00\x00\x02\xc7\x00b\xf0\xd5\x00b\xf0\xd5\x00b\xf0\xd4\xff\xff\xff\xff\xa8z\xc7W\xd0&\xab\xb2\xf1{\xbfq\xac<\xaf6W\x06q\x81\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?C\x97\x00\x00\x00\x00\x053\x00\x00\x06\xce\x00b\xf0\xd6\x00b\xf0\xd6\x00b\xf0\xd5\xff\xff\xff\xff\xa3I\x19+\xe2\x0f\xae\xd2\x95\x14\x8a\xde\x19\x18\xf0\x8cUQu\xf1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?H\xca\x00\x00\x00\x00\x02\xe4\x00\x00\x03\x9e\x00b\xf0\xd7\x00b\xf0\xd7\x00b\xf0\xd6\xff\xff\xff\xffx\xd6}\x12nt\xb9\xbc(\x83\xfb\xfa\xcc\xc1o?\xde\xcc\x06L\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?K\xae\x00\x00\x00\x00\x02j\x00\x00\x02\xb5\x00b\xf0\xd8\x00b\xf0\xd8\x00b\xf0\xd7\xff\xff\xff\xff\x04"\xfcw6\'M\xba\xf1f\xdb\x02\xbeE\x93:\xc8\x17\x88P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01B?N\x18\x00\x00\x00\x00\x03\xbb\x00\x00\x04\xb8\x00b\xf0\xd9\x00b\xf0\xd9\x00b\xf0\xd8\xff\xff\xff\xff\xb9\x15*p/\xa4*\x00\x9dZw\x01B\x87L\x8f\x08\x11\x89\xe0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0000changelog.d\x005406413267\n'
```
as the result of the debugging it is turned out that we are sending more data than expected, to have better error next time if we have any corruption of the `streaming_changelog_chunks` table
Reviewed By: StanislavGlebik
Differential Revision: D20763738
fbshipit-source-id: 6f6fa9f9a29909e044d9ba42fe84916ddcb62e8f
Summary: Each benchmark takes about 3 minutes to run. We've already got 16 benchmarks, and we're going to grow. Allow you to limit the number of benchmarks we run at once..
Reviewed By: ahornby
Differential Revision: D20735795
fbshipit-source-id: 241184085b35da8ab85314fef1c6a08404bdb769
Summary: We're going to be doing a variety of changes to sqlblob - let's enable working against a known baseline each time, instead of incremental changes.
Reviewed By: ahornby
Differential Revision: D20735796
fbshipit-source-id: 86f15dac1f004b2f3c83ced829a65f3f6e111d6b
Summary:
We want to be able to benchmark blobstore stacks to allow us to compare blobstore stacks, and ensure that we're not regressing as we improve SQLblob to production state.
Use Criterion to benchmark a few basic workflows - starting with writes, but reads can be added later.
Reviewed By: ahornby
Differential Revision: D20720789
fbshipit-source-id: e8b10664a9d08a1aa7e646e1ebde251bec0db991
Summary: Use the client identity middleware from gotham_ext in the EdenAPI server. This middleware parses validated client identities from HTTP headers inserted by Proxygen; these identities can then be used to enforce repo ACLs.
Reviewed By: HarveyHunt
Differential Revision: D20744887
fbshipit-source-id: 651e171d1b20448b3e99bfc938d118fb6dddea91
Summary:
It turns out that eden integration tests do not always set HGRCPATH for hg
commands it executes. The testing library (integration/lib/hgrepo.py) sets
HGRCPATH for hg commands it runs. However there are other code paths that
bypass the HGRCPATH setting. For example:
- hg debugimporthelper started by edenfs does not have HGRCPATH set.
- hg log -r . -T{node} started by fs/cli/util.py does not have HGRCPATH set.
That caused issues. For example, without HGRCPATH, the real system hgrc is
loaded, and that can be different from the in-repo hgrc, causing issues like
migrating "narrow-heads" down and up, which pollutes stderr and fails tests.
Increase test robustness by adding an `%include $HGRCPATH` in the repo hgrc
so commands without HGRCPATH still load the desired system hgrc.
Reviewed By: chadaustin
Differential Revision: D20769184
fbshipit-source-id: 8992bfb95250b6cccb9ebcf4bbaa0a76d50480dd
Summary: Add a hint that points to customization and troubleshooting guides for the new graph renderer.
Reviewed By: xavierd
Differential Revision: D20763338
fbshipit-source-id: ee6d2464ae5955f0f0bf52d1994adfa2b74b3367
Summary:
Instead of using Sha256::from_slice, just use Sha256::from with a correctly
sized array.
Reviewed By: quark-zju
Differential Revision: D20756181
fbshipit-source-id: 17c869325025078e4c91a564fc57ac1d9345dd15
Summary: We are changing some of the internals of how the status is working. Few tests are broken and would need some work to get them working. Will enable them once all the Inode support is landed and the tests are fixed.
Reviewed By: wez
Differential Revision: D20602112
fbshipit-source-id: eadae818deaf62bcf0f9ec51d9dd531a6d452aa4
Summary:
Eden on Windows doesn't support setting a file as executable or creating symlinks.
Windows doesn't need executable mode bit to execute. It can execute the files with executable extension, or the responsible program can run it like Python3.exe can run python script.
Reviewed By: chadaustin
Differential Revision: D19956268
fbshipit-source-id: c22416db2a9da78e3a5c4392d1537eb7cbf9bfd0
Summary: Looks like it's not dead anymore
Reviewed By: krallin
Differential Revision: D20766497
fbshipit-source-id: c49ae3b6c8a660b33e61e65adda94f78addd1498
Summary:
It is preferable to use the higher-level API of cached_config instead of ConfigeratorAPI whenever possible since the higher-level API supports OSS builds.
For `ConfigStore` let `poll_interval` be None so that for one-off reading of configs the ConfigStore doesn't needlessly spawn an updating thread.
Also this update is with compliance to the discussion in D19026190.
Reviewed By: ahornby
Differential Revision: D20670224
fbshipit-source-id: 24fc124d440fd458a9fa88a906fc3a1cfdbd827e
Summary: In the process the blobstore/factory/lib.rs was split into submodules - this way it was easier to untangle the dependencies and refactor it, so I've left the split in this diff.
Reviewed By: markbt
Differential Revision: D20302068
fbshipit-source-id: caa3a2b5487c30198c62f7e4f4e9cb7c488dc8de
Summary:
Replace explicit casts working around T38947910 (Pyre not understanding
decorators) with `pyre-ignore` comments. This will hopefully help ensure that
these are tracked more visibly, and are removed when this task is fixed.
Reviewed By: wez
Differential Revision: D20434082
fbshipit-source-id: da4e4d11e4e029a11984c0efcad2aecd0d3094bf
Summary:
Update `hg status` to print errors that were returned by EdenFS's
getScmStatusV2() call, and to exit unsuccessfully if there were any errors.
Previously errors were silently ignored.
Reviewed By: quark-zju
Differential Revision: D19958503
fbshipit-source-id: cb3109df40eb86a5bf7e3818ddfb8da74d670405
Summary:
Fix the `test_status()` function to properly canonicalize the input paths, so
that it works successfully on Windows. Previously this function would fail on
Windows as some paths would end up as UNC-style paths and some would be plain
paths, causing the equality comparison to fail even though the paths were
equivalent. This canonicalizes the repo path so that they both use the same
format.
Reviewed By: quark-zju
Differential Revision: D20662921
fbshipit-source-id: fdd36bac755f9694b4a482615d3dca43ff21e05e
Summary:
All of the callers are already using an Arc, so instead of forcing the remote
store to be cloneable, and thus wrap an inner self with an Arc, let's just pass
self as an Arc.
Reviewed By: DurhamG
Differential Revision: D20715580
fbshipit-source-id: 1bef23ae7da7b314d99cb3436a94d04134f1c0e4
Summary:
When LFS will be enabled on fbsource, the enablement will rolled out server,
with the server serving pointers (or not). In the catastrophic scenario where
Mononoke has to be rolled out, the Mononoke LFS server will be unable to serve
blobs, but some clients may still have LFS pointers locally but not the
corresponding blob. For this, we need to be able to fallback to fetching the
blob via the getpackv2 protocol.
Reviewed By: DurhamG
Differential Revision: D20662667
fbshipit-source-id: 4ac45558f6d205cbd1db33c21c6fb137a81bdbd5
Summary:
The LFS server might be temporarily having issues, let's retry a bit before
giving up.
Reviewed By: DurhamG
Differential Revision: D20686659
fbshipit-source-id: 90dabd19e45a681d6eae5cd50c72b635d44c0517
Summary:
Since we have all the integer types, let's also allow float types in the
config.
Reviewed By: kulshrax
Differential Revision: D20697007
fbshipit-source-id: 21fa264d24c0f63c233f47c3bcfb2448b4c05c70
Summary:
This demonstrates that both the legacy extension and the new implementation can
co-exist, and we can enable/disable the new extension.
Reviewed By: DurhamG
Differential Revision: D20677443
fbshipit-source-id: 10896023f536984645371d557c3ad20daa8526dd
Summary:
When repacking for the purpose of file format changes, a single packfile may
contain data that needs to be moved out of it, and thus, we need to do a repack
then.
Reviewed By: DurhamG
Differential Revision: D20677442
fbshipit-source-id: c621dd2e657f5a4565b37d4b029731415b899117
Summary:
Remotestores can implement get_missing properly by simply querying the
underlying store that they will be writing to. This may prevent double fetching
some blobs in `hg prefetch` that we already have.
Reviewed By: DurhamG
Differential Revision: D20662668
fbshipit-source-id: 22140b5b7200c687e0ec723dd8879dc8fbea6fb9
Summary:
There are cases where the user of the abstraction needs to know if this is a
local store, this will simplify the caller code.
Reviewed By: DurhamG
Differential Revision: D20662666
fbshipit-source-id: e0bde7eb0dc3484979732a7c4cdf888fedc70e13
Summary:
By regularly flushing the blob store, we avoid keeping too many LFS blobs in
memory, which could cause OOM issues.
The default size is chosen to be 1GB, but is configurable for more control.
Reviewed By: DurhamG
Differential Revision: D20646213
fbshipit-source-id: 12c06fd0212ef3974bea10c82026b6e74fb5bf21
Summary:
In the legacy lfs extension, LFS blobs were stored as loosefiles on disk, and
as we saw with loosefiles for remotefilelog, they can incur a significant
overhead to maintain. Due to LFS blobs being large by definition, the number of
loose LFS blobs should be reasonable for repack to walk over all of them to
chose which one to throw away.
A different approach would be to simply store the blobs in an on-disk format
that allows automatic size management, and simple indexing. That format is an
IndexedLog. This of course doesn't come without drawbacks, the main one being
that the IndexedLog API mandate that the full blob is present on insertion,
preventing streaming writes to it, the solution is to simply chunk the blobs
before writing them to it. While proper streaming is not done just yet, the
storage format no longer prevent it from being implemented.
Reviewed By: DurhamG
Differential Revision: D20633783
fbshipit-source-id: 37a88331e747cf22511aa348da2d30edfa481a60
Summary:
As suggested in D20680173, we can reduce the overall need to copy things by
storing refs in the resolver.
Reviewed By: krallin
Differential Revision: D20696588
fbshipit-source-id: 9456e2e208cfef6faed57fc52ca59fafdccfc68c
Summary:
See bottom diff of this stack for overview.
This diff in particular asyncifies the `upload_changeset` fn. Apart from that,
it also makes sure it can accept `&RevlogChangeset` instead of
`RevlogChangeset`, which helps us to get rid of cloning.
Reviewed By: krallin
Differential Revision: D20693932
fbshipit-source-id: b0e5e1604cbfb6f6b6e269c85a79208115325734
Summary: Same as the bottom diff of this stack, but for another file.
Reviewed By: krallin
Differential Revision: D20693934
fbshipit-source-id: 4c2d12bf9d9ab272898a7830ece6d9f563adb8fb
Summary:
This diff focuses on the following:
- replaces clones with references, both when this decreases the total sum of
clones, and when it causes the only clone to be on the boundary with the
compat code. This, when those boundaries are pushed further, we can only fix
one place in resolver
- removes a weird wrapping of a closure into an `Arc` and just calls
`upload_changesets` directly instead
- in cases when `BundleResolver` methods take `ctx` as an argument removes it
and makes those methods use the one stored in the struct
Reviewed By: StanislavGlebik
Differential Revision: D20680173
fbshipit-source-id: c397c4ade57a07cbbc9206fa8a44f4225426778c
Summary:
RotateLog loads older logs lazily. If an older log is broken, remember that and avoid
loading the broken log again.
Reviewed By: DurhamG
Differential Revision: D20663899
fbshipit-source-id: 7a4b5279cc6387c19329a51048bfe1be2e0bc1f8
Summary:
This updates hg rage to disable colors while producing the hg rage. This makes
the pastes a little more readable since they won't contain escape codes.
Reviewed By: farnz
Differential Revision: D20738945
fbshipit-source-id: e323834beb6458083642e34a40afa3d896288b42
Summary:
Instead, it's present in the fileslog. Use that instead.
This is the error I just got while doing a `hg top`:
error updating fbcode/eden/scm/lib/revisionstore/src/contentstore.rs: facebook::eden::HgImportPyError: AttributeError: 'bindings.revisionstore.contentstore' object has no attribute 'commitpending'
Reviewed By: quark-zju
Differential Revision: D20715560
fbshipit-source-id: 59c4b62ae8d0f182824e126e68b174c0ef3cdba0
Summary:
Due to the Mononoke LFS server only being available on FB's network, the tests
using them cannot run outside of FB, including in the github workflows.
Reviewed By: quark-zju
Differential Revision: D20698062
fbshipit-source-id: f780c35665cf8dc314d1f20a637ed615705fd6cf
Summary:
This bitrot with two different changes:
- D19473960 put it on a v2 runtime, but the I/O is v1 so that doesn't work (it
panics).
- The clap update a couple months ago made duplicate arguments illegal, and a
month before that we had put `debug` in the logger args (arguably where it
belong), so this binary was now setting `debug` twice, which would now panic.
Evidently, just static typing wasn't quite enough to keep this working through
time (though that's perhaps due to the fact that both of those changes were
invisible to the type system), so I also added a smoke test for this.
Reviewed By: farnz
Differential Revision: D20618785
fbshipit-source-id: a1bf33783885c1bb2fe99d3746d1b73853bcdf38
Summary:
As the name indicates, this updates unbundle_replay to run hooks. Hook failures
don't block the replay, but they're logged to Scuba.
Differential Revision: D20693851
fbshipit-source-id: 4357bb0d6869a658026dbc5421a694bc4b39816f
Summary:
Setting up a derived data tailer for this is a better approach (see D20668301
for context).
Reviewed By: StanislavGlebik
Differential Revision: D20693270
fbshipit-source-id: 7a06ffe059c41c4e100f8b0f8837978717293829
Summary:
Since we do those concurrently, it makes sense to do them on their own task.
Besides, since those are still old futures that need ownership, there is
effectively no tradeoff here.
Differential Revision: D20691373
fbshipit-source-id: 1a45e43ec857d91bed1614568b4354d56a2b0848
Summary: This will make it easier to compare performance.
Differential Revision: D20674164
fbshipit-source-id: eb1a037b0b060c373c1e87635f52dd228f728c89
Summary: This adds some Scuba reporting to unbundle_replay.
Differential Revision: D20674162
fbshipit-source-id: 59e12de90f5fca8a7c341478048e68a53ff0cdc1
Summary:
This updates unbundle_replay to do things concurrently where possible.
Concretely, this means we do ingest unbundles concurrently, and filenodes
derivation concurrently, and only do the actual pushrebase sequentially. This
lets us get ahead on work wherever we can, and makes the process faster.
Doing unbundles concurrently isn't actually guaranteed to succeed, since it's
*possible* that an unbundle coming in immediately after a pushrebase actually
depends the commits created in said pushrebase. In this case, we simply retry
the unbundle when we're ready to proceed with the pushrebase (in the code, this
is the `Deferred` variant). This is fine from a performance perspective
As part of this, I've also moved the loading of the bundle to processing, as
opposed to the hg recording client (the motivation for this is that we want to
do this loading in parallel as well).
This will also let us run hooks in parallel once I add this in.
Reviewed By: StanislavGlebik
Differential Revision: D20668301
fbshipit-source-id: fe2c62ca543f29254b4c5a3e138538e8a3647daa
Summary:
pushrebase_errmsg is NULL when we have conflicts, but we still shouldn't replay
the entry (because it'll fail, with conflicts). Let's exclude those.
Reviewed By: StanislavGlebik
Differential Revision: D20668304
fbshipit-source-id: a058bb466e0a8a53ec81e41db7ba138d6aedf3f9
Summary:
This updates unbundle_replay to support sleeping when watching for updates in a
bookmark and said bookmark isn't moving. This will be useful so it can run as a
service.
Reviewed By: StanislavGlebik
Differential Revision: D20645157
fbshipit-source-id: 6edeb66b65b2ef8b88c8db5e664982756acbfaf1
Summary:
I accidentally forgot to insert the entry, so that made this test a bit
useless. Let's make it not useless.
Reviewed By: StanislavGlebik
Differential Revision: D20645158
fbshipit-source-id: 0f0eb0cf9d16e8c346897088891aa3277b4d9c07
Summary:
This adds support for replaying the updates to a bookmark through unbundle
replay. The goal is to be able to run this as a process that keeps a bookmark
continuously updated.
There is still a bit of work here, since we don't yet allow the stream to pause
until bookmark update becomes available (i.e. once caught up, it will exit).
I'll introduce this in another diff.
Note that this is only guaranteed to work if there is a single bookmark in the
repo. With more, it could fail if a commit is first introduced in a bookmark that
isn't the one being replayed here, and later gets introduced in said bookmark.
Reviewed By: StanislavGlebik
Differential Revision: D20645159
fbshipit-source-id: 0aa11195079fa6ac4553b0c1acc8aef610824747
Summary:
I'm going to update this to run in a loop, so to do that it would be nice to
represent the things to replay as a stream. This does that change, but for now
all our streams have just one element.
Reviewed By: StanislavGlebik
Differential Revision: D20645156
fbshipit-source-id: fce7536d0ccbc1911335704816b71c17e80f2116
Summary:
We normally derive those lazily when accepting pushrebase, but we do derive
them eagerly in blobimport. For now, let's be consistent with blobimport.
This ensures that we don't lazily generate them, which would require read traffic,
and gives a picture a little more consistent with what an actual push would look like.
Reviewed By: ikostia
Differential Revision: D20623966
fbshipit-source-id: 2209877e9f07126b7b40561abf3e6067f7a613e6
Summary:
This makes it easier to realize if you used the wrong entry ID when replaying
(instead of telling you the bookmark isn't at `None` as expected, it tells you
the Hg Changeset could not be mapped to a Bonsai).
Reviewed By: ikostia
Differential Revision: D20623847
fbshipit-source-id: aaa66e7825f12373742efd4f779ae20ff21f0b46
Summary:
This updates unbundle_replay to account for pushrebase hooks, notably to assign
globalrevs.
To do so, I've extracted the creation of pushrebase hooks in repo_client and
reused it in unbundle_replay. I also had to update unbundle_replay to no longer
use `args::get_repo` since that doesn't give us access to the config (which we
need to know what pushrebase hooks to enable).
Reviewed By: ikostia
Differential Revision: D20622723
fbshipit-source-id: c74068c920822ac9d25e86289a28eeb0568768fc
Summary:
This adds a unbundle_replay Rust binary. Conceptually, this is similar to the
old unbundle replay Python script we used to have, but there are a few
important differences:
- It runs fully in-process, as opposed to pushing to a Mononoke host.
- It will validate that the pushrebase being produced is consistent with what
is expected before moving the bookmark.
- It can find sources to replay from the bookmarks update log (which is
convenient for testing).
Basically, this is to writes and to the old unbundle replay mechanism what
Fastreplay is to reads and to the traffic replay script.
There is still a bit of work to do here, notably:
- Make it possible to run this in a loop to ingest updates iteratively.
- Run hooks.
- Log to Scuba!
- Add the necessary hooks (notably globalrevs)
- Set up pushrebase flags.
I would also like to see if we can disable the presence cache here, which would
let us also use this as a framework for benchmarking work on push performance,
if / when we need that.
Reviewed By: StanislavGlebik
Differential Revision: D20603306
fbshipit-source-id: 187c228832fc81bdd30f3288021bba12f5aca69c
Summary: I'd like to get the timestamps here without needing to clone them.
Reviewed By: StanislavGlebik
Differential Revision: D20603308
fbshipit-source-id: 2d8f72b4fb3a3eed33b58dc2f0fb1a857bb3f5b9
Summary:
This updates pushrebase hooks to allow into_transaction_hook to be async (the
reason I hadn't made it async is because it hadn't been needed yet).
Currently, this is a no-op, but I'm going to use this later in this stack.
Reviewed By: StanislavGlebik
Differential Revision: D20603307
fbshipit-source-id: 79651184dbe08322c4cab03d7119a31036391852
Summary:
A few of our tasks failed on startup and most likely it was during warmup
though we are not sure (see attached task).
Let's add move logging
Reviewed By: farnz
Differential Revision: D20698273
fbshipit-source-id: 4facd21a94d2917103e417a014b820c893da4718
Summary:
The _NAME_TO_VALUES and _VALUES_TO_NAMES fields are almost deprecated,
so instead use TEnumTraits.
Reviewed By: simpkins
Differential Revision: D20708754
fbshipit-source-id: 2a5107ee1299b4160f38ca6cddf90416745f6141
Summary:
The IdDag provides graph algorithms using Segments.
The IdMap allows converting from the SegmentedChangelogId domain to the
ChangesetId domain.
The Dag struct wraps IdDag and IdMap in order to provide graph algorithms using
the common application level identifiers for commits (ChangesetId).
The construction of the Dag is currently mocked with something that can only be
used in a test environment (unit tests but also integration tests).
This diff also implements a location_to_name function. This is the most
important new functionality that segmented changelog clients require. It
recovers the hash of a commit for which the client only has a segmented
changelog Id. The current assumption is that clients have identifiers for all
merge commit parents so the path to a known commit always follow a set
of first parents.
The IdMap queries will have to be changed to async in the future, but IdDag
queries we expect to stay sync.
Reviewed By: quark-zju
Differential Revision: D20635577
fbshipit-source-id: 4f9bd8dd4a5bd9b0de55f51086f3434ff507963c
Summary:
Making room to new components. Individual files will be easier to read than
all code in lib.rs.
Reviewed By: krallin
Differential Revision: D20635579
fbshipit-source-id: 3966b03658b039e9d46e400a00fc50416d60780b
Summary: The interesting observation is that InProcessStore is not public.
Reviewed By: quark-zju
Differential Revision: D20635578
fbshipit-source-id: a0149929c8059ff77f047fd385bf3b26dc738dfd
Summary:
Similar to the previous diff. Sometimes the narrow-heads config is on but the
dependencies are off (ex. remotenames, visibility). In that case narrow-heads
should be considered off. The repo init function takes care of that but code
paths before repo init needs special care.
Reviewed By: simpkins
Differential Revision: D20699684
fbshipit-source-id: befac335bde0be6b8a42a5a9f7cee1edbbef3c55
Summary:
Some tests fail because `hg init` sets the wrong initial store requirements and
then perform a narrow-head migration down, which prints extra messages. Fix them
by making sure the initial store requirements are the same as what the migration
code path expects.
Reviewed By: simpkins
Differential Revision: D20698637
fbshipit-source-id: 1422a4ea78222617d0e3f9631ad883d5a3fe6bb7
Summary:
Update the LinuxProcessFinder code to ignore the command line arguments when
looking for the state directory, and instead only look at the open file
descriptors.
The path found from the command line arguments could be incorrect: if the
EdenFS state directory had been deleted or renamed the information extracted
from the command line would be incorrect.
The information found by examining the open file descriptors will point to the
new directory location if the state directory has been renamed. We can also
tell if the lock file had been deleted in this case as well. This adds
another field to the `EdenFSProcess` class to indicate if the lock file is no
longer valid.
Reviewed By: wez
Differential Revision: D20638621
fbshipit-source-id: e9db1390f97baf50ad3246de4a964eaaf92622e7
Summary: This diff makes it possible to relay on the thrift structures from configerator in OSS.
Reviewed By: ahornby
Differential Revision: D20279457
fbshipit-source-id: 39df1c7a6f78b10a0a5bdeeebe476249ab674cc8
Summary:
This shifts the responsibility of mocking all facebook-specific code
to mononoke's sql_ext crate. If OSS code calls into any of that code it will
most likely result in a panic.
Reviewed By: ahornby
Differential Revision: D20247580
fbshipit-source-id: 43f158d91aa32adaa5df6e3786243fb89c9ce961
Summary:
See bottom diff of the stack for overview.
This diff asyncifies the resolver's entrypoint: `resolve` fn, and provides a compatibility shim `resolve_compat` to call from old-style code. Alternatively, we could just do `async move { resolve(...).await }.boxed().compat()` at a callsite. This did not seem too important to do one way or the other, let me know if you feel strongly about it.
Reviewed By: StanislavGlebik
Differential Revision: D20634406
fbshipit-source-id: ee3c73a7a2c65c333e95194bd90ca7330b225011
Summary: See bottom diff of the stack for overview.
Reviewed By: StanislavGlebik
Differential Revision: D20633227
fbshipit-source-id: 16b3f3a764a75261da0585c9724a17853e865681
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `resolve_push`.
Reviewed By: StanislavGlebik
Differential Revision: D20620100
fbshipit-source-id: 1109933e388d9485f42c63638621a7b9227f157f
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `resolve_bookmark_only_pushrebase`.
Reviewed By: StanislavGlebik
Differential Revision: D20610253
fbshipit-source-id: 2a79ac9e8bdca18401ed95d98a0e1b3e92fee4fe
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `return_with_rest_of_bundle`.
Reviewed By: StanislavGlebik
Differential Revision: D20605807
fbshipit-source-id: 0df7d18d06720ff166cdc3e9b981932a819cb0aa
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `next_item`
Reviewed By: StanislavGlebik
Differential Revision: D20605817
fbshipit-source-id: 99fc0100736f0a9307448c6f2ead91da81a531cb
Summary:
See the bottom diff of the stack for overview.
This diff asyncifies two fns: `maybe_save_full_content_bundle2 ` and `is_next_part_pushkey`
Reviewed By: StanislavGlebik
Differential Revision: D20605814
fbshipit-source-id: 1daea901e7620638fa9b8d0b69c18b0ff4e967da
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `maybe_resolve_commonheads` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605808
fbshipit-source-id: 6b628a4c66970e468839732db8ffb11c961be591
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `maybe_resolve_pushvars` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605812
fbshipit-source-id: e68f9d878a294a9980e53b104aa1035c0d47ae65
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `maybe_resolve_changegroup`.
Reviewed By: StanislavGlebik
Differential Revision: D20605810
fbshipit-source-id: fbf5e9d93b355dfa23a3a7657edb96b033535f9d
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `ensure_stream_finished` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605805
fbshipit-source-id: 853a6d5d1afeee0f8f841eec51302fb3ceb701c7
Summary:
See bottom diff of the stack for motivation.
This diff asyncifies `resolver_multiple_parts` and `maybe_resolve_pushkey` functions.
Reviewed By: StanislavGlebik
Differential Revision: D20605815
fbshipit-source-id: 90768f4495632ec83b79ca9fcf982b0ec5c277cf
Summary:
In rare situations users end up manually deleting or removing their `.eden`
state directory without ever killing their running `edenfs` process. This can
leave this old process running indefinitely despite the fact that it's state
directory is no longer present (or has perhaps even been replaced with new
data).
This updates edenfs to periodically check if its lock file is still valid, and
quit if it isn't. This will help prevent old `edenfs` processes from running
indefinitely after their state directory is no longer valid.
Reviewed By: wez
Differential Revision: D20613841
fbshipit-source-id: d9a3a1e7e9b05806e086e794ebbc36e1cc71831a
Summary:
One of the main drawback of the current version of repack is that it writes
back the data to a packfile, making it hard to change file format. Currently, 2
file format changes are ongoing: moving away from packfiles entirely, and
moving from having LFS pointers stored in the packfiles, to a separate storage.
While an ad-hoc solution could be designed for this purpose, repack can
fullfill this goal easily by simply writing to the ContentStore, the
configuration of the ContentStore will then decide where this data will
be written into.
The main drawback of this code is the unfortunate added duplication of code.
I'm sure there is a way to avoid it by having new traits, I decided against it
for now from a code readability point of view.
Reviewed By: DurhamG
Differential Revision: D20567118
fbshipit-source-id: d67282dae31db93739e50f8cc64f9ecce92d2d30
Summary: There is no need to pass the path twice, once is enough.
Reviewed By: DurhamG
Differential Revision: D20567117
fbshipit-source-id: 169a088aa7558b4c25f8fbdecfe59bdd0d7575ef
Summary:
While the primary (for now) way of addressing an LFS blob is via its sha256,
being able to address them via different hash schemes (sha1 for Eden/Buck,
blake2, etc) will be helpful down the line. Thus, let's store a HashMap of
ContentHash in the pointer store.
Reviewed By: DurhamG
Differential Revision: D20560197
fbshipit-source-id: 8bdc4fc4cd7fc19c7eed6a27d11953c4eedf9195
Summary:
The hgcache will soon contain an LFS subdirectory, this is uninteresting for
most of the tests, let's change them a bit so they only look at the packs
subdirectory.
With this, enabling remotefilelog.useruststore only has 2 failures. One due a
difference in handling corrupt data, with the ruststore being more explicit
that some data is missing instead of a generic "stream ended unexpectedly". And
the other due to some ordering difference when dealing with LFS data. The
latter will go away with the new LFS implementation.
Reviewed By: DurhamG
Differential Revision: D20543146
fbshipit-source-id: 09c76cacfb4687fd699b82cdf5057665ac6bd521
Summary: No locking is required for this one due to being loose files on disk.
Reviewed By: DurhamG
Differential Revision: D20522890
fbshipit-source-id: 72b7ebc063060a89f54976a1128977a3b7501053
Summary:
In preparation for building a single binary that can run inside and
outside of the corp environment, restructure EdenMain to allow
selecting the implementation at runtime.
Reviewed By: simpkins
Differential Revision: D20613464
fbshipit-source-id: 56dbf006f6e9fd74af9c490a47c832a5367d3d3d
Summary:
See bottom diff of the stack for overview.
This diff asyncifies `resolve_b2xtreegroup2`.
Reviewed By: StanislavGlebik
Differential Revision: D20605806
fbshipit-source-id: 2e667d19f2014d051d25a74353e9ebd2e6a93c72
Summary:
See bottom diff of the stack for the overview.
This particular diff asyncifies the `maybe_resolve_infinitepush_bookmarks` fn.
Reviewed By: StanislavGlebik
Differential Revision: D20605816
fbshipit-source-id: 11b6e9c5dd7423bcc4ecc988efd581a3e970ccdc
Summary:
See the bottom diff of the stack for the overview.
This diff specifically migrates the `upload_changesets` function.
Reviewed By: StanislavGlebik
Differential Revision: D20605809
fbshipit-source-id: 36a11a72fb828d494bd18c7737e2682cb3b7cb9a
Summary:
See D20605813 for the overview of the stack.
This diff migrates a few leaf functions.
Reviewed By: krallin
Differential Revision: D20605811
fbshipit-source-id: 2f5d5e5fba3a00afd61a4eb58c505658ac82943a
Summary:
A wider goal of this stack is to migrate `repo_client/unbundle/src/resolver.rs` to async/await and new futures.
The approach is as follows:
- rename old futures upon import into `OldFuture`, `OldBoxFuture`, etc. so that it is easily visible where we use what [this diff]
- implement a bunch of mechanical conversions of continuation-passing-style code to the imperative async/await based code without worrying about clones-vs-references or excessive boxification. Keep individual diffs as small and as mechanical as possible, so that it is easier to review.
- once the `resolve` fn is migrated, introduce `resolve_compat`, which is used from `repo_client/client/src/mod.rs`
- then go through the codebase and see where we can remove clones of resolver/ctx and excessive `boxed()/boxify()` if any are left
Note: `Bundle2` is an `OldStream` and I postpone its migration till the high-level structure of the `resolver.rs` is migrated, since the main value is in allowing imperative-style code in the file
Reviewed By: krallin
Differential Revision: D20605813
fbshipit-source-id: 32255d7b3573f87f74a496e6e40b842e553242a7
Summary:
SCS log accepts two dates/timestamps to filter history by the commit creation time. Each timestamp is a `i64` and zero or negative timestamp still represents a pretty valid time in past.
The time filters are pretty expensive: they require sequential changeset info fetching and checking the date.
It turned out that some of the requests have time filters but seem not meaning it: their both after and before timestamps equals to zero. And there are lots of such queries: https://fburl.com/scuba/mononoke_scs_server/g345na72. This cause SCS log traverse the whole history for a path, which turns into hours of fetching cs infos and fastlog batches.
I've decided to consider a valid timestamp only the timestamp greater than 0: only after 1970-01-01 00:00:00 UTC.
Reviewed By: StanislavGlebik
Differential Revision: D20670210
fbshipit-source-id: f59c425779a37ecac489dbba2ed3fd547987ee62
Summary: Enable the CLI tests on macOS when run with Buck.
Reviewed By: wez
Differential Revision: D20524347
fbshipit-source-id: cf3e302256b6b0e6958999cf83c5be5d48f65907
Summary:
In the open source and macOS builds of EdenFS, enable Thrift function
statistics.
Reviewed By: simpkins
Differential Revision: D20550819
fbshipit-source-id: bb6fe2f1f413b89f344cb3725a7382fdb3d50a5d
Summary:
Fix a problem with the handling of short reads
which would result in returning trailing garbage.
Implement setting timeouts.
Implement sendall (not strictly needly, but added here
for symmetry with the pywatchman equivalent code.
Reviewed By: chadaustin
Differential Revision: D20584508
fbshipit-source-id: a27a7392b3335458c5e56a91376220ab045ad30b
Summary:
This enforces certain selective pull logic in core. Namely, rewrite `pull -r X`
to `pull -r X -B master`.
Unlike selectivepull in remotenames, `pull` (pulls everything) won't be
rewritten to `pull -B master` (which pulls less commits and names).
Therefore this change always adds more commits to pull, and therefore should not
break existing users. Eventually we want the "not pulling everything" behavior,
but right now this just fixes `pull -r X` to also update important remote names.
Reviewed By: markbt
Differential Revision: D20531121
fbshipit-source-id: af457b5ddb1265b61956eb2ee6afb7b7208293e0
Summary:
fixes up some functions to avoid narrowing warnings
and others to adopt `folly::to_narrow` to accept them.
This reduces the number of warnings in the windows build.
Reviewed By: chadaustin
Differential Revision: D20562026
fbshipit-source-id: f18ba50f914e142415d1efefebc6297e8c68d38e
Summary:
I noticed that the output was `{backing_repo}` in a couple
of user reports, rather than the path to the backing repo.
Reviewed By: genevievehelsel
Differential Revision: D20669756
fbshipit-source-id: c9f1dbb4f4b2ad3de03c54f1e3cb5f688ee433ac
Summary:
This is the first step of asyncifying the `bonsai_verify`
command. For this diff I'm doing a literal translation: convert a big
legacy future into a big async future.
Reviewed By: farnz
Differential Revision: D20632015
fbshipit-source-id: 9e663d36c25c6f0c90db88ff84e550fd04bb2ab7
Summary:
Changed all of the high-level interfaces and most of the
code to use new-style futures.
Reviewed By: farnz
Differential Revision: D20625467
fbshipit-source-id: e85a87f9acdaaf7671617f3f76e66a316884977c
Summary: Now we have functions to get selectivepull names in core. Use that instead.
Reviewed By: markbt
Differential Revision: D20531118
fbshipit-source-id: a0c20c491baf1b0ad71e80f870703bb4b983f19c
Summary: This makes it possible for core to always pull related bookmarks.
Reviewed By: markbt
Differential Revision: D20531120
fbshipit-source-id: 52f0834b517e03567e240f57616370d79a227abe
Summary:
This makes it possible to use template like `{remotenames}` or revset like
`remotenames()` without enabling the remotename extension.
Rarely used revsets like `upstream()` and `pushed()` are not moved.
Reviewed By: markbt
Differential Revision: D20529360
fbshipit-source-id: ea95b3324f974e112909cdd79ce662940a4f9b7c
Summary:
This makes it possible to resolve remotenames without enabling the remotenames
extension.
The config check `if repo.ui.configbool("remotenames", "bookmarks")` is dropped
intentionally as we only use remotenames for bookmarks, not named branches.
Since this only enables resolving more names, without disabling or changing
other features, and the remotename namespace priority is lower than local
bookmarks (ex. if a local `master` exists, then `master` will be using the
local bookmark, not the hoisted remote name), it should not cause breaking
changes.
Reviewed By: markbt
Differential Revision: D20529359
fbshipit-source-id: 4126faee1bb7f43ba547fab05dd6197b2e65c1fc
Summary:
This moves part of remotenames that provides the `repo._remotenames` object to
core. It should not change behaviors but merely makes `repo._remotenames`
available in core.
Reviewed By: markbt
Differential Revision: D20529358
fbshipit-source-id: 11c8538a0190101b09a4cb082018e73643a257e2
Summary:
All the `blobimport.rs` logic has now been asyncified. We still have
a `.compat()` sandwich for now around the `.traced()` call that we can
get rid of when tracing::Traced moves over to the new API
Reviewed By: farnz
Differential Revision: D20615191
fbshipit-source-id: da3f8a66b18d0dbef6895f6f08952b563f7f8680
Summary:
This diff creates a new async function `run_blobimport` to handle
most of the logic that was formerly in `main`, so that we can use
async-style code.
Reviewed By: farnz
Differential Revision: D20614860
fbshipit-source-id: bd8d3e230f53201d2dec53a3033f048572f2a67e
Summary:
I'm about to add another reference to our service name, so instead of
hardcoding "Eden", add a kServiceName constant. While I'm at it, I
renamed it to EdenFS.
Reviewed By: genevievehelsel
Differential Revision: D20500668
fbshipit-source-id: 4974363156ba2934a1a5cd86d9e8fd7e93d89181
Summary:
This diff - which asyncifies `prefetch_content` - isn't a big win on readabiliy,
but by switching to async (taking references) and then changing upstream functions
to references we're able to save two layers of cloning in the `warmup` functions,
which ought to reduce heap allocations
Reviewed By: farnz
Differential Revision: D20612696
fbshipit-source-id: a52d6246789e964d3b02d0bdc1bfafbded8f25dd
Summary:
Straightforward asyncification, which lets us pass down refs and save
a couple of clones.
Reviewed By: farnz
Differential Revision: D20612725
fbshipit-source-id: 83aa30e2954101faa40d9427ad267e8e42f57d46
Summary:
The api is now asyncified, and we no longer have to clone when calling
`subcommand_tail`.
The inner stream logic is still using the old streams api - untangling that code was difficult for me, so I'll try to tackle it in a later, isolated diff
Reviewed By: farnz
Differential Revision: D20611556
fbshipit-source-id: acd79f8048b5e611c316fc1986160b6395ee6fb9
Summary:
Asyncified `subcommand_backfill`. Some of the inner code is still using
old futures because I was not able to find a new-futures version of
`future_stats::Timed`, but most of the logic is using new futures /
new streams apis.
Reviewed By: farnz
Differential Revision: D20612724
fbshipit-source-id: 00e524f606fc02f4706c02347c0bd6feba46c3eb
Summary:
While failures in the Rust updater aren't expected, at least one valid case
requires requires retrying the operation in Python: old-style LFS pointers.
When these are stored in packfiles/indexedlog, only the Python code knows how
to deal with them, and thus the operation needs to be retried there.
Reviewed By: DurhamG
Differential Revision: D20603709
fbshipit-source-id: 7d24ba573f0ff540906d909f1b4440fd4d3469a6
Summary:
The ContentStore cannot deserialize LFS pointers stored in packfiles, to avoid
potential damage, let's refuse to update LFS blobs. A proper solution will be
built in a separate diff.
Reviewed By: DurhamG
Differential Revision: D20576575
fbshipit-source-id: 4e4ce6a9432157e2ce69881c0079e943ea3f3acd
Summary:
Instead of having the magic number 0x2000 all over the place, let's move the
logic to this method.
Reviewed By: DurhamG
Differential Revision: D20637749
fbshipit-source-id: bf666f8787e37e6d6c58ad8982a5679b7e3e717b
Summary:
On Unix, pretend that NTFS doesn't support symlinks. While this isn't
technically true, NTFS on Linux is only used to alleviate performance issues
with `hg update` on Windows. With the pyworker code, I'm expecting these
performance issues to disappear allowing this code to be removed.
Reviewed By: ikostia
Differential Revision: D20527976
fbshipit-source-id: 4194f4b5af065de2e293b41b9d03e9d4ab6ea006
Summary:
Move them into a VFS struct, a future step will move the VFS code into its own
crate.
Reviewed By: DurhamG
Differential Revision: D20527977
fbshipit-source-id: 3250b05840688db72e1c43c72ec6defbc7f20851
Summary:
In this diff I create a function to handle the logic for choosing a subcommand
to run, returning a Future that we execute. By using `helpers::block_execute`
we're able to extract the spinning up of the fb303 server, which is the key
to making this an async function.
Within each of the subcommands in the new function `choose_subcmd`, this change
also asyncifies the logic inside the `async` block.
Reviewed By: krallin
Differential Revision: D20590742
fbshipit-source-id: a6298e6da3827638cc23369f72ebdd66b5051768
Summary:
This diff moves blame integration tests out from the main `test-scs.t`.
The change makes `test-scs.t` be able to complete and not time out anymore.
Reviewed By: ikostia
Differential Revision: D20629281
fbshipit-source-id: 67ba047442e7216a8addd0945c94d2f932eca08a
Summary: This diff moves lookup integration tests out from the `test-scs.t`.
Reviewed By: krallin
Differential Revision: D20620537
fbshipit-source-id: a8e1020901271b0e66dd4caa43ad3eddbf887a41
