Summary:
Running the glob benchmark, creating the future causes EdenFS's CPU usage to
explode to 1000% for a single threaded glob query. Removing this significantly
reduce the CPU usage to 100% while not affecting performance, despite what the
comment claimed.
Reviewed By: kmancini
Differential Revision: D28617400
fbshipit-source-id: 42ff3401ff1da9d7a7f63bd1c91c8b5b30b182fd
Summary:
`rm -A` means removing files that are "deleted" (`rm`-ed but not `hg rm`-ed).
It does not need to list clean files. Listing clean files can be very slow
in a large repo.
Avoid listing clean files so `rm -A` can be faster.
This has a side effect that we no longer maintain the exit value (0: repo
becomes empty, 1: repo is not empty) like before. But I guess nobody really
cares the 1 exit value (and it does not really make sense in the `rm -A`
case).
Reviewed By: DurhamG
Differential Revision: D28622558
fbshipit-source-id: 2087d6508932905564a8307e9438895538ecede9
Summary:
This is the last piece that allows both FUSE and NFS to not allocate futures
when replying to getattr calls.
Reviewed By: chadaustin
Differential Revision: D28396819
fbshipit-source-id: 9a4f8d5d03991dbacc4c56116bd56a38773dcd63
Summary:
Since Buck issues glob requests via Watchman, I was curious to understand the
overhead that Watchman adds to see if that is worth optimizing or bypassing.
From the results shown below, it looks like talking to EdenFS directly would
yield a ~4x globbing improvement.
Reviewed By: chadaustin
Differential Revision: D28644517
fbshipit-source-id: 1aa4c43d45248626adde31491fe40836ad6c436a
Summary:
When walking in OldestFirst direction, if we restart from a checkpoint there are some nodes that will never be part of a chunk visited by the current run. e.g. if we start at commit E and parent points to commit D it will be deferred, but as we are walking from old towards new, commit D will never be visited.
Fix is to not record deferred edges when walking in OldestFirst direction.
Previous similar change did this for unodes and filenodes, but also needed here.
Differential Revision: D28749514
fbshipit-source-id: e14a67cad38dfebba7a0b4cf9ffa3de63d8c41ce
Summary:
The usage of bytes for paths and environment variables makes this entire file hacky and makes it not work on Windows. Let's remove all of that.
We still use bytes for test output and other file content type cases.
Reviewed By: andll
Differential Revision: D28227825
fbshipit-source-id: b15993601db501160c9fa4eb2463678cde1fa554
Summary:
## Idea
When `fetch_skiplist_index` is called, it will also create a task in the background that every 2 minutes, reloads the skiplist from the blobstore using the same key.
So that we can change the skiplist, we use an `ArcSwap` around it. I noticed a similar thing is already done in [here](https://fburl.com/code/non7ht60), maybe that's something to refactor into a single abstraction later.
It's possible to configure interval and enable/disable via [`MononokeTunables`](https://www.internalfb.com/code/fbsource/[8db13bf20639]/fbcode/eden/mononoke/tunables/src/lib.rs?lines=67), defaulting to enabled with 120s interval (no particular reason for the chosen interval, I'm happy to change it).
This some stuff that I won't do by default because I'm not sure it's necessary but feel free to ask and I'll add it:
- Add random delay when waiting for next iteration, so all repos don't reload mostly in sync
Reviewed By: markbt
Differential Revision: D28685596
fbshipit-source-id: 6ce2388ba92da07d5a968c1bdc446b73211155d3
Summary:
This diff has a few **refactor only** changes. They make the next diff, with actual new behaviour, smaller and easier to review.
- Move most behaviour of `fetch_skiplist_index` inside `SkiplistIndex::from_blobstore`.
- Make most of the loading logic actually return `SkiplistEdgeMapping`, the actual important content inside `SkiplistIndex`. Will be useful on the next diff because we'll reload only this part.
- Add some `derive(Debug)`s
- Make constructors a bit simpler by calling each other. Also makes next diff simpler as we'll add more fields to the struct.
Reviewed By: markbt
Differential Revision: D28711087
fbshipit-source-id: bfb5b0bab932e5079b27cd39d2001a62bd047cb9
Summary:
Refactor only.
Moves the loading behaviour inside a new struct, which will be easier to reuse on the following diffs, where I'll reload the skiplist.
Reviewed By: StanislavGlebik
Differential Revision: D28681737
fbshipit-source-id: f943ccb7623a82b2de24ef6afc2485afa376d357
Summary:
When walking in OldestFirst direction, if we restart from a checkpoint there are some nodes that will never be part of a chunk visited by the current run. e.g. if we start at commit E and walk filenode that points to commit D it will be deferred, but as we are walking from old towards new, commit D will never be visited.
Fix is to not record deferred edges when walking in OldestFirst direction.
Reviewed By: farnz
Differential Revision: D28714771
fbshipit-source-id: 11faa2be62e588a284045bbdfffeb8f2a4efdf70
Summary:
An operation that takes a result of computations and saves it so it can be
polled.
Reviewed By: krallin
Differential Revision: D28606343
fbshipit-source-id: 4fdd48f40812c55842669ea3f62865fc8244f6cc
Summary:
This abstracts the constant poolling for new items that megarepo workers will
be doing.
Reviewed By: krallin
Differential Revision: D28606344
fbshipit-source-id: 159c78c207956bcf893ffb68f7f7e93fc2426c8b
Summary: a dequeue operation abstracts popping a request from the queue for processing
Reviewed By: krallin
Differential Revision: D28606339
fbshipit-source-id: d8eba568eab7b6751e422cd11e8ffa5bcb7712b0
Summary: This diff switches megarepo_api to use the types created in the previous diffs
Reviewed By: krallin
Differential Revision: D28606338
fbshipit-source-id: c048f274395239528b3a0280172dd21052055111
Summary:
Instead of storing params separate types for each type of request let's store
them in a single union type. This is going to simplify things and improve the
type safety (as we will no longer need to trust in the "request_type" string
store in sql to safely deserialize the type).
Reviewed By: krallin
Differential Revision: D28606341
fbshipit-source-id: cfa8ee15213e6582891906e326d0e258c3a918a1
Summary: This operation is primary way of getting new jobs from queue to tailer.
Reviewed By: krallin
Differential Revision: D28441669
fbshipit-source-id: e4563d74d537d826ad4eba654fe583ea55aee974
Summary: We do the same thing in many places which isn't great for code readability.
Reviewed By: krallin
Differential Revision: D28441674
fbshipit-source-id: bec69d3bc8a000ac9442020f5054e5c79b3392de
Summary:
Previously, migrating to lazy means repo requirement changes. This diff uses
the new API to actually make the changelog lazy.
Reviewed By: DurhamG
Differential Revision: D28700896
fbshipit-source-id: 82cfd70645230cd67223195e25ef07ae5abe7df6
Summary:
Some tests would clone using FUSE, which would thus not tests NFS. Let's make
the EdenFS class aware of NFS and force cloning with --nfs in that case.
Reviewed By: kmancini
Differential Revision: D28491492
fbshipit-source-id: ce7d609f187c21937626106a16a1f352ba88ceca
Summary:
With NFS, the server is stateless, and thus to support unlinking opened files,
the unlinked file is renamed to a "silly name". For that reason, EdenFS will
see a .nfsXXX file being created on disk.
Reviewed By: chadaustin
Differential Revision: D28484877
fbshipit-source-id: 0f4b82f97102e6efaf8899f41f644e0f47d96b1d
Summary:
Switch debugrebuildchangelog from using revlog stream clone to lazy segment clone.
This removes the revlog techdebt and can be used as a way to repair
repos with broken segmented changelog. As we migrate off double-write backend we
can no longer migrate down to revlog then migrate up, and a full reclone can be
slow. So a partial reclone command that just recreates the segmented changelog
seems useful.
This command is one of the two commands that handle emergency situations
when segmented changelog related logic goes wrong. The other command
is the emergency clone mode, added by D27897892 (d1413bbbad), which assumes everything
related to segmented changelog is broken on Mononoke side and we still
need to commit and push. This command relies on segmented changelog
related features, such as hash<->location lookup, and clone on Mononoke
to work properly and the server having a compatible IdMap. So it might
not be able to address all issues if Mononoke goes wrong.
Reviewed By: DurhamG
Differential Revision: D28430885
fbshipit-source-id: 17357a33f6fda4a67d46e2c7e7be6653b530f499
Summary:
Use the interruptible block_on API so the Python methods can be interrupted by Ctrl+C.
This is especially useful if some operation triggers lots of expensive network fetches.
Reviewed By: DurhamG
Differential Revision: D28723008
fbshipit-source-id: b6c692de6290a49852eabcd960ebd9b2fb68685a
Summary:
This will be used by the next change to test migrating from a non-lazy
changelog to a lazy changelog actually makes commits lazy.
More commits were added to the graph to test laziness. The old graph
does not have commits that will be made lazy by the current standard
(parents of merges are not lazy).
Reviewed By: DurhamG
Differential Revision: D28700897
fbshipit-source-id: 527c3ce672327ed5e2398c0d89a8e9e92e5b244f
Summary:
This will be used by the next change to migrate from a non-lazy changelog to a
lazy changelog.
Reviewed By: DurhamG
Differential Revision: D28700898
fbshipit-source-id: ff12754f224586b9d0d62f73b41bbb07fc7a6eea
Summary: These are only used from tests, so tag them as such
Reviewed By: farnz
Differential Revision: D28649988
fbshipit-source-id: a8f32fa85c44ac72765387ed4848edfb2e637ee7
Summary:
For non-UTF8 names, the PathComponent constructor would raise an exception, and
since that exception wasn't caught by the handler itself, it would bubble up to
the RPC server and a generic "server IO error" would be sent back to the
client. Since non-UTF8 names aren't a server error, but an invalid argument, we
should instead return a different error.
Unfortunately, EILSEQ isn't an error that an NFS server can return, instead
let's use EINVAL as the argument is clearly invalid.
Reviewed By: chadaustin
Differential Revision: D28482032
fbshipit-source-id: b59044f1a76f7eac79e2df07356a0aeafa22e3c5
Summary: The test is passing, let's enable it.
Reviewed By: genevievehelsel
Differential Revision: D28482029
fbshipit-source-id: d8efe5fb996deb02d33a38f992e4cb18cbe97f4d
Summary:
The modeToNfsMode simply didn't consider all the mode bits to be translated to
the proper NFS mode bits. It now does.
Reviewed By: chadaustin
Differential Revision: D28459428
fbshipit-source-id: d879fb1be2085e44110ba552bc47d2770637fc86
Summary:
Similarly to the work that kmancini did for the integration tests, this
enables the Mercurial integration tests to run against NFS. Only 3 tests are
failing due to 3 separate reasons: graceful restart isn't support, permissions
given to the clients are wrong, and a lifetime issue for the Nfsd3 object.
As a bonus, I've cleaned the code a bit since TreeOnly was the only supported
configuration (and will likely stay that way).
Reviewed By: chadaustin
Differential Revision: D28459245
fbshipit-source-id: 21f043bb054d684eacaee57179f0bdf4853b0a35
Summary:
An NFS client caches the attributes of files to avoid having to request these
very frequently. What this means is that a file changed by another client (or
by the server itself) may take some time to be reflected on the client, that
time depends on the attribute caching configuration of the mount point.
For EdenFS, files can changed in 2 ways:
- Either it is changed by the user via the mount point,
- Or the user runs an `hg update`
For the first one, the client will simply update its attributes appropriately,
but for the second one, the cached attributes will only be updated when the
user does opens the file, any calls to stat prior will return the old
attributes. Since EdenFS runs on the same host, we can force the attributes
caches to be discarded by simply issuing an open call on the file that changed.
Reviewed By: chadaustin
Differential Revision: D28456482
fbshipit-source-id: 91022d35a33e436c47d94403d0c139992f880cf9
Summary:
If a patch declared the length of it's last hunk as N lines, but it
only contained N-1 lines, the Rust code would enter into an infinite loop. This
could happen if a text editor remove the trailing spaces from a patch file.
Let's fix it and add a test
Reviewed By: kulshrax
Differential Revision: D28683977
fbshipit-source-id: 0a999ae108676531a2cf18e77a3b426ba4647164
Summary: They are not being used anymore AFAIK, so it's best to get rid of dead code.
Reviewed By: farnz
Differential Revision: D28677902
fbshipit-source-id: a2de5a1ca3908128b57afc07227a46bba0e45b1a
Summary:
We are not using the async methods to run the megarepo requests yet but we'll
start doing so soon.
For now let's just fetch the current value of the bookmark and return it to the
client.
Reviewed By: mitrandir77
Differential Revision: D28679134
fbshipit-source-id: 3ab390ce1ff995c98ea150aa309c16c02dcaee5c
Summary:
This is the very first version of add_sync_target method. It's very naive - it
just creates a single merge commit, and it doesn't handle all the necessary
functionality (e.g. linkfiles) and it doesn't do all the necessary checks yet.
TODOs show the missing functionality.
However this implementation it should be good enough as the first version, and
we can continue to extend it.
Reviewed By: mitrandir77
Differential Revision: D28677365
fbshipit-source-id: a8af6b92ced285ce4d012843f591d5c29584669d
Summary: It will be used in the next diffs, let's move it to a common file
Reviewed By: mitrandir77
Differential Revision: D28675768
fbshipit-source-id: 78c21e378053a157d09ee9d032576a434a52c912
Summary: They will be used in the next diffs
Reviewed By: mitrandir77
Differential Revision: D28675756
fbshipit-source-id: 2f41191cb83921fb3ff3374c0ccb7a884b7b3a25
Summary: Sometimes things take longer, make sure we are able to distinguish whether that's due to networking, tls handshake, http parsing, or Mononoke wireproto handling.
Reviewed By: markbt
Differential Revision: D28705508
fbshipit-source-id: 1bafda7fc447f2e429690f47fe7ab81cec511494
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: fanzeyi
Differential Revision: D28663227
fbshipit-source-id: cee18ec28283c0aef18151465a3c13e9be604d7d
Summary:
Extends the `FileScmStoreBuilder` to construct two new indexedlog stores for caching aux data. The stores will be created in a directory adjacent to the normal non-LFS indexedlog stores.
Currently aux data stores will not be constructed for production users, a configuration option will be introduced to gate this before `.store_aux_data()` is called in the `filescmstore` constructor bindings.
Reviewed By: DurhamG
Differential Revision: D28689693
fbshipit-source-id: e3ad1594e5beee00b1a8b9fe489e3b6af3a2e93e
Summary:
Modify `FileStore` to introduce basic aux data fetching. Aux data is currently read from a separate IndexedLog store, serialized with `serde_json` (chosen for expediency / ease of debugging, I intend to optimize the storage format before releasing this, at the very least to avoid unnecessarily serializing the key path).
Currently aux data fetching will never succeed, as aux data fetching is not supported in the EdenApi "files" API and nothing else exists to populate the local aux data stores. Later in this stack, computing aux data (currently only content sha256) to populate the aux data storage is implemented.
Reviewed By: DurhamG
Differential Revision: D28526788
fbshipit-source-id: c8e21a1377689d7913a68426a3a480d53148da66
Summary:
Simplify tracking of incomplete fetches in preparation for attributes support in the next change.
Now, all keys which have not been completely and successfully fetched are recorded in `pending`, and are removed only when the complete fetch is recorded in `found`. Keys are now removed from `lfs_pointers` and `pointer_origin` as they are completed, as they aren't needed for anything other than fetching from local LFS and remote LFS respectively.
Reviewed By: DurhamG
Differential Revision: D28546515
fbshipit-source-id: c657e5c6350cadc8da970f57bb7694ed71022efb
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28610639
fbshipit-source-id: c85535669f84695217125af5de7cc4848fba6b2d
Summary: FileDescriptor.cp throws some warning on comparison of signed long and unsigned long. So fix these warning and follow how ```folly/portability/SysUio.cpp:doVecOperation``` did
Reviewed By: fanzeyi
Differential Revision: D28608581
fbshipit-source-id: 3b6e35e9764548ce470634f16a1f6eec5118d7ed
Summary:
Now metalog can no longer be `None`. Let's just remove logic handling its
`None` case.
This changes the commitcloud-sync-race test because the metalog itself has
internal locking and changes are atomic.
Reviewed By: DurhamG
Differential Revision: D28595292
fbshipit-source-id: bd9851f5f3bb25f28f15d673f608af2863953c46
Summary:
fncache and store have been default on for years. Enable them unconditionally.
This also makes sure that metalog is always available.
Practically, the only place that does not use fncache is hgsql server repos and
they are irrelevant now.
Reviewed By: DurhamG
Differential Revision: D28595289
fbshipit-source-id: 32b9906c179518acdb17a206b54f98a3dc994921
