Summary: 2 seconds is apparently too short for some of the big directory and files we have in our repository.
Reviewed By: wez
Differential Revision: D15387381
fbshipit-source-id: 6ebf850c5fbad8d60c2e8c9f8cb19b1c28297d47
Summary:
The `LocalStore` constructor was updated to accept a `ReloadableConfig`
argument in D12949577, but this was never used anywhere. Remove it for now to
help simplify the code. If we do want to add it back in the future I think we
should make it required, rather than allowing a null config to be specified.
Reviewed By: wez
Differential Revision: D15350217
fbshipit-source-id: 8571b48dff8c8d079ba6b25821dd0b1d77ffe791
Summary:
Implement the `show_sizes` function which reports the approximate size on disk
used by each column family.
The initial skeleton for this command was accidentally included in D15162813.
Reviewed By: wez
Differential Revision: D15307391
fbshipit-source-id: 03efee939d65500e17b48047552dec1dbb7aff7e
Summary:
Add a `name` field to KeySpaceRecord. This allows the `RocksDbLocalStore` and
`SqliteLocalStore` code to use the name from the `kKeySpaceRecords` data
structure rather than keeping their own array of names that must be maintained
in sync with the `LocalStore` code.
Reviewed By: wez
Differential Revision: D15307390
fbshipit-source-id: b53bca7d53d630f9527a810f145380288c54198e
Summary:
Update the KeySpace enum values to start at 0 instead of 1. This simplifies
the code to avoid having to skip over 0 in a few places.
This also makes the `kKeySpaceRecords` array slightly less confusing. Unlike
the `columns` array used by `RocksDbLocalStore` and the `tableNames` array
used by `SqliteLocalStore`, the `kKeySpaceRecords` array was not previously
indexed by the `KeySpace` enum values.
Reviewed By: wez, strager
Differential Revision: D15307393
fbshipit-source-id: ae8392d02396b4dc3c18e9ee94b198fcbb9b1a34
Summary:
Move the KeySpace record definitions out of LocalStore.cpp and into a header
file. This will make it possible to access the key space information from
other files in the future.
I would like to have access to this data directly in the various LocalStore
implementations, as well as in eden_store_util.cpp. Currently both
RocksDbLocalStore.cpp and SqliteLocalStore.cpp have their own copies of key
space information that should really be shared.
Reviewed By: chadaustin, strager
Differential Revision: D15307392
fbshipit-source-id: ad7cdc705d3dc9834592fe0f14b38d82f91a3b77
Summary:
Now Eden on Windows could be build using
python.exe .\opensource\fbcode_builder\getdeps.py build eden
Reviewed By: strager
Differential Revision: D15251103
fbshipit-source-id: 8ac1097b234bb27ce478101bfbc9591251e2e359
Summary: MSVC doesn't like the ImporterOptions declared as class and defined as struct.
Reviewed By: wez
Differential Revision: D15251102
fbshipit-source-id: e7725df5db8ee39a20a1940a8d045369800aa4c3
Summary:
Move the `ReloadableConfig` implementation out of `ServerState` and into the
`ReloadableConfig` class itself. This also updates `ServerState` to simply
contain a `ReloadableConfig` member variable rather than inheriting from
`ReloadableConfig`.
This makes it easier to build other utilities that need a `ReloadableConfig`
object but do not have a full `ServerState` object, such as the
`eden_store_util` class and the `zero_blob` helper tool used by the
integration tests.
Reviewed By: chadaustin
Differential Revision: D15162813
fbshipit-source-id: 1ebc0ac6c475f4aa1090dd933ebce16bc222674c
Summary:
beholdunittests
This enables some plumbing for running some of the
tests using the gtest/gmock machinery in cmake.
Part of this diff is removing the FindGMock.cmake file from the
eden repo; we now pull this in from the shared cmake library
that is populated by shipit.
Reviewed By: simpkins
Differential Revision: D14993344
fbshipit-source-id: 51caf9518c7f3a083a3b90cda10324c3a8170359
Summary:
This argument is unnecessary since D14222323, which should be deployed
everywhere by now. Passing in the extra repo argument forces hg to open the
repository object twice, which is unnecessary.
We already set the current working directory to the repository when invoking
the import helper, so hg will find the repository correctly from that.
Reviewed By: pkaush
Differential Revision: D15179667
fbshipit-source-id: 838cbee91748c41c713731187608a9823b971a53
Summary:
Update HgImporter to use `hg debugedenimporthelper` instead of the separate
`hg_import_helper.py` script by default on all platforms now. The
`debugedenimporthelper` subcommand has existed in our deployed versions of hg
for a while now.
Reviewed By: pkaush
Differential Revision: D15179668
fbshipit-source-id: 2fb8c4c9f92aed54c84899d6643f746baac73327
Summary:
This diff tries to improve the Thrift Mononoke backing store implementation by:
* Re-creating thrift client for every request since thrift has builtin connection pooling
* Using Eden's CPU thread pool for processing data fetched from remote
* Calling thrift methods within correct executor
Reviewed By: wez
Differential Revision: D15170818
fbshipit-source-id: c8be70755a851f63fb62e4d4f36833703283565e
Summary: We intend to break support for flatmanifest hg, so require that treemanifest is available in the build.
Reviewed By: wez
Differential Revision: D15057150
fbshipit-source-id: 449399cfb9d018f3b722598091eead1bd5d7c77d
Summary: Flatmanifest is on its way out. Remove support for falling back to it if a tree import fails.
Reviewed By: pkaush
Differential Revision: D15056459
fbshipit-source-id: a4df820322ee354d77f50a0ec92e9705d0f152ec
Summary: this fixes linking the tests when building with cmake in D14993344
Reviewed By: chadaustin
Differential Revision: D15167223
fbshipit-source-id: 1dda3beed3b6ff3788f3992783c58b4b92f697f2
Summary:
D15043209 missed fixing up the name of the variable
containing the list of sources.
The result is that we compile the `main` from `eden_store_util`
in to the tests and break gtest discovery in D14993344
Reviewed By: chadaustin
Differential Revision: D15167219
fbshipit-source-id: 95e9dc8ba74965f41bfebed1c1776605d8c9443a
Summary:
D14677339 added tracing to all HgImporter requests. Each trace loads a thread-local variable through EdenStats::getStatsForCurrentThread(). Each of these loads has a small but non-trivial cost.
Because each instance of HgImporter is used only on one thread, each getStatsForCurrentThread() call returns the same EdenThreadStats object. Avoid the thread-local-variable lookups by caching the result of getStatsForCurrentThread() in HgImporter.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14794284
fbshipit-source-id: d1609a1720d44c680dc0ebaa2536779def2a6f37
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
Reviewed By: fanzeyi
Differential Revision: D15078040
fbshipit-source-id: 8ef270e8c687535e9d4aa2208beff4517e121814
Summary:
Add a command to force a call to RepairDB() on the local store.
This is similar to using `ldb repair`, but invokes `RepairDB()` with the same
set of column family options as normally used by edenfs.
Reviewed By: chadaustin
Differential Revision: D15043210
fbshipit-source-id: 2c4c0e2d3410a50cb1e523611f569f1701604ae6
Summary:
Change eden_store_util to take a subcommand argument, and define 3 separate
subcommands: "gc", "clear", and "compact"
The "gc" was the old default behavior. The "clear" and "compact" subcommands
simply clear cached entries and compact the storage, respectively.
This also adds a `--keySpace` argument. If this argument is specified the
command will process only this single key space, instead of all key spaces.
Reviewed By: chadaustin
Differential Revision: D15043207
fbshipit-source-id: 7e76bc89862d9647edde631ed3d82c9afda09f41
Summary:
This moves some logic from the RocksHandles class up to RocksDbLocalStore.
The main thing moved here is the logic to automatically try and repair the DB
if opening fails. This will make it easier in a subsequent diff to make the
repair logic a bit smarter and more aware of our column family semantics.
This keeps RocksHandles a pretty dumb wrapper around the RocksDB object and
column family handles, whose only purpose is to manage destroying these two
things in the correct order.
Reviewed By: chadaustin
Differential Revision: D15043208
fbshipit-source-id: ee2d5619ac7781a892e1ba151712eee9e3ebfb14
Summary:
Move this utility from eden/fs/service to eden/fs/store, and rename it to
`eden_store_util`.
I originally put it in the `service` subdirectory since it depends on library
code from `service/`, and most of the other code in `store/` shouldn't depend
on code in `service/`. However logically it makes sense to be in the `store/`
directory.
I plan to update in the future to be able to perform some additionally utility
operations besides just garbage collection, so rename it to the more generic
`eden_store_util` name.
Reviewed By: chadaustin
Differential Revision: D15043209
fbshipit-source-id: e6c4a874ddd05bd1576ea2e8a6c6156ccdf4628f
Summary:
Build the store tests on mode/mac. RocksDB doesn't build yet (and we
will likely move away from it anyway) so split it into its own target.
Reviewed By: simpkins
Differential Revision: D15002840
fbshipit-source-id: fa0fea1d52df06526032a6fd585e2da7273910ef
Summary: These allow more tests to compile
Reviewed By: chadaustin
Differential Revision: D14994582
fbshipit-source-id: 6b2a0b276fda64c7f27e28ea9e6d548aaaa1db7c
Summary: This diff add four metrics to the stats we are collecting in Eden. Both time cost on calling `get_tree` and `get_blob` from hg and mononoke backing store.
Reviewed By: chadaustin
Differential Revision: D14780349
fbshipit-source-id: f0e902c7f7523fac2ae576a67f0c85555064e26c
Summary:
This effectively reverts D14452214, which caused Eden to write an `id` entry
to each RocksDB column family and then flush the column family each time
edenfs started.
There was relatively little benefit to this in practice. It only matters in
cases where the RocksDB column families never had enough data written to them
to get flushed automatically and then a repair is required.
On the other hand it does have some material downsides: it flushing the column
families can be fairly expensive, and can require a substantial amount of free
disk space. This flush caused some users to not be able to start up edenfs
when they did not have enough free disk space.
Reviewed By: chadaustin
Differential Revision: D14947235
fbshipit-source-id: a29f98163fa87185b028bb47945b6fab75700fd6
Summary:
Change the `eden gc` implementation to use RocksDB's `DeleteFilesInRange()`
function, as well as its new-ish `DeleteRange()` method.
This makes the garbage collection much faster, and also require much less free
disk space than previously.
`DeleteFilesInRange()` asks RocksDB to simply delete all SST files from disk
if they only contain keys in the specified range. Since the range we specify
should include all keys in the DB this should simply drop all SST files for
this column family.
We also call `DeleteRange()` after this, just in case. This API is relatively
recent, and writes a single tombstone saying that the specified range has been
deleted.
Reviewed By: wez
Differential Revision: D14910345
fbshipit-source-id: c76bdc1c8e07cb2def66673ea892e7f455c9dc7a
Summary:
EdenStats is currently an alias for ThreadLocal<EdenThreadStats>. I want to split EdenThreadStats into two structs which are allocated independently, but EdenStats's interface makes this impossible.
Refactor EdenStats from an alias to a class and encapsulate the underlying ThreadLocal<EdenThreadStats> member.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14822272
fbshipit-source-id: 691f4731aa22ecbdcd3535ee0bb0b99c816ffc3d
Summary:
I'm confused by the naming of EdenThreadStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenThreadStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement my preferred scheme: rename ThreadLocalEdenStats to EdenStats.
This diff should not change behavior.
Note: Prior to D14822274, EdenThreadStats was called EdenStats.
Reviewed By: simpkins
Differential Revision: D14822271
fbshipit-source-id: bd20179b1010588e3fc16dc9ed0657d458606f16
Summary: This diff makes `MononokeHttpBackingStore` uses `ServiceAddress` to manage remote address of the Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14855661
fbshipit-source-id: 1c102f7b2c547dc6a5b5c2cba65a513313d90bdc
Summary:
Sometimes, EdenFS goes bonkers and talks to 'hg debugedenimporthelper' a lot for seemingly no reason.
Make these situations easier to debug by counting how many requests EdenFS makes to 'hg debugedenimporthelper'. These counters let us answer questions such as the following:
* When performance sucks, is EdenFS is making a lot of requests or only a few requests? I.e. is Hg slow at responding or is EdenFS very demanding?
* Does a recent performance issue correlate with EdenFS communicating with 'hg debugedenimporthelper'?
* Which engineers are outliers having orders of magnitude more 'hg debugedenimporthelper' requests than p50 engineers?
We could get fancier with these counters and include the number of bytes received, the duration of the request, etc. For now, just having a request count is useful.
Reviewed By: simpkins
Differential Revision: D14677339
fbshipit-source-id: 7f8f394fb0096aef65d6a8a45d7da5936db539a0
Summary: Move Mononoke backing store initialization process to every request so we can adjust the configuration value without restarting the process.
Reviewed By: strager
Differential Revision: D14583357
fbshipit-source-id: 49ce2736229ce3062d34337757ebda6bb6eae16a
Summary:
We updating the encoding that mercurial uses for storing paths to UTF-8.
The current step is preventing files that have invalid names from being
committed. To that end we are updating the HgPrefetch test to submit
valid file names (valid UTF-8). This allows us to start using UTF-8
encoded string for the internal representation of file paths.
Doing encoding transformation between internal storage and system locale will
be implemented later. Normalization should also be handled at that point.
Reviewed By: simpkins
Differential Revision: D14812780
fbshipit-source-id: 1689910500391eed941df185ba92aa2d2f3f0960
Summary:
This message was added in D14337058. It is logged at the `INFO` level, which
is enabled by default, but doesn't seem to add much value to normal production
logs.
Reviewed By: chadaustin
Differential Revision: D14712654
fbshipit-source-id: 5a86d883ace30e22d299046e33a6cd6247432857
Summary:
While testing out the new getdeps code I found that none
of the include directories from the probed libraries were being used.
The new getdeps installs each dep into its own prefix, whereas the
existing getdeps script installed them into the installation
prefix for eden itself. That meant that they were being implicitly
found from a single include directory.
In addition to this, I encountered linker failures for the pretty
printers; the solution to those was to add appropriate deps for
the modules that depend upon the pretty printers.
Reviewed By: pkaush
Differential Revision: D14638758
fbshipit-source-id: a4c2b4c79603c268e1b1c707a05c3cb0e3f2757b
Summary:
This function depends on things that are not available
when EDEN_HAVE_HG_TREEMANIFEST is not enabled.
Ensure that the entire function is disabled if that is the case.
Reviewed By: chadaustin
Differential Revision: D14638759
fbshipit-source-id: 3fe83b7b42357c2b818469214b016ff591058e35
Summary: This diff enables thrift as a type of connection when talks to Mononoke API Server.
Reviewed By: strager, pkaush
Differential Revision: D14508956
fbshipit-source-id: 32e130c714be164f19452dec554976ea1cb594d3
Summary: Implement a backing store that uses thrift protocol to communicate with Mononoke API Server.
Reviewed By: strager
Differential Revision: D13966575
fbshipit-source-id: 66f66dda6b17aecd6c6b4475ab6b004c608f457f
Summary:
When opening the RocksDB, write one entry to each column family and then
explicitly flush that column family. This ensures that the column family
information has actually been flushed to an SST file. Without this some
column families may only have been written out to the write-ahead log files.
(Even calling `db->Flush()` does not appear to be sufficient; each column
family has to be explicitly flushed.)
The RocksDB' `RepairDB()` function (used by `ldb repair`) currently ends up
deleting column families that do not have any data defined in an SST file.
The repair tool ends up deleting column families that only have data in log
files.
The fact that we haven't been doing these explicit flushes previously probably
isn't too much of a concern in practice: once we write out enough data RocksDB
will automatically trigger a flush. This only matters in cases where we have
not yet written out enough data to trigger an automatic flush.
Note that with this change we re-write these `id` keys each time we open the
RocksDB store, even if they were already present.
Reviewed By: chadaustin, strager
Differential Revision: D14452214
fbshipit-source-id: 3f1b17e240cc89fe00e3d31105d16452795e754d
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:
1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.
Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.
Reviewed By: simpkins
Differential Revision: D14337058
fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
Summary: Since `CurlClient` is going to be used in multiple threads and each thread has its own instance of CurlClient, we are now able to assign one curl_easy handle for each of the client. So curl would be able to reuse the existing connections to Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14334486
fbshipit-source-id: aacd210773c44dea6f1d9dd7f9b09d765a9c7a0a
Summary: Put `CurlClient` into its own thread pool (like how `HgBackingStore` uses `HgImporter`)
Reviewed By: chadaustin
Differential Revision: D14334483
fbshipit-source-id: 3ce2a8245761c11bfa0f66691d10a66004b46fe6
Summary: This diff enables `MononokeCurlBackingStore` in HgBackingStore. It also adds a new config value `mononoke:connection-type` (values: `http`, `curl` and `thrift`).
Reviewed By: chadaustin
Differential Revision: D14135028
fbshipit-source-id: cc6fb95d94b3d98996d872d22ad1a95a97d562ab
Summary: This diff add a MononokeBackingStore implemented based on libcurl so we could add Mononoke support on macOS.
Reviewed By: chadaustin
Differential Revision: D14096313
fbshipit-source-id: 2bcb0c49002dcbb0a0190ba505a6e814e74db747
Summary:
This diff added a configurable value "mononoke:connector" to EdenConfig, so that we can specify a particular type of connection we need when using Mononoke as backing store.
This diff also moved the logic of initializing the existing `MononokeHttpBackingStore` to `initializeHttpMononokeBackingStore`.
Reviewed By: wez, strager
Differential Revision: D14141725
fbshipit-source-id: c7208295d7b3853740d7b0e5166f8854457fcf8e
Summary:
Throw a slightly nicer error message if the hg import helper exits before
returning a `CMD_STARTED` response or an error response.
When switching from the `hg_import_helper.py` script to the
`hg debugedenimporthelper` subcommand we are slightly more likely to encounter
errors during startup. For instance, if the repository path was not a valid
repository the `hg_import_helper.py` code would send an error message back
that the `HgImporter.cpp` code could parse. However, we unfortunately can't
catch errors loading the repository the same way when using an `hg`
subcommand. If it fails to load the repository it will simply print an error
message to stderr and then bail out before it invokes the
`debugedenimporthelper` code.
Reviewed By: chadaustin
Differential Revision: D14222266
fbshipit-source-id: cd60e5a61725d45a816b74f63b9969b29ade2160
Summary:
This reverts the change in D14188312 that moved `hg_import_helper.py` from
`bin/` to `libexec/eden/`, and instead updates edenfs to look for
`hg_import_helper.py` at `../../bin/hg_import_helper.py`
We cannot remove `hg_import_helper.py` from `/usr/local/bin` without breaking
existing running `edenfs` daemons as they will continue to look for
`hg_import_helper.py` in this location.
Rather than keeping a copy in both `bin/` and `libexec/eden` it seems better
to simply update edenfs so that it can correctly find `hg_import_helper.py`
even when edenfs itself has been installed in `libexec/eden/`
Reviewed By: strager
Differential Revision: D14197390
fbshipit-source-id: 2845bd79343bc29af9d3acbacecac4501cdb9032
Summary:
Now that `hg debugedenimporthelper` has been released for
a little while, we can remove the bundled implementation of it from
the eden release.
However, we cannot remove the script itself as there are users
with long running edenfs instances that pre-date the knowledge
of `hg debugedenimporthelper`. So, as a compatibility shim,
this diff redirects `hg_import_helper.py` and has it exec the
command in mercurial.
For extra fun, our own integration tests rely on being able
to import `hg_import_helper.py` and override portions of it,
so we cannot remove its implementation from the codebase just
yet either.
So this diff:
* Introduces `proxy_import_helper.py` which execs `hg debugedenimporthelper`
* Installs `proxy_import_helper.py` as `hg_import_helper.py` in the rpm
* Leave `hg_import_helper.py` as-is in the tree for now.
Reviewed By: simpkins
Differential Revision: D13970332
fbshipit-source-id: 717dc86a880fbbbe4a7e801a8b748abd053c7f7c
Summary:
in D13970332 we want to allow using `hg.real` to run the
import helper that is embedded into mercurial, but we don't know
the full path, so we need to resolve it from the environment.
`folly::Subprocess` has `usePath` option, but it is not compatible
with also overriding the environment for a child process.
The easiest short term thing to do is to resolve the path ourselves
using this simple function call.
I've also added the path to `hg.real` that is used on our mac systems
to the PATH we pass down to `edenfs` when starting eden.
Longer term we want to better encapsulate how we resolve and start
the import helper, but for now this unblocks starting eden on the
mac.
Reviewed By: strager
Differential Revision: D14078612
fbshipit-source-id: bc6787bbe99cd87224b783d2fb9d3255b825b976
Summary:
I don't have RocksDB building in mode/mac yet, and the inode code
only depends on the store interface, not all of the specific
implementations.
Reviewed By: simpkins, strager
Differential Revision: D14036390
fbshipit-source-id: d0b8d5c4d4f2f38dbc9bc2476c24ac99398d37b7
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13908257
fbshipit-source-id: 8b5315b019290f1c60087ca5716c31ebbf1f1be5
Summary: `hgext` was moved to `edenscm.hgext`.
Reviewed By: simpkins
Differential Revision: D13884629
fbshipit-source-id: 4bf69720cccbd43665cd350ac8a3ba4d46b10b93
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13784772
fbshipit-source-id: 1d3ede848b7d31c7a197a21b4ff2b31e840040a5
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
This requires our mercurial repo to be available during
the build; I symlink it in alongside `common` in the `oss` dir,
and point it up to `scm/hg`.
This has partial support for mononoke too, but will need to add
logic to getdeps to pull down the proxygen repo and build that.
Reviewed By: simpkins
Differential Revision: D13480146
fbshipit-source-id: 54874245015af83a259f56944d2e5f87615baee7
Summary:
In our linux deployments it was relatively straightforward
to import the mercurial runtime from a python process running the
system python executable. Our macOS deployments are a lot more
complex because they do not use the system python and do not install
the mercurial python packages in the python path of the target
python executable.
It is simpler to move the import helper functional into a mercurial
command that we can invoke instead of our own helper program.
This diff moves the script to be a debug command and adjusts its
argument parsing to match the mercurial dispatcher requirements.
There are some stylistic mismatches between this code and the
rest of mercurial; I'm suggesting that we ignore those as the
medium term solution is that this command is replaced by eden
directly consuming the rust config parsing code and by native
rust code to perform the data fetching that we need.
Reviewed By: pkaush
Differential Revision: D13522225
fbshipit-source-id: 28d751c5de4228491924df4df88ab382cfbf146a
Summary:
In the future, we will likely coalesce redundant fetches at the
BlobAccess/ObjectStore layer. To measure what we actually want,
populate a normal ObjectStore with a NullLocalStore and add counters
to FakeBackingStore.
Reviewed By: wez
Differential Revision: D13454331
fbshipit-source-id: 2fbf393d159f94e84c24ac53ccc207162fa754b7
Summary:
a couple of parts of the rust, datapack and mononoke code
didn't have appropriate definitions.
Allow `useDatapackGetBlob_` to be default initialized per its
definition in the header file rather than replicating the relatively
complex ifdef logic in the implementation.
Reviewed By: strager
Differential Revision: D13475710
fbshipit-source-id: d2955c9b22f1186f5897aa8bdbd9046b8f6b5f7a
Summary:
ScopedHTTPServer allows callers to specify a simple request handler function
instead of having to define their own handler factory. This updates the
MononokeBackingStore tests to change the `Handler` class in to a simpler
functor object, and delete the `HandlerFactory` class.
Reviewed By: wez
Differential Revision: D13476411
fbshipit-source-id: 0ede232ff9570c95e877b689272ea8eb26d97d83
Summary:
Change the code to use ScopedHTTPServer instead of its own custom logic that
starts a server and runs all of the test logic inside the `onSuccess()`
callback.
Doing all of the test logic inside `onSuccess()` is generally bad, as most of
the tests perform blocking `get()` calls on `folly::Future` objects. Blocking
work generally should not be done from inside EventBase threads.
When built with ASAN these tests occasionally crash with heap-use-after-free
errors during the HTTP server shutdown. I didn't track down the exact bad
behavior that was causing this, but attempting to stop the server from inside
the `onSuccess()` callback does seem a little fishy. Hopefully simply
switching to `ScopedHTTPServer` will fix this issue.
Reviewed By: wez
Differential Revision: D13476413
fbshipit-source-id: ab92cc16a5bf99a5e7b52529012a03786495c319
Summary:
Switch to ScopedEventBaseThread for the standalone EventBase thread create in
the unit tests. This way the test does not need its own custom logic for
starting and joining this thread. `ScopedEventBaseThread` also ensures that
the thread is actually running and ready to process events before its
constructor returns.
Reviewed By: wez
Differential Revision: D13476412
fbshipit-source-id: 260a40d93050e2e9b92ef9efd1549633679f36f7
Summary:
Remove some more of the double-checking code left over from debugging
T24406969. We fixed the root cause of that task and haven't seen any errors
reported by this code. The higher log levels used for reporting information
about empty files just causes extra noise in the logs.
Reviewed By: strager
Differential Revision: D13403531
fbshipit-source-id: bcb7b29a1d687ffcefd6ecf586130b96f87b4820
Summary:
There was a bug in BlobCache where, if you had an interest handle to a
blob, but that blob was evicted anyway and then something else caused
it to be reloaded, dropping your interest handle would cause the blob
to be incorrectly evicted since the reference counts were no longer
compatible. Add a version to cache items and only decrement the
reference count on an item if the interest handle and item agree.
Reviewed By: strager
Differential Revision: D13405144
fbshipit-source-id: aee052bf777e7225551c3ae2b8b69a99f4f77691
Summary:
Have `eden stats` print the size of the blob cache if the running
edenfs has information about it.
Reviewed By: strager
Differential Revision: D13349220
fbshipit-source-id: 9f59f4399f2d4283aa80bcb54ba73c51d555d502
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary: In support of making Eden's file access stateless, add a facade that connects the BlobCache and ObjectStore, allowing FileInode to fetch blobs, minimizing reloads and memory usage.
Reviewed By: strager
Differential Revision: D10850143
fbshipit-source-id: 4307f7c1143aecad1284ea3cadf3e4efca9a3925
Summary: Change the Windows pipe read and write functions to read/write in a loop. Plus changed the functions prototype to match the POSIX version.
Reviewed By: strager
Differential Revision: D13091785
fbshipit-source-id: 375b22bd9e62d371a78d410f42068945b966a743
Summary: Add a BlobCache with a maximum cache size and a minimum entry count and interest-based eviction.
Reviewed By: strager
Differential Revision: D12972062
fbshipit-source-id: 1958f7f500c051a5bc0b39b5b89a6f0fc1774b0f
Summary:
Update the HgUI object used by hg_import_helper to always return true from the
`plain()` function, regardless of whether HGPLAIN is set in the environment or
not.
This should help ensure that this script is never affected by user-defined
configuration settings. This should also help ensure that mercurial won't try
to print progress bars or other strange things.
Reviewed By: quark-zju
Differential Revision: D13008639
fbshipit-source-id: afe581958470c4c4b89825a259c460ece4e20fe7
Summary:
With the mononoke service being configured with proxygen
in front of the servers we need to ensure that we're correctly
setting up our http request if we want them not to fail with a
400 bad request error.
This diff allows setting mononoke.hostname to a DNS resolvable
name rather than an IP address and passing that hostname through
so that we can set it in the HTTP request.
Reviewed By: strager
Differential Revision: D12906498
fbshipit-source-id: b5aaabfd6f2f4c48d45128eaad8406e648477f75
Summary:
Update all of the C++ unit tests that create temporary files and directories
to use the new `facebook::eden::makeTempFile()` and
`facebook::eden::makeTempDir()` functions.
Note that this likely changes the behavior of some code paths in meaningful
ways: `/dev/shm` normally does not support `getxattr()`, and Eden's overlay
code attempts to store the SHA-1 for materialized files as using extended
attributes. This means that the tests will now typically hit the fallback
code path and will not store SHA-1 data in the overlay.
Reviewed By: chadaustin, strager
Differential Revision: D12971162
fbshipit-source-id: 6cc5eba2e04be7e9a13a30e90883feadfb78f9ce
Summary:
This function behaves similarly to the python `os.path.expanduser`
function, except that it is restricted to expanding only information for the
current user.
Use this new function to process the hgcache path returned via the
rust config parsing code.
In the longer term I want to centralize and add accessors for
the rust config and move this stuff into EdenConfig, but for
now this is the one place that does this and it is OK to
process it this way here.
Reviewed By: strager
Differential Revision: D12988902
fbshipit-source-id: 96b10640359c3b8c0adac1e14cd42dd592023c3d
Summary:
This makes it possible to change configuration options
for the LocalStore while the server is running.
As you'll see in the next diff, our current layering makes using
the config a bit more awkward, but at least this diff doesn't
look gross :-p
This diff doesn't introduce any new functionality or configuration.
Reviewed By: strager
Differential Revision: D12949577
fbshipit-source-id: cf897ba676b9359f92865170faa42ff17329b85f
Summary:
I've included a missing `.via` for the mononoke tree import path. It's hard to
measure the difference here, but it is more technically correct than letting
that second stage run in an event base thread. desired.
Reviewed By: chadaustin
Differential Revision: D12896023
fbshipit-source-id: 0af0f7f94c3274a3e4b570fed0af4d4d3c783bc3
Summary:
This diff implements getBlob on top of the mercurial rust
datapack code. It adds a C++ binding on top of the rust code to
make it easier to use and hooks it up in the hg backing store.
Need to figure this out for our opensource and windows builds:
* Need to teach them how to build and link the rust code
* need to add a windows version of the methods that accept paths;
this is just a matter of adding a WCHAR version of the functions.
Reviewed By: strager
Differential Revision: D10433450
fbshipit-source-id: 45ce34fb9c383ea6018a0ca858581e0fe11ef3b5
Summary:
Nothing populates these fields yet today, but will in a later diff.
If present in the json data for an entry, populate them in our TreeEntry.
When processing trees from mononoke, check for these fields being set
and emit them into the current writeBatch so that we set the BlobMetadata.
Reviewed By: chadaustin
Differential Revision: D12814793
fbshipit-source-id: cd19d3f553c22462adc58c024a90cfeb5b8da224
Summary:
I want to reuse this outside the LocalStore implementation
in a later diff. This has no functional changes.
Reviewed By: chadaustin
Differential Revision: D12814792
fbshipit-source-id: 2cc34b449a93afb2e71bd49ed6587009ebf3e419
Summary:
While this doesn't deliver any gains immediately, it is desirable
because the shortest path to getting trees with pre-populated size and
content sha1 information is via the mononoke api server fetch.
If we use the local trees we will populate the basic metadata slightly
faster but will then pay more to fetch and hash the data.
The calculus is a little more complex really, as having the contents of
a dir locally in a local datapack is probably faster, but we don't have
a good way to know if that is the case right now.
Reviewed By: strager
Differential Revision: D12813490
fbshipit-source-id: 545b90bf20940dab193bef8120712dd9227033e1
Summary:
This partially wires up handling mononoke related config
changes if they are made after starting the server. We don't currently
have logic to enable mononoke after starting the server, but can disable/enable
it if it was configured at process start.
Reviewed By: strager
Differential Revision: D12813346
fbshipit-source-id: ee29e91fe714a65a3b61c4e23929b67d6b7cb521
Summary:
To facilitate testing against a test instance rather than selecting
from a tier, add a config option to return the host override.
The semantics are that a hostname string will override the tier name setting.
An empty host name string is treated as missing and will not replace the
tier name.
Reviewed By: chadaustin
Differential Revision: D12813050
fbshipit-source-id: 67773d772d78d3f7882825ce54f4313c97ab0a33
Summary:
Remove the handful of (but growing) parameters that
are threaded down to the HgBackingStore in favor of the ReloadableConfig
object and use that to configure mononoke support.
This doesn't notice changes to the config on the fly today, but
does simplify the amount of stuff we need to pass around.
Reviewed By: chadaustin
Differential Revision: D12813053
fbshipit-source-id: b8309328fab4980889624257cfd387c8c1335001
Summary:
This allows detecting changes to the edenrc configuration
and adjusting how we read data accordingly.
To avoid a huge gnarly diff with a lot of dependencies, this diff
introduces a little interface class that defines the method to obtain
a possibly reloaded EdenConfig instance.
This diff shouldn't change any functionality.
Reviewed By: chadaustin
Differential Revision: D12813051
fbshipit-source-id: 35390dc28671ba46a03d098c4f280e2f567dbdc9
Summary:
There may be a metadata header in the contents and we currently
do not have any code to process this, so we want to disable this.
Add a constructor boolean to disable it and default to off while we
work up a proper solution to this.
Reviewed By: chadaustin
Differential Revision: D10523393
fbshipit-source-id: ec96b63b6897841ee5a6b848834b5a0ada145223
Summary:
Because Mercurial blob IDs change without the contents changing, and
because files get unloaded upon checkout, rebasing across a large
distance in history can result in status fetching a lot of
metadata. Keep a smallish LRU cache for SHA-1 and size by blob ID.
Reviewed By: strager
Differential Revision: D10419965
fbshipit-source-id: 81499573814775471913db05f924767c3bab300e
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary:
Noticed that the filenames in the logging in the error
case were sometimes garbled; capture a copy of the filename
rather than a reference to it.
Reviewed By: bolinfest
Differential Revision: D10473798
fbshipit-source-id: ee218a2ad9d6edfad6ba7246107c152e79a9efd8
Summary:
This uses the existing unionstore to attempt to load blobs directly from
the hgcache. This is a pretty simplistic implementation:
1. We unconditionally add the filepack path into the list of treepack paths
in the import helper startup. This is not 100% good, but should be good
enough in practice. This does mean that we'll only successfully find
a filepack if we have enabled treemanifests.
2. The load is performed in context of the calling thread, which means that
we may block. Not sure if that is critical at this point.
In the longer run, we want to migrate this to the newer rust code; this is
just early stage proof of concept.
Reviewed By: strager
Differential Revision: D10277985
fbshipit-source-id: 9b953bbf3983bdb777635c6ff0b78efc101a17cc
Summary:
Make sure all code that requires treemanifest is properly gated with
`EDEN_HAVE_HG_TREEMANIFEST` checks.
Reviewed By: chadaustin
Differential Revision: D10436528
fbshipit-source-id: b57005980242a3152f08b654306bff6e9c3ea4d3
Summary:
Eden performance can degrade when something (a status operation in
particular) queues a ton of work on the hg importer queue. In that
case, the 12 RocksDB threads are all blocked waiting to insert more
work into the importer queue. Thus, even RocksDB queries that would
hit cache cannot make progress until the importer queue clears up.
In the long term, we need a more comprehensive approach to managing
our concurrency and boundedness of scheduled work. In the short term,
though, just make the importer queue unbounded.
Reviewed By: bolinfest, strager
Differential Revision: D10368779
fbshipit-source-id: 9200002338bea091fb353b2a96ac543df98701c2
Summary:
This diff clarifies that the errors are a result of speaking with
mononoke in their exception reason, and includes the original error context
in the case of: eg: a connection failure.
Reviewed By: chadaustin
Differential Revision: D10298353
fbshipit-source-id: fbccc39706819270cf19ef6ca8397734440f525e
Summary:
This comment is instructional but is better situated in the
header file.
Reviewed By: singhsrb
Differential Revision: D10298365
fbshipit-source-id: 07ad9757a6be06a97f6aae81cff94088b776c11f
Summary: This was previously hard coded and making it configurable is operationally desirable.
Reviewed By: chadaustin
Differential Revision: D10298279
fbshipit-source-id: 124c3fa319c44aacb3d17ef6e380e9dec49db8be
Summary:
This little function helps reduce the amount of boilerplate
and casting in the main body of the code. It should not change
and functionality.
Reviewed By: chadaustin
Differential Revision: D10297205
fbshipit-source-id: ea74c294cf2ca258e3e9abb0e47209626d58f2a5
Summary:
We were catching exceptions that occurred during setup of
the mononoke futures rather than catching errors that occurred during
processing of the mononoke futures themselves.
This diff switches the try/catch blocks to `onError` handlers on
the future chains, as well as improves an ambiguous error message.
Reviewed By: chadaustin
Differential Revision: D10297428
fbshipit-source-id: c87e4a23a70e010646b07d9d8728851bdfdbcc2a
Summary:
Instead of calling getBlobMetadata in multiple places and only using
the .sha1 field, add a getSha1 function directly to ObjectStore. This
gives ObjectStore the latitude to fetch it and store it in different ways.
Reviewed By: wez
Differential Revision: D10227935
fbshipit-source-id: 180830534db3c42c07f04216599e496406af5ced
Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
future<T>.then(callable with operator()(folly::Try<T>)) to future<T>.thenTry(callable)
Reviewed By: Orvid
Differential Revision: D10227086
fbshipit-source-id: 1bb31c91cc65e28291e39302627f97801bfde15c
Summary: We've diverged in a few places from clang-format, so run it across the entirety of Eden.
Reviewed By: wez
Differential Revision: D10137785
fbshipit-source-id: 9603c2eeddc7472c33041ae60e3e280065095eb7
Summary:
Restructured the Windows code to align with the eden code layout. Plus changed the build location to eden/win/build directory, which is generated by the Windows build script.
eden
\_fs
\_ ...
\_ win
\_fs
\_service
\_utils
\_build (generated by the build script)
Reviewed By: strager
Differential Revision: D10081143
fbshipit-source-id: db9fb25f963d1a9cccb8a8f83646e7e45c87d409
Summary: Taking (#ifdef'ing) out some of the mononoke code from the Windows builds. Mononoke support will come once we compile the proxygen on Windows.
Reviewed By: strager
Differential Revision: D10079379
fbshipit-source-id: df431f9c57832d43811af41d4512674f1108cacf
Summary:
There's a lot of logic here that need not be offloaded to the
importer threads
Reviewed By: chadaustin
Differential Revision: D9539811
fbshipit-source-id: 1379eef390df6400291a61263d003b493a474b81
Summary: Override `txnutil.mayhavesharedpending` in the same way as `txnutil.mayhavepending`.
Reviewed By: strager
Differential Revision: D9834375
fbshipit-source-id: 7cd9d35121957343e8b15728485457072de047b5
Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
future<T>.then(callable with operator()(folly::Try<T>)) to future<T>.thenTry(callable)
Reviewed By: Orvid
Differential Revision: D9819578
fbshipit-source-id: f9e31f47354c041ecbf0a90953cbe50ebfda6adc
Summary: This hardcoded list is blocking using eden + mononoke with wwww
Reviewed By: chadaustin
Differential Revision: D9762557
fbshipit-source-id: 40991b26b702aa87a86a66934a069995d70f9f34
Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
Reviewed By: Orvid
Differential Revision: D9696716
fbshipit-source-id: d71433c75af8422b2f16733c0b18a417d5a4cf2e
Summary:
Modified the build scripts to use dependencies from D:\edenwin64. This is the location where we will mount the edenwin64.iso.
It also contains changes to compile it with the latest Eden Linux changes plus removed some POC stuff and dependencies from my laptop.
Reviewed By: strager
Differential Revision: D9545688
fbshipit-source-id: e92e34d0af07974845faf9f729e0861fde5af459
Summary:
This test seems to fail frequently while waiting on the tree data for the
"src" tree (at line 182). The code previously was using a 1s timeout for this
operation; this changes it to use a 10s timeout.
I'm not sure what would have changed that makes this take longer now. It's
possible something changed with the treemanifest code on the mercurial side.
It is somewhat surprising to me that this would regularly take more than 1s
though since this is only computing data locally about a pretty small test
repository.
Reviewed By: wez
Differential Revision: D9583645
fbshipit-source-id: 24695cc1b940540ec32461e2d80c30a3fd87a3e3
Summary:
Add a new unit test that exercises HgImporter and HgBackingStore using an
alternate hg_import_helper.py script that can inject errors into the response
data stream. This tests the behavior of HgImporter when there are
serialization errors with the data, as opposed to legitimately-serialized
error responses containing a python exception message.
Reviewed By: wez
Differential Revision: D9510568
fbshipit-source-id: ae86f69d5559701e9e8a9a18569986d64488eaf4
Summary:
Update hg_import_helper.py to throw a new ResetRepoError type when it decides
that its local mercurial state is invalid and that it needs to reopen the
repository. The C++ code will catch this exception, restart the
hg_import_helper.py and retry the request. (It will only retry once for each
request.)
The previous behavior of simply closing and re-opening the repository object
in Python has been problematic, as this has resulted in resource leaks. The
hg_import_helper.py process itself ends up leaking memory when re-opening the
repository. This also appears to result in many scmmemcache helper processes
being created and not cleaned up. Presumably these are associated with the
old repository state and not cleaned up properly when we reopen the
repository.
Reviewed By: quark-zju
Differential Revision: D9510664
fbshipit-source-id: 449dfa9e2e21aabf8b3ce640749d32aa8f8e4052
Summary:
Update the HgImporter code to actually check the transaction ID field in the
response headers from the hg_import_helper script.
Reviewed By: wez
Differential Revision: D9507288
fbshipit-source-id: 0a170bd6b39426176c938a135508fa4bb928053a
Summary:
Update the HgImporter code to check the return value from
`folly::writevFull()`
Reviewed By: wez
Differential Revision: D9507289
fbshipit-source-id: 00985f50ebb4717f7b97a199bf9efba6b4f5f628
Summary:
Previously the HgImporter code was not checking the return value of
`folly::readFull()`. This function does not throw exceptions: it may return
-1 on error or a partial number of bytes read. Previously the HgImporter code
did not perform any checking of the result from this function.
Reviewed By: wez
Differential Revision: D9507287
fbshipit-source-id: 80d8bdb54375eb287d1473675fffb48c4cd34c56
Summary:
These methods were only used in the test utility in one location. I plan to
clean up how the HgImporter code reads from the underlying python process, and
it will be easier if these are non-static methods.
This removes the ability from the test utility to read from a pre-computed
manifest data file. I don't think we care too much about this functionality
any more--most of our repositories are exclusively using treemanifest mode now
anyway.
Reviewed By: wez
Differential Revision: D9507290
fbshipit-source-id: 97fe3fc9d8574c97a41328ed42e7afdacb3734d1
Summary:
Add a wrapper class around HgImporter that can catch errors communicating with
the underlying `hg_import_helper.py` script. If something goes wrong
communicating with the python code we restart the import helper and retry
the operation once.
This code only retries on HgImporterError exceptions, and nothing throws these
at the moment. Therefore for now this should not result in any behavior
changes. In the future I will update our error handling to throw
HgImporterError when we fail to communicate with the underlying
hg_import_helper.py script.
Reviewed By: wez
Differential Revision: D9507291
fbshipit-source-id: 0167959df6292a577942e049576669fb6711ce76
Summary:
D9494651 and D9415207 logically conflicted with each other, but did not have
conflicts at the source code level. They both landed independently which
resulted in a broken build even though CI had passed on their individual
diffs. This updates them to be compatible.
Reviewed By: wez, strager
Differential Revision: D9505360
fbshipit-source-id: 72743167d934d1d57044cda8d5a7af03346ab142
Summary:
Change HgImporter::importFileContents() to return a Blob object rather than
just the IOBuf.
This shouldn't have any behavior changes. However, it might help track down
some of the cases where we have seen files get incorrectly imported with empty
contents in some rare cases. We're not sure what is causing that, but
incorrectly using a moved-from IOBuf object would be one thing that might
produce this behavior. By changing this code to a `unique_ptr<Blob>` will
will eliminate the possibility of seeing empty buffer contents if some code
path is somehow ever able to use one of these objects after they have been
moved away from. (The code would end up with a null pointer in this case
rather than an empty buffer. If this ever does occur in practice a null
pointer would make the problem more obvious than an empty buffer.)
Reviewed By: chadaustin
Differential Revision: D9494651
fbshipit-source-id: 799cc1b1d0e2ac32ae2013dc80c84ec17cc1c491
Summary:
This updates the `hg_import_helper.py` code so that when importing files it
also includes the file length in the response, in addition to the response
contents itself. This information is redundant and exists purely to help
the receiving C++ code verify that the response is in fact completely valid.
This is implemented with a new command number, and `hg_import_helper.py` still
supports the old command number with the old behavior so that it will still
work if old edenfs daemon instances start new instances of
`hg_import_helper.py`.
Reviewed By: chadaustin
Differential Revision: D9494653
fbshipit-source-id: 34d1221b6048d6016b30fd7f606ca0b140552d80
Summary: This commit makes eden ask Mononoke API Server first when it is fetching blobs.
Reviewed By: chadaustin
Differential Revision: D9415207
fbshipit-source-id: 1146a9b6295e8ddb15e4bcb098379ea75886baa5
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.
We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import. We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.
Reviewed By: chadaustin
Differential Revision: D9476522
fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
Summary:
`F14NodeMap` has proven to be a safe and more performant drop-in replacement
for `std::unordered_map`. With the ability to do heterogeneous lookup and
mutation, we no longer need `StringKeyedUnorderedMap`, whose main purpose was
to avoid unnecessary string copies during lookups and inserts.
Reviewed By: nbronson
Differential Revision: D9124737
fbshipit-source-id: 81d5be1df9096ac258a3dc1523a6a0f5d6b9e862
Summary: If the OS is Windows it will expect the in-fd and out-fd as Windows handles and convert them to CRT fds. This will not change the behavior if it's not Windows.
Reviewed By: wez
Differential Revision: D9344414
fbshipit-source-id: 8b30b57d979dc45af06676979e75e00b02d2b89b
Summary:
This diff is first in the series to make Eden work on Windows. It includes:
1. HG backing store and Object store, which provides the capability to talk to mercurial and fetch the file and folder contents on Windows.
2. Subprocess and Pipe definition for Windows.
3. The Visual studio solution and projects files to compile Eden and scm datapack.
Few Important points:
1. Most of the changes to existing code is done under a macro EDEN_WIN so that it doesn't impact on other platform.
2. Sqlite is used for caching the fetched contents. We are not using Rocksdb on Windows.
3. The main function only calls some test code and exit after printing the output.
4. The initializeMononoke code is disabled for Windows because it needs Proxygen to talk HTTP. Will enable this once I get Proxygen and other dependencies working.
5. HgImporter pass Windows handles to hg_import_helper as command line args. The code to convert these handles into fds is in a separate diff.
Reviewed By: wez
Differential Revision: D8653992
fbshipit-source-id: 52a3c3750425fb92c2a7158c2c214a9372661e13
Summary:
Update the hg_import_helper.py code to catch exceptions from `repo.close()`
when we are re-opening the repository after an error. Also set `self.repo` to
`None` before we start, so that it will be left as None if anything goes wrong
either closing or re-opening the repository.
I ran into a situation where `repo.close()` threw an error, and previously
this would leave `hg_import_helper.py` stuck in a bad state since it would
still be pointing to the old repo object. The next time it received a command
it would fail and try to call `self.repo.close()` again, which would still
fail since the repository was already in a halfway closed state.
Now the code will always forget about the old repository object and create a
new object.
Reviewed By: wez
Differential Revision: D9419109
fbshipit-source-id: 15bb296ba19d9d3d2a2b90169bf25b0e8e197c1f
Summary: This commit let `HgImporter` pick up configuration values from `EdenConfig`.
Reviewed By: chadaustin
Differential Revision: D9346293
fbshipit-source-id: cb63f7d13a86058e9bf076eddb52212560a64cb1
Summary:
We have seen issues with Eden occasionally getting empty file contents back
from Mercurial. We have not been able to reproduce this issue directly
yet--the overall incidence rate relative to the number of users is fairly
small, but the problem is pretty problematic when it does occur.
This updates the hg_import_helper.py script to perform additional checking
when it finds a file that is empty.
Empty files that have never been modified or renamed all have the same file
revlog hash. If the rev hash in question is this known hash, we know that the
file is in fact empty. Otherwise, if remotefilelog is in use we check the
remotefilelog metadata to confirm the size, and we log an error if it is
non-zero. We then try re-opening the repository and re-importing the file.
These additional checks and log messages should hopefully help narrow down the
problem the next time we see a report of the issue. This should help us
identify if the remotefilelog metadata has the correct file size or not.
Reviewed By: wez
Differential Revision: D9260788
fbshipit-source-id: 29615b32632946cd319aa837bec3c68b757d3ee0
Summary:
Add the beginnings of an eden gc command. Today it's equivalent to
`eden debug clear_local_caches` followed by `eden
debug_compact_local_storage`, except that it compacts each column as
they're cleared to minimize peak disk consumption.
Eventually, it will also unload in-memory inodes, flush data from the
overlay, and clear the kernel's VFS cache too.
Reviewed By: wez
Differential Revision: D9138305
fbshipit-source-id: b303a63f601014cf38ca94c9e6f7c04394159ea8
Summary:
This commit integrates Mononoke API Server with eden:
* Added two new command line options: `--client_certificate` and `--use_mononoke`.
* Added two new config values: `ssl.client-certificate` and `mononoke.use-mononoke`.
* Made `HgImporter` return the repo name along with treemanifest options.
* Make `HgImporter` ask Mononoke API Server for tree data when the desired tree is not present in local Mercurial store.
Reviewed By: chadaustin
Differential Revision: D9183035
fbshipit-source-id: e328fb3237d10c545c8af71f856007ad6c487061
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
6/n: Codemod rvalue-future<T>.then(...) to rvalue-future<T>.then(...).
Reviewed By: yfeldblum
Differential Revision: D9152002
fbshipit-source-id: 166475c1dcafb29a11154cbfbdf7e2e1feaf745b
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary: This commit changes MononokeBackingStore to use new APIs provided by Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D8882789
fbshipit-source-id: 0f06ca5f850af9fb52f1d593b9abd715a541488a
