Summary:
This is the very first version of add_sync_target method. It's very naive - it
just creates a single merge commit, and it doesn't handle all the necessary
functionality (e.g. linkfiles) and it doesn't do all the necessary checks yet.
TODOs show the missing functionality.
However this implementation it should be good enough as the first version, and
we can continue to extend it.
Reviewed By: mitrandir77
Differential Revision: D28677365
fbshipit-source-id: a8af6b92ced285ce4d012843f591d5c29584669d
Summary: It will be used in the next diffs, let's move it to a common file
Reviewed By: mitrandir77
Differential Revision: D28675768
fbshipit-source-id: 78c21e378053a157d09ee9d032576a434a52c912
Summary: They will be used in the next diffs
Reviewed By: mitrandir77
Differential Revision: D28675756
fbshipit-source-id: 2f41191cb83921fb3ff3374c0ccb7a884b7b3a25
Summary: Sometimes things take longer, make sure we are able to distinguish whether that's due to networking, tls handshake, http parsing, or Mononoke wireproto handling.
Reviewed By: markbt
Differential Revision: D28705508
fbshipit-source-id: 1bafda7fc447f2e429690f47fe7ab81cec511494
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: fanzeyi
Differential Revision: D28663227
fbshipit-source-id: cee18ec28283c0aef18151465a3c13e9be604d7d
Summary:
Extends the `FileScmStoreBuilder` to construct two new indexedlog stores for caching aux data. The stores will be created in a directory adjacent to the normal non-LFS indexedlog stores.
Currently aux data stores will not be constructed for production users, a configuration option will be introduced to gate this before `.store_aux_data()` is called in the `filescmstore` constructor bindings.
Reviewed By: DurhamG
Differential Revision: D28689693
fbshipit-source-id: e3ad1594e5beee00b1a8b9fe489e3b6af3a2e93e
Summary:
Modify `FileStore` to introduce basic aux data fetching. Aux data is currently read from a separate IndexedLog store, serialized with `serde_json` (chosen for expediency / ease of debugging, I intend to optimize the storage format before releasing this, at the very least to avoid unnecessarily serializing the key path).
Currently aux data fetching will never succeed, as aux data fetching is not supported in the EdenApi "files" API and nothing else exists to populate the local aux data stores. Later in this stack, computing aux data (currently only content sha256) to populate the aux data storage is implemented.
Reviewed By: DurhamG
Differential Revision: D28526788
fbshipit-source-id: c8e21a1377689d7913a68426a3a480d53148da66
Summary:
Simplify tracking of incomplete fetches in preparation for attributes support in the next change.
Now, all keys which have not been completely and successfully fetched are recorded in `pending`, and are removed only when the complete fetch is recorded in `found`. Keys are now removed from `lfs_pointers` and `pointer_origin` as they are completed, as they aren't needed for anything other than fetching from local LFS and remote LFS respectively.
Reviewed By: DurhamG
Differential Revision: D28546515
fbshipit-source-id: c657e5c6350cadc8da970f57bb7694ed71022efb
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28610639
fbshipit-source-id: c85535669f84695217125af5de7cc4848fba6b2d
Summary: FileDescriptor.cp throws some warning on comparison of signed long and unsigned long. So fix these warning and follow how ```folly/portability/SysUio.cpp:doVecOperation``` did
Reviewed By: fanzeyi
Differential Revision: D28608581
fbshipit-source-id: 3b6e35e9764548ce470634f16a1f6eec5118d7ed
Summary:
Now metalog can no longer be `None`. Let's just remove logic handling its
`None` case.
This changes the commitcloud-sync-race test because the metalog itself has
internal locking and changes are atomic.
Reviewed By: DurhamG
Differential Revision: D28595292
fbshipit-source-id: bd9851f5f3bb25f28f15d673f608af2863953c46
Summary:
fncache and store have been default on for years. Enable them unconditionally.
This also makes sure that metalog is always available.
Practically, the only place that does not use fncache is hgsql server repos and
they are irrelevant now.
Reviewed By: DurhamG
Differential Revision: D28595289
fbshipit-source-id: 32b9906c179518acdb17a206b54f98a3dc994921
Summary:
Now that makeImmediateFutureWith exists, we can simply use it instead of
constructing a ImmediateFuture<folly::Unit> and calling thenValue on it.
Reviewed By: chadaustin
Differential Revision: D28518059
fbshipit-source-id: 0041cf863fb32efab274f11c77c76109ca9b454f
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28665465
fbshipit-source-id: 9281de7fd62f38e09d91435bb53c819bb98fb4ec
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28595085
fbshipit-source-id: abc03d210b2e9c5aa19a8925be6d4c426311e826
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28594404
fbshipit-source-id: f3dec92403739d67df3ecd091f2d8283a11ea0db
Summary: This will be used to replace calls to folly::makeFutureWith.
Reviewed By: chadaustin
Differential Revision: D28515786
fbshipit-source-id: 2c2c542392e8e57b8f865173d6878cb9d00ba376
Summary:
For debugging and better error messages it would be nice to know who worked on
a given request.
I'll do AOSC schema change once accepted.
Reviewed By: krallin
Differential Revision: D28441673
fbshipit-source-id: ba146d7f43dde26d9433f76af7fe982da14b5b82
Summary:
Those will be used for two purposes:
* to limit a scope of given tailer to just given repo (this way we could use
different tailer binaries for different repos or disable processing for a
single repo etc...)
* to enforce a single in-flight request per repo (to prevent client from
accidentally scheduling duplicate requests etc...)
I'll do AOSC schema change once accepted.
Reviewed By: krallin
Differential Revision: D28441670
fbshipit-source-id: 7b35a1c7034707d7cf54220c559edd6e03f430c3
Summary: I want to put more things in the async_requests crate.
Reviewed By: krallin
Differential Revision: D28441671
fbshipit-source-id: 19233c2c5b697cc1e27107cd9904666baf8f10b7
Summary: I have modified the places where most of the errors were raised that users reported and were resolved by renewal of certificates.
Reviewed By: krallin
Differential Revision: D28568561
fbshipit-source-id: 44fb127a49bde83efee1c934e0435b31f8602a8d
Summary: This diff removes the gflag based batch size option and promote it into a EdenFS Configuration so we can experiment with different batch size easily.
Reviewed By: chadaustin
Differential Revision: D28555280
fbshipit-source-id: 6d3a7be3cd880f0aaa3f427c0328222efa2d37ea
Summary:
The telemetry wrapper didn't validate the SNAPSHOT header, which makes
migrating to a new format harder. Fortunately, it doesn't even need to
read the SNAPSHOT file. The dirstate file is maintained even in EdenFS
checkouts.
Reviewed By: quark-zju
Differential Revision: D28650333
fbshipit-source-id: 174cf7039adcbb28224ec528c2462e0a9232b6cd
Summary: Upcoming changes will force enable metalog so there will be no way to migrate down.
Reviewed By: DurhamG
Differential Revision: D28595290
fbshipit-source-id: a130b3c60c5b553d024868f28a28e48c50d44783
Summary:
It was added by D8527475 (72c3d8afc1) to workaround hgsql with no-fncache and long file
names synced from svn. Upcoming changes will force fncache to simplify
configuration and the hgsql server code was forked. So let's just delete
the test.
Reviewed By: DurhamG
Differential Revision: D28595291
fbshipit-source-id: 60d2449cca7af46b8b5b3c3b557a36507ff1576e
Summary: This will be used by fbclone to ship lazy commit hash backend.
Reviewed By: DurhamG
Differential Revision: D28554445
fbshipit-source-id: a263ae7683124b3b86f4025b02c7de20dcb9813e
Summary:
Add Dockerfile build for openr.thrift python module.
The python module is built by:
1. Building and installing Facebook libraries with fbcode_builder
2. Building Open/R
3. Generating Cython files from thrift files with the FB thrift compiler
4. Generating C++ files from the Cython modules with the Cython compiler
5. Compiling the C++ modules into shared objects
Future work for building and distributing Breeze:
- Fix the hacks in build_breeze.sh, see comments therein
- Use a staged Dockerfile build for the Open/R and Breeze build
- Install openr.thrift. The openr.thrift shared objects are build and
stored in the Docker image generated by Dockerfile, but are unused.
- Install all the openr python submodules in a single openr site-package
- Add cross-compilation to the openr.thrift build. This is needed for
Terragraph
- Upload the openr python package to PyPi
Reviewed By: saifhhasan
Differential Revision: D28614443
fbshipit-source-id: 38b7e7c5594fd4bb5a338f19c69e5fc3b3b95863
Summary: This makes it possible to use non-debugshell to compact the metalog.
Reviewed By: DurhamG
Differential Revision: D28550902
fbshipit-source-id: 789830ba35243d248397e6a52ee343584c1e01a9
Summary:
The "compact" API rebuilds the metalog by removing older history. It could be
useful to reduce the size overhead of the metalog.
This is also useful if we're doing other "rebuild" work, such as rebuilding the
changelog.
Reviewed By: DurhamG
Differential Revision: D28550903
fbshipit-source-id: 56f875bd955247181236a976dcce6163d126a4b6
Summary: I'm going to reuse this for AOSP import logic speedups, and I do not want my low QPS limit overridden by a higher QPS limit set for backfilling. Push the rate limiter out
Reviewed By: StanislavGlebik
Differential Revision: D28638180
fbshipit-source-id: ef3a783d4b1993614a146f534337f719958a1f36
Summary: We don't need to load the chunk data or update the chunk generation if the blobstore key is already present in IfAbsent mode.
Reviewed By: farnz
Differential Revision: D28640820
fbshipit-source-id: 3eab255ebfc896d4950935e3d7350b19f9a280b9
Summary:
The zipimport logic requires the pyc mtime to match its source. However, the
Windows system time zone can invalidate it and cause slow startups.
Workaround it by making the zipimport mtime function return a fallback value so
the mtime check is then bypassed.
# zipimport.py, _unmarshal_code
source_mtime, source_size = \
_get_mtime_and_size_of_source(self, fullpath)
if source_mtime: # if source_mtime is false, then the check is bypassed.
# We don't use _bootstrap_external._validate_timestamp_pyc
# to allow for a more lenient timestamp check.
if (not _eq_mtime(_unpack_uint32(data[8:12]), source_mtime) or
_unpack_uint32(data[12:16]) != source_size):
_bootstrap._verbose_message(
f'bytecode is stale for {fullname!r}')
return None
Change my Windows time zone from GMT-7 to GMT-4. Set PYTHONVERBOSE and
PYTHONDEBUG to 1. Ran `hg init -h` and check its stderr. It prints:
# bytecode is stale for 'edenscm.traceimport'
and alike before this change, and no longer after replacing the `__init__.py`
in the zip with the new version.
Reviewed By: DurhamG
Differential Revision: D28622287
fbshipit-source-id: bb3e8e378ea168e4f83f4b6aa9713103b2c90ef8
