Summary:
It now works with python 3. I dropped some test output. In one case
it was confirming that various path encoding work, but we only support utf8 now.
I also dropped some bit that enabled debug output, since the output order was
not the same between python 2 and 3 and it seemed non-trivial to fix.
Reviewed By: xavierd
Differential Revision: D22101132
fbshipit-source-id: 682e7c61f1dd18cce34bdc014c85d560f883bd18
Summary:
We support unicode file paths, and in python 3 those get passed to
python libraries as unicode strings. The tests set LANG=C which mean the python
library tries to convert the path to ascii, but fails for any non-ascii
characters. Let's switch to LANG="en_US.UTF-8" to match our production
behavior and make tests about unicode paths work.
Reviewed By: xavierd
Differential Revision: D22098359
fbshipit-source-id: c3057edc66e6e32f7b8b49374e622d02bd05711f
Summary:
This module was using binascii.hex directly instead of using Mercurial hex
function.
Reviewed By: xavierd
Differential Revision: D22096490
fbshipit-source-id: 32c912c16415ffd94371ad945c4df2f5a82b92e5
Summary: Replace the term 'blacklisted' with 'redacted'.
Reviewed By: quark-zju
Differential Revision: D22142842
fbshipit-source-id: 461faa09a5d3958a4575c48f03e2286eec917681
Summary: Add a crate-level doc comment clarifying the purpose of this crate. In particular, the motivation behind moving some of EdenAPI's types into this crate was to make it easier to share types between the server and client. As such, types that are not shared (which may pull in more complex dependencies and therefore cause build issues for either the server or client) should not be put in this crate.
Reviewed By: quark-zju
Differential Revision: D22142180
fbshipit-source-id: 7258ef33b73a87acf72d4f6bcbe8b27cbc361735
Summary: For python3 compatibility, explicitly open subprocesses in text mode (universal_newlines=True instead of text=True for backwards compatibility), and do not explicitly .decode("utf-8"), as all strings are already utf-8.
Reviewed By: xavierd
Differential Revision: D22123138
fbshipit-source-id: d91cee7909f4e87b4a04b85fa2a2929aec052da3
Summary:
Context: while syncing ovrsource commits into fbsource as a preparation for
megarepo we noticed that there are some commits that are particularly large
(e.g. up to 4Gb). Syncing them causes hgsql to become unavailable for 10-20
mins until all hg servers catch up - during that time hg log -r tip --forcesync
would just throw mysql unavailable errors.
As a mitigation for this we tried to make hgsql read from replicas instead of
master and it worked! (see D22094976 for more details). However the approach we
chose was quite hacky.
This diff tries to make it less hacky. It add --syncfromreplica option which
will be used by remote_backup.sh script on hg servers. if we don't have a lot
of read traffic on hg servers (and it should be the case for repos that are
hosted on Mononoke) then hg servers should try to read mostly from replica.
One important note here - the repository's state *might* go back in time with --syncfromreplica. I.e. consider the case where we move a bookmark without adding a new commit. Now, let's say local repository got the latest state from mysql leader. Then when we try to sync from replica if this replica is behind the local state of repository then we'd fetch the bookmarks from replica even though they are older.
With our current setup where we have a lot of different repos on different hg servers in completely different state this problem should be acceptable.
Reviewed By: ikostia
Differential Revision: D22118491
fbshipit-source-id: 4bfa869b697326441de6c004cd548df483049b9d
Summary: Moved GitimportPreferences and GitimportTarget objects from gitimport to import_tools. Reasoning: will need to move gitimport function which needs these objects. Since it's a rust_binary, we can't add dependency on it in import_tools. Therefore, I moved the objects to import_tools.
Reviewed By: StanislavGlebik
Differential Revision: D22135765
fbshipit-source-id: f31f96e689dccaac171a9d0573019b1fff757885
Summary: A bunch of our dependencies weren't really used, and this fact has recently became a source of hard failures. This diff is an attempt to fix it.
Reviewed By: StanislavGlebik
Differential Revision: D22136288
fbshipit-source-id: 4ae265a93e155312ae086647a27ecf1cbbd57e9c
Summary:
On Windows (haven't verified on other platforms), ThreadLocalPtr don't appear
to be releasing resources when a thread die. This means that when the importer
thread dies, the actual importer (hg.real) would still run and use resources,
with no way of talking to it.
To fix this, let's manually reset it when the main thread function returns,
this forces the importer to be destroyed and therefore the various handles to
hg.real to be released, effectively terminating it.
I'm not sure if this is the proper fix, but delving into folly feels a bit
daunting. Keeping a TODO for later to go back to it and fix it properly in
folly.
Reviewed By: chadaustin
Differential Revision: D22012540
fbshipit-source-id: 99f994bb5128b38ccf8474031763b8a21055759a
Summary:
The write method returns the number of bytes written, and python3 print these
if not captured.
Reviewed By: quark-zju
Differential Revision: D22096665
fbshipit-source-id: 2b1abd994df4c705515a6dc3d01c720f056f0411
Summary:
The ui.ferr expects bytes data, and we were passing a str, causing the progress
bar to crash.
Reviewed By: quark-zju
Differential Revision: D22094687
fbshipit-source-id: fddbfdb4314661554ae90dc0dd8bb16ce080cbfd
Summary:
The expected message are printed via repo.ui.debug, which may not be the same
device as print uses, and thus the output may be interleaved in a different
way. By using the same way of printing, this issue is avoided.
Reviewed By: ikostia
Differential Revision: D22097487
fbshipit-source-id: 6e49f44987f5660a00e72c082f0a5a483dded9ae
Summary:
Python 3.8 gives a warning when asking for line buffering on a binary file:
```
line buffering (buffering=1) isn't supported in binary mode,
the default buffer size will be used
```
Reviewed By: xavierd
Differential Revision: D22120968
fbshipit-source-id: 9b1f98f5bfa8ac9ed4f0b4e62389051f665f5d4f
Summary: Add a command to print out the current status of visibility tracking.
Reviewed By: quark-zju
Differential Revision: D22089413
fbshipit-source-id: 729e9f36fbc13ba90518dcd428a44c02ce358167
Summary: Commit cloud sync works best when visibleheads are enabled. Default to requiring it.
Reviewed By: quark-zju
Differential Revision: D22089414
fbshipit-source-id: b786103d5c5f65a7908eb97a973af9eca5990b22
Summary:
Somehow reading the file with `open("r")` will try to decode it as ascii, but
that file may contain utf-8 characters, so open it as bytes, and then decode
it.
Reviewed By: DurhamG
Differential Revision: D22105188
fbshipit-source-id: 0cfbd8c5417b637dd76391f86226e24ee663baf7
Summary:
This diff makes `lookup` to use `RequestData` as `ObjectFetchContext` in `getattr` calls.
This will make sure we correctly record backing store fetches in `eden top`
Reviewed By: chadaustin
Differential Revision: D21780969
fbshipit-source-id: 468e2fadcebf4a00477bc5de434e6c658b99d1ce
Summary: This diff makes `InodeBase::stat()` to be able to accept `ObjectFetchContext` as a parameter.
Reviewed By: chadaustin
Differential Revision: D21780883
fbshipit-source-id: 9b1db2e2268cb98663bcf902ea61897da593ea05
Summary: This diff starts to use `RequestData` as `ObjectFetchContext` in our `read()` methods. This ensures EdenFS could track backing store fetches happened in FUSE requests.
Reviewed By: chadaustin
Differential Revision: D21792503
fbshipit-source-id: 9509a1bc8f28100a0dfe196e312c4785c7842345
Summary:
Previously we check if a request is a fuse request when we fetch anything from backing store, so we can collect number of fetches happened for each process in eden top.
This is creating a dependency from store to fuse, which is a little awkward. Instead, we could make `RequestData` a `ObjectFetchContext` and record the fetches when that happens.
Similarly in the future we should also have something equivalent in our Thrift layer.
Reviewed By: kmancini
Differential Revision: D21775919
fbshipit-source-id: 95056830ddbe7c999051c43e0d8eca9a67350904
Summary:
All of these are instances of `CommitSyncer`, named `commit_sync_config`. It
makes a reader think they may be instances of `CommitSyncConfig`.
Reviewed By: farnz
Differential Revision: D22113917
fbshipit-source-id: bb682b24d7cadf87ca7721f341dbc8231636c99a
Summary: Megarepo is simplified if we can avoid copying hooks everywhere - run megarepo hooks as well as small repo hooks during pushredirection.
Reviewed By: StanislavGlebik
Differential Revision: D20652331
fbshipit-source-id: f42216797b9061db10b50c1440253de1f56d6b85
Summary:
Tooling can't handle named_deps yet, but it can warn about them
P133451794
Reviewed By: StanislavGlebik
Differential Revision: D22083499
fbshipit-source-id: 46de533c19b13b2469e912165c1577ddb63d15cd
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary:
Two changes here:
1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches.
2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file.
Reviewed By: farnz
Differential Revision: D22023144
fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
Summary:
The goal of the stack is to support hot reloading of `CommitSyncConfig`s everywhere: in `push_redirector`, `backsyncer`, `x-repo sync job` and so forth.
This diff in particular is a refactoring of how we instantiate the `PushRedirector` struct for the `unbundle` flow. Previously the struct would be instantiated when `RepoHandler` struct was built and would later be reused by `RepoClient`. Now we want to instantiate `PushRedirector` before we start processing the `unbundle` request, so that we can request the newest `CommitSyncConfig`. Note that this diff does not introduce the hot reload itself, it just lays the groundwork: instantiation of `PushRedirector` at request start.
To achieve this goal, `RepoClient` now contains a somewhat modified `PushRedirectorArgs` struct, whose goal is to own the unchanging stuff, needed to create a full `PushRedirector`.
Here are a few explicit non-goals for this hot reloading:
- the overall decision whether the repo is part of any `CommitSyncConfig` or not is still made at `RepoHandler` creation time. What this means is that if `CommitSyncConfig` is changed to have another small repo and Mononoke servers happens to know about that repo, it would not automatically pick up the fact that the repo should be a part of `CommitSyncConfig`
- same for removal (stopping push redirector is already possible via a different hot-reloaded config)
- changing anything about a large/small relationship is likely to be very complicated under the best circumstances of everything being down, let alone during a hot reload. This means that target repo cannot be changed via this mechanizm.
Essentially, the goal is only to be able to do a live change of how paths change between repos.
Reviewed By: StanislavGlebik
Differential Revision: D21904799
fbshipit-source-id: e40e6a9c39f4f03a436bd974f3cba26c690c5f27
Summary: Added basic main and target files for the repo import tool
Reviewed By: StanislavGlebik
Differential Revision: D22067642
fbshipit-source-id: 5ed97e3f2446c86314918f57103c2ecb911e30b0
Summary:
Add optional compress on put controlled by a command line option.
Other than costing some CPU time, this may be a good option when populating repos from existing uncompressed stores to new stores.
Reviewed By: farnz
Differential Revision: D22037756
fbshipit-source-id: e75190ddf9cfd4ed3ea9a18a0ec6d9342a90707b
Summary: Add zstd decoding support to packblob so that if store contains individually zstd compressed blobs we can load them on get()
Reviewed By: farnz
Differential Revision: D22037755
fbshipit-source-id: 41d85be6fcccf14fb198f6ea33a7ca26c4527a46
Summary:
Add a regex for repo prefix detection, and use it in prefixblob for removal of repo prefix from blob-embedded keys.
This is important to keep blobs copyable between repos, and to allow dedupe between same blob in two repos.
I looked at alternate approaches of passing down the prefix from PrefixBlob::new(), but that was fragile and didn't cover the use cases from things like blobstore_healer where no prefixblob is configured at all but packblob will be in the blobstore stack. Using a pattern is the only real option in such "all repo" cases.
The aim of binding the pattern and its tests closely to the prefix generation is to make it hard for someone to get them out of sync, and provide a clear local test failure if they do.
Reviewed By: farnz
Differential Revision: D22034638
fbshipit-source-id: 95a1c2e1ef81432cba606c22afa16b026f59fd5f
Summary:
Add pack and unpack logic to packblob.
Loading a packed form is possible from a regular get(), as the store may contain packed data.
Storing a packed form is only possible from a new put_packed() method, intended for use from the packer (and tests).
NB As noted in the TODO in get, this does not yet handle prefix removal on get, will address that in a separate diff.
Reviewed By: StanislavGlebik
Differential Revision: D21980498
fbshipit-source-id: f534b0e754aa29c42bf00bb7e764f93f1446c14c
Summary: Add blobstore link trait so we can use hardlink style links in fileblob and memblob for testing and later sqlblob et al for prod.
Reviewed By: StanislavGlebik
Differential Revision: D21935647
fbshipit-source-id: f76eaca26b6b226c77d6e39e9c64e02b4145b614
Summary: Fix the capitalization in the error message to match the other variants.
Reviewed By: singhsrb
Differential Revision: D22078125
fbshipit-source-id: 4a1de0111fa4dceea2d401c47ef145669a3cc6f5
