Summary:
`const_cstr::ConstCStr` is represented internally as a fat pointer with fixed size: `&'static str`. See https://docs.rs/const-cstr/0.3.0/const_cstr/struct.ConstCStr.html. Notably this is **different** from the representation of `std::ffi::CStr`, which is a dynamically sized type and normally passed around behind a reference, as `&CStr`. Using `&ConstCStr` in signatures, which is effectively like `&'a &'static CStr`, is confusing due to the discrepancy between the two relatedly named types. Additionally having two different lifetimes involved -- the static lifetime of the underlying bytes, and the short lifetime of the fat pointer -- is unnecessarily confusing when async code and a language boundary are involved.
The utf8-cstr crate uses what seems like a better representation to me than the const-cstr crate. See https://docs.rs/utf8-cstr/0.1.6/utf8_cstr/struct.Utf8CStr.html. `Utf8CStr` is the dynamically sized type, just like `CStr`. Then `&'static Utf8CStr` is how it would commonly be passed around, just like `&CStr`.
Reviewed By: krallin
Differential Revision: D27698169
fbshipit-source-id: ffe172c2c2fc77aeab6b0a0a8aed3e3c196098cc
Summary: Migrate the codebase away from the ad-hoc `folly::uint64ToBufferUnsafe` and to `folly::to_ascii_decimal` which is intended for these cases.
Reviewed By: WillerZ
Differential Revision: D27281875
fbshipit-source-id: 0c98749e4aed9c873853eed2221cf54a89279ff4
Summary:
The hashes that are passed in as parameters to the hash-to-location function
may not be hashes that actually exist. This change updates the code so that
we don't return an error when an unknown hash is passed in. The unknown
hash will be skipped in the list of results.
Reviewed By: quark-zju
Differential Revision: D27526758
fbshipit-source-id: 8bf9b7a134a6a8a4f78fa0df276f847d922472f5
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the client code to thread the list of master commits to the
EdenApi client.
Reviewed By: quark-zju
Differential Revision: D27523868
fbshipit-source-id: db4148a3f1d0e8b0b162e0ecc934e07f041c5511
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the request object for HashToLocation to send over all the
master heads.
When the server builds non-master commits, we will want to send over non-master
heads too. We may consider having one big list of heads but I think that we
would still want to distinguish the non-master commit case in order to optimize
that use-case.
Reviewed By: quark-zju
Differential Revision: D27521778
fbshipit-source-id: cc83119b47ee90f902c186528186ad57bf023804
Summary:
This scenario appears when master moves backwards. Since the master group in
segmented changelog is append-only. Non-fast-forward master move will cause
multiple heads in the master group.
Since Segmented Changelog was updated to handle multiple master heads, we can
propagate the full list that we get from the client.
This diff makes the assumption that Mononoke will know to convert all client "master head"
hashes from HgChangesetId (Sha1) form to ChangesetId (Blake2). If any of the master
heads cannot be converted then it means the server might not be reliably answer the
client's question (in "ancestors(master_heads)", translate "this hash" to a path, or tell me
confidently that the "hash" is outside "ancestors(master_heads)"). That's an error case.
Reviewed By: quark-zju
Differential Revision: D27521779
fbshipit-source-id: 219e08a66aac17ac06d2cf02676a43c7f37e8e26
Summary:
This scenario appears when master moves backwards.
Since the IdDag can handle multiple master heads, the server can piggy-back on that
functionality and support multiple master heads when translating location to hash.
Reviewed By: quark-zju
Differential Revision: D27521780
fbshipit-source-id: c27541890d4fda13648857f010c11a25bf96ef67
Summary:
`panic!()`, and things which use `panic!()` like `assert!()`, take a literal format
string, and whatever parameters to format. There's no need to use `format!()`
with it, and it is incorrect to pass a non-literal string.
Mostly it's harmless, but there are some genuinely confusing asserts which
trigger this warning.
Reviewed By: dtolnay
Differential Revision: D27672891
fbshipit-source-id: 73929cc77c4b3d354bd315d8d4b91ed093d2946b
Summary:
Modify the `Debug` implementation for `minibytes::Bytes` to use `std::ascii::escape_default` to debug print a `Bytes` as an ascii-escaped bytestring.
For comparison, the `bytes` crate `Bytes` type provides the same functionality, though it doesn't use the standard library `escape_default` function: https://docs.rs/bytes/1.0.1/src/bytes/fmt/debug.rs.html#39-43
This change greatly improves the output of the `debugscmstore` command. If we don't want to make this the default behavior, we can provide a formatting wrapper type or I can specialize the output in `debugscmstore`, but I can't see any real downsides, especially given the `bytes` crate does the same thing, and we have a similar specialization for `HgId` (hex format in that case).
Reviewed By: quark-zju
Differential Revision: D27642721
fbshipit-source-id: 8faba421fa5082a2098b13ef7d286e05eccb6400
Summary: Add the `with_key` function to `Entry`, which replaces it's key with a provided key. Currently, scmstore returns incorrect results when multiple entries exist with different paths but the same HgId (as scmstore directly returns the path found on disk locally). This isn't a problem in the legacy API, which returns a bare `Vec<u8>` content, which is implicitly associated with the requesting key because it is the result of a single `get` call, or is irrelevant because the `prefetch` method doesn't directly return the results.
Reviewed By: andll
Differential Revision: D27664025
fbshipit-source-id: 014d44ca9a1dc2721685622fd2b077ed3483838f
Summary:
We are moving our skelton for eden top a bit further, by creating the view
structure for eden top. This creates the widget for the banner of eden top,
the help page and the main page as well as a couple sections of the main
page.
The main page is displayed on default and they are toggled with `h` for help
page and `esc` to return to the mage page. Visable and hidden widgets are not
implemented in termwiz yet so we have to do a bit of hacking to hide and
display widgets for our selves.
Each of the sections is stubed with place holder text for testing.
Reviewed By: xavierd
Differential Revision: D26892620
fbshipit-source-id: a7bb4d0e11f3a8968ef071e7f585d07a9c286880
Summary:
D27659634 (8e8aaa61d6) removed these files, so let's drop their exclusions from
test-check-code.t
Reviewed By: sfilipco
Differential Revision: D27682136
fbshipit-source-id: f8e10fac37ea90fb2782b960faf4536f1ff9133b
Summary:
fctx is not guaranteed to have the _path and _filenode attributes. Those are
specific to implementations, e.g. `absentfilectx` does not have them.
`basefilectx` instead defines the `path()` and `filenode()` for general fctx
use.
Reviewed By: quark-zju
Differential Revision: D27667176
fbshipit-source-id: 1d7889d264b597665ef05f84a752323f078cb455
Summary:
Create a fork of the Mercurial code that we can use to build server
rpms. The hg servers will continue to exist for a few more months while we move
the darkstorm and ediscovery use cases off them. In the mean time, we want to
start making breaking changes to the client, so let's create a stable copy of
the hg code to produce rpms for the hg servers.
The fork is based off c7770c78d, the latest hg release.
This copies the files as is, then adds some minor tweaks to get it to build:
- Disables some lint checks that appear to be bypassed by path
- sed replace eden/scm with eden/hg-server
- Removed a dependency on scm/telemetry from the edenfs-client tests since
scm/telemetry pulls in the original eden/scm/lib/configparser which conflicts
with the hg-server conflict parser.
allow-large-files
Reviewed By: quark-zju
Differential Revision: D27632557
fbshipit-source-id: b2f442f4ec000ea08e4d62de068750832198e1f4
Summary:
Sometimes we can hit an Idle timeout error while talking to MySQL, because we open a connection and go idle for a long time. Then when we finally send a query, server returns an error: the connection is expired. This is the issue we found and fixed in D27503062 (a856799489) that blocked MySQL client release.
## Future starvation
Imagine you have a stream in which you're connecting to a server, fetching and preparing some values:
```
let v = vec![1u32, 2, 3, 4, 5];
let mut s = stream::iter(v)
.map(|key| async move {
let conn = connect(..).await?;
conn.fetch(..).await
})
.buffered(2)
.map(|item| async move { prepare(..) })
.buffered(2);
```
Now you want to asynchronously process those prepared values one by one:
```
while let Some(result) = s.next().await {
let value = result?;
process(value).await?;
}
```
This async `process(..)` call can be talking to some service to take these values or something else that doesn't require much of a CPU time. Although the operation can be long.
**Now what happens when we do s.next().await?**
Because the stream is `buffered(2)` we wait for the first 2 futures. When the first item is ready, it returns the result and polls next stream item with a key - 3. The third future only makes a `connect(..)` call and gets switched.
When we've got a next value from the stream, we're waiting on the `process(value)` call and not polling the underlying stream till the processing is done.
**As I mentioned earlier, it is not expensive...**
But what if it takes > 10s to complete anyway?
The third future from the stream, that was polled earlier, **will wait for all these > 10s till it is polled again**.
More details [in this post](https://fb.workplace.com/groups/learningrust/permalink/2890621307875402/).
## Solution
In this case spawning a future with connection and query steps is a way to fix the issue.
This diff spawns queries in `shed::sql::sql_common::mysql_wrapper` - this covers all the places in Mononoke where we talk to MySQL. Also I removed the spawn from hg sync code, because it is not needed anymore and to illustrate that this approach works.
Reviewed By: StanislavGlebik
Differential Revision: D27639629
fbshipit-source-id: edaa2ce8f5948bf44e1899a19b443935920e33ef
Summary:
`getdeps` builds sometimes fail - try a higher compression level to
see if it's a different default internally and in open source
Reviewed By: ahornby
Differential Revision: D27659420
fbshipit-source-id: 702341e6b288ab79584bfa8de5b1ccd5ed6bc57a
Summary:
Somehow, rmtree can fail to open the file being removed, in which case the
function is `os.open` that requires a flag on top of the path. When that
happens a `TypeError` will be raised:
```
✗ Fail: eden/integration:integration - test_eden_start_fails_if_edenfs_fails_during_startup (eden.integration.start_test.StartFakeEdenFSTestSystemdEdenCLI) (4.363)
Test output:
> TypeError: open() missing required argument 'flags' (pos 2)
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/case.py", line 717, in doCleanups
self._callCleanup(function, *args, **kwargs)
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/async_case.py", line 72, in _callCleanup
self._callMaybeAsync(function, *args, **kwargs)
File "/usr/local/fbcode/platform009/lib/python3.8/unittest/async_case.py", line 84, in _callMaybeAsync
ret = func(*args, **kwargs)
File "/usr/local/fbcode/platform009/lib/python3.8/contextlib.py", line 533, in close
self.__exit__(None, None, None)
File "/usr/local/fbcode/platform009/lib/python3.8/contextlib.py", line 525, in __exit__
raise exc_details[1]
File "/usr/local/fbcode/platform009/lib/python3.8/contextlib.py", line 510, in __exit__
if cb(*exc_details):
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 104, in __exit__
self.cleanup(exc_type is not None)
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 117, in cleanup
cleanup_tmp_dir(temp_dir)
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 71, in cleanup_tmp_dir
shutil.rmtree(tmp_dir, onerror=_remove_readonly)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 715, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 652, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 652, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 652, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/fbcode/platform009/lib/python3.8/shutil.py", line 648, in _rmtree_safe_fd
onerror(os.open, fullname, sys.exc_info())
File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/aab7ed39/eden/integration/integration#binary,link-tree/eden/test_support/temporary_directory.py", line 66, in _remove_readonly
func(path)
```
To bypass this, let's simply test if the function is os.unlink or os.rmdir
first.
Reviewed By: chadaustin
Differential Revision: D27654126
fbshipit-source-id: 318c547f363c020e98e21d5f07a9c7a9b2e5102d
Summary:
Our logs are filled with this message, making it almost impossible to read,
this doesn't need to be DBG1, let's reduce it to DBG5 similarly to what we do
for blobs.
Reviewed By: kmancini
Differential Revision: D27652526
fbshipit-source-id: c61d502a9f17d8180bad857ff6c1a9033763947d
Summary:
Those are not used. Recently we saw build issues like:
lib/third-party/sha1dc/sha1.c:8:10: fatal error: string.h: No such file or directory
#include <string.h>
^~~~~~~~~~
Possibly by some compiler flags disabling stdlib. Since we don't need
the C code let's just remove them.
Reviewed By: singhsrb
Differential Revision: D27659634
fbshipit-source-id: b8fcac4f05be95cac470f9b4830baf76e06f98ad
Summary:
Defend against the possibility of `nodeinfo["remote_bookmarks"]` being `None`
by treating it the same as an empty list.
Reviewed By: quark-zju
Differential Revision: D27658994
fbshipit-source-id: 06c31bdba2cc6cbd82f5bd718befd6be9bb233f2
Summary:
The `python` in `PATH` might be the fbcode Python that has `-nostdinc` in its
`CFLAGS`. That Python is problematic because it cannot compile C extensions.
Reviewed By: DurhamG, markbt
Differential Revision: D27661488
fbshipit-source-id: 243205522fcaf53d5af6a3c9afc4d28160072de5
Summary:
a [git worktree](https://git-scm.com/docs/git-worktree) allows for a single repository to have multiple local checkouts in different directories (allowing parallel work on multiple branches of the same repo).
in such a worktree, `.git` is a file and not a directory:
```
$ cat .git
gitdir: /path/to/actual/repo/.git/worktrees/name
```
the `_git_prompt` helper in `scm-prompt.sh` expects only directories:
```
$ echo "$(_git_prompt /path/to/worktree/.git)"
# (output is empty)
```
this amends the helper to support them:
```
$ echo "$(_git_prompt /path/to/worktree/.git)"
f140443f|REBASE-i|T123456
$ echo "$(_git_prompt /path/to/parent/.git)"
T54321
```
Reviewed By: quark-zju
Differential Revision: D27509147
fbshipit-source-id: 9a4ab55af99538c6425ad5721c3266386dbda453
Summary:
The "filter" set's filter function might not be prepared for inputs outside
the parent set. So let's the "contains" function to test against the parent
set first, then test the filter function.
This fixes the "merge()" set's "contains" check using the revlog backend:
In [1]: v=repo.revs('draft() & merge()')
In [2]: v._set
Out[2]: <meta ?>
In [3]: m.node.nullid in v._set
Out[3]: False
Before this diff it would be:
In [3]: m.node.nullid in v._set
CommitLookupError: '0000000000000000000000000000000000000000 cannot be found'
Note: Segmented changelog backend is not affected because it does not use
filter sets.
Reviewed By: xavierd
Differential Revision: D27657502
fbshipit-source-id: 30bb261fea59bdc5644580e98796f52fa93c2705
Summary: Update the `curl` and `curl-sys` crates to use a patched version that supports `CURLOPT_SSLCERT_BLOB` and similar config options that allow the use of in-memory TLS credentials. These options were added last year in libcurl version `7.71.0`, but the Rust bindings have not yet been updated to support them. I intend to upstream this patch, but until then, this will allow us to use these options in fbcode.
Reviewed By: quark-zju
Differential Revision: D27633208
fbshipit-source-id: 911e0b8809bc0144ad8b32749e71208bd08458fd
Summary: Currently no matter through which VIP clients are connecting to, they get a response with `"href": "https://monooke-lfs.internal.tfbnw.net"` which prevents us from enabling c2p vip in corp.
Reviewed By: krallin
Differential Revision: D27331945
fbshipit-source-id: f215cce2f64a2a38accd6d55d5100d8d364ce77b
Summary:
Previously we ran into an issue where client has sent us too large known
request, and we passed it all the way to the mysql.
Mysql slow log shows that we have quite a few slow queries
(https://fburl.com/scuba/mysql_slow_log/w0ugmc1i), so it might be that these
requests are still coming, but because of the problems in the logging (see
previous diff), we can't know it for sure.
In any case, adding a knob like that can be useful
Reviewed By: farnz
Differential Revision: D27650806
fbshipit-source-id: c4c82b7b5781a85c349abb4e5fa534b5e8f125a0
Summary:
the code is almost the same, so it would be good to deduplicate it. The
duplication let to the annoying differences in logging - i.e. we logged how
many nodes were sent to us in `known` call but not in `knownnodes` call.
Reviewed By: farnz
Differential Revision: D27650583
fbshipit-source-id: 5e2e3be3b9fd66631364d23f34d241c27e370340
Summary:
Now that the `hg_external_sync` jobs are gone we can delete the code
in Mononoke that behaves differently when a sync job connects.
Reviewed By: StanislavGlebik
Differential Revision: D27500506
fbshipit-source-id: 443fb54577833dbf44ece6ae90a5f25ffed38cd5
Summary: This was added in D27503062 (a856799489) as a debug info and is very spammy, let's remove it.
Reviewed By: StanislavGlebik
Differential Revision: D27647927
fbshipit-source-id: 12c6b2d4cb8b1bae2d987fd8ff461bd480b7dc18
Summary:
Currently if we failed to fetch the repo status we only see "Repo is marked as read-only: Failed to fetch repo lock status" error, which is not very informative. Example of the error in production: P385612782.
Let's log the error.
Reviewed By: krallin
Differential Revision: D27621996
fbshipit-source-id: 85d9f0fe39397759da1b51e197f9188761678715
Summary:
Add support for returning unhydrated draft commits if requested by the client using a config option 'wantsunhydratedcommits'
This is needed to support slow enabling it for some clients like OnDemand.
Reviewed By: StanislavGlebik
Differential Revision: D27621442
fbshipit-source-id: 672129c8bfcbcdb4cee3ba1b092dac16c0b1877d
Summary:
We already log file count, but file sizes is another useful piece of
information.
I evaluated two options - either do as I did in this diff or just change ScribeToScuba
logging python script to query scs to get file sizes. I opted for option #1
mainly because scs doesn't have a method to query file sizes for many files at once, and
querying one by one might be too expensive. We can add a method like that, but
that's a bit of a bigger change that I'd like.
Reviewed By: andll
Differential Revision: D27620507
fbshipit-source-id: 2618e60845bc293535b190d4da85df5667a7ab60
Summary: This API has moved in Python 3. This is only used for util.popen4 calls, which aren't super frequent, so it escaped notice until now.
Reviewed By: sfilipco
Differential Revision: D27603246
fbshipit-source-id: 9ee6af4f66380480fe6c1c736287231464d0b21e
Summary: We were accidentally not setting the TLS key here; this diff fixes it.
Reviewed By: quark-zju
Differential Revision: D27634276
fbshipit-source-id: 9aac3a34b6f6655059a8d3332eea8ba02d062651
Summary:
Usage is hidden under config experimental.nativerebase
When config is set, we attempt to execute native rebase, and check if it has conflicts
If there are no conflicts, we convert native rebase result into Python actions
In case of conflicts, we currently just fallback to python merge
Reviewed By: DurhamG
Differential Revision: D27482909
fbshipit-source-id: 5705d372095f64ce0c6ee93c783e1c8dccf8b11a
Summary:
This diff contains proposal for basic rebase model for new checkout.
This diff introduces two types of file changes - basic no conflict Actions(that will replace CheckoutPlan actions) and ConflictAction for actions that contain conflicts
No conflict actions are produced when comparing two manifests
Conflict actions can be produced when rebasing manifests.
Eventually we will adjust checkout code to take either list of Actions or Conflict actions and perform corresponding operations on the file.
Rebase::rebase is used to generate list of conflict actions when rebasing.
Currently `Rebase::rebase` produces list of actions on top of 'base' commit.
Before doing actual checkout we will need to take into account what current commit is and adjust this list of actions accordingly (however, this adjustment will be trivial, it won't produce more conflicts)
After getting adjusted list of actions, checkout code can roll it out to fs and call merge drivers if needed.
In the future, if we will want to allow check in conflicts, we will need to adjust `Rebase::resolve_conflict` to take `ConflictAction` as input, instead of an Action.
Reviewed By: DurhamG
Differential Revision: D27028001
fbshipit-source-id: b02b7ad8030fcf146b53639d69e66fdc8f38a62d
Summary:
The nameset._set.iter() used by nameset.{fastdesc,fastdesc} do not know how to
prefetch. Just make them return `None` in this case. This addresses issues
where `log` can trigger one-by-one fetching.
hg log -r "limit(all(), 100000, 100000) & date('2021-04-01')"
The stack is a bit deep, it is:
# cmdutil.getlogrevs
if not revs:
...
# smartset.filteredset.__nonzero__
fast = None
candidates = [self.fastasc, self.fastdesc]
...
if fast is not None:
it = fast()
Basically filteredset calls into its _subset (nameset)'s fast{asc,desc},
which does not do prefetch properly and that cause filteredset.__nonzero__
to be slow.
Reviewed By: sfilipco
Differential Revision: D27634020
fbshipit-source-id: c3a44a1eb81c32f4dccb319ce42a15c56121c2ec
Summary:
Fixes MySQL syntax errors we've seen in some cases. No reason to call to the
database if we have no items to query for. It seems that empty queries can come
from the caching layer under certain configurations.
Reviewed By: krallin
Differential Revision: D27624798
fbshipit-source-id: 2febeff127f2fbdc739368ff1d1065c8f64723f8
Summary:
Remove the `DangerousOverride` mechanism for bookmarks traits and config.
Bookmarks overrides were only used to make unit test bookmarks use the same
metadata database as `SqlCounters`, which can now be done safely by getting the
metadata database from the factory.
Config overrides can be performed by the factory at repo construction.
Reviewed By: krallin
Differential Revision: D27424695
fbshipit-source-id: 0259da3abedb7ed4944fe945ba89800eea76ebff
Summary:
This dangerous override was being used to override
derived data config. Replace it with customizing the
config in the factory.
Reviewed By: krallin
Differential Revision: D27424696
fbshipit-source-id: 6dcf0c1397e217f09c0b82cf4700743c943f506f
Summary: This has been superseded by `RepoFactory`.
Reviewed By: krallin
Differential Revision: D27400617
fbshipit-source-id: e029df8be6cd2b7f3a5917050520b83bce5630e9
Summary:
Use `RepoFactory` to construct repositories in the walker.
The walker previously had special handling to allow repositories to
share metadata database and blobstore connections. This is now
implemented in `RepoFactory` itself.
Reviewed By: krallin
Differential Revision: D27400616
fbshipit-source-id: e16b6bdba624727977f4e58be64f8741b91500da
Summary: Add a way for users of `RepoFactory` to customize the blobstore that repos use.
Reviewed By: krallin
Differential Revision: D27400615
fbshipit-source-id: e3e515756c56dc78b8de8cf7b929109d05cec243
Summary:
Remove the dependency on blobrepo factory by defining a custom facet factory
for benchmark repositories.
Reviewed By: krallin
Differential Revision: D27400618
fbshipit-source-id: 626e19f09914545fb72053d91635635b2bfb6e51
Summary: Use `RepoFactory` to construct repositories in the LFS server.
Reviewed By: krallin
Differential Revision: D27363465
fbshipit-source-id: 09d5d32a133f166c6f308d56b2fb02f00031a179
Summary: Use `RepoFactory` to construct repositories in the microwave builder.
Reviewed By: krallin
Differential Revision: D27363468
fbshipit-source-id: 25bf2f7ee1ac0e52e1c6d4bda0c50ba67bc03110
Summary: Use the equivalent function from `repo_factory`.
Reviewed By: krallin
Differential Revision: D27363470
fbshipit-source-id: dce3cf843174caa2f9ef7083409e7935749be4cd
Summary:
This import is only used for the `ReadOnlyStorage` type, which is canonically
defined in `blobstore_factory`.
Reviewed By: krallin
Differential Revision: D27363474
fbshipit-source-id: 78fb1866d8a1223564357eea27ec0cdbe54fb5db
Summary:
This import is only used for the `ReadOnlyStorage` type, which is canonically
defined in `blobstore_factory`.
Reviewed By: krallin
Differential Revision: D27363466
fbshipit-source-id: 7cb1effcee6d39de92b471fecfde56724d24a6a4
Summary: Use `RepoFactory` to construct repositories for the hook tailer.
Reviewed By: krallin
Differential Revision: D27363472
fbshipit-source-id: 337664d7be317d2cfc35c7cd0f1f1230e39b6b43
Summary:
This import is only used for the `ReadOnlyStorage` type, which is canonically
defined in `blobstore_factory`.
Reviewed By: krallin
Differential Revision: D27363467
fbshipit-source-id: ed1388e661453e1b434c83af63c76da1eea1bce1
Summary: Use `RepoFactory` to construct repositories for all users of `cmdlib`.
Reviewed By: krallin
Differential Revision: D27363471
fbshipit-source-id: c9a483b41709fd90406c6600936671bf9ba61625
Summary:
Switch from `blobrepo_factory` to the new `RepoFactory` to construct `BlobRepo`
in `mononoke_api`.
The factory is part of the `MononokeEnvironment`, and is used to build all of the repos.
Reviewed By: krallin
Differential Revision: D27363473
fbshipit-source-id: 81345969e5899467f01d285c232a510b8edddb17
Summary:
To facilitate migration from `blobrepo_factory` to `repo_factory`, make common
types the same by re-exporting them from `repo_factory` in `blobrepo_factory`.
Reviewed By: ahornby
Differential Revision: D27323371
fbshipit-source-id: 9b0d98fe067de7905fc923d173ba8ae24eaa0d75
Summary:
Add a factory for building development and production repositories.
This factory can be re-used to build many repositories, and they will share
metadata database factories and blobstores if their configs match.
Similarly, the factory will only load redacted blobs once per metadata
database config, rather than once per repo.
Reviewed By: krallin
Differential Revision: D27323369
fbshipit-source-id: 951f7343af97f5e507b76fb822ad2e66f4d8f3bd
Summary: I found the Writer based zstd::Encoder api was doing a lot more allocations than the buffer based Compressor api, so switched as its both faster and better fit to the usecase.
Differential Revision: D27588448
fbshipit-source-id: ee8f72180045308a2e16709b9b5aa7bcf3b5cafd
Summary: Reduce amount of manual steps needed to restart a manual scrub by checkpointing where it has got to to a file.
Differential Revision: D27588450
fbshipit-source-id: cb0eda7d6ff57f3bb18a6669d38f5114ca9196b0
Summary: Here we further add prefetch-metadata support to prefetching profiles
Reviewed By: genevievehelsel
Differential Revision: D27568542
fbshipit-source-id: 64507125f47cf093c0133c82fcab941ed6495f32
Summary:
This is ... a stopgap :( There is probably some slow polling happening in
unbundle_future, and this causes us to fail to use our connection in time in
check_lock_repo...
Reviewed By: ahornby, StanislavGlebik
Differential Revision: D27620728
fbshipit-source-id: b747011405328b60419a99f0e5dbbaf64d53196a
Summary:
This one is a little bit trickier since we want to use Tokio inside a
Quickcheck function. That said, this is basically the expansion `tokio::main`
does, so we can simply use it.
Reviewed By: farnz
Differential Revision: D27619146
fbshipit-source-id: 1e3ea2d119913900d9b55c0a6d33de8a6ed5781c
Summary:
I'd like to just get rid of that library since it's one more place where we
specify the Tokio version and that's a little annoying with the Tokio 1.x
update. Besides, this library is largely obsoleted by `#[fbinit::test]` and
`#[tokio::test]`.
Reviewed By: farnz
Differential Revision: D27619147
fbshipit-source-id: 4a316b81d882ea83c43bed05e873cabd2100b758
Summary:
The intention is that the packer decides what to pack and in what order, PackBlob provides the methods needed to do the packing as requested by a packer.
Change the API so that a packer cannot make mistakes
Reviewed By: ahornby
Differential Revision: D27476427
fbshipit-source-id: 7dd534302c62b2432a2aca474f49da8ab9cbef1a
Summary:
It is useful to have latency stats grouped by the shardmap and label to easily identify where the problem comes from if something is broken.
This diff switches a single histogram used for all the MySQL use-cases into a set of histograms: one per `shardmap:label`. Ans also makes the histograms a bit smaller as we don't actually have such big numbers as 10s per conn/query.
There is only one case when the histogram is created per shard instead of a shardmap, it is `xdb.hgsql` DB with 9 shards. The reason why it happens it's because we connect to each shard as to an individual tier: https://fburl.com/diffusion/um8lt7cr.
{F582699426}
Reviewed By: farnz
Differential Revision: D27503833
fbshipit-source-id: 40c7eb64df7ae0694f63d3644231f240df8212ec
Summary: introduce a way of requesting unhydrated commits using client telemetry
Reviewed By: StanislavGlebik
Differential Revision: D27591868
fbshipit-source-id: 7616f9d3b222c2c7f43c2ba930086eb0ec9e15bc
Summary:
Only used by one test that can define the constaint itself.
The problem with having it on the trait is that it's a bit noisy when
things operate on ToApi at the generic level. It adds to the list of
constaints they these users of the ToApi trait need to add.
Reviewed By: kulshrax
Differential Revision: D27549922
fbshipit-source-id: fff9e513eb4c06862111ce6eecc84ab981eea893
Summary:
This is only used in one utility which can define the constaint itself.
I am looking to simplify the Requirements for ToWire so that we can more
easily iterate on them. Debug as a requirement produces too much noise.
There is the risk of ending up in a scenario where we want to print the Wire
type but it's more practical to annotate structs with derive Debug when that
happens than to add the constaint in the trait.
Reviewed By: kulshrax
Differential Revision: D27549925
fbshipit-source-id: aacf7c1c465c94414be02aa143187897c7084980
Summary:
There is no use for it outside of one test which can describe that constraint
itself.
I think that requiring ToWire and ToApi to use the same objects is too much
for the general use case. We regularly convert between different object types
that are the same logical thing but have different representations. A good
example for that is the HgId. It makes sense to implement ToWire for all HgId
variations.
Reviewed By: kulshrax
Differential Revision: D27549924
fbshipit-source-id: d76d7a4beb528634bed46ae93dbd634d850547e5
Summary:
For async requests, we perform a blocking request in a separate thread, and stream the results back through a channel. However, if the curl handle for the request is dropped before starting the request (for example, because of a configuration error), this function would return a `oneshot::Canceled` error (from the channel consumer) instead of the real error message from the IO thread.
This diff fixes the issue by ensuring that the function waits for and returns the error message from the IO thread in the event that the IO thread returns before starting the request.
Reviewed By: quark-zju
Differential Revision: D27584502
fbshipit-source-id: 8447c158d253c3f28f03fcc4c36a28698fe6e83d
Summary:
This adds command line argument `-I` that supplies \0-separated list of files to add to commit.
Added files can be ignored/untracked.
No limit on total size for now, still waiting to hear from mononoke team on max files size
Reviewed By: quark-zju
Differential Revision: D27547822
fbshipit-source-id: 8bb755db5dd6e557e2752381dbeb5f1035073725
Summary: This will be used in ephemeral commit, since by default it does not need to include untracked files
Reviewed By: quark-zju
Differential Revision: D27580975
fbshipit-source-id: 16c4faa92e9afe472ff1677e5b92507bebaee247
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
* use `std::nullopt`
* TODO about sandcastle_instance_id in opensource version
Reviewed By: chadaustin
Differential Revision: D27575732
fbshipit-source-id: bf76970a15fee5a3dc1e4e411ea70f5af7248496
Summary:
When creating a commit via scs api we need to validate a few things (e.g. that
the file that a commit is trying to delete existed in the parent), and in order
to do that we need to use a manifest. Previously we were using fsnodes
manifests, however fsnodes is the slowest manifest to derive, and that makes
the whole create_commit operation much slower. Let's try to use skeleton
manifest, which is the fastest to derive.
Reviewed By: markbt
Differential Revision: D27587664
fbshipit-source-id: a60cab4956063bf26c0f1ec8c9cfa05233bb3cc0
Summary:
Previously ChangesetPathContext was holding both fsnode_id and unode_id, however it made it easier to misuse api and trigger expensive fsnodes or unodes path traversal (see more info in the comments for D27587664).
This diff splits it in two separate types.
Also I noticed that ChangesetPathContext wasn't using the `unode_id` future that it stored, so I just deleted it.
Reviewed By: markbt
Differential Revision: D27590997
fbshipit-source-id: 08fc14d33c82357275413c4cf2698f97620503ea
Summary: The default limit for commit cloud interactive history should be two weeks, not two days.
Reviewed By: farnz
Differential Revision: D27589697
fbshipit-source-id: 4314621fa7f06dac9243eb9b826acc1c7b0c0b10
Summary:
Hg sync jobs were frequently failing due to the task performing MySQL query being starved.
It acquired a connection but then waited for many seconds until it could finally send a query. At that time the server returned error: the connection was open idle for >12s and now timed out:
```
I0401 11:08:32.085223 390 [main] eden/mononoke/mononoke_hg_sync_job/src/main.rs:355] error without entry
E0401 11:08:32.086126 390 [main] eden/mononoke/cmdlib/src/helpers.rs:336] Execution error: While executing ReadNextBookmarkLogEntries query
Caused by:
0: While making query 'SELECT id, repo_id, name, to_changeset_id, from_changeset_id, reason, timestamp,
replay.bundle_handle, replay.commit_hashes_json
FROM bookmarks_update_log log
LEFT JOIN bundle_replay_data replay ON log.id = replay.bookmark_update_log_id
WHERE log.id > 19896395 AND log.repo_id = 2106
ORDER BY id asc
LIMIT 20'
1: Mysql Query failed: Failed (MyRouter) Idle timeout after 12 seconds see https://fburl.com/wait_timeout
I0401 11:08:32.172088 390 ClientSingletonManager.cpp:95] Shutting down Manifold ClientSingletonManager
remote: pushkey hooks finished (after 0.00s)
Error: Execution failed
```
Link to the full logs in a timeframe: https://fburl.com/tupperware/16th1yk7 (I added a debug output when `ReadNextBookmarkLogEntries` query runs).
Hg sync job initiates an infinite loop to look for the new commits to synchronize. In the async stream it runs `ReadNextBookmarkLogEntries` query and then prepares bundle and synchronizes it. The stream is [buffered](https://fburl.com/diffusion/z1r7648f) by [5 (link)](https://fburl.com/diffusion/surn37hx).
My guess is that the `ReadNextBookmarkLogEntries` query starts executing, while the previously discovered bundles are being prepared. The query opens a connection and then gets switched, now the bundles are being synced. But sometimes those bundles take too long to sync while the query task is waiting till it be executed again.
The sync finishes and the query task finally tries to send a MySQL query but hits an idle timeout error on the server.
This diff:
* Spawns the MySQL query and `apply_bundle` call.
* Adds watchdog on futures to help debug issues if they occur later, although I couldn't see any slow polls in the logs.
Reviewed By: StanislavGlebik
Differential Revision: D27503062
fbshipit-source-id: 6d1d9166b99487c056f3fb217502f8a9d3d46228
Summary:
Some manual scrub runs can take a long time. Provide progress feedback logging.
Includes a --quiet option for when progress reporting not required.
Reviewed By: farnz
Differential Revision: D27588449
fbshipit-source-id: 00840cdf2022358bc10398f08b3bbf3eeec2b299
Summary: D27591073 (a1e2833377) made the histogram smaller, so this is sufficiently fast to call directly.
Reviewed By: krallin
Differential Revision: D27592432
fbshipit-source-id: 50d3d594b237b87cc9d0a90910a6f022b7c40f2a
Summary:
There was no reason for this to be this large, and it's causing issues with
repo construction since it's pretty expensive to construct as a result
(D27501915 (69896e90b5)).
Let's just make it much smaller.
Reviewed By: StanislavGlebik
Differential Revision: D27591073
fbshipit-source-id: 1c986cb922d70b10c39711c57ac9f5899ed7496c
Summary:
This warmer hasn't been used, and we aren't going to use it going further.
Let's remove it
Reviewed By: krallin
Differential Revision: D27533802
fbshipit-source-id: deaaf954ed535789ab6b5dc89b8da9967c40e84f
Summary:
The comment seem to be incorrect - it wasn't checking that all files from a
replaced directory were removed (in fact, we allow to not remove them, see
https://fburl.com/diffusion/wdhu5arg).
Instead it was checking whether a file that was replaced with a directory was indeed
removed.
Reviewed By: markbt
Differential Revision: D27534121
fbshipit-source-id: 4f0d53096c2665fec8bb575ee183411c8edf2ccb
Summary:
This crashes with `ui.ignorerevnum=1`.
Rev number should use `%d`. Or use node and `%n`. `%s` is not the right way.
Reviewed By: kulshrax
Differential Revision: D27527470
fbshipit-source-id: 115385d8bb8dd006fcbf62dee1099b8f9d5262c7
Summary:
Since we're rolling out native checkout and resumable checkout around
the same time, let's make resumable checkout optional so we can turn it off it
causes issues, without turning off native checkout.
Reviewed By: quark-zju
Differential Revision: D27481986
fbshipit-source-id: a0a3e68567ca2a468e852dce95c03c4b606aaf22
Summary: This makes it easier to use.
Reviewed By: kulshrax
Differential Revision: D27406589
fbshipit-source-id: 11bef407ab620859381c6ee952e6ef00494551e1
Summary:
The issue is that `mut i: usize` is no longer shared across multiple `async
move` blocks (introduced by D27308798 (0df4efa969)).
Rewrite the logic to collect `ids` first, then use `vertex_name_batch`
query instead.
Reviewed By: sfilipco
Differential Revision: D27406586
fbshipit-source-id: b41fe3a13114dc34aa5763e6e2bebe0571decc87
Summary:
Merge paths like `x~n` and `x~(n+1)` to `x~n (batch_size = 2)`.
This could be more efficient bandwidth-wise and algorithm-wise.
Reviewed By: sfilipco
Differential Revision: D27406587
fbshipit-source-id: f2a67352ad627945685e33667e8299a2bc652930
Summary: IdSet is more compact. This changes the order a bit.
Reviewed By: sfilipco
Differential Revision: D27339279
fbshipit-source-id: e9b50a47beba081b892eccd7711dbd6ab5c3a886
Summary: This will be used by the next change.
Reviewed By: sfilipco
Differential Revision: D27406591
fbshipit-source-id: fcacc35a9ae8ed96cebb2af804d26d1e5e83ad9e
Summary:
Add a way to flush the overlay map to disk so we can avoid network fetches over
and over.
Reviewed By: sfilipco
Differential Revision: D27406592
fbshipit-source-id: 7086ad665119cc3a0834f533690325c7a2363442
Summary: It will be reused elsewhere.
Reviewed By: sfilipco
Differential Revision: D27406593
fbshipit-source-id: 296cf5f50830bb7285e0cb9c7c15a9b374689819
Summary:
I spent some time thinking about how to flush the "overlay_map" to disk.
It is a bit tricky because the on-disk IdMap might have changed in an
incompatible way. I tried a few ways to verify the on-disk IdMap remains
compatible and didn't find a way that looks good (either slow - calculating
universal_ids, or is not 100% correct in corner cases).
Now I come up with this "just track x~n" idea. It trades memory usage (~2x
overlay_map) for easy-to-verify correctness, and efficient overlay_map
flush.
Reviewed By: sfilipco
Differential Revision: D27406583
fbshipit-source-id: 0b7fb3186a9c15f376c1dc4afe7f0516c25d3dec
Summary: It is not obvious. So let's add more comments.
Reviewed By: sfilipco
Differential Revision: D27406584
fbshipit-source-id: 9ce1215efc1a6d4849180c6693616613c08f2a51
Summary:
A sparse dag does not have full IdMap. Its IdMap only contains "universally known" entries.
Add a basic test about cloning from a sparse clone data and resolve vertex <-> id mapping
on the fly.
Reviewed By: sfilipco
Differential Revision: D27352018
fbshipit-source-id: 4a3f5f50be52e91bf7b2021cdc858bcab9c99e80
Summary:
The `NameDag::flush` API will actually rebuild the graph using a "parent" function.
That is not necessary if we got clone data, and won't work well for a lazy graph
(since the parent function talks about vertex names and some names are missing).
Let's bypass the `flush` function and write data directly in `import_clone_data`.
Reviewed By: sfilipco
Differential Revision: D27352019
fbshipit-source-id: a79569d25d858447b8c5eb86902b8d39ae0429a3
Summary: This will be used in tests.
Reviewed By: sfilipco
Differential Revision: D27343882
fbshipit-source-id: 5a2d94a9f755eed0fc27e5a11093b55c810dc8da
Summary:
Implement logic to export the clone data. This allows us to construct a sparse/lazy
dag via export + import CloneData.
Reviewed By: sfilipco
Differential Revision: D27343885
fbshipit-source-id: 71dc0d31e36876a8b6a8c3d7f3498be3262ce297
Summary:
Clone data can only be imported to an empty Dag and universally known vertexes
should be present in the IdMap.
Reviewed By: sfilipco
Differential Revision: D27343888
fbshipit-source-id: ba150d6afdbe15f0902ec20ff150a70657e24c80
Summary: Collect "missing" items and only use one request to fetch them.
Reviewed By: sfilipco
Differential Revision: D27406588
fbshipit-source-id: a5cd091b39d90c1ad0e7c5d509673c4665232304
Summary:
This will be used by upcoming changes. Sparse/Lazy NameSet will override it
to reduce network round-trips.
Reviewed By: sfilipco
Differential Revision: D27406590
fbshipit-source-id: a44a73b4aec6e14d6e82d55285fe1cfc0fcfd482
Summary: This will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27343884
fbshipit-source-id: 0938b1fb3d90b35f9d51c468cffca53e3f421bb8
Summary: The `mut` was unused. Remove it to make the interface more flexible.
Reviewed By: sfilipco
Differential Revision: D27406594
fbshipit-source-id: 1cfa4921015fc89b6c71ed4a97d9c351f56c7370
Summary:
Remove some TODOs. This serves as fallback paths where batch prefetch didn't happen.
I'd expect most use-cases will trigger IdStatic set's batch prefetch logic (to be
added).
I haven't decided what to do exactly with "contains". Fetching remotely there seems
to require some kind of negative cache (ex. in mutation records there might be nodes
not in the graph). But it _might_ be okay to say the "contains" is a local-only
operation, too. I leave it as-is and we'll see how the Python world uses "contains"
later.
Reviewed By: sfilipco
Differential Revision: D27339275
fbshipit-source-id: ba70b3c84a391a8e395c73ccd1d7e08f92b0cbd0
Summary:
Put everything together. I used "programming error" extensively to
provide more context if we have to investigate issues in the future.
Reviewed By: sfilipco
Differential Revision: D27339278
fbshipit-source-id: 574a2c048dc1d24dbe690f862fec3e5078cb067a
Summary: Provide more context about what invariants we expect. Not just show "vertex not found".
Reviewed By: sfilipco
Differential Revision: D27339273
fbshipit-source-id: 1c6c92537ff37666ff603783adfd8f9ea770fbaa
Summary:
Makes NameDag own the state (logic) about how to send remote requests.
So NameDag can send requests on its own.
Reviewed By: sfilipco
Differential Revision: D27339282
fbshipit-source-id: 3cb6327dfeaefae45d4e7b88a3535463a84b195b
Summary: This will make the overlay IdMap effective when query against the NameDag.
Reviewed By: sfilipco
Differential Revision: D27339276
fbshipit-source-id: 80712bf651beb6c7e9f23bd4233c6d916101696a
Summary: To be able to quickly find all activity related to given Sandcastle job as discussed in [this workplace post](https://fb.workplace.com/groups/2120196508269853/permalink/2899040243718805).
Reviewed By: kmancini
Differential Revision: D27541803
fbshipit-source-id: a55900064bbee92da902de785ebe0c0e8738c3a2
Summary:
When watchman is in use, removing tons of files very quickly leads to FSEvents
losing events, causing watchman to recrawl the entire repository. It's not
entirely clear today what the maximum number of threads to use is, let's make
it configurable.
Reviewed By: andll
Differential Revision: D27536577
fbshipit-source-id: 58f2bc453321fd4e7468d3324ee4df2b79b96d5d
Summary:
We already checked size and mtime, which should allow us to identify if
a file has changed since the last update, but we need to check hgid as well, to
make sure the previously-written content is supposed to be the same as the
about-to-be-written content
Reviewed By: andll
Differential Revision: D26955401
fbshipit-source-id: 859ad35b008e68d699601217f2a4398c6789913e
Summary:
Updates native checkout to store which files have already been written
in .hg/upgradeprogress. Then enables it to load that file and skip writing those
files if they're already on disk and their mtime and size match the previously
written values. In theory we could record and check file hashes as well, but
that'd likely slow things down quite a bit.
Future diffs will add:
- Recording and checking the hgid that was written before vs what is about to be
written. Just an hgid comparison, not a full hash computation.
- Some UX to inform the user when hg checkout can be continued, and possibly to
implement 'hg checkout --continue'.
Reviewed By: andll
Differential Revision: D26830249
fbshipit-source-id: 88a75080966dae5241550ed7eedbc057c65966dd
Summary:
Currently eden on startup prints:
```
Starting edenfs (dev build), pid 190
Opening local RocksDB store...
Opened RocksDB store in 0.073 seconds.
Could not parse config.json file: couldn't read /var/twsvcscm/local/.eden/config.json: No such file or directory
Skipping remount step.
Started edenfs (pid 190)
Logs available at /var/twsvcscm/local/.eden/logs/edenfs.log
```
Would be convenient if it also printed session id, to be able to query scuba straight away.
Reviewed By: chadaustin
Differential Revision: D27522665
fbshipit-source-id: d7d4cf6c97bc551061761f2653375f208e393498
Summary: Refactoring to make the following diff smaller.
Reviewed By: chadaustin
Differential Revision: D27522581
fbshipit-source-id: 8f858714fcbfe4b8f8b1c3678bb2003623abbd94
Summary:
Still trying to enable Scuba logging in ovrsource TD to figure out what caused Buck regression when we started working with multiple watchman instances.
Did not try previous fix of pgrp (deploying stuff on Sandcastle is not trivial), although now I'm not sure that pgrp was the issue. Hard to say.
But now I'm launching CI differently and observe different symptoms.
`eden start` command finishes in 30 seconds (according to logs), but Sandcastle waits for something for 10 minutes and then somebody kills `scribe_cat` and Sandcastle continues. I don't really know what that means, but there's another issue I discovered:
`scribe_cat` inherits a copy of stderr fs created during daemon startup.
This diff fixes the issue.
Reviewed By: kmancini
Differential Revision: D27494520
fbshipit-source-id: 069f4e9ea1efb553cf7a7f18e20ae92c27da808d
Summary:
I see in Sandcastle logs it was 10 minutes between `eden start` command and the following command.
So I'm curious, is it eden took 10 minutes to start or Sandcastle did something else but did not log it?
This little message in log will help to understand that.
Reviewed By: kmancini
Differential Revision: D27491709
fbshipit-source-id: 796c8db5abe49b056bd81b03ea57585a761c3cb6
Summary:
`RendezVousConnection::new` can block for some time doing work on the CPU,
specifically creating the stats objects. This causes problems for other
futures during repo construction.
Instead, move rendez-vous construction to a `spawn_blocking` closure, so that
it doesn't interfere with the other futures.
Since `SqlBonsaiHgMapping::from_sql_connections` is not async, and is part of
the SqlConstruct trait, we must convert this to the builder pattern so that we
can defer rendez-vous construction to later on.
Reviewed By: farnz
Differential Revision: D27501915
fbshipit-source-id: 9c58c32411301128424985deeab127d052c43532
Summary: Vendor updated zstd crates. Started on this as a prerequisite for vendoring updated async-compression crate, however dependency resolution means this actually updates async-compression as well.
Reviewed By: farnz
Differential Revision: D27467760
fbshipit-source-id: 74ca9e4da9f336cf609cf06c62559c9ef913c9a2
Summary:
The extensions loading logic built up an _order list to track what order extensions should be loaded in. It also treated that list as the list of what has already been loaded, if we load extensions multiple times (like once at dispatch, then later for repo creation). Unfortunately some extensions, like tweakdefaults, modify the _order list, so when extensions uses len(_order) to decide which extensions are new, it sometimes gets the wrong results because tweakdefaults has inserted values at the beginning of _order which messes up the offset.
This causes some extension's uisetup and extsetup to be invoked twice. With dynamicconfig on Windows, I was seeing pushrebase load twice, which caused an error due to double registering bundle2 parts.
Extension loading is a mess, so this whole ordering concept really doesn't make sense, but for now let's fix it by not relying on the _order offset.
Reviewed By: quark-zju
Differential Revision: D27470828
fbshipit-source-id: e7ab8f87dfde22d82e5e080e8c01b884458b30c3
Summary:
We call `setsid` too late, [on successful startup](https://fburl.com/diffusion/p88am07t), after `scribe_cat` binary launched.
As a result, `scribe_cat` does not belong to process group of eden, it belongs to the process group of launcher process, so it is killed by `timeout` command in ovrsource CI.
To deal with it, this diff proposes additionally calling `setpgrp` call early during initialization.
Reviewed By: chadaustin
Differential Revision: D27436474
fbshipit-source-id: acb168c1ab5b78b1598c34ebece88847a9a6480d
Summary:
Let's give some time for bookmarks to warm up before we start checking the
location of the master bookmark.
Reviewed By: quark-zju
Differential Revision: D27472520
fbshipit-source-id: f20e6f04acadaf8b7caf6aa38a6ab9d244c6f8ea
Summary:
In the case where EdenFS crashes, any client performing IO to EdenFS would be
stuck in the D state waiting for the server to be back up and running. Since on
restart EdenFS will chose different ports, this makes it hard/impossible to
actually recover and will keep some process unkillable, often time requiring
rebooting a host.
To avoid this, we can make the mount "soft" which aborts the IO after some time
has elapsed and the server isn't back up. This should enable us to kill these
processes and recover more gracefully.
Reviewed By: chadaustin
Differential Revision: D27402817
fbshipit-source-id: ff81a4360900c4c94665174e3c0cf63402f1533e
Summary:
Add a `set_global_config` function to `hg-http` that gets called prior to command execution. This function can be used to configure the global behavior of all HTTP requests by setting up a callback that modifies every `Request` prior to sending.
Right now, the use case for this is to support the `--insecure` flag, as can be seen in the code.
Reviewed By: quark-zju
Differential Revision: D27243638
fbshipit-source-id: 61a52c1f4b56fffd860c2a7e9c7b03e146b2240a
Summary: Add the `--insecure` flag to the list of global hg flags. This flag is already supported by Python, and adding it here will make it available to Rust code as well. This is used later in the stack.
Reviewed By: quark-zju
Differential Revision: D27243029
fbshipit-source-id: 150d42ee96e1e3194ff1be6a33d9b36887d86f2c
Summary:
The `focusedbranch` revset should only include commits that are related via
draft commits. If a commit is landed, its public successor shouldn't cause
focused smartlog to include descendants of the landed commit.
Reviewed By: quark-zju
Differential Revision: D27461850
fbshipit-source-id: c0eb4fbcd6cf1cd78a88e3f796b5a16257dfb190
Summary:
When running `sudo mkscratch path /home/markamendoza`, the folder `/data/users/markamendoza/scratch/homeZmarkamendoza` will be created, including all parents that did not already exist. All these new directories will be created with owner = root.
Anticipating this kind of call, `mkscratch` already goes in and manually modifies the owner of the leaf directory (e.g. `/data/users/markamendoza/scratch/homeZmarkamendoza`), **but will not do so for any of the parent folders it created**.
This is normally not a problem, as long as `/data/users/markamendoza/scratch/` already exists. If it doesn't then these directories are created with the wrong ownership, which causes problems if we (for example) call `mkscratch` again, without sudo.
This diff remedies this by calling the same `set_file_owner` function (that's already called on the leaf directory) on all parents that were created by `mkscratch`
Reviewed By: chadaustin
Differential Revision: D27378153
fbshipit-source-id: 26c40f4ca478cacf9117093c7b70cbedd679cea6
Summary:
Preserving that state causing some bugs.
Also it leads to accumulating lots of .eden-backing-repos/fbsource/.hg/store/commitcloudstate.user files.
It is cleaner to remove it.
Reviewed By: markbt
Differential Revision: D27461467
fbshipit-source-id: 21eebe799afcb919539ae059fc8b707abc74971c
Summary:
We want to protect against races where we may have multiple writers with
different IdMap versions. We want IdMapVersions to be monotonous increasing.
The common race scenario is between tailers and seeder. We may have a tailer
that starts to updates while a seeder writes a version with a new idmap. The
tailer may try to write a version that has an old version of the idmap. We want
the tailer to fail in this case.
Reviewed By: krallin
Differential Revision: D27210035
fbshipit-source-id: 168c7c6f25622587071337df1172a12b799e0285
Summary:
This subcommand had the same help text as `content-fetch`. Update the
help text to describe what the command actually does.
Reviewed By: ahornby
Differential Revision: D27463645
fbshipit-source-id: ac4fba62f7836b55c94410889aafbf76a3cad74f
Summary:
Support the `--history` flag (and other flags from `hg cloud sl`) on `hg cloud
ssl`. The only real difference between the two is the template.
Fetching the phabricator status for large number of commits can take a while,
so show a progress spinner while we do it.
Reviewed By: liubov-dmitrieva
Differential Revision: D27460144
fbshipit-source-id: c80830b85fd0766ad9c56afa9afea0c2d21f9b36
Summary:
Replace the implementation of `hg cloud sl --history` with one that uses
streampager. This allows the user to scroll around the rendered smartlog while
also switching from one version to the next.
The old (broken) history is available by setting `commitcloud.useoldhistorytui` to true.
Reviewed By: quark-zju
Differential Revision: D27268251
fbshipit-source-id: 77501e4b7f3da9506cd5a9cabb9cba0388743723
Summary:
Add bindings for `pysptui`. This allows using `streampager` as a TUI, using
streampager's controlled file mode to manage the display.
Reviewed By: quark-zju
Differential Revision: D27268252
fbshipit-source-id: d191a09c44ca4ed013647feb81e6f031d553b2f2
Summary:
We weren't validating any hashes in walker, and so it was entirely possible to
have corrupt content in the repo. Let's start hash validation it by validating hg
filenodes
Reviewed By: ahornby
Differential Revision: D27459516
fbshipit-source-id: 495d59436773d76fcd2ed572e1e724b01012be93
Summary:
When going into a root unode we check if fastlog is derived or not. if it's not
derived we don't even try to check fastlog dirs/files, and that's correct.
However that means that if fastlog batch for a given commit is derived than
we could (and should!) check if all fastlog batches referenced from this
commit exist.
Previously we weren't doing that and this diff fixes it. Same applies for hg
filenode.
Reviewed By: ahornby
Differential Revision: D27401597
fbshipit-source-id: 71ad2744eee33208c44447163cf77bc95ffe98d0
Summary:
During an `hg update`, all the loaded inodes are unloaded and forgotten, except
the ones that had a positive refcount, these are forgotten when FUSE calls the
`forget` API. For NFS, this `forget` API doesn't exist but the protocol relies
on file handle to stay valid for some time. The duration for which a file
handle need to stay alive depends on several factor. The root one needs to
always be valid, the rest relies on the lookup cache that the client keeps.
This lookup cache is invalidated when the mtime for the parent directory
changes, but mtime changes are often not immediately detected, as attributes
are cached client side with the `acdirmin` and `acregmin` for directories and
files.
For now, this diff doesn't attempt to deal with files being out of date right
after an `hg update`, it merely accounts for the NFS client being able to pass
old file handles to EdenFS. The astude reader will have noticed that
InodeNumber are never reclaimed for now, for that, a time-based mechanism will
need to be added to forget InodeNumbers that have expired.
Reviewed By: kmancini
Differential Revision: D27405295
fbshipit-source-id: af4a4ce9e31bfcc335608da91f0247b50ab87b3f
Summary: In a later diff, I plan to make this `pub` in order to parse HTTP versions from the user's config.
Reviewed By: quark-zju
Differential Revision: D27449576
fbshipit-source-id: 28a60080393eff73399c65b9e808647b39603719
Summary:
Production confidence team wants a simple tool to capture current state of the repo and get a commit hash in commitcloud
ephemeralcommit achieves that by making a hidden commit and pushing it to commitcloud
Main purpose of this is to use with ephemeral fbpkg, this produces relatively low volume of commits (comparing to total commit count).
This currently does not add untracked files, but removes missing files
Usage:
```
# note that when invoking from automatic tools need to specify -q
$ hg debugephemeralcommit -q
<hash of the commit>
```
Reviewed By: quark-zju
Differential Revision: D27339320
fbshipit-source-id: 07a4ea8ff80b80ce620fb609096db97f46d383dc
Summary: WSAEACCES is included in Windows only. Only check it if the os is windows.
Reviewed By: quark-zju
Differential Revision: D27434606
fbshipit-source-id: 25eb8036363b42629fbd010f7637a404dccff236
Summary:
`isinstance(i, int)` is problematic on Python 2 Windows:
In [1]: isinstance(72057594037927936,long)
Out[1]: True
In [2]: isinstance(72057594037927936,int)
Out[2]: False
Fix it by replacing `isinstance(_, int)` with a pycmpat function.
Reviewed By: DurhamG
Differential Revision: D27436107
fbshipit-source-id: 9d9d9f9359546a91a564d948718078a9aa594420
Summary: Integration test and macOS tests did not run at diff time.
Reviewed By: DurhamG
Differential Revision: D27436108
fbshipit-source-id: ab94ec88bad8de42025f539023ab426002b9bed5
Summary: On macOS, the mount argument are encoded with XDR, let's add them first before using them.
Reviewed By: genevievehelsel
Differential Revision: D27306770
fbshipit-source-id: 727824f05d3874119858af60c263267adfb3e61e
Summary:
The returned value was never used by any callers, let's simply not return any
value.
Reviewed By: kmancini
Differential Revision: D27418015
fbshipit-source-id: 2a6f15eee01052cdfa9ae334c34e69f2f0a74407
Summary: Expand the IdConvert implementation so we can change it.
Reviewed By: sfilipco
Differential Revision: D27339280
fbshipit-source-id: eb55c63529c895502a25bb279bcba3c13737452a
Summary:
Add an overlay IdMap field for NameDag to store temporary remote IdMap results.
This diff just adds the field. It's not used yet.
Reviewed By: sfilipco
Differential Revision: D27339274
fbshipit-source-id: dbbde227f26de15d10c84f5d7c61ca8054577752
Summary:
In a future diff, AbstractNameDag wants an "overlay" IdMap to store temporary
remote IdMap results. The MemIdMap is suitable but has extra features (next
free id, version). Extract the core in-memory IdMap logic for the "overlay"
purpose.
Reviewed By: sfilipco
Differential Revision: D27339277
fbshipit-source-id: 4e73032b8bc6670264e3fa1dd5515ea3bc853d10
Summary:
The `Process::process` contains logic to resolve Id <-> Vertex using remote
service. The remote service is async so let's make `Process` async.
Reviewed By: sfilipco
Differential Revision: D27308798
fbshipit-source-id: 30c2c3eda124d542d0867d278ce56a7a174f33e0
Summary:
The config turns on fsync for all indexedlog writes. It can be useful in places
where we want reliability more than performance with unknown kernel issues.
Reviewed By: sfilipco
Differential Revision: D27347595
fbshipit-source-id: c0b31928684e8805a9e6441062f96b05ad311ea2
Summary:
Add a global flag, if turned on, ensure all atomic files, and indexes and
primary logs use fsync.
Also enhance fsync so it syncs the directory too.
Reviewed By: sfilipco
Differential Revision: D27347596
fbshipit-source-id: 831e27e494cc343a33ca675619c030ead8023210
Summary: test-lfs-placeholders.t fails on windows. The code isn't used on Windows so mark the test as no-windows.
Reviewed By: sfilipco
Differential Revision: D27433793
fbshipit-source-id: 4cbf70efae655ca318d776f6a2d6b79e83c78cbc
Summary:
Current we treat missing just as any other error, and that makes it hard to
have an alarm on blobs that are missing, because any transient error might make
this alarm go off.
Let's instead differentiate between a blob being missing and failed to fetch a
blob because of any other error (i.e. manifold is unavaialble). Turned out this
is not too difficult to do because a lot of the types implement Loadable trait,
which has Missing variant in its errors.
side-note:
It looks like we have 3 steps which treat missing blobs as not being an
error. These steps are:
1) fastlog_dir_step
2) fastlog_file_step
3) hg_filenode_step
I think this is wrong, and I plan to fix it in the next diff
Reviewed By: ahornby
Differential Revision: D27400280
fbshipit-source-id: e79fff25c41e4d03d77b72b410d6d2f0822c28fd
Summary:
Add a way to disable revnum resolution for non-automation (non-HGPLAIN)
usecases. Automation (ex. nuclide-core) still use revision numbers.
Some tests still use revnums. Put them in a list so this feature
does not break them.
Reviewed By: DurhamG
Differential Revision: D27144492
fbshipit-source-id: fba97fc90c7942e53914c29354938178b2637f44
Summary:
Manifest parent triggers an unimplemented tree history fetch path and is
generally prone to errors. See D9013996 (2b7e9e5e8b) and D9013995 (9e51fdef40).
Reviewed By: DurhamG
Differential Revision: D27411626
fbshipit-source-id: aee79f7928f0eb7fd39f68d12ec3ca33873f4e0b
Summary: Use Edenapi book request and response type in bookmarks edenapi endpoint. Serialize the response as cbor.
Reviewed By: kulshrax
Differential Revision: D27174122
fbshipit-source-id: 6bc7295c25bd355db4625da3c1f8c68349e7b0b7
Summary: Add edenapi types for the bookmarks endpoint. Now the endpoint can handle a request for a batch of bookmarks instead of a single bookmark. The request type still needs to be modified at some point to allow for bookmark prefix listing.
Reviewed By: kulshrax
Differential Revision: D27133284
fbshipit-source-id: c3960629cad76504e222f726a151eb3390850276
Summary: We have number of small repos which doesn't have much traffic and load so it can be combined in one single tailer job.
Reviewed By: StanislavGlebik
Differential Revision: D27361196
fbshipit-source-id: 3326a7445cf4f0f0cb27fe0f4545cf8ee3357ff2
Summary:
When EdenFS is mounted via FUSE, the changed directory and files are flushed so
the kernel is forced to re-read them to obtain new InodeNumber. When NFS is
used, EdenFS can't directly invalidate the client view of the FS, instead we
need to rely on when and how is this invalidated. On Linux, a file handle won't
be refreshed if its parent directory hasn't changed, which is done by looking
at the mtime of the directory by performing a GETATTR call. It is thus crucial
that the mtime is updated during an `hg update`.
Reviewed By: chadaustin
Differential Revision: D27403729
fbshipit-source-id: 1f7195c6c33e2a34c3bb73145f404d652302d828
Summary:
We can move the ifdef in the function itself. This makes the functions calling
these easier to read, and it's less likely for someone to forgot to add the
proper ifdef. If we also ever want to make these do something on Windows, it
will be easier to do too.
Reviewed By: chadaustin
Differential Revision: D27403730
fbshipit-source-id: c5b78ea7c7eb70eaf8d4974e5bec14296f91576f
Summary: This fixes various tests on Windows. Many of these fixes were ported from upstream.
Reviewed By: markbt
Differential Revision: D27174617
fbshipit-source-id: b9f36ad0714793f2b76db32c1d840284b744a841
Summary: This will cause a compile error if a new variant is added to `Value` - I have looked at the existing types, and none of the ones called out here would be used by MySQL for an integer field or an enum field.
Reviewed By: StanislavGlebik
Differential Revision: D27402174
fbshipit-source-id: e1fd8dd821f66094225f1ea4c38cd22626c8ab64
Summary:
The placeholder mode is meant to be used when talking to LFS server is not available in
given environment. It allows the user to work in degraded mode where
all of the LFS files are replaced with placeholders informing about their
original sizes and hashes.
Reviewed By: xavierd
Differential Revision: D27294603
fbshipit-source-id: 2bb8e2cb74ffccefcd90d618d6791ce5c45755d6
Summary:
This situation is not normally possible - every hg changeset should have a
corresponding bonsai. So let's return an error if that's not the case.
Reviewed By: farnz
Differential Revision: D27400281
fbshipit-source-id: 4b01b973eeef0e3336c187fb90dd2ab4853b5c02
Summary:
Non-list optional data can be present in some XDR description, let's special
case it so the intent is clear when declaring XDR datastructures.
Reviewed By: fanzeyi
Differential Revision: D27306768
fbshipit-source-id: 9d4d18bf8deff16f859c6d28a2579341dac8ee6f
Summary:
After receiving a network packet, it's possible that more than one fragment
were received as part of it. We thus need to service all of them before
returning.
This would typically be seen when running `rg` in the repository, which would
cause hangs due to some requests not being serviced as they would stay in the
iobuf queue until a new packet was received.
Reviewed By: kmancini
Differential Revision: D27194038
fbshipit-source-id: 3d81c797b5be7d0466d4acad7208f6a82593b4ca
Summary:
Computing the length of an iobuf chain can be expensive due to having to walk
its entirety. Thanksfully, IOBufQueue can cache the total length when data is
appended to it, which makes computing the length a constant operation.
Reviewed By: kmancini
Differential Revision: D27194037
fbshipit-source-id: af659c162ada61f2796bf407f419f5f15e918c02
Summary:
By moving the work to a background threadpool, we can more quickly go back to
servicing incoming NFS requests and thus allow more work to be done
concurrently. This would allow tools like ripgrep to being able to use multiple
cores to search in the code base.
Reviewed By: genevievehelsel
Differential Revision: D27194040
fbshipit-source-id: 7f1775ddaaa7eaf8776a06d05951cb936cd3fbb5
Summary:
D26744888 added use of Value::UInt to the MySQL client. This breaks the ChunkingMethod conversion, which depended on it being supplied Value::Int.
Add UInt support to the list - and keep the extra debugging data used to track this down
Differential Revision: D27396544
fbshipit-source-id: d9b8f1ef37f112d9a7d560e4b2a039480997ea0b
Summary:
Manual import from https://github.com/facebookexperimental/eden/pull/81 as the auto import mapped the .yml paths incorrectly
Updates eden/scm/Makefile to use python3 so we don't need to install multiple py versions
Adds hgext.convert.repo to setup3.py packages as mononoke tests showed it was missing
Updates github actions python versions
Reviewed By: quark-zju
Differential Revision: D27367568
fbshipit-source-id: 3817bdc1c48a8f7bfa8e29b5f7ec87d0eed579a9
Summary:
Like it says in the title. We don't use this anymore.
Context in D27268419
Reviewed By: markbt
Differential Revision: D27268635
fbshipit-source-id: 236adb5e68bc67612610d99f626344f4d592b5f9
Summary:
`create_mysql_connections_sharded` takes over a second to construct
4,000 shards (over 250 µs per shard). It's also blocking; run it in a blocking
thread so that other tasks are not waiting for it.
Reviewed By: StanislavGlebik
Differential Revision: D27360221
fbshipit-source-id: 79065bf4a8cd60dddbb5c1e8bf871872fd52f428
Summary: We don't use it, so let's remove it
Reviewed By: farnz
Differential Revision: D27359959
fbshipit-source-id: 42ce7da16fd0359bbceeab9d1f99712f45a80314
Summary: Can reduce number of allocations and copies by sharing the underlying thrift buffer
Reviewed By: markbt
Differential Revision: D27043232
fbshipit-source-id: a6e58c53035cb07f7b205df465a9ba2f7a78d52e
Summary: Can reduce number of allocations and copies by sharing the underlying thrift buffer
Reviewed By: krallin
Differential Revision: D27043231
fbshipit-source-id: 90731ac0a94d50ec28c9082f4e878c2ba24fcffd
Summary: Add a test target to the eden_scm getdeps manifest and underlying Makefile
Reviewed By: markbt
Differential Revision: D27336805
fbshipit-source-id: 07ec4be1ff03c6a384451ce138d88938dd4bf86e
Summary:
The `remotebookmarks` field in the local commit cloud state should always be one of:
* The empty set, if the previous sync was performed with remotebookmarks sync disabled; or
* The cloud workspace's remote bookmarks for that version.
Currently when processing remote bookmarks, we may store in the local state the
outcome of conflict resolution for the remote bookmarks. This is the wrong
thing to do, as it means we won't then upload those conflict resolutions as a
new cloud version, which means they may get lost and rolled back.
Change application of cloud remote bookmarks to store the cloud remote bookmarks
in the local state, even if we changed them through conflict resolution. This
means we will always upload the newly updated remote bookmarks to the server,
and things will stay more in sync.
Reviewed By: quark-zju
Differential Revision: D27291238
fbshipit-source-id: 8e6a0ab150da5907d32b8127aa0e6ccb17df4eea
Summary:
When connecting to a commit cloud workspace where there are no draft commits to
pull, no local bookmarks to sync, but the remote bookmarks in the local repo
are ahead of the ones in the commit cloud workspace, we fail to sync the remote
bookmarks to the server.
This results in the remote bookmark rewinding on the next sync.
Reviewed By: quark-zju
Differential Revision: D27291237
fbshipit-source-id: 8ba56542492fda26b9cecb6726ddd1b85ed5c180
Summary:
In the next diff I'd like to add support for syncing globalrev to our darkstorm
repos. Doing it the same way we do it for hgsql isn't going to work, because
darkstorm repos stores globalrevs the same way mononoke does it (i.e. per
commit entry in mysql) and not the way hgsql does (i.e. one row per repo).
In this diff I do a small refactoring that remembers which bonsai commits were pushed
in a bundle, so that in the next diff we can start writing them to darkstorm
db.
Reviewed By: krallin
Differential Revision: D27268778
fbshipit-source-id: bbb39de233719c8435d11d00980f6eaf5b755ba6
Summary:
Original commit changeset: 0708a4b0dc37
It seem to be the reason of sql timeouts on mononoke startup
Differential Revision: D27337030
fbshipit-source-id: 7b154c09397b0e297e18b186a6338ab801b1769d
Summary:
In production, all repos are instantiated roughly the same time so all reload
processes are started roughly the same time. Reload makes a bunch of requests
and could potentially cause load. Jitter spreads out the load of the reload.
Avoiding the load spike will make overall server behavior more predictable.
Reviewed By: krallin
Differential Revision: D27280117
fbshipit-source-id: 0727af2e7f231a5b6c948424022788a8e7071f82
Summary:
We would like to distribute the load of update process when many repositories
have Segmented Changelog enabled. Without the jitter all enabled repositories
start their update at roughly the same time. The jitter smooth out the load and
reduces variance in the update process.
Reviewed By: krallin
Differential Revision: D27280118
fbshipit-source-id: 41ad83b09700da1ef70c09dd5d284977e53a95a2
Summary:
`build_from_scratch` only called in `run_with_idmap_version` so we can inline
the code so that the seed process reads better.
This function used to be used as a shortcut towards getting a built dag but now
we prefer to fetch the dags from the store that the seeder writes to.
Reviewed By: krallin
Differential Revision: D27210036
fbshipit-source-id: 0b31ff1126a0f4904578da333cf6d34d69b2782c
Summary:
Removing the last callsite for SegmentedChangelogBuilder means that
the whole class goes away.
Reviewed By: krallin
Differential Revision: D27208339
fbshipit-source-id: 006aa91ed9656e4c33b082cbed82f9a497b0d267
Summary:
We are removing SegmentedChangelogBuilder.
Remove the last uses of Builder in the tests module.
Reviewed By: krallin
Differential Revision: D27208341
fbshipit-source-id: 00f1aaa2376ee5d68dbf7c1256b312cfe0b96d86
Summary:
Any functions that returns SegmentedChangelog is a valid argument for
reloading.
Reviewed By: krallin
Differential Revision: D27202520
fbshipit-source-id: fe903c6be4646c8ec98058d1a025829268c36619
Summary:
PeriodReloading is not fundamentally tied to the Manager. A future change will
update the load function.
Reviewed By: krallin
Differential Revision: D27202524
fbshipit-source-id: a0e4b08cb8605d071d5f30be8c3054f75321aa9c
Summary:
Let's look at these test from a higher perspective. Right now the tests use
internal APIs because not all components were ready when they were added. We
now have components for all the parts of the lifecycle so we can set up tests
in the same form that we would set up the production workflow.
This simplifies the API structure of the components since they can be catered
to one workflow.
Reviewed By: krallin
Differential Revision: D27202530
fbshipit-source-id: 6ec10a0b1ae49da13cfbe803e120a4e754b35fc7
Summary:
The broad goal is to get rid of SegmentedChangelogBuilder.
We will have a new constructor for Seeder, one that uses non segmented_changelog
dependencies as input.
Reviewed By: krallin
Differential Revision: D27202523
fbshipit-source-id: d420507502925d4440d5c3058efef0a4d2dbe895
Summary:
For tests that don't care about the bookmarks specifically, we want to use the
default bookmark name that we defined in BOOKMARK_NAME.
FWIW, it makes sense for this bookmark to be set by blobrepo even... at least
the fixtures. They set a bookmark and it makes sense for us to have a reference
to the bookmark that they set. Something to think about.
Reviewed By: krallin
Differential Revision: D27202522
fbshipit-source-id: 7615e4978dded491dd04ae44ce0b85134a252feb
Summary:
This gets rid of the odd builder for the Seeder.
We can get into design discussions with this one. What is struct and what is
function? For real structures that provide some behavior, I prefer to put
dependencies in owned data. Things that are part of the request go into
function parameters. In mononoke, RepositoryId is the common exception.
Anyway, IdMapVersion is part of the request for seeding. It is useful to have
that as a parameter when starting the seeder.
Reviewed By: krallin
Differential Revision: D27202528
fbshipit-source-id: a67b33493b20d2813fd0a144b9bb7f4510635ae8
Summary:
With the mix of external pager and progress suspension, the progress might
be enabled by accident:
# pager: disable forever
disable_progress(True)
# suspension
with progress.suspend()
...
# on __exit__, re-enables pager
Update the pager disabling logic be nested to avoid the potential issue.
Reviewed By: andll
Differential Revision: D27275016
fbshipit-source-id: 35ca7aef1890a981e6a1f0e5313b6a482ed43368
Summary:
Currently dropping progress bar panics if `__exit__` returns error
This happens, for example, when handling interrupts. Best course of actions just do not panic in this case
Reviewed By: sfilipco
Differential Revision: D27334897
fbshipit-source-id: c879fb14cfd4c16c0f9caede552129f8117defdc
Summary:
The progress total can be expensive to calculate for generatorset. Do not
calculate it. This will preserve laziness of certain sets (ex. generatorset
used by fastlog and average directory log).
Reviewed By: sfilipco
Differential Revision: D27327127
fbshipit-source-id: b0e3655e33b9e89ee2100941af18a769315f25bb
Summary:
Buffering the stream can provide suboptimal UX if the stream is slow.
Detect slow streams and avoid full buffering.
Reviewed By: sfilipco
Differential Revision: D27327128
fbshipit-source-id: a7b8037b7ba28fccc10661ffd15fd68f191d0048
Summary:
Remove use of dangerous_override from the repo client tests.
Previously this was used to override filestore config, so just use the existing
config override mechanism to set the filestore params this is generated from.
Reviewed By: ahornby
Differential Revision: D27169424
fbshipit-source-id: 7d17437f0e218d1cf19cc64d48e1efdd7012e927
Summary:
Remove use of dangerous_override in the pushrebase tests.
This was being used to ensure the bookmarks db in the test repo was shared with the mutable counters db. This is now directly possible by accessing the metadata database from the factory.
Reviewed By: ahornby
Differential Revision: D27169435
fbshipit-source-id: 1412231bdd9214bc869a3bfa7f63bf6c14db6836
Summary:
Remove uses of dangerous_override from derived data tests.
These were being used to override the derived data config or the lease object.
Extend the factory to allow customization of the config or the lease object,
and use that instead.
Reviewed By: StanislavGlebik
Differential Revision: D27169438
fbshipit-source-id: e8d0be248391d02bb054e19fdb9a90005db09c84
Summary:
Remove uses of dangerous_override in the commit rewriting tests.
Previously, the test was using dangerous_override to replace the bookmarks
attributes with ones that share a backing database with a `SqlMutableCounters`
instance.
With TestRepoFactory, all attributes share one metadata database. Obtain that
from the factory and use it to initialize the `SqlMutableCounters` instance.
Reviewed By: StanislavGlebik
Differential Revision: D27169429
fbshipit-source-id: 3c1b285db38a96deca7029d37e6692cb49356d31
Summary:
Remove dangerous overrides used in the tests in the bookmarks crate.
The test was using dangerous overrides to change the blobstore, which is now
supported directly by the test repo factory.
It was also using dangerous overrides to override the bookmark update log to
reset its database so it looks like the log is empty. Instead, clear out the
bookmark update log database as part of the test.
Reviewed By: StanislavGlebik
Differential Revision: D27169426
fbshipit-source-id: 64e1e89e31f62dcb585741ea728ebbe45f60fd38
Summary: This has been superseded by `test_repo_factory::TestRepoFactory`.
Reviewed By: StanislavGlebik
Differential Revision: D27169434
fbshipit-source-id: 97fcd400c5e3c6e8f86c9acfc5e979909e2eda31
Summary: Use the test factory for the remaining existing tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169443
fbshipit-source-id: 00d62d7794b66f5d3b053e8079f09f2532d757e7
Summary: Use the test factory for existing bookmarks and pushrebase tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169430
fbshipit-source-id: 65a7c87bc37cfa2b3b42873bc733cec177d8c1b0
Summary: Use the test factory for test fixtures that are used in many existing tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169432
fbshipit-source-id: bb3cbfa95b330cf6572d1009c507271a7b008dec
Summary: Use the test factory for existing lfs_server tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169427
fbshipit-source-id: 004867780ad6a41c3b17963006a7d3b0e5b82113
Summary: Use the test factory for existing commit_rewriting tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169442
fbshipit-source-id: df2447b2b6423d172e684d7e702752ad717a2a4b
Summary: Use the test factory for existing repo_client and repo_import tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169425
fbshipit-source-id: 2d0c34f129447232cec8faee42056d83613de179
Summary: Use the test factory for existing hooks and mapping tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169436
fbshipit-source-id: f814e462bb2244e117c83bd355042016f5bfde8e
Summary: Use the test factory for existing mononoke_api tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169444
fbshipit-source-id: d940bfa494dfe89fdb891d5248b73ba764f74faf
Summary: Use the test factory for existing derived data tests.
Reviewed By: farnz
Differential Revision: D27169433
fbshipit-source-id: b9b57a97f73aeb639162c359ab60c6fd92da14a8
Summary:
Create a factory that can be used to build repositories in tests.
The test repo factory will be kept in a separate crate to the production repo factory, so that it can depend on a smaller set of dependencies: just those needed for in-memory test repos. This should eventually help make compilation speeds faster for tests.
A notable difference between the test repos produced by this factory and the ones produced by `blobrepo_factory` is that the new repos share the in-memory metadata database. This is closer to what we use in production, and in a couple of places it is relied upon and existing tests must use `dangerous_override` to make it happen.
Reviewed By: ahornby
Differential Revision: D27169441
fbshipit-source-id: 82541a2ae71746f5e3b1a2a8a19c46bf29dd261c
Summary:
Convert `BlobRepo` to a `facet::container`. This will allow it to be built
from an appropriate facet factory.
This only changes the definition of the structure: we still use
`blobrepo_factory` to construct it. The main difference is in the types
of the attributes, which change from `Arc<dyn Trait>` to
`Arc<dyn Trait + Send + Sync + 'static`>, specified by the `ArcTrait` alias
generated by the `#[facet::facet]` macro.
Reviewed By: StanislavGlebik
Differential Revision: D27169437
fbshipit-source-id: 3496b6ee2f0d1e72a36c9e9eb9bd3d0bb7beba8b
Summary: To prepare for making `RepoBlobstore` a facet, convert it to a newtype wrapper.
Reviewed By: ahornby
Differential Revision: D27169439
fbshipit-source-id: ceefe307e962c03c3b89be660b5b6c18d79acf3e
Summary: We have support for backup-repo-id, but tw blobimport doesn't have id and have source repo name to use. Let's add support similar to other repo-id/source-repo-id etc.
Reviewed By: StanislavGlebik
Differential Revision: D27325583
fbshipit-source-id: 44b5ec7f99005355b8eaa4c066cb7168ec858049
Summary:
I'm trying to track down an issue in SQLBlob construction, where it takes over 3 wall-clock seconds to construct.
This is indicative of a missing `blocking` annotation somewhere; to make it easier to add the correct annotation, switch construction to modern futures and `async`
Reviewed By: krallin
Differential Revision: D27275801
fbshipit-source-id: 2b516b4eca7143e4be17c50c6542a9da601d6ac6
Summary:
The pack key needs to be removed using unlink() after target keys are pointed to it so that old unused packs can be GCed. e.g.
packer run 1: keys A and B are packed to P1
packer run 2: keys A and C are packed to P2
packer run 3: keys B and D are packed to P3
If we don't unlink P1 during run1 then GC can't collect the old pack after run 3 as the key will keep it live
Reviewed By: farnz
Differential Revision: D27188108
fbshipit-source-id: bbe22a011abbf85370a8c548413401ffb54b16b6
Summary:
Refactor `scmstore::types` into separate `file` and `tree` modules and introduce a new `StoreTree` type to represent trees in the scmstore API.
Introduce a minimal `StoreTree` type. Importantly, for now this type does not provide any methods for deserializing the tree manifest and inspecting it's contents. This functionality won't be too hard to implement, though - it'll require some revisions to the `manifest-tree` crate and / or moving the `StoreTree` and `StoreFile` types to `revisionstore_types`.
Reviewed By: kulshrax
Differential Revision: D27310878
fbshipit-source-id: 712330fba87f33c49587fa895efea3601ce377af
Summary:
The revset is not optimized for the (to be migrated to segments) revlog backend.
Optimize it.
Reviewed By: jteosw
Differential Revision: D27317708
fbshipit-source-id: cec9d6aad0f6c30c69a931898f8e1cc7c904b3f8
Summary:
We've seen occasional index timeouts on inserts in MySQL. This is very
reminiscent of D19158550 (fef360b284).
aida points out this seems to have started to happen (occasionally) recently.
That would make sense: we used to insert one-by-one so we wouldn't have ordering
issues (because bug), but we also recently started inserting in batches again (because bugfix).
There's little reason to expect we couldn't run into the same bug there as well.
So let's give filenodes & copydata the same sorting treatment we give paths.
Worst case, this does nothing. Best case, it fixes the issue.
Reviewed By: StanislavGlebik
Differential Revision: D27301588
fbshipit-source-id: 2b24ddd68e1a1c4e31fe33e03efcef47dad3657d
Summary: Previously, accessing both repo.filelog and repo.manifestlog in a test debug command, under buck builds only, would cause a "memcache singleton leaked" error to be printed after the command completes (see https://www.internalfb.com/intern/qa/87181/runtime-warnings-about-memcacheconnectionmanagerim ). There are still some unanswered questions about this issue, as noted in the QA post, but this fixes the main issue at hand.
Reviewed By: sfilipco
Differential Revision: D27297498
fbshipit-source-id: e19665333bae9f91e1c3c6db370962a3aea2727d
Summary:
Rename "newstore" to "scmstore"
The three main options I'm considering are "edenstore", "scmstore", and "hgstore". They all describe the project sufficiently well, in my opinion. "edenstore" might be a stretch, as I have no reals plans for Mononoke to use it, while "hgstore" might be a bit limited, as integration with EdenFS is a core goal, and it'll be used by EdenFS to fetch remote data via EdenApi, not just Mercurial's local storage. I feel like "scmstore" rolls off the tongue the easiest, particularly vs. "hgstore" (both "H.G. Store" and "Mercurial Store"), and is a bit easier to type than "edenstore" (which has "ede" all with the same finger). I don't have a strong opinion on this matter, so If you'd like a different name I'm open to suggestions.
Speak now or forever hold your peace.
Reviewed By: andll
Differential Revision: D27180332
fbshipit-source-id: 19e6972ea0f6527e671792845dcfd339cf1ab767
Summary:
Introduce a new `StoreFile` type for the `revisionstore` crate. This is an enum of `File`, `LfsPointer`, and `RedactedFile`, which represent the different cases the `Entry` type's `content` might actually represent. The `File` variant also handles stripping the copy header, if present, and stores both the `copied_from`, raw unstripped `content`, and stripped `content`, though only the latter can be accessed through the public API right now. Ideally, we'll move copy information into the history API and never actually need to make it public here.
Conversions are provided from `Entry` and EdenApi's `FileEntry` type (which also supports redaction and LFS pointers, but as errors instead of first-class values).
Modify the output type used in the Python bindings to use this new `StoreFile` type.
Currently, the `LfsPointer` variant is unused. This will be used later when we add first-class LFS support.
Reviewed By: kulshrax
Differential Revision: D26862710
fbshipit-source-id: 8326921f3ee43bf2e253847d5735c61f5a50bfa6
Summary: Adds `ExtStoredPolicy` support to the EdenApi and IndexedLog ReadStore implementations. This flag controls how LfsPointers found during a file query are handled (either returned as a result, or treated as "not found").
Reviewed By: kulshrax
Differential Revision: D27171814
fbshipit-source-id: 14dda47f32184c3ee703fbc77106885ca4d3ea27
Summary:
Add an unlink operation to BlobstoreWithLink for use from packblob in the next diff.
The reason an unlink operation is needed is so that pack keys don't keep old packed data live longer than it should be. e.g.
Before the packer runs, keys A, B, C, D each point to content A, B, C, D
packer run 1: keys A and B are packed to P1. Keys A, B, P1 now all point to pack P1
GC run here can remove content A and B
packer run 2: keys A and C are packed to P2. Keys A, C, P2 now all point to P2.
GC run here can remove content C
packer run 3: keys B and D are packed to P3. Keys B, D, P3 now all point to P3.
GC run here can remove content B and D.
At this point, keys A, C point to pack P2, keys B, D point to pack P3.
If we don't unlink pack P1 during packer run 1 then GC can't collect the old pack after run 3 as the key P1 will keep it live; if we unlink key P1 after pointing keys A and B to point at P1, then when keys A and B stop pointing to pack P1, there will be no further references.
The unlink method is on the BlobstoreWithLink trait which most Mononoke code does not reference and therefore can't easily accidentally call.
Its modelled on posix unlink(2), it will error if the key being unlinked does not exist.
Reviewed By: farnz
Differential Revision: D27188109
fbshipit-source-id: 628698dd5894895e2aff5cf7db979d32e1bc2b3e
Summary: For batch deriviation of filenodes we can first dervie non-root filenodes and write them into db and after that write all root-filenodes for that batch of commits.
Reviewed By: StanislavGlebik
Differential Revision: D27044302
fbshipit-source-id: 2a2eb8e336c787c4210192500b4d4a2a0663a29c
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
