Summary:
The name `rev` is used for revision numbers. It was used as a hex node in the
code. Let's make it use binary `node` explicitly to avoid confusion.
Reviewed By: muirdm
Differential Revision: D45898261
fbshipit-source-id: 7f6fc04e1e5a6aaef46dd31c3aca9d494cbf488a
Summary:
Make fastlog uses a regular generator and just one generatorset, instead of a
background thread + queue + nested generatorsets. The "combinator" that takes
threads becomes unnecessary after D45898263 and is removed.
This avoids the SIGABRT issue which could be reproduced relatively easily like:
export CHGDISABLE=1
# likely SIGABRT in the first 10 runs
for i in `seq 100`; do hg log -r 'max(follow(fbcode/buck2,master))' --pager=off || { echo FAIL!; return 1 }; done
Example SIGABRT backtraces in gdb:
(gdb) r log -r 'max(follow(fbcode/buck2,master))' --pager=off --verbose --debug --config fastlog.debug=remote
Starting program: hg log -r 'max(follow(fbcode/buck2,master))' --pager=off --verbose --debug --config fastlog.debug=remote
[Thread debugging using libthread_db enabled]
...
thread '<unnamed>' panicked at 'panic in a function that cannot unwind', library/core/src/panicking.rs:123:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread caused non-unwinding panic. aborting.
Thread 14 "hg" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffc7a00700 (LWP 785327)]
0x00007ffff6a4eacf in raise () from /lib64/libc.so.6
(gdb) bt
0 0x00007ffff6a4eacf in raise () from /lib64/libc.so.6
1 0x00007ffff6a21ea5 in abort () from /lib64/libc.so.6
2 0x0000555557fb9677 in std::sys::unix::abort_internal () at library/std/src/sys/unix/mod.rs:350
3 0x0000555557fb1eaa in std::panicking::rust_panic_with_hook () at library/std/src/panicking.rs:711
4 0x0000555557fb1b42 in std::panicking::begin_panic_handler::{{closure}} () at library/std/src/panicking.rs:581
5 0x0000555557fb0226 in std::sys_common::backtrace::__rust_end_short_backtrace () at library/std/src/sys_common/backtrace.rs:150
6 0x0000555557fb1892 in rust_begin_unwind () at library/std/src/panicking.rs:579
7 0x0000555557fdb223 in core::panicking::panic_nounwind_fmt () at library/core/src/panicking.rs:93
8 0x0000555557fdb2d1 in core::panicking::panic_nounwind () at library/core/src/panicking.rs:123
9 0x0000555557fdb3f3 in core::panicking::panic_cannot_unwind () at library/core/src/panicking.rs:171
10 0x00005555563ae8c4 in cpython::conversion::ToPyObject::with_borrowed_ptr (self=0x7fffc79ff158, f=..., py=...) at ../../../third-party/rust/vendor/cpython-0.7.1/src/conversion.rs:55
11 cpython::objectprotocol::ObjectProtocol::call (py=..., args=..., kwargs=..., self=<optimized out>) at ../../../third-party/rust/vendor/cpython-0.7.1/src/objectprotocol.rs:214
12 pythreading::RGeneratorIter::next_internal (self=<optimized out>, py=..., iternext=<optimized out>) at src/lib.rs:405
13 0x00005555563b2264 in pythreading::RGeneratorIter::__next__ (py=...) at src/lib.rs:389
14 0x00005555563b3714 in pythreading::RGeneratorIter::create_instance::TYPE_OBJECT::wrap_unary::{{closure}} (py=...)
at ../../../third-party/rust/vendor/cpython-0.7.1/src/py_class/slots.rs:321
15 cpython::function::handle_callback::{{closure}} () at ../../../third-party/rust/vendor/cpython-0.7.1/src/function.rs:220
16 std::panicking::try::do_call (data=<optimized out>) at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:487
17 std::panicking::try (f=...) at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:451
18 0x00005555563b2609 in std::panic::catch_unwind (f=...) at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panic.rs:140
19 cpython::function::handle_callback (_location=..., f=..., _c=...) at ../../../third-party/rust/vendor/cpython-0.7.1/src/function.rs:218
20 pythreading::RGeneratorIter::create_instance::TYPE_OBJECT::wrap_unary (slf=0x7fffefa613d0) at ../../../third-party/rust/vendor/cpython-0.7.1/src/py_class/slots.rs:318
21 0x00007ffff65e1518 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
22 0x00007ffff659f02f in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
23 0x00007ffff65e11dd in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
24 0x00007ffff659f02f in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
25 0x00007ffff65e11dd in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
26 0x00007ffff659f02f in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
27 0x00007ffff659fb24 in method_vectorcall () from /lib64/libpython3.8.so.1.0
28 0x00007ffff659798f in PyObject_Call () from /lib64/libpython3.8.so.1.0
29 0x00007ffff662ec5e in t_bootstrap () from /lib64/libpython3.8.so.1.0
30 0x00007ffff65ee5a4 in pythread_wrapper () from /lib64/libpython3.8.so.1.0
31 0x00007ffff5c081ca in start_thread () from /lib64/libpthread.so.0
32 0x00007ffff6a39e73 in clone () from /lib64/libc.so.6
(gdb) info threads
Id Target Id Frame
1 Thread 0x7ffff7fe6d80 (LWP 784691) "hg" 0x00007ffff650f274 in visit_decref () from /lib64/libpython3.8.so.1.0
2 Thread 0x7ffff4a00700 (LWP 785063) "ctrl-c" 0x00007ffff6b1f9b4 in read () from /lib64/libc.so.6
3 Thread 0x7ffff4600700 (LWP 785066) "tokio-runtime-w" 0x00007ffff6b2ee87 in epoll_wait () from /lib64/libc.so.6
....
10 Thread 0x7fffee600700 (LWP 785073) "tokio-runtime-w" 0x00007ffff6a399bd in syscall () from /lib64/libc.so.6
11 Thread 0x7fffee200700 (LWP 785074) "rust-progress" 0x00007ffff6af9868 in nanosleep () from /lib64/libc.so.6
13 Thread 0x7fffd4a00700 (LWP 785135) "hg" 0x00007ffff6b2667f in select () from /lib64/libc.so.6
* 14 Thread 0x7fffc7a00700 (LWP 785327) "hg" (Exiting) 0x00007ffff6a4eacf in raise () from /lib64/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff7fe6d80 (LWP 784691))]
0 0x00007ffff650f274 in visit_decref () from /lib64/libpython3.8.so.1.0
(gdb) bt
0 0x00007ffff650f274 in visit_decref () from /lib64/libpython3.8.so.1.0
1 0x00007ffff650bc0f in func_traverse () from /lib64/libpython3.8.so.1.0
2 0x00007ffff6528e94 in collect.constprop () from /lib64/libpython3.8.so.1.0
3 0x00007ffff664f192 in _PyGC_CollectNoFail () from /lib64/libpython3.8.so.1.0
4 0x00007ffff664f43a in PyImport_Cleanup () from /lib64/libpython3.8.so.1.0
5 0x00007ffff66519ff in Py_FinalizeEx () from /lib64/libpython3.8.so.1.0
6 0x000055555627f9d5 in hgcommands::python::py_finalize () at src/python.rs:136
7 <hgcommands::hgpython::HgPython as core::ops::drop::Drop>::drop::{{closure}} () at src/hgpython.rs:237
8 tracing::span::Span::in_scope (self=0x7fffffffafc0, f=...) at ../../../third-party/rust/vendor/tracing-0.1.37/src/span.rs:1102
9 0x000055555615651a in <hgcommands::hgpython::HgPython as core::ops::drop::Drop>::drop (self=<optimized out>) at src/hgpython.rs:237
10 0x0000555556130936 in core::ptr::drop_in_place<hgcommands::hgpython::HgPython> () at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/ptr/mod.rs:490
11 hgcommands::run::dispatch_command (io=0x7fffffffcde0, dispatcher=..., cwd=..., in_scope=..., start_time=...) at src/run.rs:240
12 hgcommands::run::run_command::{{closure}} () at src/run.rs:148
13 0x000055555612dbbd in hgcommands::run::run_command (args=..., io=0x7fffffffcde0) at src/run.rs:133
14 0x00005555561242da in hgmain::main () at src/main.rs:114
Note it involves threads, GC, rust-cpython, and `Py_Finalize`.
See also https://github.com/dgrunwald/rust-cpython/pull/244.
Reviewed By: muirdm
Differential Revision: D45898265
fbshipit-source-id: ce94763c7249f83237472f1fc4b4abad1011230c
Summary:
It does not seem like the code path would ever need to handle len(paths) > 1
cases. So let's simplify the code accordingly.
For example, when you run `hg log PATH1 PATH2`, the underlying revset is
`follow(PATH1) or follow(PATH2)`. Each follow revset only gets 1 path.
I guess the multi-path support was needed before migrating to the follow revset
in D14764074.
Reviewed By: muirdm
Differential Revision: D45898264
fbshipit-source-id: 05d1aed8d08741fe555e7eda4e4b3a2fd797833e
Summary:
The `RGenerator` and `RGeneratorIter` types keep Python objects internally.
Let's be a good citizen and tell CPython the object graph so CPython can
properly GC related objects.
Reviewed By: muirdm
Differential Revision: D45898262
fbshipit-source-id: da2b04ec6f7928c486f3f691885bf2ed8cb06055
Summary: Add a function to query the (optional) ObjectId from VirtualInode.
Reviewed By: kmancini
Differential Revision: D45673165
fbshipit-source-id: 72dadaf4cbd5120c75548734aca2bb8ae715b8e2
Summary:
Add a function to query the (optional) ObjectId from an arbitrary
inode.
Reviewed By: kmancini
Differential Revision: D45673053
fbshipit-source-id: 23630638ba94bbd1925a39193fa125df04abebad
Summary:
I noticed some situations where we don't hold the root inode for the
duration of an asynchronous operation. To make that easier, I
introduced an EdenMountHandle.
Port EdenServer::getMount() to return EdenMountHandle, which
simplifies some Thrift service handlers.
Reviewed By: kmancini
Differential Revision: D45891887
fbshipit-source-id: 99dfaa4aaf609c8b237cceeaa15b91bd26981d02
Summary:
Asynchronous operations that must ensure an EdenMount (and its root)
are usable for the duration must hold a reference to the root's
InodePtr, even if they only need the EdenMount.
To make this relationship less implicit, introduce an EdenMountHandle
which is effectively the same as the previous
`pair<EdenMountPtr, TreeInodePtr>`, but more clearly documents the intent
is to preserve the usability of EdenMount and its root inode.
This also allows us to decouple that relationship in the future,
should we want to.
For now, only use EdenMountHandle in getMountPoints. Next, I'll switch
`getMount` to use it.
Reviewed By: kmancini
Differential Revision: D45830176
fbshipit-source-id: 832dd96b9a6fe04fabbb5af0ecbed0173f9d2a2b
Summary: This diff implements the `create_git_tag` endpoint introduced in the last diff. The primary logic is implemeted as part of the `mononoke_api::create_annotated_tag` method so no new logic is introduced here. The `create_annotated_tag` does not yet store the mapping of the created changeset object with the tag name
Reviewed By: mitrandir77
Differential Revision: D45654021
fbshipit-source-id: 3ebd8d7b67a59935f646f5ba4287e14a096f82d1
Summary: The `create_git_tag` endpoint introduced in this diff is responsible for creating a changeset that represents the metadata of a git tag and create a DB-level mapping between the changeset-representing-the-tag and the changeset that the tag bookmark points to. This diff just introduces the skeleton, the ne xt diff in the stack will implement the core logic
Reviewed By: mitrandir77
Differential Revision: D45643823
fbshipit-source-id: 3b25c8a9382a364667c262961ed8506344988d49
Summary: This test was failing on Windows due to `working_copy.remove` not failing silently anymore. Given that the exact thing that it is testing doesn't work, it's better to just remove it.
Reviewed By: quark-zju
Differential Revision: D45833748
fbshipit-source-id: 952ca8a49fcdde8b5345d27eab20cedc3b979d7a
Summary: There are some cases where the Curl crate might not support HTTP V2, but will still try to use V2 regardless. This diff makes it fall back to Any (whatever the Curl crate considers best and available) instead.
Reviewed By: muirdm
Differential Revision: D45778450
fbshipit-source-id: 7811d1f30f75fab09f7bb0b478f44c730bbfe12b
Summary: Sometimes the log size changes a bit, which was making this test flaky on Linux and fail in most cases on Windows. This diff fixes that issue.
Reviewed By: muirdm
Differential Revision: D45762283
fbshipit-source-id: d16aedc2e0bfd858f1c5d3060339ae6e106fcb5b
Summary:
By default, `debugexportstack -r wdir()` exports tracked file changes. In the
ISL use-case, we'd like to add a "chunk selector" for files in the working copy.
The "chunk selector" would like to know the file contents regardless of whether
files are in `?`, `M`, `A`, `!`, `R` states.
This diff adds a `--assume-tracked` flag to support this use-case. Basically,
this flag can be used to make files in `!` or `?` states exported as if they
were `R` and `A` states.
Reviewed By: muirdm
Differential Revision: D45787979
fbshipit-source-id: 812a389ffe6d01b8a4200754bdc38c49b775c67e
Summary:
I noticed that `workingfilectx` handles renames properly. There is no need to
use `wctx`.
Reviewed By: muirdm
Differential Revision: D45787980
fbshipit-source-id: b46364305bd551b462445a8719fde1d77b694581
Summary:
The EdenMount::getObjectStore gave the assumption to callers that the
ObjectStore lived longer than the EdenMount. Since this assumption is implicit
and not explicit, there is no way to guarantee it by just reading the code in a
local fashion. It forces the reader/auditor to understand the code in its
entirety. Unfortunately, EdenFS contains more of these than can be kept in
memory which leads to bugs and use-after-free.
This can be avoided by breaking the implicit lifetime assumption and simply
returning the shared_ptr. If the caller needs to keep the ObjectStore around,
they can do so explicitely which makes reading the code easier.
Reviewed By: chadaustin
Differential Revision: D45831168
fbshipit-source-id: 8bdb8e708308a8a220e3130d2fc51854c8556cb1
Summary:
This will be used in a later diff outside of EdenMount.cpp, thus let's move it
to its own file.
Reviewed By: chadaustin
Differential Revision: D45831170
fbshipit-source-id: 1cc55e1ae580ee7f02e3d9c6a717da4751c955e6
Summary:
When a BlobMetadata fails to fetch, the fetch promise is set to a `nullptr` to
force the higher level BackinStore to fetch the full blob. Unfortunately,
`HgImportRequestQueue::markImportAsFinished` attempts to deep copy that
`nullptr` leading to a crash.
Reviewed By: chadaustin
Differential Revision: D45873681
fbshipit-source-id: 3f5b777a3671cf53dd9d605dca2e119eabe3e625
Summary: Here we implement the logic needed to bump QPS counters so we can shift traffic cross regionally with cslb
Reviewed By: mzr
Differential Revision: D45731719
fbshipit-source-id: d5df034d40588d47f19e121b21d606637dcdc41f
Summary: Add fallback logic for MetadataRenameFinder in case users forget to record copy information in the header
Reviewed By: muirdm
Differential Revision: D45865166
fbshipit-source-id: 991a71718b9e149fa1cd23fe33dd254f22e2f366
Summary: Those functions will be used for fallback logic in next diff
Reviewed By: muirdm
Differential Revision: D45865165
fbshipit-source-id: 73a4df33ff1dc407c53bf98fc95c10e1bff2501c
Summary:
RenameFinderInner is like a base struct for MetadataRenameFinder and ContentSimilarityRenameFinder.
It is introduced for code reuse between those two file based rename finders.
Reviewed By: muirdm
Differential Revision: D45865167
fbshipit-source-id: 66302c7dcad87fcd36e19ad2649d7fc4d42098b6
Summary: We already have a ContentSimilarityRenameFinder, MetadataRenameFinder sounds more appropriate than the original SaplingRenameFinder, since both rename finders now describes what information they are using for finding renames.
Reviewed By: muirdm
Differential Revision: D45827713
fbshipit-source-id: 9d9e5576cec676afbdd661333591085bef70d996
Summary: We have timeout in seconds (not a limit of number of tries), and it's usually around a minute. Doesn't make a lot of sense to sleep with 0.1. It also spams the logs a lot.
Reviewed By: YousefSalama
Differential Revision: D45810132
fbshipit-source-id: b638d086526da2450251fe412acead20c0bf5579
Summary:
The user doesn't really care if a runlog update failed. This is expected to happen sometimes on Windows since the update will fail if the file is underlying runlog file is open (by another process).
Also, add a couple retries to the final runlog update on Windows to increase the chance we close out the entry.
Reviewed By: sggutier
Differential Revision: D45831199
fbshipit-source-id: 664e14695fdf3a7363dafc7523cef78c9fafd84c
Summary:
I noticed this special path in `eden rm` misspelled "still". I also
cleaned up the grammar. Sentence fragments aren't capitalized and, to
make the path clearer, I removed the trailing period and split the
message across multiple lines.
(Separately, I think this is special path is not hit as often as was
intended.)
Reviewed By: xavierd
Differential Revision: D45828487
fbshipit-source-id: 183ccde081cb022abbc9e972f00bae8e5b9f33c0
Summary: There are a few categories that get collected into the "hg_errors" table. Make the fields more consistent so they always include exception_type, exception_msg and a new field "source" to make it more obvious where the error was emitted.
Reviewed By: grakkpl
Differential Revision: D45817135
fbshipit-source-id: a568e4c68fc51806fc5016b7a3e7e0a51ed9047d
Summary:
The pattern of grepping command output is very popular in our integration
tests keeping out tests shorter but has one big problem: when the underlying
command fails (oftentime with big stacktrace) the test output is empty.
Finding out actual failure culprit requires modifying the test to remove greps
wasting time. The function introduced in this diff attempts to solve that
problem.
Differential Revision: D44229554
fbshipit-source-id: eb173ff6177e811de72347e44beeb0eb3e2e7e92
Summary:
No need for this anymore.
Removing in configerator here: D45394588
Reviewed By: mitrandir77
Differential Revision: D45769295
fbshipit-source-id: c254cac81f931a02f075cf6b37dbf82257afd7eb
Summary:
We don't have a kext on macOS anymore, so we can remove the
configuration setting that enables it.
If we ever build our own kernel module on Linux, we can bring this back.
Reviewed By: kmancini
Differential Revision: D45260142
fbshipit-source-id: 6f99bdfb3fce236b7a17a665818327f00b5884c0
Summary:
ServerState.h is a widely-included header, as it's the dependency
injection seam used by EdenFS's inodes layer.
It included several unnecessary headers. I noticed this by seeing too
many files recompile when changing PrivHelper.h.
Replace the includes with forward-declarations.
Reviewed By: kmancini
Differential Revision: D45260059
fbshipit-source-id: 31a73030c9eb592107e8786be026e32eea80ceb4
Summary:
EdenMount used to track the set of bind mounts for a checkout. That's
long been removed, and bind mounts are now managed by the CLI (and
rectified against the mount table).
Move these vestigial EdenMount methods into EdenServiceHandler,
simplifying lifetimes and moving us closer to breaking EdenMount's
dependency on PrivHelper.
Reviewed By: kmancini
Differential Revision: D45259970
fbshipit-source-id: e9f1bb1c44fb74e49ece08416fbe1913dd1a6ea1
Summary:
We use SpawnedProcess on all platforms to start the privhelper, so we no longer
need to include sys/wait.h. This removes a conditional #include.
Reviewed By: genevievehelsel
Differential Revision: D45259880
fbshipit-source-id: dd54c60c13e34fc819eec4c4143e3c7da37329b0
Summary:
We use privhelper for NFS too, so move fs/fuse/privhelper into
fs/privhelper.
Reviewed By: kmancini
Differential Revision: D45243273
fbshipit-source-id: cfc93b324768809c87075b0085905e873c7ea88d
Summary:
I was hacking on EdenServer.h in an inner loop, and noticed my laptop
was spending time recompiling PeriodicTask every time. PeriodicTask
only needs an EventBase, so break that dependency.
Reviewed By: genevievehelsel
Differential Revision: D45788655
fbshipit-source-id: 9f61601906428dcdf4bff515c623f4b61a763d15
Summary:
all responses in edenapi are zstd compressed on _all_ platforms.
this is enabled here:
https://www.internalfb.com/code/configerator/[645398df35269eed7047d9d1c39015c39c038d26]/source/scm/hg/hgclientconf/hgclient.cinc?lines=NaN-NaN%2C1031%2C1033%2C1035
This isn't really needed for many many of our responses since they are anyway
very small. On the other hand, it is probably not a big deal but for commit
graph endpoint I have implemented a special progress bar that updates much nicer if we disable
such compression. So, I would like to disable it explicitly.
Reviewed By: muirdm
Differential Revision: D45691469
fbshipit-source-id: 67c3debe473638dfffc96084b7ad438870a26951
Summary:
It's unfortunate that changing NFS code causes most of the inode layer
to recompile. One benefit of FsChannel is we can begin to decouple
EdenMount from FUSE, NFS, and PrjFS. Remove the Nfsd3.h include from
EdenMount.h.
Reviewed By: kmancini
Differential Revision: D45167461
fbshipit-source-id: 41650390cbd053233dfe3baf051c592bc203de56
Summary:
waitForPendingWrites is general to any userspace filesystem with
asynchronous write notifications. This could even include FUSE in
writeback caching mode. To remove another ifdef, move
waitForPendingWrites into FsChannel.
Reviewed By: kmancini
Differential Revision: D45167206
fbshipit-source-id: 4daec70845864d90b94739fe2011e8ed90a6a200
Summary:
Even though PrjfsChannel has no definition on unix, we can still offer
the function. This removes an ifdef. Any attempt to PrjfsChannel
remains limited to Windows.
Reviewed By: kmancini
Differential Revision: D45167158
fbshipit-source-id: b2769ba003c3decf964c9cea06b8b4238ed5887c
Summary:
There's no need setting a test FsChannel needs to be prjfs-specific,
so remove the ifdef and introduce setTestFsChannel.
In the future, it would be nice if the FsChannel was specified at
construction so EdenMount can just be given an appropriate one during
tests.
Reviewed By: kmancini
Differential Revision: D45167055
fbshipit-source-id: 4274e827a8cf0578b88b63214558236aceb12604
Summary:
The only remaining use of EdenMount::getMountProtocol() is for
stringifying into the FinishedMount StructuredLog event. We can use
FsChannel::getName() instead. The only difference is that
FsChannel::getName() is lowercase, but I did not see anything using
the logged value.
Reviewed By: kmancini
Differential Revision: D45159419
fbshipit-source-id: 08b6a21d910bdf9c27f91c3e021c7ca8c902d478
Summary:
Also includes patch that jsgf had included, so I'm removing the patch here
UPDATE: Bumping to 0.8.4 - Includes div-by-0 potential bug in factory stats
Reviewed By: Imxset21
Differential Revision: D45779506
fbshipit-source-id: f7cd2946ae6e2c81b0a6c708165c19b218645b4c
