Summary: publish per-node-type progrss stats so we can correlate storage access/load to type of node traversed
Reviewed By: farnz
Differential Revision: D20181064
fbshipit-source-id: c741b526c50e86a3eee105fab57fd7bc3ecc063b
Summary: Add tests for existing default block casefolding_check behaviour, plus test demonstrating problem with casefolding_check=false
Reviewed By: farnz
Differential Revision: D20192412
fbshipit-source-id: 1aea0fc5581e0c44388a4224ca693698731d3cd5
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.
rs changes performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs,
rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
intersect
rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
)" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```
Reviewed By: jsgf
Differential Revision: D20168958
fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0
Summary: Recently there are increased reports on EdenFS's backing repo stuck in interrupted transaction state, and the user has to manually run `hg recover` in their backing repo to fix the problem. This diff teaches `eden doctor` to automatically run that command for the users.
Reviewed By: simpkins
Differential Revision: D20109567
fbshipit-source-id: a7427834e98425be388741c7f214b9d7354ac44e
Summary:
Enable pyre-strict type checking for `process_finder.py`, and split it into
its own library.
Reviewed By: genevievehelsel
Differential Revision: D20178483
fbshipit-source-id: e6c62ca5d84c7b7e599ae00fb51df6f7e4c55a65
Summary:
A couple places in the CLI code (mostly used by `eden doctor`) were checking
`sys.platform` to tell if we were on Linux. Unfortunately these checks both
expected the value `linux2`. However, since Python 3.3 `sys.platform` is just
`linux` on Linux, and not `linux2`. This meant we were always hitting the
non-Linux code paths and skipping these checks.
This updates the code to check `platform.system()`. Based on the
documentation it sounds like this is intended to give a bit more consistent
behavior across different platforms and OS versions.
Reviewed By: genevievehelsel
Differential Revision: D20178488
fbshipit-source-id: c908d5133a9c41e6a239a8893742d03f6c08527c
Summary:
Call `folly::setThreadName()` in the privhelper process when it starts. This
changes the command name reported in `/proc/PID/comm` and in `ps`
The process name is limited to 15 bytes, so this shows up as `edenfs_privhelp`
Reviewed By: fanzeyi
Differential Revision: D20199409
fbshipit-source-id: a5349bfab9230174aaa99c87f0db73fe31659186
Summary: One small step towards typing
Reviewed By: thatch
Differential Revision: D20090620
fbshipit-source-id: 811bb54159ab91e5560d115c20373eaf6542b2f9
Summary:
Small cleanup that removes a bunch of duplicate code.
That should make it easier to add other types of derived data to the warmer
Reviewed By: krallin
Differential Revision: D20193169
fbshipit-source-id: 437fe7981d8a71164dc9edfcc423e8c41cbe0967
Summary:
Add a a new `hg` module to the `mononoke_api` crate that provides a `HgRepoContext` type, which can be used to query the repo for data in Mercurial-specific formats. This will be used in the EdenAPI server.
Initially, the `HgRepoContext`'s functionality is limited to just getting the content of individual files. It will be expanded to support querying more things in later diffs.
Reviewed By: markbt
Differential Revision: D20117038
fbshipit-source-id: 23dd0c727b9e3d80bd6dc873804e41c7772f3146
Summary:
This updates our middleware stack and introduces two new pieces of functinality:
- Middleware can now be async.
- Middleware can now preempt requests and dispatch a response.
The underlying motivation for this is to allow implementing Mononoke LFS's rate
limiting middleware in our existing middleware stack.
Reviewed By: kulshrax
Differential Revision: D20191213
fbshipit-source-id: fc1df7a14eb0bbefd965e32c1fca5557124076b5
Summary: D20121350 changed the methods for accessing file content on `FileContext` to no longer return `Stream`s. We should update the comments accordingly.
Reviewed By: ahornby
Differential Revision: D20160128
fbshipit-source-id: f5bfd7e31bc7e6db63f56b8f4fc238893aa09a90
Summary:
A bunch of files include folly/executors/GlobalExecutors.h transitively through thrift/lib/cpp2/async/Stream.h, which is going away. Explicitly include the header (and add dependency to target) in preparation for deleting Stream.h
drop-conflicts
Reviewed By: vitaut
Differential Revision: D20141838
fbshipit-source-id: 21c58cf82136287fc2d84ba5badec6b872106015
Summary:
This updates the hg_sync_job to update Globalrevs in hgsql before attempting to
sync bundles. This means that if we're syncing successfully, hg is in sync with
Mononoke, and if we fail (which should be very uncommon to begin with!), hg
might skip a little bit ahead, but that's OK.
This only makes sense when generating bundles — when doing pushrebase, hg would
be updating its own globalrevs.
Reviewed By: StanislavGlebik
Differential Revision: D20159262
fbshipit-source-id: 6736f8592682da1001c7c9c4c9444462b71913c2
Summary:
Our previous implementation of unodes had a problem with diamond merges -
essentially because p1 and p2 might have the same file but with different
content unode will always create a merge unode which can be unexpected.
(code comment in unodes/derive.rs has more info about it).
This diff fixes the problem by introducing unodes v2. This allows us to import
new repos with new unode implementation while keeping the old repos with unode
v1.
This implementation uses a heuristic which should be fast and should do the
correct thing most of the time. In some cases it might exclude some parts of
the history completely. For example:
O <- merge commit, doesn't change anything
/ \
P1 | <- modified "file.txt" to "B"
| P2 <- modified "file.txt" to "B"
\ /
ROOT <- created "file.txt" with content "A"
In that case history of "file.txt" starting from merge commit will contain only (P1, ROOT),
but it won't contain P2.
We also considered other options:
1) Move this heuristic to fastlog batch derived data. See D19973553 for more
details about why we decided not to do it.
2) Filter out parent unodes that are ancestors of other parent unodes. This should
always be correct, but it will be hard to implement, it wil be even harder to make
sure it always have good performance.
Reviewed By: krallin
Differential Revision: D19978157
fbshipit-source-id: 445ddd5629669d987e7aa88c35fecf0b34a40da0
Summary: I'd like to log all derivations to a single place so that's it's easier to understand what was derived and where
Reviewed By: aslpavel
Differential Revision: D20140004
fbshipit-source-id: 305ea533031a04ff95995a6fe2a6e57e95a87026
Summary: Log the source node when validating so that we can more quickly reproduce any issues in a single step via the --walk-root option, rather than needing to run the entire walk again.
Differential Revision: D20098200
fbshipit-source-id: 6b0d7d151c97f25080953d6c0fbf431dc2cec6a8
Summary:
Also patch aho-corasick to fix the issue.
The issue was introduced by [an optimization path](063ca0d253) added in aho-corasick 0.7 series (used by globset 0.4.3).
aho-corasick 0.6.x (globset 0.4.2) are not affected.
The next aho-corasick release (0.7.9) contains the fix.
See https://github.com/BurntSushi/aho-corasick/issues/53 for more context.
Reported by: yns88
Reviewed By: DurhamG
Differential Revision: D20125697
fbshipit-source-id: 592375b43d7ee494bb3e916a1cb11c18f9ebe425
Summary:
`parentfunc` is only needed when adding new nodes to the DAG.
Move it to `addheads` methods instead.
Reviewed By: sfilipco
Differential Revision: D20155398
fbshipit-source-id: 0bddd5f46e84c44891928b9f598a38206917aecb
Summary:
Migrate away from some uses of revision numbers.
Some dead code in discovery.py is removed.
I also fixed some test issues when I run tests locally.
Reviewed By: sfilipco
Differential Revision: D20155399
fbshipit-source-id: bfdcb57f06374f9f27be51b0980652ef50a2c8e0
Summary:
`hiddenoverride` is a hacky implementation that preserves part of another hacky
`inhibit` extension. With our modern setup (inhibit or narrow-heads),
`hiddenoverride` is less useful. Therefore just remove it.
Reviewed By: sfilipco
Differential Revision: D20148011
fbshipit-source-id: f4a5f05b67ae6f315e9b07d50ef03018d6d05df5
Summary:
This makes it so that DAG calculations in NameDag are all using commit hashes.
The `id2node`, `node2id` APIs are still using integer ids, and hopefully their
usage can eventually be removed.
Reviewed By: sfilipco
Differential Revision: D20020527
fbshipit-source-id: ee32b1ccacabd5174ff1556e426b5ed32d2b8507
Summary:
This exposes the NameSet type to the Python world.
The code is similar to the SpanSet wrapper that exists in pydag.
Reviewed By: sfilipco
Differential Revision: D20020521
fbshipit-source-id: 840e009eadca7154f11ca61561da4c48022088f6
Summary: This makes it possible to use NameIter in py_class.
Reviewed By: sfilipco
Differential Revision: D20020529
fbshipit-source-id: b9147b7dccb38d18d8361b420507fcbe97e01351
Summary:
Mercurial has a special case that b'\0' * 20 maps to rev -1 and means
"an empty commit". This cannot be cleanly supported by the zstore commit data,
since sha1("") is not '\0' * 20 and zstore does not allow faked SHA1 keys.
Therefore let's add the special case in the bindings layer. It's possible to
do this check in Python, but that'll be slower.
Reviewed By: sfilipco
Differential Revision: D20020520
fbshipit-source-id: 0686832666646f2e201035992e3951b47c32eb5a
Summary: Use the new NameDag as the backing structure and expose its APIs.
Reviewed By: sfilipco
Differential Revision: D20020528
fbshipit-source-id: ccb49e1a5e757bd35a3f71cfb54ceccfb544664e
Summary: This will be used by commit hash prefix lookup.
Reviewed By: sfilipco
Differential Revision: D20020523
fbshipit-source-id: f2905ddf63098704b08dad8eb48272c3ffba7e25
Summary: Export common types at the top-level of the crate so it's easier to use.
Reviewed By: sfilipco
Differential Revision: D20020526
fbshipit-source-id: e9a0a8bc3cc91f81d0bc74e7530cd4613fc1dd61
Summary: Those just delegate to IdDag for the actual calculation.
Reviewed By: sfilipco
Differential Revision: D20020522
fbshipit-source-id: 272828c520097c993ab50dac6ecc94dc370c8e8b
Summary: This will be used to produce NameSet.
Reviewed By: sfilipco
Differential Revision: D20020519
fbshipit-source-id: abf6d73f2b985b74560d6b5db2800ff25450f02e
Summary: DagSet's SpanSet has fast paths for set operations. Use them.
Reviewed By: sfilipco
Differential Revision: D19912104
fbshipit-source-id: 24b55aa14d03be2f1be59c923e0b8e79d6bcbe6d
Summary: This is similar to hg's fullreposet. It'll be useful as a dummy "subset".
Reviewed By: sfilipco
Differential Revision: D19912108
fbshipit-source-id: 33a95bcb3cf5931a431a1201d1a1f3c627cec7a1
Summary: SortedSet is a wrapper to other sets that marks it as topologically sorted.
Reviewed By: sfilipco
Differential Revision: D19912111
fbshipit-source-id: 2637e8fd29b97f6db0c5bae3f0decd7ac382eeb1
Summary:
Wraps SpanSet + IdMap so it only exposes commit names without ids.
There is no equivalent smartset in Mercurial.
Reviewed By: sfilipco
Differential Revision: D19912112
fbshipit-source-id: 0d257de11527dfa8836065ac94f652730a97a468
Summary: Similar to Mercurial's smartset.baseset. All names are statically known.
Reviewed By: sfilipco
Differential Revision: D19912105
fbshipit-source-id: e4fcf2d59291adb3ca01b3b90f1ac32c65ad7eaa
