Commit Graph

16876 Commits

Author SHA1 Message Date
Chad Austin
142166abaf light EdenStats refactoring
Summary:
Rename "channel" to "fs channel", rename StatPtr to DurationStatPtr,
and generalize DurationStatPtr to every stat category.

Reviewed By: xavierd

Differential Revision: D39631597

fbshipit-source-id: 147f9a9d9fe97a39c0d38ccecf57e73c93af0d8c
2022-09-23 17:45:28 -07:00
Chad Austin
5a27ae9aab measure blob metadata lookup time
Summary: Add a counter for blob metadata lookup from hg.

Reviewed By: xavierd

Differential Revision: D39593967

fbshipit-source-id: 544ac3a46de7ce396cefc1a52c4856425f546a7f
2022-09-23 17:45:28 -07:00
Chad Austin
02fad1a987 rename HgBackingStore fb303 counters so they indicate units
Summary:
I've been confused by some of our duration statistics not having
units, so migrate the hg backing store statistics to Duration, and
rename them to have a _us suffix.

Reviewed By: xavierd

Differential Revision: D39593787

fbshipit-source-id: d224a12f5cff436b138e5c3fc283b3b63fdec596
2022-09-23 17:45:28 -07:00
Chad Austin
f685e2813f add a DurationStat type to EdenThreadStatsBase
Summary:
I'd like to have a type-level difference between duration stats and
counter stats. As a first step, let's introduce a Duration stat class
without addValue but with addDuration.

Reviewed By: xavierd

Differential Revision: D39591210

fbshipit-source-id: 5b1cc68dc4a7d8255fb04b2076ad05fe53b781d8
2022-09-23 17:45:28 -07:00
Chad Austin
d806784cb3 add a benchmark for RocksDbLocalStore::getBlobMetadata
Summary:
I was curious how long it takes to fetch metadata from LocalStore. So
add a benchmark.

Reviewed By: xavierd

Differential Revision: D39590670

fbshipit-source-id: a95ba55adb911103a676100fec8dba9954d16307
2022-09-23 17:45:28 -07:00
Michael Cuevas
66d70ec0a5 add missing help info for prefetch-profile fetch
Reviewed By: genevievehelsel

Differential Revision: D39718221

fbshipit-source-id: e63a9ff50c64ebb3fcdb6269d40af6900a36c2e9
2022-09-23 17:33:14 -07:00
Michael Cuevas
52b4fd4195 fix some compilation warnings on macos and windows
Reviewed By: chadaustin

Differential Revision: D39644167

fbshipit-source-id: 5e81bb0a4886630624db76c9e3da27b086e68e3b
2022-09-23 17:33:14 -07:00
Michael Cuevas
698bd6266a use std::Process::Command instead of Exec::cmd
Reviewed By: kmancini

Differential Revision: D39641284

fbshipit-source-id: 19e009da6f82afbba7661ed7e8443d333f052666
2022-09-23 17:33:14 -07:00
Michael Bolin
26321dd222 introduce issensitiveargname() check
Differential Revision: D39775123

fbshipit-source-id: b08757afb0e84d7b03e2fb6060907e80853e4599
2022-09-23 15:41:41 -07:00
Michael van der Westhuizen
f21854d20e Build for aarch64 in addition to x86_64
Summary:
Allow CI/CD builds to build and publish aarch64 builds alongside the existing x86_64 builds.

The published RPMs will have the same version number as the x86_64 RPMs allowing standard slowroll logic to push the packages to the fleet.  This allows the aarch64 bringup team to stop special-casing slowroll.

This diff affects only the RPMs that are required for initial bringup.  Subsequent activity will add more RPMs.

Context about the aarch64 effort: https://fb.workplace.com/groups/1858357931081326/posts/3709554039295030

Reviewed By: rahulg

Differential Revision: D39770905

fbshipit-source-id: 3074f974ff8a75a2b4f6fffb17224eb452a56edb
2022-09-23 10:00:53 -07:00
Egor Tkachenko
f64a8c8be0 Add logging per request
Summary: This diff adds logging to the scuba for derived data service. As well as constructs the logger with logged repo_name, which will be used by downstream dependency, like derived data manager.

Reviewed By: mitrandir77

Differential Revision: D39578035

fbshipit-source-id: 136c88a07b6aeb7776bd6a5b6cb580d3485e31ae
2022-09-23 07:50:43 -07:00
Gustavo Andrade Do Vale
d51b6b301a Adding initial version of the pushrebase logic to the Land Service
Summary:
It does the necessary modifications on the land service server and thrift files to adapt the needed fields to introduce the pushrebase logic. For now, it gets the ctx and authz and prepares the code to new implementations (upcoming diff). For instance, implementations to create a LocalPushrebaseClient, call the pushrebase from it, and convert the output for the LandChansetsResponse.

We also spent some effort creating handling errors that are out of LandChangesetsExn, such as MononokeErrors and BookmarkMovementError

Reviewed By: yancouto

Differential Revision: D39707704

fbshipit-source-id: ada056bf022c38e9398db8c7aad429bfd830754a
2022-09-23 07:20:20 -07:00
Pierre Chevalier
3718d5badc Fix clippy warnings
Summary:
Fix two warnings like this one:
```
warning: this expression creates a reference which is immediately dereferenced by the compiler
   --> eden/mononoke/streaming_clone/src/main.rs:121:41
    |
121 |             let ctx = build_context(fb, &logger, &repo, &tag);
    |                                         ^^^^^^^ help: change this to: `logger`
    |
    = note: `#[warn(clippy::needless_borrow)]` on by default
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_borrow
```

Reviewed By: markbt

Differential Revision: D39767275

fbshipit-source-id: da8226a362a17688ed10022786c8a6eb4063e392
2022-09-23 07:10:22 -07:00
CodemodService Bot
e203646b89 Reviewed By: zertosh
Differential Revision: D39765405

fbshipit-source-id: f7f0f5714cf99e4b16fb60d12d546ce8009565a8
2022-09-23 04:39:39 -07:00
Mark Juggurnauth-Thomas
729513eb25 statistics_collector: ignore redacted files
Summary: When computing statistics on historical commits, if a file is redacted, just ignore it.

Differential Revision: D39742659

fbshipit-source-id: 5203e3bd1cfe628bcd09be44879d707b22de8470
2022-09-23 04:20:17 -07:00
Saul Gutierrez
a45129fd12 revsets: add bookmark resolution in Rust
Summary: Adds local and remote revset resolution to the `resolve_single` function

Differential Revision: D39582514

fbshipit-source-id: 011dcab99e7661f36b0e68e6ce2316ac42aa8f26
2022-09-22 16:29:13 -07:00
Muir Manders
4233063555 color: fix colors when using "less" pager
Summary:
When using a pager, we want to inherit the "is tty?" determination from before we fiddle with stdout to point to the pager. Python had some special logic for this we lost when we switched to Rust to answer "should color?". Streampager still worked because Rust handles that pager. "Less" is handled in Python, so Rust only saw that stdout wasn't a tty, so it disabled colors.

Interestingly, the "should color?" determination happens before we enable the pager. However, we re-initialize the "color" extension after starting a non-streampager pager, so that disables colors in this case. Fortunately, we can just remove the re-initialization because the motivating config "color.pagermode" is apparently no longer supported. This works around the issue because now we only ask Rust "should color?" before the pager is activated.

Reviewed By: sggutier

Differential Revision: D39749806

fbshipit-source-id: a5ce205200b555fbfcbadcdf030d5f2b9ab08b38
2022-09-22 16:04:07 -07:00
Genevieve (Genna) Helsel
a16690c0f1 rename OverlayType to TreeOverlayType
Summary: In preparation for the split between TreeOverlays and BlobOverlays, this diff makes it explicit that the OverlayType applies to tree storage

Reviewed By: chadaustin

Differential Revision: D38279396

fbshipit-source-id: c56805818e43471f5301329beb0b5a51e0b04a69
2022-09-22 15:04:17 -07:00
Zhaolong Zhu
818a278dc7 matcher: introducing regexmatcher
Summary: this diff is for introducing regexmatcher, its usage is in D39741975

Reviewed By: quark-zju

Differential Revision: D39731271

fbshipit-source-id: 3367aea7edc0121dba767e96597a8c521c81b7ed
2022-09-22 13:53:38 -07:00
Mateusz Kwapich
b7446438e2 fix timeouts and revise constants
Summary:
The internal fastlog batches are 100 commits each - so that's the right size
for page size here too.

The timeout constant wasn't used anywhere (which made the timeout 10s). I'm
upping it because in very mergy repos the fastlog traversal is unfortunately
0(merges on the path).

Reviewed By: quark-zju

Differential Revision: D39738299

fbshipit-source-id: 2eb2cf450f433302494ea2a523bd6af158fc6998
2022-09-22 12:32:21 -07:00
Gustavo Andrade Do Vale
17d45fa65d Integration tests making SSL certificates work
Summary: In the previous diff (D39580462 (99e4c1c935)) we made the connection between the land service and the mononoke server. However, it was missing the certificates and primary key work properly. This diff makes it works. Next, we will implement the pushrebase logic works on the land service and add the tests.

Reviewed By: yancouto

Differential Revision: D39652929

fbshipit-source-id: 29d4d502c715eeecbea329ed69e8b65ae81064df
2022-09-22 09:18:52 -07:00
Zhaolong Zhu
9ccbcb1372 match: add _kindpatstoglobsregexs to convert kindpats to globs and regexs
Summary: This diff is a preparation for using Rust RegexMatcher.

Reviewed By: quark-zju

Differential Revision: D39706345

fbshipit-source-id: ce4f978aa58f43ca3321e146b4892fd57abc67e5
2022-09-22 08:37:40 -07:00
Gustavo Andrade Do Vale
99e4c1c935 Initial version of the integration tests for the land service
Summary: It implements the first version of tests related to the land service. For short it works until the part it needs the certificates to perform the pushrebases. Next diff it should be fixed.

Reviewed By: RajivTS

Differential Revision: D39580462

fbshipit-source-id: 0f6b6e974a22dfa91c8aa7572f00cf887e14f8dc
2022-09-22 08:02:39 -07:00
Pyre Bot Jr
0157e43863 Add annotations to eden/scm/edenscm/ext/shelve.py
Reviewed By: jordanwebster

Differential Revision: D39727583

fbshipit-source-id: 7bc6d38123fd819911e539062f679c190b6da8ef
2022-09-22 05:41:50 -07:00
Ilia Medianikov
dc73b7a6d2 mononoke: remove "bundle_replay_data" SQL table references
Summary: It is unused.

Reviewed By: YousefSalama

Differential Revision: D39660635

fbshipit-source-id: d4a00e6036c7af883dc35acfbae555396fc440f4
2022-09-22 04:40:05 -07:00
Gustavo Andrade Do Vale
dfc568ff36 Changing the config to make the land service client work
Summary:
It (1) adds the repo in the Land Service, (2) includes Remote Land Service and remote land service with local fallback, (2) get the address from tier or host and calls the client in the unbundle processing

The Static analysis failed in the FBTHRIFTCOMPAT1 because i added a new field in the land_changesets method in the impl LandService (land_service_impl.rs). It does not break anything since we do not have the server in production. I mean, changing the id does not influence on previous versions of the server since there is no land service server in production.

Reviewed By: yancouto

Differential Revision: D39553570

fbshipit-source-id: bb1f09c2ab2c111734b9026109fc39e14c514516
2022-09-22 04:28:36 -07:00
Dmitry Safonov
83de9cf049 migrate new_streaming_clone to new CLI framework
Reviewed By: mitrandir77

Differential Revision: D39621622

fbshipit-source-id: 7e3bb287363cd91f83d374c38a70151106afd0cf
2022-09-22 03:31:22 -07:00
MSDK Bot
555c513f10 Update isl component on FBS:master
Differential Revision: D39720035

fbshipit-source-id: 5e777aabcb44a561b6fde9f49f1a5045da84715d
2022-09-21 22:59:53 -07:00
Zhaolong Zhu
71bb3e40cd regexmatcher: handle empty dir for testing "root" dir
Summary: handle special empty dir for testing "root" dir

Reviewed By: quark-zju

Differential Revision: D39703941

fbshipit-source-id: 59ac5b281fe3eb81af4d12e1a99a18861040559c
2022-09-21 17:45:07 -07:00
Mark Shroyer
e0779babd3 Fix rage pastry title's hostname substitution
Summary:
The default rage.reporter config assumes shell substitution of `$(hostname)`,
but as implemented this doesn't work.  So pastries created by `eden rage` all
have the same title `eden rage from $(hostname)`.

This change enables `{hostname}` variable (not shell) substitution for `eden
rage`.

Side note, it looks like we popen the rage.reporter in at least three different
places:

- https://www.internalfb.com/code/fbsource/[dcfed651fdd7]/fbcode/eden/fs/cli/main.py?lines=2024
- https://www.internalfb.com/code/fbsource/[dcfed651fdd7]/fbcode/eden/fs/cli/rage.py?lines=312
- https://www.internalfb.com/code/fbsource/[dcfed651fdd7]/fbcode/eden/fs/cli/facebook/rage.py?lines=265

The instance in main.py is the only one that seems to be invoked when running
`eden rage`.  Are the others dead code?

Reviewed By: fanzeyi

Differential Revision: D39709054

fbshipit-source-id: 6c5a45bf948b00cd730fbb065095b995e27011f6
2022-09-21 16:55:38 -07:00
Michael Cuevas
535222e634 fix Rust config dir selection
Summary:
Previously, we would simply check if --config-dir was supplied. If not, we would use the default config dir path.

This is incorrect, as we're supposed to check whether the cwd is located in an eden mount. If so, we should use the config dir for whatever Eden mount we're currently in.

Reviewed By: fanzeyi

Differential Revision: D39705519

fbshipit-source-id: 233520ff3046bfb4b004b919f917c0569c804e91
2022-09-21 16:40:40 -07:00
Genevieve (Genna) Helsel
e547b1ea8f remove Windows FSCK hacky check
Summary: We don't have SqliteOverlay anymore, so this check can be removed and instead we can just `#ifdef` the snippet

Reviewed By: xavierd

Differential Revision: D39673738

fbshipit-source-id: 04a3b755637c10c9c3cfd230aa8629cdb551acf2
2022-09-21 16:03:40 -07:00
Brandon LeBlanc
49571cd219 update indexmap from 1.8.0 to 1.9.1
Summary: Changes: https://github.com/bluss/indexmap/compare/1.8.1...1.9.1

Reviewed By: ndmitchell

Differential Revision: D39530366

fbshipit-source-id: 303a3135861882ff4e4de990c1f79f7ab05009e7
2022-09-21 13:29:45 -07:00
Muir Manders
447bbbb7c7 formatter: support "--color debug" in Rust
Summary: This adds some markup to the output so you can see what style is being applied to what parts of the output.

Reviewed By: quark-zju

Differential Revision: D39621629

fbshipit-source-id: a875bce34452dbed53ff0ab7d019ca57952ab4c0
2022-09-21 12:31:41 -07:00
Muir Manders
c5b81595c8 formatter: use proper "should color" detection
Summary:
Now users of the formatter can unconditionally use "write_styled()" and the formatter will output regular text if color is disabled.

I will fix "--color debug" output in next diff.

Reviewed By: quark-zju

Differential Revision: D39621626

fbshipit-source-id: 13b01acaf4dfdde728aa4a5153921ca3416b7c1c
2022-09-21 12:31:41 -07:00
Muir Manders
11f24b3455 color: use Rust "should color?" determination
Summary:
Replace Python color mode detection with Rust's.

Windows Virtual Terminal mode is enabled in hgmain, so we don't need the Python code anymore.

Reviewed By: quark-zju

Differential Revision: D39621631

fbshipit-source-id: eca87ea4d13b226211c601abe9b7113b9207b5d1
2022-09-21 12:31:41 -07:00
Muir Manders
d86da1f9c3 termstyle: add code to determine if we should use colors
Summary:
This will replace most of edenscm.color._modesetup in the following commit.

This logic should match Python logic closely, except Rust has a subset of the logic implemented on the io::IsTty "can_color()" method. can_color() says whether colors appear to be supported by the io handle and environment. should_color combines that information with the user's configuration to determine whether we should try coloring output.

Reviewed By: quark-zju

Differential Revision: D39621630

fbshipit-source-id: 40cfef1e4682c844c69f2905315b16bfaf4d9a0f
2022-09-21 12:31:41 -07:00
Muir Manders
7cd5f02ef9 termstyle: shuffle effects code to a separate module
Summary: Going to add some more stuff to this crate.

Reviewed By: quark-zju

Differential Revision: D39621632

fbshipit-source-id: cb78b76f741fd15dde93381c09424ba4f6018bb0
2022-09-21 12:31:41 -07:00
Muir Manders
76fcf5116e move configparser::hg::is_plain to separate crate
Summary:
configparser is a weird and annoying place for an env var utility to live.

Leave the tests in the configparser create so they continue to benefit from the env locking mechanism.

Reviewed By: quark-zju

Differential Revision: D39621628

fbshipit-source-id: 816f12e2492038d6e8a7af76c94849681ce576fd
2022-09-21 12:31:41 -07:00
Muir Manders
bf90a93084 io: check for TERM=dumb in can_color()
Summary:
This check mimics color.py. I'm putting non-config related logic in IO can_color so things like tracing output can color appropriately without access to the config.

I got rid of the separate CanColor trait and merged it into IsTty.

Reviewed By: quark-zju

Differential Revision: D39621627

fbshipit-source-id: a843fa83a5de8f18f64cc1f8b2234fc330d86192
2022-09-21 12:31:41 -07:00
Michael Bolin
3c619f2f89 include DotSlash version of ISL for Windows
Reviewed By: evangrayk

Differential Revision: D39681885

fbshipit-source-id: 70f7a88791a7c1d3e350c278378ea7bfa662cdfc
2022-09-21 09:31:08 -07:00
Muir Manders
1290f518a7 status: don't respect sparse profile on EdenFS
Summary: Don't read the sparse config if the fs type is eden. This prevents sparse profiles from accidentally working even on EdenFS working copies.

Reviewed By: quark-zju

Differential Revision: D39659620

fbshipit-source-id: e29dc563fe87680a3058230e97190a891ea8e5a7
2022-09-21 09:22:08 -07:00
Muir Manders
88fd6f8ab6 sparse: support "tempsparse" overrides in Rust
Summary: During a merge we temporarily add files that need merging to the sparse profile to make sure they show up properly. Now we read from the "tempsparse" file in Rust and respect those overrides.

Reviewed By: quark-zju

Differential Revision: D39659619

fbshipit-source-id: 3265900aac8eee621be9feea87f136664a448e61
2022-09-21 09:22:08 -07:00
Muir Manders
bca3785944 progress: fix issue rendering multiple bars
Summary:
When rendering enough progress bars, the cursor would jump too far back after rendering the bars, which would mess things up (because new output would overwrite old output).

This was caused by our use of termwiz ChangeSequence, which tracks relative cursor x and y so it is easy to jump the cursor back after outputing a bunch of progress bars. ChangeSequence interprets "\n" as y+1, but _not_ x=0, so once the cumulative length of the progress bars became greater than the terminal width, you ended up with an "extra" y+1 since ChangeSequence thought the line wrapped. When jumping the cursor back, it would go back too far.

Anyway, fix by changing our progress output to use \r\n instead of \n.

Reviewed By: sggutier

Differential Revision: D39677310

fbshipit-source-id: 5205af30dd1bac94ce2eae175ad8a05634ed1e85
2022-09-21 08:39:13 -07:00
Jan Mazur
351cc8e31d add repo_info method
Summary: I will use this to correctly infer scm_type in Hack's scmquery-proxy client library.

Differential Revision: D39492898

fbshipit-source-id: 4d9aafcb88f75b20260d4212bae70495efca22d5
2022-09-21 07:56:02 -07:00
Mateusz Kwapich
bc15663eb2 mark test-megarepo-invisible-merge test as slow
Summary: It times out in CI.

Differential Revision: D39620249

fbshipit-source-id: c4c9708d6557e87ad0f10fb9c13f88737547c0ce
2022-09-21 06:58:24 -07:00
Liubov Dmitrieva
d0fd27541d deprecate support for very old commits
Summary:
Deprecate support for very old commits.

Very old commits will be stripped off users workspaces, we can also deprecated
this redundant code. This is because the very old commits are in flat manifest
format that is not longer supported by hg client.

Reviewed By: quark-zju

Differential Revision: D39657045

fbshipit-source-id: 32318aff02bed88307737464525474b626dfcaa8
2022-09-21 04:59:17 -07:00
Xavier Deguillard
0358ba1eaf model: use a constant time hash for ObjectId
Summary:
The ObjectCache is an extremely hot code path, dominating the glob query times.
When profiling it, hashing ObjectId shows up as a big component of the
ObjectCache runtime. By convention, we know that ObjectId contain a hash
already, and thus we can use that as the ObjectId hash directly.

Reviewed By: chadaustin

Differential Revision: D39399209

fbshipit-source-id: b8113102ac94445efed74dcdd88bb0b1f3c62591
2022-09-21 03:13:51 -07:00
Zhaolong Zhu
9a51b6804d match: extract a literal constant into a constant variable
Summary:
extract a literal constant into a constant variable, and it seems also has better performance.

```
>>> dis.dis(_buildregexmatch_with_literal)
...
1627          12 LOAD_GLOBAL              1 (len)
              14 LOAD_FAST                2 (regex)
              16 CALL_FUNCTION            1
              18 LOAD_CONST               1 (20000)
              20 COMPARE_OP               4 (>)
              22 POP_JUMP_IF_FALSE       28
...

>>>  dis.dis(_buildregexmatch)
...
1597          12 LOAD_GLOBAL              1 (len)
              14 LOAD_FAST                2 (regex)
              16 CALL_FUNCTION            1
              18 LOAD_GLOBAL              2 (MAX_RE_SIZE)
              20 COMPARE_OP               4 (>)
              22 POP_JUMP_IF_FALSE       28
...

>>> timeit.timeit("_buildregexmatch_with_literal([('re', 'fbcode/.*.py', '')], '$')", globals=locals(), number=10000)
1.0288722580007743

>>> timeit.timeit("_buildregexmatch([('re', 'fbcode/.*.py', '')], '$')", globals=locals(), number=10000)
0.8637818480056012
```

Reviewed By: quark-zju

Differential Revision: D39664940

fbshipit-source-id: 4ae6fcfe3f9aa03a1bd1cb9511afe6845a347588
2022-09-20 15:41:12 -07:00
Zhaolong Zhu
34237f0ca2 matcher: refactor _buildregexmatch help function
Summary: refactor _buildregexmatch function to make a new function `_buildregex`, this will be reused when using Rust RegexMatcher.

Reviewed By: quark-zju

Differential Revision: D39657735

fbshipit-source-id: d5bac663df6dd0842951392e2cb4ef7c6812b1e1
2022-09-20 15:24:42 -07:00