Commit Graph

4281 Commits

Author SHA1 Message Date
Andrey Chursin
9712359884 pull: use fast path pull for master
Summary:
Use segmented changelog fast path during master branch pull
This change is gated by a config

Reviewed By: quark-zju

Differential Revision: D29347804

fbshipit-source-id: f0cc7633e67fb04266ce15506375873d58ea77fc
2021-06-24 20:36:13 -07:00
Jun Wu
cf020c59e7 pyerror: expose NeedSlowPathError
Summary:
Make it possible for Python to detect the error and fallback to slow path
accordingly.

Reviewed By: andll

Differential Revision: D29369067

fbshipit-source-id: 1de6162532b145469c09070bdf843ec96d38e431
2021-06-24 17:54:46 -07:00
Jun Wu
ed7c8931fa dag: make parents_and_head include roots
Summary:
An alternative to D29363808 (e396cab669). The benefit is that parents_and_head is used by
both the client and the server. So we don't need to duplicate D29363808 (e396cab669) in
Mononoke code.

Reviewed By: andll

Differential Revision: D29365079

fbshipit-source-id: bca60ba2b3df477929d8e72b2363e5a0f744b35d
2021-06-24 17:54:46 -07:00
Jun Wu
ac63c8df22 dag: forbid pull fast path with pending changes
Summary:
Pull fast path uses `reload` which drops pending changes.
To avoid misuse, raise an error if pending changes are present.

Reviewed By: andll

Differential Revision: D29363799

fbshipit-source-id: 8f520d2c5553432abc452bc7b2b59d7af80e0a99
2021-06-24 17:54:46 -07:00
Jun Wu
29d07f8f73 dag: reduce round-trip looking up remotely in pull
Summary: Reduce network round-trip by prefetching missing parents and roots.

Reviewed By: andll

Differential Revision: D29363797

fbshipit-source-id: ba9201ba4a22e08effaa82026398e16f69b7c0c6
2021-06-24 17:54:46 -07:00
Jun Wu
d4e08f7345 dag: check overlapped vertexes during pull
Summary: Resolve an issue demonstrated by test_pull_overlap.

Reviewed By: andll

Differential Revision: D29363800

fbshipit-source-id: 4ba2490e9c9f0d659974177bac19480c8aeef62b
2021-06-24 17:54:45 -07:00
Jun Wu
4ee4d0be21 dag: store resolved parents in pull fast path
Summary:
The import pull data logic used low-level locking, persisting APIs, it does not
write cached idmap to disk. So we need to manually insert the idmap remote
lookup result to the actual local idmap explicitly.

This addressed an issue that verify_missing fails in the pull fast path.

Reviewed By: andll

Differential Revision: D29363813

fbshipit-source-id: 2749855a6c8c924bd1b772173de066d400f73764
2021-06-24 16:20:14 -07:00
Jun Wu
10d6c23b83 dag: resolve names remotely in pull fast path
Summary:
For a NameDag, `IdConvert` on `self.map` cannot resolve names remotely, but
`IdConvert` on `self` can. Use the latter. This is similar to D27547584 (af3c3b3fd0) where
some `self.map` are updated to `self`.

This addressed an issue found in the pull fast path test. Note there is another
issues to solve.

Reviewed By: andll

Differential Revision: D29363810

fbshipit-source-id: 28ba583ed14bbc5d52af81d4128d965f24eef011
2021-06-24 16:20:14 -07:00
Jun Wu
677097aca1 dag: add a more complex test about pull fast path
Summary: The test pulls when the client has a lazy graph, and the server has a few merges.

Reviewed By: andll

Differential Revision: D29363806

fbshipit-source-id: 09bc3c4c3d21924f500ca86e8d86f58a15159169
2021-06-24 16:20:14 -07:00
Jun Wu
16796c9fb5 dag: add TestDag::debug_segments
Summary:
`fmt::Debug` for a NameDag is too verbose. Separate part of it so we can debug
print segments for a given (group, level).  This will be used by upcoming
changes.

Reviewed By: andll

Differential Revision: D29363805

fbshipit-source-id: e1c6713be10b8b64fc7a42178117e724e0d691d0
2021-06-24 16:20:14 -07:00
Victoria Tsai
ad9e1b0d4b Fix rust lint errors
Summary: unused import and unused variable

Differential Revision: D29365367

fbshipit-source-id: 39d625eba232deeb8e70e8ba11c6043b4ea60122
2021-06-24 15:54:45 -07:00
Jun Wu
b5b41b733c test-fb-hgext-treemanifest-peertopeer: fix test on macOS
Summary: macOS's `wc -l` prints extra spaces. Remove them.

Differential Revision: D29367789

fbshipit-source-id: 23ccacd015a1c5cdc186fcb0c282dd63f4171021
2021-06-24 15:17:06 -07:00
Jun Wu
73ab6e9e6f dag: ensure TestDag::pull updates client's remote
Summary:
The client TestDag might have outdated server Dag as the remote protocol,
because it is a static "snapshot". Ensure the remote Dag is updated when
using the pull API.

This is an issue solved by tracking down issues in tests added in upcoming
diffs.

Reviewed By: andll

Differential Revision: D29363807

fbshipit-source-id: a560b2e91999873338604907a6d83cc7d2ff5c58
2021-06-24 15:12:58 -07:00
Jun Wu
27f864cc5b dag: extract TestDag::set_remote from client
Summary: It will be used by the next change.

Reviewed By: andll

Differential Revision: D29363802

fbshipit-source-id: 842735ac05ea5fea4ea0c3625a68d06d27bc37d5
2021-06-24 15:12:58 -07:00
Jun Wu
c908ead7fa dag: add TestDag::drawdag_async
Summary:
It is useful when drawdag itself triggers remote fetches.

This was used but is not used after some refactoring. I think it might be useful
in the future so kept it.

Reviewed By: andll

Differential Revision: D29363803

fbshipit-source-id: fa178ac9783d1dc1b73525eeb8cd3d766cf46a0f
2021-06-24 15:12:58 -07:00
Jun Wu
63619fd27b dag: add test about pull checking overlapped vertexes
Summary: The test will be used to verify upcoming changes.

Reviewed By: andll

Differential Revision: D29363809

fbshipit-source-id: d34d13123914cfabb5c82dee3873b6e0c4979ee2
2021-06-24 15:12:58 -07:00
Jun Wu
367ce3dbc3 dag: add TestDag::pull_ff_master
Summary: Make it easier to write more tests around pull.

Reviewed By: andll

Differential Revision: D29363804

fbshipit-source-id: 5b2cf8675343898fabc1d8845228e240e463edf8
2021-06-24 15:12:58 -07:00
Jun Wu
d3b87c6861 indexedlog: resolve a warning
Summary: Follow up of D29035913 (63884d65c1).

Reviewed By: singhsrb

Differential Revision: D29363811

fbshipit-source-id: 1147b5e28f73bf893c30f3bea5a617fc116885bf
2021-06-24 15:12:58 -07:00
Jun Wu
e396cab669 dag: include roots in pull data
Summary:
The roots data will be useful for the client to check if the pulled commits are
going to overlap with its existed DAG.

Reviewed By: andll

Differential Revision: D29363808

fbshipit-source-id: e09d924d65537f59fd4ea209b568265d07a80e46
2021-06-24 15:12:58 -07:00
Jun Wu
21e2bdd709 dag: sever => server
Summary: Minor spelling fix.

Reviewed By: andll

Differential Revision: D29363812

fbshipit-source-id: 58393ddac058e1e63182aa5bc4a85e5dcf04c599
2021-06-24 15:00:35 -07:00
Jun Wu
f8af66d7d2 dag: x..y+1 => x..=y
Summary: Minor change to make the code a little bit more straightforward.

Reviewed By: andll

Differential Revision: D29363801

fbshipit-source-id: 2c4bd6ece07282f044622227a3c077cb31db6d17
2021-06-24 15:00:35 -07:00
Jun Wu
77e8dcc21d dag: minor updates on iddag docstring
Summary: Make the docstring a bit more consistent.

Reviewed By: andll

Differential Revision: D29363798

fbshipit-source-id: 1b4e2a7a1af4c4cffe3693e437a831bab1b43fd7
2021-06-24 15:00:35 -07:00
Andrey Chursin
568237d11c eden_api: implementation for pull_fast_forward_master client request
Reviewed By: quark-zju

Differential Revision: D29342432

fbshipit-source-id: 1a371e707b2e75b155f5abdc343bfb42ba53df5a
2021-06-24 13:58:02 -07:00
Andrey Chursin
49bd00fe2c types: introduce fast forward pull wire types
Reviewed By: quark-zju

Differential Revision: D29342139

fbshipit-source-id: 848eceaf5d4a7e88e9830155c0763321ea275c82
2021-06-24 13:58:02 -07:00
Jun Wu
03003242b0 debuginternals: new command to list or export internal states
Summary:
Sometimes we need repo internal states for deeper investigation. Right now
the process is quite adhoc - telling people to figure out where the "backing"
repo is, excluding unwanted files like 00changelog.d, etc. Let's just make
it easier so we have a dedicated command to tar up selected internal states.

Reviewed By: andll

Differential Revision: D29242716

fbshipit-source-id: 85604d0df5b3a12f1aed29e4f31fcd5c166f9066
2021-06-24 12:31:47 -07:00
Meyer Jacobs
bbe46dcf06 scmstore: avoid empty remote requests in TreeStore
Summary: Verify we actually have pending keys to fetch before attempting a remote request in scmstore TreeStore.

Reviewed By: kulshrax

Differential Revision: D29345214

fbshipit-source-id: 328bdcbc41429e59de6ceb488533bafa97518fcc
2021-06-23 19:27:26 -07:00
Arun Kulshreshtha
7ffa37c5e9 pyedenapi: use block_unless_interrupted instead of block_on_future
Summary: Previously, it was not possible to interrupt `hg` during EdenAPI fetch operations. This made it impossible to interrupt long-running fetches, which is very frustating to users. This can be simply fixed by using `block_unless_interrupted` in place of `block_on`.

Reviewed By: quark-zju

Differential Revision: D29344670

fbshipit-source-id: 3b0d36dda28f5f7cc812a07981f295f8d0fbdd8a
2021-06-23 19:01:22 -07:00
Andrey Chursin
fc47068389 debugsementpull: introduce debug command to pull using segmented chanelog
Summary:
This is simple command mostly to be used by testing before we fully integrate with hg pull

This command does not perform discovery and requires from/to revision to be passed in cmd line

Reviewed By: quark-zju

Differential Revision: D29315647

fbshipit-source-id: 26d67031e566b7c99af1e2a5ab287f02b52f7db0
2021-06-23 17:58:26 -07:00
Xavier Deguillard
27b55303ee Back out "Enable fb dynamicconfig loading inside eden backingstore"
Summary: This is breaking the Windows release, reverting.

Reviewed By: fanzeyi

Differential Revision: D29339787

fbshipit-source-id: 22d8ff5db5619194e4597754dc37343cf0bc3286
2021-06-23 16:45:43 -07:00
Meyer Jacobs
5e9a33e11e tests: fix test-fb-hgext-treemanifest-peertopeer.t on mactest
Summary: `|&` apparently fails on mactest, so I've replaced it with `2>/dev/stdout | ` which works on my devserver and macbook.

Reviewed By: andll

Differential Revision: D29337621

fbshipit-source-id: eaac2592f4c7bfda6696c2500f3b08441b596c39
2021-06-23 16:11:41 -07:00
Meyer Jacobs
640767d414 scmstore: track contentstore fallbacks
Summary:
Introduce basic contentstore fallback tracking to help monitor the scmstore shim rollout.

This will be expanded to a general fetch metrics system for scmstore in a future change.

Reviewed By: kulshrax

Differential Revision: D29305839

fbshipit-source-id: c6cc3ea15a3bb7b90f4ec298febc911ec4e2af91
2021-06-23 15:26:26 -07:00
Andrey Chursin
c37c61cc0d pydag: add importpulldata binding
Reviewed By: quark-zju

Differential Revision: D29317065

fbshipit-source-id: 19597b72cd3c210f43425df30fbe7e2312e33370
2021-06-23 14:51:39 -07:00
Andrey Chursin
7f54775d00 dag: tests for pull protocol
Reviewed By: quark-zju

Differential Revision: D29177851

fbshipit-source-id: 8c2b35306cc68d9c4d18449fd80e5504bd650031
2021-06-22 18:14:41 -07:00
Andrey Chursin
8c12f1f786 dag: implementation for import_pull_data
Reviewed By: quark-zju

Differential Revision: D29147359

fbshipit-source-id: 0508b48656fda99f37e9e3377a6ac08149bc1dcf
2021-06-22 18:14:41 -07:00
Pyre Bot Jr
ca842ced69 suppress errors in fbcode/eden - batch 1
Differential Revision: D29301094

fbshipit-source-id: 2b0c3dc44295341efb68ae15403cd2df4e837db7
2021-06-22 13:31:57 -07:00
Meyer Jacobs
b5858adee1 scmstore: update remaining tests
Summary: Update the remaining tests for scmstore. In each of these cases we're just disabling scmstore for various reasons. I think `test-lfs-bundle.t` and `test-lfs.t`'s failures represents a legitimate issue with scmstore's contentstore fallback, but I don't think it should block the rollout

Reviewed By: kulshrax

Differential Revision: D29289515

fbshipit-source-id: 10d055bf679db8efdeb16ac96b7ed597d7b6d82c
2021-06-22 13:14:58 -07:00
Meyer Jacobs
015f5864d2 scmstore: fix deadlock in write
Summary:
Prevent `FileStore` from deadlocking when a write falls back to contentstore and attempts to write to the same indexedlog_local which is held lock for the batch.

Note: this shouldn't need to block release, we current expect writing raw LFS pointers to only happen with non-remotefilelog LFS.

Reviewed By: kulshrax

Differential Revision: D29299050

fbshipit-source-id: bf39f87b9956165a558f3a19960d3d055685db9a
2021-06-22 13:14:58 -07:00
Andrey Chursin
70857500c7 sparse: fix size calculation on the file outside of sparse profile
Summary:
When user runs hg diff between revision and working copy, and if the diff contains file that is not present in working copy because of it's sparse profile, then hg diff fails

Failure happens because hg diff tries to compare file sizes and calculating file size on workingfilectx fails, because file does not exists.

This diff overwrites size function for workingfilectx in the sparse extension to protect against that, similarly how it is already done for data function

Differential Revision: D29279691

fbshipit-source-id: 55d7843d23370c31693a32a0e1df8b882db0d89d
2021-06-22 11:22:30 -07:00
Andres Suarez
fc37fea20c Update itertools 0.8.2 to 0.10.1
Reviewed By: dtolnay

Differential Revision: D29286012

fbshipit-source-id: 6923c0b750692e6932e85fd539b076b172ff43b7
2021-06-22 04:09:00 -07:00
Jan Mazur
5830589a71 Enable fb dynamicconfig loading inside eden backingstore
Summary: Enable fb dynamicconfig loading inside eden backingstore

Reviewed By: fanzeyi, xavierd

Differential Revision: D29212009

fbshipit-source-id: 79f3ab1a5764101c37dcf3e4676c0c0394d88d40
2021-06-22 02:17:32 -07:00
Meyer Jacobs
43a75431bb scmstore: update additional test
Summary: Only four tests remaining after this.

Reviewed By: kulshrax

Differential Revision: D29229656

fbshipit-source-id: 56c0a17f6585263e983ce8bc3c345b1f266422e0
2021-06-21 20:32:50 -07:00
Meyer Jacobs
88ab7198bc scmstore: update more tests
Summary: Update more tests to avoid relying on pack files and legacy LFS, and override configs in `test-inconsistent-hash.t` to continue using pack files even after the scmstore rollout to test the Mononoke's response to corruption, which is not currently as easy with indexedlog.

Reviewed By: quark-zju

Differential Revision: D29229650

fbshipit-source-id: 11fe677fcecbb19acbefc9182b17062b8e1644d8
2021-06-21 20:32:50 -07:00
Stanislau Hlebik
790986d561 setdiscovery: fix failing pulls
Summary:
Looks like D29145340 (d0e16f1a25) introduced regression - "hg pull" fails with
"TypeError: Population must be a sequence or set.  For dicts, use list(d).", stack trace - P424458181.

This diff fixing it by converted a dag.nameset to a list first

Reviewed By: mzr

Differential Revision: D29258771

fbshipit-source-id: 9ffcc756f9931d6d24b69dadf1cd2d08faccb443
2021-06-21 03:58:28 -07:00
Meyer Jacobs
c891c84967 scmstore: introduce LegacyStore trait for scmstore shim
Summary:
Introduce `LegacyStore` trait, which contains ContentStore methods not covered by other datastore traits.

Implement this trait for both contentstore and scmstore, and modify rust code which consumes `contentstore` directly to use `PyObject` and `LegacyStore` to abstract over both contentstore and scmstore instead.

Reviewed By: DurhamG

Differential Revision: D29043162

fbshipit-source-id: 26e10b23efc423265d47a8a13b25f223dbaef25c
2021-06-18 17:39:53 -07:00
Meyer Jacobs
18f2bb2b69 scmstore: introduce scmstore.enableshim config
Summary: Introduce a new config option, `scmstore.enableshim`, which replaces instances of contentstore in Python with scmstore objects instead. Currently, this config is not safe to enable. Addition fixes are incoming.

Reviewed By: DurhamG

Differential Revision: D29213190

fbshipit-source-id: 7fd4db77d55cd25cc08c40bee28798d6a6d2555c
2021-06-18 17:39:53 -07:00
Meyer Jacobs
7db0302cd8 scmstore: correctly propagate errors back to Python
Summary: Previously, we just fetched "best effort", and logged any encountered errors using `tracing`, leaving it up to the client to inspect errors if necessary. Python relies on catching these fetch errors as exceptions, though, so this change introduces some utility methods to help propagate them correctly.

Reviewed By: DurhamG

Differential Revision: D29211683

fbshipit-source-id: 5e9dee942c2b60e0f77a051624d7f393a811fc4e
2021-06-18 17:39:52 -07:00
Meyer Jacobs
b489f6b1c8 tests: modify tests to exercise non-packfile configuration
Summary:
Remove packfile-specific parts of tests and modify them to test without depending on packfiles where possible.

Currently debugindexedlogdatastore and debugindexedloghistorystore appear to be broken, and debugdumpindexedlog just dumps the raw indexedlog contents, without any semantic information, so for the time being I've simply removed most packfile inspection.

Reviewed By: DurhamG

Differential Revision: D29099241

fbshipit-source-id: 86c4f9c83520374560587b8bec5c569d9c5c6510
2021-06-18 17:39:52 -07:00
Meyer Jacobs
5b1a16077f scmstore: actually fix undesired file logging
Summary: My previous fix was actually incorrect, we now log actual remote requests, but join that with the logs from the contentstore fallback.

Reviewed By: DurhamG

Differential Revision: D29206878

fbshipit-source-id: d22e58792bf380c274e8086ce08aebe20dd9b848
2021-06-18 17:39:52 -07:00
Meyer Jacobs
ee67634fbf scmstore: gate contentstore fallback behind config
Summary: Introduce a config option to enable or disable contentstore fallback in scmstore.

Reviewed By: DurhamG

Differential Revision: D29206846

fbshipit-source-id: 58e339a8f0f7b8fa98baa53b99ef25b18a066299
2021-06-18 17:39:52 -07:00
Arun Kulshreshtha
fff9086983 edenapi: don't wait for headers from all responses before yielding entries
Summary:
Previously, when fetching data using several concurrent requests, the EdenAPI client would wait for the headers for every request to finish coming in before starting to deserialize and yield entries from the bodies of any of the requests.

Normally, this isn't a huge deal since the response headers on all of the requests are usually roughly the same size, so they all finish downloading at roughly the same time when the requests are run concurrently. However, this does become an issue when `edenapi.maxrequests` is set. This option makes EdenAPI configure libcurl to queue outgoing connections once the configured limit is hit.

This means that although from EdenAPI's perspective all of the requests are running concurrently, they are not actually running in parallel. The result is that the EdenAPI client ends up waiting for all of the queued requests to be sent before yielding any data to the caller, which forces it to buffer all of the received data, resulting in massive memory consumption.

This diff fixes the problem by rearranging the structure of the Futures/Streams involved such that the client immediately begins yielding entries when they are received from any of the underlying transfers.

Reviewed By: quark-zju

Differential Revision: D29204196

fbshipit-source-id: b6b56bb7d60457de3c4046a07a5965749e9dd371
2021-06-18 15:23:25 -07:00
Arun Kulshreshtha
ca8200e9df http-client: make send_async return a Vec of Futures instead of a Stream
Summary:
When the `send_async` method is used to dispatch multiple concurrent requests, the method needs to return an `AsyncResponse` for each request. Since `AsyncResponse`'s constructor is itself `async` (it waits for all of the headers to be received), internally the method ends up with a collection of `AsyncResponse` futures.

Previously, in an attempt to simplify the API, the method would insert all of these futures into a `FuturesUnordered`, thereby conceptually returning a `Stream` of `AsyncResponses`. Unfortunately, this API ends up making it harder to consume the resulting `AsyncResponses` concurrently, as one might want to do when streaming lots of data over several concurrent requests.

This diff changes the API to just insert the `AsyncResponse` futures into a `Vec` to allow the caller to use them as desired. To maintain compatibility with the old behavior for the sake of this diff, the one current callsite has been updated to just dump the returned `Vec` into a `FuturesUnordered`. This will be changed later in the stack.

Reviewed By: quark-zju

Differential Revision: D29204195

fbshipit-source-id: ecee8cff430badd8213c2efef62fc68fbd91fde9
2021-06-18 15:23:25 -07:00
Arun Kulshreshtha
2a13b62754 edenapi: remove metadata from Fetch<T>
Summary: Nothing was using this metadata, and removing it simplifies the subsequent diffs in this stack.

Reviewed By: quark-zju

Differential Revision: D29147228

fbshipit-source-id: aa4828b710c3ef719f4d66adec5f66cd5b7d05d1
2021-06-18 15:23:25 -07:00
Thomas Orozco
97c598ac82 fixup build after os_info crate update
Summary:
This dep got updated in D29165283 (b82c5672fc) across a major version but the code depending
on it wasn't so now it's broken.

Reviewed By: mitrandir77

Differential Revision: D29229087

fbshipit-source-id: 5f2a14dd9f0447dd4578e8321991dfb3df32dcc2
2021-06-18 07:06:14 -07:00
Jun Wu
4567de2ca6 helptext: remove reference of color.pagermode
Summary:
Follow up of D29172395 (6ed616db0c). Fixes `test-check-help.t`:

    --- test-check-help.t
    +++ test-check-help.t.err
    @@ -34,3 +34,6 @@
     to print errors.
       $ grep -v subversion $TESTTMP/topics > $TESTTMP/topics_filtered
       $ cat $TESTTMP/topics_filtered | xargs -n1 -P $NPROC hg --cwd / help >/dev/null
    +  abort: help section not found: config.color.pagermode
    +  xargs: hg: exited with status 255; aborting
    +  [124]

Reviewed By: DurhamG

Differential Revision: D29216316

fbshipit-source-id: 0df4e06b4f40de4bd2baaf47ad343f0b476ae405
2021-06-17 16:45:21 -07:00
Davide Cavalca
b82c5672fc Update several rust crate versions
Summary: Update versions for several of the crates we depend on.

Reviewed By: danobi

Differential Revision: D29165283

fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
2021-06-17 16:38:19 -07:00
Jun Wu
02a0d053e0 clone: add a way to stage rollout lazy-changelog for fbclone
Summary: Check the dest repo config to enable lazy-changelog.

Reviewed By: DurhamG

Differential Revision: D29123244

fbshipit-source-id: 5bae797fb43e94856296e62bbf6a17d1257b5f6e
2021-06-17 11:29:53 -07:00
Jun Wu
6ed616db0c pager: update helptext
Summary: Update helptext about streampager.

Reviewed By: DurhamG

Differential Revision: D29172395

fbshipit-source-id: 0aa0bd9c58f5bab3fa3d06495caf9cb3404f5a08
2021-06-17 09:33:40 -07:00
Jun Wu
2007761538 pager: make "internal:streampager" the default
Summary: It was rolled out weeks ago and there seem to be no complaints so far.

Reviewed By: DurhamG

Differential Revision: D29172396

fbshipit-source-id: 4f3310597ef1bb1abd92ee7207700a3d0039c598
2021-06-17 09:33:40 -07:00
Liubov Dmitrieva
1b818d114d add an option to pass some metadata in the token
Summary:
add an option to pass some metadata in the token

This will be used for content tokens, for example. We would like to guarantee that the specific content has been uploaded and it had the specific length. This will be used for hg filenodes upload.

Reviewed By: markbt

Differential Revision: D29136295

fbshipit-source-id: 2fbd3917ee0a55f43216351fdbc1a6686eb80176
2021-06-17 08:22:33 -07:00
Liubov Dmitrieva
98f863b323 use commitknown api for checking existing commits
Summary:
use `commitknown` edenapi api for checking the existing commits

it uses the same `lookup_commits` under the hood but a bit shorter to use

we won't need the tokens for existing changesets, so can use a simpler api

also, make `lookupfilenodes` function a bit shorter

Reviewed By: markbt

Differential Revision: D29134677

fbshipit-source-id: 257624d64480102c34761560b2bd768049cbfa83
2021-06-17 08:22:33 -07:00
Mark Juggurnauth-Thomas
759d6fc5b7 commitcloud: remove syncing of obsmarkers
Summary: Obsolescence markers have been deprecated in favour of mutation and visibility for some time.  Remove syncing of obsmarkers via the commit cloud service.

Reviewed By: liubov-dmitrieva

Differential Revision: D29159443

fbshipit-source-id: 33e1d526a9df5c02059931c880529d695036c616
2021-06-17 06:09:02 -07:00
Jun Wu
d12a1fe872 test-inherit-mode: stabilize the test
Summary:
In some envs the mode 02775 gets printed, and some other envs have 00775:

    --- test-inherit-mode.t
    +++ test-inherit-mode.t.err
    @@ -85,7 +85,7 @@
       006?0 ./.hg/store/00changelog.i (glob)
       00664 ./.hg/store/00changelog.len
       00660 ./.hg/store/00manifest.i
    -  02775 ./.hg/store/allheads/
    +  00775 ./.hg/store/allheads/
       00664 ./.hg/store/allheads/index2-node
       00664 ./.hg/store/allheads/log
       00664 ./.hg/store/allheads/meta

Allow both in the test.

Reviewed By: singhsrb

Differential Revision: D29185622

fbshipit-source-id: 40995c3941a88dbae7ad484c1a3abf9cded40bab
2021-06-16 23:33:42 -07:00
Jun Wu
d0d8e61daf setup: check Python.h existence
Summary:
Non-buck build cannot succeed if Python.h is missing.
Let's check it explicitly so we don't pick a bad Python.

Reviewed By: DurhamG

Differential Revision: D29179295

fbshipit-source-id: 421c824053d066914a6611f05815527768f257ee
2021-06-16 15:08:52 -07:00
Jun Wu
960cdcc4b9 setup: pick the right Python more aggressively
Summary:
It turns out that `python3` might be a symlink to a fbcode Python that cannot
perform the build in some (not all) environment. Let's try more names like
`python3.8`, `python3.7`, etc.

Reviewed By: DurhamG

Differential Revision: D29178933

fbshipit-source-id: da6cae351f25a90ab8a9da85282d09f79505c5e7
2021-06-16 15:07:23 -07:00
Jun Wu
ee1939e4d1 setup: downgrade IPython to 5.8
Summary:
IPython 7 has compatibility issues when bundling into zip:
- The dependency parso does not work when embedded in zip. It expects
  `python/grammar38.txt` to exist on the filesystem (not in a zip).
- It stacktraces after exiting the IPython shell.
- TAB completion does not really work.

Downgrading to IPython 5 and avoid the bundling the parso dep to resolve the
issues.

This should fix `test-autofix.t` and `test-argspans.py` failures.

Reviewed By: DurhamG

Differential Revision: D29170653

fbshipit-source-id: 14a3d16deaca72fbfb7b3acc0a4246a771c4d0aa
2021-06-16 11:51:11 -07:00
Andrey Chursin
11f0f9152d dag: separate out AbstractNameDag::{verify_missing, reload, persist}
Summary: They will be reused in import_pull_data

Reviewed By: quark-zju

Differential Revision: D29147950

fbshipit-source-id: 192bf33c30067f43c4fcaaf3054741b39efb4e25
2021-06-16 11:07:50 -07:00
Andrey Chursin
2de68ff1fd dag: introduce DagImportPullData
Summary: This is an interface for importing pull data into dag

Reviewed By: quark-zju

Differential Revision: D29142979

fbshipit-source-id: b40b94403a044c0b74d1574528aa374ec309a0cf
2021-06-16 11:07:50 -07:00
Andrey Chursin
2d76e5ce61 commits: introduce AppendCommits::import_pull_data
Summary: This will be used to import pull data into segmented changelog

Reviewed By: quark-zju

Differential Revision: D29142981

fbshipit-source-id: 2d19a035ee0b6cefef8fc0547a5dfb79f284a1de
2021-06-16 11:07:49 -07:00
Jun Wu
f7fde44264 setup3: remove duplicated deps
Summary:
It causes warnings:

  /usr/lib64/python3.6/zipfile.py:1378: UserWarning: Duplicate name: 'toml/__init__.pyc'
    return self._open_to_write(zinfo, force_zip64=force_zip64)
  /usr/lib64/python3.6/zipfile.py:1378: UserWarning: Duplicate name: 'toml/encoder.pyc'
    return self._open_to_write(zinfo, force_zip64=force_zip64)
  /usr/lib64/python3.6/zipfile.py:1378: UserWarning: Duplicate name: 'toml/decoder.pyc'
    return self._open_to_write(zinfo, force_zip64=force_zip64)
  /usr/lib64/python3.6/zipfile.py:1378: UserWarning: Duplicate name: 'toml/tz.pyc'
    return self._open_to_write(zinfo, force_zip64=force_zip64)
  /usr/lib64/python3.6/zipfile.py:1378: UserWarning: Duplicate name: 'toml/ordered.pyc'
    return self._open_to_write(zinfo, force_zip64=force_zip64)
  /usr/lib64/python3.6/zipfile.py:1378: UserWarning: Duplicate name: 'six.pyc'
    return self._open_to_write(zinfo, force_zip64=force_zip64)
  /usr/lib64/python3.6/zipfile.py:1378: UserWarning: Duplicate name: '__init__.pyc'
    return self._open_to_write(zinfo, force_zip64=force_zip64)

Reviewed By: andll

Differential Revision: D29107803

fbshipit-source-id: 0c46d739bf4f0c5e2faec835a29b3e0017f55ddd
2021-06-16 10:25:28 -07:00
Jun Wu
d0e16f1a25 setdiscovery: use native set types
Summary: Avoid converting to Python `set` for better performance.

Reviewed By: andll

Differential Revision: D29145340

fbshipit-source-id: 9c560f14404ac764184736bd09f5ea99f83cb7d5
2021-06-16 09:10:27 -07:00
Jun Wu
8afaafc486 changelog: remove code dealing with cl.userust() is False
Summary:
`cl.userust()` is always True after D29020191 (3765f8bd76) so `cl.userust() is False` code
is now dead.

Reviewed By: andll

Differential Revision: D29142464

fbshipit-source-id: f9a7e5c56641218758f12bad3de43d1cd1a71716
2021-06-16 09:10:27 -07:00
Meyer Jacobs
94be2053d5 tests: remove pack file-only tests
Summary: Pack files are no longer supported, yet we still have many tests which exercise them. In preparation for landing `scmstore` as a drop-in replacement for ContentStore, I'm removing our tests which only exist to test datapack-specific functionality.

Reviewed By: DurhamG

Differential Revision: D29099012

fbshipit-source-id: 635a913ee0d93ed8d536e71f8fa6a600b823b343
2021-06-15 19:00:58 -07:00
Andres Suarez
0f273c5ded update globset from 0.4.5 to 0.4.7
Summary:
The only real change here is: https://github.com/BurntSushi/ripgrep/pull/1756
This is a patch release but fixes a very glaring bug that others have
depended on. This diff fixes the uses to match the old behavior.

Although it's billed as a "fix", it's actually a huge perf improvement
for Linttool, which uses predominantly recursive suffix globs. The fact
that we don't have to compile ~5,000 regexps at Linttool startup anymore
makes such a huge difference that I am going to do write up soon.

Reviewed By: ndmitchell

Differential Revision: D29085977

fbshipit-source-id: 304470e5fa8cb986738aa0d9dd941641684a9194
2021-06-15 15:47:49 -07:00
Durham Goode
0e8bbd2e51 py3: remove py3 options and py2 rpm spec
Summary:
Python 2 is no longer built anywhere. Let's make the various py3
options the default, like renaming 'make local3' to 'make local' and let's get
rid of the dead setup.py and rpm spec.

Reviewed By: quark-zju

Differential Revision: D29077093

fbshipit-source-id: 0c50c2296fe10ff1db9ac8f9b0df2a4836c0ea5b
2021-06-15 14:46:42 -07:00
Andrey Chursin
c8d63eff2f async: replace block_on_future with block_on
Summary:
For a while we had two methods in async runtime: block_on_future and block_on_exclusive, due to historic reasons

Recently those methods were calling same code, and now it is time to replace both of them and rename to block_on

Reviewed By: quark-zju

Differential Revision: D29121107

fbshipit-source-id: 5faa76ae181e491b55d799c23c9de1b4e80298f3
2021-06-15 14:08:13 -07:00
Thomas Orozco
28e474eb82 hg: set ignoreautobackup on hg cloud backup
Summary:
If this opens a transaction and fails, then we start a never-ending cycle of
starting hg cloud backup and failing.

This command normally doesn't open transactions but it might via other
extensions, here is one example: P423007962.

This ends up killing people's machines:

https://fb.workplace.com/groups/scm/permalink/3957513017631622/

This fixes that by not kicking off background backups in this case.

Reviewed By: liubov-dmitrieva

Differential Revision: D29136077

fbshipit-source-id: 1c4e6de6147571dd6d728f761324506b1804f303
2021-06-15 12:54:40 -07:00
Jun Wu
be06ac3ada hgcommands: add debugdumpdynamicconfig for no-repo use-cases
Summary:
The added command gives access to read (execute) dynamicconfig without
using an on-disk repo.

It can be used by the clone script to stage rollout lazy changelog, or just to
verify dynamic config changes without using a repo.

Reviewed By: DurhamG

Differential Revision: D29123072

fbshipit-source-id: e8856d816a636fa860bfcc9694306a4a37552523
2021-06-15 10:48:01 -07:00
Liubov Dmitrieva
99a2b85f1a upload file content
Summary:
implement uploading file content via Eden API

* in this diff I aim to upload file content for the given set of filenodes
* also, the code would check with Mononoke using the lookup request what is already there and skip those
* also, this diff introduces calculation of blake2 hash (called ContentId) for file contents (we would probably need to store/cache those and the mapping from hg filenode id to the canonical Mononoke content_id)
* for every uploaded content EdenApi returns a token that we would also need to store later

Reviewed By: markbt

Differential Revision: D29063229

fbshipit-source-id: 739a44bc3ff904cb04a39514ba5efd01c80ba6d0
2021-06-15 08:09:00 -07:00
Andrey Chursin
41c476a200 pyedenapi: add pullfastforwardmaster binding
Reviewed By: quark-zju

Differential Revision: D29118724

fbshipit-source-id: f6083244dffe2cab050330e1213c98e144d278a7
2021-06-14 21:11:15 -07:00
Andrey Chursin
3cf926bcfe eagerepo: impl pull_fast_forward_master for EagerRepo
Reviewed By: quark-zju

Differential Revision: D29113217

fbshipit-source-id: f7c84d450b6c029c251474bfe1be86767979a78b
2021-06-14 21:11:15 -07:00
Andrey Chursin
6e4d6a5671 dag: pull_fast_forward_master implementation for AbstractNameDag
Summary: This will be used in eager repo integration tests

Reviewed By: quark-zju

Differential Revision: D29113218

fbshipit-source-id: a24232bd6c19010d8ac90d1305f57f1094b06323
2021-06-14 21:11:15 -07:00
Jun Wu
880b5c3cd8 mutation: avoid expensive server lookups for lazy changelog
Summary:
Similar to D29111710. Let's avoid asking servers for unknown nodes.

In theory this might affect correctness. Practically, this should only affect "landed as" markers,
because all drafts should be non-lazy. If the "landed as" correctness is an issue, we can fix
forward "landed as" later (ex. by writing down the public commit hash explicitly in
debugmarklanded).

Reviewed By: andll

Differential Revision: D29114049

fbshipit-source-id: db8dc34244feb66919cdff9433b6f18967c9ea9b
2021-06-14 16:00:58 -07:00
Jun Wu
f555f9c55d pydag: add a way to test commit existence without asking remote server
Summary: This will be used by upcoming changes.

Reviewed By: andll

Differential Revision: D29114048

fbshipit-source-id: ce963a42ebc5722ca9c61ab0a4e7a7377bbf7abb
2021-06-14 16:00:58 -07:00
Jun Wu
ac6c6cf3fa mutationstore: avoid excessive server lookups for lazy dag
Summary:
The mutationstore might contain lots of names (commit hashes) that are unknown
to the repo. When using a lazy changelog, by default, those names will be
looked up remotely, and most of the time the server just returns "name is outside
the graph".

We don't have negative cache to speed it up, because cache invalidation is
tricky - every lazy pull would in theory invalidate them.

To make things faster, let's just skip names in the mutationstore that are unknown
_locally_ without asking the server.

In theory this might affect correctness. Practically, this should only affect "landed as" markers,
because all drafts should be non-lazy. If the "landed as" correctness is an issue, we can fix
forward "landed as" later (ex. by writing down the public commit hash explicitly in
debugmarklanded).

Reviewed By: andll

Differential Revision: D29111710

fbshipit-source-id: 2c1b16a8140ca4f7195bf1206ffe1db4750185b9
2021-06-14 16:00:58 -07:00
Jun Wu
1e2b3cf87a configmodel: impl Config on BTreeMap<String, String>
Summary: This allows BTreeMap to be used as a provide of config for testing.

Reviewed By: andll

Differential Revision: D28019023

fbshipit-source-id: 94a1585139d6c6612fa163f2ff1aecc909db3a72
2021-06-13 09:19:22 -07:00
Pedro Rittner
b91da1d115 Bump crossbeam from 0.7 to 0.8, lsp from 0.3 to 0.5, lsp-types from 0.73 to 0.89
Summary: Bumping the crossbeam version so we can use `recv_deadline`. This also necessitates updating the lsp and lsp-types crates.

Reviewed By: alunyov, dtolnay

Differential Revision: D29056473

fbshipit-source-id: 9434e9e0895d82482f4c70afa01a2f77702b965f
2021-06-11 21:57:21 -07:00
Meyer Jacobs
8e79a2ce47 scmstore: add support for writing LFS pointers directly via FileStore
Summary:
This functionality is exercised in `test-lfs-copytracing.t` and a few other tests, though I'm not sure if it's necessary outside of that. I've added a `TODO` to investigate and try to eliminate such cases.

Because we'd rather not support this, I implemented it on top of ContentStore - it looks like we might have some lingering datapack-related tests where I think this comes up, too.

Reviewed By: DurhamG

Differential Revision: D29056647

fbshipit-source-id: e6bd5ec31dde06d8c509665f738cfe19270ddc76
2021-06-11 18:53:11 -07:00
Meyer Jacobs
ee1b174558 scmstore: update scmstore ExtractInnerRef types
Summary:
Modify the `ExtractInnerRef` implementations for `filescmstore` and `treescmstore` to return the new scmstore objects rather than the deprecated ones.

Modify `pycheckout` to use a new methods, `get_oldscmstore`, to access the deprecated scmstore instead (native checkout will be updated to use the new scmstore later on).

Reviewed By: DurhamG

Differential Revision: D29056498

fbshipit-source-id: 09b4655941bca048b01e6d365fcca1ccfb1cd700
2021-06-11 18:53:11 -07:00
Meyer Jacobs
41bc3699f8 scmstore: implement LFS upload for FileStore via legacy codepath
Summary: Temporarily implement `upload` on `FileStore` by forwarding to the existing `upload` implementation using `LfsRemote`. Eventually we'll want to re-implement this entirely inside `FileStore`.

Reviewed By: DurhamG

Differential Revision: D29045264

fbshipit-source-id: 8ebbaead4c1032949e5bdd1dcf9a733d0086a77c
2021-06-11 18:53:11 -07:00
Meyer Jacobs
5dfcaeefbd scmstore: add contentstore Python methods to filescmstore and treescmstore
Summary: Implement `ContentStore`'s Python methods on `filescmstore` and `treescmstore`.

Reviewed By: DurhamG

Differential Revision: D29042603

fbshipit-source-id: e077d7a2efc7780a384132703a2a1dc983612194
2021-06-11 18:53:10 -07:00
Meyer Jacobs
062fc5be4b scmstore: implement get_shared_mutable for TreeStore for compatibility with ContentStore
Summary: Add a `get_shared_mutable` method to `TreeStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).

Reviewed By: DurhamG

Differential Revision: D29042716

fbshipit-source-id: f2236ff8b47ef213b2ffc61501ca301da02dc492
2021-06-11 18:53:10 -07:00
Meyer Jacobs
7bf2eccf87 scmstore: implement get_shared_mutable for FileStore for compatibility with ContentStore
Summary: Add a `get_shared_mutable` method to `FileStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).

Reviewed By: DurhamG

Differential Revision: D29042485

fbshipit-source-id: e28af0bafac5eba87523b0ef522f32355106b467
2021-06-11 18:53:10 -07:00
Meyer Jacobs
47bbb75678 scmstore: track remote fetches in FileStore with FetchLogger
Summary: Add `FetchLogger` to `FileStore` to track remote fetches which match `remotefilelog.undesiredfileregex`.

Reviewed By: DurhamG

Differential Revision: D29042243

fbshipit-source-id: 08ec69d979d66ef93e8bbe8659171a412ade7e22
2021-06-11 18:53:10 -07:00
Meyer Jacobs
8be436187f scmstore: extract out FetchLogger from ReportingRemoteDataStore for use by scmstore
Summary: `ReportingRemoteDataStore` wraps a store and records all fetches which match `remotefilelog.undesiredfileregex`. Since scmstore doesn't use a hierarchical store composition, this change extracts out the matching/reporting logic into `FetchLogger` so that it can be re-used by both `ReportingRemoteDataStore` and scmstore.

Reviewed By: DurhamG

Differential Revision: D29041488

fbshipit-source-id: db0c4b545886ffebcf2a2841a506d301b2f2f230
2021-06-11 18:53:10 -07:00
Jun Wu
f1411b154f setup: add IPython deps for Python 3
Summary:
The files are listed by:

    python3 -m pip download IPython pip

Reviewed By: DurhamG

Differential Revision: D29072480

fbshipit-source-id: 96be7142bee4cc9ee67cbdea35a3e1e91170dbe7
2021-06-11 16:20:51 -07:00
Meyer Jacobs
a292d63d7f scmstore: update debugscmstore to use new scmstore
Summary: Straightforward update of `debugscmstore` from old scmstore to new scmstore. I'll want to improve this command a bit more in the future, but this at least enables you to easily test arbitrary fetches.

Reviewed By: DurhamG

Differential Revision: D29047527

fbshipit-source-id: 4e10cb88cba4b572d3e413640ca3d800940d675d
2021-06-11 12:40:08 -07:00
Durham Goode
63884d65c1 rotatelog: add fix cleaning up partially cleaned up logs
Summary:
Our log deletion is a two step process. First it deletes the meta file,
then it deletes the log. There was a bug where if the meta file deletion
succeeded but the log deletion failed (often because Eden has the log open) then
future attempts to delete the log will fail because it fails to delete the meta
file.

Reviewed By: quark-zju

Differential Revision: D29035913

fbshipit-source-id: 33ab9a4e46b285819c4bf14031426983d2b4342f
2021-06-11 11:36:05 -07:00
Durham Goode
60001b5161 rotatelog: add debug logging for rotatelog cleanup
Summary:
Since all the errors are eaten, we don't have good insight into the
rotatelog cleanup process. Windows users were seeing large growth in their
hgcache and this logging would've been useful, so let's add it.

Reviewed By: quark-zju

Differential Revision: D29035914

fbshipit-source-id: 54f0f05aa8bed1911b6b95438cd2d3ccc45e8ec9
2021-06-11 11:36:05 -07:00
Durham Goode
7a10894540 py3: fix crecord for non-utf8 files
Summary:
The crecord width calculation used a function that requires a string,
so we tried to force the rows from bytes to columns, which crashed if the row
wasn't utf8. Let's just use the error="replace" utf8 error handler since we
don't care if the row computation is slightly off.

Reviewed By: quark-zju

Differential Revision: D29043385

fbshipit-source-id: a00d867bca9c62007ae4f68b6198d159deeb6016
2021-06-11 11:33:15 -07:00
Andrey Chursin
c82b8867f6 dag: introduce PreparedFlatSegments::parents_and_head
Summary: This method will be used to get all Ids that needs to be included into CloneData::id_map during fast forward pull.

Reviewed By: quark-zju

Differential Revision: D29045538

fbshipit-source-id: f6eb04f537aa5365f7588391ea4c60b3dc010a53
2021-06-11 11:26:54 -07:00