Summary: Windows path like `eagerepo:///C:\foo\bar` needs special handling.
Reviewed By: kulshrax
Differential Revision: D27971119
fbshipit-source-id: 9d4b87782eca2734b708565f0ee22a7495253cff
Summary: `hg gc` does not do anything anymore, so in order to reduce confusion, let's just print a message that says it is no longer supported and provide a manual remediation.
Reviewed By: xavierd
Differential Revision: D28164614
fbshipit-source-id: 7ed2392cdb0091cd604a15b4c2382632706981f2
Summary:
This avoids issues where the tree is stored without p1, p2. It is similar to
what we do for public commits (in createtreepackpart):
if sendtrees == shallowbundle.AllTrees or ctx.phase() != phases.public:
...
Note: the trees API actually provides p1, p2, but p1, p2 are dropped when
writing to the current data store implementation.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200388
fbshipit-source-id: e1fe93d8ae8576e691077d6db432d55f7b9d498d
Summary: Add a way to fetch tree content without going through store.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200387
fbshipit-source-id: 8f5b2214aafba39c7674f0f6b27af0c985f0ea72
Summary:
The `trees` API is coupled with a store. We're going to add another API that is
not coupled with a store so let's rename `trees` to `storetrees`.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200389
fbshipit-source-id: 826116f0b461873b2f5df07e7fd35e6d1018f929
Summary:
This output is non-determenistic, and it does not seem to be important in this test
We could replace HashMap with BTreeMap to make it deterministic, as an alternative, but it is probably not justified for this test
Reviewed By: quark-zju
Differential Revision: D28204050
fbshipit-source-id: 50000671520e3bbf41849dc53c420ccab496dca0
Summary: the option has been deprecated and is not used anywhere
Reviewed By: krallin
Differential Revision: D28191314
fbshipit-source-id: f5f092b93a9644c8249628520d8d707b60854aac
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
This will consume `CloneData` from EdenApi and write to the graph.
Note `CloneData<Vertex>` and `CloneData<HgId>` has the same mincode
serialization result so there is no need to do extra type conversion.
This can be used like:
In [1]: v=api.clonedata('fbsource');
In [6]: d=bindings.dag.commits.openhybrid(None, '/tmp/seg', '/tmp/msg', repo
...: .edenapi, repo.name, lazyhash=True)
In [7]: d.importclonedata(v)
Reviewed By: kulshrax
Differential Revision: D27971125
fbshipit-source-id: 4d420c6ff0495dc184e7c9618b866a69f0a00002
Summary:
Expose NameDag's `import_clone_data` API so this can be then exposed via
`pydag`.
Reviewed By: kulshrax
Differential Revision: D27971118
fbshipit-source-id: c9d869ffbbc8ba5a7a6ae98d17a2b7ea713bc675
Summary: The `CloneData` is currently only exposed in Rust. Expose it in Python too.
Reviewed By: kulshrax
Differential Revision: D27971124
fbshipit-source-id: 1a9c63274b6e2d78a176869b6810acbc191ba314
Summary: We skip it in other places but missed this one. Skip it too.
Reviewed By: kulshrax
Differential Revision: D27957853
fbshipit-source-id: 429d25e8b692218c9bae6c10ad76d08495a4bc66
Summary: If ui.ssh is "false", then ssh cannot be used at all. Force using edenapi.
Reviewed By: kulshrax
Differential Revision: D27957312
fbshipit-source-id: 9860344779e6a6bab557d3f953ee38e40fadb78b
Summary: Make it easier to check whether APIs in EagerRepo is called or not.
Reviewed By: andll
Differential Revision: D27955426
fbshipit-source-id: 27ca505c63596368cff98642de010b5b5717454c
Summary: It has been enabled for a long time in our production config.
Reviewed By: kulshrax
Differential Revision: D27953636
fbshipit-source-id: 428f6e8a3e7eae6d44c61970624a75d7d1ab3e36
Summary: It has been enabled for a long time in our production config.
Reviewed By: kulshrax
Differential Revision: D27953635
fbshipit-source-id: a351342fbc8cffccd16967bd0e7032ac3e4f35cf
Summary:
Add "getbundle" alternative "commitgraph" for pulling from a EagerRepo.
This avoids tech-debt like bundle2 or linkrev. It depends on a lazy
(text) changelog backend.
Reviewed By: kulshrax
Differential Revision: D27951620
fbshipit-source-id: f21119d37da6505e68c6c5f3b33b9bd1f65e4d9a
Summary: It's not an error case. It just means all nodes are unknown to the repo.
Reviewed By: kulshrax
Differential Revision: D27951619
fbshipit-source-id: 672932af3a54ffa5adfa5cccbfff7edbf4f24022
Summary: It's okay to migrate to any backend if the repo is empty.
Reviewed By: kulshrax
Differential Revision: D27951626
fbshipit-source-id: 27c00c853bf73fa3c696d74f3c05eb620f35db0e
Summary:
Add "unbundle" alternative "addblobs" for pushing to a EagerRepo.
This avoids tech-debt like bundle2 and linkrev.
Reviewed By: kulshrax
Differential Revision: D27951628
fbshipit-source-id: 3315e0653ee12928993e4e9325fbe8e2c369307b
Summary:
Now EdenApi trait is moved to a separate crate, we can inline the EdenApi
backed by EagerRepo without using dynamic registration functions.
Reviewed By: andll
Differential Revision: D28006553
fbshipit-source-id: 427513da94db228745b1a7e90af0e62296056128
Summary: So that we don't duplicate the URL handling in Python.
Reviewed By: andll
Differential Revision: D28006552
fbshipit-source-id: 2efda622fe86787373fa4ec5978537588defec28
Summary:
`peer` is the interface in hg to support push/pull. Implement it for EagerRepo.
Note: `getbundle` and `unbundle` are not implemented yet. Push / Pull are not
working yet. They will be made work later.
Reviewed By: kulshrax
Differential Revision: D27951621
fbshipit-source-id: 71f9c26713a532a0712460fa2aa34125b2b67e35
Summary:
Our BigSur mactest machines have python3 defaulting to some internal
fbprojects python install. This is breaking our OSS builds. Let's change
pick_python to avoid that install.
Note, sys.stdout was changed to print because during my manual testing on Mac,
sys.stdout did not actually print the value, despite the flush. Using print()
did work.
Reviewed By: quark-zju
Differential Revision: D28101632
fbshipit-source-id: 2907d644b2baa8a53a9a2d7da176d33cd83dfbd5
Summary:
There have been a bunch of problems with the previous approach to scmstore, so I'm going to try to start simple, make it feature complete, and then add async integration and factor out generic functionality as appropriate.
This change contains a `TreeStore` implementation with a single, synchronous, batch read method (supporting local storage, memcache, and legacy fallback, with writing missing to cache).
Add `TreeStoreBuilder`, which duplicates the existing `TreeScmStoreBuilder`, which some changes that make it easier to use for this case. I intend to unify these in the future.
Add an inherent impl for `EdenApiTreeStore` that provides subset of the `BlockingEdenApi` trait, which eliminates the need to unpack this type into a different adapter as the old `scmstore` code does. This might not be the right approach here, in reality we only need a `(client: Arc<dyn EdenApi>, repo: String)` here for trees, and that + `ExtStoredPolicy` for files, so we could take the `EdenApiAdapter` approach here too. The only reason we have to do any of this is because when `pyrevisionstore` is called to construct `scmstore` / `contentstore`, all we have is `Arc<EdenApiTreeStore>`. We could also just make the `EdenApiRemoteStore` fields public, and access them through the `Arc`.
Add `add_mcdata` method to `MemcacheStore`, `impl TryFrom<Entry> for McData`, and `impl From<McData> for Entry` for convenience when working with `MemcacheStore` (so we don't need to manually unpack the type and build `Entry`, or manually build a fake `Delta` from `Entry` to write).
Reviewed By: DurhamG
Differential Revision: D28076900
fbshipit-source-id: 7fdb5e8a42d052879eff449f60d40a83cfa7145d
Summary:
Both `get_local_path` and `get_cache_path` take suffix as as `PathBuf`, even though they only ever use it as a reference. `get_local_path` also takes a reference to a `PathBuf`, even though it always clones it internally, and takes an `Option`, even though it just maps across the contents of the option.
I modified `get_local_path` to accept a `PathBuf` by move, which it uses directly, and to not take an `Option` (instead just calling `map` externally, removing some unnecessary unwraps), and for both functions to accept `impl AsRef<Path>` for suffix.
Reviewed By: DurhamG
Differential Revision: D28100527
fbshipit-source-id: df28b51c8005f3d95acc8e082b40adaab18e31c9
Summary: Add a Read/Write Guard API to IndexedLogHgIdDataStore which allows client code outside the module to perform a series of reads and writes without locking for each individually.
Reviewed By: kulshrax
Differential Revision: D28075788
fbshipit-source-id: 2a65a426f443e1a421198ad8b4c610e4822574f7
Summary:
Add get_entry, put_entry, and flush_log inherent methods to IndexedLogHgIdDataStore. Refactor callers to use them in cases where they don't lock across multiple reads / writes (to avoid performance regressions).
This should allow `ReadStore` and `WriteStore` to be moved out of the module.
Reviewed By: DurhamG
Differential Revision: D27979828
fbshipit-source-id: c9fb8c4ac68f67b285c72396509aa17928aa54ed
Summary: It has been wrong since 2014 (tweakdefault).
Reviewed By: kulshrax
Differential Revision: D28122703
fbshipit-source-id: c83ddbac2c6162e36672649c60c2e7916dc7cbd2
Summary: This is step towards unifying native merge/rebase structs with native checkout - we now construct native checkout plan from the action map, instead of directly making it from the diff
Reviewed By: quark-zju
Differential Revision: D28078156
fbshipit-source-id: 318d7e419ca9fef15a4aebf7494451f69a3bbbe5
Summary:
This diff makes concurrency of native checkout to be configurable
This config can be used to reduce concurrency on platforms that are known to cause issues with watchman due to too many checkout operations
Reviewed By: quark-zju
Differential Revision: D28074993
fbshipit-source-id: 0a09fcf3ae48d08cead36da56c06b546aecd16b4
Summary: This diff refactors out `Checkout` component from checkout plan and allows to configure parallelism in checkout
Reviewed By: quark-zju
Differential Revision: D28074994
fbshipit-source-id: 72933c757d6e27615d1ef2bb4652bc67c9c3253d
Summary:
From what I can see, this was added when EdenFS had a Mononoke store, which is
now long gone, thus we should be able to remove the Curl dependency altogether.
Reviewed By: fanzeyi
Differential Revision: D28037816
fbshipit-source-id: 834f7db64bab5dda1748ad2f033c27a2854b0ba4
Summary:
This updates hg to have a different amount of retry for backoffs requested by
the server and errors.
The rationale is that backoffs are fairly well understood and usually caused by
a surge in traffic where everybody wants the same data (in which case we should
be willing to wait to get it because there is literally no alternative),
whereas general errors aren't predictable in the same way.
We're now effectively at a point on the server side where _all_ our instances
have the exact same load, so if any server is telling you to backoff, that
pretty much guarantees that the whole tier has too much traffic to deal with.
This leaves us with two options:
- Tell clients to wait longer and smooth out the traffic surge.
- Add enough capacity that even our biggest surges don't result in _any_
throttling.
The latter is a bit unrealistic unrealistic given we routinely get egress
variations in excess of 5x (here's an example: https://fburl.com/ods/pidsrqnl),
so this does the former.
This also updates the client to tell the server how many attempts it has left
in addition to how many it used up so far. How many are left is more meaningful
for alerting!
Finally, it adds a bit of logging so that in debug mode you can see this
happening.
Reviewed By: quark-zju
Differential Revision: D28092797
fbshipit-source-id: f61410e39c4a3e3356371a3c7bd7892de4beacc8
Summary:
After D27144492 (48cd15ab14) we disabled revision number resolution. There is no need to
consider it when calculating shortest prefix.
Reviewed By: DurhamG
Differential Revision: D28072997
fbshipit-source-id: 832017c7b626265eb8cd2dd78946a03c4e7228f6
Summary:
The "known" API is expected to accept a list with ordering guarantees.
Avoid passing `set` to it. This also stabilizes debug outputs in (upcoming)
tests.
Update `setdiscovery` to pass a list with stable order too.
Reviewed By: kulshrax
Differential Revision: D27951627
fbshipit-source-id: a2718e4380f80584fc8afc35d14e524ab428f9bd
Summary: They will be useful in the pull exchange path.
Reviewed By: kulshrax
Differential Revision: D27951625
fbshipit-source-id: 31dfd54cda7a0d0d0b2565e0ecddc2e82eebcc05
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way.
Actual implementation is TBD.
Differential Revision: D27951634
fbshipit-source-id: 66210a833d8fc87452a19d1935e9d208a1d31b14
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way. Actual implementation is TBD.
I dropped the progress callback parameter since the native Rust progress bars
make it unnecessary. I also avoided the blocking API interface, since we would
like pure Rust code to not call blocking APIs which is likely a mistake, and
only use `async_runtime::block_on` in Python bindings - in that case blocking
API is not needed.
Reviewed By: andll
Differential Revision: D27951624
fbshipit-source-id: 3844dd96df265cc6e61d7cf5e79f39c891e8117d
Summary: This makes use of plan calculation logic in merge.py
Reviewed By: quark-zju
Differential Revision: D28045174
fbshipit-source-id: f2bfd1104ba2d010c2b911aa6c643b057e4cb91d
Summary: This factors out native checkout plan calculation (including handling sparse profiles) into separate fn
Reviewed By: quark-zju
Differential Revision: D28045175
fbshipit-source-id: cd034b2c335496e5e2c6994795dd5c40d3388238
Summary:
Add a way to construct the EagerRepo from Python, and use it to serve EdenApi.
Example in `hg dbsh`:
Prepare EagerRepo:
In [2]: r=b.eagerepo.EagerRepo.open('/tmp/r1')
In [3]: r.addsha1blob(b'1')
Out[3]: '5j\x19+y\x13\xb0LTWM\x18\xc2\x8dF\xe69T(\xab'
In [4]: r.getsha1blob(Out[3])
Out[4]: '1'
In [5]: r.addcommit?
Docstring:
Insert a commit. Return the commit hash.
(parents: [node], rawtext: bytes) -> node
Type: builtin_function_or_method
In [6]: c=r.addcommit([], 'commit 1')
In [7]: c
Out[7]: '\x8a\x8a\x0bq\x1e\xa4\x08\xd4\xa5\x14a\xc8\x95Y\xed\x8d$\xba1\x9a'
In [8]: r.flush()
Use it to serve EdenApi:
In [9]: e=r.edenapiclient()
In [10]: e.commitdata?
Docstring:
commitdata(repo: str, nodes: [bytes], progress=None) -> [(node: bytes, data: bytes)], stats
Fetch commit data in raw HG format (sorted([p1, p2]) + text).
Type: builtin_function_or_method
In [11]: e.commitdata('', [c])
Out[11]: (<stream at 0x7f96194bcc70>, <future at 0x7f96194bf810>)
In [12]: list(e.commitdata('', [c])[0])
Out[12]:
[{'hgid': '\x8a\x8a\x0bq\x1e\xa4\x08\xd4\xa5\x14a\xc8\x95Y\xed\x8d$\xba1\x9a',
'revlog_data': '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00commit 1'}]
Reviewed By: andll
Differential Revision: D27951623
fbshipit-source-id: c5423d55b6359146e5a416c440d85f817e43396d
Summary:
Add a way to construct the Python EdenApi wrapper from a native Rust EdenApi instance,
bypassing the config, etc.
Reviewed By: kulshrax
Differential Revision: D27951622
fbshipit-source-id: 2e3d16a91e20c6aab6baced3dea60db9834fa103
