Summary: This allows us to be more flexible in choosing authentication and expands variables used in configuration.
Reviewed By: singhsrb
Differential Revision: D25304008
fbshipit-source-id: 636893a9eaec31ca5acfa02f72931d5e56b695d0
Summary:
Update pywatchman/bser.c to define `PY_SSIZE_T_CLEAN` before including
`Python.h` and then receive parsed string lenghts as `Py_ssize_t` rather than
`int`. This eliminates some runtime Python warnings about the old `int`
behavior being deprecated.
Reviewed By: chadaustin
Differential Revision: D25294778
fbshipit-source-id: 7db678ed918db3bf4745ba716585ed6475d1c910
Summary:
The `repo.names` namespace lookup should not return `[None]`. Detect and avoid
that to avoid crashes.
Reviewed By: singhsrb
Differential Revision: D25261462
fbshipit-source-id: c7a0bba557717ab292a05d753569c68112e7f0b0
Summary:
Introduce `edenapithin` crate, which offers C bindings to EdenAPI Client.
There are two top-level `owned` types, `EdenApiClient` and `TreeEntryFetch`, which represent `Box`ed results from calls to EdenAPI's blocking client. The C / C++ code is responsible for calling the associated `free` functions for these types, as well as `OwnedString`, which is used to represent error variants of a `Result` as a string.
Most functionality is provided through functions which operate on and return references into these top-level owned types, providing access into Rust `Result`s and `Vec`s (manually-monomorphized), and EdenApi's `TreeEntry` and `TreeChildEntry`.
Additional non-pointer types are defined in the `types` module, which do not require manual memory management.
C++ bindings are not included currently, but will be introduced soon.
Reviewed By: fanzeyi
Differential Revision: D24866065
fbshipit-source-id: bb15127b84cdbc6487b2d0e1798f37ef62e5b32d
Summary:
Introduce a new API type, `TreeAttributes`, corresponding to the existing type `WireTreeAttributesRequest`, which exposes which optional attributes are available for fetching. An `Option<TreeAttributes>` parameter is added to the `trees` API, and if set to `None`, the client will make a request with TreeAttributes::default().
The Python bindings accept a dictionary for the attributes parameter, and any fields present will overwrite the default settings from TreeAttributes::default(). Unrecognized attributes will be silently ignored.
Reviewed By: kulshrax
Differential Revision: D25041255
fbshipit-source-id: 5c581c20aac06eeb0428fff42bfd93f6aecbb629
Summary: Adding `full_idmap_clone` to edenapi and usign that in `debugsegmentclone`.
Reviewed By: quark-zju
Differential Revision: D25139730
fbshipit-source-id: 682055f7c30a94a941acd16e2b8e61b9ea1d0aef
Summary:
This method reconstructs a dag from clone data.
At the moment we only have a clone data construction method in Mononoke. It's
the Dags job to construct and import the clone_data. We'll consolidate that at
a later time.
Reviewed By: quark-zju
Differential Revision: D24954823
fbshipit-source-id: fe92179ec80f71234fc8f1cf7709f5104aabb4fb
Summary:
The server expects post requests. At this time we don't want to cache this data
so POST.
Reviewed By: singhsrb
Differential Revision: D24954824
fbshipit-source-id: 433672189ad97100eee7679894a894ab1d8cff7b
Summary:
The config is something that makes sense for all commands to have access to.
Commands that don't use a repo don't have access to the config that is prepared
by the dispatches. This change is a stop-gap to allow new commands that don't
require a repository to receive the config as an argument.
The construction of the config is something that we should iterate on. I see
the current implementaiton as a workaround.
Reviewed By: quark-zju
Differential Revision: D24954822
fbshipit-source-id: 42254bb201ba8838e7cc107394e8fab53a1a95c7
Summary:
On Python 3 we only support utf8 files. Python 3 has a way of
representing non-utf8 strings in utf8 format by utilizing surrogateescape, but
these strings cause issues in other parts of the codebase that don't expect
suorrageescaped strings (like the Rust code). Since we simply do not support
these paths, let's filter them out as soon as we get them from Watchman.
Reviewed By: quark-zju
Differential Revision: D25134079
fbshipit-source-id: 8be6893a6114b816097422f4469ac317fa3795d1
Summary: As HG<->Mononoke will go through proxygen, testing showed that Proxygen forces us to use 'Upgrade: websocket' and confirm with the Websocket handshake. Adjust accordingly to do so.
Reviewed By: ahornby
Differential Revision: D25197395
fbshipit-source-id: ca00ac31be92817c6f1a99d7d492469b17b46286
Summary: editmergeps.bat was separating a filename with spaces into args[0] and args[1] when calling editmergeps.ps1. Use proper quoting to send files with spaces as a single argument.
Reviewed By: ikostia
Differential Revision: D25194324
fbshipit-source-id: 065f677c9015681b310e1cfc46f52ff563a35f99
Summary:
While trying to repro a user report (https://fburl.com/jqvm320o), I ran into a
new hg error: P151186623, i.e.:
```
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("fbcode/thrift/facebook/test/py/TARGETS"), hgid: HgId("55713728544d5955703d604299d77bb1ed50c62d") })'
```
After some investigation (and adding a lot of prints), I noticed that this
was trying to query the EdenAPI server for this filenode. That request should
succeed, given Mononoke knows about this filenode:
```
[torozco@devbig051]~/fbcode % mononoke_exec mononoke_admin fbsource --use-mysql-client filenodes by-id fbcode/thrift/facebook/test/py/TARGETS 55713728544d5955703d604299d77bb1ed50c62d
mononoke_exec: Using config stage prod (set MONONOKE_EXEC_STAGE= to override)
I1126 08:10:02.089614 3697890 [main] eden/mononoke/cmdlib/src/args/mod.rs:1097] using repo "fbsource" repoid RepositoryId(2100)
I1126 08:10:02.995172 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:137] Filenode: HgFileNodeId(HgNodeHash(Sha1(55713728544d5955703d604299d77bb1ed50c62d)))
I1126 08:10:02.995282 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:138] -- path: FilePath(MPath("fbcode/thrift/facebook/test/py/TARGETS"))
I1126 08:10:02.995341 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:139] -- p1: Some(HgFileNodeId(HgNodeHash(Sha1(ccb76adc7db0fc4a395be066fe5464873cdf57e7))))
I1126 08:10:02.995405 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:140] -- p2: None
I1126 08:10:02.995449 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:141] -- copyfrom: None
I1126 08:10:02.995486 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:142] -- linknode: HgChangesetId(HgNodeHash(Sha1(dfe46f7d6cd8bc9b03af8870aca521b1801126f0)))
```
Turns out, the success rate for this method is actually 0% — out of thousands of requests,
not a single one succeeded :(
https://fburl.com/scuba/edenapi_server/cma3c3j0
The root cause here is that the server side is not properly deserializing
requests (actually finding that was a problem of its own, I filed T80406893 for this).
If you manage to get it to print the errors, it says:
```
{"message":"Deserialization failed: missing field `path`","request_id":"f97e2c7c-a432-4696-9a4e-538ed0db0418"}
```
The reason for this is that the server side tries to deserialize the request
as if it was a `WireHistoryRequest`, but actually it's a `HistoryRequest`, so
all the fields have different names (we use numbers in `WireHistoryRequest`).
This diff fixes that. I also introduced a helper method to make this a little
less footgun-y and double-checked the other callsites. There is one callsite
right now that looks like it might be broken (the commit one), but I couldn't
find the client side interface for this (I'm guessing it's not implemented
yet), so for now I left it as-is.
Reviewed By: StanislavGlebik
Differential Revision: D25187639
fbshipit-source-id: fa993579666dda762c0d71ccb56a646c20aee606
Summary:
The write & read path use different back up state files, which means they can
go out of sync. If that happens, it's a bit annoying because:
- The output of `hg cloud check` is out of sync with that of `hg sl`
- `hg cloud backup -r` says the commit is backedup, and `hg cloud check -r`
says it isn't.
This diff fixes this by just using the `backedup()` revset, which under the
hood reads all state files.
Reviewed By: liubov-dmitrieva
Differential Revision: D25186071
fbshipit-source-id: 0ae2d43227d65a3564b2f796218b55982d5cc467
Summary:
Right now we just get a "deadline exceeded" error, which isn't very amenable to
helping us understand why we timed out. Let's add more logging. Notably,
I'd like to understnad what we've actually received at this point, if anything,
and how long we waited, as I'm starting to suspect this issue doesn't have much
to do with HTTP.
See https://fb.workplace.com/groups/scm/permalink/3361972140519049/ for
more context.
Reviewed By: quark-zju
Differential Revision: D25128159
fbshipit-source-id: b45d3415526fdf21aa80b7eeed98ee9206fbfd12
Summary:
It turns out the hggit tests weren't passing in Python 3, despite us
having removed them from our py3-broken list. Woops. This fixes them and enables
the tests.
Reviewed By: sfilipco
Differential Revision: D25095189
fbshipit-source-id: acffca34b0d5defa7575ede60621ca2ce0a2afe4
Summary:
Back in March we forced all extras to be strings. It turns out hggit
needs to write binary extras since it stores some binary patches in extras.
To support that, let's encode commit extras using surrogateescaped. That will
allow hggit to put binary data in extras in a later diff.
Reviewed By: sfilipco
Differential Revision: D25095190
fbshipit-source-id: 330bf06b15fc435f70119490557b8c22341b6ed9
Summary:
With segmented changelog, rev can exceed f64 safe range
(Number.MAX_SAFE_INTEGER in Javascript, 9007199254740991, 0x1fffffffffffff).
If rev is used in JSON, then the JSON cannot be parsed with precise rev
information.
This diff adds a compatibility mode so template will map the out-of-range revs
to safe range, and the mapped revs can be detected, and mapped back to be
resolved correctly.
Reviewed By: sfilipco
Differential Revision: D25079001
fbshipit-source-id: 52a5a2c8345a76355b8bf81435034d90c7c31610
Summary:
The `getdesignatednodes` function returns a boolean indicating whether the requested nodes were actually fetched.
In the case of SSH, this is needed because the server may not support the `designatednodes` capability (only supported by Mononoke). If the fetch fails, Mercurial will fall back to an expensive complete tree fetch.
The HTTP version of this function accidentally omitted `return True`, which meant it implicitly returned `None`, which triggered the fallback path.
Reviewed By: sfilipco
Differential Revision: D25074067
fbshipit-source-id: 089d5382dd566db89ee732cdcb82762c8d43e21a
Summary:
Skip the need to go through SSH/Mercurial servers. Directly connect to Mononoke
via TLS with X509 certificates.
Reviewed By: markbt
Differential Revision: D24756947
fbshipit-source-id: a782c894956d6a6be055eaa78287c040ceb57da7
Summary:
We've seen some hangs with http 2 in lfs. Switching to http 1.1 seems
to fix it. Let's make this configurable so we can tweak this if we see it in
edenapi. For now we continue to default to http 2.
Reviewed By: krallin
Differential Revision: D24901201
fbshipit-source-id: 9806e2c37fa299e4bd381ebdcb17d00800408de3
Summary:
We had a bug: if two files were reverted and then we try to amend one of them
mercurial will actually amend both of them.
Looks like the problem was caused by "Prune files reverted by amend" block.
Previously this block was considering every file that was changed in a commit we
are about to amend and comparing with working copy. If a file is the same in a
commit we are about to amend and in the working copy then it will amended as
well.
This diff attempts to fix it by considering only files that were selected for
amending.
Reviewed By: DurhamG
Differential Revision: D24949727
fbshipit-source-id: cf6cb95af3f67ec769e8a58db3b829945133b830
Summary:
We have an edge case - if we reverted two files and then we try to amend only a
single one then both of them will be amended.
This diff adds a test for this and other edge-cases. The next diff will fix it
Reviewed By: DurhamG
Differential Revision: D24949726
fbshipit-source-id: c5c53de1d67f161efa8564f89127e61ac2f28ac9
Summary:
When we streamclone, we snapshot the revlogs under a lock, then we start
sending. That works fine, unless we have a file whose size changes during
the sending phase. This can happen if it's promoted from a single `.i` to a
`.i` and a `.d`.
When that happens, the clone fails (the client reports it received a bunch of
corrupted data because it starts interpreting parts of files as inputs). Since
the breakage is also confusing client side, I updated the server side to also
assert that it's sending what it thinks it's sending.
Reviewed By: DurhamG
Differential Revision: D24958885
fbshipit-source-id: a0349c651b7cb63ab27546adf9944e7fba63a95d
Summary:
The wirepack sending code builds up the entire history blob in memory
before sending it. Previously we did this by appending to the string. In Python
2 this was fast, in Python 3 this is n^2 and n can be 100k+ in cases of long
history.
Let's switch to list+join.
Reviewed By: xavierd
Differential Revision: D24933183
fbshipit-source-id: 5c36d7868e7c64a2292bd68ec2ffb584d85dd98f
Summary:
osxfuse is rebranding as macfuse in 4.x.
That has ripple effects through how the filesystem is mounted and shows up in
the system.
This commit adjusts for the new opaque and undocumented mount procedure and
speculatively updates a couple of other code locations that were sensitive to
looking for "osxfuse" when examining filesystems.
Reviewed By: genevievehelsel
Differential Revision: D24769826
fbshipit-source-id: dab81256a31702587b0683079806558e891bd1d2
Summary:
We got a [report](https://fb.workplace.com/groups/scm/permalink/3379140858802177/) that a new hg build fails with an error because it can't xor None types.
```
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr {
ptype: <type 'exceptions.TypeError'>, pvalue: Some("unsupported operand type(s)
for ^: 'NoneType' and 'NoneType'"), ptraceback: Some(<traceback object at
0x00000249BB158248>) }',
```
Full stack trace is here
{P149395441}
This seems likely to be related to the diff I landed recently - D24725902 (7b1798be37).
However it's unclear why it was affecting only windows because I couldn't repro
it on linux.
Turned out that we have experimental.treematcher option disabled on windows,
which causes it to use includematcher instead of treematcher. And includematcher
returns either None or BytesMatchObject and they are impossible to xor.
This diff fixes it by converting them value to bool first, and also it adds a
test for it.
Reviewed By: singhsrb
Differential Revision: D24918192
fbshipit-source-id: 1359e8b97d26d3b1a4795b7b3d4cfa3d6d4ae843
Summary:
We've seen http 2 potentially causing hangs for users. Let's make this
configurable for lfs, so we can disable it and see if things get fixed.
Reviewed By: krallin
Differential Revision: D24898322
fbshipit-source-id: dc7842c0247dc6b9590a1f160076b17788aab1b9
Summary:
As discussed in a group thread (see link below), HTTP 2 may be causing
hangs for users. Let's start by making the http-client configurable. In
subsequent diffs we'll make edenapi and lfs configurable as well.
Reviewed By: krallin
Differential Revision: D24898323
fbshipit-source-id: f0035a1b8df3cee626ebe519e9e99358c1b3f043
Summary:
This isn't code that compiles, but the convention in Rust is that code actually
is doctests unless annotated otherwise, so if tested with Cargo, those fail.
This fixes that.
Reviewed By: farnz
Differential Revision: D24917364
fbshipit-source-id: 62fe11700ce561c13dc5498e01d15894b17b5b22
Summary:
Thread Pool fails with py3 hg build. Let's replace with a loop.
Most of the usage for the command will be for a single head anyway.
Reviewed By: krallin
Differential Revision: D24902167
fbshipit-source-id: c7af46d0d63ddd074c98788bf55520ae3f2550b8
Summary:
Transfers iddag flat segments along with the head_id that should be use to
rebuild a full fledged IdDag. It also transfers idmap details. In the current
version it only transfers universal commit mappings.
Reviewed By: krallin
Differential Revision: D24808329
fbshipit-source-id: 4de9edcab56b54b901df1ca4be5985af2539ae05
Summary:
BE: remove old subscription to save resources in IceBreaker. The client code will recreate it anyway if missing but cleaning up will help us to reduce number of unused subscriptions.
Classic example: repo opsfiles or configerator maybe needed once and then a user don't use
Another example: switching workspaces failed and it could be result in subscriptions are not cleaned up properly
Reviewed By: markbt
Differential Revision: D24859931
fbshipit-source-id: 6df6c7e5f95859946726e04bce8bc8f3ac2d03df
Summary:
Those are the tweaks I've made to make `--config devel.bundle2.debug` more
verbose to aid with my investigation. This might help somebody else in the
future so let's comit it:
* added "params" decoding to debugsendbundle
* added "message" to `error:unsupportedcontent` part (we already send it with
some other error parts)
Reviewed By: sfilipco
Differential Revision: D24840405
fbshipit-source-id: b25d5823d05f3d50230c078e8db459dc66256707
Summary:
This function is useful in the mononoke to compute the universal commit idmap
that is required for clone.
Reviewed By: quark-zju
Differential Revision: D24808327
fbshipit-source-id: 0cccd59bd7982dd0bc024d5fc85fb5aa5eafb831
Summary:
`flat_segments` are going to be used to generate CloneData. These segments will
be sent to a client repository and are going to bootstrap the iddag.
Reviewed By: quark-zju
Differential Revision: D24808331
fbshipit-source-id: 00bf9723a43bb159cd98304c2c4c6583988d75aa
Summary: This is the object that will be used to bootstrap a Dag after a clone.
Reviewed By: quark-zju
Differential Revision: D24808328
fbshipit-source-id: 2c7e97c027c84a11e8716f2e288500474990169b
Summary:
The goal is to reused the functionality provided by AssignHeadOutcome for clone
purposes.
Reviewed By: quark-zju
Differential Revision: D24717924
fbshipit-source-id: e88f21ee0d8210e805e9d6896bc8992009bd7975
Summary:
The original problem was a fastlog bug, solved by D24513444 (c3bcc1ab88).
Restores prefetching for phabricator status so `hg ssl` and `hg fssl` become fast again.
Original commit changeset: b10c4caf8fda
Reviewed By: sfilipco
Differential Revision: D24749774
fbshipit-source-id: fa14f7dde9c922733525a7ff014efc32875426fa
Summary:
The original issue was a rust-cpython bug, solved by D24698226, or https://github.com/dgrunwald/rust-cpython/pull/244.
Original commit changeset: 08f598df0892
Reviewed By: sfilipco
Differential Revision: D24759765
fbshipit-source-id: f9a1359cfce68c8754ddd1bcb8bfc54bf75af7ff
Summary:
On Windows terminal, with light color schemes, crecord text was barely visible
(sometimes invisible) due to low contrast on either the background, or the
foreground. Making the text bold makes it brighter and thus more readable.
As a bonus, I've also made the hunk lines magenta to mimic what `hg diff` does.
Reviewed By: DurhamG
Differential Revision: D24718598
fbshipit-source-id: 18c2ff03fc2a46ca45808d5061db21e1f1b501ae
Summary: This makes it clean up stale files more aggressively.
Reviewed By: DurhamG
Differential Revision: D24744461
fbshipit-source-id: 76d163c9f16d8f8d1bf628e9197a3086d7cd48aa
Summary:
The goal of this code was to divide the cache limit by the number of
logs. Instead it divided the cache limit by the default per-log size (2GB). That
results in a very small max-bytes-per-log so data was being thrown out
constantly. This fixes it and updates tests to actually demonstrate the issue.
Reviewed By: kulshrax
Differential Revision: D24712842
fbshipit-source-id: 8062758b5bfa40493e2003d5a9028d601b1522b1
