Summary:
Fixes the below type error when pulling from a git repo using hg-git on my laptop:
File "edenscm/hgext/remotenames.py", line 225, in expull
pullremotenames(repo, remote, bookmarks)
File "edenscm/hgext/remotenames.py", line 314, in pullremotenames
path = activepath(repo.ui, remote)
File "edenscm/hgext/remotenames.py", line 1464, in activepath
rpath = _normalizeremote(remote.url)
File "edenscm/hgext/remotenames.py", line 1439, in _normalizeremote
u = util.url(remote)
File "edenscm/hgext/hggit/__init__.py", line 164, in _url
if not (path.startswith(pycompat.ossep) and ":" in path):
AttributeError: 'function' object has no attribute 'startswith'
Basically, `peer.url()` is the API, `peer._url` is a private field that does
not always exist somehow.
Besides, further remove named branches that can crash hg-git with
NotImplementedError:
File "edenscm/hgext/remotenames.py", line 225, in expull
pullremotenames(repo, remote, bookmarks)
File "edenscm/hgext/remotenames.py", line 322, in pullremotenames
for branch, nodes in remote.branchmap().iteritems():
File "edenscm/hgext/hggit/gitrepo.py", line 73, in branchmap
raise NotImplementedError
Reviewed By: DurhamG
Differential Revision: D14144462
fbshipit-source-id: 2e886c639cf6689480f84626eaf0d5ec25512ea0
Summary:
Apparently Gluster really can't copy with renames, so write to a
unique name, then symlink the canonical key to it. If the symlink already
exists, then we'll assume that the file already does, and remove the unique
name.
Reviewed By: StanislavGlebik
Differential Revision: D14014167
fbshipit-source-id: 1e5e2ce989652232d67d2aaac776e35127f58fb0
Summary: Keep track of error causes so we can print better errors.
Reviewed By: StanislavGlebik
Differential Revision: D14014165
fbshipit-source-id: e9a7846256bbfbfd689e0d78f01b1ac50cf64c1b
Summary:
Reduce directory fan out levels to 2 wide ones (xxx/xxx) rather than 4 narrow
(xx/xx/xx/xx). This saves write latency cost: with 4 levels, until we get
~billions of blobs, every write will also be a mkdir, and we'll have many
single-entry directories. With 2 wide levels this will be true up to ~millions of
blobs, but then start to fill out the leaf directories.
The downside is that the leaf directory loading will be higher once we do have
billions of objects, but I think it will be managable.
This is NOT INTEROPERABLE with 4-level fanout, so invalidates existing stores.
Reviewed By: aslpavel
Differential Revision: D13923991
fbshipit-source-id: b93ddc49305c921f8a1b34606796b6d75273fb75
Summary:
Adding `.data` to the end breaks the "rsync hack" to keep files colocated on
the same node.
Reviewed By: aslpavel
Differential Revision: D14014284
fbshipit-source-id: c65b237ed03c9c8ea3a6ac239135820379b7e38d
Summary: In this final diff for verify_integrity the data authorization data is being passed from hgcli (where the ssh connection is established) to Mononoke where it is further passed to verify_integrity script.
Reviewed By: StanislavGlebik
Differential Revision: D14387759
fbshipit-source-id: 2c0f9eef4128f5af0052276a1830ea2f449fb03a
Summary: The username and ssh vars are both properties of request, so they fit nicely inside per request CoreContext. They will be used in verify_integrity hook.
Reviewed By: StanislavGlebik
Differential Revision: D14385840
fbshipit-source-id: 9fe3cb96ffa89d8b017c730e37ca9ea4124ede0c
Summary:
This additional data that is send to Mononoke will be used e.g. in verify_integrity hook.
This diff also contains change to the user_unixname, instead of passing env variable USER there we pass value produced by users crate. Since the user_unixname is not used anywhere now it is safe to do so.
Reviewed By: StanislavGlebik
Differential Revision: D14385280
fbshipit-source-id: 1e48c232fafbba5d5188c7d162e9ef21efd738f7
Summary:
This stack is about adding getpackv1 wireproto request support (see task for
more details).
This diff adds a Decoder trait implementation. It parses all parameters for a
single file (i.e. filename + list of nodes). This Decoder is combined with
the previous diff to fully parse getpackv1 request
Reviewed By: lukaspiatkowski
Differential Revision: D14401516
fbshipit-source-id: 9b5fa1f48de338e58a288eb0653bd734cd8d1623
Summary:
The stack is about adding support for getpackv1 wireproto request.
This diff make decode_getfiles_arg_stream function generic over Decoder type.
At the moment decode_getfiles_arg_stream is used for unpacking `getfiles` wireproto
request. However getpackv1 packs request in a similar way to getfiles (which
is different from any other wireproto requests). Let's re-use the same
functionality for getpackv1
Reviewed By: lukaspiatkowski
Differential Revision: D14401517
fbshipit-source-id: ef0e9abeecec61c7c3d25b4a9e36da3f05c870cb
Summary:
Copy & rename sources must be included in the list of conflict files so that if
a copy was modified between root and `onto` bookmark then pushrebase should
fail with conflicts.
Note that some some merge cases are not handled yet - see TODO in the code
Reviewed By: lukaspiatkowski
Differential Revision: D14322036
fbshipit-source-id: d69bcceaa24987dd1e9d67e77f6a3205b580a7d8
Summary:
Mononoke does not support pushrebasing over a merge commit. Previously
`find_closest_ancestor_root` didn't detect merges.
In cases like
```
o <- onto
|
o o <- commit to pushrebase
|\ /
| o
o <- main branch
...
```
`find_closest_ancestor_root` would go to the main branch and finally fail with
`RootTooFarBehind` error. By detecting merge commit earlier we could print
better error message and avoid doing useless traversal
Reviewed By: lukaspiatkowski
Differential Revision: D14321616
fbshipit-source-id: 2aa53a2627f25897a241616a429864f1cfca3100
Summary:
The problem was in using `file_changes()` of a bonsai object. If a file
replaces a directory, then it just returns an added file, but not a removed
directory.
However `changed_entry_stream` didn't return an entry if just it's mode was changed (i.e. file became executable, or file became a symlink). This diff fixes it as well
Let's use the same computing changing files method instead of `file_changes()`.
Differential Revision: D14279470
fbshipit-source-id: 976b0abd93646f7d68137c83cb07a8564922ce17
Summary:
In integration tests, the sent replycaps part can be different lengths
depending on client capabilities. Usually this is globbed, but a couple
of places are missing.
Reviewed By: aslpavel
Differential Revision: D14363789
fbshipit-source-id: 5db161bd646c7a9fa8aeef2281ee7eb4d0c7771e
Summary: This endpoint is also used in Mercurial now.
Reviewed By: StanislavGlebik
Differential Revision: D14303557
fbshipit-source-id: fe38b62d010de2846dcf800f93ba050d9c396873
Summary: The phabricator message parsing capability will be also used from check_unittests Rust hook, so it had to be extracted and adjusted to Rust standards.
Reviewed By: StanislavGlebik
Differential Revision: D14301561
fbshipit-source-id: 47b59527dfadd7b761f750825da52ffce14fdf21
Summary:
There's a bug in pushrebase. When a directory is replaced with a file, then
changed file list is computed incorrectly. This diff adds a test that
highlights the issue, next diff fixes it.
Differential Revision: D14279472
fbshipit-source-id: dbb8738e4cd0b2c025060e91b5772662c296f342
Summary:
Make compute_changed_files accept manifest id, it will be used in the next
diffs
Differential Revision: D14279468
fbshipit-source-id: eca92900ed8862bb6db38ff6c5ed5372d8206aa9
Summary:
In the future we will be introducing mutation recording in commit extras. For
pushrebase, these fields need to be stripped when the commit is rebased. This
matches the Mercurial server behaviour.
At some point we will update Mononoke pushrebase to fill in these fields with
the pushrebase information, but that can wait until it is more widely deployed.
Reviewed By: quark-zju
Differential Revision: D14263856
fbshipit-source-id: 78b682fc580695137fd1fd4a4f65210a77f16b24
Summary: This diff adds a new `hg debuggethistory` command that takes filenode/path pairs from stdin, fetches the history of the files from the API server, and writes the results to a historypack in the hg cache.
Reviewed By: quark-zju
Differential Revision: D14248082
fbshipit-source-id: 8014a758abd3a578ea213d8d3177812629b2fd51
Summary: The Eden API client in Mercurial should be a singleton. This diff assigns the client to `repo.edenapi` so that it is accessible throughout the code.
Reviewed By: quark-zju
Differential Revision: D14233314
fbshipit-source-id: 8e0ed22c32611e8f6e7d4461c3e31870d47a0e95
Summary:
Client telemetry is used to send which commands are being run on the client,
which makes debugging slow sessions easier. For the client perspective, the
server hostname is send back so that the user knows to which physical host it's
connected, which is also useful debug info.
Reviewed By: StanislavGlebik
Differential Revision: D14261097
fbshipit-source-id: f4dc752671b76483f9dcb38aa4e3d16680087850
Summary:
We want to be able to reason about the progress of the sync job.
List of fields logged to [`mononoke_hg_sync`](https://our.intern.facebook.com/intern/scuba/query/?dataset=mononoke_hg_sync&pool=uber):
- reason
- entry
- repo_id
- succes
- err
- delay
- bookmark
Reviewed By: StanislavGlebik
Differential Revision: D14243842
fbshipit-source-id: 8ee3591be8f1dc9e1adca081b7b8eb51e7c6521a
Summary:
When converting from Mononoke `HgHistoryEntry` to Mercurial's `LooseHistoryEntry`, rather than directly copying the parents, we need to implement remotefilelog's logic for handling copied files.
Specifically, a copied file will usually have 0 parents (or 1 in the case of a merge). The filenode of the original (pre-copy) file is stored separately in the file's copy info. In the remtoefilelog loose file format, the filenode of the original file is stored as the p1 for the copy file, and the merge parent (if any) of the file is stored as the p2.
Reviewed By: StanislavGlebik
Differential Revision: D14256997
fbshipit-source-id: 09a070fb95d8b6c08c26e73156123d33cbfc685d
Summary: The code was accidentally setting the linknode for every entry to be the same as the filenode. This diff fixes the issue.
Differential Revision: D14251788
fbshipit-source-id: f0512c4c6ff59e2d22df0b42b4ee52a20f77a88f
Summary: Add an optional `depth` parameter to the `getfilehistory` endpoint, allowing the caller to specify the maximum number of history entries to fetch.
Differential Revision: D14222991
fbshipit-source-id: 8b28c9674e764f0098a9780f94ef2957ef117f09
Summary: Optionally specify a max depth for history fetching.
Reviewed By: StanislavGlebik
Differential Revision: D14218337
fbshipit-source-id: b6b92181172637e58a43bf61793257559915c7f1
Summary:
Move most of the fb303 methods from `fb303.FacebookService` to
`fb303_core.BaseService`.
A few methods have been left behind. In most cases these are methods that are
either already deprecated or that we would like to deprecate in the future.
One important method that is not being moved as part of this diff is
`getStatus()`. We do want to move this to `BaseService` in the future, but
doing so is a bit more challenging than the other methods, since it also uses
a custom enum that will need to be moved from `fb303.thrift` to
`fb303_core.thrift`. In most languages this can be done relatively easily
by replacing it with a typedef in `fb303.thrift`. Unfortunately the `py3`
thrift generator does not handle typedefs well, so more work will need to be
done to enable this for python3 thrift code.
Reviewed By: yfeldblum
Differential Revision: D13688965
fbshipit-source-id: 48d3d007ef6b63b4176cab753624813b7f9d6526
Summary:
Previously we always restart replaying from the beginning. starting from this
diff we actually record the latest replayed id in the mutable_counters table.
Special flag `--loop-forever` was added. It should be set in production, but for tests we don't want a binary to run forever.
Reviewed By: ikostia
Differential Revision: D14226938
fbshipit-source-id: cbfde15a506ee94b0fa72015d9dcfd550f5b8ca3
Summary:
Some repos that are not that big but they have a lot of merges
And sometimes there are a lot of merges where one merge parent
is ancestor of another merge parent, see example below:
o <- A
|\
| o <- B
|/|
o | <- C
|\|
| o
|/|
o |
|\|
...
|/|
| ~
o
|\
~ ~
Turned out in the cases like that blobimport or push can hit a combinatorial explosion.
See an example. Commit A depends on commit B and C. So a future that creates a commit A also
need to wait until commit B and commit C are created i.e. roughly future for commit A looks like
`B.join(C).and_then(...)`. So when future A is polled it polls future B and after it polls future C.
The problem is that future B also polls future C and they do so all the way
down. So we poll every future a lot of times even when it's completely unnecessary.
This is similar to doing DFS without remembering visited nodes. In that case
event loop spends all the time polling nodes over and over again.
Note that combinatorial explosion depends and how many in-flight futures we
have. For example, reducing the number of commits that are computed simultaneously
(see `CONCURRENT_CHANGESETS` parameter of blobimport) avoids the issue, but
obviously it slows down blobimport for "normal" non-mergy repos. It's also
harder to repro in debug mode.
Differential Revision: D14100259
fbshipit-source-id: 8bcbc34a2843477f3704eb43cf5876e16ad4be61
Summary:
Small refactoring before the next diffs.
Before
mononoke_hg_sync ... --mode sync-once ssh://PATH --start-id 0
Now
mononoke_hg_sync ... ssh://PATH sync-once --start-id 0
Reviewed By: HarveyHunt
Differential Revision: D14226940
fbshipit-source-id: 1b6db6f194aecb2f4bdb7bbd9e846aaba180e098
Summary: This diff adds a new `getfilehistory` endpoint to the API server that takes a filenode/path pair and returns a stream of JSON-encoded history entries (corresponding to `types::LooseHistoryEntry`) representing the full history of that filenode. (Note that the particular serialization format can be changed relatively easily in the future; for now I've chosen JSON for ease of debugging.)
Reviewed By: StanislavGlebik
Differential Revision: D14197233
fbshipit-source-id: 390f2e528f36fa3ec4b402e2fc3def0f2d529432
Summary: Convert the API Server to Rust 2018 edition.
Reviewed By: StanislavGlebik
Differential Revision: D14222621
fbshipit-source-id: 4afd8a57c415cfd5b5923d15cb2873a1c098f861
Summary: The Cargo.toml file for the API server appears to be from a very old prototype version. It is tiny and barely captures any of the current dependencies (many of which can't be built with Cargo anyway). Let's delete it since it's not adding any value.
Reviewed By: StanislavGlebik
Differential Revision: D14222619
fbshipit-source-id: c7ce625da859ae6fa85e787a6e92a2795c8420c5
Summary:
The main module of the API server is growing pretty large, since a new method and query parameters struct is required for each new HTTP endpoint added to the server. To make this manageable, let's restructure the code so that each handler function is paired with a params struct that contains its corresponding params.
This makes it easier to make changes to individual handlers, such as switching the method GET to a POST and changing the handler to read the params from the request body.
Reviewed By: StanislavGlebik
Differential Revision: D14222620
fbshipit-source-id: 67abea2ff32a3d6cc278375efd368fa2969ab891
Summary: We want blobimport to log all its newly added commits to scribe the same way mononoke will in the future. This will also enable post_commit scribe logs for fbsource (this is necessary to keep streaming_clone_tailer updating)
Reviewed By: HarveyHunt
Differential Revision: D14225504
fbshipit-source-id: cd6ab4f0245802135adfed428e87199b8f5cc3da
Summary:
We didn't process parts like `error:abort` and so we might have easily missed
an error. This diff fixes it.
Reviewed By: quark-zju
Differential Revision: D14185378
fbshipit-source-id: e68e365fd939a4bd6a0c2835a513ebc94530aa87
Summary: This diff added "manifest" field to Mononoke API Server's `get_changeset` method, which is required to implement Eden's `BackingStore::getTreeForCommit`
Reviewed By: StanislavGlebik
Differential Revision: D13966576
fbshipit-source-id: d2444b3aab477ab76efa7baddf114d90fa7a7a77
Summary: Add a thrift method `get_tree` to Mononoke API Server
Reviewed By: StanislavGlebik
Differential Revision: D13964085
fbshipit-source-id: c0d31e4e13079bec49d480c116ebb300c7e4260f
