Summary: It can be useful to understand how many ancestors are not derived yet.
Reviewed By: Croohand
Differential Revision: D28902194
fbshipit-source-id: 87c11b3e35ba7f67122990318ff07408c47d4d6c
Summary:
Backing stores differentiate between individual tree objects and the
root of a checkout. For example, Git and Mercurial roots are commit
hashes. Allow EdenFS to track variable-width roots to better support
arbitrary backing stores.
Reviewed By: genevievehelsel
Differential Revision: D28619584
fbshipit-source-id: d94f1ecd21a0c416c1b4933341c70deabf386496
Summary:
Making it the default should make cloning using EdenFS easier and will remove
the need to manually pass the `--nfs` command line to `eden clone`.
Reviewed By: kmancini
Differential Revision: D28913818
fbshipit-source-id: 742c35c950cb5edf34fd9769fb78bf26095af0d5
Summary:
Now that we don't publish any Python 2 packages, let's drop make local.
Once we've confirmed that nothing was using make local, we can rename make
local3 to be make local.
Reviewed By: kulshrax
Differential Revision: D28647154
fbshipit-source-id: de277887e93a6dbc0324a30f592198ef7c83f818
Summary:
More things needing fixing to be python 3 compatible. Caught when
trying to remove the Python 2 build.
Reviewed By: quark-zju
Differential Revision: D28880028
fbshipit-source-id: d162c78237f330f1f931c3581b25ead24e3ea375
Summary:
The combination of metalog and the new clone pattern of first creating
the repo causes local copy clones to fail on Windows because the initial metalog
files are held open and the copy can't overwrite them.
Let's drop the destrepo before we do the local copy.
Reviewed By: quark-zju
Differential Revision: D28880029
fbshipit-source-id: 2a4ef52675eebf16afa528e645acd927a6110cb4
Summary:
The position the the lines with "count:" is undeterministic. Let's skip
matching them for now to unblock test.
Reviewed By: mzr
Differential Revision: D28940741
fbshipit-source-id: c05521722f838e7b72572a0cf67d3ebbc3b26868
Summary: This will be used for rate limiting decisions. Also, could be logged to scuba tables to get more info about clients.
Reviewed By: quark-zju
Differential Revision: D28750197
fbshipit-source-id: 83f54e38f998c9dd824ef2d3834c777a44d0ffed
Summary: Let clients connect to lfs with HTTP through unix socket so we don't have to worry about certificates presence.
Reviewed By: johansglock
Differential Revision: D28683392
fbshipit-source-id: f6228b4099ef04fe584e320cb1892e6cb513e355
Summary: The update timestamp field of the walker_checkpoints wasn't being set on updates.
Reviewed By: Croohand
Differential Revision: D28871609
fbshipit-source-id: b8873ec494f6671ef27b0243a8b2fb373a7313e8
Summary: Code inspection of derived data shows several places that are already set up to be I/O parallel; by adding `spawn` calls and looking for performance changes, I found that these two also gain from being CPU parallel. The others are not helped (or hindered) by a `spawn`, so leave them alone for now.
Reviewed By: ahornby
Differential Revision: D28901878
fbshipit-source-id: f774bdf93a11e9c0f0370612968f4f32179f3eb1
Summary:
WAL is known to be significantly faster than the default DELETE journaling
mode, let's enable it.
Reviewed By: fanzeyi
Differential Revision: D28915718
fbshipit-source-id: 3ee10d10dbaf07a01a23101a6266874a41784cfc
Summary:
These appear to be unecessary, thus there is no need to keep them in place as
it makes the code harder to follow.
Reviewed By: chadaustin
Differential Revision: D28850241
fbshipit-source-id: 129b853e88f0291bb637c9d24de4872a222a5af4
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
A few final write calls need to be threaded as well.
Reviewed By: xavierd
Differential Revision: D28844011
fbshipit-source-id: 214853eea7fa65263fe6415e1ae8b76ca21512d6
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
Some thrift methods used by watchman to get metadata implicitly are not threaded.
These can cause fetches, so let's thread the fetch context here too.
Reviewed By: genevievehelsel
Differential Revision: D28842300
fbshipit-source-id: b1e4b3aea879d6ed7b92afa26184616dedad5935
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for symlink
Reviewed By: genevievehelsel
Differential Revision: D28841453
fbshipit-source-id: 080eb62f0b562f8e0995c34e9a8302238fc59ed8
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
Most of rmdir is already threaded, not sure if this case can actuall cause
fetches in production, but might as well thread.
Reviewed By: genevievehelsel
Differential Revision: D28840211
fbshipit-source-id: 8dea08e775be470dd1730e2d32750a6912650ee0
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
this change is for rename
Reviewed By: genevievehelsel
Differential Revision: D23467437
fbshipit-source-id: e9d79c65fb5c4d686f0597550e43a0e87c4792cb
Summary:
we now want macFUSE to be installed instead of osxfuse. eden doctor will spew
errors if osxfuse is not installed or loaded, so it perpetually complains.
Update these checks for macFUSE now.
Reviewed By: xavierd
Differential Revision: D28751766
fbshipit-source-id: 4bc61349e33492aebe888a4e869ef7620c74768e
Summary:
implement filenode and tree lookup in edenapi
simple lookup in blobstore without any additional validation of blob content
assuming all the validation will be inside upload logic
here, assuming if key is known that content of blob is valid
Reviewed By: markbt
Differential Revision: D28868028
fbshipit-source-id: 590cc404f33adbec69f8adafd33365a0249d3241
Summary:
create end to end intergation for the lookup API on the client
Start prototyping of `hg cloud upload` command.
Currently, it just performs lookup for existing heads.
This way we can end to end test the new APIs.
Reviewed By: markbt
Differential Revision: D28848205
fbshipit-source-id: 730c1ed4a21c1559d5d9b54d533b0cf551c41b9c
Summary:
Files upload will be executed in 2 stages:
* check if content is already present
* upload missing files
The check api is generic, it could be used for any id type. Called 'lookup' API.
Reviewed By: markbt
Differential Revision: D28708934
fbshipit-source-id: 654c73b054790d5a4c6e76f7dac6c97091a4311f
Summary:
Previously we set this in the rpm spec, but we need to set it in make
local as well since sometimes hgbuild invokes make local directly.
Ideally we'd put this in setup.py, since make and rpmspecs go through that, but
we need this environment also set for the dulwich build, which we don't really
control the setup.py for.
Reviewed By: singhsrb
Differential Revision: D28902015
fbshipit-source-id: bfc170c3027cc43b24c6a517512a63a71f433d23
Summary:
Recently we had an issue with `connectivity-lab` repo where 3 keys P416141335 had different values because of parent ordering P416094337.
Walker can detect difference between keys in the multiplex inner blobstores and repair them, however it doesn't have notion of the copy keys (there isn't concept of source and the target). We have a copy_blobstore_keys tool, which is used for restoring keys from the backup and with small modification it can handle copy between innerstore.
Reviewed By: StanislavGlebik
Differential Revision: D28707364
fbshipit-source-id: 3d5a4f39999623023539b9159fa7310d430f0ee4
Summary:
All megarepo methods write to a repository, so we need to check if write is
allowed to a given repo, and previously we weren't checking that.
Let's fix it and Let's start doing so now by trying to get RepoWriteContext
for the target repo. If writes are not allowed then RepoWriteContext fails.
Reviewed By: farnz
Differential Revision: D28838994
fbshipit-source-id: e45d4fe72603e7fe2755141874fc4125998bfed8
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for mkdir
Reviewed By: genevievehelsel
Differential Revision: D23458622
fbshipit-source-id: f3914a4f692490434882143664a5d5f1701e93ba
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
This change is for create
Reviewed By: genevievehelsel
Differential Revision: D23457862
fbshipit-source-id: d4c9cc658c26b3119b2b2a1da061e299eaf510c9
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
Most of lookup is already threaded. This finishes the threading for lookup.
Reviewed By: xavierd
Differential Revision: D23456910
fbshipit-source-id: fab7397caeee19f921d8fba1fb6528baa5cf2960
Summary:
The recent change to make run-tests work with Python 3 broke the
allow/deny list functionality because it started testing the full test name
instead of the base. This fixes that.
Reviewed By: quark-zju
Differential Revision: D28885125
fbshipit-source-id: 586a71e66e0f094b79e6a3e07e27813db6f662d3
Summary: create `uncopy` command to unmark files that were copied using `copy`.
Reviewed By: quark-zju
Differential Revision: D28821574
fbshipit-source-id: c1c15f6fb2837cec529860aba70b516ddd794f10
Summary:
Time 0.2 is current, and 0.1 is long obsolete. Unfortunately there's a
large 0.1 -> 0.2 API change, so I preserved 0.1 and updated the targets of its
users. Also unfortunate that `chrono` has `oldtime` as a default feature, which
makes it use `time-0.1`'s `Duration` type. Excluding it from the features
doesn't help because every other user is specifying it by default.
Reviewed By: dtolnay
Differential Revision: D28854148
fbshipit-source-id: 0c41ac6b998dfbdcddc85a22178aadb05e2b2f2b
Summary:
The worker should be able to process the requests from the queue no matter
which repo it is and what are it ACLs. It's during the request scheduling when
we should check the identity of the entity scheduling request.
Reviewed By: StanislavGlebik
Differential Revision: D28866807
fbshipit-source-id: 5d57eb9ba86e10d477be5cfc51dfb8f62ea16b9e
Summary:
BackingStore and LocalStore are no longer tied at the hip, so decouple
FakeBackingStore from LocalStore.
Reviewed By: kmancini
Differential Revision: D28615431
fbshipit-source-id: ee6bc807da6de4ed8fba8ab6d52ff5aeff34e8ae
Summary:
The meaning of the root ID is defined by the BackingStore, so move
parsing and rendering into the BackingStore interface.
Reviewed By: xavierd
Differential Revision: D28560426
fbshipit-source-id: 7cfed4870d48016811b604348742754f6cdbd842
Summary:
Currently bookmark warmers receives a `BlobRepo`. This diff makes it receive an `InnerRepo`, but does no logic changes.
This will be useful as fields are moved from `BlobRepo` to `InnerRepo`, and will also be useful for accessing skiplists from warmers, as I plan to do on the next diff.
Reviewed By: StanislavGlebik
Differential Revision: D28796543
fbshipit-source-id: dbe5bec9fc34da3ae51e645ea09b03e2bb620445
Summary:
This diff simply extracts `InnerRepo` object type to a separate target.
This will be used on the next diff where we need to access `InnerRepo` from `BookmarkWarmer`, which needed to be split to a different target in order to not create a circular dependency in buck.
Reviewed By: StanislavGlebik
Differential Revision: D28796406
fbshipit-source-id: 4fdadbbde31719b809abb6b8a9ba8fa24b426299
Summary:
This diff creates a new `InnerRepo` container, that contains `BlobRepo` as well as the skiplist index.
The plan here is:
- As of code organisation, `InnerRepo` will eventually contain most of the fields currently in `Repo`, as well as the fields of `BlobRepo` that are only used in binaries that use `Repo`. This way each binary will only build the "attribute fields" it needs to, but the code to build them can still be neatly shared.
- As for `SkiplistIndex`, the plan is to be able to modify it inside `WarmBookmarksCache`, that's why I'm moving it to `InnerRepo` as well. I'll make bookmark warmers receive `InnerRepo` instead of `BlobRepo`, so they can access the skiplist index if wanted, and then modify it (this is an attempt to try to make skiplists faster on bookmarks).
Reviewed By: StanislavGlebik
Differential Revision: D28748221
fbshipit-source-id: bca31c14a6789a715a215cc69ad0a69b5e73404c
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for mknod
Reviewed By: chadaustin
Differential Revision: D23452153
fbshipit-source-id: 7b9bc6b624fbe81b91770bc65a0d27bc9d397032
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for getxattr
Reviewed By: chadaustin
Differential Revision: D23451954
fbshipit-source-id: bae73878754d59661cddf7c0b001e506bbc88d13
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for readlink
Reviewed By: chadaustin
Differential Revision: D23451821
fbshipit-source-id: 1f8ee369a992ab3489a9366f9a972f67461970de
Summary: The run_start in the pack info logging is the run start for a given checkpoint name when checkpointing, so let's include that to provide context.
Reviewed By: farnz
Differential Revision: D28867962
fbshipit-source-id: 113b9e10f5b8e1869702b3ea83374d0d08a8792e
Summary: This makes it easier to understand what each move and merge commits are for.
Reviewed By: mitrandir77
Differential Revision: D28839677
fbshipit-source-id: 1a42205c164224b64c773cff80b690b251a48381
Summary:
When deriving `hgchangesets` for merge commits, we try to get filenodes right.
Speed this up a little by recognising that history checks are unnecessary if both parents have the same filenode.
Reviewed By: StanislavGlebik
Differential Revision: D28866024
fbshipit-source-id: 6da4c162abce5b426269630f82e9e0b84eea2b33
Summary: Sometimes it is useful to search/group by commit hash.
Reviewed By: krallin
Differential Revision: D28834644
fbshipit-source-id: 93f650e19ae512450e33542cf74b8aa3333c6c35
Summary:
Fetching all history of both filenodes to see if there's common history either side of a merge is wasteful, and in some megarepo work is causing long delays deriving merge changesets.
Where we have already derived filenodes for a given merge's ancestors, we can go faster; we can use the linknodes to determine the older of the two filenodes, and fetch only history for the newer of the two.
This is imperfect for the import use case, since filenodes depend on hgchangesets, and the batching in use at the moment prefers to generate all derived data of a given type before moving onto another type, but it's an improvement for cases where some filenodes are already derived (e.g. due to import of a repo with a similar history).
Reviewed By: StanislavGlebik
Differential Revision: D28796253
fbshipit-source-id: 5384b5d2841844794a518c321dbf995891374d3a
Summary:
This is a precaution. add_sync_target can create a very branchy repository, and
I'm not sure how well Mononoke is going to handle deriving of these commits by
all mononoke hosts at once (in particular, I'm worried about traversal that has
to be done by all hosts in parallel). In theory it should work fine, but
deriving data during add_sync_target call should be a reasonable thing to do
anyway.
While working on that I noticed that "git" derived data is not supported by derived data utils, and it was causing test failures. I don't think there's any reason to not have TreeHandle in derived data utils, so I'm adding it now.
Reviewed By: farnz
Differential Revision: D28831052
fbshipit-source-id: 5f60ac5b93d2c38b4afa0f725c6908edc8b98b18
