Commit Graph

287 Commits

Author SHA1 Message Date
Jun Wu
047b1a05f4 exchange: kill checkheads
Summary:
We had `ui.checkheads=false` set for a long time. Let's remove the feature
entirely and update the tests.

This is necessary before killing the branch cache, as some tests still use
different branches and there would be suddently more "heads" that cause
test issues.

Reviewed By: sfilipco

Differential Revision: D14179861

fbshipit-source-id: 0de76566799a9560b45e823cc5f49cfda9e3dd30
2019-02-22 21:02:41 -08:00
Liubov Dmitrieva
6ee9cf11e5 infinitepush + commitcloud: strip the marker to get build the
Summary: Mononoke uses the markers but we shouldn't look at them for Commit Cloud.

Differential Revision: D14188356

fbshipit-source-id: e5dee581728a9bc83d2f7a17575b3ae6b3183d39
2019-02-22 13:16:24 -08:00
Liubov Dmitrieva
65b628a472 commit cloud: check that the secondary path is not the same as the first.
Summary:
add this check to avoid overhead

we don't need to backup to the secondary place if it is the same as the first.

Reviewed By: singhsrb

Differential Revision: D14187754

fbshipit-source-id: 6ee59ae2f0846716ca99253958af7088d0538df9
2019-02-22 12:55:14 -08:00
Liubov Dmitrieva
ac751a0ac1 fix format for broken check-code test
Summary: Just renaming, and another variable as well to look similar

Reviewed By: DurhamG, quark-zju

Differential Revision: D14185033

fbshipit-source-id: e34de690274afd2f2c6e51db21c7b158f6c3452a
2019-02-22 11:44:51 -08:00
Johan Schuijt-Li
877d1315da treemanifest/remotenames: use write path for reads after push
Summary:
Treemanifest uses it's own fallbackpath for reads in all cases, but
particularly in the case of remotenames it should stay on the path that is
actively being used by the push. It is possible to have remotenames which are
mirrored and selected by query strings in the repo path. In this case it is
possible that the mirror is still out of date when reading back data from our
push. Ensure that when doing a push the remote server this is considered
'sticky' so that we read back from the remote we pushed to, rather then
determening the path ourselves.

To disable, please use:
  treemanifest.stickypushpath=False

Reviewed By: DurhamG, markbt, quark-zju

Differential Revision: D14165444

fbshipit-source-id: 75a53ffab895d87a4c52814f7887145c134868b5
2019-02-22 06:03:18 -08:00
Stanislau Hlebik
9aaad3788f pushrebase: limit pushrebase error message
Summary: We'll start recording hook output, which can be rather long. Let's limit that.

Reviewed By: quark-zju

Differential Revision: D14164228

fbshipit-source-id: efb50d89c2c78e717c4eb340f53fe049fe69f932
2019-02-22 05:10:46 -08:00
Stanislau Hlebik
8922f8a4b1 hook: make it possible to record stderr of failed python hooks
Summary:
Context: in pushrebase replay job we run on Mononoke we also want to test our
hooks i.e. replay pushrebase requests that were denied on mercurial because of
a hook failure. The problem is that at the moment the only error message we have
is `prepushrebase.driver hook failed` (see xdb table `xdb.mononoke_production`,
query `select * from pushrebaserecording where pushrebase_errmsg like '%hook%'
limit 10;`). This error message tells us nothing, not even what hook failed.

To change it I suggest to make it possible for python hooks to raise HookAbort
and provide more information in the `hook_failure_reason` field.

Reviewed By: quark-zju

Differential Revision: D14131359

fbshipit-source-id: 69a2b51b30c78efadf3ba0d3332f46a777022568
2019-02-22 05:10:46 -08:00
Liubov Dmitrieva
db6d74a622 pushbackup - add a message when we truncate heads that needs to be backed
Summary: In chef there is currently an override to 50, so if someone got more due to some reason, it is nice to add a message.

Reviewed By: quark-zju

Differential Revision: D14170749

fbshipit-source-id: 3f3c79e4e6103523c14c8d9c1600f104e3d5d3d8
2019-02-21 20:27:32 -08:00
Arun Kulshreshtha
fd1ccf44e3 rage: use pastry instead of arc paste
Summary: `pastry` is the modern replacement for `arc paste`. Let's use it in `hg rage` instead of `arc paste`.

Reviewed By: singhsrb

Differential Revision: D14172108

fbshipit-source-id: 6586b9a8d8b90bac55d91baed852edbc7c1d9db9
2019-02-21 14:00:58 -08:00
Saurabh Singh
1c782a4f47 stackpush: make specifying the original node optional
Summary:
Currently, stackpush requires specifying the original node for each
new node to be created. This inhibits the creation of new commits which are not
replacing any commit via stackpush. This commit changes the behaviour such that
specifying the original node is optional.

Reviewed By: quark-zju

Differential Revision: D14153674

fbshipit-source-id: b3d150a8a8044ac1740937f2dc058ce542ee13e4
2019-02-21 10:01:19 -08:00
Stanislau Hlebik
ddd0f0ed99 mononoke: mononoke -> hg sync job
Summary:
First stab at a job that will keep hg in sync with Mononoke when Mononoke
becomes a source of truth.

Reviewed By: ikostia

Differential Revision: D14018269

fbshipit-source-id: f88c5eba8bf5482f2f162b7807ca8e41a3b4291d
2019-02-21 05:46:40 -08:00
Stanislau Hlebik
e36da58ac9 pushrebase: fix choosing commit timezone
Summary: There was a typo, the timezone is `[1]` and not at `[0]`

Reviewed By: ikostia

Differential Revision: D14147329

fbshipit-source-id: 8ad4bff810ed949a9f8e86d03ef62bc63aaf11bd
2019-02-21 00:59:13 -08:00
Liubov Dmitrieva
bf0fec10f0 infinitepush: priority for scratch pushes should start with infinitepush path
Summary: the push-path will be used for Mononoke write path roll out.

Reviewed By: markbt

Differential Revision: D14153207

fbshipit-source-id: 158e6e08dfa017c98e668fdfc37138dd7b7d76c7
2019-02-20 15:36:29 -08:00
Liubov Dmitrieva
ac401b9d9f commitcloud: trigger a backup to a secondary storage after cloud sync in the background
Summary:
We can't run in parallel at the moment as the log file and the lock file are
shared.

Every path maintains independent backup state (the previous diff).

The secondary backup state doesn't affect smartlog (only the main one)

The issue with this approach is that we maintain backup lock a bit longer.

Unfortunately, the progress in smartlog doesn't show anything about the second backup.

I added 'finished', it makes it easier to compare in the logs.

Reviewed By: markbt

Differential Revision: D14149399

fbshipit-source-id: f90e8aac6cb8dee53d5c7468bd6adba067e13362
2019-02-20 15:36:29 -08:00
Liubov Dmitrieva
ae7b506f80 infinitepush backup: separate depends on the remote infinitepush path
Summary:
We are going to support 2 different backends of Commit Cloud: Mercurial and
Mononoke.

Each of them should maintain local backup state separately.

Output of some tests have been slightly changes, this is because a separate backup
state, the same error appears earlier when we are trying the backup stacks.

The idea is to have separate backup states for different remote paths, but
there will be only one cloud sync state for the current source of
truth. We could include there the remote path and then validate that cloud sync
state is correct if the remote path has been changed.

However, for backup states it is much easier to have them separately (and we
will backup in 2 places)

Reviewed By: markbt

Differential Revision: D14138496

fbshipit-source-id: 0a7a763a395be5456cbd724bff7ebc069f03fb0e
2019-02-20 15:36:29 -08:00
Arun Kulshreshtha
a15b016f89 movement: add short flags for --clean and --merge
Summary: The `hg update` command has short flags for `--merge` and `--clean`, namely `-m` and `-C`. Let's add the short versions to `hg next` and `hg prev` as well.

Reviewed By: quark-zju

Differential Revision: D14155218

fbshipit-source-id: d51f5f658b525809e4c512ffa300dea4b4cabcdf
2019-02-20 14:49:53 -08:00
Durham Goode
8d31ac8424 simplecache: add checksum to simplecache values
Summary:
We're encountering an issue that I think is caused by invalid data
coming back from memcache, possibly due to our recently introduced connection
reusing. Let's add some checksums to verify that the data we put in is identical
to the data we get out for a given key.

Reviewed By: kulshrax

Differential Revision: D14141683

fbshipit-source-id: 206b51b862db7d54def02f5310b90f473d5a0d03
2019-02-20 13:56:33 -08:00
Arun Kulshreshtha
aaf26c23ca movement: pass --merge flag to update from prev/next
Summary: We weren't passing the `--merge` flag from `hg prev` and `hg next` to the underlying invocation of `hg update`. This diff fixes the problem so that `--merge` now works as expected.

Reviewed By: DurhamG

Differential Revision: D14154244

fbshipit-source-id: 4a2412cca714ec8f269eb5f2989e39821642fbb3
2019-02-20 13:49:08 -08:00
Durham Goode
1935d046b8 pushrebase: backout change to preload the manifest
Summary:
Backs out D13944829 that add preloading the manifest to pushrebase. In
a situation where you receive tree commits, but the repo is hybrid, this causes
it to try to lookup the bundled tree manifests in the flat manifest revlog.

Let's just back this out for now until we can come up with an appropriate fix,
or move all our repos to treeonly.

Reviewed By: quark-zju

Differential Revision: D14151876

fbshipit-source-id: 0f7419d5b9bcd9d7ce363f4349e9e2e4a86cf713
2019-02-20 13:44:51 -08:00
Mark Thomas
23aa7bda11 rage: read fsmonitor state from treestate
Summary:
If treestate is in use, fsmonitor state is stored in the treestate, rather than
in a separate fsmonitor.state file.  Update rage to understand this.

Reviewed By: quark-zju

Differential Revision: D14131131

fbshipit-source-id: d80d766625d7915b6a76f66f33e763eed23677e9
2019-02-20 11:01:52 -08:00
Kostia Balytskyi
4a3a890f0d unbundlereplay: do not produce changegroup response part
Summary:
This is a perf optimization. `unbundlereplay.respondlightly` instructs the
server to not produce the pushback parts regardless of what `replycaps` part
of the incoming bundle says. This is important, since the mononoke-hg sync will
send all the bundles in a searialized way, so we want to optimize time where
possible.

Reviewed By: StanislavGlebik

Differential Revision: D14131575

fbshipit-source-id: afec15347d43fa52b1ec64b4ac8ece5b227ccf7d
2019-02-20 08:47:25 -08:00
Xavier Deguillard
126de4655e revisionstore: add a Store trait
Summary:
A lot of code is duplicated between data stores, and history stores, and one
reason for it is the absence of common trait between these 2. By adding a new
Store trait it will make it easier to write generic code that works accross
data and history store.

Reviewed By: quark-zju

Differential Revision: D14091899

fbshipit-source-id: deef1d43a7d300cb3607c67554ad54f20c870e23
2019-02-19 12:18:27 -08:00
Liubov Dmitrieva
6f13efb795 infinitepush - prepare the code to use 'known' wireprotocol request for
Summary:
Prepare the code to use 'known' wireprotocol command to check what is backed up
on the server instead of slower 'lookup'. We don't need to reestablish ssh
connection, and we can test all the nodes in one go.

Also test the 'known' request with Mononoke is working (all commands have been
tested with isbackedupnodes => isbackedupnodes2)

Reviewed By: markbt

Differential Revision: D14131383

fbshipit-source-id: 5a150b64d0e84a02357c2367879b2da8493d9167
2019-02-19 08:48:07 -08:00
Liubov Dmitrieva
5e7afa31ff commit cloud / pushbackup - use the same way to check what is backedup with Mononoke and Mercurial
Summary:
Use the same way to check if the commits have been backed up in Mononoke and Mercurial.

The only common way to check is 'lookup' command because Mercurial doesn't support discovery for commit cloud commits, the command 'known' is also not supported.

Also we have to go one by one because lookup doesn't got any better API.
It is still much faster than backup commits that are already there.

Introduce such check for pushbackup as well.

Remove hacky way to check it from cloud sync.

For commit cloud in Mononoke we will have backfill, so the server side check  will be heavily used when you go to Mononoke at the first time.

Unfortunately connection pool module in mercurial is not good enough in detecting closed connections and can easily return a broken connection on the next call.

Reviewed By: markbt

Differential Revision: D14085849

fbshipit-source-id: d76d9a71f9efdbdfec4de3198cd428b6b693418d
2019-02-19 05:49:07 -08:00
Kostia Balytskyi
da3e429150 wireproto: add unbundlereplay command
Summary:
This is to be used from Mononoke->hg sync.
Currently expects only `pushrebase` bundles, e.g. one head and one book to
move.

Reviewed By: StanislavGlebik

Differential Revision: D14116130

fbshipit-source-id: 959a6e51f51e21da5592c84188e294a33057ffaa
2019-02-19 01:57:39 -08:00
Mateusz Kwapich
4db0750027 fsmonitor: use correct length for number of files changed
Summary: When comparing the number of files changed, fsmonitor was incorrectly using the length of the result dict (always 4), rather than the number of files watchman returned.  Use the right list of files instead.

Reviewed By: markbt

Differential Revision: D14123604

fbshipit-source-id: 94684f1f189d045b2f6a880180b15e52ba9bba8c
2019-02-18 10:03:12 -08:00
Liubov Dmitrieva
1c47c775c0 inifinitepush: add new capability
Summary:
this will be used to understand if server supports 'listkeyspatterns'

(Mononoke doesn't support this)

Reviewed By: DurhamG

Differential Revision: D14107742

fbshipit-source-id: c9a42e8516eb5660ab2f498996b211db7086bcb1
2019-02-18 04:30:32 -08:00
Arun Kulshreshtha
9c6b914a22 types: move Key and NodeInfo out of revisionstore
Summary:
In order to move the types in `edenapi-types` (containing types shared between Mercurial and Mononoke) to the `types` crate, we need to move a few types from the  `revisionstore` crate into this crate first, because `revisionstore` depends on `types`, which would create a circular dependency since `edenapi-types` uses types from `revisionstore`.

In particular, this diff moves the `Key` and `NodeInfo` types into their own modules in the `types` crate.

Reviewed By: quark-zju

Differential Revision: D14114166

fbshipit-source-id: 8f9e78d610425faec9dc89ecc9e450651d24177a
2019-02-15 22:51:04 -08:00
Liubov Dmitrieva
4454d6eeb5 pushbackup: fix early exit logic
Summary:
the early exit logic was incorrect, basically if there is at least 1 bookmark
in the repo and the backup state was not empty, it didn't catch that nothing has been changed.

bookmarks are dicts, so it is fine to compare them

if any bookmarks in the repo, everytime `hg pushbackup` established a connection to mercurial

Reviewed By: quark-zju

Differential Revision: D14103938

fbshipit-source-id: 0edc4d9e186199670765fd2362236330e831c7d9
2019-02-15 10:04:00 -08:00
Jun Wu
702ed9a453 test-revset: do not use "hg branch" directly
Summary:
Migrate away from using the "hg branch" command to "hg commit --extra branch="
instead. In the future the branch namespace would be removed, so we create
local tags instead.

`test-revset2` was split from `test-revset` and has the same header. Do the
same change to it. Besides, de-duplicate tests about the `tag()` revset.

Reviewed By: ikostia

Differential Revision: D14058435

fbshipit-source-id: b59fadc43939d85d14bbf9f81227c523b65557a0
2019-02-14 17:44:39 -08:00
Xavier Deguillard
76316fbf9d revisionstore: verify repacked keys before deleting pack files
Summary:
During repack, the repacked files are deleted without any verification. Since
Adam saw some data loss, it's possible that somehow repack didn't fully repack
a packfile but it was deleted. Let's verify that the entire packfile was
repacked before deleting it.

Since repack is mostly a background operation, we don't have a way to notify
the user, but we can log the error to a scuba table to analyse further.

Reviewed By: DurhamG

Differential Revision: D14069766

fbshipit-source-id: 4358a87deeb9732eec1afdfb742e8d81db41cd87
2019-02-14 13:03:09 -08:00
Michael Liu
8747ff025f Apply modernize-use-override
Summary:
Use C++11’s override and remove virtual where applicable.
Change are automatically generated.

Reviewed By: Orvid

Differential Revision: D14052667

fbshipit-source-id: 119e237155a5318e08e31065c8f87e6b7f3f7c52
2019-02-14 11:57:55 -08:00
Xavier Deguillard
e5a7da32da revisionstore: rename the packfile before removal on windows
Summary:
Removing files on Windows is hard. It can fail for many reasons, many of which
involves another process having the file opened in some way. One way to solve
this problem is that renaming the file isn't as restrictive as removing it.

Since hg repack will attempt removing any temporary files it will also try to
remove the packfiles that we failed to remove earlier.

Reviewed By: DurhamG

Differential Revision: D14030445

fbshipit-source-id: 1f3799e021c2e0451943a1d5bd4cd25ed608ffb6
2019-02-14 10:34:52 -08:00
Xavier Deguillard
dc6cbcdfe6 remotefilelog: do not preload packs
Summary:
Preloading all the pack files on initialization ties the lifetime of the
packfiles to the repo. For normal operations, this is fine, as packfiles are
mostly read. During a repack however, we need to be able to remove them, and
while having an open file handle allows deletion on unix OSes, it prevents it
on Windows.

The Rust repack now succeeds on Windows.

Reviewed By: DurhamG

Differential Revision: D14013786

fbshipit-source-id: 99279d4af67a0dfe8679159e9409186f56a09296
2019-02-14 10:34:52 -08:00
Johan Schuijt-Li
09d54f3cf4 remotenames: treat query strings and fragments as parameters
Summary:
When using query strings or fragments in an URL we should treat repository paths
as the same repo. This allows servers to use query strings for metadata, without
treating the urls as different servers. By introducing normalization in our grouping,
we will consider the normalized result to be the qualifier for what is the same repo,
rather then the full absolute path - which includes query strings and fragments.

Reviewed By: DurhamG

Differential Revision: D14051479

fbshipit-source-id: c82fe041467f6bd6af210688c0be873e2da93781
2019-02-14 03:21:53 -08:00
Arun Kulshreshtha
a4a155c025 edenapi: add configitem call for edenapi.url
Summary: We were reading the  `edenapi.url` config item without explicitly setting it up with the `configitem()` function. Not sure what negative impact this would have, but it's probably a good idea to have the explicit call in place.

Reviewed By: quark-zju

Differential Revision: D14075080

fbshipit-source-id: bb4e25de273341768f850f1d5aab6ac21e7f2fc5
2019-02-13 17:41:54 -08:00
Arun Kulshreshtha
c68b388bed edenapi: remove try around import
Summary: Now that the `edenapi` module in bindings is always available for all platforms, we no longer need a try block around the import.

Reviewed By: quark-zju

Differential Revision: D14075082

fbshipit-source-id: e3f45e67ef4572e58f85875af12390ea5d697d43
2019-02-13 17:41:54 -08:00
Arun Kulshreshtha
575e570a28 bindings: move pyedenapi into bindings crate
Summary: Move the edenapi Python bindings into the common `bindings` crate.

Reviewed By: quark-zju

Differential Revision: D13963179

fbshipit-source-id: 76dead82af992615a9e452ee6fbb9f66639c822c
2019-02-13 16:07:00 -08:00
Liubov Dmitrieva
380f425385 infinitepush: fix issue with discovery
Summary:
if server already have everything, the function should return True

otherwise we treat it as fail to backup in pushbackupbundlestacks function

failed to push stack bundle rooted at ...

Reviewed By: quark-zju

Differential Revision: D14048141

fbshipit-source-id: 288291db2f31b0d284c97fecbd61049a58c045de
2019-02-13 08:20:17 -08:00
Jun Wu
9d21b98c57 convert: drop monotone support
Summary:
Monotone repos are rare these days. Drop support for it.
This also solves an issue that test-convert-mtn.t uses named branches.

Differential Revision: D14059836

fbshipit-source-id: 1e9d4fe6fdc295393ff67c5e068b230b9ed0c0af
2019-02-12 21:45:13 -08:00
Jun Wu
4cd0b5c098 transplant: remove the extension
Summary:
The (disabled by default) `transplant` command is similar to `graft` or
`rebase`. It makes sense to removeit.

This diff removes the extension.

Reviewed By: singhsrb

Differential Revision: D14000099

fbshipit-source-id: a03e4925cefa4236bd9d62cfe9d33d140707bd7c
2019-02-12 21:45:11 -08:00
Jun Wu
87b87d5490 rebase: remove --keepbranches flag
Summary: Branches are going away. Remove related features.

Differential Revision: D13993824

fbshipit-source-id: 8afb01912df016bda08be5bc22ac52be7f5168eb
2019-02-12 21:45:10 -08:00
Jun Wu
fbccd19ed7 patchbomb: remove the extension
Summary:
`test-patchbomb.t` uses named branches and does not look simple to fix.
We don't use email patches internally, and it's not hard to write scripts
around `hg export`. Therefore drop the extension and its tests.

Reviewed By: singhsrb

Differential Revision: D13978577

fbshipit-source-id: 19867ae68c19c996bfce064eb2234705939db9ea
2019-02-12 21:45:10 -08:00
Jun Wu
04f6795519 remotenames: remove logic about named branches
Summary: Named branches are going away. Remove the logic around it.

Reviewed By: phillco

Differential Revision: D13978575

fbshipit-source-id: d6e28d7cadffa612f74a2afc12800829d6113dfa
2019-02-12 19:54:30 -08:00
Xavier Deguillard
e241404290 remotefilelog: always refresh before runonpacks
Summary:
The refresh function is intended to load on-disk packfiles that aren't yet
present in the cache, but it had several issues that resulted in unexpected
behavior. Mainly, the new files would be added in the wrong order in the cache,
and runonpacks would not update the last used packs.

Differential Revision: D14013785

fbshipit-source-id: da4151b859441f0ba51863f3907852922c255f82
2019-02-12 11:21:34 -08:00
Xavier Deguillard
f5d7209d39 remotefilelog: move rustrepack
Summary:
Previously, the logic for chosing between the rust repack, and the python one
was deeply integrated with the python repack. This resulted in work being
performed unnecessarily.

The drawbacks of this method is that in the case of fallback to Python, the
repacklock will be dropped and re-obtained, as well as the prerepack hook will
be ran twice. Since the python code will soon be entirely replaced this is
probably not important.

Differential Revision: D14013790

fbshipit-source-id: 7c754db3fb984cc9e6b7df1a4e32b72fa4a1531b
2019-02-12 11:21:34 -08:00
Xavier Deguillard
f134a4fe92 remotefilelog: refactor incremental/full repack
Summary: Most of the code is identical between these 2 functions, so let's merge them.

Differential Revision: D14013787

fbshipit-source-id: f0469e05fcf723db02d9f4c28097e61f958284a5
2019-02-12 11:21:34 -08:00
Jun Wu
946fa43efd fsmonitor: update watchman clock if too many files are returned
Summary:
This should auto recover from a state where watchman returns too many files,
which makes "hg status" slow.

The default config value 200 is not chosen very scientifically, but it should
cover the 6s status case where there are 90k changed files reported.

Differential Revision: D14036494

fbshipit-source-id: 368f53e99e2e54343a6ac9145cbb86e4ac65e4ac
2019-02-12 11:16:12 -08:00
Jun Wu
98bb16cace hgsql: fix hggit sync
Summary: Teach mysql connector to deal with bytearray.

Reviewed By: DurhamG, StanislavGlebik

Differential Revision: D14025816

fbshipit-source-id: 738d9ef37b985afad5fe62815be134a4a5913b9a
2019-02-11 13:47:12 -08:00
Aida Getoeva
3aee83d26a prefetch: replaced store path with the cache path
Summary:
To connect to the memcache client it used a path from a loose files data store (`remotefilelogcontentstore._path`), which is a path to the hg cache, but pack files store doesn't have the same field.
I replaced the path with cache path from shallowutil.

Reviewed By: singhsrb, liubov-dmitrieva

Differential Revision: D14020122

fbshipit-source-id: 29bc8a01fc42d43eddc6ae67cf4e41fc552e117c
2019-02-11 09:38:43 -08:00
Xavier Deguillard
0ce06567ab remotefilelog: properly declare CacheConnectionError
Summary:
The super python builtin expects a type as an argument, while a string was
passed, this was causing trouble for some users.

Reviewed By: DurhamG

Differential Revision: D14012716

fbshipit-source-id: 6714eb20745428818721ec3df85588a7b4c7ebb7
2019-02-08 17:16:00 -08:00
Liubov Dmitrieva
ca7f631006 lazy connection open for memcache has an issue with SIGPIPE
Summary: My earlier diff broke master, this is the fix.

Reviewed By: markbt

Differential Revision: D14005894

fbshipit-source-id: 6ce43913fb501791592512f2f12cd67ef27f0457
2019-02-08 09:44:06 -08:00
Liubov Dmitrieva
261da2a786 fix remotenames for infinitepush
Summary:
Currently if default path and infinitepush both point to the remote path
from the repo, the activepath function returns 'infinitepush' rather than
'default' because of the logic I have fixed.

It makes your smartlog looks like.

```
o  037dddb5  62 minutes ago  infinitepush/fbobjc/stable remote/fbobjc/stable
.
o  66e0e00c  Today at 05:58  infinitepush/fbandroid/stable
remote/fbandroid/stable
.
@  28c3b5b0  Today at 05:40  mlesyk  D14003881  T27597699
.  [fbar][prod_role] fix region detection part
.
o  407504d6  Today at 04:58  infinitepush/fbsource/stable
remote/fbsource/stable
```

When you pull it updates infinitepush/master, etc instead of remote/master,
etc.

This only happens if they both point to the same url.

This is not desirable for infinitepush.

I also introduced infinitepush-other that we will use for the secondary commit cloud backend.

It would have the same issue if it matches the url of the default path.

Reviewed By: markbt

Differential Revision: D14005639

fbshipit-source-id: f3ea8098e1c418e2efd43b5430bb4252f6ad6b2c
2019-02-08 09:24:20 -08:00
Wez Furlong
35e2b7c6ff hg: fix lfs and journal extension postshare wrapping
Summary:
this fixes `hg-new-workdir` by matching the function signature
and propagating the arguments.

Reviewed By: singhsrb

Differential Revision: D13992348

fbshipit-source-id: 582025affe2e0511f239e82bc880b2fecea84ac7
2019-02-07 18:58:40 -08:00
Durham Goode
20f9ac38f6 remotefilelog: handle corrupt loose files more gracefully
Summary:
If the history portion of a loose file was corrupt, users would often
get errors like "ValueError: substring not found" when trying to split on the
null separators. Let's catch that and handle it where appropriate. For cache
stores, we move it out of the way and return a KeyError, so the union store will
move on to try to fetch it from the server again. For local stores, we return a
ValueError and surface it to the user since their local data may be corrupt.

Reviewed By: quark-zju

Differential Revision: D13976499

fbshipit-source-id: 39e63f19fc752ca0179fbc5b0908a28d46de3a3a
2019-02-07 13:40:42 -08:00
Durham Goode
2def6874f5 remotefilelog: default file blobs to version 1
Summary:
We're seeing cache corruption issues due to mixing and matching loose
file versions. Let's just default to 1.

Reviewed By: quark-zju

Differential Revision: D13980892

fbshipit-source-id: 1e8cc9a53b8b92e05dc08b202505bb8f453a3074
2019-02-07 11:08:01 -08:00
Liubov Dmitrieva
da5a666d2f use simplecache store on the server side to store remotefilelog cache
Summary:
simplecache can be configured with memcache backend

this will potentially save us a lot of disk space and allow us to have a shared cache between hg hosts.

getting rid of the caches on disks will save 966GB of disk space per machine.
P60876105

Reviewed By: quark-zju

Differential Revision: D13950137

fbshipit-source-id: 015e971f1bfed334edb4fe2381c2c6336b84b161
2019-02-07 05:15:11 -08:00
Liubov Dmitrieva
638959056d simplecache - reuse the connection to memcache
Summary:
reuse the connection for other requests
I made it global

Reviewed By: quark-zju

Differential Revision: D13964705

fbshipit-source-id: ed759e36f15b91a838dc2c20b58384ae54ad5d48
2019-02-07 04:59:48 -08:00
Liubov Dmitrieva
13abfafb8d treemanifest: use simplecache as a cache store
Summary:
simplecache extension support both memcache and local disk as a cache store.

we are going to use memcache on hg servers

getting rid of the caches on disks will save 966GB of disk space per machine.

P60876105

this will also allow hg machines to share the same cached data

Reviewed By: DurhamG

Differential Revision: D13927163

fbshipit-source-id: 3155c61da4a82dd7a790a8ccf8dbc701fe3957e1
2019-02-07 04:59:48 -08:00
Adam Simpkins
8d35b90969 fix a crash in remotefilelog prefetch
Summary:
It looks like D13924759 accidentally replace `prog.value` with `prog` in one
location, which is causing crashes of the form:

  TypeError: unsupported operand type(s) for +=: 'nullbar' and 'int'

Reviewed By: quark-zju

Differential Revision: D13984835

fbshipit-source-id: cbf1a52d8048b623beac6048224767f0abb7f5c0
2019-02-06 22:03:19 -08:00
Durham Goode
af61927d32 remotefilelog: add documentation for remotefilelog.getfilesstep
Summary:
This is breaking the tests. It has existed for years, so not sure why
the recent reformatting broke it now.

Reviewed By: quark-zju

Differential Revision: D13976462

fbshipit-source-id: 08f3a9f08ec54449b4554017ea9806f3ee51afcb
2019-02-06 13:02:59 -08:00
Aida Getoeva
562b9d1884 prefetch: remove check misses after fetching from server
Summary: We don't really need to check misses here as neither of request fucnctions return misses after fetching from the server. They raise exception if there are any misses.

Differential Revision: D13908252

fbshipit-source-id: 80b6e4fb79b419b5e6d3b343b3bd46d3f8dc7d3c
2019-02-06 10:43:15 -08:00
Aida Getoeva
996ac6da37 prefetch: add requestpacks
Summary: Implemented logic for fetching data from the scmmemcache and the server in pack files format. By default `prefetch` will still use loose files, unless the `remotefilelog.fetchpacks` config flag will be set to true.

Differential Revision: D13905733

fbshipit-source-id: cc559f5bc600067199a91efe6cf44b1bde6bf69b
2019-02-06 10:43:15 -08:00
Aida Getoeva
d853db727d prefetch: add get/set for pack files and remove hitcount
Summary: New get/set protocols for pack files. Also removed hitcount from `receive` method, because it is not used.

Differential Revision: D13924759

fbshipit-source-id: 1382b6f2bc865a19125f3ccd17a779cd16f08b50
2019-02-06 10:43:15 -08:00
Aida Getoeva
2b23a376f6 prefetch: get packs' paths after fetching from server
Summary: To add the data fetched from the server to the scmmemcache, we need to know pasths to the packfiles created.

Differential Revision: D13905734

fbshipit-source-id: 7c470b560e32e63e7df4f960be92bc9b6da9de3d
2019-02-06 10:43:15 -08:00
Aida Getoeva
3ea6881ea7 prefetch: move "fetch from server" code to the requestloose
Summary: Moved fetching loose files from the server to the separate function

Reviewed By: DurhamG

Differential Revision: D13855807

fbshipit-source-id: 39821644abcdc7375ba1b9ef9519fd2083eb0a71
2019-02-06 10:43:15 -08:00
Aida Getoeva
e218de341c prefetch: add set and get request
Summary: Move scmmemcache set/get commands protocol to the cache wrapper to isolate it from the main logic

Reviewed By: DurhamG

Differential Revision: D13855809

fbshipit-source-id: 95b70f7f9d10d97beff107f9f6e5e4f5a8a307c1
2019-02-06 10:43:14 -08:00
Aida Getoeva
79a1bd7770 prefetch: return set of misses instead of list
Summary: List of missed files is converted to a set many times. So changing the type to a set.

Differential Revision: D13924760

fbshipit-source-id: 65fe52ff6113724d8f7f7a8d5b3c1501f245eedc
2019-02-06 10:43:14 -08:00
Aida Getoeva
494608cf81 prefetch: add receive
Summary: Move scmmemcache receive logic to the cache wrapper to isolate it from the main prefetch logic

Reviewed By: DurhamG

Differential Revision: D13855808

fbshipit-source-id: d0da84857797e3e155507fcf284b39a146677e54
2019-02-06 10:43:14 -08:00
Durham Goode
163704f80a disablesymlinks: add extension to disable symlinks
Summary:
In some cases we are mounting ntfs in a linux host using ntfs.3g and we
want to prevent it from writing symlinks that can't be read from Windows.

Some background, ntfs.3g supports symlinks, but it produces IntxLNK which cause
problems when mounted into a real Windows machine. We don't support symlinks on
Windows anyway, so we want to just disable this. Unfortunately it's difficult to
detect if we're on a ntfs.3g fuse mount, so instead we're just going to have the
machine set up the repo with this extension enabled.

Differential Revision: D13964546

fbshipit-source-id: 7545538e51be4c986a50161165d7cc2e64280555
2019-02-06 08:41:34 -08:00
Arun Kulshreshtha
70aff50986 edenapi: rename mononokeapi to edenapi
Summary: Rename Mononoke API to Eden API, per war room discussion.

Reviewed By: quark-zju

Differential Revision: D13908195

fbshipit-source-id: 94a2fe93f8a89d0c5e9b6a24939cc4760cfaade0
2019-02-05 21:22:48 -08:00
Arun Kulshreshtha
6eab7a5222 mononokeapi: use auth section for TLS configuration
Summary:
Use the hgrc auth section to configure HTTP data fetching. For example:

```
[mononoke-api]
enabled = true
url = https://mononoke-api.internal.tfbnw.net

[auth]
mononoke-api.prefix = mononoke-api.internal.tfbnw.net
mononoke-api.schemes = https
mononoke-api.key = /var/facebook/credentials/$USER/x509/$USER.pem
mononoke-api.cert = /var/facebook/credentials/$USER/x509/$USER.pem
```

Reviewed By: DurhamG

Differential Revision: D13907185

fbshipit-source-id: 3b607f25912a85438b56c4ea7ab9cfed91ab7e85
2019-02-05 21:22:48 -08:00
Durham Goode
3309a8090a pushrebase: read the manifest before looping through files
Summary:
pushrebase has logic that loops through all the files and builds a list
of what has changed. Unfortunately, Mercurial has some optimizations where if
the manifest isn't loaded already, it tries to only load the manifest delta, and
checks if this might be a good idea by first checking if the file is in the
commit metadata file list. The commit metadata is a list, which makes it a O(n)
scan to check containment. Since we do this for every file, it becomes O(n^2).

To avoid this, let's just make sure the actual manifest is loaded.

Once every repo is a tree repo, we can get rid of the manifest delta
optimization and get rid of the need to prefetch here.

Reviewed By: singhsrb

Differential Revision: D13944829

fbshipit-source-id: c0f33ca650b7956a1f39e961c94678a6f7f380b6
2019-02-05 16:05:50 -08:00
Stanislau Hlebik
ff74d1cc82 remotefilelog: add an option to verify all the hashes
Reviewed By: DurhamG

Differential Revision: D13838772

fbshipit-source-id: e7220ba8a0e07f3003693199086e3a3512023857
2019-02-05 15:03:24 -08:00
Mark Thomas
f402b4f525 lfs: include name of file being downloaded in progress bar
Summary:
When doing lfs prefetch, pass the filenames that correspond to each oid to lfs
so that it can include them in the progress bar.

Reviewed By: DurhamG

Differential Revision: D13716078

fbshipit-source-id: c91eca6b342996c0c3d1ee43f3b1e98db4ed64d4
2019-02-04 11:44:28 -08:00
Durham Goode
5eced4c533 watchman: canonicalize repo root before querying watchman
Summary:
If the repository was mounted into a different directory than the
source of truth, our watchman integration would not work. To fix this, Mercurial
now applies the same canonicalization logic to the path before passing it to
watchman. This logic is stolen verbatim from watchmans integration test path
utility.

This also requires updating our copy of pywatchman to match the upstream
pywatchman.

Reviewed By: quark-zju

Differential Revision: D13886207

fbshipit-source-id: ea72166edf2eb30f2bee992d9aa936e2e9729e63
2019-02-04 10:34:08 -08:00
Mark Thomas
b44e2103d7 simplecache: don't use memcache in tests
Summary:
The simplecache test attempts to connect to a memcache instance on localhost.
This means the test fails if a memcache instance is not available.

Remove the use of memcache in the test.  We still test local files, and assume
that memcache itself works.

Also corrupt the correct key when testing corruption handling, and document the
config options.

Reviewed By: quark-zju

Differential Revision: D13818471

fbshipit-source-id: 6ca7f575aa813f95773144be5337796f029ffd90
2019-02-01 19:19:36 -08:00
Mark Thomas
ae0a81f2c2 rust: move bindings to a single python extension
Summary:
Move all Rust bindings to a single python extension, `bindings`.  This should
improve compilation time and make things simpler.

Reviewed By: quark-zju

Differential Revision: D13923866

fbshipit-source-id: 560592b5a6c0c4f1b836c755ef123666a1059164
2019-02-01 17:53:22 -08:00
Stanislau Hlebik
6adfda0e99 pushrebase: move selecting commit date in a separate file
Summary:
Add an option that let's us specify the date of each commit in a pushrebase.
It should be specified via a config option, which accepts a name of a file in
which json dictionary should be stored. Key of this dictionary is a commit hash, value
is a date.

Reviewed By: ikostia

Differential Revision: D13803174

fbshipit-source-id: 6271c18c61399b89c92dce7a4fe63c8fae8dae7c
2019-02-01 09:42:57 -08:00
Arun Kulshreshtha
78ba8186f3 remotefilelog: add debug command to fetch multiple files over HTTP
Summary: Previously, `hg debuggetfile` would allow fetching a single file from the API server. This diff updates the command to `hg debuggetfiles`, which accepts a list of filenode/path pairs on stdin, and fetches all the files concurrently, writing the contents to a single datapack.

Reviewed By: DurhamG

Differential Revision: D13893894

fbshipit-source-id: 36fc54f1870273ab4b447de5d615b3fb6aaa3378
2019-02-01 01:41:31 -08:00
Arun Kulshreshtha
50c293b94e pymononokeapi: add Python API for batch file downloads
Summary: Expose `MononokeClient`'s multi-get functionality through the Python bindings using a builder-style API.

Differential Revision: D13893748

fbshipit-source-id: 312e7449eb8300432b1f1de28e0c6fc3e47d1d53
2019-02-01 01:41:31 -08:00
Arun Kulshreshtha
1b74af2ace mononokeapi: add ability to fetch a file and write it to a datapack
Summary: Add a new `get_file()` method to `MononokeClient` that fetches Mercurial file content from the API server and writes it to a datapack in the cache. This functionality is exposed via the new `hg debuggetfile` debug command, which takes a filenode and file path and fetches the corresponding file.

Differential Revision: D13889829

fbshipit-source-id: 2b68bf114ee72d641de7a1043cca1975e34cf4e6
2019-02-01 01:41:31 -08:00
Arun Kulshreshtha
f19e1e6771 remotefilelog: rename debugmononokeapi to debughttphealthcheck
Summary: Rename this debug command to make it obvious what it does (i.e., perform a health check) and make it print more useful output (i.e., the name of the server for which it performs the health check).

Reviewed By: DurhamG

Differential Revision: D13890867

fbshipit-source-id: 8fc96bcc06d04611a308ecc0b870049936f1cb29
2019-01-30 18:30:49 -08:00
Jun Wu
b557ea167b doc: fix rst format
Summary: Fix ill-formatted content that breaks `make doc`.

Reviewed By: singhsrb

Differential Revision: D13885565

fbshipit-source-id: 1b4a771e535f76679d73d9081714ad4c36b1db8c
2019-01-30 14:57:42 -08:00
Jun Wu
9dc21f8d0b codemod: import from the edenscm package
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.

Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.

Reviewed By: phillco

Differential Revision: D13868981

fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
2019-01-29 17:25:32 -08:00
Wez Furlong
83f00ab318 hg: restore spawning watchman on-demand
Summary:
A while back, as a startup optimization, we started to hard code
the path to the watchman socket in the mercurial config.  This meant
that we could avoid running a subprocess for every command.

Today, a chef configuration change was rolled out that caused watchman
to be stopped on all of the corpnet macs with the result being that
all mercurial commands are slow because it never uses and never restarts
watchman.

This diff adds some fallback logic to allow spawning the server in
the case that we have a connection error.

Reviewed By: DurhamG

Differential Revision: D13855696

fbshipit-source-id: 44c0ad36b6e80520912e42bcea9e04f4f165b588
2019-01-29 15:49:37 -08:00
Jun Wu
c12e300bb8 codemod: move Python packages to edenscm
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.

To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.

Reviewed By: phillco, ikostia

Differential Revision: D13853115

fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
2019-01-28 18:35:41 -08:00