Summary:
Our reverse sync job is too slow when pushing to svn. It seems like the slowness comes
keep doing push-pull-rebase which doesn't seem necessary in our case.
From my understanding, we need to do push-pull-rebase because we have multiple writers.
In our sync job, we only have one writer and therefore can we skip the pulls and rebases?
Reviewed By: DurhamG
Differential Revision: D16442559
fbshipit-source-id: 926d1c516e8e6d59298d310fc67927ace37f72c9
Summary:
Util method to generate Enum, since some machines don't have the
python library installed.
Reviewed By: suitingtseng
Differential Revision: D16491020
fbshipit-source-id: e06470a609605c482f591418583b516918c49d93
Summary: There no users left, let's get rid of it.
Reviewed By: kulshrax
Differential Revision: D16423137
fbshipit-source-id: 6898e681f800ab677010d7b6cdd36d377e3ef644
Summary: The Rust code is the default.
Reviewed By: kulshrax
Differential Revision: D16423134
fbshipit-source-id: 30baf041a5e7c786a28b31cfe5025e893b34594c
Summary:
The Rust code is now always taken, let's start removing all the references to
the python mutabledatapack code.
Reviewed By: kulshrax
Differential Revision: D16392287
fbshipit-source-id: dfccd4f4ec052a46975b6f9144106b51c3282988
Summary: This is now enabled for the entire fleet, let's hardcode this in the code now.
Reviewed By: kulshrax
Differential Revision: D16392289
fbshipit-source-id: 462152ded12d00cf8218526d51a911d6fe5975ca
Summary: Checks if the retrieved content of a file is equal to a `magic string` representing a blacklisted file. If so, then the content is replaced by a readable text which suggests a `rebase` or `update` to a newer commit.
Reviewed By: ikostia
Differential Revision: D16260011
fbshipit-source-id: ac1d40132b9c947927271d8e6efda98b19dce984
Summary: The debug messages from Eden API provide Source Control team members with useful diagnostic information about HTTP data fetching, but they have the potential to be spammy when written to log files. To prevent log spam, let's only print these messages during interactive usage.
Reviewed By: quark-zju
Differential Revision: D16445346
fbshipit-source-id: 001dc75e440eaf797f4f953648453086421f624e
Summary:
When using fetchpacks, memcache will write to an indexedlog, which can't be
repacked. I'm also suspecting that there is a race between this repack, and
hg_memcache_client opening the newly created packfiles which can cause the
misses to not be sent to memcache.
Reviewed By: kulshrax
Differential Revision: D16432538
fbshipit-source-id: 62362682474883bcd58249791c02b9fed5cb8fea
Summary: Bindings so that the rust manifest code can be used in Python.
Reviewed By: quark-zju
Differential Revision: D16352532
fbshipit-source-id: 34d4522f5e084f531f31bcd21770950f15f2fe13
Summary:
Copytrace modified the global definitions table which was making it very difficult to keep track of side-effects as the code was executed, as well as making it harder to replace the fancyopts calls with native Rust.
Since the copytrace behavior can be achieved through a configuration, it now will no longer modify the global definitions table, and will display the correct flag for a user to use in order to get this same behavior.
Reviewed By: quark-zju
Differential Revision: D15902449
fbshipit-source-id: 1c254162d56823e65085b7047bb37513f187b487
Summary:
When rebasing or showing a diff for a commit which moved files, remotefilectx.ancestors (called by _tracefile) calculates the linkrev for each ancestor. Sometimes [1], this is a disaster:
* remotefilectx._linkrev executes its slow path and scans the change log.
* For each entry in the change log, remotefilectx._linkrev downloads trees if needed.
* Hg downloads trees one-by-one (as of D15964145).
remotefilectx.ancestors is only calculating linkrevs so it can sort the ancestors topologically. _tracefile only needs the ancestors to be ordered topologically, not by linkrev. remotefilectx.ancestors's calls to remotefilectx._linkrev are redundant.
Optimize _tracefile's use of remotefilectx.ancestors: order remotefilectx objects topologically (breadth-first) without sorting by linkrev. Create a new function for this purpose (topological_ancestors) to avoid possibly breaking other callers of remotefilectx.ancestors. As a side effect, make this new function return remotefilectx objects lazily, similar to the filectx.ancestors function.
On my machine, with warm caches, this speeds up 'hg diff -c' and 'hg rebase' for a modestly-sized commit. 'hg diff -c' takes 0.64 seconds, down from 65.6 seconds.
[1] Hypothesis: After 'hg amend', 'hg bundle' packages linkrevs which refer to the pre-amend commit (which is not serialized into the bundle) rather than the post-amend commit. 'hg unbundle' thus creates linkrevs referring to a missing commit.
Reviewed By: DurhamG
Differential Revision: D16297426
fbshipit-source-id: 407597d5e36fc06b33719c28f5ea5052e01dc7a3
Summary:
If one of the repo checkouts doesn't have remote bookmarks synchronization enabled, it must not affect the Cloud's remote bookmarks state.
I also changed `localserver` (is used for our tests) because Commit Cloud server deletes the "old" bookmark names from the db and inserts "new". However, the `localserver` just set the "new" books dictionary as a new state, which is not quite correct.
Reviewed By: simpkins
Differential Revision: D16338257
fbshipit-source-id: d77d9218b1c35ea1a097bbe7393d0910ce7b4d38
Summary: While checking if one public commit is an ancestor of another public commit, I used string hashes instead of binary nodes. Fixing.
Reviewed By: farnz
Differential Revision: D16338256
fbshipit-source-id: e27ed8d79ec9ed3a188cdee93366ab2f96526152
Summary:
This commit is just a simple refactor of the method for the resolving
the `svnrev` template keyword. In particular, we split the method so that it
can be easily wrapped by other extensions like globalrevs in D16361887.
Reviewed By: quark-zju
Differential Revision: D16361888
fbshipit-source-id: 9f20fb33afd2b286c4f30571fa257b8284f2bb54
Summary:
When we are migrating www to hg as source of truth, we would like to maintain the
reverse sync for some time in case we need to rollback.
In order to achieve that, we need to know the latest svn and hg commit to operate on. We
would like to record this information in the svn commit itself so it doesn't require
extra syncing and transaction.
In ordre to get this info, we can run `svn log -l 1` and parse the commit message from
there.
Reviewed By: quark-zju
Differential Revision: D16337012
fbshipit-source-id: acf66babdb48c07f95e9eb49daac0d3d3e6a96a0
Summary:
There are multiple places that want to know whether the code is running inside
tests or not. Add `util.istest` for that and migrate those callsites to use it.
Reviewed By: xavierd
Differential Revision: D16344715
fbshipit-source-id: faa4261c83e4d889d59d63968b440954c4cac2ce
Summary:
We've had cases where hg_memcache_client failed to write to the
indexedlogdatastore, but since it was returning a success code, hg wouldn't
find the node, and crash.
Let's add a new error code to signify that all the requested objects should be
considered as misses.
Reviewed By: quark-zju
Differential Revision: D16344530
fbshipit-source-id: 4e627fee1603f9e4d5f542ef2d1f0e881a085bad
Summary:
the default value was being set to type int, changed it in both occasions to "".
in cloudsync I had to do some parsing for the version into an int not to break the rest of the code
Reviewed By: mitrandir77
Differential Revision: D16340048
fbshipit-source-id: 7f7142ea4c7862c0a093a68d5c8ceb1d2e22df24
Summary: Updated help text for hg undo and fixed test
Reviewed By: kulshrax
Differential Revision: D15937058
fbshipit-source-id: 577d05fb9c16237f8b879ef9ebf60341d5fdf37e
Summary: Modified the hg cloud sl command to have two options --date and --version_number that display that historical version of the smartlog
Reviewed By: mitrandir77
Differential Revision: D16120273
fbshipit-source-id: 4f202ed49488247f43bd682574ff52dcf17c958e
Summary:
Bad callers could try to add nodes that had the same value as the
parent. Let's catch this and prevent it.
Reviewed By: quark-zju
Differential Revision: D16189602
fbshipit-source-id: 08220c18ee96743e4eda00e5e953a5946b40d95c
Summary:
The python runtime update in D15894694 broke ctypes for all APIs that accept a
Windows `HANDLE`. This was used by the `fsmonitor` code that tries to connect
to watchman on windows.
As a result Mercurial would silently fail to use watchman. Commands like
`hg status` became very slow, with no user-visible output to indicate that
anything was wrong. (It wouldn't print a progress bar or any other
information either.)
Reviewed By: quark-zju
Differential Revision: D16198869
fbshipit-source-id: 675007674e0ac742c5b62739d766b8c3016e0c1b
Summary:
Following D16198896, we can really simplify the logic for resolving
globalrev/svnrev to commit hash. In particular, we can make much more stronger
assumptions about what is indexed now and therefore, narrow the search to only
the commits that are not cached.
Reviewed By: quark-zju
Differential Revision: D16199072
fbshipit-source-id: cec66cbda24ee6df200fc77aeed42c11c29edeba
Summary:
When looking for the matching filenode when searching for the linkrev of a
filectx, override fetchdepth to 1 so that we don't download the whole manifest
just to examine a single file, and add tracing and a progress bar, as this can
be slow.
Reviewed By: mitrandir77
Differential Revision: D15964145
fbshipit-source-id: d338de330c82d4240bed5569caf25769778458aa
Summary:
With treemanifest and the eden api, descendantrevfastpath (which downloads all
the manifests) is slower than just downloading the history from the eden api.
We still want to record the descendantrev if we already know the real linkrev
of the current filenode, as that will still be fast.
Reviewed By: quark-zju
Differential Revision: D15943076
fbshipit-source-id: c6013801822bdfa3196e60cbcd34a9ce184c5c5f
Summary:
Adjusting linknodes can sometimes take a long time. Add a progress bar and perftrace
for when this happens.
Reviewed By: sfilipco
Differential Revision: D15943077
fbshipit-source-id: f91bc6bb9f7c6d39ffe9ac7b2bf4ab0f32b4d9d3
Summary:
During `cloud sync`, disable the following options, as they slow sync down and
aren't really necessary.
* Set `pull.automigrate` to False, as there is no need to perform pull migrations
in the sync pull.
* Set `treemanifest.pullprefetchrevs` to False, as we won't have updated the
remote bookmarks, so there is no need to try to prefetch the trees.
* Set `treemanifest.prefetchdraftparents` to False, as synced commits are not
usually checked out immediately, so there is no need to prefetch the tree.
Reviewed By: quark-zju
Differential Revision: D16054353
fbshipit-source-id: 0eeab49e5242fdddc593843ed67e6a17281ec912
Summary:
Add `treemanifest.prefetchdraftparents`, which controls whether or not to
prefetch the trees for the parents of newly added draft commits.
We will want to avoid this in commit cloud, as the prefetch may take too
long, and the user might not access the commits anyway.
Reviewed By: quark-zju
Differential Revision: D16054354
fbshipit-source-id: cf590451fb5252d215748567d6ad421c589ed581
Summary:
Previously I tried to not call to SVN if we have the 2 configs set. However I made a
mistake that the args will always be evulated anyway (computer science 101).
Let make it really lazy and add a test for that.
Reviewed By: ikostia
Differential Revision: D16163470
fbshipit-source-id: 6ec0f855b10164ae9a210bc70789b2f59fd19858
Summary:
On a shared host (like the devgpu), the shared cache will be used by many
users, and thus the repacklock needs to be able to be read/write by all the
users. Despite the code trying to open it as 664, the umask downgraded it to
644, only allowing the file creator to run repack.
Reviewed By: quark-zju
Differential Revision: D16151053
fbshipit-source-id: 4d3ecc19f2f6fbf9a8b831f7ea39d0a53bc4b8e9
Summary:
Fixing bugs I've discovered by enabling remote bookmarks sync for myself:
* turned out, adding "None" to the dict field in Python doesn't work as a null-value of optional field in C++ Thrift structure
* I didn't update lastsyncstate in some of the cases, also fixing
Reviewed By: mitrandir77
Differential Revision: D16131799
fbshipit-source-id: bdee423f519820d702719b3dfe76865a7547221e
Summary: It will be used in non-fsmonitor cases.
Reviewed By: markbt
Differential Revision: D15755286
fbshipit-source-id: 75ba7c5f227e1c9224c5d092b7e766afdf45c428
Summary:
The new log is typed and concise.
Old log:
[fsmonitor]> clock='c:1559600325:3956709:1:34762' len(nonnormal)=0
[fsmonitor]> setlastclock 'c:1559600325:3956709:1:36405'
[fsmonitor]> setlastisfresh False
[fsmonitor]> watchman returned ["x"]
[fsmonitor]> getlastclock 'c:1559600325:3956709:1:36405'
[fsmonitor]> set clock='c:1559600325:3956709:1:36405' notefiles=["x"]
New log:
[fsmonitor] clock: "c:1559600325:3956709:1:34762" -> "c:1559600325:3956709:1:36405"; need check: [] + ["x"]
In JSON form:
{"fsmonitor":{"new_clock":"c:1559600325:3956709:1:36425","new_files":{"len":1,"short_list":["x"]},"old_clock":"c:1559600325:3956709:1:34762"}
The new logging does not cover every information exposed by the old logging.
For example:
- Non-treestate events like why fsmonitor state gets invalidated.
Assuming most clients are on treestate now. These are removed.
- "fsmonitor_state = normal | unavailable | fresh" scuba logging. This can be
inferred, and will be added in a later diff.
- New "notefiles". The next "fsmoniotr" event will log the information.
Reviewed By: markbt
Differential Revision: D15710672
fbshipit-source-id: 5c4cad08c0072c7dc711e5c1e65aa7552940699e
Summary:
This has the exact same behavior as `hg debugdatapack` but on
indexedlogdatastore. Since nodes can be added concurrently to the indexedlog,
we can't verify that nodes aren't duplicated, so disable that logic.
Reviewed By: kulshrax
Differential Revision: D16073582
fbshipit-source-id: b7bbefdca8062c8b797432cb76b0de9df850148e
Summary: All the users are now using the Rust code.
Reviewed By: kulshrax
Differential Revision: D16039092
fbshipit-source-id: 1341ed33ab9a17f30b541b02eaa577db56959e31
Summary:
This allows switching hg debughistorypack to using Rust historypack code
instead of Python. Similarly to the iterentries for datapack, we collect all
the data in memory instead of using an iterator.
Reviewed By: kulshrax
Differential Revision: D16039098
fbshipit-source-id: 8b61c04e43a319a4601cc1aaae2eb46922d4a851
Summary:
This has now been entierely replaced by Rust code.
Goodbye.
Reviewed By: kulshrax
Differential Revision: D16039096
fbshipit-source-id: 8c3ce2eb5595a5f90c3ecab413a242a3d0afcb89
Summary: This is the last use of Python datapack in treemanifest
Reviewed By: kulshrax
Differential Revision: D16039101
fbshipit-source-id: 205b6fed0811ae86542f72aeed95cec81a61d741
