Summary:
The progress bar thread is a daemon thread, so that it doesn't need to be
deactivated manually. However, this means the thread terminates and the lock
is released during interpreter shutdown. Because Python clears module
attributes at the start of interpreter shutdown (see [1]), releasing the lock
fails as it can no longer get the thread identity.
To mitigate this, we register an atexit handler to terminate and join to the
progress bar thread.
[1] https://stackoverflow.com/questions/25649676/where-is-pythons-shutdown-procedure-documented
Differential Revision: D7568155
fbshipit-source-id: 85ef10af6c1576d5beceb78f8514e0e440cdab7f
Summary:
We disabled `runworker` for every importer (fastimport, syncimport, seqimport) when we hotfixed a customized file transaction breakage in fastimport D7108127.
Since
- we're not using fastimport, which relies on the customized file transaction
- seqimport does not rely on the customized file transaction mechanism
- syncimport is super slow without `runworker` and syncimport does not rely on the customized file transaction mechanism
- the manual work involved in adding mapping to ovrsource (we have to do that if we don't have syncimport) is not trivial
I think it's a good idea to bring back `runworker`, which is a configurable config, and it's explicitly set to `false` for fastimport and seqimport, `force` for syncimport.
This diff
- make the `useworker` config available again
- remove the unnecessary test on the customized file transaction, which is still broken with `runworker`
- log the time it takes to finish one sync import
Differential Revision: D7557856
fbshipit-source-id: 6d4105cc38b182e027512730901ce3b2a4e1d449
Summary: We want to know what is the reason why we think the lock file can't be removed.
Reviewed By: farnz
Differential Revision: D7513954
fbshipit-source-id: fd1668e7a614e5e24e250018fbc880ba87821aa8
Summary:
This helps with PID reuse, which can cause false positives when checking
whether the lock-owning process is still alive.
Reviewed By: mjpieters
Differential Revision: D7513955
fbshipit-source-id: d3df4b4afa53cd1e9633d71c294cf7014b7b65c5
Summary:
The goal of this diff is to introduce a central place for all logic related
to parsing and handling the contents of the lock files. The current state of
things is hard to manage and even harder to extend.
NB: the stack is not complete, the changes that actually use this refactoring are yet to come.
Reviewed By: quark-zju
Differential Revision: D7489326
fbshipit-source-id: aa79d964411e3f5b61e24aa9babece05d4f0bd60
Summary:
This will later be used to add a bit of uniqueness to PIDs on Windows, to
address the stale lock problem.
Reviewed By: quark-zju
Differential Revision: D7489325
fbshipit-source-id: b866bd239eacfc7fe64433f0989a73dff1246644
Summary: We need this to support producing sparse profile stats in a null-revision working copy.
Reviewed By: quark-zju
Differential Revision: D7553022
fbshipit-source-id: b1643f443de5a748bf81ff2df16ace984d59a2c2
Summary: A followup diff will add more in the way of testing; this is easier with an actual usecase.
Reviewed By: quark-zju
Differential Revision: D7553020
fbshipit-source-id: a6e8b86154d802f8cdbc846f5ea14fa809ac5264
Summary:
hg sparse explain wasn't supposed to recurse, not without additional support.
It was the process of gathering stats that overwrote shared mutable structures.
Prevent this from happening again by making these structures immutable.
Differential Revision: D7516570
fbshipit-source-id: 247c60b4bd8d2d22b42c41871e1eb15988820025
Summary:
We were ignoring the actual profile data, and all profiles were given the same .hg/sparse-based hash signature instead.
Error introduced in D7415720
Differential Revision: D7514087
fbshipit-source-id: 56288aaaa2065b031318e7c065ec6310f6cecd37
Summary:
This will either include system dirent on POSIX or a vendored dirent from
folly on Windows.
`/no-check-code` is here because it's everywhere across hg's .c codebase.
Differential Revision: D7555759
fbshipit-source-id: dc55926e83e17976930522277ed7fe6ce41f32f7
Summary:
Our tests use features that are provided by the bash shell to function
properly. I ran into issues with our tests on Ubuntu because the default shell
was `dash` which did not even support the `source` command. Therefore, this
commit changes the default hg shell for the tests to `bash`. Clients can always
override the default hg shell for the tests to another shell.
Reviewed By: quark-zju
Differential Revision: D7563733
fbshipit-source-id: e9c16c002793a919a49292a9aa0a876ba232c293
Summary:
`run-tests.py` has an option to specify the shell for running the
tests. However, when run with the `--shell bash` configuration, several tests
fail with the `testrepohg: command not found` error. I could not find the
specific cause for this but changing the alias to a function resolved this
error. Also, fixed another related error by using double quotes during variable
expansion.
Reviewed By: quark-zju
Differential Revision: D7563731
fbshipit-source-id: 58e1b5b996ccdc20b8375dcd5f4f8e071bd9cdc1
Summary:
This allows us to turn on and off hgignore support directly without changing
files in the working copy (which could be hard to revert cleanly).
Reviewed By: mjpieters
Differential Revision: D7544529
fbshipit-source-id: 14cc41e2ae361070f91bf3b8aa28dd5808e7fe99
Summary: When I run this test (test-fb-hgext-p4fastimport-seqimport.t) on my dev server, which has the same `p4d` as what we have in prod (P59384966), it fails the test due to reading the file from a different source P59381563. Since both `p4` and `rcs` are valid options for a read, this diff updates the test to not restrict to `p4` only.
Differential Revision: D7556357
fbshipit-source-id: b82d254841a31fe447452ee408bdb8e157854aab
Summary:
sl assumed that the masterrev is missing from revset if any revset was
specified so it was failing back to current commit. This fixes this
behaviour by checking if masterrev is available.
Differential Revision: D7535692
fbshipit-source-id: 033088149b52293dce219b1dc7a04186bfc461b9
Summary:
This avoids building shared dependencies (ex. regex) over and over. The only
downside is cargo will take a lock and cannot build projects in parallel.
But `setup.py` does not support building extensions in parallel. So it's
fine.
Changed `matcher` to also enable lto like existing extensions, so `cpython`
build result can be reused.
Before (on devserver):
$ time python setup.py build_rust_ext
real 2m19.401s
user 3m35.118s
sys 0m8.277s
$ du -hs build/temp.linux-x86_64-2.7/
115M build/temp.linux-x86_64-2.7/
After:
$ time python setup.py build_rust_ext
real 2m4.371s
user 2m25.864s
sys 0m5.198s
$ du -hs build/temp.linux-x86_64-2.7/
58M build/temp.linux-x86_64-2.7/
`cargo` builds things in parallel. The speed improvement would be more
significant on laptops.
Differential Revision: D7512429
fbshipit-source-id: 378e721890bdfe53c8adbe364ad5f0b374023ff5
Summary:
Previously `hg graft -r 'ancestor::descendant` printed no error message at all.
This diff fixes it.
Reviewed By: mjpieters
Differential Revision: D7533498
fbshipit-source-id: 5c4e41ecc3178495ad2f41ef53ef65f7fbb70212
Summary:
A side effect is, the hint won't be printed out if fbamend is not enabled,
which is more "correct".
Reviewed By: markbt
Differential Revision: D7392130
fbshipit-source-id: 5b7aa4cc3083b03546c54965ce51040fab958b87
Summary: This allows people to silence the hint.
Reviewed By: markbt
Differential Revision: D7392127
fbshipit-source-id: ac16f952a178d567ce13e22946127456972ebe85
Summary:
This allows users to silence the "hide" advice.
In the future, we might want to change "hide/unhide" to only affect visibility
without changing obsolesce. So "strip" is not fully deprecated yet.
Reviewed By: markbt
Differential Revision: D7392131
fbshipit-source-id: 2448d4c91dffce31d29e2dd99078cb555c9a8f8c
Summary:
This allows people to silence hints as they like. It's done by modifying
user hgrc.
Reviewed By: markbt
Differential Revision: D7392133
fbshipit-source-id: 1365294217db92dfb3a0c81332a9fefd164795d4
Summary: This allows us to change configs using code.
Reviewed By: mjpieters
Differential Revision: D7392129
fbshipit-source-id: 0960d4a322d6906469a79976031ab2b619dfa183
Summary:
This allows us to have a unified way to print hint messages at the end of a
command. It would be helpful for feature discovery in general.
Reviewed By: mjpieters
Differential Revision: D7392132
fbshipit-source-id: 8b4e94cc2176266652459ecca3428bd86d95bfe2
Summary:
I did some extra xdiff changes in upstream, namely:
- Remove unused features
- Replace "long" (32-bit in MSVC) with int64_t to support large files
- Add comment on some key variables
This backports them. It also includes Matt's fixes about Windows compatibility.
Reviewed By: ryanmce
Differential Revision: D7223939
fbshipit-source-id: 9287d5be22dae4ab41b05b3a4c160d836b5714a6
Summary:
The correct gitignore matcher needs O(N^2) time to check a path which is N
directory deep. For example, to check "a/b/c/d", it needs to check:
- Whether .gitignore matches a/b/c/d
- Whether a/.gitignore matches b/c/d
- Whether a/b/.gitignore matches c/d
- Whether a/b/c/.gitignore matches d
- Whether .gitignore matches a/b/c
- Whether a/.gitignore matches b/c
- Whether a/b/.gitignore matches c
- Whether .gitignore matches a/b
- Whether a/.gitignore matches b
- Whether .gitignore matches a
It might not look that bad because N=4 for the above example. But when N is
larger (ex. node_modules/../node_modules/../node_modules/..), things get much
worse.
This patch adds "caching" about whether a directory is ignored or not. For
example, if "a/b/" is ignored, the new code would skip checking subdirectories
(ex. "a/b/c/"). The time complexity is now roughly O(N) gitignore tests instead
of O(N^2), since we only did a gitignore check for a parent directory of a path
being tested once, and then cache the parent directory result in a boolean
value.
To be clear, for the first time checking a path which is not ignored, it still
needs O(N^2) for initializing the trees. But once it's initialized, the next
time checking a file in a same directory, will be O(N).
`LruCache` is replaced by `HashMap` since it does not support `.get` and the
code needs that to work.
The perf issue was previously documented as a "PERF" comment.
This diff removes it.
Reviewed By: DurhamG
Differential Revision: D7496058
fbshipit-source-id: f10895b8f0d7dcdde6faf9daeec5cd78a1f15a2b
Summary:
Previously every time a directory was deleted in subversion it would
call delete_entry which contained a ctx.walk() that would iterate over the
entire manifest. When many directories were deleted, this was
O(directories_deleted * size_of_repo) and very slow.
The new algorithm uses bisect to avoid iterating over the entire manifest.
Reviewed By: quark-zju
Differential Revision: D7529670
fbshipit-source-id: 763c1ad5ad46929d878848d150542f255ad96dd4
Summary:
We bailifchanged() later on, but the abort does an `up -C originalwc` before that. Let's bailifchanged() immediately.
A better version: reset the transaction and clear the statefile, instead of calling `abort()`.
Reviewed By: quark-zju
Differential Revision: D7538017
fbshipit-source-id: 8c490b366e495bb269c4d8c75b6144c535c8d54f
Summary:
Minor fixes to how the `previous locations of %s` line is printed:
- Start the pager before printing this line, so it gets included in the pager
output correctly.
- Avoid printing this line when using a custom output template. Previously it
was only skipped when using the `json` template. This now matches the logic
used to skip the `no recorded locations` line that was recommended in
D7512030.
Reviewed By: ryanmce
Differential Revision: D7537661
fbshipit-source-id: eb695dd98c06149701cf96acf5ec2eb277ea9cf3
Summary:
By using low-level APIs, `environ` could contain ill-formed entries that do
not have the `NAME=VAL` form:
#define _GNU_SOURCE
#include <unistd.h>
char *envp[] = { "X", 0 };
char *argv[] = { "hg", "log", 0 };
int main() {
return execvpe("chg", argv, envp);
}
Ignore them silently so they won't break chg.
Reviewed By: DurhamG
Differential Revision: D7537406
fbshipit-source-id: 3fef3d656383723d451fbfa29ba9a9fa170311d0
Summary:
Avoid printing "no recorded locations" directly to stdout when a format
template was specified. In particular this avoids printing non-JSON data
when using `-Tjson`.
We potentially could change this to print to stderr instead. However for now
I just followed the same pattern of checking the template as was done above for
the "previous locations" message.
Reviewed By: ryanmce
Differential Revision: D7512030
fbshipit-source-id: 2c32f07962fac4ca3d6bfd8f2ca3c4840b2a8a9b
Summary:
On Windows, the `mman` and `dirent` functionality can be provided by external
libs, which we would not like to mention in the `setup.py`. This change allows
us to to call `setup.py` with the right env and link with the right libs, same
approach as with headers.
Reviewed By: quark-zju
Differential Revision: D7535358
fbshipit-source-id: d0364cb305a9114260106844f9a3575a731419e9
Summary: We don't really need this separation.
Reviewed By: quark-zju
Differential Revision: D7535364
fbshipit-source-id: a480ddb62883e2e9044e06ff7b27201151632b71
Summary:
Run single DELETE query instead of using separate single DELETE statements because it is much faster and the common way in mysql world.
One query is better because:
* Only one plan will need to be generated
* Only one transaction will be involved
* Less overhead with ODBC calls and network traffic
* Any indexes will need to be refreshed just once, not many times.
Reviewed By: StanislavGlebik
Differential Revision: D7526427
fbshipit-source-id: 39f1d4605adb26d8690a9b39e783c792dee6ad87
Summary:
Reduce number of queries for infinitepush
Use bulk INSERT instead of using separate single-row INSERT statements, that is many many times faster.
('executemany' builds a single query for us)
(out max_allowed_packet is about 134M)
One query is better than 'for loop' because:
* Only one plan will need to be generated
* Only one transaction will be involved
* Less overhead with ODBC calls and network traffic
* Any indexes will need to be refreshed just once, not many times.
Reviewed By: StanislavGlebik
Differential Revision: D7504452
fbshipit-source-id: 9b0beb1ff572bf6c7b28892878d91e0cf4897e1f
Summary: Fix long line which breaks `test-check-code.t`.
Reviewed By: farnz
Differential Revision: D7532586
fbshipit-source-id: e5add9c9407b34bb3bd4cd701ba491e0e0ce1079
Summary:
fix `hg ssl` for Commit Cloud
In Commit Cloud world, obstore can sometimes have more knowledge than the unfiltered repo.
For example, in case of many amends on one client, some stuff can not be backed up
and not fetched to another client.
But we don't need to have all obsoleted versions in both repos,
obstore helps us to provide enough information to a user of what has happened
but we need to be carefull with operator []
Reviewed By: DurhamG
Differential Revision: D7516465
fbshipit-source-id: 7e109741829b26c30f20a7528b21b5ebc9b31dde
Summary:
Using windows libs like `_kernel32 = ctypes.windll.kernel32` means that
the `GetLastError` calls are useless.
(1317 is a non-existent process)
```
In [44]: k32 = ctypes.WinDLL('kernel32', use_last_error=True)
In [52]: ctypes.windll.kernel32.OpenProcess(0x0001, False, 1317)
Out[52]: 0
In [53]: ctypes.windll.kernel32.GetLastError()
Out[53]: 0
In [54]: k32.OpenProcess(0x0001, False, 1317)
Out[54]: 0
In [55]: k32.GetLastError()
Out[55]: 5
```
This behavior can explain why we can't auto-delete the stale lock file. See the `testpid` code:
```
def testpid(pid):
'''return True if pid is still running or unable to
determine, False otherwise'''
h = _kernel32.OpenProcess(_PROCESS_QUERY_INFORMATION, False, pid)
if h:
try:
status = _DWORD()
if _kernel32.GetExitCodeProcess(h, ctypes.byref(status)):
return status.value == _STILL_ACTIVE
finally:
_kernel32.CloseHandle(h)
return _kernel32.GetLastError() != _ERROR_INVALID_PARAMETER
```
The non-existent `pid` causes `h` to be `0` and `_kernel.GetLastError()` to also be `0` (see above why). This means that we think that this process exists!
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D7519457
fbshipit-source-id: c08f727228073962359e93db13f1fdb76f3699e6
Summary:
When a .backupfiles file is invalid, hg recover gives a not very
useful stack trace. This makes it a bit easier to debug.
Reviewed By: DurhamG
Differential Revision: D7482578
fbshipit-source-id: e8c7ad73bb14a38d2d43b636d60a3b77cd947331
Summary: When no cache has been set (`simplecache.caches=`) then there's no `name` set either. This exposed a logic error in this section of the code.
Reviewed By: ryanmce
Differential Revision: D7513976
fbshipit-source-id: ecbc7a8ac8c6eb23010d64ab8cbf9f9fb7d8f497
Summary:
Currently hg doesn't return anything from processing infinitepush parts.
However, mononoke will treat infinitepush parts just like normal hg parts.
So let's not panic if we see reply:changegroup part
Reviewed By: markbt
Differential Revision: D7513729
fbshipit-source-id: 8417a285bd49efadf2f2ef9ed80ad2f2b6d6bc94
Summary:
Currently, the `build_deb.py` writes the packages to the directory
from which the script was invoked. This leads to issues in integrating with our
continuous build setup. Therefore, this commit changes the script to write to a
known location.
Reviewed By: quark-zju
Differential Revision: D7511832
fbshipit-source-id: 85beaf37e92418cae932fd81c72065cf5d64c249
Summary:
move logic to update current rev to its new location (optional)
we are trying to update only if it is unambiguous
Reviewed By: DurhamG
Differential Revision: D7431940
fbshipit-source-id: 72e7fea7365a231c4d98ceb4cf4872a4db02d9ca
