Commit Graph

43253 Commits

Author SHA1 Message Date
Jun Wu
8f8adff716 treestate: add getter and setter of watchman clock to TreeState
Summary: Previously they cannot be changed.

Reviewed By: markbt

Differential Revision: D7769658

fbshipit-source-id: 4548eb90a82e9bd85fadf6a6f356cca7352fff0d
2018-04-30 19:10:45 -07:00
Jun Wu
224fe91344 treestate: add write methods to TreeState
Summary:
This allows writing `TreeState` state in two ways - save as a new file, or
incrementally update an existing file.

Reviewed By: markbt

Differential Revision: D7748822

fbshipit-source-id: 472b78af6cf7ea79968460a51ec824eaa96e4973
2018-04-30 19:10:45 -07:00
Jun Wu
a34b11ec8d treestate: make Tree write methods return BlockId
Summary: They are used by the next diff.

Reviewed By: markbt

Differential Revision: D7748834

fbshipit-source-id: 9562204975d83a8dce6eb80d2677387e24f8f0a0
2018-04-30 19:10:45 -07:00
Jun Wu
0923d108be treestate: add map-like operations to TreeState
Summary:
The method names are inspired by std HashMap. The types are slightly
different due to `Tree` implementation details.

Reviewed By: markbt

Differential Revision: D7748828

fbshipit-source-id: fc24481cdf0054c8e879d760082e192e52afc7f5
2018-04-30 19:10:45 -07:00
Jun Wu
b15a1b747f treestate: add Tree.get_mut
Summary:
The `Tree` object can return an `&mut` entry easily. Let's expose the
interface. This could be useful when the caller only wants to modify part of
the file state. For example, changing `copied` without touching anything
else.

Reviewed By: markbt

Differential Revision: D7748820

fbshipit-source-id: 430fa8ee310297c61866695a692134daf519e78d
2018-04-30 19:10:45 -07:00
Jun Wu
e07e4a99a1 treestate: add a TreeState struct
Summary:
Unlike TreeDirstate, this struct does not have two trees, and uses
FileStateV2.

Reviewed By: markbt

Differential Revision: D7748826

fbshipit-source-id: e637fad64e6b3e9b2a122e26a29fd04014181d6b
2018-04-30 19:10:45 -07:00
Martijn Pieters
a9f4167218 sparse: add filtering on field contents
Summary: This lets you select on substrings in fields and paths.

Reviewed By: quark-zju

Differential Revision: D7788826

fbshipit-source-id: f92b8cc646fd36f4cb3b8a4dc6116576db80eb42
2018-04-30 14:16:44 -07:00
Martijn Pieters
796523487a sparse: add a hint about using --verbose on hg sparse list
Summary: Calculation on how many profiles were hidden is only done when the hint is actually shown.

Reviewed By: quark-zju

Differential Revision: D7774192

fbshipit-source-id: a7196b2cc5640d0a7cb9c4d572e1a31ebfa41598
2018-04-30 14:16:44 -07:00
Martijn Pieters
2c10ce3145 sparse: add filtering options for fields present or missing
Summary:
This lets us produce a list of profiles with the title set, or find all profiles that are missing a description.

Convert the 'hidden' filter to use this feature, rather than special-case it in `_discover()`.

Reviewed By: quark-zju

Differential Revision: D7774029

fbshipit-source-id: 3bcba75e6da97bf0e560e11ce1ae7cbcee49ee45
2018-04-30 14:16:44 -07:00
Mark Thomas
cec43c1e77 infinitepush: enable compression of bundles
Summary:
Infinitepush stores bundles uncompressed, which can make them quite large.
Enable zstd compression of bundles, both when sending them to the server during
backups, and when storing the bundle in the bundle store on the server.

Reviewed By: quark-zju

Differential Revision: D7777371

fbshipit-source-id: ef9360bf05bf2a6d2b76e26cd40ad1e4ee8ae076
2018-04-30 02:49:18 -07:00
Jun Wu
cfec85ab00 fastannotate: disable forcefollow by default
Reviewed By: sid0

Differential Revision: D7764010

fbshipit-source-id: 16604e9dbeea382bc0e1feb55d30986d0e4ee0d1
2018-04-27 23:40:00 -07:00
Jun Wu
e21cf00d2e sparse: move some documentation to extension-level
Summary:
check-seclevel.py complains about using section markers inside a container.
The help text about sparse file format and config options can be put at
extension-level.

Reviewed By: mjpieters

Differential Revision: D7776355

fbshipit-source-id: 0b7d813c5eee352a054b21897682f6064f384829
2018-04-27 23:40:00 -07:00
Durham Goode
b1027ac19e make: use homebrew python path when available
Summary:
On OSX developers kept having to set their $PATH or change the make
file to actually build against the homebrew python. Let's build against it by
default if it exists.

Reviewed By: ryanmce

Differential Revision: D7791395

fbshipit-source-id: c69e41a469c5f94825814b4b30bc8ea144112167
2018-04-27 11:19:28 -07:00
Jun Wu
94e0297817 tests: fix test-clone-uncompressed.t
Summary: Got the other output on OSX builders.

Reviewed By: singhsrb

Differential Revision: D7790153

fbshipit-source-id: 913b5cf78c2e90933cb77d14a77e244b1ef063ba
2018-04-27 10:00:40 -07:00
Jun Wu
c0994a212f tests: stabilize test-fastmanifest-blackbox
Reviewed By: mjpieters

Differential Revision: D7729157

fbshipit-source-id: 1d546e974520c76e532152802e22fe01b091a9d2
2018-04-27 09:16:02 -07:00
Kaley Huang
c32047e854 p4syncimport: eliminate "p4 where" calls
Summary:
We currently call "p4 where" on all files in the new client spec, which is roughly O(all files in ovrsource), and it's only going to get worse as ovrsource grows :(
This diff makes the number of calls O(files added & removed between two client specs) by filtering them in the following way:
- get a full list of files from old client (L1)
- get a full list of files from new client  (L2)
- files_added = L2 - L1
- files_removed = L1 - L2
- `p4 where` on (files_added + files_removed)

How much speedup do we get?
from
  [2018-04-25 16:07:27,725]INFO:root: Start sync import in ovrsource-master
  [2018-04-25 17:35:35,873]INFO:root: Finish sync import in ovrsource-master after 5288.147963762283 seconds
to
  [2018-04-26 13:43:28,819]INFO:root: Start sync import in ovrsource-master
  [2018-04-26 13:51:35,575]INFO:root: Finish sync import in ovrsource-master after 486.7560772895813 seconds
on this change D7722798

Differential Revision: D7772403

fbshipit-source-id: 05a16343264007ee3ee466621da9da888c2368d7
2018-04-27 08:33:47 -07:00
Jun Wu
2b625d0753 wireproto: write eol for the remotefilelog error message
Summary:
"\n" will flush the line and that would probably solve the OSX test failure.

```
    --- tests/test-clone-uncompressed.t
    +++ tests/test-clone-uncompressed.t.err
    @@ -37,8 +37,8 @@
       > EOF
       $ hg clone --stream -U ssh://user@dummy/server blockedclone
       streaming all changes
    -  remote: unable to perform an implicit streaming clone - make sure remotefilelog is enabled
       abort: locking the remote repository failed
    +  remote: unable to perform an implicit streaming clone - make sure remotefilelog is enabled (no-eol)
       [255]
       $ hg clone --stream --config clone.requestfullclone=True -U ssh://user@dummy/server blockedclone
       streaming all changes
    ok
```

Reviewed By: DurhamG

Differential Revision: D7776998

fbshipit-source-id: f21c26e1bf7aa547cd79892f66521fb27cb2e77f
2018-04-26 13:34:29 -07:00
Jun Wu
9ce759a99a treestate: expose visit filtering via Tree.visit_advanced
Reviewed By: markbt

Differential Revision: D7748830

fbshipit-source-id: f3b41531e015fef90c01773ab65a4523ee72e7df
2018-04-25 17:38:20 -07:00
Jun Wu
53e7ab2a6c treestate: add file state filtering to Node.visit
Reviewed By: markbt

Differential Revision: D7748825

fbshipit-source-id: 2395ac8cc25fb4f4d3e6bdb5770616d859fcfab0
2018-04-25 17:38:20 -07:00
Jun Wu
23506bde19 treestate: merge Node.visit and Node.visit_changed
Summary:
They are similar. Merge into one single method. The `visit` method will be
extended to support other filtering features.

Reviewed By: markbt

Differential Revision: D7748829

fbshipit-source-id: 4388291945668a684808fe384341328ffd4ad2a8
2018-04-25 17:38:20 -07:00
Jun Wu
8cbe4d45c0 treestate: add serialization for Node<FileStateV2>
Reviewed By: markbt

Differential Revision: D7748832

fbshipit-source-id: bd7c6e8fce5b512068d86e16d441564e36565459
2018-04-25 17:38:20 -07:00
Jun Wu
b3da9a0262 treestate: add a compatibility layer for Node
Summary:
Allow `Node` type to work with both versions of file states. This is the
static dispatch approach that does not introduce runtime overhead.

Reviewed By: markbt

Differential Revision: D7748831

fbshipit-source-id: 4ac0386f9f93e55af1102b97a3510c8e872444a2
2018-04-25 17:38:20 -07:00
Jun Wu
cc0390192f treestate: add serialization for FileStateV2
Reviewed By: markbt

Differential Revision: D7748821

fbshipit-source-id: 56c7d7d81c86a8db05f6db2c8f6f02993cd07989
2018-04-25 17:38:20 -07:00
Jun Wu
75d50004f4 treestate: add serialization for StateFlags
Summary: It's just a thin wrapper about writing VLQ integers.

Reviewed By: markbt

Differential Revision: D7748835

fbshipit-source-id: 53a302afe51d551e49ac341901e2767d1a044946
2018-04-25 17:38:20 -07:00
Jun Wu
b796901d0c treestate: add StateFlags to Node
Summary:
This field stores the pre-computed aggregated state that helps fast path
traversal - if a state does not match the aggregated state, we can now skip
an entire tree quickly.

Reviewed By: markbt

Differential Revision: D7748823

fbshipit-source-id: 4b81ef5b911b4a21fdd46f8845ec217a75f5af8c
2018-04-25 17:38:20 -07:00
Jun Wu
71b99ae067 treestate: define a new file state object
Summary: The new `FileState` has a state bitflags field, and a "copied" information.

Reviewed By: markbt

Differential Revision: D7748824

fbshipit-source-id: a68687764e1b0c13252cb914673f2b16fa22d4ef
2018-04-25 17:38:19 -07:00
Mateusz Kwapich
052d0e3708 Command trait in rust
Summary:
We (me and Aida)  wanted to have something that could be used instead of Command decorator that we have in python
and we came up with the following trait.

Differential Revision: D7754930

fbshipit-source-id: 15f412d07045e7d8b229801ec3094664f78f801b
2018-04-25 16:00:36 -07:00
Mateusz Kwapich
86d3de94e5 demo binary
Summary: Let's commit a stub of demo binary depending on argparse that we'll work on this wekk

Differential Revision: D7752614

fbshipit-source-id: a811ea363d49e0fd56cc755b0abb74d89b4a3112
2018-04-25 16:00:36 -07:00
Mateusz Kwapich
21069062f4 argparse crate (derived from fbcode/scm/telemetry/telemetry/src)
Summary:
let's turn the wez's argparse library from telemetry into a separate
crate withing hg/lib. We'll experiment on it and if things go well we'll make
`telemetry` depend on that.

Reviewed By: quark-zju

Differential Revision: D7752615

fbshipit-source-id: 0814d91d704abdb746894a0289bf082e8d799b73
2018-04-25 16:00:36 -07:00
Durham Goode
e3f8f7fc38 clone: add stream_clone_options wireprotocol endpoint
Summary:
We want to allow blocking full repo streaming clones in certain
repositories (since they can take the lock and take a very long time) unless the
client has explicitly asked for it. The existing stream_out wire protocol has no
way of passing an option, so let's create a new endpoint.

Reviewed By: quark-zju

Differential Revision: D7763717

fbshipit-source-id: eace47143f8fdcc4c6e302b5c26678ccf56ca5d4
2018-04-25 13:54:54 -07:00
Martijn Pieters
ed857415c5 sparse: allow listing of profiles at a different revision
Summary: This is needed to be able to list profiles via SCMQuery, where there is a working copy at the null revision.

Reviewed By: quark-zju

Differential Revision: D7745434

fbshipit-source-id: b5ac236c36f9bafc9e0f305bc0892e0cd8bec628
2018-04-25 08:50:27 -07:00
Jun Wu
d15213f6f5 treedirstate: move non-Python part to a separate crate
Summary:
This makes it easier to modify and test the core logic without coupling with
the Python logic.

Reviewed By: markbt

Differential Revision: D7734012

fbshipit-source-id: 0d7b19198d85f6ca7314611256e9271be60070d1
2018-04-24 15:59:07 -07:00
Durham Goode
b66618d8c5 testing: no-op change
Summary:
I need a no-op change to test landing logic.

(Note: this ignores all push blocking failures!)

Reviewed By: phillco

Differential Revision: D7747338

fbshipit-source-id: 49388721418572b97b99971def951704059fc36a
2018-04-24 12:36:19 -07:00
Siddharth Agarwal
be7307cb48 hgsql: work around mysql-python-connector null byte mangling
Summary: See the comments for more.

Reviewed By: quark-zju

Differential Revision: D7738340

fbshipit-source-id: 20660c2ff7fdba1850c0fb9f34548db0084abf3d
2018-04-24 11:05:55 -07:00
Martijn Pieters
51dae5969e sparse: ignore non-existing profiles when listing
Summary: If there is a non-existing sparse profile active in .hg/sparse, skip it rather than bail out.

Reviewed By: quark-zju

Differential Revision: D7744957

fbshipit-source-id: 4e6a046c95f5fe6cf49a7093d3443340f70e8e2a
2018-04-24 10:21:22 -07:00
Liubov Dmitrieva
c0ddf154fb watchman client: add missing crates
Summary: add missing vendor crates as initial step

Reviewed By: DurhamG

Differential Revision: D7738538

fbshipit-source-id: 9c1c55a62ce28da755f98cc5100e5958db064d77
2018-04-23 23:34:28 -07:00
Kaley Huang
d29116477e p4syncimporter: cleanup
Summary:
as title: remove unused/old sync importer code

deadcodeskull

Differential Revision: D7714319

fbshipit-source-id: 38eb382027214b879ffb43f7049b83e38064ca8e
2018-04-23 16:32:41 -07:00
Kaley Huang
e701ca4c6e p4syncimoprter: reimplement
Summary: Curernt sync importer modifies mercurial storage data structures directly, which is error prone and hard to debug. This reimplements the sync importer with a higher level mercurial API (what `p4seqmport` uses).

Differential Revision: D7714322

fbshipit-source-id: 0269839b5ee3a4b45f166dce74dfd29c8ec5135a
2018-04-23 16:32:41 -07:00
Alexandre Marin
8a6a1da5d7 p4fastimport - debugscanlfs - remove client option
Summary:
D7676378 would break this as it changes how parse_where behaves. This is the only
callsite left behind. AFAICT it actually wouldn't work, as it passes the hg path
to p4 where.

This change removes the option and related code.

Differential Revision: D7696956

fbshipit-source-id: a27077a9540369b5c77692185f317cf5395789ba
2018-04-23 13:21:18 -07:00
Durham Goode
e1ac4b80e0 metrics: add performance logging to important locations
Summary:
This logs timing data for:
- update
  - merge.update
  - mergedriver
  - applyupdates
  - workers
- fetches
  - pulls
  - remotefilelog prefetches
  - treemanifest prefetches
- status
  - dirstate walks
  - fsmonitor walks
  - watchman queries

Hopefully this will let us narrow down where time is going in a number of cases
where the profile data is ambigiuous or hard to come by.

Reviewed By: quark-zju

Differential Revision: D7681026

fbshipit-source-id: e6fe65c9a4d2f4e128f62ccb767a7cbe73b2649a
2018-04-23 10:51:21 -07:00
Jun Wu
4b81417575 setup: remove Cargo.lock automatically
Summary:
When building Rust extensions, remove Cargo.lock automatically so they
always get regenerated. This avoids issues when vendored crates are upgraded
but the local Cargo.lock is still using old versions.

Reviewed By: DurhamG

Differential Revision: D7562864

fbshipit-source-id: 25187ef2c5a9be8d6059f02a1113f3ddb55ebe99
2018-04-21 03:42:58 -07:00
Jun Wu
58029547d6 cleanup: remove Cargo.lock from check-in
Summary:
They cause a lot of churn when updating vendored crates. It's unnecessary to
check them in because the vendored crates basically pinned versions of all
dependencies.

Also clean up .gitignore, since it does not support "syntax:re".

Reviewed By: DurhamG

Differential Revision: D7562533

fbshipit-source-id: b9fcade6f38c0652a1b19ac4f90827dac7eed181
2018-04-21 03:42:58 -07:00
Liubov Dmitrieva
a0beff1460 commitcloud: use cloudsync related phrases in smartlog
Summary:
when a user is joined to commitcloud workspace, we should provide
commit cloud related messages in the smartlog

Reviewed By: markbt, StanislavGlebik

Differential Revision: D7705709

fbshipit-source-id: ba2c246f115542e4f49446284cd831d8c266245f
2018-04-20 11:29:24 -07:00
Mark Thomas
53c721643a setup.py: add commitcloud to setup.py
Differential Revision: D7709932

fbshipit-source-id: 5ab6c193243b73a188c52dd5e90921ce26e525d5
2018-04-20 11:15:40 -07:00
Mark Thomas
3902417b93 infinitepush: limit the list of backups to just the most recent few
Summary:
Make `hg getavailablebackups` limit the list to 5 (configurable) backups by
default.  Add a new `--all` option to list them all.

Reviewed By: quark-zju

Differential Revision: D7671210

fbshipit-source-id: 743e0032e11b5d21b1544b695d28e8818c44085e
2018-04-20 10:10:04 -07:00
Mark Thomas
6f3e39c8ee infinitepush: add backupdelete command to delete backups
Summary: Add the `hg backupdelete` commands, that lets users delete their old backups.

Differential Revision: D7670524

fbshipit-source-id: d0a12e1aa7c770cf9313e792d859c9a76ff43d9f
2018-04-20 10:10:04 -07:00
Mark Thomas
1a9ca888a8 infinitepush: don't push empty backups in new clones
Summary:
If the infinitepush backup state is empty (indicating this is probably a new
clone of a repository), and there are no commits or bookmarks to be backed up,
don't push a backupbundle that clears all the heads and bookmarks of any
backups that have the same servername and path.  This might be a reclone of the
repository, and the user might want to restore the backup.

When the first commit or bookmark is made in the new repository, the old
backup will be cleared at that point.

Differential Revision: D7669728

fbshipit-source-id: bec859bbbbee6b631efb4483a8fedbad8f3bcbac
2018-04-20 10:10:03 -07:00
Jun Wu
9e4122b90d setup: do not use system re2 include path
Summary:
Swap the include paths so it looks for re2 source specified by `--re2-path`
first. This should resolve one of the recent OSX build failure issues.

Reviewed By: singhsrb

Differential Revision: D7702478

fbshipit-source-id: 303f9efc6009da3b876b63b0d5162437d06e7a04
2018-04-19 17:47:37 -07:00
Alexandre Marin
69005698f4 importer - call p4 where with multiple paths
Summary:
We currently call p4 where with one path at a time, but it accepts a list.
This change takes advantage of that, batching p4 where calls, which speeds up
importing.

Differential Revision: D7676378

fbshipit-source-id: 4a6747458555a60dd5f385604f2a25d595af947d
2018-04-19 17:08:30 -07:00
Jun Wu
9cf21d7754 obsshelve: remove 100 patches per branch limit
Summary:
Backports the `shelve.py` change [1] to `obsshelve.py`.

[1]: 4107eb8a56

Reviewed By: DurhamG

Differential Revision: D7691913

fbshipit-source-id: 642eade39c50f6a161dae5af51b7ab54025de322
2018-04-19 15:05:28 -07:00