Commit Graph

135 Commits

Author SHA1 Message Date
Saurabh Singh
a4f5e8aefc add tests for empty union date store
Summary:
These tests just test the expected outcome when the union data store
is empty.

Reviewed By: quark-zju

Differential Revision: D8018975

fbshipit-source-id: a2cc4c87509b857dbf5f6af506f165ea62080db8
2018-05-15 18:37:47 -07:00
Saurabh Singh
cab75fbb90 add common traits for Key
Summary:
This commit derives the common traits for the Key type just as we did
for the Node type in D7872300.

Reviewed By: quark-zju

Differential Revision: D8018973

fbshipit-source-id: 566a69be16d74529c6eb5f157b84de25835f780f
2018-05-15 18:37:47 -07:00
Saurabh Singh
88a2d8ff6b implement quickcheck::Arbitrary for Key in revisionstore
Summary:
We need to implement `quickcheck::Arbitrary` for Key so that it can be
used for the quickcheck tests.

Reviewed By: quark-zju

Differential Revision: D8018977

fbshipit-source-id: dbdbb34fbd7eaeb18321eafec4604d752f496a4d
2018-05-15 18:37:47 -07:00
Saurabh Singh
0b5b994973 implement quickcheck::Arbitrary for Node type in revisionstore
Summary:
We need to implement `quickcheck::Arbitrary` for Node so that it can
be used for quickcheck tests.

Reviewed By: quark-zju

Differential Revision: D8018978

fbshipit-source-id: ceda99622370bee6e9d05b839f9856c0526f553c
2018-05-15 18:37:47 -07:00
Saurabh Singh
411736dc7c add 'quickcheck' crate to revisionstore
Summary:
I am planning to use the `quickcheck` crate for testing the union data
store. This commit just adds the crate to the revisionstore.

Reviewed By: quark-zju

Differential Revision: D8018974

fbshipit-source-id: d390deeb01aa7d1bf1e66bb5bc948d48bd3f269e
2018-05-15 18:37:47 -07:00
Saurabh Singh
8688e1cc5e union data store: union data store implementation in Rust
Summary:
This commit just introduces the `UnionDataStore` and implements the
`DataStore` trait for it.

Reviewed By: quark-zju

Differential Revision: D7801615

fbshipit-source-id: 14eabd2aa1b1e085de94aec126a7108231ec6e8d
2018-05-15 18:37:47 -07:00
Saurabh Singh
d2b9c6c6ac union store: introduce common type for the union store implmentation
Summary:
We will be implmenting multiple union stores and therefore, it makes
sense to encapsulate the common logic in its own type. This also abstracts the
usage of `RefCell` within the union store.

Reviewed By: jsgf

Differential Revision: D7884651

fbshipit-source-id: a74b6d9df5ee0d7d04359219e276fd5713b3a00b
2018-05-15 18:37:47 -07:00
Saurabh Singh
bcb7ac0b32 node: add common traits for the node type
Summary:
Based on the review comments for D7801615, this commit adds the common traits
for the `Node` type

Reviewed By: jsgf

Differential Revision: D7872300

fbshipit-source-id: 44dedfc3ec0e18ac0dee0dcfc5f7dfc4aff2511d
2018-05-15 18:37:47 -07:00
Durham Goode
8c6e5fd964 mpatch: add rust bindings
Summary:
Adds rust bindings around the existing mpatch c library.

Also fixes a bug in mpatch where it could reference uninitialized memory.

Reviewed By: quark-zju

Differential Revision: D7769299

fbshipit-source-id: bcc21df85c97ef6f5537ebff8fbf1b350ee64fc3
2018-05-14 16:06:32 -07:00
Durham Goode
18697e3fb3 hg: implement getmissing() for Rust hgstore
Summary:
Initial implementation of getmissing for a simple Rust pass through
data store. Future diffs will extend this to implement the union data store
completely in Rust.

Reviewed By: quark-zju

Differential Revision: D7632405

fbshipit-source-id: e660d33f8231410805cfaba6d77c56f27b002f8e
2018-05-14 12:05:13 -07:00
Durham Goode
f2b4d7f2e0 hg: implement getmeta() for Rust hgstore
Summary:
An initial implementation of getmeta for the Rust hg data store. Future
diffs will add more functionality.

Reviewed By: quark-zju

Differential Revision: D7632404

fbshipit-source-id: 53bd3b96b777bc3c5aef2b4d07ce1a9d9a5a52ed
2018-05-14 12:05:13 -07:00
Durham Goode
39dde8552d hg: implement getdeltachain() for Rust hgstore
Summary:
An initial implementation of getdeltachain for a simple pass through
data store. Future diffs will add additional functionality.

Reviewed By: quark-zju

Differential Revision: D7632407

fbshipit-source-id: 1a38089ba8ea70f8772af95afd871ee493082d80
2018-05-14 12:05:13 -07:00
Durham Goode
880ff5d0a9 hg: implement datastore.get() for Rust hgstore
Summary:
Implements the get function for a simple pass through rust data store
layer. Future diffs will implement more functions, and then later we will
implement the entire union data store in Rust.

Reviewed By: quark-zju

Differential Revision: D7632403

fbshipit-source-id: 3a1d0a8500e3110213d70dc1cff637cf8eadd809
2018-05-14 12:05:13 -07:00
Durham Goode
a97e97e413 hg: initial boiler plate for new hgstore crate
Summary:
This will contain all the Python centric hg store code that will let
Python call into the Rust storage layer.

Reviewed By: quark-zju

Differential Revision: D7632406

fbshipit-source-id: 6b7bcc8f47a23e9c0121e1f92de1137369bf584e
2018-05-14 12:05:12 -07:00
Jun Wu
a2e7d1cfe3 treestate: implement Rand for FileStateV2
Summary: This will be used in tests.

Differential Revision: D7769655

fbshipit-source-id: 27647685848c03f56740f49361dd286abdef8e33
2018-05-10 16:40:25 -07:00
Mateusz Kwapich
be9c5d754b fix pyflakes.t failure
Summary: Our CI didn't catch when I landed the previous diff.

Reviewed By: singhsrb

Differential Revision: D7834066

fbshipit-source-id: a51c2a294ea550917836f8b1eede2570838b60b7
2018-05-01 13:44:40 -07:00
Mateusz Kwapich
626759e564 add short_list support to argparse
Summary:
We mark some hg commands as appearing in short help, let's add this
to our parser

Reviewed By: quark-zju

Differential Revision: D7779270

fbshipit-source-id: 0c2b790f1994205ae4dbf7cd12ac3ba7f5ef39ad
2018-05-01 04:26:26 -07:00
Mateusz Kwapich
cc3ed51d7f importer of command defninitions from python hg to rust
Summary:
Let's check-in the definitions for now. In the future those should be generated
every build with all the extensions enabled - like in prod.

Reviewed By: quark-zju

Differential Revision: D7779273

fbshipit-source-id: f0d5c5260be74c5f64c0945004bf60399a6e8c4c
2018-05-01 04:26:26 -07:00
Jun Wu
8f8adff716 treestate: add getter and setter of watchman clock to TreeState
Summary: Previously they cannot be changed.

Reviewed By: markbt

Differential Revision: D7769658

fbshipit-source-id: 4548eb90a82e9bd85fadf6a6f356cca7352fff0d
2018-04-30 19:10:45 -07:00
Jun Wu
224fe91344 treestate: add write methods to TreeState
Summary:
This allows writing `TreeState` state in two ways - save as a new file, or
incrementally update an existing file.

Reviewed By: markbt

Differential Revision: D7748822

fbshipit-source-id: 472b78af6cf7ea79968460a51ec824eaa96e4973
2018-04-30 19:10:45 -07:00
Jun Wu
a34b11ec8d treestate: make Tree write methods return BlockId
Summary: They are used by the next diff.

Reviewed By: markbt

Differential Revision: D7748834

fbshipit-source-id: 9562204975d83a8dce6eb80d2677387e24f8f0a0
2018-04-30 19:10:45 -07:00
Jun Wu
0923d108be treestate: add map-like operations to TreeState
Summary:
The method names are inspired by std HashMap. The types are slightly
different due to `Tree` implementation details.

Reviewed By: markbt

Differential Revision: D7748828

fbshipit-source-id: fc24481cdf0054c8e879d760082e192e52afc7f5
2018-04-30 19:10:45 -07:00
Jun Wu
b15a1b747f treestate: add Tree.get_mut
Summary:
The `Tree` object can return an `&mut` entry easily. Let's expose the
interface. This could be useful when the caller only wants to modify part of
the file state. For example, changing `copied` without touching anything
else.

Reviewed By: markbt

Differential Revision: D7748820

fbshipit-source-id: 430fa8ee310297c61866695a692134daf519e78d
2018-04-30 19:10:45 -07:00
Jun Wu
e07e4a99a1 treestate: add a TreeState struct
Summary:
Unlike TreeDirstate, this struct does not have two trees, and uses
FileStateV2.

Reviewed By: markbt

Differential Revision: D7748826

fbshipit-source-id: e637fad64e6b3e9b2a122e26a29fd04014181d6b
2018-04-30 19:10:45 -07:00
Jun Wu
9ce759a99a treestate: expose visit filtering via Tree.visit_advanced
Reviewed By: markbt

Differential Revision: D7748830

fbshipit-source-id: f3b41531e015fef90c01773ab65a4523ee72e7df
2018-04-25 17:38:20 -07:00
Jun Wu
53e7ab2a6c treestate: add file state filtering to Node.visit
Reviewed By: markbt

Differential Revision: D7748825

fbshipit-source-id: 2395ac8cc25fb4f4d3e6bdb5770616d859fcfab0
2018-04-25 17:38:20 -07:00
Jun Wu
23506bde19 treestate: merge Node.visit and Node.visit_changed
Summary:
They are similar. Merge into one single method. The `visit` method will be
extended to support other filtering features.

Reviewed By: markbt

Differential Revision: D7748829

fbshipit-source-id: 4388291945668a684808fe384341328ffd4ad2a8
2018-04-25 17:38:20 -07:00
Jun Wu
8cbe4d45c0 treestate: add serialization for Node<FileStateV2>
Reviewed By: markbt

Differential Revision: D7748832

fbshipit-source-id: bd7c6e8fce5b512068d86e16d441564e36565459
2018-04-25 17:38:20 -07:00
Jun Wu
b3da9a0262 treestate: add a compatibility layer for Node
Summary:
Allow `Node` type to work with both versions of file states. This is the
static dispatch approach that does not introduce runtime overhead.

Reviewed By: markbt

Differential Revision: D7748831

fbshipit-source-id: 4ac0386f9f93e55af1102b97a3510c8e872444a2
2018-04-25 17:38:20 -07:00
Jun Wu
cc0390192f treestate: add serialization for FileStateV2
Reviewed By: markbt

Differential Revision: D7748821

fbshipit-source-id: 56c7d7d81c86a8db05f6db2c8f6f02993cd07989
2018-04-25 17:38:20 -07:00
Jun Wu
75d50004f4 treestate: add serialization for StateFlags
Summary: It's just a thin wrapper about writing VLQ integers.

Reviewed By: markbt

Differential Revision: D7748835

fbshipit-source-id: 53a302afe51d551e49ac341901e2767d1a044946
2018-04-25 17:38:20 -07:00
Jun Wu
b796901d0c treestate: add StateFlags to Node
Summary:
This field stores the pre-computed aggregated state that helps fast path
traversal - if a state does not match the aggregated state, we can now skip
an entire tree quickly.

Reviewed By: markbt

Differential Revision: D7748823

fbshipit-source-id: 4b81ef5b911b4a21fdd46f8845ec217a75f5af8c
2018-04-25 17:38:20 -07:00
Jun Wu
71b99ae067 treestate: define a new file state object
Summary: The new `FileState` has a state bitflags field, and a "copied" information.

Reviewed By: markbt

Differential Revision: D7748824

fbshipit-source-id: a68687764e1b0c13252cb914673f2b16fa22d4ef
2018-04-25 17:38:19 -07:00
Mateusz Kwapich
052d0e3708 Command trait in rust
Summary:
We (me and Aida)  wanted to have something that could be used instead of Command decorator that we have in python
and we came up with the following trait.

Differential Revision: D7754930

fbshipit-source-id: 15f412d07045e7d8b229801ec3094664f78f801b
2018-04-25 16:00:36 -07:00
Mateusz Kwapich
86d3de94e5 demo binary
Summary: Let's commit a stub of demo binary depending on argparse that we'll work on this wekk

Differential Revision: D7752614

fbshipit-source-id: a811ea363d49e0fd56cc755b0abb74d89b4a3112
2018-04-25 16:00:36 -07:00
Mateusz Kwapich
21069062f4 argparse crate (derived from fbcode/scm/telemetry/telemetry/src)
Summary:
let's turn the wez's argparse library from telemetry into a separate
crate withing hg/lib. We'll experiment on it and if things go well we'll make
`telemetry` depend on that.

Reviewed By: quark-zju

Differential Revision: D7752615

fbshipit-source-id: 0814d91d704abdb746894a0289bf082e8d799b73
2018-04-25 16:00:36 -07:00
Jun Wu
d15213f6f5 treedirstate: move non-Python part to a separate crate
Summary:
This makes it easier to modify and test the core logic without coupling with
the Python logic.

Reviewed By: markbt

Differential Revision: D7734012

fbshipit-source-id: 0d7b19198d85f6ca7314611256e9271be60070d1
2018-04-24 15:59:07 -07:00
Liubov Dmitrieva
c0ddf154fb watchman client: add missing crates
Summary: add missing vendor crates as initial step

Reviewed By: DurhamG

Differential Revision: D7738538

fbshipit-source-id: 9c1c55a62ce28da755f98cc5100e5958db064d77
2018-04-23 23:34:28 -07:00
Jun Wu
00a9659536 radixbuf: replace criterion with minibench
Summary: Similar with D7440249. See D7440254 for motivation.

Reviewed By: DurhamG

Differential Revision: D7562195

fbshipit-source-id: b11eb4f47375a2a2d70be96ebcfe2fefe1e0aaad
2018-04-17 18:54:39 -07:00
Jun Wu
3408c53051 vlqencoding: replace criterion with minibench
Summary: Similar with D7440249. See D7440254 for motivation.

Reviewed By: DurhamG

Differential Revision: D7562196

fbshipit-source-id: e90c623bd9576de49c3d4990ac93c105238d219c
2018-04-17 18:54:39 -07:00
Jun Wu
3ec8c3af00 minibench: simple test filtering support
Summary:
Now it's possible to filter tests like:

  cargo bench --bench index --verbose -- TEST_KEYWORD

Useful for profiling specific test.

Reviewed By: DurhamG

Differential Revision: D7562174

fbshipit-source-id: 9c7fe13a0541bd3dda7a9c1acf95c91513b633f2
2018-04-17 18:54:39 -07:00
Jun Wu
40a88364be indexedlog: replace div with shr to make checksum faster
Summary:
Spot `div` slowness using Linux's `perf` tool.

        |    Disassembly of section .text:
        |
        |    0000000000018990 <indexedlog::checksum_table::ChecksumTable::check_range>:
        |    _ZN10indexedlog14checksum_table13ChecksumTable11check_range17h2303c96b1e035e20E():
   1.36 |      push   %rax
   0.18 |      mov    %rdx,%r8
   0.54 |      mov    $0x1,%cl
        |      test   %r8,%r8
        |      je     60
   0.54 |      add    %rsi,%r8
   0.72 |      cmp    0x30(%rdi),%r8
        |      ja     64
   0.27 |      mov    0x28(%rdi),%r9
   0.27 |      test   %r9,%r9
        |      je     6a
   0.36 |      add    $0xffffffffffffffff,%r8
   0.18 |      xor    %edx,%edx
   0.45 |      mov    %rsi,%rax
   0.36 |      div    %r9
  43.72 |      mov    %rax,%rsi
        |      xor    %edx,%edx
        |      mov    %r8,%rax
   0.18 |      div    %r9
  42.82 |      add    $0x1,%rax
   0.09 |      cmp    %rax,%rsi
        |      jae    60
   2.17 |      cmpq   $0x0,0x60(%rdi)
        |      je     78
        |      mov    0x50(%rdi),%rcx
        |      cmpb   $0x0,(%rcx)
   1.63 |      sete   %cl
   0.18 |      xchg   %ax,%ax
        |50:   test   $0x1,%cl
        |      je     64
   0.45 |      add    $0x1,%rsi
   0.81 |      mov    $0x1,%cl
   0.09 |      cmp    %rax,%rsi
        |      jb     50
        |60:   mov    %ecx,%eax
        |      pop    %rcx
   2.62 |      retq
        |64:   xor    %ecx,%ecx
        |      mov    %ecx,%eax
        |      pop    %rcx
        |      retq
        |6a:   lea    panic_loc.a.llvm.9800112514578621117,%rdi
        |      callq  core::panicking::panic
        |      ud2
        |78:   lea    panic_bounds_check_loc.7.llvm.9800112514578621117,%rdi
        |      xor    %esi,%esi
        |      xor    %edx,%edx
        |      callq  core::panicking::panic_bounds_check
        |      ud2

Change `chunk_size` to `chunk_size_log`. Replace `div` with `shr` to make it
significantly faster:

Before:

  index lookup (memory)           1.118 ms
  index lookup (disk, no verify)  2.078 ms
  index lookup (disk, verified)   7.687 ms

After:

  index lookup (memory)           1.066 ms
  index lookup (disk, no verify)  1.992 ms
  index lookup (disk, verified)   3.591 ms

Reviewed By: DurhamG, markbt

Differential Revision: D7554992

fbshipit-source-id: c24189ced722d880af6ca0d64967eb762363d9e3
2018-04-17 18:54:39 -07:00
Jun Wu
f25c152d01 indexedlog: add a test about checksum
Summary:
Add a test that bitflips the index content, and make sure reading the index
would trigger an error.

Due to run-time performance difference, the release version tests 2-byte key
while the debug version only tests 1-byte key.

The header byte was not verified. Now it is verified.

Reviewed By: DurhamG

Differential Revision: D7517134

fbshipit-source-id: b3d8665ff4ac08c1a70db8d21122ba241913a2ed
2018-04-17 18:54:39 -07:00
Jun Wu
9ce455769c indexedlog: avoid writing unused entries due to leaf split
Summary:
In "split_leaf" "Example 3" case, the old leaf entry (and its key) becomes
unused. Writing them to disk is unnecessary. This patch adds "unused" marker
so they could be marked and skipped inside flush().

No visible performance change:

  index insertion                 3.710 ms
  index flush                     3.717 ms
  index lookup (memory)           1.128 ms
  index lookup (disk, no verify)  1.993 ms
  index lookup (disk, verified)   7.866 ms

Reviewed By: DurhamG

Differential Revision: D7517139

fbshipit-source-id: 253c878bc4b3762382c424777dfa779b3868e851
2018-04-17 18:54:38 -07:00
Kostia Balytskyi
fa03821500 hg: fix test-checks broken in windows treemanifest stack
Reviewed By: quark-zju

Differential Revision: D7560133

fbshipit-source-id: 98b016d0911aaecc1058263c134a5e4ecd0be9e5
2018-04-13 21:51:50 -07:00
Kostia Balytskyi
171ca13ebe hg: add a dirent.h portability header
Summary:
This will either include system dirent on POSIX or a vendored dirent from
folly on Windows.

`/no-check-code` is here because it's everywhere across hg's .c codebase.

Differential Revision: D7555759

fbshipit-source-id: dc55926e83e17976930522277ed7fe6ce41f32f7
2018-04-13 21:51:50 -07:00
Kostia Balytskyi
6be12d9dae hg: install mman as a third-party dep before building hg
Summary: This is needed for `treemanifest`.

Differential Revision: D7555758

fbshipit-source-id: 24d7dac292a62b0f3cabed1cbc0cd39e0b19a470
2018-04-13 21:51:50 -07:00
Jun Wu
bdbf60f28d xdiff: backport upstream changes
Summary:
I did some extra xdiff changes in upstream, namely:

  - Remove unused features
  - Replace "long" (32-bit in MSVC) with int64_t to support large files
  - Add comment on some key variables

This backports them. It also includes Matt's fixes about Windows compatibility.

Reviewed By: ryanmce

Differential Revision: D7223939

fbshipit-source-id: 9287d5be22dae4ab41b05b3a4c160d836b5714a6
2018-04-13 21:51:48 -07:00
Jun Wu
3ffa0f28e2 gitignore: avoid quadratic behavior
Summary:
The correct gitignore matcher needs O(N^2) time to check a path which is N
directory deep. For example, to check "a/b/c/d", it needs to check:

  - Whether .gitignore matches a/b/c/d
  - Whether a/.gitignore matches b/c/d
  - Whether a/b/.gitignore matches c/d
  - Whether a/b/c/.gitignore matches d

  - Whether .gitignore matches a/b/c
  - Whether a/.gitignore matches b/c
  - Whether a/b/.gitignore matches c

  - Whether .gitignore matches a/b
  - Whether a/.gitignore matches b

  - Whether .gitignore matches a

It might not look that bad because N=4 for the above example. But when N is
larger (ex. node_modules/../node_modules/../node_modules/..), things get much
worse.

This patch adds "caching" about whether a directory is ignored or not. For
example, if "a/b/" is ignored, the new code would skip checking subdirectories
(ex. "a/b/c/"). The time complexity is now roughly O(N) gitignore tests instead
of O(N^2), since we only did a gitignore check for a parent directory of a path
being tested once, and then cache the parent directory result in a boolean
value.

To be clear, for the first time checking a path which is not ignored, it still
needs O(N^2) for initializing the trees. But once it's initialized, the next
time checking a file in a same directory, will be O(N).

`LruCache` is replaced by `HashMap` since it does not support `.get` and the
code needs that to work.

The perf issue was previously documented as a "PERF" comment.
This diff removes it.

Reviewed By: DurhamG

Differential Revision: D7496058

fbshipit-source-id: f10895b8f0d7dcdde6faf9daeec5cd78a1f15a2b
2018-04-13 21:51:48 -07:00
Jun Wu
ac52e4a6fb indexedlog: add a test against std hashmap for multi-values
Summary: Since we now have the ability to store multiple values. Add a test.

Reviewed By: DurhamG

Differential Revision: D7472880

fbshipit-source-id: 85b1c69245ac7f0c4702daf22a02f5e5072f0924
2018-04-13 21:51:46 -07:00