Commit Graph

15 Commits

Author SHA1 Message Date
Durham Goode
d633ef7216 arc: update config to work with external phabricator
Summary:
Let's switch to upstream phabricator for our code reviews. We keep the
internal configs around, and they can be accessed by swapping the .arcconfig
symlink if necessary.

Test Plan: hgarc lint, hgarc unit, hgarc diff

Reviewers: sid0, mitrandir, quark

Reviewed By: mitrandir, quark

Differential Revision: https://hg-phab.durin42.com/D1
2017-06-29 13:42:07 -07:00
Jun Wu
f1be725a71 traceprof: new extension to perform accurate profiling
Summary:
Instead of sampling periodically like the default profiler `statprof`, trace
every function call so the report is accurate. The output is designed to be
similar to `statprof`.

Written in C++ and inline assembly to minimize overhead. Cython is used as a
glue layer.

Comparing to `lsprof`, the overhead is similarly small. The major difference is
this profiler cares about "frame object" and records `frame->f_back` (parent
frame) so it could draw a statprof-like tree of the callgraph. `lsprof` records
"code object" instead and could have difficulty to deal with recursive calls.

The following code could demostrate the difference:

```
def fib(x):
    if x < 2:
        return x
    return fib(x - 1) + fib(x - 2)

# lsprof - recorded 1 code object for fib and no way to figure out callgraph
import _lsprof
p = _lsprof.Profiler()
p.enable(subcalls=True)
fib(5)
p.disable()
for s in p.getstats():
    print(s)

'''output:
_lsprof.profiler_entry(code="<method 'disable' of '_lsprof.Profiler' objects>", callcount=1, reccallcount=0, totaltime=0.0, inlinetime=0.0, calls=None)
_lsprof.profiler_entry(code=<code object fib at 0x7f27ca587530, file "a.py", line 1>, callcount=15, reccallcount=14, totaltime=8e-06, inlinetime=8e-06, calls=[_lsprof.profiler_subentry(code=<code object fib at 0x7f27ca587530, file "a.py", line 1>, callcount=14, reccallcount=12, totaltime=6e-06, inlinetime=6e-06)])
'''

# traceprof (with timethreshold set to 0) - callgraph drawn correctly
import sys
from hgext3rd import traceprof
with traceprof.profile(None, sys.stderr):
    fib(5)

'''output:
    | <module>                                  a.py:1
  0  \ fib                                      a.py:1
  0    \ fib                                    a.py:1
  0      \ fib                                  a.py:1
  0        \ fib                                a.py:1
  0         | fib                               a.py:1
  0        \ fib                                a.py:1
  0      \ fib                                  a.py:1
  0       | fib                                 a.py:1
  0    \ fib                                    a.py:1
  0     | fib                                   a.py:1
     \ __exit__                                 contextlib.py:21
'''
```

There are some performance / correctness tradeoffs. The current implementation
takes `frame` address and several (but not theoretically complete) fields to
hash a frame.  That could in theory lead to inaccurate frame information being
recorded. But hashing all fields recursively incurs significant overhead. The
related logic is the `hashframe` function, which could be tweaked later.

This is an initial version. It probably does not cover all corner cases like
exception handling well. And it may duplicate a same frame multiple times
because of the way it hashes a frame. The duplicated frame issue could be
solved by adding post-profiling filtering though.

Here is a real-world example of `hg log -r .` on `hg-committed`:

```
      | <module>                                hg:10
      | run                                     dispatch.py:81
      | dispatch                                dispatch.py:132
      | _runcatch                               dispatch.py:202
      | _callcatch                              dispatch.py:309
      | callcatch                               scmutil.py:133
      | _runcatchfunc                           dispatch.py:215
      | _dispatch                               dispatch.py:744
    2  \ repository                             hg.py:166
    2   | _peerorrepo                           hg.py:151
    2   | instance                              localrepo.py:2025
    2   | __init__                              localrepo.py:275
  863  \ runcommand                             dispatch.py:656
  863   | _runcommand                           dispatch.py:926
  863   | <lambda>                              dispatch.py:918
  863   | check                                 util.py:1059
  863   | log                                   commands.py:3262
  480    \ getlogrevs                           cmdutil.py:2236
  479     | _logrevs                            cmdutil.py:2190
  479     | revrange                            scmutil.py:429
  479     | anyrevs                             localrepo.py:622
  478     | mfunc                               revset.py:2275
  478     | __init__ (4 times)                  smartset.py:1033
  478     | __init__ (4 times)                  smartset.py:919
  478     | __len__ (4 times)                   localrepo.py:587
  478     | changelog                           repoview.py:309
  477     | filterrevs (11 times)               repoview.py:260
  477     | computehidden                       repoview.py:171
  455      \ hideablerevs                       repoview.py:24
  455       | getrevs                           obsolete.py:1114
  455       | _computeobsoleteset               obsolete.py:1143
    3        \ __get__ (2 times)                localrepo.py:76
    7        \ node (5760 times)                changelog.py:359
    4         | node (8938 times)               revlog.py:482
    9        \ __get__ (5760 times)             localrepo.py:76
  428        \ __get__                          util.py:798
  428         | successors                      obsolete.py:672
  225          \ __get__                        util.py:798
  224           | _all                          obsolete.py:662
    6            \ tryread                      vfs.py:32
    6             | read                        vfs.py:60
  200            \ _readmarkers                 obsolete.py:442
* 199             | _fm1readmarkers             obsolete.py:430
   15            \ _checkinvalidmarkers         obsolete.py:523
* 203          \ _addsuccessors                 obsolete.py:504
    3      \ tryreadcache                       repoview.py:166
    3       | cachehash                         repoview.py:98
    3       | heads                             localrepo.py:1884
   17      \ <genexpr> (5225 times)             repoview.py:191
   15       | __contains__ (5226 times)         ancestor.py:334
    5       | parentrevs (4010 times)           changelog.py:371
    2       | parentrevs (4332 times)           revlog.py:479
    9    \ pager                                ui.py:843
    9     | _runpager                           ui.py:906
    8     | __init__                            subprocess.py:330
    8     | _execute_child                      subprocess.py:880
    5     | _eintr_retry_call                   subprocess.py:118
  374    \ show                                 cmdutil.py:1308
  374     | _show                               cmdutil.py:1316
  365      \ _changesetlabels                   cmdutil.py:1266
  365       | troubled                          context.py:225
   12        \ unstable (2 times)               context.py:207
   12         | getrevs                         obsolete.py:1114
   12         | _computeunstableset             obsolete.py:1154
    3          \ set (322 times)                localrepo.py:610
    3            \ revs (3 times)               localrepo.py:593
    8            \ __getitem__ (963 times)      localrepo.py:566
    4          \ parents (321 times)            context.py:246
    4           | __get__ (321 times)           util.py:798
    4           | _parents (321 times)          context.py:562
    3          \ <genexpr> (642 times)          obsolete.py:1164
    3           | obsolete (321 times)          context.py:199
    2           | getrevs (321 times)           obsolete.py:1114
  319        \ bumped (2 times)                 context.py:211
  319         | getrevs                         obsolete.py:1114
  319         | _computebumpedset               obsolete.py:1181
    4          \ set (322 times)                localrepo.py:610
  312          \ allprecursors (2021 times)     obsolete.py:850
  308           | __get__                       util.py:798
  308           | precursors                    obsolete.py:678
* 308           | _addprecursors                obsolete.py:509
   34        \ divergent (2 times)              context.py:218
   34         | getrevs                         obsolete.py:1114
   33         | _computedivergentset            obsolete.py:1204
    4          \ set (322 times)                localrepo.py:610
   24          \ successorssets (1368 times)    obsolete.py:899
    3            \ __get__ (1368 times)         localrepo.py:76
    4            \ __contains__ (200 times)     localrepo.py:575
    9      \ names (2 times)                    namespaces.py:184
    9       | <lambda>                          namespaces.py:43
    9       | nodetags                          localrepo.py:741
    8       | __get__                           util.py:798
    8       | _tagscache                        localrepo.py:646
    8       | _findtags                         localrepo.py:685
    7       | findglobaltags                    tags.py:170
    7       | _readtagcache                     tags.py:370
    6       | filteredhash                      scmutil.py:307
       \ __exit__                               contextlib.py:21
        | maybeprofile                          profiling.py:199
        | __exit__                              contextlib.py:21
        | profile                               profiling.py:148
        | __exit__                              contextlib.py:21
Total time: 867 ms
```

Some example conclusions from reading the above output include:

  1. `_fm1readmarkers`, `_addsuccessors`, `_addprecursors` are slow
  2. `changelog.node` and `changelog.parentrevs` are called thousands of times,
     but they are just a few milliseconds.
  3. Reading the `obsstore` file (`vfs.read`) takes 5ms exactly

While `statprof` may get 1 right, `lsprof` may get 2 right, neither of them is
helpful for 3. `statprof` is not accurate enough to be confident about 5ms,
`lsprof` will include other `vfs.read` calls so it's impossible to see just the
read for obsstore.


Test Plan:
With the upstream [patch](https://patchwork.mercurial-scm.org/patch/20822/), run:

```
 HGPROF=traceprof hg id --config extensions.traceprof= --profile
```

Reviewers: #mercurial, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters, jiangyuan

Differential Revision: https://phabricator.intern.facebook.com/D5104851

Signature: t1:5104851:1495480386:666847ea08bb6a94658bc10d7f0e91c225d56836
2017-05-22 15:49:45 -07:00
Jun Wu
4240bd017e remotefilelog: let content stores support metadata
Summary:
This diffs add a `getmeta` method to all content stores. The cdatapack code is
modified to pass the tests, it needs further change to support `getmeta`.

The datapack format is bumped to v1 from v0. For v1, we append a `metadata`
dict at the end of each revision. The dict is currently used to store revlog
flags and rawsize of raw revlog fulltext. In the future we can put more data
like a second hash etc, without changing API or format again.

This diff focuses on correctness. A datapack caching layer to speed up
`getmeta` will be added later.

Tests are updated since we write new v1 packfile now and the format change
leads to different content and packfile names.

`Makefile`, `ls-l.py` are added to make tests easier to maintain.

Test Plan: Updated existing tests.

Reviewers: #mercurial, rmcelroy, durham

Reviewed By: durham

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4903917

Signature: t1:4903917:1493255844:7ef5d487096cd2f78f2aaae672a68d49f33632ee
2017-04-26 19:50:36 -07:00
Jun Wu
ca4723ad22 patchrmdir: new extension to workaround rmdir kernel issues
Summary:
We have encountered a kernel issue where `rmdir` a non-empty directory may race
with other things and hang in kernel for a long time.

This patch changes `os.rmdir` to avoid `rmdir` non-empty directories. It is
written in Cython calling the low-level `readdir` libc friends to make overhead
minimal.

Test Plan: Added a new test

Reviewers: #sourcecontrol, clm, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, simpkins, osandov, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4716711

Tasks: 16647532

Signature: t1:4716711:1489627923:7c7432748c1fd8c070ce257bd172feebd3807f65
2017-03-15 18:55:48 -07:00
Jun Wu
71c96fe8ea arc: implement an arcanist unittest engine for mercurial
Summary:
This makes `arc unit` work for the repo.
Hopefully it helps improve the code quality.

Test Plan:
Run `arc unit`. Make some changes locally to confirm
it can handle skipped and failed tests.

Reviewers: #sourcecontrol, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3889495

Signature: t1:3889495:1474454314:b3fb7e538979c775f8301d9abdad0d6eda71e411
2016-09-19 22:41:58 +01:00
Ryan McElroy
86dad1d355 merge .hgignore contents
Test Plan: hg st

Reviewers: #mercurial, ttung, simonfar

Reviewed By: simonfar

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3777380

Tasks: 12855049

Signature: t1:3777380:1472216748:24e80031279d53a5db0743cbbbfa5afcfeeefdf9
2016-08-26 06:27:59 -07:00
Jun Wu
0d3a392454 hgignore: ignore linelog non-source files
Summary: This diff adds an `.hgignore` file for the `linelog` directory.

Test Plan: `make local`, and run `hg status`, make sure no unwanted files are printed.

Reviewers: #mercurial, ttung, akushner

Reviewed By: akushner

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3699575

Signature: t1:3699575:1471151389:7e4de7b75d671d3f0db18337e57c65e9972de059
2016-08-10 22:32:21 +01:00
Tony Tung
fc8c0d61a1 [fastmanifest] rename fastmanifest c library directory to cfastmanifest
Summary: This allows us to use fastmanifest as a directory to drop in the python module.

Test Plan: compiles, passes existing tests.

Reviewers: lcharignon

Reviewed By: lcharignon

Subscribers: mitrandir, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3351021

Signature: t1:3351021:1464284417:6cbcde514ab1fd7b5caa6c83cb5577f3502dbc58
2016-05-26 11:33:07 -07:00
Tony Tung
d8830b241e add .testtimes to .hgignore
Summary: arc diff adds this

Test Plan: meh

Reviewers: #sourcecontrol, ikostia

Reviewed By: ikostia

Subscribers: mitrandir, mjpieters

Differential Revision: https://phabricator.fb.com/D3207936

Signature: t1:3207936:1461258994:ae945f903b51271dd3d0d460ccf1185cd993f387
2016-04-21 10:22:48 -07:00
Tony Tung
14a8f27f55 [fastmanifest] initial checkin
Summary:
Still missing:

== Code and tests: ==
* remove path
* contains path (but this is essentially the same as get path)
* read/write from file

== Tests: ==
* checksumming directories

Test Plan: well, it kind of builds.

Reviewers: durham, simpkins, lcharignon

Reviewed By: lcharignon

Subscribers: net-systems-diffs@, mitrandir, mjpieters, akushner, rmcelroy

Differential Revision: https://phabricator.fb.com/D3120166

Signature: t1:3120166:1459464981:60f19dd1e36d62776a69fa88c018122a4be27d87
2016-04-01 00:46:28 -07:00
Adam Simpkins
b18929a474 [tests] ignore the .testtimes file created by run-tests.py
Summary:
Ignore the .testtimes file created by run-tests.py
This is the same ignore line used in the main hg repository's .hgignore file.

Test Plan: Ran "hg status", didn't see tests/.testtimes listed as untracked any more.

Reviewers: #sourcecontrol, rmcelroy, ttung, durham

Reviewed By: durham

Subscribers: net-systems-diffs@, yogeshwer, mjpieters

Differential Revision: https://phabricator.fb.com/D3014987

Signature: t1:3014987:1457138188:066392d386341e7304b875d9e0aa3d5f4b368a6a
2016-03-04 16:39:51 -08:00
Cecile Berillon
6763341497 dbutil.py: remote database is SQL
Summary: Added all the sql commands for the MySQL remote database, and gathered all the commands in a single function

Test Plan: Created a serverrepo, two client ones, did push and pulls between them

Reviewers: #sourcecontrol, rmcelroy

Reviewed By: rmcelroy

Differential Revision: https://phabricator.fb.com/D2661606

Tasks: 8660367

Signature: t1:2661606:1447875185:0e9d3876b2f509028585349a9d05b447876b3da0
2015-11-20 09:38:21 -08:00
Durham Goode
df38507f4e Ignore dist/ directory 2015-11-13 10:44:11 -08:00
James Mills
c6bf360c5c Use setuptools if available.
Summary: Because working in a virtualenv w/ setuptools makes things a little easier :)

Test Plan: workon hg && python setup.py develop

Reviewers: rmcelroy, durham

Reviewed By: durham

Differential Revision: https://phabricator.fb.com/D2570341

Signature: t1:2570341:1445542319:0f30cb4aefb7c59753decda4535ca335a4f423cc
2015-10-22 08:02:30 -07:00
Siddharth Agarwal
70693db843 add some convenience features 2014-09-08 12:49:25 -07:00