Summary:
My jq has 2 spaces after "Usage:", so this fails. Let's make the matching a bit
more lenient.
Reviewed By: StanislavGlebik
Differential Revision: D21572028
fbshipit-source-id: 1948af8bbee6486df78299c3cc08209ce3ef3e12
Summary: This might provide some useful insights for debugging hanging processes.
Reviewed By: kulshrax
Differential Revision: D21580307
fbshipit-source-id: 424992cdd9851bfd782a3333ed22ac1193c33fe3
Summary:
Make `prev` and `next` match the behavior of `update` and allow movement with pending changes. A new `--check` flag has been added to enforce a clean working copy (similar to `update`).
Note that these commands literally just call `update` under the hood, so this was just a matter of removing the existing dirtiness check. (The check was originally put there because `update` originally required a clean working copy by default.)
Reviewed By: quark-zju
Differential Revision: D21581805
fbshipit-source-id: 33d0ac4d36a795713054af7e8776d077fd353048
Summary: Previously, we would only allow BFS prefetching when HTTP was enabled (since prior to the `designatednodes` server capability, SSH had no way of supporting BFS). That has now changed, so we should allow BFS even if HTTP is disabled.
Reviewed By: quark-zju
Differential Revision: D21581215
fbshipit-source-id: b9d5945730e2c718681004f7d5e5f319449fcf6a
Summary:
Now we have a virtual DAG that can be queried. Use that to figure out what
heads to hide and add instead of manually traversal through the graph.
Also did some tweaks to fix Python 3 compatibility and make the parent graph
style a bit more correct.
Reviewed By: markbt
Differential Revision: D21554672
fbshipit-source-id: 749d7938a8612e21c5975d9b80a275a059de022d
Summary:
The `getsmartlog` APIs return a generator suitable for passing to the graphlog
rendering layer, but not suitable for doing other things such as querying the
DAG. Split that API into two APIs:
- getsmartlog -> SmartlogInfo
- makedagwalker(SmartlogInfo, Repo) -> generator suitable for graph rendering
Reviewed By: markbt
Differential Revision: D21554673
fbshipit-source-id: c709b7a41572eb3e9c9f91b74fc2bc8226bc481f
Summary:
This simplifies a lot of the code, and makes it possible to do DAG queries like
ancestors, descendants on the commit cloud graph.
Reviewed By: markbt
Differential Revision: D21554674
fbshipit-source-id: ee08cddfc162a7546d63d4bf385f2948fc799fd3
Summary:
The only reason that we keep the legacy renderer is that `hg-sl-up` parses
`hg sl` output and breaks with the new Rust renderer. `cloud sl` does not
have such issues so we can switch to the new renderer unconditionally.
Together with the previous change, this allows the upcoming change to render
the graph without using revision numbers.
Reviewed By: markbt
Differential Revision: D21554670
fbshipit-source-id: eee5fbd641096b38e206298f9e716df358a7ab7e
Summary:
Change pyrenderdag to accept non-revision-number graph vertexes so it can
render a graph even if the graph does not use revision numbers.
The next diff wants this behavior so it can just emit commit hashes to
the renderer without caring about revision numbers. The type is made
so it can still support revision numbers, since the legacy graphlog
interface would still use revision numbers.
Reviewed By: markbt
Differential Revision: D21554671
fbshipit-source-id: 20572683b831f7cecb03957c83f278ff3903eff0
Summary:
The previous code was wrong - it converts the PyObject to iterator every time
(ex. if the PyObject is a set, then it calls `set.__iter__` every time, and
will only get the first element of the set).
For example, it will enter an infinite loop for evaluating this:
bindings.dag.nameset({'1', '2'})
Fix it by calling `__iter__`, to get the iterator object and use that instead
of the original PyObject.
Reviewed By: markbt
Differential Revision: D21554676
fbshipit-source-id: 0f2adae8f123530cee2d473da37ca1a93a941fde
Summary:
This function reorders commits so the graph looks better.
It will be used to optimize graph rendering for cloud smartlog (and perhaps
smartlog in the future).
Reviewed By: markbt
Differential Revision: D21554675
fbshipit-source-id: d3f0f27c7935c49581cfa6e87d7c32eb5a075f75
Summary: This makes it easier to do `a & b`, `a | b`, `a - b`.
Reviewed By: markbt
Differential Revision: D21554677
fbshipit-source-id: e1e2571a3dc83f80a1ec7a056f2c8f71ab292d9e
Summary:
This adds support for periodically logging that a command is in progress in
Mononoke. The underlying motivation is to make sure that if something is taking
a long time, we can still show some feedback to the user (and log to Scuba). We
might want to log this every 30 seconds.
That said, this is more of an RFC at this stage. I'm thinking it might make
sense to log to Scuba more often and to users less often. It might make sense
to also restrict this to specific commands, such as unbundle:
https://fburl.com/scuba/mononoke_test_perf/atik5959
Reviewed By: StanislavGlebik
Differential Revision: D21549862
fbshipit-source-id: 1d02c5c926abc7e491ac5b8ae0244b5f4620c93e
Summary: Same as the previous diff, but for commands that return a stream.
Reviewed By: StanislavGlebik
Differential Revision: D21549864
fbshipit-source-id: ba8c14db34a651cd4ddbc1c8b9ad382c08cc775d
Summary:
This doesn't do anything on its own, but it's refactoring I need for later in
this stack. It wraps all our commands in a command_future call that gives us an
opportunity to wrap the future being returned. We used to use `start_command`
to get the context, so this just replaces that.
Reviewed By: StanislavGlebik
Differential Revision: D21549863
fbshipit-source-id: 0e613bb1db876d27d662fd6c993d7b7d954b5f2b
Summary:
This allows us to find the closest globalrev (i.e. the highest that is lower
than or equal to the one being requested). Conceptually, this is similar to
prefix lookups, which is why we're including it in the prefix lookup function
method!
Reviewed By: markbt
Differential Revision: D21553081
fbshipit-source-id: 4b08f7e2478210d406b1a043e95f8768a3dcc45a
Summary:
The output streams from the walks are of the form (Key, Payload, Stats).
Where the Key is Node and the Payload is NodeData this is ok, but with the key and payload both tuples it gets hard to read, so this introduces named tuple-like structs for clarity.
Reviewed By: StanislavGlebik
Differential Revision: D21504916
fbshipit-source-id: a856d34af4117d3183ef0741b311c1c34cf9dacc
Summary:
Add a --sample-path-regex option for use in the corpus dumper so we can dump out just a subset of directories from a repo.
This is most useful on large repos.
Reviewed By: farnz
Differential Revision: D21325548
fbshipit-source-id: bfda87aa76fbd325e4e01c2df90b5dcfc906a8f6
Summary:
Track path mtime, as being able to order by mtime is important to be able to use the on disk corpus to evaluate delta compression approaches
The dumped blobs mtime is set based on the last traversed bonsai or hg commit's timestamp. For Bonsai it prefers committer_time if present and if not falls back to author_time.
Reviewed By: farnz
Differential Revision: D21312223
fbshipit-source-id: fa14615603f78675ca54a0f4946cc8480b8eade5
Summary:
Update the corpus walker to dump the sampled bytes as early as possible to the Inflight area of the output dir, then move them to final location once path is known.
When walking large files and manifests this uses a lot less memory that holding the bytes in a map!
Layout is changed is to make comparison by file type easier. we get a top level dir per extension, e.g. all .json files are under FileContent/byext/json
This also reduces the number of bytes taken from the sampling fingerprint used to make directories, 8 was overkill. 3 is enough to limit directory size.
Reviewed By: farnz
Differential Revision: D21168633
fbshipit-source-id: e0e108736611d552302e085d91707cca48436a01
Summary:
Add corpus dumper for space analysis
This reuses the path based tracking from compression-benefit and the size sampling from scrub.
The core new functionality is the dump to disk from inside corpus stream.
Reviewed By: StanislavGlebik
Differential Revision: D20815125
fbshipit-source-id: 01fdc9dd69050baa8488177782cbed9e445aa3f7
Summary:
Make it so that remote bookmarks like `foo/name` or `bar/name` would pull from
different sources `paths.foo` or `paths.bar`.
Reviewed By: DurhamG
Differential Revision: D21526666
fbshipit-source-id: 6791ab047840c6c49df0c96ff1f56ae7bd1aeeba
Summary:
We now have auto pull logic that covers most unknown rev use-cases. The hint
message is no longer necessary. It's also unclear how to use `hg pull`
correctly. For example, should it be `-r`, `-B remote/foo` or `-B foo`?
Reviewed By: DurhamG
Differential Revision: D21526667
fbshipit-source-id: 40583bfb094e52939130250dd71b96db4d725ad5
Summary: The original purpose of this test was to verify that `hg restack` would work correctly with the `inihibt` extension disabled. `inhibit` has not been relevant at FB for years, so this test has no value.
Reviewed By: quark-zju
Differential Revision: D21555411
fbshipit-source-id: 475ed37439ed71aee08ad1b23ebe1770c3324890
Summary:
I want to reuse the functionality provided by `fetch_all_public_changesets`
in building Segmented Changelog. To share the code I am adding a new crate
intended to store utilities in dealing with bulk fetches.
Reviewed By: krallin
Differential Revision: D21471477
fbshipit-source-id: 609907c95b438504d3a0dee64ab5a8b8b3ab3f24
Summary:
This is helpful, when we have raw unbundle bytes and a server path and just
want to send these bytes server's way.
Very similar to `sendunbundlereplay`, but does not do anything additional,
and reads from stdin.
Reviewed By: markbt
Differential Revision: D21527243
fbshipit-source-id: 97726cb40a32c7e44f47e0f56d8c8eabc4faf209
Summary:
As a developpers is working on large blobs and iterating on them, the local LFS
store will be growing significantly over time, and that growth is unfortunately
unbounded and will never be cleaned up. Thankfully, one the guarantee that the
server is making is that an uploaded LFS blob will never be removed[0]. By using
this property, we can simply move blobs from the local store to the shared
store after uploading the blob is complete.
[0]: As long as it is not censored.
Reviewed By: DurhamG
Differential Revision: D21134191
fbshipit-source-id: ca43ddeb2322a953aca023b49589baa0237bbbc5
Summary:
In order to build a StringPiece from a C-string, the length of the that
string needs to be known, and a constexp strlen is performed on it. That
strlen is however a recursive one, causing the stack to blow up on big file.
Interestingly enough, this also means that binary files potentially had a
wrong sha1 being computed, potentially causing `hg status` to report some
files as being modified, while they aren't. By switching to using a vector
instead of a string, the intent should of this should be more obvious.
Reviewed By: simpkins
Differential Revision: D21551331
fbshipit-source-id: 2dc1f08d96f49d310593e0e934a03215be2b5cbb
Summary:
If a push or pull operation doesn't involve any changesets for which mutation
information might be relevant, don't spend any time querying the database, and
instead exit early.
Reviewed By: krallin
Differential Revision: D21549937
fbshipit-source-id: a6f992e621456b826acd1bddde3591e751d23b31
Summary:
MySQL returns an error for a query of the form `WHERE value IN ()`. Avoid
these by checking that collections are not empty before making the query.
Reviewed By: krallin
Differential Revision: D21549690
fbshipit-source-id: 1507d36e81f7a743d2a1efb046e52a5479633ab9
Summary:
The `test-infinitepush-mutation.t` test covers the new mutation database, so
add it to the mysql tests.
Reviewed By: krallin
Differential Revision: D21548966
fbshipit-source-id: 0dc1f90129fa61fb6db1c1b5a747efa3d20041f5
Summary:
The `support_bundle2_listkeys` flag controls at runtime whether we support
`listkeys` in bundle2. Since this was added before tunables were available,
it uses a value in the mutable counters SQL store.
We could migrate this to tunables, but in practice we have never disabled it,
so let's just make it the default.
Reviewed By: krallin
Differential Revision: D21546246
fbshipit-source-id: 066a375693757ea841ecf0fddb0cc91dc144fd6f
Summary:
When the client pulls draft commits, include mutation information in the bundle
response.
Reviewed By: farnz
Differential Revision: D20871339
fbshipit-source-id: a89a50426fbd8f9ec08bbe43f16fd0e4e3424e0b
Summary:
Advertise support for `b2x:infinitepushmutation`. When the client sends us
mutation information, store it in the mutation store.
Reviewed By: mitrandir77
Differential Revision: D20871340
fbshipit-source-id: ab0b3a20f43a7d97b3c51dcc10035bf7115579af
Summary: Add the mutation store to blobrepo.
Reviewed By: krallin
Differential Revision: D20871336
fbshipit-source-id: 777cba6c2bdcfb16b711dbad61fc6d0d2f337117
Summary: What it says in the title
Reviewed By: StanislavGlebik
Differential Revision: D21549635
fbshipit-source-id: 75939ebbfb317a9beaa9acd1fc1a7c6f41b0f88f
Summary: The compiler is warning about it.
Reviewed By: singhsrb
Differential Revision: D21550266
fbshipit-source-id: 4e66b0dda0e443ed63aeccd888d38a8fcb5e4066
Summary: Expose the API that returns a real graph.
Reviewed By: DurhamG
Differential Revision: D21486520
fbshipit-source-id: 4ebdb4011df8971c54930173c4e77503cd2dac47
Summary:
Part of the mutation graph (excluding split and fold) can fit in the DAG
abstraction. Add a method to do that. This allows cross-dag calculations
like:
changelogdag = ... # suppose available by segmented changelog
# mutdag and changelogdag are independent (might have different nodes),
# with full DAG operations on either of them.
mutdag = mutation.getdag(...)
mutdag.heads(mutdag.descendants([node])) & changelogdag.descendants([node2]) # now possible
Comparing to the current situation, this has some advantages:
- No need to couple the "visibility", "filtered node" logic to the mutation
layer. The unknown nodes can be filtered out naturally by a set "&"
operation.
- DAG operations like heads, roots can be performed on mutdag when it's
previously impossible. We also get operations like visualization for free.
There are some limitations, though:
- The DAG cannot represent non 1:1 modifications (fold, split) losslessly.
Those relationships are simply ignored for now.
- The MemNameDag is not lazy. Reading a long chain of amends might be slow.
For most normal use-cases it is probably okay. If it becomes an issue we
can seek for other solutions, for example, store part of mutationstore
directly in a DAG format on disk, or have fast paths to bypass long
predecessor chain calculation.
Reviewed By: DurhamG
Differential Revision: D21486521
fbshipit-source-id: 03624c8e9803eb1852b3034b8f245555ec582e85
