Summary:
This adds the `lfs_url` to our logs, which makes it easier to identify the
vip we're trying to talk to, and will make it easier to monitor for errors if
any come up.
Reviewed By: HarveyHunt
Differential Revision: D17474943
fbshipit-source-id: 5f67665cffa878966e5d49fae672ff1be571690e
Summary:
When the rebase destination has renamed a file, and copytrace failed to figure
out anything useful, make it possible to manually input the rename destination.
I'm using it to resolve conflicts caused by landing the bindings rename.
Reviewed By: xavierd
Differential Revision: D17367857
fbshipit-source-id: 55d9ed3a9641e40cc43518e9662fd803cbc00620
Summary:
The cacheprocess config is of the form: memcache_client key. With the memcache
client moving in process, the memcache_client binary is not of interest, but
the key is.
Reviewed By: quark-zju
Differential Revision: D17438697
fbshipit-source-id: 925c6e4c390f975825afef01fbd87a9331894e81
Summary:
This is useful when writing merge tools that are checked into the repo. We can now reference a tool in the repo. Suggested use case:
[merge-tools]
mytool =
mytool.executable = scripts/mytool.py
Reviewed By: quark-zju
Differential Revision: D17342897
fbshipit-source-id: ffdb160dfea8404721c4fc94fdfcab2d86cca08d
Summary:
Per team meeting, we want to remove whole changelog scans that are incompatible
with the upcoming dag changes.
Heads calculation is one of such "whole changelog scans".
The plan is to use visibility heads + remote names to answer `head()`. However,
remote names are not guarnateed to be heads. For example, `stable` might be
an ancestor of `master`. To get the right answer about `head()`, some
calculation like `heads(::(remotenames() + visible-heads()))` needs to be done.
Calculating `heads(ancestors(...))` in Python is quite slow. This diff provides
a native fast path for it. It still requires a partial changelog scan, but will be
compatible with the future dag-based commit graph.
Reviewed By: sfilipco
Differential Revision: D17199841
fbshipit-source-id: 6ea4367b8877209899d56094f8d8ee1aff1ad6f3
Summary:
Add a function to do "head-based phases" calculation on the revlog. So we can
experiment the breaking change, since phases are no longer root-based, and
are probably defined by remotenames and visibility heads.
The segmented changelog structure will drop support for root-based phases for
performance.
Reviewed By: sfilipco
Differential Revision: D17199844
fbshipit-source-id: 4a4dba183bb5f751b0cf454b9fc2b7e601e8c491
Summary:
This module is inteneded to have native paths for some operations that need to
scan the whole changelog. It allows us to experiment some breaking changes,
namely, head-based visibility without "filtered revs", head-based phases on
the revlog format, before the more advanced structure taking over.
This diff adds a revlog index reader that can answer do simple queries like
"length", "parents".
Reviewed By: sfilipco
Differential Revision: D17199837
fbshipit-source-id: 2574f64c980419fa966200fd52fa5ddf873baae4
Summary:
Expose more methods in Rust to Python.
As we're here, change `__contains__` to take a signed int so `-1 in set` test
won't trigger an error.
Reviewed By: sfilipco
Differential Revision: D17244562
fbshipit-source-id: 0b8b9069bd0a35615066d1328933ca50b09b4a25
Summary:
One step towards redefining heads with remote heads.
Debugshell was teached to skip IPython for non-interactive input.
(For some reason most part of this diff got folded into the previous one)
Reviewed By: sfilipco
Differential Revision: D17199842
fbshipit-source-id: 1594658e3e5f2f2c51f817bc1074d28329df7786
Summary:
This is to match APIs like `ui.configbool` in the Python land.
The new APIs use string types to make them easier to use.
Reviewed By: xavierd
Differential Revision: D17423798
fbshipit-source-id: 747b094c4cc6afef2e818a1862e820647328c4c2
Summary: The implemenation is based on `range`.
Reviewed By: sfilipco
Differential Revision: D17002967
fbshipit-source-id: f59e1686306b9fc08dbca9891c2b6d6c42be2717
Summary:
This function sets the proper "upper bound" in the Dag. It can be used together
with other functions, like `roots` and `heads`.
Reviewed By: sfilipco
Differential Revision: D17002968
fbshipit-source-id: 581a2acea73d6439ffb47442f12e8f009ba6aa8c
Summary: Make it clear that the index takes `level`.
Reviewed By: sfilipco
Differential Revision: D17002966
fbshipit-source-id: 376347964b684579b49d40fc202a37797131ac65
Summary:
The function is capable of handling all levels. But practically only used for
the flat level. Rename it and drop the `level` parameter.
Reviewed By: singhsrb
Differential Revision: D17002969
fbshipit-source-id: e5b99142b6612a80237b4af2706eba51f3794c88
Summary:
Previously, `SyncableDag` and `Dag` can co-exist. Dropping SyncableDag involves
error handling and is not panic-free. If we want to make sure `Dag` has complete
high-level segments, then it would have been implemented in `SyncableDag::drop`,
making it more sensitive to panic.
Change the API so `SyncableDag` is independent from `Dag`, so `Dag` always
has complete segments, and changes to `SyncableDag` are invisible to `Dag`,
so `SyncableDag` cannot mess up existing `Dag` structures.
Reviewed By: sfilipco
Differential Revision: D17000969
fbshipit-source-id: 1ceed4ea335d3d64848b7430d48076846b90695d
Summary: Similar to Index::clone, this makes it possible to clone a Log.
Reviewed By: sfilipco
Differential Revision: D17379310
fbshipit-source-id: f28c4b60d256b282491a93c3cd39138ebbe71131
Summary:
This provides a way to check if the hg process was gone unexpectedly.
The motivation was to make it less likely that Nuclide shows an hg operation
was ongoing but the process was long gone.
Reviewed By: xavierd
Differential Revision: D17372808
fbshipit-source-id: 19e51f8bbd0fdf14b7e0a4bdb8247d1b4277ee48
Summary: This makes it possible to decode VLQ from a stream.
Reviewed By: alexeyqu
Differential Revision: D17404066
fbshipit-source-id: 4a3b0e5333664c3cfc0f76bbdc7db80c25a3a49c
Summary:
The function just wraps `std::fs::create_dir` and ignore the errors when the
directory already exists.
Reviewed By: quark-zju
Differential Revision: D17357936
fbshipit-source-id: 3a0b2d027aa9425877b288e43feeb9dbeba78651
Summary:
Similarly to the IndexedLog based store, let's also enable writing to the
packstore. One big benefit of doing that is that it will greatly reduce the
need for rescanning the filesystem when new packfiles are created.
The read-only version are for now kept as they are being exposed to Python,
long term, the read-only will be made private and the Mutable version renamed.
Reviewed By: kulshrax
Differential Revision: D17357937
fbshipit-source-id: 0f440d5668ca8b412300ff3519a233d1449a464d
Summary:
$HOME and $USER can vary depending on whether sudo is used or
not. In the mixed owner situations that tend to crop up at FB we
generally want to consider the user and home dir of the owner of the
repo rather than whatever the current user is.
This diff adds a `REPO_OWNER_HOME` replacement token that resolves
the owner of the repo and then looks up their home directory.
Reviewed By: simpkins
Differential Revision: D17335435
fbshipit-source-id: bf81565abdf66084b93c3411ae6b58af0c4a6143
Summary:
If we are run as root then set the owner of any files or
directories that we create to the owner of the repo.
This helps to avoid some classes of problems if there are mixed
invocations with root and repo owner.
Note that I picked a string to represent the user largely because
I can see us wanting to specify the user in the config file
using a string at some point in the future.
It is also slightly more portable (even though it isn't native)
than doing something funky with uid/gid on posix and something
involving security descriptors or DACLs on windows.
Reviewed By: chadaustin
Differential Revision: D17335433
fbshipit-source-id: 33c1d1cdbe4d07ce4d4f2581206f1308e3b29290
Summary:
This is a trivial change to use the new helpers added
in the previous diff
Reviewed By: chadaustin
Differential Revision: D17335434
fbshipit-source-id: feb6fa5cee2e2775e89ba330916327e52ffe9394
Summary:
These make it slightly more ergonomic to lookup
user information either by name or uid on unix systems
in the diffs that follow in this stack.
Reviewed By: chadaustin
Differential Revision: D17335436
fbshipit-source-id: 61cabe368504ad32dfa9ca52bbdd106ddbca289a
Summary:
In the future the changelog would need to access 'remotenames' to figure out
public heads. Move the state file to svfs so it can be read by the changelog
object, which only has access to svfs.
Reviewed By: sfilipco
Differential Revision: D17199834
fbshipit-source-id: 9000e0d8e8bb8d398d6c77b5b395da904fef6418
Summary:
This will be used in a later change.
I found ChecksumTable::clone crash when the file is empty. That was fixed as
part of the change.
Reviewed By: sfilipco
Differential Revision: D17379311
fbshipit-source-id: 0e43e8559f0628008e10f1e619a0523474c4094f
Summary:
SyncableDag might have lagging high-level segemnts that break the "high-level
segements cover all flat segments" assumption by some DAG algorithms (children,
range).
Remove Deref so those DAG algorithms cannot run on a SyncableDag.
Reviewed By: sfilipco
Differential Revision: D17000967
fbshipit-source-id: 953589ddd8388340a2913b530f442028810296de
Summary: After D17000965, the parameter is no longer used.
Reviewed By: sfilipco
Differential Revision: D17379312
fbshipit-source-id: c6a12ee27a54efcec5caa1769f04095e16dd5995
Summary:
Previously, the `Dag` has 2 low-level `build_segemnts` APIs:
- Dag::build_flat_segments(..., last_threshold)
- Dag::build_high_level_segments(..., drop_last)
They allow customization about whether the segments are lagging or not.
However, certain algorithms (ex. children and range) now require the high level
segments to cover everything covered by the flat segments. The above APIs
wouldn't ensure that.
This diff refactors the segment building APIs so that:
- Make `build_flat_segments`, and `build_high_level_segments` private to
prevent misuse.
- Ensure high level segments cover flat segments at `Dag::open` and
`Dag::build_segments_volatile`, the only ways to change `Dag`.
- Provide different APIs suitable for different (one-time in-memory vs
on-disk) use-cases. The on-disk `build_segments_persistent` API makes high
level segments lagging to avoid fragmentation, while the in-memory
`build_segments_volatile` does not.
To satisfy the existing test need, a `set_segment_size` API was added to
override the default segment size.
Most callsites become simpler because they no longer need to figure out
details about segment size, level, and lagging.
Reviewed By: sfilipco
Differential Revision: D17000965
fbshipit-source-id: 78bb0c7674c99e91be6011bb7e623cd4f63b1521
Summary:
Parents are ordered and the order can be important. The flat segments preserve
the order but there are no APIs exposing the parent order. Define one.
Reviewed By: markbt
Differential Revision: D17000966
fbshipit-source-id: 66beb53d9cef651a53391707c0b690a1e3b76ce2
Summary: See the previous diff for motivation.
Reviewed By: sfilipco
Differential Revision: D17000964
fbshipit-source-id: 3bf112e4b608cf06bf9082d993635df1d65a81fa
Summary:
Segmented changelog wants:
- Entries stored on disk is lagging (i.e. the last segment per each level is
dropped to avoid fragmentation)
- Entries in memory is not lagging.
To ensure that, the easiest way seems to be:
- Upon initializing the dag structure, build missing segments in memory.
- When preparing writing to disk, drop the in-memory segments.
This diff makes it possible to drop in-memory index changes.
Reviewed By: sfilipco
Differential Revision: D17000970
fbshipit-source-id: 554746c9a868cc3881dd836dc3a3cda265b54287
Summary:
Higher level segments need to be producing fewer segments. If they cannot do that,
there is no point creating those levels.
This makes it possible to auto-detect the maximum segment level.
Reviewed By: sfilipco
Differential Revision: D17000968
fbshipit-source-id: bfb35f7451a40612803cb232be163db12bcab2fd
Summary:
The fast path takes advantage of high-level segments and avoids visiting lower
level segments if possible.
`cargo bench --bench dag_ops children` before:
children 16.842 s
after:
children 427.577 ms
Reviewed By: markbt
Differential Revision: D16976385
fbshipit-source-id: ae1b5f26b6ce2213f5ce1f28c4ac1cf49678992f
Summary: This allows us to check if the "HAS_ROOT" flag is correctly set or not.
Reviewed By: sfilipco
Differential Revision: D16976384
fbshipit-source-id: a5bb73b0bc414e422afd7c8690c55ee1076ef9f2
Summary:
This would allow us to implement the fast path in `children`.
This is a breaking change in file format. But we haven't been really using the
segmented dag format in production yet. So it's fine to not have a migration
path.
This potentailly makes the future format migration easier since there are 7
bits left to use.
Reviewed By: markbt
Differential Revision: D16976382
fbshipit-source-id: 0110c03d4ff2ce0a882b3fb8ce8ee12d2087f41f
Summary:
The children operation is now O(flat segments) = O(merges). But it has
potential to be faster with more precalculated information in segments.
See added comments for details.
Reviewed By: markbt
Differential Revision: D16976383
fbshipit-source-id: 80bfd2cefa5aa5ceabfd7e40d46d06f7e2b64d34
Summary:
This makes dag_ops benchmark runnable.
Example benchmark result run with `sudo nice -20`:
building segments 489.076 ms
ancestors 378.471 ms
common_ancestors (spans) 495.074 ms
gca_one (2 ids) 607.611 ms
gca_one (spans) 509.784 ms
gca_all (2 ids) 977.349 ms
gca_all (spans) 570.267 ms
heads 248.168 ms
heads_ancestors 248.524 ms
is_ancestor 295.376 ms
parents 246.623 ms
Reviewed By: sfilipco
Differential Revision: D16992731
fbshipit-source-id: 838f9240ca942f442fa395c1cf7bf914c52652c2
Summary:
This will guide optimizations.
Right now, gca_one "large sets" can take forever to run.
The code to build the mozilla DAG was extracted to a single file to be sharable.
Reviewed By: sfilipco
Differential Revision: D16992730
fbshipit-source-id: 1538f5b0098cd06cb179bd556df285055e1d62b6
Summary:
This is the mutable operation that allows other code paths to add spans to a
SpanSet. It will be used in a later change.
Reviewed By: markbt
Differential Revision: D17169483
fbshipit-source-id: be9286e67bfa64e961ee75482432aa61dfc0e6ed
Summary:
We don't use fncache. It does not scale. The test break with narrow-heads.
This diff just removes the fncache test.
Reviewed By: singhsrb
Differential Revision: D17293478
fbshipit-source-id: b41dba333276ad63973b7556dfc400f4b33f6f5d
Summary:
Ondemand has been complaining that their hgcache was growing very quickly, and
looking at logs and scuba data, this appears to be caused by refetching
everything from the network and writing it to the indexedlogdatastore.
Looking at the prefetch code, I realized that the indexedlogdatastore wasn't
present in the repo.fileslogs.shareddatastore list, and therefore may be
omitted from the stores in which we look for data.
Reviewed By: kulshrax
Differential Revision: D17374761
fbshipit-source-id: 9e9f279d4c8154e2491312a57d24bc5fd7da79fc
Summary: This is no longer used, let's remove it.
Reviewed By: quark-zju
Differential Revision: D17337442
fbshipit-source-id: 2272c2662440bc2be2a3ff29ef11fc4e0eb3605f
Summary:
The parking_lot crate is more convenient to use than std::sync, on top
of everything else listed at https://crates.io/crates/parking_lot. Let's
use it everywhere.
Reviewed By: quark-zju
Differential Revision: D17337444
fbshipit-source-id: b5489be0b7d2bd5f6a6edc5d1d6eea366a6c05b9
Summary:
I hit this when editing the dag stack. This resolves a "BUG" in the
test-amend-nextrebase.t test.
Reviewed By: singhsrb
Differential Revision: D17357387
fbshipit-source-id: 309efe34dee71180abdbdb5a9fe2e7b731230051
Summary: Now that several of the submodules in the Manifest crate have their own unit tests, let's de-duplicate the test helper functions across those tests by moving them to a `testutil` submodule (which only gets built in `cfg(test)`).
Reviewed By: xavierd
Differential Revision: D17352890
fbshipit-source-id: 1b7d7cb754ee501def8d1d508f2af5733d548f4d
