Summary:
Codemoding imports from libfb.py of the format "from libfb import X".
This is part of a larger codemod to remove the mapping from libfb/py to libfb,
in the interest of enabling static typechecking in fbcode.
Reviewed By: dark
Differential Revision: D7335696
fbshipit-source-id: f1a1c3b11bec15610ef2bb24cd1c2941f9409b49
Summary:
A Mononoke tiny changeset is going to have a list of these file
changes.
Reviewed By: farnz
Differential Revision: D7336315
fbshipit-source-id: 25d9908cf825e665ac7605435ce5a4056f221e17
Summary:
For now this just wraps a blob, but in the future this will have
special support for large files.
Reviewed By: farnz
Differential Revision: D7336121
fbshipit-source-id: f4c9e216efdd250fde90307264c1931de97d8592
Summary: Represented as a chrono datetime in Rust and an (i64, i32) pair in Thrift.
Reviewed By: jsgf
Differential Revision: D7325332
fbshipit-source-id: 22c3b17961ffb0b4fdb4e6e8aece3b257b3c718e
Summary:
External code should consider these identifiers opaque -- the fact
that they're hashes is an implementation detail.
Also a "blob" is too generic here -- what we're talking about is file contents.
In the future (e.g. for 10GB+ files) the content hash might point to something
more complicated than file contents.
Reviewed By: farnz
Differential Revision: D7320932
fbshipit-source-id: 11a781345a1973f0af8222b809004daa948c3a47
Summary: just to excersise our RocksDb blobstore which might come in handy more often than file based as it should be more efficient
Reviewed By: StanislavGlebik
Differential Revision: D7290069
fbshipit-source-id: ce776cfa14e43dc45cca796ef187655ba665d177
Summary: The new error contexts contain not only static information, but also the runtime values for which the error happen, so are easier to act upon
Reviewed By: StanislavGlebik
Differential Revision: D7289693
fbshipit-source-id: e0fd2d9872343093beef23a508d5672ac2c06f8e
Summary: The parent information is very valuable when using RevlogManifest to blobimport manifests (actual usage happens in pending diff)
Reviewed By: farnz
Differential Revision: D7289650
fbshipit-source-id: 2483dd6085af6b16afefee03cc93d55494795368
Summary: This test should excercise the situation when a path to a file is hashed when stored in revlog and for a large file the index revlog is split into index and data revlog
Reviewed By: StanislavGlebik
Differential Revision: D7299332
fbshipit-source-id: 6b0f0385e391c8396d5a3702ced5feb1aba2163c
Summary:
When Mononoke server is running, it can log to Scuba; repo logs are
meant to go to Scuba or similar, so send them there, too.
Reviewed By: kulshrax
Differential Revision: D7261637
fbshipit-source-id: 99f6e948eec53b0287b939fc5d1d637ba1e4ccc9
Summary:
To diagnose slow changeset creation, we need to know which chunk was
slow. As with upload_blob and process_entries in past diffs, measure key
phases.
The idea is that we can combine all these times (5 per changeset, 2 per blob)
to work out what needs tracing when we're looking for slowness
Reviewed By: StanislavGlebik
Differential Revision: D7195013
fbshipit-source-id: 9769ead981f02d6ecf0258190763e90762936387
Summary:
Processing entries does two things:
1. Polls all the upload futures until they're all complete.
2. Works out what entries are required to be present for this changeset.
This is the bulk of the Blobstore operations in creating a changeset, so we
know that if this is slow, then we either have a slow Blobstore, *or* we have a
bug. Further, we can combine this with the metrics for upload_blob to know
whether we're doing uploads nicely in parallel, or whether there's
serialization we didn't intended (time taken here greater than sum of time
taken in blob uploads)
Reviewed By: StanislavGlebik
Differential Revision: D7182445
fbshipit-source-id: 205b43700f28ff7e5461235e16a6955b8c597a52
Summary:
I think it makes a lot of logical sense to have three separate hash
types:
* `ChangesetId` for changesets
* `UnodeHash` for unodes
* `BlobHash` for file blobs
These three are never going to mix. Note that manifests and file unodes are
going to mix in e.g. tree manifest unodes.
Reviewed By: farnz
Differential Revision: D7300742
fbshipit-source-id: 688b410f7a5304fdd5ab3a6800137632887f97b9
Summary:
As part of conversion from Thrift to native Rust data structures,
verify that they're valid.
Reviewed By: farnz
Differential Revision: D7297264
fbshipit-source-id: d0bb46a5a5ed2113d9616a56fc100943802cbc1f
Summary:
Also add conversions to and from the Thrift type to the Rust type.
There are tradeoffs around having parallel sets of data structures vs using
Thrift as the only data structure, but I believe that for these hashes (and
transitively, anything that includes them), having a separate type is best.
That's because:
* these hashes are going to be used all over the place, and using a
heap-allocated vector is just a waste. Even a `SmallVec` that's inline will
consume some extra memory for storing the fact that it's inline.
* something has to verify that the vector is valid, and it's nice for that
to be encoded in the type system.
Reviewed By: farnz
Differential Revision: D7285116
fbshipit-source-id: e887414862821076ef0b11e2d0cb0783be897d4a
Summary:
While writing Thrift deserialization code I realized there was nothing
that actually checked that MPathElement instances don't have embedded nulls or
slashes.
Reviewed By: farnz
Differential Revision: D7296838
fbshipit-source-id: 6a23d559da11e5e935e23d7b9a13f58894efaf62
Summary:
Pushkey part is used to send bookmark updates from hg client to the server.
This diff does all the wireproto parsing, but doesn't actually apply bookmark
updates on the server.
Also this diff "implements" branchmap method. We have no plans to support it,
but currently remotenames extension calls it. So this diff adds a fake
implementation that always returns empty response.
Reviewed By: farnz
Differential Revision: D7150973
fbshipit-source-id: 6889c02a1105127b1805ef1fafa6fbe9c2d57e7d
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary: Overview of the various SQL schema Mononoke will use. These are pseudo-schema for documentation purposes, and not directly usable (in particular, foreign keys and "engine = 'manifold'").
Reviewed By: kulshrax
Differential Revision: D6965062
fbshipit-source-id: dc73f7612b10c50a35b27b927abcf2a093cd92ef
Summary:
Empty chunks results in '\x00\x00\x00\x00' being sent to the client. However
this is also a marker for end of the bundle2 part. So previously if listkeys
part were empty we were sending corrupted bundles - it had additional
'\x00\x00\x00\x00' in the end. This wasn't a problem earlier, because getbundle
was the last called wireproto command. However we want to support remotenames,
and remotenames sends listkeys wireproto request after the getbundle.
Reviewed By: farnz
Differential Revision: D7271598
fbshipit-source-id: 1304e2e74725a75726b78bcd4a9664995025191f
Summary:
First draft of the new bookmarks trait. Note that it doesn't support Bookmarks
& Heads simultaneous transactions.
There are a few notable differences from old bookmarks trait:
1) New `Transaction` trait was added that handles all write operations.
2) It has two different set methods: `set()` and `set_unconditionally()`.
`set()` checks the current bookmark value before updating it,
and `set_unconditionally()` doesn't.
Reviewed By: farnz
Differential Revision: D7138038
fbshipit-source-id: 8cbec5caa972d847a30baa0a351df651306b4fd1
Summary:
These hashes will be the used throughout Mononoke's data model.
Also add mocks.
Note that a `UnodeHash` cannot be null.
Reviewed By: farnz
Differential Revision: D7260955
fbshipit-source-id: d5be5c67848e7b93218a26282e0a757708df7e62
Summary:
I've spent quite some time but wasn't able to get to the bottom of why
test-infinitepush.t has been flaky. For some reason error message
```
Expected Bundle2 Changegroup
```
hasn't made it's way to the hg client. Adding a multisecond delay before
sending the error "fixes" the problem, but I haven't found the actual reason
behind the problem. So since that's not a #1 priority now I suggest to postpone
fixing it for now.
Reviewed By: farnz
Differential Revision: D7246511
fbshipit-source-id: a385130e6bdc978765e04f44c6a536405ee12e02
Summary:
The reason for flakiness was in unpredictable order of the tree items from
Mononoke. In the first gettreepack call we request two different revisions, and
Mononoke can return them in any order. hg handles it just fine (see updated
test with `hgmn up ...`), but hashes of the tree packs may change.
To remove flakiness let's not rely on the treepack files hash
Reviewed By: farnz
Differential Revision: D7237559
fbshipit-source-id: c04e9d45c41f1d288a90706d0ecc27ede36f8008
Summary:
I'm going to reuse this for unit-testing changeset timings. Make it a
macro so that I don't keep repeating myself.
Reviewed By: StanislavGlebik
Differential Revision: D7182442
fbshipit-source-id: de40e0f10892b2268c4d39cf771b7a8be6e1cf76
Summary:
It was flaky because server may not yet be ready to accept connections when we
send first request. Sometimes delay needs to be > 1 sec.
Let's query eden server in a loop until it responds.
Reviewed By: farnz
Differential Revision: D7233069
fbshipit-source-id: 8bcb5b2b8ebdc52d2447b33e18580e50c1e27031
Summary:
I want to be able to work in a private Manifold namespace when testing
that my hunches about Manifold/Mononoke interactions are good, and when testing
changes to the ManifoldBlob that should make life better. Make that possible.
In the future, once I have a Blobstore that simulates Manifold locally, this
also makes it possible to compare the simulation to Manifold without tripping
over other users.
Reviewed By: StanislavGlebik
Differential Revision: D7194754
fbshipit-source-id: 601bf1c2ded1af5db6a123fdd05600bc3eb5d7cf
Summary:
This is ripped off pretty much directly from the `Sha1` type in
`mercurial-types`.
Reviewed By: StanislavGlebik
Differential Revision: D7205225
fbshipit-source-id: facb243735d858c68886ad932920024eaf3de536
Summary: SharedError has a default implementation for std Error, but the parameter must implement std Error, that is why I wrapped failure::Error in failure::Compat. Thanks to that we can use SharedError as any other normal std Error (including the fact that it implements failure::Fail)
Reviewed By: farnz
Differential Revision: D7213459
fbshipit-source-id: 54899c64c2627dfdba276630d986a3d6007ea59a
Summary:
These are types that are going to be used throughout Mononoke -- I'm hoping to
avoid references to current Mercurial data structures in here.
For now, the only module I've moved is part of the `path` module. The
Mercurial-specific `fsencode` bits have been kept in mercurial-types (though
maybe they should move to `mercurial`?)
In the future, this module will also include definitions for unodes, etc.
Reviewed By: jsgf
Differential Revision: D7188722
fbshipit-source-id: fc097ca12c38a787f83e35af9b8dd308f2b910ea
Summary:
We want to be able to measure the time it takes to upload individual
blobs, to confirm that we don't have a concurrency issue to chase down (e.g.
blobs accidentally uploaded in series).
Measure content upload time separately, so that we know not to dive down a
rabbit hole if the measured slowness is just the time spent uploading content
Reviewed By: StanislavGlebik
Differential Revision: D7172154
fbshipit-source-id: 08729a8ffaa69a364a64f6277edfa591a8712592
Summary:
I'm going to be adding more to test the timing features, so split this
up to make code sharing easier
Reviewed By: StanislavGlebik
Differential Revision: D7172156
fbshipit-source-id: 056be70268dd1c8a37aff8e8d53342b8cea4a355
Summary: Add in the missing code to establish a MySQL connection to the `changesets` crate. Also add unit tests for `MysqlChangesets`.
Reviewed By: jsgf
Differential Revision: D7174371
fbshipit-source-id: b531c5d5efb95eeb5e39188c4d608540f0c0261f
Summary: I'm going to need a logger to log future-stats output to (and later trace output). Thread one through to BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D7167450
fbshipit-source-id: 4ed729e4d448b66e491cefa19380d3be9bc99091
Summary:
We had two separate issues in blob size, which were breaking `test-eden-server.t`:
1. The value documented as "blob size without metadata" was calculated with metadata included (because we didn't strip the metadata from the content blob).
2. If the BlobNode was clean (i.e. had both data and a hash), the size was the size of the hash, not the data.
Reviewed By: StanislavGlebik
Differential Revision: D7167581
fbshipit-source-id: 965778c1d053af36c760a988f0d34130052e8a1a
Summary:
Add a struct that helds parsed data from changegroup part. In the next diffs
we'll start parsing pushkey parts with bookmarks, and ChangegroupPush struct
knocks down the number of parameters that we need to pass from one function to
another.
Reviewed By: farnz
Differential Revision: D7150974
fbshipit-source-id: a8c0e260a4421168903894930befbab430a6418d
Summary:
We are going to significantly change Bookmark trait and implementation.
We are going to use Diesel, and new Bookmark trait won't use Version. Instead
of changing the current implementation, let's just move it to the separate
directory which will soon be deleted.
Reviewed By: lukaspiatkowski
Differential Revision: D7138039
fbshipit-source-id: 72b81bf7f6524052b3a360bac23ab56a978b08cb
Summary:
Add basic telemetry for some of hg operations that return a future. This is a draft I'm requesting feedback on (as advised by stash@).
1) The code is a bit repetitive. Is there a way to refactor it (for example, python decorator-like style)?
2) What is the best way to add telemetry for operations that return a BoxStream (and not a future)?
Reviewed By: StanislavGlebik
Differential Revision: D7056236
fbshipit-source-id: 7fd61b5533d0fe3857cc7865491b5e9e7240a886
Summary: Replace the generic types if `Blob` and `BlobNode` with `Bytes`.
Reviewed By: lukaspiatkowski
Differential Revision: D7115361
fbshipit-source-id: 924d347377569c6d1b3b4aed14d584510598da7b
Summary:
This is the part that sends client bookmarks to the server. Quite likely we are
not going to process it, so we are just reading the part and ignoring it.
Reviewed By: farnz
Differential Revision: D7123758
fbshipit-source-id: 7ad39bde77b6f77cf6e440e726ac3bdb9f340cea
Summary:
Support b2x:infinitepush part. It contains changegroup v2, so just reuse the
normal push path for it.
Note that pushbackup still fails because we don't support
b2x:infinitepushscratchbookmarks part. Also all wireproto params are ignored
for b2x:infinitepush.
Reviewed By: farnz
Differential Revision: D7086120
fbshipit-source-id: 2f98e5d59059ca3c2b82842c98e6dc771c70c6f0
Summary:
Asyncmemo re-implementation using Shared future. The reasons for doing it are:
1 Now it's safe to drop MemoFuture
2 Now there will be no deadlocks where one future evict another future from the
cache, and then the second future evicts the first futures and so on.
Since Shared returns SharedError, I had to do a hack to work around that. More
details in the code comments.
Reviewed By: farnz
Differential Revision: D7099852
fbshipit-source-id: e08cb859ef06f28339ac9314edb8e53e92480874
Summary: This starts porting uses of Vec<u8> for file contents to the Bytes type.
Reviewed By: jsgf
Differential Revision: D7106766
fbshipit-source-id: 15d531836132317cede7a6f9d6b047a423deb5bb
Summary:
This is a hacky way of getting the push/pull working. We should instead remove the commits that are no longer heads from HeadStore and add those that are the new heads.
In this diff though we add all the commits as heads, just because this does not break the client
Reviewed By: farnz
Differential Revision: D7112279
fbshipit-source-id: 036f0fd230de52e96cbf4168c2cda7c2a1c5bd89
Summary: Now both RepoPath and MPath are showing human readable paths when printed
Reviewed By: farnz
Differential Revision: D7112280
fbshipit-source-id: ba7698d2ffa90a125176d70d665cdef669d63b93
