Summary:
Native checkout avoids the initial large prefetch, and instead does
small batch prefetches and immediately writes them to disk. This will prevent
the issue where data in the cache is evicted before it's used and has to be
redownloaded in a non-batch fashion later.
In a future diff, using native checkout will also allow us to serialize and
deserialize it's CheckoutPlan, so we can make clones resumable.
Reviewed By: quark-zju
Differential Revision: D26825773
fbshipit-source-id: 6fde90cd7578f6af5fdd4ac78004f7f63af4f287
Summary: the list of configured default selective pull bookmarks are actually related to the hoist namespace I think when we consider what is protected and what is not.
Reviewed By: quark-zju
Differential Revision: D26814708
fbshipit-source-id: 566f363f1e6b88dd2ebc6e33cfb6eba596493988
Summary:
Mononoke hg sync job does the following operation in the loop:
1) check if we got new bookmark update log entries in db
2) Sync these log entries to hg server
3) update the counter that says what was the latest synced entry
It's possible that step 2 was successful but step 3 failed. After that sync job restarts and tries to replay already-replayed bundle again, and this fails.
We had a protection against this situation - after every failed bundle we sent a request to hg server to check what's the location of bookmark is - however it stopped working after we added `--combine-bundles` option which syncs more than 1 bookmark update log entry at once.
To be precise, this situation could happen
1) Sync job syncs log entries [1, 2] successfully
2) Sync job tries to update the `latest-replayed-request` counter to 2, and fails
3) New log entry 3 comes into queue
4) sync job restarts and tries to sync [1, 2, 3]. This fails because [1, 2] were already synced. Because of that sync job tries to verify that the location of the bookmark on the server is at the position 3, but actually it's at 2. So sync job fails again and it keeps crashlooping.
This diff attempts to fix it by trying to detect if any of the first entries in the first bundle were already synced. It does so by taking the value of the bookmark from BookmarkOverlay (it's equivalent to the value of the bookmark on hg server), and comparing with to_changeset_id/from_changeset_id values from the batch, and skipping a first few. See code comments for more details on how it does it.
A few notes:
1) BookmarkOverlay might actually contain outdated values i.e. not the latest snapshot of hgsql. This happens because there's a delay between commit appearing on hgsql and propagating to hg server repositories. In a case like that it's possible that adjusting the first batch might not kick in, and sync job will fail to sync the bundle. However my assumption is that in practice it should be ok - sync job will restart and try to repeat the bundle adjustment, and eventually it should succeed. Also the current `verify-server-bookmark-on-failure` likely to have the same problem.
2) Note that this still might not work if we try to decrease the combine-bundles value when sync job counter is pointing to the wrong log entry. I expect it to be rare enough so that we don't need to worry about it
Note that it still worth keeping the `verify-server-bookmark-on-failure` option, since it can detect other kinds of failure (e.g. hg command returning non-zero exit, but actually successfully syncing the entry).
Reviewed By: krallin
Differential Revision: D26753763
fbshipit-source-id: bea9da9ab1ceede19666c99e28553e74edb0ed2a
Summary:
We use itertools group_by() to group filenodes. Unfortunately that only works
if they're sorted, which they're not. With a hundred shards, we easily end
up sending one query per filenode.
This fixes that to group the filenodes properly. This means we'll make 1 query
for 1000 filenodes, i.e. make 1000x fewer queries.
Differential Revision: D26821973
fbshipit-source-id: b83ccaff608f38adefe86c8f5018e4b525cbde11
Summary:
On POSIX systems it's a good practice to fsync directories to ensure metadata
about a file is properly written.
Reviewed By: DurhamG
Differential Revision: D26822211
fbshipit-source-id: fca10c702b480f22020ad184eb55c8879999f0ee
Summary:
Typical mistake of `reposetup` - the `repo` object might be remote and lack of
local repo methods.
Reviewed By: DurhamG
Differential Revision: D26822960
fbshipit-source-id: 54fb95c51b09ef5021b0a10a93b4509e7aa4a115
Summary:
We're missing some metrics to measure how much queries we are making here, and
our error reporting isn't very comprehensive. Let's improve this.
Reviewed By: StanislavGlebik
Differential Revision: D26819845
fbshipit-source-id: c78b8dee02d58dc09ea5f5bd9712e5dfa0bb3f29
Summary:
The recent change to make `list-bookmarks` read from the warm bookmark cache
has made this test flaky, as we must wait for the warm bookmark cache to update
before the new values are visible.
Update `wait_for_bookmark_update` to use `list-bookmarks` which always uses
the warm bookmark cache. A similar function is also needed for bookmark deletion.
Reviewed By: StanislavGlebik
Differential Revision: D26816627
fbshipit-source-id: e99b83454070d565646b069177e18bd6a01946ae
Summary: This is merely adding the types needed to implement the REMOVE procedure.
Reviewed By: kmancini
Differential Revision: D26694441
fbshipit-source-id: 736d06ed7bfae40f827fbec6017e069fe11d7464
Summary:
Since EdenFS doesn't support hardlinks, this is merely collecting the attribute
of the file before always failing.
Reviewed By: kmancini
Differential Revision: D26682014
fbshipit-source-id: 8c39170147023a03e87ed7a0fae66f9184b4f690
Summary:
The RPC simply queries various filesystem attributes, we merely forward what
statfs on the overlayfs gives us.
Reviewed By: kmancini
Differential Revision: D26681613
fbshipit-source-id: 5b94d05cafff8d77390fe60a3b5cf1dc3e022f42
Summary: This merely adds the types for the RPC.
Reviewed By: kmancini
Differential Revision: D26681615
fbshipit-source-id: d092cf0b6b5bb7435702d125b5c6ea7ee68356dc
Summary:
This simply writes the data passed in to the inode. Note that the current
implementation has a protocol violation since it doesn't sync the data written
to disk but advertise to the client that it did. This is completely wrong from
a data consistency guarantee, but is probably fine for now. Once the code
becomes closer to being production ready, this will be changed to honor the
client asks.
Reviewed By: kmancini
Differential Revision: D26681614
fbshipit-source-id: 82ad7a141be3bbe365363b1f6692ae62f253423f
Summary:
The Appender API doesn't allow us to simply append an IOBuf chain to it forcing
the entire chain to be copied to it. For the most part, this isn't an issue,
but to reduce the overhead of the READ NFS procedure, this will be a nice
benefit.
For the most part, this diff is a codemod, with a couple of places that needed
to be manually fixed. These were in the rpc/Server.{cpp,h} and
rpc/StreamClient.{cpp,h}.
Reviewed By: genevievehelsel
Differential Revision: D26675988
fbshipit-source-id: 04feef8623fcddd02ff7aea0b68a17598ab1d0f8
Summary:
The ser, de and roundtrip are duplicated 3 times in the various tests, let's
move them into a common library.
Reviewed By: chadaustin
Differential Revision: D26675989
fbshipit-source-id: 1da0bc33429795a889b72c76fa18e5fa3cb7df6f
Summary: This merely adds the various types for the WRITE RPC.
Reviewed By: chadaustin
Differential Revision: D26671895
fbshipit-source-id: 8409c8a1f90e97478aed7c9f19b881c46234b539
Summary:
For the READ/WRITE RPC calls, copying data in and out of an IOBuf chain can be
fairly expensive, to avoid this overhead, we can simply clone the data out of
the IOBuf chain directly, saving on the cost of copy.
Since the code uses a folly::io::Appender that doesn't support adding an IOBuf
to it, we still pay the cost of copying data to it, switching to
folly::io::QueueAppender may solve this.
Reviewed By: chadaustin
Differential Revision: D26671896
fbshipit-source-id: 0161f04cb820bf27ef66fdef6b4a1ce4eb778b96
Summary:
The CREATE RPC is intended to create regular files on the server. It supports 3
ways of creating them: unchecked, guarded and exclusive. The exclusive mode
allows for a client to being able to re-transmit the creation and for it to not
fail when that happens. The way this is implemented requires the server to
store a cookie passed in by the client and on retransmission to compare it in
order to not fail the file creation. This complexity may not be warranted and
needed in the case of EdenFS, and the spec allows for a server to not support
this operation, thus EdenFS claims to not support it.
The 2 other modes are more traditional, guarded will simply fail if the file
already exist, while unchecked succeed in that same situation. It's not
entirely clear in the RFC what the behavior is supposed to be in the unchecked
case, and whether the attributes needs to be applied. For now, the error is
just ignored, a warning message is logged and a success is returned in this
case. In the future, setting the file attribute (in particular the mode and
size) may be necessary.
Reviewed By: kmancini
Differential Revision: D26654766
fbshipit-source-id: c2e8142b8a4ff756a868e5859fdda4b07e53bddf
Summary:
I moved the source of the sphinx project to newdoc for newdoc/dev.
I updated the sphinx config for markdown to something that works for recent
versions.
PNG images rendered better than SVG for me.
I moved the TARGETS to newdoc.
Reviewed By: quark-zju
Differential Revision: D26801426
fbshipit-source-id: 3ae51e886d27f848f0f6a48c96056da607d8da45
Summary:
Mercurial uses Makefile here to build stuff. We want to track Makefile under
scm.
Reviewed By: quark-zju
Differential Revision: D26802165
fbshipit-source-id: 1fe8db13d50c07a6a0681180959eba22eaf8d486
Summary:
This won't be very long lived, as I'll remove the 0.2 version once Mononoke is
updated (we are the only users), and stop using a fork as soon as Gotham
releases. However, for now, I need both versions in order to avoid to do it all
in 1 diff.
A couple things worth noting here:
- See the previous diff for why I am using 2 Git repos for this.
- I can't easily use 2 forks for this because there are some extra crates they
vendor, notably `borrow_bag`. I removed it from both forks and pulled it from
crates.io instead.
Reviewed By: mitrandir77
Differential Revision: D26781291
fbshipit-source-id: 0b9824c07b880dbd5b94cd6d62d2cb2a68e348e9
Summary:
There is a bug in Hyper with 101 Continue responses: it sends a Content-Length.
This makes Proxygen unhappy with websocket upgrades. We used to have this
patched in hyper-02, but since Mononoke is about to update to Tokio 1.x, we
also need in the matching Hyper.
One thing that's a bit awkward is you might notice I changed the fork where the
patch comes from. This is because `cargo vendor` cannot deal with 2 overriding
sources coming from the same upstream. I posted about my adventures in this
here: https://fb.workplace.com/groups/rust.language/permalink/5278514098863829/
Reviewed By: HarveyHunt
Differential Revision: D26780899
fbshipit-source-id: e775b7151427898d63d8767acaa53f5f68229db6
Summary: This is used in Metagit and I'd like to decouple those 2 Tokio 1.x migrations.
Reviewed By: HarveyHunt
Differential Revision: D26813352
fbshipit-source-id: 7bc34e1cad00c83bf66edce559b07104d44a7357
Summary: Now the queries macros are asynced, lets do the same with the Transaction api exposed from them.
Reviewed By: krallin
Differential Revision: D26730195
fbshipit-source-id: 278753a5d0401f602ce50519138164bb5e49d550
Summary: Migrate to the std futures version of sql::queries!
Reviewed By: krallin
Differential Revision: D26700359
fbshipit-source-id: 39c75d7896a5975e53dd3af53860ce486683b4ed
Summary: Migrate to the std futures version of sql::queries!
Reviewed By: krallin
Differential Revision: D26700357
fbshipit-source-id: ea9382028b2e5abfa1946e1c5de344e32ac60d04
Summary: Migrate to the std futures version of sql::queries!
Reviewed By: krallin
Differential Revision: D26700360
fbshipit-source-id: 9ed2664d522bde8d0e923142357ca876a7de2613
Summary: Migrate to the std futures version of sql::queries!
Reviewed By: krallin
Differential Revision: D26700358
fbshipit-source-id: 4a100705c43d77d67fb784afbb6b44b57904cba0
Summary:
Async the query macros. This change also migrates most callsites, with a few more complicated ones handle as separate diffs, which temporarily use sql01::queries in this diff.
With this change the query string is computed lazily (async fn/blocks being lazy) so we're not holding the extra memory of query string as well as query params for quite as long. This is of most interest for queries doing writes where the query string can be large when large values passed (e.g. Mononoke sqlblob blobstore )
Reviewed By: krallin
Differential Revision: D26586715
fbshipit-source-id: e299932457682b0678734f44bb4bfb0b966edeec
Summary:
The previous fix D23357655 (d60e80796a) actually only fixes py2 absorb -i. On Python 3,
`b"-"[0]` is `45`, not `b"-"` like Python 2. Fix it again using `b"-"[0:1]`.
Reviewed By: singhsrb
Differential Revision: D26805315
fbshipit-source-id: 07ca850373a6bc49b561466ead478024631ce051
Summary:
Memcache is dogslow to initialize, taking >30s on debug build. As a
consequence, this slows down every single test by that amount time, with the
guarantee that no blobs will be found in memcache, ie: a total waste of time.
On release builds, Memcache is significantly faster to initialize, so let's
only disable initializing Memcache for debug builds only.
Reviewed By: fanzeyi
Differential Revision: D26800265
fbshipit-source-id: 8b393c603414de68268fdadb385de177e214a328
Summary:
Adds a new hg py3 windows package to hgbuild for publishing. Currently
the tests don't run. I'll do that next.
Reviewed By: quark-zju
Differential Revision: D26768336
fbshipit-source-id: bd4533abbbc1e1c861aa9995b7a3868a7f6a1a22
Summary: Trigger the cleanup logic automatically if there are too many remote bookmarks.
Reviewed By: sfilipco
Differential Revision: D26802251
fbshipit-source-id: 1ab40c7be98c507865ea17001dc9775b6edf4446
Summary: This is handy to make the `sl` output cleaner.
Reviewed By: sfilipco
Differential Revision: D26802250
fbshipit-source-id: 1b74f3d9a42ab6b0780f07bec59b03a5cd0ea6a9
