Summary:
Fast-track some tricky-to-avoid getxattr requests from the kernel
itself. It would be best to avoid the context switch entirely, but in
lieu of that, at least don't log anything or enter our more expensive
FUSE handling path.
This is follow-up to D24039130 (37f030ba72), which apparently did not completely
eliminate POSIX ACL lookups.
Reviewed By: xavierd
Differential Revision: D26803589
fbshipit-source-id: 18cba8e3ffc45516e6458872e408ed29a562c7a8
Summary:
Taking a fuse_setattr_in arg means that it can only be used on FUSE, and while
FUSE has been the only backend on UNIX for a while, this is changing with NFS
being added. Thus, we need to find a solution that isn't tied to FUSE.
The solution is fairly simple, let's simply have a struct with optional fields
that needs changing, FUSE and NFS will set these to what needs changing.
Reviewed By: chadaustin
Differential Revision: D26742213
fbshipit-source-id: 16e3e8cdd22d88ace43485d9e3744280de1ee8ad
Summary:
As its name implies FuseDispatcher::Attr is purely FUSE related and thus
shouldn't affect the inodes. It also removes a blocker to using setattr for
NFS.
Reviewed By: chadaustin
Differential Revision: D26742214
fbshipit-source-id: 41a67af0c948d969d5a427f24718de5b134980da
Summary:
This merely adds the types needed for the SYMLINK RPC, the implementation will
follow.
Reviewed By: kmancini
Differential Revision: D26737273
fbshipit-source-id: 4ed3029304fe64892e88bc05a64b4b3b19fd5460
Summary: This is merely adding the type, the implementation will follow.
Reviewed By: kmancini
Differential Revision: D26737271
fbshipit-source-id: 42de7873b271a2bf9499f1274bca50f23dc1016b
Summary:
Both READDIR and READDIRPLUS RPC (and other) are using lists as a way to have
an unsized array of values. From a behavior perspective, this is similar to an
array of XdrOptionalValue but without it being prefixed by a size, and with an
additional last element being empty.
To simplify writing these RPC, let's add this XdrList type.
Reviewed By: kmancini
Differential Revision: D26729816
fbshipit-source-id: 8d14bbc6f0513aac51d65625e5751cedc2071a0b
Summary:
The recent change to make list-bookmarks be served from the warm bookmark cache
made this test flaky.
Ensure the warm bookmark cache has been populated with all of the bookmarks
before starting the tests.
Reviewed By: StanislavGlebik
Differential Revision: D26847951
fbshipit-source-id: 7e26c3745600fa8c1a337e8a8b47af6cca2f3eff
Summary:
Turned out that some of our commit cloud commits dont have an entry in changesets table,
but they do have an entry in bonsai_hg_mapping. It happens because of transient mysql errors.
Because of the mess we have in how we determine if a given hg changeset is present in mononoke or not,
this broken commit is considered backed up by commit cloud, while in reality its impossible to pull this commit.
This diff fixes it by first inserting changeset entry, and then bonsai_hg_mapping.
This way if an insertion of a changeset entry fails then the commit is not
considered to exist at all.
If an insertion of bonsai hg mapping fails, then commit is considered to be inserted but the whole
operation (it can be "hg cloud sync" for example) is considered failed. In this case hg client will
try to reupload this commit later.
But there's one caveat.
Hg changesets and bonsai changesets are not always round-trippable i.e.
generating from bonsai to hg and hg to bonsai might not always
produce the same results. So it's possible to push a commit which inserts changeset entry but not hg bonsai entry,
then we accidentally derive hg changeset from this bonsai and it's different from what user has pushed to mononoke.
This could lead to problems later (i.e. re-pushing the same commit again later would fail).
Another solution (inserting bonsai hg mapping and changeset entry in the same transaction) could make this issue
less likely, but it won't eliminate it completely. We discussed it, and I think it probably make sense to go
with the simpler fix first, since this problem should happen rarely in practice
(and if it happens often then we should prioritize fixing it).
Reviewed By: krallin
Differential Revision: D26845779
fbshipit-source-id: 61f8b0d835dc4aa0130e6be632c43307ff4a771f
Summary: Like it says in the title. I also replaced one of our status codes that was wrong.
Reviewed By: johansglock
Differential Revision: D26844865
fbshipit-source-id: b8c1261d0077cf5dc006827e16667e382db7d189
Summary: This diff removes the split between manually managed and autocargo managed Cargo.toml files in `eden/scm/lib`, now all files are autogenerated.
Reviewed By: quark-zju
Differential Revision: D26830884
fbshipit-source-id: 3a5d8409a61347c7650cc7d8192fa426c03733dc
Summary:
The GitTreeDict throws when we're trying to remove the root directory but there
are valid situations where the Tree becomes briefly empty. One of such is a
commit where all the files in the repo are moved. We process the deletions
first. Then we proceed with additions.
Reviewed By: DurhamG
Differential Revision: D26778717
fbshipit-source-id: 6caa1709ff2c5e78d4745336b0527af4bb20ec60
Summary:
These aren't tied to the RPC protocol but more to how XDR works, therefore it
makes more sense to move these to the Xdr.h file. It also enables tests to use
it instead of having to write the serialization by hand.
Reviewed By: chadaustin
Differential Revision: D26729815
fbshipit-source-id: 227299b94e2f6fa5ecc747a8d5705f0931a7d1f8
Summary: This merely adds the types for the SETATTR RPC.
Reviewed By: kmancini
Differential Revision: D26707362
fbshipit-source-id: 89cbf7a501cf4e13878cd84b6c36691d2a08594f
Summary:
Now that a large part of the RPC is implemented, being able to see what
arguments are being passed in makes it convenient for debugging.
Reviewed By: kmancini
Differential Revision: D26707363
fbshipit-source-id: 676adef996bf584140b324df1419b139f4600c91
Summary:
This reads the data from the file. One of the optimization that isn't done in
this diff would be to detect the EOF without having read return a 0-bytes long
buffer. For now this isn't done, but more plumbing in the FileInode would be
required to achieve this.
Reviewed By: kmancini
Differential Revision: D26704138
fbshipit-source-id: 82e6d4994dd5dc46372778c10d9f68f7ca1e3a87
Summary: This merely adds the types for the READ RPC.
Reviewed By: kmancini
Differential Revision: D26704137
fbshipit-source-id: a803fc9082cbec6ce0c6b9db2bfede55bf478df1
Summary:
While creating the .hg directory, Mercurial seems to want to remove some files
too, thus let's thread this to the inode layer to allow Mercurial to continue.
Reviewed By: kmancini
Differential Revision: D26699272
fbshipit-source-id: e6d283b92e18962698c0df3c6361adfbd0c50223
Summary:
All the callers of these methods have a real context in hand, let's use it
instead of creating a null one.
Reviewed By: genevievehelsel
Differential Revision: D26699271
fbshipit-source-id: 9fb268c9b3194d7e951e30ab5e90c4f2f0388e81
Summary:
Native checkout avoids the initial large prefetch, and instead does
small batch prefetches and immediately writes them to disk. This will prevent
the issue where data in the cache is evicted before it's used and has to be
redownloaded in a non-batch fashion later.
In a future diff, using native checkout will also allow us to serialize and
deserialize it's CheckoutPlan, so we can make clones resumable.
Reviewed By: quark-zju
Differential Revision: D26825773
fbshipit-source-id: 6fde90cd7578f6af5fdd4ac78004f7f63af4f287
Summary: the list of configured default selective pull bookmarks are actually related to the hoist namespace I think when we consider what is protected and what is not.
Reviewed By: quark-zju
Differential Revision: D26814708
fbshipit-source-id: 566f363f1e6b88dd2ebc6e33cfb6eba596493988
Summary:
Mononoke hg sync job does the following operation in the loop:
1) check if we got new bookmark update log entries in db
2) Sync these log entries to hg server
3) update the counter that says what was the latest synced entry
It's possible that step 2 was successful but step 3 failed. After that sync job restarts and tries to replay already-replayed bundle again, and this fails.
We had a protection against this situation - after every failed bundle we sent a request to hg server to check what's the location of bookmark is - however it stopped working after we added `--combine-bundles` option which syncs more than 1 bookmark update log entry at once.
To be precise, this situation could happen
1) Sync job syncs log entries [1, 2] successfully
2) Sync job tries to update the `latest-replayed-request` counter to 2, and fails
3) New log entry 3 comes into queue
4) sync job restarts and tries to sync [1, 2, 3]. This fails because [1, 2] were already synced. Because of that sync job tries to verify that the location of the bookmark on the server is at the position 3, but actually it's at 2. So sync job fails again and it keeps crashlooping.
This diff attempts to fix it by trying to detect if any of the first entries in the first bundle were already synced. It does so by taking the value of the bookmark from BookmarkOverlay (it's equivalent to the value of the bookmark on hg server), and comparing with to_changeset_id/from_changeset_id values from the batch, and skipping a first few. See code comments for more details on how it does it.
A few notes:
1) BookmarkOverlay might actually contain outdated values i.e. not the latest snapshot of hgsql. This happens because there's a delay between commit appearing on hgsql and propagating to hg server repositories. In a case like that it's possible that adjusting the first batch might not kick in, and sync job will fail to sync the bundle. However my assumption is that in practice it should be ok - sync job will restart and try to repeat the bundle adjustment, and eventually it should succeed. Also the current `verify-server-bookmark-on-failure` likely to have the same problem.
2) Note that this still might not work if we try to decrease the combine-bundles value when sync job counter is pointing to the wrong log entry. I expect it to be rare enough so that we don't need to worry about it
Note that it still worth keeping the `verify-server-bookmark-on-failure` option, since it can detect other kinds of failure (e.g. hg command returning non-zero exit, but actually successfully syncing the entry).
Reviewed By: krallin
Differential Revision: D26753763
fbshipit-source-id: bea9da9ab1ceede19666c99e28553e74edb0ed2a
Summary:
We use itertools group_by() to group filenodes. Unfortunately that only works
if they're sorted, which they're not. With a hundred shards, we easily end
up sending one query per filenode.
This fixes that to group the filenodes properly. This means we'll make 1 query
for 1000 filenodes, i.e. make 1000x fewer queries.
Differential Revision: D26821973
fbshipit-source-id: b83ccaff608f38adefe86c8f5018e4b525cbde11
Summary:
On POSIX systems it's a good practice to fsync directories to ensure metadata
about a file is properly written.
Reviewed By: DurhamG
Differential Revision: D26822211
fbshipit-source-id: fca10c702b480f22020ad184eb55c8879999f0ee
Summary:
Typical mistake of `reposetup` - the `repo` object might be remote and lack of
local repo methods.
Reviewed By: DurhamG
Differential Revision: D26822960
fbshipit-source-id: 54fb95c51b09ef5021b0a10a93b4509e7aa4a115
Summary:
We're missing some metrics to measure how much queries we are making here, and
our error reporting isn't very comprehensive. Let's improve this.
Reviewed By: StanislavGlebik
Differential Revision: D26819845
fbshipit-source-id: c78b8dee02d58dc09ea5f5bd9712e5dfa0bb3f29
Summary:
The recent change to make `list-bookmarks` read from the warm bookmark cache
has made this test flaky, as we must wait for the warm bookmark cache to update
before the new values are visible.
Update `wait_for_bookmark_update` to use `list-bookmarks` which always uses
the warm bookmark cache. A similar function is also needed for bookmark deletion.
Reviewed By: StanislavGlebik
Differential Revision: D26816627
fbshipit-source-id: e99b83454070d565646b069177e18bd6a01946ae
Summary: This is merely adding the types needed to implement the REMOVE procedure.
Reviewed By: kmancini
Differential Revision: D26694441
fbshipit-source-id: 736d06ed7bfae40f827fbec6017e069fe11d7464
Summary:
Since EdenFS doesn't support hardlinks, this is merely collecting the attribute
of the file before always failing.
Reviewed By: kmancini
Differential Revision: D26682014
fbshipit-source-id: 8c39170147023a03e87ed7a0fae66f9184b4f690
Summary:
The RPC simply queries various filesystem attributes, we merely forward what
statfs on the overlayfs gives us.
Reviewed By: kmancini
Differential Revision: D26681613
fbshipit-source-id: 5b94d05cafff8d77390fe60a3b5cf1dc3e022f42
Summary: This merely adds the types for the RPC.
Reviewed By: kmancini
Differential Revision: D26681615
fbshipit-source-id: d092cf0b6b5bb7435702d125b5c6ea7ee68356dc
Summary:
This simply writes the data passed in to the inode. Note that the current
implementation has a protocol violation since it doesn't sync the data written
to disk but advertise to the client that it did. This is completely wrong from
a data consistency guarantee, but is probably fine for now. Once the code
becomes closer to being production ready, this will be changed to honor the
client asks.
Reviewed By: kmancini
Differential Revision: D26681614
fbshipit-source-id: 82ad7a141be3bbe365363b1f6692ae62f253423f
Summary:
The Appender API doesn't allow us to simply append an IOBuf chain to it forcing
the entire chain to be copied to it. For the most part, this isn't an issue,
but to reduce the overhead of the READ NFS procedure, this will be a nice
benefit.
For the most part, this diff is a codemod, with a couple of places that needed
to be manually fixed. These were in the rpc/Server.{cpp,h} and
rpc/StreamClient.{cpp,h}.
Reviewed By: genevievehelsel
Differential Revision: D26675988
fbshipit-source-id: 04feef8623fcddd02ff7aea0b68a17598ab1d0f8
Summary:
The ser, de and roundtrip are duplicated 3 times in the various tests, let's
move them into a common library.
Reviewed By: chadaustin
Differential Revision: D26675989
fbshipit-source-id: 1da0bc33429795a889b72c76fa18e5fa3cb7df6f
Summary: This merely adds the various types for the WRITE RPC.
Reviewed By: chadaustin
Differential Revision: D26671895
fbshipit-source-id: 8409c8a1f90e97478aed7c9f19b881c46234b539
Summary:
For the READ/WRITE RPC calls, copying data in and out of an IOBuf chain can be
fairly expensive, to avoid this overhead, we can simply clone the data out of
the IOBuf chain directly, saving on the cost of copy.
Since the code uses a folly::io::Appender that doesn't support adding an IOBuf
to it, we still pay the cost of copying data to it, switching to
folly::io::QueueAppender may solve this.
Reviewed By: chadaustin
Differential Revision: D26671896
fbshipit-source-id: 0161f04cb820bf27ef66fdef6b4a1ce4eb778b96
Summary:
The CREATE RPC is intended to create regular files on the server. It supports 3
ways of creating them: unchecked, guarded and exclusive. The exclusive mode
allows for a client to being able to re-transmit the creation and for it to not
fail when that happens. The way this is implemented requires the server to
store a cookie passed in by the client and on retransmission to compare it in
order to not fail the file creation. This complexity may not be warranted and
needed in the case of EdenFS, and the spec allows for a server to not support
this operation, thus EdenFS claims to not support it.
The 2 other modes are more traditional, guarded will simply fail if the file
already exist, while unchecked succeed in that same situation. It's not
entirely clear in the RFC what the behavior is supposed to be in the unchecked
case, and whether the attributes needs to be applied. For now, the error is
just ignored, a warning message is logged and a success is returned in this
case. In the future, setting the file attribute (in particular the mode and
size) may be necessary.
Reviewed By: kmancini
Differential Revision: D26654766
fbshipit-source-id: c2e8142b8a4ff756a868e5859fdda4b07e53bddf
