Summary:
update thrift interface to include a function that returns attributes for a given list of files.
to support this, we add several structures that define the functions parameters and return type.
- FileAttributes: a "bitmask" enum. Each attribute is a power of 2 to enable bitwise ORing attributes together to decide which attributes to fetch from files.
- FileAttributeData: struct to hold the attributes of an individual file
- FileAttributeDataOrError: union that is either a FileAttributeData or EdenError thrown when accessing the file attributes
- GetAttributesFromFilesParameters: self explanatory.
- GetAttributesFromFilesResult: self explanatory. Map of file -> FileAttributeDataOrError
- getAttributesFromFile(): actual function that will be implemented in EdenServiceHandler.cpp
Reviewed By: chadaustin
Differential Revision: D32155168
fbshipit-source-id: 6dd768f3bcee1e32a2362df231feebfb1297dbf6
Summary:
It would be nice to be able to quickly turn it off in case of problems.
Hopefully we'll never need this lever, but better not need it but have it, then
have it not need it.
Reviewed By: yancouto
Differential Revision: D32491454
fbshipit-source-id: 8a84bc0cd955e0121aa632eab5ba7b98fef724c8
Summary:
Previously we've been just looking at the small repos in a given version to
decide if this is related or not. This is wrong - versions for all pre-merge fbsource
commits have no small repos in it, and yet we want to find this versions when
trying to sync a commit from fbsource to ovrsource.
FWIW just looking at repos in given version is not enough to understand if a
repo and version are related. What tells them if they are related is common
config - if common config contains a given repo, then it's indeed related.
This is what this diff does.
Also I had to debug some of the failing tests, and pretty_assertions made them easier to debug, so I decided to leave pretty_assertions here.
Reviewed By: mitrandir77
Differential Revision: D32461769
fbshipit-source-id: 9c9a57aa71e20cec8448602dfbe0248a6bb850f7
Summary:
Thanks to the previous diffs we can continue removing methods that use current
version, and this is what this diff does.
Reviewed By: mitrandir77
Differential Revision: D32428739
fbshipit-source-id: 047d07f6ceea5dcf9dd1f8f71de6fd8aeccc624d
Summary:
Previously SyncedAncestorsVersions struct had to keep track of NotSyncCandidate
separately because they didn't have a version. Now that they have it we can
remove a separate boolean field in SyncedAncestorsVersions and treat
NotSyncCandidate in the same way
Reviewed By: mitrandir77
Differential Revision: D32490221
fbshipit-source-id: 718ee6d3877a7755d8bef60b7b27ce4a00e6a4d8
Summary:
This diff is the diff that actually (and finally!) makes it easier to add new
small repos to an existing large repo.
The key change is in
eden/mononoke/commit_rewriting/cross_repo_sync/src/commit_sync_outcome.rs,
function get_plural_commit_sync_outcome. Now when syncing in large->small
direction if there's no entry in synced_working_copy_equivalence mapping then
we check if a mapping for a given large repo commit contains a given small repo
at all. If no then we just consider it a NotSyncCandidate. The upside of that
is that adding a new small repo wouldn't require adding new mapping for existing large repo commits.
The rest of the diff does two main things:
1) Add a version field to NotSyncCandidate.
2) Threads CommitSyncDataProvider and CommitSyncDirection to
get_plural_commit_sync_outcome.
Both of these changes are necessary and splitting them in separate diffs was
impractical.
Reviewed By: mitrandir77
Differential Revision: D32428738
fbshipit-source-id: cf6ff07cd8c7e6253d8be33e2b3ed36d964ab014
Summary: This checks the error at the beginning of the function and is slightly simpler, and matches the style of the other error checking.
Reviewed By: markbt
Differential Revision: D32471435
fbshipit-source-id: 1f1ff6fd4312083fb94ba5fc5ddf631ee0cc1ceb
Summary:
This diff allows specifying a `copy_from_bubble` on a lookup request. If the file is not found in your usual bubble during a request, it will check the `copy_from` bubble, and if the file is available there, will copy its content, so it doesn't need to be reuploaded.
This will be used to optimise uploads on snapshots, which often share a lot of files.
Reviewed By: markbt
Differential Revision: D32471433
fbshipit-source-id: 6c913ae79fbf19f8162d4faac2bad216294f6fb9
Summary: We currently have 3 functions for this, one for each type of hash. This diff makes it a single function, using `FetchKey` which is used to hold all types of hash. This makes code simpler and it still works the same.
Reviewed By: markbt
Differential Revision: D32467694
fbshipit-source-id: f7ed501df129e5f2883d2041c0517409457e48e6
Summary: This simply moves the logic to lookup files to another function, as I plan to make it more complicated, and don't want the main function to get more complicated.
Reviewed By: markbt
Differential Revision: D32467695
fbshipit-source-id: 97df6d1b3fb8fa4bb62e5d367e8a28209420ce72
Summary:
See D31935606 (82bc644036) for high level explanation.
## What this diff
Remove logic on client and server to deal with `index` and `token` fields.
## Land safety
This needs to be landed after the previous diff deploys everywhere. It is safe to deploy in any order because:
- If server deploys first, it will stop sending `usize` and ìndex` fields, which will be defaulted by serde, but the client code already uses `result` token if it's not default, which it's already not as the previous diff deployed.
- If client deploys first, we stop looking at `usize` and `index` fields, but since previous diff already deployed, `result` always has a value so the behaviour is the same.
Reviewed By: StanislavGlebik
Differential Revision: D32000221
fbshipit-source-id: ded7591448c16b4e1c1264cf0409d996a8834d97
Summary: I was a little afraid there might be corner cases for the snapshot update optimisation, so I added a bunch more tests on snapshot update, to make sure it always restores the snapshot correctly, regardless of how the working directory state is.
Reviewed By: StanislavGlebik
Differential Revision: D32314413
fbshipit-source-id: 2e42a5fe2eff499fd53a06be110b69eb7c1ad59d
Summary:
This is an optimisation for snapshots. If we're restoring an snapshot, and the files are already downloaded with correct content, we don't need to delete and redownload them.
This is useful for "syncing" mechanisms that quickly create and restore snapshots, as most of the files will be the same between restores, and thus don't need redownloading.
Reviewed By: StanislavGlebik
Differential Revision: D32288548
fbshipit-source-id: 6572c3d49584d350f06c2f8e2a3ac57e7c8696c9
Summary: It's used in a few tests, and I don't see a reason to not initialize it here
Reviewed By: mitrandir77
Differential Revision: D32490222
fbshipit-source-id: 58c1a08379636b529247e1b3cde64098e536d468
Summary:
Add support for nullable value_len fields to sqlblob chunk tables and population of them from sqlblob_gc mark phase.
For put() this change adds a chunk_generation insert if the UpdateGeneration on put shows no rows affected. This is so that that chunk_generation.value_len can be relied on to be present for new rows. The insert is done after chunk and data put so as to not introduce a "chunk_generation without data key" state.
Reviewed By: farnz
Differential Revision: D32317235
fbshipit-source-id: 83de98350a59aab6905a7e6cf9f64d8f89c7682e
Summary:
The commit cloud service uses hex nodes, as it uses JSON as its usual
serialization format. However, historically these have leaked into the
Mercurial internals.
In D21554672 (b2c1d90f22) we converted the fake dag for cloud smartlog to use the `dag`
module, however since this requires binary nodes, we converted the hex
nodes to binary by encoding them (i.e. they ended up with hex-encoded nodes
stored in a bytes object).
This leads to further issues when other things are expecting real binary
nodes, but instead get binary-encoded hex nodes. For example, if these
nodes are passed to the mutation store to do look-ups.
Instead, let's use the real binary node as the keys in the fake dag.
Reviewed By: DurhamG
Differential Revision: D32361889
fbshipit-source-id: bfeab9ea25741ca76db91617d6bc6bf094adf679
Summary:
Cast EDEN_MICRO to something fmt::format can handle: char* (from std::basic_string<char8_t>); but that means this is no longer a literal, so cannot concat. Move to its own fmt argument.
This avoids the following errors:
eden/fs/service/EdenServiceHandler.cpp:227:60: error: invalid operands to binary expression ('basic_ostream<char>' and 'std::basic_string<char8_t>')
Differential Revision: D32486517
fbshipit-source-id: c799bdd73be1b46cc938b2076ebc78d0bdb3dded
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
reinterpret_cast another u8"..." literal to const char*
This avoids the following errors:
```
stderr: In file included from eden/fs/utils/test/Utf8Test.cpp:9:
In file included from folly/portability/GTest.h:32:
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1545:11: error: invalid operands to binary expression ('char8_t const[4]' and 'const std::basic_string<char>')
if (lhs == rhs) {
~~~ ^ ~~~
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1572:12: note: in instantiation of function template specialization 'testing::internal::CmpHelperEQ<char8_t [4], std::basic_string<char>>' requested here
return CmpHelperEQ(lhs_expression, rhs_expression, lhs, rhs);
^
eden/fs/utils/test/Utf8Test.cpp:43:3: note: in instantiation of function template specialization 'testing::internal::EqHelper::Compare<char8_t [4], std::basic_string<char>, nullptr>' requested here
EXPECT_EQ(u8"\uFFFD", ensureValidUtf8("\xff"));
```
Differential Revision: D32484896
fbshipit-source-id: b271292eb121e0e183bec56ab51fc6acb791b052
Summary:
It turns out repo.sparsematch() is called in a tight loop in some
places, like during checkout's record update phase. Let's cache it in memory.
This is a little different from the old caching mechanism because 1) it's only
ever in memory, so we don't need to worry about persisting it over time and
invalidating it when the sparse profile configs change, and 2) it only hashes
the inputs, and not the whole config file. The one input that is not part of the
cache key is the dynamic sparse profile config. Since that input has strict
invalidation points, we can just invalidate the sparse cache when the dynamic
sparse profile config invalidated.
Differential Revision: D32474918
fbshipit-source-id: 35593d06dac420e7a7fee47c5e1602413541de5f
Summary:
On Windows, directory removal notifications can be received out of order, and
sometimes duplicated. The next diff will rework the way EdenFS handle
notifications on Windows by looking at the on-disk state and applying it to the
inode hierarchy. In some cases, an entire directory hierarchy will need to be
removed which the added function will make it easier to do.
Reviewed By: chadaustin
Differential Revision: D32329806
fbshipit-source-id: ad9af389c993420184c0e59dd3d25188adddb77a
Summary: As the title says. More details can be found in the associated task.
Reviewed By: quark-zju
Differential Revision: D32346505
fbshipit-source-id: 00494165a57714dd8f8e6d3f36920a1342c121c7
Summary: A default impl that errors was added to EdenApi trait, but this impl still has a bunch of similar implementations, let's clean them.
Reviewed By: DurhamG
Differential Revision: D32314851
fbshipit-source-id: 7899c9ea23dc56bb83962d3f8625023f14b5cb4f
Summary:
This is an attempt to import `move-vm-runtime` and some other core crates of the Move programming language into fbsource.
The imported Move crates contains a few patches and are different from the official release in the following ways:
1. `diem-workspace-hack` regenerated in disabled mode
2. `diem-types` does not enable the default features for its dependency `diem-crypto`
3. `bytecode-interpreter` does not enable the default features for its dependency `bytecode-interpreter-crypto`
All of these patches are needed to help avoid enabling conflicting features at the same time.
#buildmore
Reviewed By: shazqadeer
Differential Revision: D29804394
fbshipit-source-id: 4fe3ba153437062a8744ffdf8cfe528bab817a91
Summary:
In the next diff we are going to refactor CommitSyncOutcome::NotSyncCandidate
to have a version, which would mean that the version should always be present
even for NotSyncCandidate
So in this diff let's make mapping version non-optional. Note that at the moment our
prod tables have some of the mapping entries with a non-empty version - I'm
going to fix it before landing this diff.
Reviewed By: mitrandir77
Differential Revision: D32428740
fbshipit-source-id: be9f8b104a14497bf5a524c68c0fd53beba22ee1
Summary:
Use the more efficient implementation.
This is about 100x faster. In my repo with `len(ml.roots())` being 10k+:
Before:
In [1]: %time [ml.checkout(r) for r in ml.roots()];1
CPU times: user 4.97 s, sys: 2.75 s, total: 7.72 s
Wall time: 8.72 s
After:
In [1]: %time [ml.checkout(r) for r in ml.roots()];1
CPU times: user 76.9 ms, sys: 16.5 ms, total: 93.4 ms
Wall time: 92.2 ms
Reviewed By: DurhamG
Differential Revision: D32436360
fbshipit-source-id: 6c993ea70bea5f45d0e2d1b410180f791c9fdd96
Summary: This will be used to checkout previous versions of metalog relatively cheaply.
Reviewed By: DurhamG
Differential Revision: D32436359
fbshipit-source-id: 97ca4512f63831dcc4dc109bd62f734a3e5c80c2
Summary:
This will be used to implement an efficient `checkout` without re-opening the
logs.
Reviewed By: DurhamG
Differential Revision: D32436358
fbshipit-source-id: 12edb685d88c140ffc9f76cf996ee8f3b146d4aa
