Summary: More hooks will come in next diffs.
Reviewed By: aslpavel
Differential Revision: D23449755
fbshipit-source-id: 451fdb7a759140f2d6df8f3a18493c700fa2b761
Summary:
That's one of the sev followups. Before redacting a file content let's check if
it exists in "main-bookmark" (which is be default master), and refuse to redact
if it actually exists.
If this check passes (i.e. the content we are about to redact is not reachable
from master) that doesn't mean that we are 100% safe. E.g. this comment can be
in ancestor of master, or in any other repo or it can be added in the next
commit.
This check is a best-effort check to prevent shooting ourselves in the foot.
Reviewed By: aslpavel
Differential Revision: D23476278
fbshipit-source-id: 5a4cd10964a65b8503ba9a6391f17319f0ce37d8
Summary:
The loop took care to advance `b` to match the amount
of data that it had processed, but was still passing `buf`
(the unadjusted start of the buffer) to the syscalls.
This meant that in situations where a `readFull` might
encounter a partial read, it would scribble over the start
of the buffer and leave junk at the end.
For example:
write("hell");
write("o");
could produce "oell?" in the buffer when `readFull` consumes
the other end of the pipe.
Reviewed By: xavierd
Differential Revision: D23486270
fbshipit-source-id: 0848f6789b44421b609b91fe08890768ff59f7f5
Summary:
Currently we use a single path prefix to configure data fetch logging in eden
(i.e if the path of a file which we fetch is an extension of our configured
path, then we log that data fetch. )
There is some interest in extending this to multiple path prefixes, so that we
can log separate parts repo.
Reviewed By: StanislavGlebik
Differential Revision: D22877942
fbshipit-source-id: f6eb3dcb4fa460b4acab09677e972caf9421ddff
Summary:
We use Re2 in D22877942 for parsing multiple path prefix data fetch logging,
this introduces the dependency for eden's opensource builds.
Reviewed By: chadaustin
Differential Revision: D23431175
fbshipit-source-id: 44b399e92cb89ba1403295ecd10bc8f8d769b02c
Summary: This code can be used on Mac as well, so I can just move it to `UnixProcUtils` to be shared. I think to start it we can just try using this before trying to add special idleness detection with looking for active screensavers etc.
Reviewed By: fanzeyi
Differential Revision: D23183163
fbshipit-source-id: fffad8314e70f8726836c482f7a5e30e57a75c0d
Summary: We don't need to restart users if their running version is the same as their installed version, so we should check that when deciding if we should restart. This will give us more freedom in restarts since we won't have to play with `min_uptime`. I will add a flag to skip this check in case for some reason we need to do so on the fly.
Reviewed By: wez
Differential Revision: D23438306
fbshipit-source-id: b17c0e13789071b8b7c1b15ac5a8deb74a4fd091
Summary: I want to be able to reverse engineer an EdenInstance in the `edenfs_restarter` given the cmdline of the process. I think this best lives in the `config.py` file.
Reviewed By: fanzeyi
Differential Revision: D23438318
fbshipit-source-id: b3d9ac3981d3fb2bb8045b07b8d949cd601f6898
Summary: The latter will not strip new lines from the system error message, while the former does.
Reviewed By: genevievehelsel
Differential Revision: D23480435
fbshipit-source-id: 44742b960935552fa1781ed19f38ff446a8c9403
Summary:
Change dag_ops benchmarks to use different IdDagStores. An example run shows:
benchmarking dag::iddagstore::indexedlog_store::IndexedLogStore
building segments (old) 856.803 ms
building segments (new) 127.831 ms
ancestors 54.288 ms
children (spans) 619.966 ms
children (1 id) 12.596 ms
common_ancestors (spans) 3.050 s
descendants (small subset) 35.652 ms
gca_one (2 ids) 164.296 ms
gca_one (spans) 3.132 s
gca_all (2 ids) 270.542 ms
gca_all (spans) 2.817 s
heads 247.504 ms
heads_ancestors 40.106 ms
is_ancestor 108.719 ms
parents 243.317 ms
parent_ids 10.752 ms
range (2 ids) 7.370 ms
range (spans) 23.933 ms
roots 620.150 ms
benchmarking dag::iddagstore::in_process_store::InProcessStore
building segments (old) 790.429 ms
building segments (new) 55.007 ms
ancestors 8.618 ms
children (spans) 196.562 ms
children (1 id) 2.488 ms
common_ancestors (spans) 545.344 ms
descendants (small subset) 8.093 ms
gca_one (2 ids) 24.569 ms
gca_one (spans) 529.080 ms
gca_all (2 ids) 38.462 ms
gca_all (spans) 540.486 ms
heads 103.930 ms
heads_ancestors 6.763 ms
is_ancestor 16.208 ms
parents 103.889 ms
parent_ids 0.822 ms
range (2 ids) 1.748 ms
range (spans) 6.157 ms
roots 197.924 ms
benchmarking dag::iddagstore::bytes_store::BytesStore
building segments (old) 724.467 ms
building segments (new) 90.207 ms
ancestors 23.812 ms
children (spans) 348.237 ms
children (1 id) 4.609 ms
common_ancestors (spans) 1.315 s
descendants (small subset) 20.819 ms
gca_one (2 ids) 72.423 ms
gca_one (spans) 1.346 s
gca_all (2 ids) 116.025 ms
gca_all (spans) 1.470 s
heads 155.667 ms
heads_ancestors 19.486 ms
is_ancestor 51.529 ms
parents 157.285 ms
parent_ids 5.427 ms
range (2 ids) 4.448 ms
range (spans) 13.874 ms
roots 365.568 ms
Overall, InProcessStore > BytesStore > IndexedLogStore. The InProcessStore
uses `Vec<BTreeMap<Id, StoreId>>` for the level-head index, which is more
efficient on the "Level" lookup (Vec), and more cache efficient (BTree).
BytesStore outperforms IndexedLogStore because it does not need to verify
checksum on every read access - the checksum was verified at store creation
(IdDag::from_bytes).
Note: The `BytesStore` is something optimized for serialization, and hasn't been sent.
Reviewed By: sfilipco
Differential Revision: D23438174
fbshipit-source-id: 6e5f15188e3b935659ccde25fac573e9b963b78f
Summary: This allows them to use the SyncableIdDag APIs.
Reviewed By: sfilipco
Differential Revision: D23438170
fbshipit-source-id: 7ec7288cfb8186b88f85f0212a913cb0dffe7345
Summary: Other IdDagStores can also use the API. This will be used in benchmarks.
Reviewed By: sfilipco
Differential Revision: D23438180
fbshipit-source-id: 565552b66372dcfbb268c397883f627491d6e154
Summary:
Similar to `IdDagStore::sync` -> `GetLock::persist`, `reload` is more related
to filesystem/internal state exchange, and should be protected by a lock. So
let's move the API there, and requires a lock.
Reviewed By: sfilipco
Differential Revision: D23438169
fbshipit-source-id: 4228106b7739a1a758677adfddd213ad54aa4b6a
Summary:
`NameDag::reload` is used in `flush` to get a "fresh" NameDag.
In a future diff the `IdDag::reload` API gets changed, so let's
remove NameDag's use of it.
Instead, let's just re-`open` the path again to get a fresh NameDag.
It's a bit more expensive but probably okay, and easier to understand.
`get_new_segment_size()` was added as an internal API to preserve tests.
This also solves an issue where `NameDag` cannot recover properly if its
`flush` fails, because the old `NameDag` state is not lost.
After removing `NameDag::reload`, `idMap::reload` is no longer used publicly
and was made private.
Reviewed By: sfilipco
Differential Revision: D23438179
fbshipit-source-id: 0a32556a2cd786919c233d7efcae1cb9cbc5fb09
Summary:
The word "sync" is bi-directional: flush + reload. It was indexedlog::Log's
behavior. However, in the IdDag context "sync" is confusing - it is actually
only used to write data out, with protection from lock. Rename to `persist`
to clarify it's memory -> disk. Besides, requires a reference to a lock object
as a lightweight prove that some lock is held.
Reviewed By: sfilipco
Differential Revision: D23438175
fbshipit-source-id: 3d9ccd7431691d1c4e2ee74f3c80d95f5e7243b5
Summary:
This removes the need of cloning `IdMap`.
SyncableIdMap is a bit tricky. I added some comments to clarify things.
Reviewed By: sfilipco
Differential Revision: D23438176
fbshipit-source-id: fe66071da07067ed6c53a6437790af1d81b28586
Summary:
Make the test cover IndexedLogIdDagStore. The only change is the parent index
returns children in a different order.
Reviewed By: sfilipco
Differential Revision: D23438173
fbshipit-source-id: bcfabcd329e45bbc5e7e773103fa42307c23c35d
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: simpkins
Differential Revision: D23465292
fbshipit-source-id: bb9df3ad183685fae28173da7275e6ecd34df048
Summary:
There aren't too many thigs that we can do with the responses that we get back
from the server. Thigs are somewhat application specific for this endpoint.
One option that is not available right now and might make sense to add is
limiting the number of entries that are printed for a given location.
Reviewed By: kulshrax
Differential Revision: D23456220
fbshipit-source-id: eb24602c3dea39b568859b82fc27b7f6acc77600
Summary:
To reduce the size over the wire on cases where we would be traversing the
changelog on the client, we want to allow the endpoint to return a whole parent
chain with their hashes.
Reviewed By: kulshrax
Differential Revision: D23456216
fbshipit-source-id: d048462fa8415d0466dd8e814144347df7a3452a
Summary:
Renaming all the LocationToHash related structures to CommitLocationToHash.
This is done for consistency. I realized the issue when the command for reading
the request from cbor was not what I was expecting it to be. The reason was that
the commit prefix was used inconsistently for LocationToHash.
Reviewed By: kulshrax
Differential Revision: D23456221
fbshipit-source-id: 0181dcaf81368b978902d8ca79c5405838e4b184
Summary:
Segmented Changelog is a component that has multiple components of each own
that each can be configured in different ways. It seems that it already is
more complicated than other components in how it is set up and it will probably
evolve to have more knobs (caching comes to mind).
Right now we have 3 ways of instantiating SegmentedChangelog:
- Disabled, all requests return errors
- ReadOnly, requests to unprocessed commits return errors
- OnDemandUpdate, requests trigger commit processing when required
Reviewed By: aslpavel
Differential Revision: D23456217
fbshipit-source-id: a6016f05197abbc3722764fa8e9056190a767b36
Summary:
Parsing is done in the SegmentedChangelogConfig structure which will inform
how to construct the SegmentedChangelog in Mononoke.
Reviewed By: aslpavel
Differential Revision: D23456222
fbshipit-source-id: a7d5d81f4c166909164026e81af57f1c2ea32347
Summary:
The Segmented Changelog must be built somewhere. One of the simplest deployments
of involves the on-demand update of the graph. When a commit that wasn't yet
processed is encountered, we sent it to processing along with all of it's
ancestors.
At this time not much attention was paid to the distinction of master commit
versus non-master commit. For now the expectation is that only commits from
master will exercise this code path. The current expectation is that clients
will only call location-to-hash using commits from master.
Let me know if there is an easy way to check if a commit is part of master.
Later changes will invest more in handling non-master commits.
Reviewed By: aslpavel
Differential Revision: D23456218
fbshipit-source-id: 28c70f589cdd13d08b83928c1968372b758c81ad
Summary:
This builders implements SqlConstruct and SqlConstuctFromMetadataDatabaseConfig
to make handling the Sql connection for IdMap consistent with what happens in
Mononoke in general.
Reviewed By: aslpavel
Differential Revision: D23456219
fbshipit-source-id: 6998afbbfaf1e0690a40be6e706aca1a3b47829f
Summary:
The trait provides two methods for location to hash translation. The first
returns a single hash and is existing functionality. The second returns a
list of hashes and represents new functionality. This diff also adds this
functionality to the Dag structure which is currently the only real
implementation for SegmentedChangelog.
Reviewed By: aslpavel
Differential Revision: D23456215
fbshipit-source-id: 0c2ca91672cf23129342c585f98446c0ebbdf7ef
Summary:
I am planning to add Segmented Changelog to attributes.
I am writing an integration test for an EdenApi endpoint that depends on
Segmented Changelog and I would like to set it up to update on demand. When a
request comes in for a commit that we haven't parsed for Segmented Changelog we
want to update the structure on demand. This means that we probably need to
fetch commits. This means that we want to pass the ChangesetFetcher to Segmented
Changelog when it is built. Since Segmented Changelog fits well as an attribute
we want the ChangesetFetcher as an attribute.
I wonder how much thought has been given to attributes behaving as a dependency
injector in the `guice` sense.
Reviewed By: aslpavel
Differential Revision: D23428201
fbshipit-source-id: 7003c018ba806fd657dd8f071e0e83d35058b10f
Summary:
This helps make RequestData slightly more generic by depending less on Fuse
specific types/code.
Reviewed By: chadaustin
Differential Revision: D23467487
fbshipit-source-id: 830f8269e2114c2968dcc49d3b5574e687191e4d
Summary:
This is to be able to automatically report progress: how many merges has been
done already.
Note: this intentionally uses the same logic as regular `gradual-merge`, so that we always report correct numbers.
Reviewed By: StanislavGlebik
Differential Revision: D23478448
fbshipit-source-id: 3deb081ab99ad34dbdac1057682096b8faebca41
Summary:
This is unused and is sadly Fuse specific, making it harder to move to inodes/,
thus, let's remove it.
Reviewed By: chadaustin
Differential Revision: D23467486
fbshipit-source-id: c34e1abe245cfb79b9414002fd1055b8c0a5f1d3
Summary:
None of these were used, let's remove them.
ps: I thought we had a system to detect unused headers and lint about them?
Reviewed By: chadaustin
Differential Revision: D23465783
fbshipit-source-id: c21a34c9838db29f4fd0057d3be4e0fcb527cd6d
Summary:
This is not per-se fuse related, thus move it to a common location and remove
the duplicated define in FileInode.h
Reviewed By: chadaustin
Differential Revision: D23465192
fbshipit-source-id: 5fa7709f127c2d3372ee5ea3aeb89e793ea5b9f7
Summary: This file is not fuse specific, therefore, let's move it to a non-fuse folder.
Reviewed By: chadaustin
Differential Revision: D23464460
fbshipit-source-id: f70e94bb0ecc37bd74798fd230dee2058484f31b
Summary:
In the code base, "channel" is used to denote the OS mechanism that sends
EdenFS notifications. In macOS and Linux, that's Fuse, on Windows, that's
ProjectedFS. To avoid platform specific naming in ObjectedFetchContext, let's
rename the fetch cause enum.
Reviewed By: kmancini
Differential Revision: D23462460
fbshipit-source-id: 3ac68cdf4999e6a3b4ff4ee266f94e1f9736df39
Summary:
The default archive behavior archives the entire working copy. That is
undesirable and easy to accidentally trigger in a large repository. Let's
prevent it and require users to specify what they want archived.
Reviewed By: quark-zju
Differential Revision: D23464818
fbshipit-source-id: c39a631d618c2007e442e691cda542400cf8f4c3
