Summary:
In the next diff I'm going to add log_only mode for redaction.
And in this diff I make a small refactoring that makes next diff simpler.
find_files_with_given_content_id_blobstore_keys don't accept tasks anymore,
just content keys.
Reviewed By: aslpavel
Differential Revision: D23535829
fbshipit-source-id: 1dac37f5ea7038fc779ad51192a290fcc23e6556
Summary: Use the new fields from RawHookConfig in HookConfig
Reviewed By: StanislavGlebik
Differential Revision: D23499766
fbshipit-source-id: 43e9d2dfdcfb0fa0dd4de6310ea0013db1b69474
Summary:
`C:/tools/eden` will be overriden whenever a new EdenFS package is installed, therefore making it unsuitable to be managed by Chef.
Changing the default configuration directory to `C:\ProgramData\eden` that aligns with other programs.
Reviewed By: xavierd
Differential Revision: D23484626
fbshipit-source-id: 763518c608b24caa08e089a738f5c3577a0d6483
Summary:
Removing Fuse from the enum name makes it non tied to Fuse and thus makes it
more portable. This also eliminates the last platform specific bit from
RequestData.
Reviewed By: chadaustin
Differential Revision: D23467773
fbshipit-source-id: 52515522c8ac51d0c4b56dc5e42d4b6593df6623
Summary:
The version is going to be used to seamlessly upgrade the IdMap. We can
generate the IdMap in a variety of ways. Naturally, algorithms for generating
the IdMap may change, so we want a mechanism for updating the shared IdMap.
A generated IdDag is going to require a specific IdMap version. To be more
precise, the IdDag is going to specify which version of IdMap it has to be
interpreted with.
Reviewed By: quark-zju
Differential Revision: D23501158
fbshipit-source-id: 370e6d9f87c433645d2a6b3336b139bea456c1a0
Summary:
Separate the operational bits of the IdMap from the core SegmentedChangelog
requirements.
I debaded whether it make sense to add repo_id to SqlIdMap. Given the current
architecture I don't see a reason not to do it. On the contrary separating
two objects felt convoluted.
Reviewed By: quark-zju
Differential Revision: D23501160
fbshipit-source-id: dab076ab65286d625d2b33476569da99c7b733d9
Summary:
Planning to add a trait for core idmap functionality (that's just translating
cs_id to vertex and back). The current IdMap will then be an implementation of
that trait.
Reviewed By: quark-zju
Differential Revision: D23501159
fbshipit-source-id: 34e3b26744e4b5465cd108cca362c38070317920
Summary:
Replacing places where the tokio runtime is instantiated inside the edenapi
client crate.
Reviewed By: quark-zju
Differential Revision: D23468596
fbshipit-source-id: ef68718c7d5b89b6477a2946daaa51618b53d06a
Summary:
At open time, it's pointless to attempt to create new levels. So let's just
read the existing max_level and do not try to build max_level + 1.
This turns out to save 300ms in profiling result.
Reviewed By: sfilipco
Differential Revision: D23494509
fbshipit-source-id: 4ea326a3cc21792790ea0b87e5bf608a94ae382b
Summary:
With MultiLog, per-log meta was previously entirely ignored. However, they can
be useful for updated indexes. For example, application defines a new index,
and opens a Log via MultiLog. The application would expect the new index is
built only once. Without MultiLog, per-log meta is updated at open time in
place. With MultiLog, the updated index meta is not written back to the
multimeta so the new index would be rebuilt multiple times undesirably.
Update MultiLog to reuse the per-log meta if it's compatible so it can pick up
new indexes.
Reviewed By: sfilipco
Differential Revision: D23488212
fbshipit-source-id: c8b3e6b5589dbda2e76a143d15085862a93dae22
Summary:
The poisoned meta makes investigation harder. ex. `debugdumpindexlog` won't
work on those logs.
Reviewed By: sfilipco
Differential Revision: D23488213
fbshipit-source-id: b33894d8c605694b6adf5afdaed45707fbd7357e
Summary:
I'm going to change this function soon, so it's nice to asyncify it to make
next diffs simpler and also remove duplicated logic.
Also remove unnecessary `logger` parameter - we can always get logger from CoreContext
Reviewed By: krallin
Differential Revision: D23501634
fbshipit-source-id: 7ad2fc17167e4107481ceb230e0b7cb3e7f2549a
Summary: This closely replicates EscapeZero work in D23328638 and will allow us to issue requests to SCS using Thrift Fiddle (https://www.internalfb.com/thrift_fiddle).
Reviewed By: EscapeZero
Differential Revision: D23475864
fbshipit-source-id: fb286e3fcd6ea79704fa2e7e1ed9ab5595ff7b81
Summary: Now that post-request callbacks are available in `gotham_ext`, we can make `StreamBody` use them directly instead of using an LFS-specific wrapper (previously required to access the LFS server's `RequestContext`). This also means that the EdenAPI server will get this behavior for free.
Reviewed By: krallin
Differential Revision: D23402969
fbshipit-source-id: 56ab710473f13e8983b136664af364af6884bd3f
Summary: Add `LogMiddleware` to the EdenAPI server, which will print a log message whenever a request is received or has completed.
Reviewed By: DurhamG
Differential Revision: D23299902
fbshipit-source-id: f44ef1b01692f0e4f9b109917fcee89a84ca4208
Summary: Use `LoadMiddleware` to track the number of outstanding requests in the server.
Reviewed By: DurhamG
Differential Revision: D23298415
fbshipit-source-id: bdcdb0f657d8deac593d356c87ac0d8d3f39e322
Summary: Now that `LogMiddleware` no longer depends on `RequestContext`, it can be moved into `gotham_ext`.
Reviewed By: DurhamG
Differential Revision: D23298412
fbshipit-source-id: d5288decba98c3dd4605b9a44e41eba0f47fee37
Summary: Now that `LoadMiddleware` no longer depends on `RequestContext`, it can be moved into `gotham_ext`.
Reviewed By: DurhamG
Differential Revision: D23298416
fbshipit-source-id: 5d29da492e39beb5621daf0570d9b3e657cbfc04
Summary: This diff removes the post-request callback functionality from the LFS server's `RequestContext` and replaces it with the new `PostRequestMiddleware`. The middleware is directly based on `RequestContext`, so the underlying behavior is essentially the same as before.
Reviewed By: krallin
Differential Revision: D23298413
fbshipit-source-id: 1e58a40f6ce6d526456dbd9ae3a8efc85768bf04
Summary: Make `PostRequestMiddleware` generic over a user-provided config struct which can be used to dynamically configure the behavior of post-request callback dispatching. Right now this is only used to support disabling hostname logging, but could be easily extended to cover more uses in the future.
Reviewed By: krallin
Differential Revision: D23495005
fbshipit-source-id: 3d59a8346f449775ec76d03c260d973d04fb90a9
Summary: Add new middleware that allows HTTP handlers and other middleware to register callbacks that will be run once the current request completes. This is heavily based on the post-request callback functionality from the LFS server's `RequestContext`. The intention here is to expose this functionality in a manner that's independent of other, application-specific logic.
Reviewed By: krallin
Differential Revision: D23298419
fbshipit-source-id: e4b1534b02c35f685ce544de13e331947e187818
Summary:
I pattern matched off of this for the previous diff in this stack, and spotted
a bit of clean up that might make sense here:
- Using `.help()` for a subcommand overrides the whole help text. We meant to
use `.about()` here. I fixed this in some copy-pasted code as well.
- Printing debug output alongside real output makes it harder to select the
real output. I fixed this by logging debug output to stderr instead.
Reviewed By: StanislavGlebik
Differential Revision: D23471560
fbshipit-source-id: 7900cfe65613c48abd77faad6d6a45a7aa523b36
Summary:
This adds a subcommand for dumping all the paths in a repository. This is
helpful when you have a Content ID, limited imagination and time on your hands,
and you'd like to turn those into a file path where that Content ID lives.
This uses fsnodes for the traversal because that's O(# directories) as opposed
top O(# files). I had an earlier implementation that used unodes, but that was
really slow.
Reviewed By: StanislavGlebik
Differential Revision: D23471561
fbshipit-source-id: 948bfd20939adf4de0fb1e4b2852ad4d12182f16
Summary:
add backsyncing to rewrite file paths:
After setting the variables for large repo (D23294833 (d6895d837d)), we try to import the git commits into large repo and rewrite the file paths.
Following this, repo import tool should back-sync the commits into small_repo.
next step: derive all the data types for both small and large repos. Currently, we only derive it for the large repo.
==============
remove backup file:
The backup file was a last-minute addition when trying to import a repo for the first time.
Removed it, because we shouldn't write to external files. Future plan is to include
better process recoverability across the whole tool and not just rewrite file paths functionality.
Reviewed By: StanislavGlebik
Differential Revision: D23452571
fbshipit-source-id: bda39694fa34788218be795319dbbfd014ba85ff
Summary: More hooks will come in next diffs.
Reviewed By: aslpavel
Differential Revision: D23449755
fbshipit-source-id: 451fdb7a759140f2d6df8f3a18493c700fa2b761
Summary:
That's one of the sev followups. Before redacting a file content let's check if
it exists in "main-bookmark" (which is be default master), and refuse to redact
if it actually exists.
If this check passes (i.e. the content we are about to redact is not reachable
from master) that doesn't mean that we are 100% safe. E.g. this comment can be
in ancestor of master, or in any other repo or it can be added in the next
commit.
This check is a best-effort check to prevent shooting ourselves in the foot.
Reviewed By: aslpavel
Differential Revision: D23476278
fbshipit-source-id: 5a4cd10964a65b8503ba9a6391f17319f0ce37d8
Summary:
The loop took care to advance `b` to match the amount
of data that it had processed, but was still passing `buf`
(the unadjusted start of the buffer) to the syscalls.
This meant that in situations where a `readFull` might
encounter a partial read, it would scribble over the start
of the buffer and leave junk at the end.
For example:
write("hell");
write("o");
could produce "oell?" in the buffer when `readFull` consumes
the other end of the pipe.
Reviewed By: xavierd
Differential Revision: D23486270
fbshipit-source-id: 0848f6789b44421b609b91fe08890768ff59f7f5
Summary:
Currently we use a single path prefix to configure data fetch logging in eden
(i.e if the path of a file which we fetch is an extension of our configured
path, then we log that data fetch. )
There is some interest in extending this to multiple path prefixes, so that we
can log separate parts repo.
Reviewed By: StanislavGlebik
Differential Revision: D22877942
fbshipit-source-id: f6eb3dcb4fa460b4acab09677e972caf9421ddff
Summary:
We use Re2 in D22877942 for parsing multiple path prefix data fetch logging,
this introduces the dependency for eden's opensource builds.
Reviewed By: chadaustin
Differential Revision: D23431175
fbshipit-source-id: 44b399e92cb89ba1403295ecd10bc8f8d769b02c
Summary: This code can be used on Mac as well, so I can just move it to `UnixProcUtils` to be shared. I think to start it we can just try using this before trying to add special idleness detection with looking for active screensavers etc.
Reviewed By: fanzeyi
Differential Revision: D23183163
fbshipit-source-id: fffad8314e70f8726836c482f7a5e30e57a75c0d
Summary: We don't need to restart users if their running version is the same as their installed version, so we should check that when deciding if we should restart. This will give us more freedom in restarts since we won't have to play with `min_uptime`. I will add a flag to skip this check in case for some reason we need to do so on the fly.
Reviewed By: wez
Differential Revision: D23438306
fbshipit-source-id: b17c0e13789071b8b7c1b15ac5a8deb74a4fd091
Summary: I want to be able to reverse engineer an EdenInstance in the `edenfs_restarter` given the cmdline of the process. I think this best lives in the `config.py` file.
Reviewed By: fanzeyi
Differential Revision: D23438318
fbshipit-source-id: b3d9ac3981d3fb2bb8045b07b8d949cd601f6898
