Summary: This allow to render metrics in the real time, similar to how we render network usage
Reviewed By: DurhamG
Differential Revision: D30137295
fbshipit-source-id: 8843b574971c1ccaaae0727be6f9d46d1d90625f
Summary: See docstring to init_from_env for the description
Reviewed By: quark-zju
Differential Revision: D30137304
fbshipit-source-id: ad3fb5e0cbb52ca405959eb0d394f5a2c9761ad9
Summary: This allows to count number of parallel operations, like parallel entrance to some fn or parallel requests
Reviewed By: quark-zju
Differential Revision: D30137297
fbshipit-source-id: 74585dcab353091619a0a1f217548121079b463c
Summary:
This moves some Prjfs logic into the channel code, which allows for
de-duplicating a bit of code. This will also make a subsequent change in the
rename code easier to do.
Differential Revision: D30023970
fbshipit-source-id: 7efa6dcc4318213e9d266932527b5a56edacefd7
Summary: Allow this to be used by more utilities.
Reviewed By: DurhamG
Differential Revision: D29958512
fbshipit-source-id: b6a1a7017102a4ff4ad252246d3252903bbb793f
Summary: Prepare before moving to a common directory
Reviewed By: StanislavGlebik
Differential Revision: D29958513
fbshipit-source-id: 48c04a9bf6e8c3800b3f187e7741d240ecff9e9c
Summary: This allows us to remove older files to free disk space.
Reviewed By: DurhamG
Differential Revision: D30092445
fbshipit-source-id: 20770968cfaeba4fb6f6c45b247cc1cdafb79a08
Summary: This makes upcoming changes a bit easier.
Reviewed By: yancouto
Differential Revision: D30092446
fbshipit-source-id: f62f1ced45c46dc48c3b1112af661d8fbcd80706
Summary: This was a "builder" class, but it was highly unnecessary. It had two fields, but only one was ever used, and it made things simpler to just replace it with "constructor" methods.
Differential Revision: D30162340
fbshipit-source-id: ed75d9979794c00ca18aa95fdb01688831ec4b5a
Summary: It makes it clearer whether the push was successful or not.
Reviewed By: krallin
Differential Revision: D30217873
fbshipit-source-id: 6b7c3af5d794ce53504e5f92fd4d5cd6e763acc0
Summary: This removes some redundant lookups when using the `*_batch` APIs.
Reviewed By: andll
Differential Revision: D30094155
fbshipit-source-id: f64702a09f67b67a10a2d51545f3b6940e80022f
Summary: The test exposes issues that the local cache isn't used in some cases.
Reviewed By: andll
Differential Revision: D30094158
fbshipit-source-id: 0122154b8bd2f9d321e248720632e28b6ca0559e
Summary:
Similar to D29888937 (8cb3d3dd75). This helps getting the Rust traceback of why certain
code path gets triggered.
Reviewed By: andll
Differential Revision: D30094157
fbshipit-source-id: ef5bacbf4612ed115fd631056f6c004fe6d1d93f
Summary: This will be used to debug some over fetching issues.
Reviewed By: andll
Differential Revision: D30094159
fbshipit-source-id: cfbdff497fbf53f97f1c957bf77c25e26ee79286
Summary:
Prefetch vertex on checkout (which is more expected to require network to download
files) so committing on them does not trigger remote lookup.
Reviewed By: andll
Differential Revision: D30094156
fbshipit-source-id: 917df2bbad7cee51a06f2fa38172775423893c37
Summary:
See the added comments. In some cases we can avoid remote lookups. This would
help make commit/amend more offline friendly.
Reviewed By: DurhamG
Differential Revision: D30004908
fbshipit-source-id: 94fbc7934a1eb3ae1058d8c542211a885d5ad8e6
Summary:
Similar to D30094157. This is helpful to get a Rust backtrace on "not found"
errors. This is useful because the Python traceback might not have Rust
stack.
Ideally, it'd be better if we can integrate backtrace into the error types.
However, the stdlib backtrace is still unstable and the thirdparty backtrace
seems too heavyweight.
Reviewed By: DurhamG
Differential Revision: D30123355
fbshipit-source-id: 7af76422143340194d8499a75e8ed65e9bf1fcae
Summary:
We have seen slicing panics on NFS, like:
thread 'main' panicked at 'range end index 14079379 out of range for slice of length 0', eden/scm/lib/indexedlog/src/log/mod.rs:1678:14
Possibly when some files were stale or out of sync (ex. `log` got shorter than
expected somehow) due to NFS weirdness (bad locking?). Let's make them Rust
errors instead of panics for easier error reporting and investigation.
Reviewed By: DurhamG
Differential Revision: D30166177
fbshipit-source-id: 25cbd9c178b24580c176fe9469362fe0020ede95
Summary:
commitcloud CLEAN UP: remove last bits code related to an old interngraph commit cloud proxy
The configs are not used, the service_url in scm daemon is not used, the interngraph proxy is not used
Reviewed By: singhsrb
Differential Revision: D30160471
fbshipit-source-id: 07fee2b8733afa29659467cb296df3b47fe1b54b
Summary:
I'd like to replace our current python streaming changelog builder with new
rust streaming changelog builder. One thing that's still missing is monitoring
and alarms. This diff adds a basic support that - let's log to scuba every time
we update streaming changelog. Later we can add a detector that would alarm if scuba had no updates in a while
Reviewed By: Croohand
Differential Revision: D30157560
fbshipit-source-id: 9740c8462ca2edf18adfe1b65b271fa0a8618cb4
Summary: Due to lifetime issues of FetchContext& w.r.t. background prefetching, we can just create the helper in `_globFiles` and use that to maintain lifetime
Reviewed By: xavierd
Differential Revision: D30175224
fbshipit-source-id: b2fccb76f9d4011139e80bd5bc52c40bbab08b94
Summary: Added a Thrift method that tells EdenFS to prefetch files from a user's most used directories using an endpoint that talks to the edenfs/edenfs_service SmartPlatform service to get the directory list. The default number of directories is set to 10,000.
Reviewed By: genevievehelsel
Differential Revision: D29909976
fbshipit-source-id: bfb1a411d50d7355ff604de5bc090a9e2c3100a0
Summary: These were accidentally left over from a previous diff.
Reviewed By: singhsrb
Differential Revision: D30133799
fbshipit-source-id: 311f746848fc823a6fbae9e58bafda6641104b1e
Summary:
file upload: pass content size as a parameter
We shouldn't rely on body size, because it will not allow us to build compression and also it won't allow us to implement streaming.
Reviewed By: yancouto
Differential Revision: D30100887
fbshipit-source-id: c16f79fa71fe320f61d15e1328b67026f586a1dc
Summary:
The changes in this diff makes the code to "run" and skip correctly according to our rules.
All integration tests are now runnable on Windows with mode/win
Differential Revision: D30143255
fbshipit-source-id: b2ddbff7268f182274b3755f4b28df6ac6cdeef4
Summary: This was an oversight. With this edenfsctl should be able to run on Windows and talk with EdenFS daemon via UNIX domain socket.
Differential Revision: D30089140
fbshipit-source-id: abe12aeee06d91f7a7323a6042d2b69c31963bbb
Summary:
On Windows, EPIPE's raw os error might be 232, different from libc::EPIPE (32).
That means the Python world cannot check the errno properly to hide the EPIPE
error message. Fix it by reading io::ErrorKind first, and fallback to
raw_os_error.
Reviewed By: yancouto
Differential Revision: D30118006
fbshipit-source-id: b34a2142b24714e68d11a03a0771a5f9c037ed0f
Summary:
The "local log" algorithm in the fastlog extension is poorly implemented.
It can be much slower than the non-fastlog algorithm. This means, if the
"remote" fastlog thread does not work in production, then enabling fastlog
can actually seriously regress log performance.
The fastlog test (already) cannot meaningfully test the remote behavior. So it
is removed for now.
Differential Revision: D30145651
fbshipit-source-id: 601a38e03920fbcae86e23262949ae3e3a16b9af
Summary:
I'm changing the semantics of `metadata.thrift` here slightly.
---
Before:
- `ThriftServiceContext` contains `ThriftService` inline.
- `ThriftMetadata` only contains other services *referred* to by the primary service (in `ThriftServiceContext`). This includes all base classes **but not the service itself**.
- Understanding the service class hierarchy requires traversing through each `parent` field and looking up the name in `ThriftMetadata`.
---
After:
- `ThriftServiceContextRef` contains just the service name.
- `ThriftMetadata` now includes the service itself (previously inlined in `ThriftServiceContext`).
- `services` field lists all names of services in the class hierarchy in order from most to least derived. **These semantics are needed to support `MultiplexAsyncProcessorFactory` where the concept of a single `parent` falls apart**.
---
After migrating all clients, we can remove `ThriftServiceContext` completely. It's now deprecated.
For `py3`, I've removed `extractMetadataFromServiceContext` because it's no longer needed. All it was doing was adding the inline `ThriftServiceContext` into the metadata's map... which we do by default now.
Reviewed By: yfeldblum
Differential Revision: D29952004
fbshipit-source-id: 13c62aafabbfc287ad64489c02104dd977be71ce
Summary:
Original commit changeset: 34fe02ddf580
We are seeing reports of EdenFS only showing partial directories to our user. Local testing shows this commit seems to be causing the issue. Reverting it for now.
Reviewed By: kmancini
Differential Revision: D30136949
fbshipit-source-id: 7fcc81506c132055a5b639a383b8c9be68118dc5
Summary:
Practically only checking the "root" vertexes is also "correct". Let's do
that to reduce some remote lookups.
Reviewed By: andll
Differential Revision: D30004909
fbshipit-source-id: e46e8ee6b072270b860c5275b13f1e2103ef2b6a
Summary:
The test shows the offline commit problem that will be addressed by upcoming
changes.
Reviewed By: andll
Differential Revision: D30094154
fbshipit-source-id: c5f38ff6f0d33d241501050a1f6257633b90b31e
Summary:
IPython assumes time.clock exists but it is missing on Windows. Polyfill it
on Windows.
Reviewed By: yancouto
Differential Revision: D30106738
fbshipit-source-id: fd49417917b0b17e81eba599a474fc96f5624b42
Summary:
With lazy changelog (and lazy tree, files, etc), scanning the local repo is
painfully slow. Moreover, the remote request is fast and fastlog does not
seem to know to end the local thread immediately if the remote request has
ended.
Let's just disable the local scan by default to avoid excessive remote
lookups.
Reviewed By: singhsrb
Differential Revision: D30137868
fbshipit-source-id: 23eeb2a88d83ebb2bbea4993408da181d2e8003d
Summary:
It turns out that D30052693 (fd10938296) was not enough. The FailScenario cannot be setup
from different threads too. Doing that would cause deadlock. So let's avoid
that too.
Reviewed By: DurhamG
Differential Revision: D30124930
fbshipit-source-id: 0095bcf5ad3a99831d9b9c75a1e9f2c50729819b
Summary:
Now we validate the bonsai changeset uploaded via edenapi, by using the `RepoWriteContext.create_changeset` function, instead of directly creating the changeset using `BonsaiChangesetMut`.
I left a comment with a possible future improvement, where we can use upload tokens on `create_changeset` to avoid querying the blobstore for file size.
Differential Revision: D30045939
fbshipit-source-id: 84bb383879f8a25464044487eb99bd38b2849537
Summary:
This simplifies both client and server code to make bonsai changeset uploading be simpler for snapshots, as we only need a single commit, no mutations, etc.
This will make it easier to validate the bonsai changeset on the next diff.
It is fine to change both client and server as this code is not still in production, so we don't need to worry about rollout order.
Reviewed By: StanislavGlebik
Differential Revision: D30044542
fbshipit-source-id: d14bf58d671bc3bb5ff54b07c21f1781a043e0cf
Summary:
Using bonsai upload on commitcloud was useful to get things started, but it's not really necessary, and by removing we can simplify things a bit, since for snapshots it's a single commit.
This diff removes the client code. The server code is still used by snapshots, but will be simplified in a following diff.
Reviewed By: liubov-dmitrieva
Differential Revision: D30018157
fbshipit-source-id: 98bf0d30959de0c63b01bd11442f0bb3c538fcc7
Summary:
This diff addresses [this comment](https://www.internalfb.com/diff/D29849964 (4bde7b7488)?dst_version_fbid=244353817531864&transaction_fbid=342353780770798).
- It removes the bit of code in `process_files_upload` that adds file size to the metadata.
- In order for this not to break the bonsai upload, I made it so the lookup call returns upload tokens with file size when looking up a file.
- Took the opportunity to do some refactoring
- Consolidated duplicated functions in `convert_file_to_content_id`, and added some helpful From implementations to make calling it more ergonomic.
- `convert_file_to_content_id` now doesn't fail when the file doesn't exist, instead returns option (also fixed the callsite)
Reviewed By: liubov-dmitrieva
Differential Revision: D30016963
fbshipit-source-id: aae8a085d7a207e50679bb1210277a9e21a32de8
Summary:
These fields had placeholder values and are now populated properly. I found out about `workingctx` which has most information I need about uncommitted changes that will create the snapshot.
TODO on next diffs:
- Remove code for uploading commit cloud stuff via bonsai, and simplify API.
- Verify created bonsai is valid before persisting it
- Store files and bonsai blobs in the created ephemeral bubble
- Modify bonsai changeset objects so they can contain snapshot information
Reviewed By: liubov-dmitrieva
Differential Revision: D30013238
fbshipit-source-id: c509d25c59d6396d922435cc3a97fc837d3fe51b
Summary: Using changes from D29995429, this returns the upload token of the changeset upload in the uploadsnapshot response.
Reviewed By: StanislavGlebik
Differential Revision: D30012368
fbshipit-source-id: 5ca54763153a474d1ce3c38ddeaa0efff071b09c
Summary:
Using the new macros from previous diffs, this creates a new `ChangesetId` edenapi type and adds it to AnyId, which allows it to be used from UploadToken.
It then adds the lookup method for it, and returns it from upload_bonsai_changeset call (instead of a fake HgId UploadToken).
This will be used so that the client can know the changeset id of the uploaded snapshot.
Reviewed By: StanislavGlebik
Differential Revision: D29995429
fbshipit-source-id: e2ee4b9b0ac21d6f5394afacbfed1802da64013b
