Summary: We will use it in a separate subcmd later.
Reviewed By: StanislavGlebik
Differential Revision: D25944075
fbshipit-source-id: 211063ef74b192c18b5d006e9e0592085fdfc811
Summary: This enum no longer serves a purpose, delete it
Reviewed By: ikostia
Differential Revision: D25927254
fbshipit-source-id: 6731d6a57313946ee70b301ebc4f5cd634a90e61
Summary:
I've been seeing this hundreds of times / day in my Eden logs:
```
2021-01-07 09:19:19,702 Fetching failed, continuing as this may be spurious: Expected type that converts to PySequence but received set
```
Unfortunately, those logs are really not very helpful, because that error
doesn't tell you anything about where it occurred. I do however care about them
because they have correlated with slowness in Eden for me :)
At the same time, this error is a bit important: The last time we had an issue
here, it turned into a SEV. That was because the fallback was buggy, but it
does suggest that we do not expect to be routinely falling back.
So, let's log more.
Reviewed By: xavierd
Differential Revision: D25924077
fbshipit-source-id: f59d81cc51786f94f747142ec4baaca78f3072c2
Summary:
To ensure that a specific content gets sharded to multiple tasks (by tasks_per_content config parameter) each content can have multiple routing keys. These are the SHA256 (as before) or the SHA256 suffixed with "-1", "-2" etc..
When the routing key is generated for an item, one of them is picked randomly.
Added config constraint to ensure that the number of tasks parameter is at least 1.
Reviewed By: krallin
Differential Revision: D25886402
fbshipit-source-id: fb8911dad07d2f0b6bbf57b4ede084428fe6c49d
Summary:
Include changes since the last progress update (if any) in the final delta and total time logged.
When not chunking this was a minor inaccuracy, but when chunking in small chunks it would mean a lot of stats for each chunk were missing as if a chunk took less than 5 seconds there might not even have been a progress update before this end state.
Reviewed By: StanislavGlebik
Differential Revision: D25852274
fbshipit-source-id: 6cea76e3abd37908475052947794eed442a1ac82
Summary:
When the root type is e.g. Changeset allow edges from ChangesetToX
This is important for chunking, doing it independently to check it doesn't break non-chunked behaviour.
Reviewed By: krallin
Differential Revision: D25742294
fbshipit-source-id: ea4c989e9f61b30094d0fd83e543fe14a38254fd
Summary:
Calling the count Final doesn't make sense when chunking as it can appear more than once per tail, so update it to something more appropriate.
Seen is the count of Nodes seen in the stream. Loaded is the count of nodes where NodeData was loaded. When chunking some deferred Nodes will be seen but not Loaded.
Reviewed By: krallin
Differential Revision: D25742283
fbshipit-source-id: 1f10007d94ad2dbd750bfa53bab3e46a2caad7fa
Summary: This logging was being globbed away in tests. Disable it to make tests easier to read
Reviewed By: krallin
Differential Revision: D25742287
fbshipit-source-id: dfba05a688c9b1a56d6ab8746df2df3ea885c239
Summary: Add ability to turn on or off various types of logging, allows removing boilerplate in test cases.
Reviewed By: krallin
Differential Revision: D25742284
fbshipit-source-id: bbbe7f477156fc49ff6779f9a09e1b397ff6f618
Summary: This logging was always being globbed away, so remove it
Reviewed By: krallin
Differential Revision: D25742292
fbshipit-source-id: 75e004d3fdadc617f479beee44999692c267d2a9
Summary:
Add arguments to cmdlib so we can filter log messages by the slog tag, using new Drains added in slog_ext.
To use tagging from slog the form is:
```
const FOO_TAG: &str = "foo";
info!(logger, #FOO_TAG, "hello foo!");
```
Reviewed By: krallin
Differential Revision: D25837627
fbshipit-source-id: b164d508a2e82a80c4ff6f5f35c0c722257b9a2a
Summary:
This took a lot of cmake fuckery. I am guessing it will break the current travis-ci.org based build, since it no longer downloads gflags.
As I put this diff up, mvfst (a dependency of quic (a dependency of proxygen)) is not building right now due to another diff - a fix is in progress but isn't landed yet. I applied the fix locally for the test plan, but didn't include it in the diff.
Reviewed By: snarkmaster
Differential Revision: D25651851
fbshipit-source-id: b4f15ac7fa5293123710fe367c20d355c0970c42
Summary:
override_ctx() sets different SessionClass while data is derived. This should
reduce the number of entries we write to blobstore sync queue. See previous
diff for more motivation.
This diff uses override_ctx() for batch_derive() method
Reviewed By: krallin
Differential Revision: D25910465
fbshipit-source-id: b8a3e729c059cad5716b1b09bd2f1cc618273627
Summary:
Previously we weren't initializing tunables at all in derived_data_tailer and
probably in lots of other binaries as well.
This is unfortunate, since derived_data_tailer runs for a long time and it
would be good to be able to control its behaviour.
Let's fix it by using init_mononoke() function.
Reviewed By: ikostia
Differential Revision: D25912156
fbshipit-source-id: eabd1c56120d087a169746077c8a7d36855c2b84
Summary:
We had issues with mononoke writing too much blobstore sync queue entries while
deriving data for large commits. We've contemplated a few solutions, and
decided to give this one a go.
This approach forces derive data to use Background SessionClass which has an
effect of not writing data to blobstore sync queue if the write to blobstore
was successful (it still writes data to the queue otherwise). This should
reduce the number of entries we write to the blobstore sync queue
significantly. The downside is that writes might get a bit slower - our
assumption is that this slowdown is acceptable. If that's not the case we can
always disable this option.
This diff overrides SessionClass for normal ::derive() method. However there's
also batch_derive() - this one will be addressed in the next diff.
One thing to note - we still write derived data mapping to blobstore sync queue. That should be find as we have a constant number of writes per commits.
Reviewed By: krallin
Differential Revision: D25910464
fbshipit-source-id: 4113d00bc0efe560fd14a5d4319b743d0a100dfa
Summary:
We ran into issues when we write too much data to our blobstore sync queue. The
idea we want to try is to prefer to not write to the queue if puts were
successful.
In the next diff we'll start overriding session class when we derive data.
This diff makes it possible to do so.
Reviewed By: krallin
Differential Revision: D25910463
fbshipit-source-id: 2ead5291acbce4014fbb833cac1dc53a6eb61b13
Summary:
This might be controversial, so I'd like to hear opinions.
We are running low on retention once again, and there's client_ip and
source_hostname field that have largely duplicated info and together use ~14%
of space.
I suggest to log source_hostname if it exists and client_ip if source_hostname
is not set. This should save ~7% of space since most of the time both of these
fields are set.
Reviewed By: krallin
Differential Revision: D25900993
fbshipit-source-id: f3db59e0dde44a5117059c829df7a1a1c811f641
Summary: Allow us to return arg parsing errors rather than panicing
Reviewed By: krallin
Differential Revision: D25837626
fbshipit-source-id: 87e39de140b1dcd3b13a529602fdafc31233175d
Summary: ChangesetEntry query is a more constrained query where we know the ids, can issue query in chunks separately
Differential Revision: D25804024
fbshipit-source-id: 6627fa17ee155182285997cb0642c7d7f033da26
Summary:
Allow use of read replica when fetching bulk ids. Bulkops clients not needing most up to date bounds can use this mode providing they are not checkpointing the repo min/max ids.
Existing default behaviour is unchanged.
Differential Revision: D25804028
fbshipit-source-id: ca14e929ea94c351e27eed2aa012fe914c8c691e
Summary: I was seeing mysql timeouts (where client doesn't consume query results within 10s) in walker runs using fetch_id(). Spawning the query means the results are collected and ready when we do end up polling.
Differential Revision: D25804025
fbshipit-source-id: 443dd87028fe68de16c174deb7b017d7ce5439d1
Summary: Simplify the code by allowing us to remove the old windowing logic.
Differential Revision: D25804022
fbshipit-source-id: 1f2837c2f83adcb3afdb453a9220ac68509a36ec
Summary:
When querying for changesets in a repo, often there are very few changesets for a repo inside a 65536 wide range of ids, which means multiple round trips to the database.
This change adds a LIMIT based mysql query that can always return up to the specified limit rows if the repo has them, and then using bounded_traversal_stream to unfold from the highest id loaded from that query to the next chunk to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D25804023
fbshipit-source-id: 46df2ea48d01bc4143d96642e45066f520faa4d6
Summary: **expected_sha1** was declared but not used in **xattr_test.py** file, removing this unused variable and variables which was used to creat **expected_sha1**
Reviewed By: xavierd
Differential Revision: D25899401
fbshipit-source-id: 26f0bb06d2c96e7f6754a4b821ffe4cf59a2f35e
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary: Small improvement, can pass by reference
Reviewed By: krallin
Differential Revision: D25898714
fbshipit-source-id: f1012b7d947e1ead00cd9c032fea3f3aa04a3072
Summary:
We seem to get cpu spikes. The theory is that it happens because of a commit with
low generation number lands which triggers a slow path in getbundle code. Note that I've landed two
optimizations (D23824204 (609c2ac257) and D23599866 (54d43b7f95)) which *should* help, however at the
moment threshold for what to consider a log generation number is too low so the
optimization doesn't kick in.
I'd like to verify this theory, hence adding this logging.
Reviewed By: ahornby
Differential Revision: D25884345
fbshipit-source-id: 9686933726ff0a3ae11b541b3738eb08d011abe0
Summary:
In this diff I've replaced non-transparent error definition error("{0}") with error(transparent).
The reason is non-transparent errors print the same thing as the original errors:
```
Error: failed to complete task
Caused by: other error description <-- this duplicate shouldn't be here
Caused by: other error description
```
Reviewed By: krallin
Differential Revision: D25899411
fbshipit-source-id: e586af86b635a7e2fbf8952297171c546b859300
Summary:
Use the new API to clean up stale logs at open time. This hopefully helps
releasing disk space on Windows if the normal rotation fails to remove
old files being mmap-ed by other processes.
Reviewed By: xavierd
Differential Revision: D25894282
fbshipit-source-id: a3d8247b737dd451ee68b58cc5a38fdd2822c0c3
Summary:
Previously rotation only happens at flush time and file deletion is a best
effort (it might fail on Windows). For use-cases that are sensitive about
space usage that's suboptimal. This diff adds an API to remove old files
manually so high level logic can choose to clean up old files after open().
Reviewed By: xavierd
Differential Revision: D25894283
fbshipit-source-id: fbffff426544b39349ddf3537d46954d3cab5d12
Summary:
Previously, the fsmonitor state update logic will skip updating treestate if the wlock
cannot be obtained. D17468790 (8d4d0a66a2) made it wait for wlock for the painful "watchman fresh
instance" case. But things can still suck if it's not a "fresh instance" but there are just
too many nonnormal files.
This diff makes it that exceeding a threshold of nonnormal files will trigger a fsmonitor
state write as an attempt to to reduce the number of nonnormal files. In additional,
`--debug` was changed to print more internal states for debugging.
This would hopefully address issues where people have a large "nonnormal"
treestate, suffers from the bad performance issue and cannot recover from it
automatically.
Reviewed By: DurhamG
Differential Revision: D25794083
fbshipit-source-id: 741426cf31484d9318f9cfcab11d38da33ab5067
