Summary:
This might be controversial, so I'd like to hear opinions.
We are running low on retention once again, and there's client_ip and
source_hostname field that have largely duplicated info and together use ~14%
of space.
I suggest to log source_hostname if it exists and client_ip if source_hostname
is not set. This should save ~7% of space since most of the time both of these
fields are set.
Reviewed By: krallin
Differential Revision: D25900993
fbshipit-source-id: f3db59e0dde44a5117059c829df7a1a1c811f641
Summary: Allow us to return arg parsing errors rather than panicing
Reviewed By: krallin
Differential Revision: D25837626
fbshipit-source-id: 87e39de140b1dcd3b13a529602fdafc31233175d
Summary: ChangesetEntry query is a more constrained query where we know the ids, can issue query in chunks separately
Differential Revision: D25804024
fbshipit-source-id: 6627fa17ee155182285997cb0642c7d7f033da26
Summary:
Allow use of read replica when fetching bulk ids. Bulkops clients not needing most up to date bounds can use this mode providing they are not checkpointing the repo min/max ids.
Existing default behaviour is unchanged.
Differential Revision: D25804028
fbshipit-source-id: ca14e929ea94c351e27eed2aa012fe914c8c691e
Summary: I was seeing mysql timeouts (where client doesn't consume query results within 10s) in walker runs using fetch_id(). Spawning the query means the results are collected and ready when we do end up polling.
Differential Revision: D25804025
fbshipit-source-id: 443dd87028fe68de16c174deb7b017d7ce5439d1
Summary: Simplify the code by allowing us to remove the old windowing logic.
Differential Revision: D25804022
fbshipit-source-id: 1f2837c2f83adcb3afdb453a9220ac68509a36ec
Summary:
When querying for changesets in a repo, often there are very few changesets for a repo inside a 65536 wide range of ids, which means multiple round trips to the database.
This change adds a LIMIT based mysql query that can always return up to the specified limit rows if the repo has them, and then using bounded_traversal_stream to unfold from the highest id loaded from that query to the next chunk to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D25804023
fbshipit-source-id: 46df2ea48d01bc4143d96642e45066f520faa4d6
Summary: **expected_sha1** was declared but not used in **xattr_test.py** file, removing this unused variable and variables which was used to creat **expected_sha1**
Reviewed By: xavierd
Differential Revision: D25899401
fbshipit-source-id: 26f0bb06d2c96e7f6754a4b821ffe4cf59a2f35e
Summary:
`extra_cmake_defines` are extra flags that are passed to cmake when
compiling each one of the dependencies. For instance:
```
$ ./opensource/fbcode_builder/getdeps.py build f4d --extra-cmake-define='{"CMAKE_CXX_FLAGS": "-mavx2 -mfma -mavx -mf16c -march=native"}'
```
It takes a json map as input, which can take a list of defines (key value
pairs).
Reviewed By: wez
Differential Revision: D25855781
fbshipit-source-id: 7f4fef2c66f4d12f23c8d7086d6a4f24fcc01ff7
Summary: Small improvement, can pass by reference
Reviewed By: krallin
Differential Revision: D25898714
fbshipit-source-id: f1012b7d947e1ead00cd9c032fea3f3aa04a3072
Summary:
We seem to get cpu spikes. The theory is that it happens because of a commit with
low generation number lands which triggers a slow path in getbundle code. Note that I've landed two
optimizations (D23824204 (609c2ac257) and D23599866 (54d43b7f95)) which *should* help, however at the
moment threshold for what to consider a log generation number is too low so the
optimization doesn't kick in.
I'd like to verify this theory, hence adding this logging.
Reviewed By: ahornby
Differential Revision: D25884345
fbshipit-source-id: 9686933726ff0a3ae11b541b3738eb08d011abe0
Summary:
In this diff I've replaced non-transparent error definition error("{0}") with error(transparent).
The reason is non-transparent errors print the same thing as the original errors:
```
Error: failed to complete task
Caused by: other error description <-- this duplicate shouldn't be here
Caused by: other error description
```
Reviewed By: krallin
Differential Revision: D25899411
fbshipit-source-id: e586af86b635a7e2fbf8952297171c546b859300
Summary:
Use the new API to clean up stale logs at open time. This hopefully helps
releasing disk space on Windows if the normal rotation fails to remove
old files being mmap-ed by other processes.
Reviewed By: xavierd
Differential Revision: D25894282
fbshipit-source-id: a3d8247b737dd451ee68b58cc5a38fdd2822c0c3
Summary:
Previously rotation only happens at flush time and file deletion is a best
effort (it might fail on Windows). For use-cases that are sensitive about
space usage that's suboptimal. This diff adds an API to remove old files
manually so high level logic can choose to clean up old files after open().
Reviewed By: xavierd
Differential Revision: D25894283
fbshipit-source-id: fbffff426544b39349ddf3537d46954d3cab5d12
Summary:
Previously, the fsmonitor state update logic will skip updating treestate if the wlock
cannot be obtained. D17468790 (8d4d0a66a2) made it wait for wlock for the painful "watchman fresh
instance" case. But things can still suck if it's not a "fresh instance" but there are just
too many nonnormal files.
This diff makes it that exceeding a threshold of nonnormal files will trigger a fsmonitor
state write as an attempt to to reduce the number of nonnormal files. In additional,
`--debug` was changed to print more internal states for debugging.
This would hopefully address issues where people have a large "nonnormal"
treestate, suffers from the bad performance issue and cannot recover from it
automatically.
Reviewed By: DurhamG
Differential Revision: D25794083
fbshipit-source-id: 741426cf31484d9318f9cfcab11d38da33ab5067
Summary:
We are going to write docs on how to unbind repositories and bind them later.
This integration test is a "test for documentation" i.e. it should run the same
steps as we'd run ourselves whenever we need to rebind.
The key part here is that we do a merge of unbound small repo into large repo commits
Reviewed By: ikostia
Differential Revision: D25883873
fbshipit-source-id: fac1d7871e52f4e1aa7f15d32d39f2755b803cd3
Summary:
Add a command that will unconditionally rewrite commits from one repo to
another. This is similar to what x-repo lookup thrift api does, however this
command doesn't do a few safety checks (in particular, it allows to rewrite
public commits).
This is going to be used for unbinding procedure.
Reviewed By: ikostia
Differential Revision: D25883874
fbshipit-source-id: a40eda9aa40ef7ad63e2403d956871940dd1601d
Summary: We don't use obsstore. Its size is irrelevant.
Reviewed By: singhsrb
Differential Revision: D25876066
fbshipit-source-id: 5657c3ca08f5ed1cb5a3d1a5b3395ab74756b7e5
Summary: It's used for malloc / realloc which take size_t instead of unsigned int.
Reviewed By: StanislavGlebik, xavierd
Differential Revision: D25890562
fbshipit-source-id: e2787786e9b995431c50b411d77cbce438a82c98
Summary: This is a bit nicer to read
Reviewed By: ahornby
Differential Revision: D25881919
fbshipit-source-id: 3c97390a96410a18e8fdc6cb6279b2d46e407cd3
Summary:
I think it's nice to group them together. Also while there I sorted both groups
lexicographically
Reviewed By: ahornby
Differential Revision: D25881918
fbshipit-source-id: 05bb9f07ee7799c3d5c19a8ffaabadaca635fef2
Summary: Certain doctor items are not meaningful outside Facebook. This diff adds the ability to have Facebook-specific doctors.
Reviewed By: xavierd
Differential Revision: D25789275
fbshipit-source-id: 94160be741a8fc3e8d01e86beaa0d2428952db21
Summary:
Before this diff, we did DNS lookups using a crate called `dns_lookup`. This crate is a thin layer over libc DNS lookups. Those lookups are blocking (i.e. they hold a thread), so they're not very friendly to asynchronous code. We currently offload them on a dedicated thread pool to mitigate the issue, but this isn't ideal: if we experience e.g. slow DNS responses, we could saturate this thread pool pretty easily.
I updated it to use the trust-dns crate, which provides an asynchronous implementation of DNS lookups, and is currently used in other parts of Mononoke.
Reviewed By: krallin
Differential Revision: D25849872
fbshipit-source-id: 826ab4e5618844f3b48e5def4ad9bd163753ebb1
Summary:
D25850262 (772680f2f7) updated a dep but only did a partial regen. This broke
other diffs.
This diff does a full regen to sync up again.
Differential Revision: D25872444
fbshipit-source-id: b5749b3d634ffe355a5a3cf6e94066ca5158001f
