2019-11-02 03:38:10 +03:00
|
|
|
/*
|
|
|
|
* Copyright (c) Facebook, Inc. and its affiliates.
|
|
|
|
*
|
|
|
|
* This software may be used and distributed according to the terms of the
|
|
|
|
* GNU General Public License version 2.
|
|
|
|
*/
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
|
2019-11-22 19:51:58 +03:00
|
|
|
use anyhow::Result;
|
2019-12-18 06:25:48 +03:00
|
|
|
use dag::{idmap::IdMap, segment::Dag, Group, Id};
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
use minibench::{
|
|
|
|
bench, elapsed,
|
|
|
|
measure::{self, Measure},
|
|
|
|
};
|
|
|
|
use tempfile::tempdir;
|
|
|
|
|
2019-09-14 05:26:34 +03:00
|
|
|
mod bindag;
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
|
|
|
|
fn main() {
|
2019-09-14 05:26:34 +03:00
|
|
|
let parents = bindag::parse_bindag(bindag::MOZILLA);
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
|
|
|
|
let head_name = format!("{}", parents.len() - 1).as_bytes().to_vec();
|
2019-11-15 01:09:18 +03:00
|
|
|
let parents_by_name = |name: &[u8]| -> Result<Vec<Box<[u8]>>> {
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
let i = String::from_utf8(name.to_vec())
|
|
|
|
.unwrap()
|
|
|
|
.parse::<usize>()
|
|
|
|
.unwrap();
|
|
|
|
Ok(parents[i]
|
|
|
|
.iter()
|
|
|
|
.map(|p| format!("{}", p).as_bytes().to_vec().into())
|
|
|
|
.collect())
|
|
|
|
};
|
|
|
|
|
|
|
|
let id_map_dir = tempdir().unwrap();
|
|
|
|
let mut id_map = IdMap::open(id_map_dir.path()).unwrap();
|
2019-12-05 09:41:13 +03:00
|
|
|
id_map
|
2019-12-18 06:25:48 +03:00
|
|
|
.assign_head(&head_name, &parents_by_name, Group::MASTER)
|
2019-12-05 09:41:13 +03:00
|
|
|
.unwrap();
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
|
|
|
|
let head_id = id_map.find_id_by_slice(&head_name).unwrap().unwrap();
|
|
|
|
let parents_by_id = id_map.build_get_parents_by_id(&parents_by_name);
|
|
|
|
|
|
|
|
// Test the size, and generation speed, and ancestor calcuation speed
|
|
|
|
// with some different segment sizes.
|
|
|
|
for &segment_size in [4, 8, 10, 12, 14, 16, 18, 20, 22, 24, 32, 64, 128].iter() {
|
|
|
|
let dag_dir = tempdir().unwrap();
|
|
|
|
let mut built = false;
|
|
|
|
bench(format!("building segment_size={}", segment_size), || {
|
|
|
|
built = true;
|
|
|
|
measure::Both::<measure::WallClock, String>::measure(|| {
|
|
|
|
let mut dag = Dag::open(&dag_dir.path()).unwrap();
|
2019-09-17 22:34:26 +03:00
|
|
|
dag.set_new_segment_size(segment_size);
|
|
|
|
let mut syncable = dag.prepare_filesystem_sync().unwrap();
|
|
|
|
let segment_len = syncable
|
dag: refactor segment building APIs
Summary:
Previously, the `Dag` has 2 low-level `build_segemnts` APIs:
- Dag::build_flat_segments(..., last_threshold)
- Dag::build_high_level_segments(..., drop_last)
They allow customization about whether the segments are lagging or not.
However, certain algorithms (ex. children and range) now require the high level
segments to cover everything covered by the flat segments. The above APIs
wouldn't ensure that.
This diff refactors the segment building APIs so that:
- Make `build_flat_segments`, and `build_high_level_segments` private to
prevent misuse.
- Ensure high level segments cover flat segments at `Dag::open` and
`Dag::build_segments_volatile`, the only ways to change `Dag`.
- Provide different APIs suitable for different (one-time in-memory vs
on-disk) use-cases. The on-disk `build_segments_persistent` API makes high
level segments lagging to avoid fragmentation, while the in-memory
`build_segments_volatile` does not.
To satisfy the existing test need, a `set_segment_size` API was added to
override the default segment size.
Most callsites become simpler because they no longer need to figure out
details about segment size, level, and lagging.
Reviewed By: sfilipco
Differential Revision: D17000965
fbshipit-source-id: 78bb0c7674c99e91be6011bb7e623cd4f63b1521
2019-09-14 05:26:34 +03:00
|
|
|
.build_segments_persistent(head_id, &parents_by_id)
|
|
|
|
.unwrap();
|
2019-09-17 22:34:26 +03:00
|
|
|
syncable.sync(std::iter::once(&mut dag)).unwrap();
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
let log_len = dag_dir.path().join("log").metadata().unwrap().len();
|
dag: refactor segment building APIs
Summary:
Previously, the `Dag` has 2 low-level `build_segemnts` APIs:
- Dag::build_flat_segments(..., last_threshold)
- Dag::build_high_level_segments(..., drop_last)
They allow customization about whether the segments are lagging or not.
However, certain algorithms (ex. children and range) now require the high level
segments to cover everything covered by the flat segments. The above APIs
wouldn't ensure that.
This diff refactors the segment building APIs so that:
- Make `build_flat_segments`, and `build_high_level_segments` private to
prevent misuse.
- Ensure high level segments cover flat segments at `Dag::open` and
`Dag::build_segments_volatile`, the only ways to change `Dag`.
- Provide different APIs suitable for different (one-time in-memory vs
on-disk) use-cases. The on-disk `build_segments_persistent` API makes high
level segments lagging to avoid fragmentation, while the in-memory
`build_segments_volatile` does not.
To satisfy the existing test need, a `set_segment_size` API was added to
override the default segment size.
Most callsites become simpler because they no longer need to figure out
details about segment size, level, and lagging.
Reviewed By: sfilipco
Differential Revision: D17000965
fbshipit-source-id: 78bb0c7674c99e91be6011bb7e623cd4f63b1521
2019-09-14 05:26:34 +03:00
|
|
|
format!("segments: {} log len: {}", segment_len, log_len)
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
})
|
|
|
|
});
|
|
|
|
|
|
|
|
bench(
|
|
|
|
format!("ancestor calcuation segment_size={}", segment_size),
|
|
|
|
|| {
|
|
|
|
assert!(built, "segments must be built to run this benchmak");
|
|
|
|
let dag = Dag::open(&dag_dir.path()).unwrap();
|
|
|
|
elapsed(|| {
|
|
|
|
for i in (0..parents.len() as u64).step_by(10079) {
|
|
|
|
for j in (1..parents.len() as u64).step_by(2351) {
|
2019-12-05 09:41:13 +03:00
|
|
|
dag.gca_one((Id(i), Id(j))).unwrap();
|
dag: add benchmark about various "segment size" choices
Summary:
Add a benchmark testing various segment sizes - time to build segments, log
size, and ancestor calcuation performance. An initial output looks like:
building segment_size=4 0.223 ms segments: [34859, 10972, 3156, 834, 218, 55, 13, 3] log len: 1194609
ancestor calcuation segment_size=4 1.510 s
building segment_size=8 0.193 ms segments: [34859, 5472, 737, 95, 11, 1] log len: 958272
ancestor calcuation segment_size=8 1.630 s
building segment_size=10 0.213 ms segments: [34859, 4339, 460, 46, 4] log len: 918468
ancestor calcuation segment_size=10 1.739 s
building segment_size=12 0.187 ms segments: [34859, 3613, 316, 26, 2] log len: 893901
ancestor calcuation segment_size=12 1.934 s
building segment_size=14 0.202 ms segments: [34859, 3034, 228, 16, 1] log len: 873677
ancestor calcuation segment_size=14 2.311 s
building segment_size=16 0.199 ms segments: [34859, 2601, 172, 10] log len: 858211
ancestor calcuation segment_size=16 2.139 s
building segment_size=18 0.193 ms segments: [34859, 2280, 133, 7] log len: 847505
ancestor calcuation segment_size=18 2.301 s
building segment_size=20 0.175 ms segments: [34859, 2043, 105, 5] log len: 839458
ancestor calcuation segment_size=20 2.335 s
building segment_size=22 0.187 ms segments: [34859, 1847, 85, 3] log len: 832764
ancestor calcuation segment_size=22 2.500 s
building segment_size=24 0.206 ms segments: [34859, 1668, 71, 2] log len: 826775
ancestor calcuation segment_size=24 2.570 s
building segment_size=32 0.194 ms segments: [34859, 1230, 38, 1] log len: 811701
ancestor calcuation segment_size=32 3.091 s
building segment_size=64 0.215 ms segments: [34859, 598, 9] log len: 790230
ancestor calcuation segment_size=64 4.215 s
building segment_size=128 0.281 ms segments: [34859, 293, 2] log len: 779475
ancestor calcuation segment_size=128 7.573 s
It seems segment size = 8 to 20 might be a reasonable choice.
Reviewed By: sfilipco
Differential Revision: D16660078
fbshipit-source-id: f8af64c703ce0209b9b4c09112c9bdc4a1371172
2019-08-10 02:12:58 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
})
|
|
|
|
},
|
|
|
|
);
|
|
|
|
}
|
|
|
|
}
|