rui314/mold
1
1
Fork 0
mirror of https://github.com/rui314/mold.git synced 2024-08-16 08:20:23 +03:00
Code Issues Projects Releases Wiki Activity
f180ac21c8
mold/common/hyperloglog.cc

21 lines
453 B
C++
Raw Normal View History

Optimize string merging Linking clang-13 with debug info takes ~3.6 seconds on a simulated 10-core/20-threads machine. mold spends most of its time (~2.3 seconds) merging string literals in .debug_str. Input .debug_str sections contain 70 million string literals in total, which is reduced to 2 million after de-duplication. The input object files contain a lot of duplicates. clang-13 with debug info is enormous -- it is ~3.1 GiB after linking. It looks like TBB's concurrent hashmap doesn't scale well with the input. In this patch, I implemented our own concurrent hashmap. The hashmap is extremely lightweight and support only the key-value insertion operation. It doesn't even support rehashing. It aborts once the hash table becomes full. In order to know the correct size for the hashmap before inserting strings into it, I also implemented HyperLogLog algorithm in this patch. HyperLogLog is an algorithm that gives a fairly accurate estimate on the number of unique elements. With this patch, mold can link clang-13 in ~2.5 seconds, which is ~30% faster than before. https://github.com/rui314/mold/issues/73
2021-07-12 05:00:10 +03:00
// This file implements HyperLogLog algorithm, which estimates
// the number of unique items in a given multiset.
//
// For more info, read
// https://engineering.fb.com/2018/12/13/data-infrastructure/hyperloglog
Reorganize files
2023-01-16 10:26:52 +03:00
#include "common.h"
Optimize string merging Linking clang-13 with debug info takes ~3.6 seconds on a simulated 10-core/20-threads machine. mold spends most of its time (~2.3 seconds) merging string literals in .debug_str. Input .debug_str sections contain 70 million string literals in total, which is reduced to 2 million after de-duplication. The input object files contain a lot of duplicates. clang-13 with debug info is enormous -- it is ~3.1 GiB after linking. It looks like TBB's concurrent hashmap doesn't scale well with the input. In this patch, I implemented our own concurrent hashmap. The hashmap is extremely lightweight and support only the key-value insertion operation. It doesn't even support rehashing. It aborts once the hash table becomes full. In order to know the correct size for the hashmap before inserting strings into it, I also implemented HyperLogLog algorithm in this patch. HyperLogLog is an algorithm that gives a fairly accurate estimate on the number of unique elements. With this patch, mold can link clang-13 in ~2.5 seconds, which is ~30% faster than before. https://github.com/rui314/mold/issues/73
2021-07-12 05:00:10 +03:00
#include <cmath>
Move target-independent files to the top directory
2021-09-08 13:49:51 +03:00
namespace mold {
Move everything into mold::elf namespace
2021-09-02 12:19:43 +03:00
Optimize string merging Linking clang-13 with debug info takes ~3.6 seconds on a simulated 10-core/20-threads machine. mold spends most of its time (~2.3 seconds) merging string literals in .debug_str. Input .debug_str sections contain 70 million string literals in total, which is reduced to 2 million after de-duplication. The input object files contain a lot of duplicates. clang-13 with debug info is enormous -- it is ~3.1 GiB after linking. It looks like TBB's concurrent hashmap doesn't scale well with the input. In this patch, I implemented our own concurrent hashmap. The hashmap is extremely lightweight and support only the key-value insertion operation. It doesn't even support rehashing. It aborts once the hash table becomes full. In order to know the correct size for the hashmap before inserting strings into it, I also implemented HyperLogLog algorithm in this patch. HyperLogLog is an algorithm that gives a fairly accurate estimate on the number of unique elements. With this patch, mold can link clang-13 in ~2.5 seconds, which is ~30% faster than before. https://github.com/rui314/mold/issues/73
2021-07-12 05:00:10 +03:00
i64 HyperLogLog::get_cardinality() const {
double z = 0;
for (i64 val : buckets)
replace pow(2,x) by ldexp(1.0,x) for performance
2023-07-20 10:28:35 +03:00
z += std::ldexp(1.0, -val);
Optimize string merging Linking clang-13 with debug info takes ~3.6 seconds on a simulated 10-core/20-threads machine. mold spends most of its time (~2.3 seconds) merging string literals in .debug_str. Input .debug_str sections contain 70 million string literals in total, which is reduced to 2 million after de-duplication. The input object files contain a lot of duplicates. clang-13 with debug info is enormous -- it is ~3.1 GiB after linking. It looks like TBB's concurrent hashmap doesn't scale well with the input. In this patch, I implemented our own concurrent hashmap. The hashmap is extremely lightweight and support only the key-value insertion operation. It doesn't even support rehashing. It aborts once the hash table becomes full. In order to know the correct size for the hashmap before inserting strings into it, I also implemented HyperLogLog algorithm in this patch. HyperLogLog is an algorithm that gives a fairly accurate estimate on the number of unique elements. With this patch, mold can link clang-13 in ~2.5 seconds, which is ~30% faster than before. https://github.com/rui314/mold/issues/73
2021-07-12 05:00:10 +03:00
return ALPHA * NBUCKETS * NBUCKETS / z;
}
Move target-independent files to the top directory
2021-09-08 13:49:51 +03:00
} // namespace mold
Reference in New Issue Copy Permalink