blackbox: initial library for native logging
Summary:
The blackbox library is intended to be the native logging library, with the
goals:
- Unifying Rust and Python logging. Currently, there are no good story in the
Rust logging land.
- Unifying cloud logging and local logging. Currently, there are very different
implementations and both sides have data that are unavailable in the other
side. My plan is to make local logging complete and cloud logging is just
extracting from the local logs.
- Typed data, binary format, with indexes. So the data is easier for code to
consume (ex. no more regex parsing text logs).
For the initial version, I just added the basic features: `log` (write) and
`filter` (read). I was optimizing for "code is easier to write", so I have made
decisions like:
- Silence errors and lose data. No `Fallible` in return type.
- Use `Vec` instead of complex iterator (can be changed later).
- Use a single `enum` type for different event types instead of originally
designed a complex "register" framework to register different event types [1].
(based on the assumption that we know all event types)
Although I did not optimize raw performance too much here, it might be good
enough with the previous improvements to `Log::sync`. I'd expect the fast path
to be able to log 10 entries per millisecond. If `Log::sync` becomes an issue,
we can move it to a background thread in the future.
There are also some places where the logic can be smarter and try to auto
recover from error cases (ex. by automatically rebuilding indexes for Log),
but they are not added in this diff. For now, such index corruption might just
delete the blackbox logs.
The indexedlog code was changed slightly to make the iterator less sensitive
to data corruptions, and continue with potentially good data. It affects one of
the tests added in this diff.
[1]: I later found https://github.com/dtolnay/typetag provides such feature. It
is not adopted for 2 reasons:
- It depends on rust-ctor, which is a bit hacky (ex. does not work for all platforms,
and might break with certain build flags).
See also https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions/9840
- Event definitions are in individual crates, instead of a centric one. There are pros
and cons. The cons are, programs (ex. the cloud logging executable) would have
unnecessary dependencies, and it's harder to reuse structures for different events.
Reviewed By: xavierd
Differential Revision: D15588431
fbshipit-source-id: 264a6b9c5fb587cf425ec2fe14cf1fe930c39892
2019-06-05 02:05:32 +03:00
|
|
|
[package]
|
|
|
|
|
name = "blackbox"
|
|
|
|
|
version = "0.1.0"
|
|
|
|
|
edition = "2018"
|
|
|
|
|
|
|
|
|
|
[dependencies]
|
2019-11-22 19:51:58 +03:00
|
|
|
anyhow = "1.0.20"
|
blackbox: initial library for native logging
Summary:
The blackbox library is intended to be the native logging library, with the
goals:
- Unifying Rust and Python logging. Currently, there are no good story in the
Rust logging land.
- Unifying cloud logging and local logging. Currently, there are very different
implementations and both sides have data that are unavailable in the other
side. My plan is to make local logging complete and cloud logging is just
extracting from the local logs.
- Typed data, binary format, with indexes. So the data is easier for code to
consume (ex. no more regex parsing text logs).
For the initial version, I just added the basic features: `log` (write) and
`filter` (read). I was optimizing for "code is easier to write", so I have made
decisions like:
- Silence errors and lose data. No `Fallible` in return type.
- Use `Vec` instead of complex iterator (can be changed later).
- Use a single `enum` type for different event types instead of originally
designed a complex "register" framework to register different event types [1].
(based on the assumption that we know all event types)
Although I did not optimize raw performance too much here, it might be good
enough with the previous improvements to `Log::sync`. I'd expect the fast path
to be able to log 10 entries per millisecond. If `Log::sync` becomes an issue,
we can move it to a background thread in the future.
There are also some places where the logic can be smarter and try to auto
recover from error cases (ex. by automatically rebuilding indexes for Log),
but they are not added in this diff. For now, such index corruption might just
delete the blackbox logs.
The indexedlog code was changed slightly to make the iterator less sensitive
to data corruptions, and continue with potentially good data. It affects one of
the tests added in this diff.
[1]: I later found https://github.com/dtolnay/typetag provides such feature. It
is not adopted for 2 reasons:
- It depends on rust-ctor, which is a bit hacky (ex. does not work for all platforms,
and might break with certain build flags).
See also https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions/9840
- Event definitions are in individual crates, instead of a centric one. There are pros
and cons. The cons are, programs (ex. the cloud logging executable) would have
unnecessary dependencies, and it's harder to reuse structures for different events.
Reviewed By: xavierd
Differential Revision: D15588431
fbshipit-source-id: 264a6b9c5fb587cf425ec2fe14cf1fe930c39892
2019-06-05 02:05:32 +03:00
|
|
|
byteorder = "1"
|
|
|
|
|
indexedlog = { path = "../indexedlog" }
|
2019-06-05 02:05:32 +03:00
|
|
|
lazy_static = "1"
|
blackbox: initial library for native logging
Summary:
The blackbox library is intended to be the native logging library, with the
goals:
- Unifying Rust and Python logging. Currently, there are no good story in the
Rust logging land.
- Unifying cloud logging and local logging. Currently, there are very different
implementations and both sides have data that are unavailable in the other
side. My plan is to make local logging complete and cloud logging is just
extracting from the local logs.
- Typed data, binary format, with indexes. So the data is easier for code to
consume (ex. no more regex parsing text logs).
For the initial version, I just added the basic features: `log` (write) and
`filter` (read). I was optimizing for "code is easier to write", so I have made
decisions like:
- Silence errors and lose data. No `Fallible` in return type.
- Use `Vec` instead of complex iterator (can be changed later).
- Use a single `enum` type for different event types instead of originally
designed a complex "register" framework to register different event types [1].
(based on the assumption that we know all event types)
Although I did not optimize raw performance too much here, it might be good
enough with the previous improvements to `Log::sync`. I'd expect the fast path
to be able to log 10 entries per millisecond. If `Log::sync` becomes an issue,
we can move it to a background thread in the future.
There are also some places where the logic can be smarter and try to auto
recover from error cases (ex. by automatically rebuilding indexes for Log),
but they are not added in this diff. For now, such index corruption might just
delete the blackbox logs.
The indexedlog code was changed slightly to make the iterator less sensitive
to data corruptions, and continue with potentially good data. It affects one of
the tests added in this diff.
[1]: I later found https://github.com/dtolnay/typetag provides such feature. It
is not adopted for 2 reasons:
- It depends on rust-ctor, which is a bit hacky (ex. does not work for all platforms,
and might break with certain build flags).
See also https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions/9840
- Event definitions are in individual crates, instead of a centric one. There are pros
and cons. The cons are, programs (ex. the cloud logging executable) would have
unnecessary dependencies, and it's harder to reuse structures for different events.
Reviewed By: xavierd
Differential Revision: D15588431
fbshipit-source-id: 264a6b9c5fb587cf425ec2fe14cf1fe930c39892
2019-06-05 02:05:32 +03:00
|
|
|
libc = "0.2"
|
2019-09-14 01:14:26 +03:00
|
|
|
parking_lot = "0.9"
|
blackbox: initial library for native logging
Summary:
The blackbox library is intended to be the native logging library, with the
goals:
- Unifying Rust and Python logging. Currently, there are no good story in the
Rust logging land.
- Unifying cloud logging and local logging. Currently, there are very different
implementations and both sides have data that are unavailable in the other
side. My plan is to make local logging complete and cloud logging is just
extracting from the local logs.
- Typed data, binary format, with indexes. So the data is easier for code to
consume (ex. no more regex parsing text logs).
For the initial version, I just added the basic features: `log` (write) and
`filter` (read). I was optimizing for "code is easier to write", so I have made
decisions like:
- Silence errors and lose data. No `Fallible` in return type.
- Use `Vec` instead of complex iterator (can be changed later).
- Use a single `enum` type for different event types instead of originally
designed a complex "register" framework to register different event types [1].
(based on the assumption that we know all event types)
Although I did not optimize raw performance too much here, it might be good
enough with the previous improvements to `Log::sync`. I'd expect the fast path
to be able to log 10 entries per millisecond. If `Log::sync` becomes an issue,
we can move it to a background thread in the future.
There are also some places where the logic can be smarter and try to auto
recover from error cases (ex. by automatically rebuilding indexes for Log),
but they are not added in this diff. For now, such index corruption might just
delete the blackbox logs.
The indexedlog code was changed slightly to make the iterator less sensitive
to data corruptions, and continue with potentially good data. It affects one of
the tests added in this diff.
[1]: I later found https://github.com/dtolnay/typetag provides such feature. It
is not adopted for 2 reasons:
- It depends on rust-ctor, which is a bit hacky (ex. does not work for all platforms,
and might break with certain build flags).
See also https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions/9840
- Event definitions are in individual crates, instead of a centric one. There are pros
and cons. The cons are, programs (ex. the cloud logging executable) would have
unnecessary dependencies, and it's harder to reuse structures for different events.
Reviewed By: xavierd
Differential Revision: D15588431
fbshipit-source-id: 264a6b9c5fb587cf425ec2fe14cf1fe930c39892
2019-06-05 02:05:32 +03:00
|
|
|
serde = "1"
|
2019-06-26 20:59:13 +03:00
|
|
|
serde_alt = { path = "serde_alt" }
|
2020-02-22 01:07:01 +03:00
|
|
|
serde_cbor = "0.11"
|
blackbox: initial library for native logging
Summary:
The blackbox library is intended to be the native logging library, with the
goals:
- Unifying Rust and Python logging. Currently, there are no good story in the
Rust logging land.
- Unifying cloud logging and local logging. Currently, there are very different
implementations and both sides have data that are unavailable in the other
side. My plan is to make local logging complete and cloud logging is just
extracting from the local logs.
- Typed data, binary format, with indexes. So the data is easier for code to
consume (ex. no more regex parsing text logs).
For the initial version, I just added the basic features: `log` (write) and
`filter` (read). I was optimizing for "code is easier to write", so I have made
decisions like:
- Silence errors and lose data. No `Fallible` in return type.
- Use `Vec` instead of complex iterator (can be changed later).
- Use a single `enum` type for different event types instead of originally
designed a complex "register" framework to register different event types [1].
(based on the assumption that we know all event types)
Although I did not optimize raw performance too much here, it might be good
enough with the previous improvements to `Log::sync`. I'd expect the fast path
to be able to log 10 entries per millisecond. If `Log::sync` becomes an issue,
we can move it to a background thread in the future.
There are also some places where the logic can be smarter and try to auto
recover from error cases (ex. by automatically rebuilding indexes for Log),
but they are not added in this diff. For now, such index corruption might just
delete the blackbox logs.
The indexedlog code was changed slightly to make the iterator less sensitive
to data corruptions, and continue with potentially good data. It affects one of
the tests added in this diff.
[1]: I later found https://github.com/dtolnay/typetag provides such feature. It
is not adopted for 2 reasons:
- It depends on rust-ctor, which is a bit hacky (ex. does not work for all platforms,
and might break with certain build flags).
See also https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions/9840
- Event definitions are in individual crates, instead of a centric one. There are pros
and cons. The cons are, programs (ex. the cloud logging executable) would have
unnecessary dependencies, and it's harder to reuse structures for different events.
Reviewed By: xavierd
Differential Revision: D15588431
fbshipit-source-id: 264a6b9c5fb587cf425ec2fe14cf1fe930c39892
2019-06-05 02:05:32 +03:00
|
|
|
serde_derive = "1"
|
2019-06-05 02:05:32 +03:00
|
|
|
serde_json = "1"
|
blackbox: initial library for native logging
Summary:
The blackbox library is intended to be the native logging library, with the
goals:
- Unifying Rust and Python logging. Currently, there are no good story in the
Rust logging land.
- Unifying cloud logging and local logging. Currently, there are very different
implementations and both sides have data that are unavailable in the other
side. My plan is to make local logging complete and cloud logging is just
extracting from the local logs.
- Typed data, binary format, with indexes. So the data is easier for code to
consume (ex. no more regex parsing text logs).
For the initial version, I just added the basic features: `log` (write) and
`filter` (read). I was optimizing for "code is easier to write", so I have made
decisions like:
- Silence errors and lose data. No `Fallible` in return type.
- Use `Vec` instead of complex iterator (can be changed later).
- Use a single `enum` type for different event types instead of originally
designed a complex "register" framework to register different event types [1].
(based on the assumption that we know all event types)
Although I did not optimize raw performance too much here, it might be good
enough with the previous improvements to `Log::sync`. I'd expect the fast path
to be able to log 10 entries per millisecond. If `Log::sync` becomes an issue,
we can move it to a background thread in the future.
There are also some places where the logic can be smarter and try to auto
recover from error cases (ex. by automatically rebuilding indexes for Log),
but they are not added in this diff. For now, such index corruption might just
delete the blackbox logs.
The indexedlog code was changed slightly to make the iterator less sensitive
to data corruptions, and continue with potentially good data. It affects one of
the tests added in this diff.
[1]: I later found https://github.com/dtolnay/typetag provides such feature. It
is not adopted for 2 reasons:
- It depends on rust-ctor, which is a bit hacky (ex. does not work for all platforms,
and might break with certain build flags).
See also https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions/9840
- Event definitions are in individual crates, instead of a centric one. There are pros
and cons. The cons are, programs (ex. the cloud logging executable) would have
unnecessary dependencies, and it's harder to reuse structures for different events.
Reviewed By: xavierd
Differential Revision: D15588431
fbshipit-source-id: 264a6b9c5fb587cf425ec2fe14cf1fe930c39892
2019-06-05 02:05:32 +03:00
|
|
|
|
|
|
|
|
[dev-dependencies]
|
2019-06-26 20:59:13 +03:00
|
|
|
minibench = { path = "../minibench" }
|
blackbox: initial library for native logging
Summary:
The blackbox library is intended to be the native logging library, with the
goals:
- Unifying Rust and Python logging. Currently, there are no good story in the
Rust logging land.
- Unifying cloud logging and local logging. Currently, there are very different
implementations and both sides have data that are unavailable in the other
side. My plan is to make local logging complete and cloud logging is just
extracting from the local logs.
- Typed data, binary format, with indexes. So the data is easier for code to
consume (ex. no more regex parsing text logs).
For the initial version, I just added the basic features: `log` (write) and
`filter` (read). I was optimizing for "code is easier to write", so I have made
decisions like:
- Silence errors and lose data. No `Fallible` in return type.
- Use `Vec` instead of complex iterator (can be changed later).
- Use a single `enum` type for different event types instead of originally
designed a complex "register" framework to register different event types [1].
(based on the assumption that we know all event types)
Although I did not optimize raw performance too much here, it might be good
enough with the previous improvements to `Log::sync`. I'd expect the fast path
to be able to log 10 entries per millisecond. If `Log::sync` becomes an issue,
we can move it to a background thread in the future.
There are also some places where the logic can be smarter and try to auto
recover from error cases (ex. by automatically rebuilding indexes for Log),
but they are not added in this diff. For now, such index corruption might just
delete the blackbox logs.
The indexedlog code was changed slightly to make the iterator less sensitive
to data corruptions, and continue with potentially good data. It affects one of
the tests added in this diff.
[1]: I later found https://github.com/dtolnay/typetag provides such feature. It
is not adopted for 2 reasons:
- It depends on rust-ctor, which is a bit hacky (ex. does not work for all platforms,
and might break with certain build flags).
See also https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions/9840
- Event definitions are in individual crates, instead of a centric one. There are pros
and cons. The cons are, programs (ex. the cloud logging executable) would have
unnecessary dependencies, and it's harder to reuse structures for different events.
Reviewed By: xavierd
Differential Revision: D15588431
fbshipit-source-id: 264a6b9c5fb587cf425ec2fe14cf1fe930c39892
2019-06-05 02:05:32 +03:00
|
|
|
tempfile = "3"
|
2019-06-26 20:59:13 +03:00
|
|
|
|
|
|
|
|
[[bench]]
|
|
|
|
|
name = "blackbox"
|
|
|
|
|
harness = false
|
