Summary:
**Context**: We want to roll out the new WAL multiplexed blobstore and need final tweaks before doing that. See https://fb.quip.com/vyTWAgGZSA2Z
Operation key was initially added on D20557702 (3d314761c3) because we had the problem of doing `select * where blobstore_key=X` and getting entries from multiple writings to the same key.
This is not a problem on WAL anymore, because we never do this type of select, each row is enough information for us and there's no need for follow up queries.
After we fully remove old multiplex, we can delete all code for operation key.
Note: Since I was also touching the table schema, I also fixed some columns, made `blob_size` not null, since we never insert a NULL value, it being nullable was a remnant from the old sync queue, where it was added after the table already existed.
Differential Revision: D40718122
fbshipit-source-id: 661c6f28d1c13f94fe757b7c13f5832187146208
Summary:
**Context**: We want to roll out the new WAL multiplexed blobstore and need final tweaks before doing that. See https://fb.quip.com/vyTWAgGZSA2Z
Currently, when the healer doesn't find a blob in any blobstore, it puts it back in the queue.
However, it the blob did fail to write on all blobstores, it will stay in the queue forever. This diff adds a limit to how many times an item can be put back in the queue (magic number 20).
This makes it necessary to change the SQL schema, but that's fine because we haven't started putting this in prod yet.
Differential Revision: D40715865
fbshipit-source-id: 55a6e4dd0c7bb6ae36ba6b2d724e7a2260367b56
Summary:
This is just an outline for the WAL, implementation of which is coming in the further diffs.
The diff for the actual MySQL table creation - D37107784.
Reviewed By: markbt
Differential Revision: D37040270
fbshipit-source-id: 69dda1a19cbdab4c53eca04565ab0dc97e130bc1
Summary: This simplifies test setup and means can stop throwing away creation SQL errors
Reviewed By: StanislavGlebik
Differential Revision: D31536408
fbshipit-source-id: 4d3df0f69c5b49719196c8a1b20d65c965d88869
Summary:
Schema for `xdb.mononoke_blobstore_sync_queue : blobstore_sync_queue` wasn't altered yet as AOSC [wasn't working](https://fb.workplace.com/groups/mysql.users/permalink/4991036464278262/).
This will allow the blobstore healer to make decisions about how many blobs it should copy at once. The goal is to eliminate the OOMs that were regularly seen by giving the blobstore healer a memory budget.
Reviewed By: krallin
Differential Revision: D25925522
fbshipit-source-id: d3714dbadc74274a4c4a0e66fa732b84bef89227
Summary:
From time to time we're experiencing the blobstore healer to crash
because its SQL queries timing out. The rootcause of the problem
is that the same blob_key may show up on the queue many times repeatedly
and the query is trying to select all occurences.
But, the original intention of blobstore healer is to act on a single
put operation across all blobstores. To be able to identify which
puts in the healer queue are part of the same operation we need
some unique id that we'll use per such operation, let's call it OperationKey.
corresponding configerator change to create db column: D20557659
NOTE: This diff has to be landed and rolled out first, before D20557700 is rolled out. I'm assuming that after some time since rolling out this diff all the rows in the production db will have proper `operation_key` value set.
Reviewed By: krallin
Differential Revision: D20557702
fbshipit-source-id: 404d9fdea6796b38193292d1bbd4b8cd4b5b3eb8
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
Summary:
D19767626 added an original_timestamp column to the
blobstore_sync_queue. Update the sqlite schema to keep it in sync.
Reviewed By: krallin
Differential Revision: D19787488
fbshipit-source-id: ad576e2ec99349953e2ab69e3defb73d1ff556c0
Summary:
Modify the multiplexed blobstore implementation so that the
multiplex_id is written to the healer queue after a put. Further, update the
blobstore healer to only look at entries with the same multiplex ID as it's
configured to run with.
Reviewed By: ahornby
Differential Revision: D19770057
fbshipit-source-id: 41db19f0b0f84c048d49ab9e6258cccc89cf4195
