Summary:
At the moment our memcache lease implementation has surprising side effect -
releasing the lease with `put_success = true` sets a memcache key, which means
that until this key is evicted from memcache it's not possible to take this
lease again (`try_add_put_lease()` fails with because it uses
memcache.add_with_ttl() method, which returns false if a key already exists in
memcache).
That caused a deadlock in our derived data generation:
1) `derive_may_panic()` function does a few things - it actually derives the
data, then releases the lease but it DOES NOT update derived data mapping
(mapping is updated later in a batch for performance reasons).
2) So the following situation is possible - data was successfully derived, memcache
lease key was updated (hence taking the lease one more time is not possible) but
mapping update failed.
This is an invalid state - any attempt to get the lease fails, but mapping doesn't
exist either, and workers just loop forever.
There seems to be a few ways to fix the deadlock, but changing release_lease()
logic seems to cause less surprises in future - we expect memcache leases to be
a "soft lock", and when a lock was unlocked it should be possible to lock it
again.
So currently release_lease() checks the value that's set in the memcache key. If this value
is LockState::uploaded(...) (which can be set only by MemcacheLease::put()`), then nothing
is done to this key, otherwise the key is deleted. this ensure that
1) No changes to the current behaviour of the CachingBlobstore - lease is invalidated on inserting a new value in the cache
2) Deadlock issue is fixed, because release_lease() now deletes the key.
Reviewed By: krallin
Differential Revision: D19468234
fbshipit-source-id: 85525bc11802a0fd4d94a692bf7c6dd4569a6f46
Summary:
This updates hg derivation to minimally handle octopus merges. The underlying
goal is to allow converting a Git repository to a Mercurial repository by going
through Bonsais (and to produce something meaningful in doing so).
Since hg doesn't currently have a notion of parents beyond p1 and p2, this
support is (purposefully) very minimal. What we seek to guarantee is that the
files in the hg manifest are logically identical to what is represented by the
Bonsai changeset. However, we do not preserve parents in filenodes referencing
stepparents (it might be confusing for hg to encounter files).
Implementation-wise, this leaves most of the existing code unchanged: the only
place where we actually to handle > 2 parents is when deriving the Manifest.
It does however remove assertions that assume 2 parents.
To make this work, I added a `Traced` wrapper in the `manifest` crate that lets
us find which parent manifest a given child manifest or filenode came from.
This lets us filter out ancestors for parents that aren't p1 or p2.
Finally, for now, I've recorded stepparents in commit extras (we just
concatenate their hexified IDs together). This seems consistent with things
like rebase_source that I've seen here or there, but let me know if not.
Reviewed By: StanislavGlebik
Differential Revision: D19348533
fbshipit-source-id: a40cb81483b0741f09c026055c719c0c1ed8e809
