Merge pull request #3404 from urbit/philip/safe-merge

clay: add safe merge strategies

pkg/arvo/gen/hood/merge.hoon

@@ -3,6 +3,7 @@
 ::::  /hoon/merge/hood/gen
   ::
 /?    310
+/*  help-text  %txt  /gen/hood/merge/help/txt
 =,  clay
 ::
 |%
@@ -14,12 +15,13 @@
   ::
 :-  %say
 |=  $:  {now/@da eny/@uvJ bek/beak}
-        {arg/{?(sorc {syd/$@(desk beaky) sorc})} cas/case gem/?(germ $auto)}
+        {arg/{?(~ sorc {syd/$@(desk beaky) sorc})} cas/case gem/?(germ $auto)}
     ==
 =*  our  p.bek
 |^  :-  %kiln-merge
-    ^-  {syd/desk her/ship sud/desk cas/case gem/?(germ $auto)}
+    ^-  $@(~ {syd/desk her/ship sud/desk cas/case gem/?(germ $auto)})
     ?-  arg
+      ~  ((slog (turn help-text |=(=@t leaf+(trip t)))) ~)
       {@ @ ~}
         =+(arg [sud ?.(=(our her) her (sein:title p.bek now her)) sud (opt-case da+now) gem])
     ::

pkg/arvo/gen/hood/merge/help.txt

@@ -0,0 +1,110 @@
+Usage:
+
+    |merge %destination-desk ~source-ship %source-desk
+    |merge %destination-desk ~source-ship %source-desk, =gem %strategy
+    |merge %destination-desk ~source-ship %source-desk, =cas ud+5
+
+We support various merge strategies.  A "commit" is a snapshot of
+the files with a list of parents plus a date.   Most commits have
+one parent; a "merge" commit is a commit with two parents.  The
+%home desk starts with an initial commit with no parents; commits
+with several parents ("octopus merges") are possible but we don't
+generate them right now.
+                                                                      
+Unless otherwise specified, all of the following create a new commit
+with the source and destination commits as parents.
+                                                                      
+Several strategies need a "merge-base".  They find it by identifying
+the most recent common ancestor of the two desks.  If none, fail
+with %merge-no-merge-base; if there are two or more, pick one.
+                                                                      
+%init: the only way to create a desk.  Not a true merge, since it
+simply assigns the source commit to the destination.
+                                                                      
+%fine: if source or destination are in the ancestry of each other,
+use the newer one; else abort.  If the destination is ahead of the
+source, succeed but do nothing.  If the source is ahead of the
+destination, assign the next revision number to the source commit.
+Some call this "fast-forward".
+                                                                      
+%meet: combine changes, failing if both sides changed the same file.
+Specifically, take diff(merge-base,source) and
+diff(merge-base,destination) and combine them as long as those diffs
+touch different files.
+                                                                      
+%mate: combine changes, failing if both sides changed the same part
+of a file.  Identical to %meet, except that some marks, like %hoon,
+allow intelligent merge of changes to different parts of a file.
+                                                                      
+%meld: combine changes; if both sides changed the same part of a
+file, use the version of the file in the merge-base.
+                                                                      
+%only-this: create a merge commit with exactly the contents of the
+destination desk.
+                                                                      
+%only-that: create a merge commit with exactly the contents of the
+source commit.
+                                                                      
+%take-this: create a merge commit with exactly the contents of the
+destination desk except take any files from the source commit which
+are not in the destination desk.
+                                                                      
+%take-that: create a merge commit with exactly the contents of the
+source commit except preserve any files from the destination desk
+which are not in the source commit.
+                                                                      
+%meet-this: merge as in %meet, except if both sides changed the same
+file, use the version in the destination desk.
+                                                                      
+%meet-that: merge as in %meet, except if both sides changed the same
+file, use the version in the source commit.
+                                                                      
+# Examples and notes:
+                                                                      
+The most common merge strategy is %mate, which is a normal 3-way
+merge which aborts on conflict.
+                                                                      
+%take-that is useful to "force" an OTA.  After running %take-that,
+you're guaranteed to have exactly the files in the source commit plus
+any files you separately added.
+                                                                      
+We speak of merging into a destination *desk* from a source *commit*
+because while you can only merge on top of a desk, you can merge from
+historical commits.  For example,
+                                                                      
+    |merge %old our %home, =cas ud+5, =gem %init
+                                                                      
+will create a new desk called %old with the 5th commit in %home.
+You can revert the contents of a desk to what they were yesterday
+with
+                                                                      
+    |merge %home our %home, =cas da+(sub now ~d1), =gem %only-that
+                                                                      
+Note this is a normal %only-that merge, which means you're creating a
+*new* commit with the old *contents*.
+                                                                      
+%meld is rarely used on its own, however if you specify %auto or
+omit the merge strategy, %kiln will run a %meld merge into a scratch
+desk and then annotate the conflicts there.
+                                                                      
+If you resolve merge conflicts manually, for example by mounting the
+desks, copying the files in unix and then running |commit, you
+should usually run an %only-this merge.  This will not change the
+newly-fixed contents of your desk, but it will record that the merge
+happened so that those conflicts don't reappear in later merges.
+                                                                      
+If you get a %merge-no-merge-base error, this means you're trying to
+merge two desks which have no common ancestors.  You need to give
+them a common ancestor by choosing a merge strategy which doesn't
+need a merge-base, like %only-this, %only-that, %take-this, or
+%take-that.
+                                                                      
+%take-this could be useful to install 3rd party software, but you
+couldn't get subsequent updates this way, since the files would
+already exist in the destination desk.  Something like "take only
+the files which aren't in my OTA source or any other 3rd party app"
+would be basically correct.  This would require a parameter listing
+the desks to not conflict with.
+                                                                      
+%meet-this and %meet-that imply the existence of %mate-this and
+%mate-that, but those don't exist yet.

pkg/arvo/lib/hood/kiln.hoon

@@ -25,7 +25,7 @@
   ==
 +$  per-desk                                            ::  per-desk state
   $:  auto=?                                            ::  escalate on failure
-      gem=germ                                          ::  strategy
+      gem=?(%this %that germ)                           ::  strategy
       her=@p                                            ::  from ship
       sud=@tas                                          ::  from desk
       cas=case                                          ::  at case
@@ -47,11 +47,12 @@
       sud=desk                                          ::
   ==
 +$  kiln-merge                                          ::
+  $@  ~
   $:  syd=desk                                          ::
       ali=ship                                          ::
       sud=desk                                          ::
       cas=case                                          ::
-      gim=?($auto germ)                                 ::
+      gim=?(%auto germ)                                 ::
   ==
 --
 |=  [bowl:gall state]
@@ -204,14 +205,14 @@
   ::
   ::  If destination desk doesn't exist, need a %init merge.  If this is
   ::  its first revision, it probably doesn't have a mergebase yet, so
-  ::  use %that.
+  ::  use %take-that.
   ::
   ++  get-germ
     |=  =desk
     =+  .^(=cass:clay %cw /(scot %p our)/[desk]/(scot %da now))
     ?-  ud.cass
       %0  %init
-      %1  %that
+      %1  %take-that
       *   %mate
     ==
   ::
@@ -372,6 +373,7 @@
 ::
 ++  poke-merge                                        ::
   |=  kiln-merge
+  ?~  +<  abet
   abet:abet:(merge:(work syd) ali sud cas gim)
 ::
 ++  poke-cancel
@@ -603,20 +605,21 @@
     ::    Initial merges from any source must use the %init germ.
     ::    Subsequent merges may use any germ, but if the source is
     ::    a remote ship with which we have not yet merged, we won't
-    ::    share a merge-base commit and all germs but %that will fail.
+    ::    share a merge-base commit and all germs but %only-that will
+    ::    fail.
     ::
-    ::    We want to always use %that for the first remote merge.
+    ::    We want to always use %only-that for the first remote merge.
-    ::    But we also want local syncs (%base to %home or %kids)
+    ::    But we also want local syncs (%base to %home or %kids) to
-    ::    to succeed after that first remote sync. To accomplish both
+    ::    succeed after that first remote sync. To accomplish both we
-    ::    we simply use %that for the first three sync merges.
+    ::    simply use %only-that for the first three sync merges.  (The
-    ::    (The first two are from the pill.)
+    ::    first two are from the pill.)
     ::
     =/  =germ
       =/  =cass
         .^(cass:clay %cw /(scot %p our)/[syd]/(scot %da now))
       ?:  =(0 ud.cass)
         %init
-      ?:((gth 2 ud.cass) %that %mate)
+      ?:((gth 2 ud.cass) %only-that %mate)
     =<  %-  spam
         ?:  =(our her)  ~
         [(render "beginning sync" sud her syd) ~]
@@ -677,6 +680,8 @@
   ::
   ++  perform                                         ::
     ^+  .
+    ?<  ?=(%this gem)
+    ?<  ?=(%that gem)
     (blab [%pass /kiln/[syd] %arvo %c [%merg syd her sud cas gem]] ~)
   ::
   ++  fancy-merge                                     ::  send to self
@@ -812,6 +817,8 @@
       =.  ..mere  (fancy-merge tic our syd %init)
       =>  (spam leaf+"%melding %{(trip sud)} into scratch space" ~)
       %-  blab  :_  ~
+      ?<  ?=(%this gem)
+      ?<  ?=(%that gem)
       =/  note  [%merg (cat 3 syd '-scratch') her sud cas gem]
       [%pass /kiln/[syd] %arvo %c note]
     ==

pkg/arvo/sys/vane/clay.hoon

@@ -2032,13 +2032,13 @@
       ^-  (each (unit merge-result) [term tang])
       ?-    germ
       ::
-      ::  If this is a %this merge, we check to see if ali's and bob's
+      ::  If this is a %only-this merge, we check to see if ali's and bob's
       ::  commits are the same, in which case we're done.  Otherwise, we
       ::  check to see if ali's commit is in the ancestry of bob's, in
       ::  which case we're done.  Otherwise, we create a new commit with
       ::  bob's data plus ali and bob as parents.
       ::
-          %this
+          %only-this
         ?:  =(r.ali-yaki r.bob-yaki)
           &+~
         ?:  (~(has in (reachable-takos:ze r.bob-yaki)) r.ali-yaki)
@@ -2049,12 +2049,12 @@
             lat=~
         ==
       ::
-      ::  If this is a %that merge, we check to see if ali's and bob's
+      ::  If this is a %only-that merge, we check to see if ali's and bob's
       ::  commits are the same, in which case we're done.  Otherwise, we
       ::  create a new commit with ali's data plus ali and bob as
       ::  parents.
       ::
-          %that
+          %only-that
         ?:  =(r.ali-yaki r.bob-yaki)
           &+~
         :*  %&  ~
@@ -2063,6 +2063,36 @@
             lat=~
         ==
       ::
+      ::  Create a merge commit with exactly the contents of the
+      ::  destination desk except take any files from the source commit
+      ::  which are not in the destination desk.
+      ::
+          %take-this
+        ?:  =(r.ali-yaki r.bob-yaki)
+          &+~
+        ?:  (~(has in (reachable-takos:ze r.bob-yaki)) r.ali-yaki)
+          &+~
+        =/  new-data  (~(uni by q.ali-yaki) q.bob-yaki)
+        :*  %&  ~
+            conflicts=~
+            new=&+[[r.bob-yaki r.ali-yaki ~] (to-yuki new-data)]
+            lat=~
+        ==
+      ::
+      ::  Create a merge commit with exactly the contents of the source
+      ::  commit except preserve any files from the destination desk
+      ::  which are not in the source commit.
+      ::
+          %take-that
+        ?:  =(r.ali-yaki r.bob-yaki)
+          &+~
+        =/  new-data  (~(uni by q.bob-yaki) q.ali-yaki)
+        :*  %&  ~
+            conflicts=~
+            new=&+[[r.bob-yaki r.ali-yaki ~] (to-yuki new-data)]
+            lat=~
+        ==
+      ::
       ::  If this is a %fine merge, we check to see if ali's and bob's
       ::  commits are the same, in which case we're done.  Otherwise, we
       ::  check to see if ali's commit is in the ancestry of bob's, in
@@ -2082,7 +2112,7 @@
           ==
         &+`[conflicts=~ new=|+ali-yaki lat=~]
       ::
-          ?(%meet %mate %meld)
+          ?(%meet %mate %meld %meet-this %meet-that)
         ?:  =(r.ali-yaki r.bob-yaki)
           &+~
         ?:  (~(has in (reachable-takos:ze r.bob-yaki)) r.ali-yaki)
@@ -2095,7 +2125,7 @@
               leaf+"consider a %this or %that merge to get a mergebase"
           ==
         =/  merge-point=yaki  n.merge-points
-        ?.  ?=(%meet germ)
+        ?:  ?=(?(%mate %meld) germ)
           =/  ali-diffs=cane  (diff-base ali-yaki bob-yaki merge-point)
           =/  bob-diffs=cane  (diff-base bob-yaki ali-yaki merge-point)
           =/  bof=(map path (unit cage))
@@ -2113,13 +2143,35 @@
           %-  ~(uni by `(map path *)`cal.bob-diffs)
           %-  ~(uni by `(map path *)`can.bob-diffs)
           `(map path *)`old.bob-diffs
-        ?.  =(~ both-diffs)
+        ?:  &(?=(%meet germ) !=(~ both-diffs))
           :~  %|  %meet-conflict
             >~(key by both-diffs)<
             leaf+"consider a %mate merge"
           ==
+        =/  both-done=(map path lobe)
+          |^
+          ?-  germ
+            %meet       ~
+            %meet-this  (resolve (~(uni by new.bob-diffs) cal.bob-diffs))
+            %meet-that  (resolve (~(uni by new.ali-diffs) cal.ali-diffs))
+          ==
+          ++  resolve
+            |=  news=(map path lobe)
+            %-  malt  ^-  (list [path lobe])
+            %+  murn  ~(tap by both-diffs)
+            |=  [=path *]
+            ^-  (unit [^path lobe])
+            =/  new  (~(get by news) path)
+            ?~  new
+              ~
+            `[path u.new]
+          --
+        ::
+        =/  deleted
+          %-  ~(dif by (~(uni by old.ali-diffs) old.bob-diffs))
+          (~(run by both-done) |=(* ~))
         =/  not-deleted=(map path lobe)
-          %+  roll  ~(tap by (~(uni by old.ali-diffs) old.bob-diffs))
+          %+  roll  ~(tap by deleted)
           =<  .(not-deleted q.merge-point)
           |=  [[pax=path ~] not-deleted=(map path lobe)]
           (~(del by not-deleted) pax)
@@ -2401,7 +2453,7 @@
           [%| %mate-conflict -]
         =/  old=(map path lobe)                         ::  oldies but goodies
           %+  roll  ~(tap by (~(uni by old.dal) old.dob))
-          =<  .(old q.bas)
+          =<  .(old q.bob)
           |=  [[pax=path ~] old=(map path lobe)]
           (~(del by old) pax)
         =/  [hot=(map path lobe) lat=(map lobe blob)]   ::  new content
@@ -3855,7 +3907,7 @@
 ::
 ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
 =|                                                    ::  instrument state
-    $:  ver=%4                                        ::  vane version
+    $:  ver=%5                                        ::  vane version
         ruf=raft                                      ::  revision tree
     ==                                                ::
 |=  [our=ship now=@da eny=@uvJ ski=sley]              ::  current invocation
@@ -4114,9 +4166,15 @@
   ~!  [old=old new=*state-4]
   =?  old  ?=(%2 -.old)  (load-2-to-3 old)
   =?  old  ?=(%3 -.old)  (load-3-to-4 old)
-  ?>  ?=(%4 -.old)
+  =?  old  ?=(%4 -.old)  (load-4-to-5 old)
+  ?>  ?=(%5 -.old)
   ..^^$(ruf +.old)
   ::
+  ++  load-4-to-5
+    |=  =state-4
+    ^-  state-5
+    state-4(- %5, pun ~)
+  ::
   ++  load-3-to-4
     |=  =state-3
     ^-  state-4
@@ -4324,8 +4382,19 @@
       --
     --
   ::
-  +$  any-state  $%(state-4 state-3 state-2)
+  +$  any-state  $%(state-5 state-4 state-3 state-2)
-  +$  state-4  [%4 raft]
+  +$  state-5  [%5 raft]
+  +$  state-4
+    $:  %4
+        rom=room
+        hoy=(map ship rung)
+        ran=rang
+        mon=(map term beam)
+        hez=(unit duct)
+        cez=(map @ta crew)
+        pud=(unit [=desk =yoki])
+        pun=(list *)
+    ==
   +$  state-3
     $:  %3
         rom=room-3
@@ -4335,7 +4404,7 @@
         hez=(unit duct)
         cez=(map @ta crew)
         pud=(unit [=desk =yoki])
-        pun=(list move)
+        pun=(list *)
     ==
   +$  rung-3  rus=(map desk rede-3)
   +$  rede-3

pkg/arvo/sys/zuse.hoon

@@ -904,18 +904,17 @@
         lab/(map @tas @ud)                              ::  labels
     ==                                                  ::
   ++  germ                                              ::  merge style
-    $?  $init                                           ::  new desk
+    $?  %init                                           ::  new desk
-        $this                                           ::  ours with parents
+        %fine                                           ::  fast forward
-        $that                                           ::  hers with parents
+        %meet                                           ::  orthogonal files
-        $fine                                           ::  fast forward
+        %mate                                           ::  orthogonal changes
-        $meet                                           ::  orthogonal files
+        %meld                                           ::  force merge
-        $mate                                           ::  orthogonal changes
+        %only-this                                      ::  ours with parents
-        $meld                                           ::  force merge
+        %only-that                                      ::  hers with parents
-    ==                                                  ::
+        %take-this                                      ::  ours unless absent
-  ++  khan                                              ::
+        %take-that                                      ::  hers unless absent
-    $~  [~ ~]
+        %meet-this                                      ::  ours if conflict
-    $:  fil/(unit (unit cage))                          ::  see ++khan-to-soba
+        %meet-that                                      ::  hers if conflict
-        dir/(unit (map @ta (unit khan)))                ::
     ==                                                  ::
   ++  lobe  @uvI                                        ::  blob ref
   ++  maki  {p/@ta q/@ta r/@ta s/path}                  ::

pkg/arvo/ted/ph/breach-sudden.hoon

@@ -17,7 +17,8 @@
 ;<  ~              bind:m  (check-file-touched ~marbud %home file)
 ;<  ~              bind:m  (breach az ~bud)
 ;<  ~              bind:m  (real-ship az ~bud)
-;<  ~              bind:m  (dojo ~bud "|merge %home ~marbud %kids, =gem %this")
+;<  ~              bind:m
+  (dojo ~bud "|merge %home ~marbud %kids, =gem %only-this")
 ;<  file=@t        bind:m  (touch-file ~bud %kids %bar)
 ;<  file=@t        bind:m  (touch-file ~bud %kids %baz)
 ;<  ~              bind:m  (check-file-touched ~marbud %home file)

pkg/arvo/ted/ph/breach-sync.hoon

@@ -13,13 +13,14 @@
 ;<  ~              bind:m  (real-ship az ~marbud)
 ;<  file=@t        bind:m  (touch-file ~bud %kids %foo)
 ;<  ~              bind:m  (check-file-touched ~marbud %home file)
-::  Merge so that when we unify history with the %this merge later, we
+::  Merge so that when we unify history with the %only-this merge later, we
 ::  don't get a spurious conflict in %home
 ::
 ;<  ~              bind:m  (dojo ~marbud "|merge %kids our %home")
 ;<  ~              bind:m  (breach-and-hear az ~bud ~marbud)
 ;<  ~              bind:m  (real-ship az ~bud)
-;<  ~              bind:m  (dojo ~bud "|merge %kids ~marbud %kids, =gem %this")
+;<  ~              bind:m
+  (dojo ~bud "|merge %kids ~marbud %kids, =gem %only-this")
 ;<  file=@t        bind:m  (touch-file ~bud %kids %bar)
 ;<  file=@t        bind:m  (touch-file ~bud %kids %baz)
 ;<  ~              bind:m  (check-file-touched ~marbud %home file)