on master help sign in
Merge branch 'urbytes'
fde0252C. Guy Yarvin
main/blog/blockchain.mdnew file+620
@@ -0,0 +1,620 @@
1+---
2+date: ~2016.7.14
3+type: post
4+title: Why Urbit (probably) doesn't need a blockchain
5+author: Curtis Yarvin
6+layout: urbit,post
7+comments: true
8+navmode: navbar
9+navdpad: false
10+navselect: blog
11+navpath: /
12+navhome: /
13+navclass: urbit
14+---
15+<br /><br />
16+TLDR: Urbit (probably) doesn't need a blockchain, because the Urbit
17+address-space PKI is a special case of a consensus ledger (itself a
18+special case of Byzantine agreement).
19+
20+Special cases are often easier to solve than general problems. The
21+"New Jersey" or "worse-is-better" school of systems design emphasizes
22+solving any problem with the weakest possible tools.
23+
24+## The worse-is-better school
25+
26+Culturally, Urbit is best described as a functional stack built in the
27+"New Jersey" school of system software design, as described in Richard
28+Gabriel's famous ["worse is
29+better"](https://en.wikipedia.org/wiki/Worse_is_better) essay.
30+
31+There are two kinds of mindsets when approaching hard technical
32+problems. The "MIT" mindset, more typical among mathematicians and
33+scientists, wants to attack the problem as its simplest and broadest
34+case, producing the most general and powerful solution available. In
35+short: better-is-better.
36+
37+The "New Jersey" worse-is-better mentality, more typical among
38+programmers and engineers, wants to do everything possible to *not*
39+attack the problem. Can I produce a serviceable approximation to a
40+true solution? Do I actually have to solve the general problem, or
41+can I reduce it to an easier special case? Can I change the product
42+so it doesn't have this problem?
43+
44+Ideally, anyone building anything has both these mindsets in their
45+toolbox. But it's worth noting that (a) Jersey designs (meaning the
46+Bell Labs school that produced Unix, C, and Go) tend to beat MIT
47+designs in the marketplace, and (b) there are very few pure-functional
48+systems built with the true Jersey spirit.
49+
50+## Why Urbit (probably) doesn't need a blockchain
51+
52+Nakamoto consensus is an example of better-is-better. The blockchain
53+design is actually more powerful than it needs to be. It's not just a
54+trustless decentralized ledger; it's a trustless decentralized
55+database and (as Ethereum has realized; but this idea is clearly
56+present in Bitcoin's scripting language) a general-purpose, zero-trust
57+decentralized computer.
58+
59+Again, Urbit is not a general-purpose decentralized database or
60+consensus computer. It's a special case of the former, which is a
61+special case of the latter. Urbit is nothing but a digital title
62+chain for network addresses. Our belief is that the special case is
63+much easier than the general case, which is why Urbit (probably)
64+doesn't need a blockchain.
65+
66+The rest of this essay describes what makes this special case special,
67+and how we can use that to not need full Nakamoto consensus, or any
68+kind of general-purpose Byzantine agreement.
69+
70+## The double-sell problem
71+
72+Urbit has the same basic "double-spend" issue as Bitcoin. As Satoshi
73+puts it: "The problem of course is the payee can't verify that one of
74+the owners did not double-spend the coin." But Urbit is not a
75+currency, so we say "double-selling."
76+
77+"Double-selling" means signing over an Urbit address (*ship*) to two
78+different parties at the same time, presumably with the intent to sell
79+it to both and skip town with the loot.
80+
81+Again, the logical problem is the same. Bitcoin attacks it as a broad
82+theoretical breakthrough: better-is-better. Urbit attacks it as a
83+narrow special case: worse-is-better.
84+
85+Intuitively, Urbit's answer is that a real-estate transaction is
86+substantively a very different thing from a credit-card payment. No
87+real-estate agent ever has ever leaned over the desk, quickly grabbed
88+*both* the check and the keys, and run laughing out the door.
89+
90+To understand more rigorously how Urbit is a special case, let's take
91+a quick look at the Urbit address space and PKI.
92+
93+## The Urbit address space and PKI
94+
95+(NB: this description differs in minor ways from the running code.
96+Where they differ, the description is right. Urbit is not at present
97+secure: use for entertainment purpose only.)
98+
99+An Urbit ship is an unsigned integer that serves four purposes: a
100+cryptographic identity, a packet routing address, a stateful virtual
101+computer, and a unique, human-memorable name.
102+
103+Each ship is owned by a private key. To transfer a ship to a new
104+keypair, sign the new public key with the old private key. This
105+revokes the old key and activates the new key. Propagating this
106+update globally makes the revocation global. This is essentially the
107+digital title system a smart 12-year-old would design.
108+
109+### Ships and wills
110+
111+Ships are 128-bit numbers classified into tiers by bitwidth. An 8-bit
112+ship (a number less than 256) is a *galaxy*. A 16-bit ship is a
113+*star*. A 32-bit ship is a *planet*. A 64-bit ship is a *moon*. A
114+128-bit ship is a *comet*.
115+
116+The chain of signatures for each ship is called a *will*. Unlike most
117+certificate systems, there is no metadata in an Urbit will. Nothing is
118+signed but the public-key fingerprint. The length of the will is the
119+keypair version that owns the ship. This version is called its
120+*life*. Each new signed fingerprint in the will is a *deed*.
121+
122+Who gets to create a ship? Comets are fingerprints of a random
123+keypair. Moons, planets and stars are created by their *prefix*, the
124+ship whose bits are the bottom half of their formal width. For
125+instance, the star `0xbeef` is the prefix of the planet `0xdeadbeef`.
126+Galaxies have their fingerprints hardcoded in the official kernel.
127+
128+Ship numbers are not generally written in hex. They're written in
129+Hoon's `@p` syntax, a phonemic base-256. `@p` also permutes within
130+tiers, to superficially conceal prefix structure; for instance,
131+`0xdead.beef` is `~firbyr-napbes`, while `0xcafe.beef` is
132+`~nissep-sopnub`.
133+
134+### Parent hierarchy
135+
136+The prefix of a moon, planet, or star also becomes its *parent*. The
137+parent of a ship is its natural locus of federated services. The most
138+basic such service is to mediate P2P routing, as a so-called
139+"supernode." But the set of potential federated services is
140+unbounded.
141+
142+For moons, each new key update is signed by the parent. Planets and
143+stars sign their own updates, making them independent. To prevent
144+denial-of-service attacks by parents against children, stars and
145+planets can *exit*, choosing a new parent. An exit deed is a key
146+update also signed by the new parent.
147+
148+The full public cryptographic record of a ship is its *dossier*. The
149+dossier is the set of ship-will pairs, including all parents and
150+prefixes recursively, needed to validate the ship's current key. A
151+ship can process a dossier from an untrusted source.
152+
153+The full set of ship-will pairs a ship knows is its *cabinet*. So the
154+crucial question for PKI propagation is whether a submitted dossier is
155+accepted into a ship's cabinet.
156+
157+### Propagation race
158+
159+The rule is simple: any deeds in the dossier which are new (not
160+already in the cabinet) must not use keys superseded in the cabinet.
161+Otherwise revocation doesn't work. But revocation affects only
162+dossiers received in the future; nothing already in the cabinet can be
163+revoked.
164+
165+Intuitively, this algorithm creates a propagation race. Life `2` of
166+`~firbyr-napbes` revokes life `1` of `~firbyr-napbes`, so propagating
167+life `2` to the whole network ensures that anyone who steals the
168+private key for life `1` can't use it in any way -- whether to launch
169+children, or to sign a new, bogus life `2`.
170+
171+## Solving the special case
172+
173+So how can we solve the double-sell problem, for the special case of
174+the system designed above?
175+
176+### Double-spend relaying in Bitcoin
177+
178+The first thing we need to notice about our special case is that *even
179+Bitcoin*, not considered as a general consensus problem but just as a
180+decentralized payment rail, isn't that far from *not* needing a
181+blockchain.
182+
183+As the Karame ["Two Bitcoins at the Price of
184+One"](http://eprint.iacr.org/2012/248.pdf) paper points out:
185+
186+> An efficient countermeasure to combat double-spending on fast
187+Bitcoin payments would be for Bitcoin peers to propagate alerts
188+whenever they receive two or more transactions that share common
189+inputs and different outputs.
190+
191+"Fast Bitcoin payments" means zero-confirmation, ie, Bitcoin without
192+the blockchain. These double-spend alerts have never been integrated
193+into Bitcoin Core, despite frequent proposals. Given that Bitcoin
194+*does* have a blockchain, it seems quite sensible for purists to shun
195+hacks that make zero-conf transactions somewhat more secure. Also,
196+double-spend alerts in Bitcoin are a DoS vulnerability; Bitcoin has
197+fees to control the cost of propagating transactions, but who pays for
198+an alert?
199+
200+In principle, however, we could imagine a version of Bitcoin without a
201+blockchain, in which the mempool was the ledger, and there was an
202+alerting and/or conflict model to detect and resolve double spends.
203+An accepted transaction is simply one that propagates to one or more
204+randomly chosen observer nodes.
205+
206+This design, for the problem that Bitcoin is solving, intuitively
207+feels weak and a little hacky. The Karame paper is not actually
208+advocating for removing the blockchain from Bitcoin. At the same
209+time, low-value zero-confirmation payments are commonly used.
210+
211+One way to think about security is to flip the tables and think of the
212+defender as the attacker, and vice versa. Just imagine you have some
213+moral right to double spend. The rest of the world is unfairly trying
214+to stop you. Can you defend yourself?
215+
216+In the context of a gossip network that shares all the things, a
217+procedure that involves telling one person one thing, and another
218+person another conflicting thing, and not getting "hacked" by the
219+unfair world which believes there should be only one truth, is
220+inherently quite fragile. And where the attack is fragile, the
221+defense is robust.
222+
223+Most of the security in any digital title system comes from the PKI
224+itself. The double-spend problem, the general issue of Byzantine
225+failures in information propagation, is already a corner case. So
226+perhaps we can solve it as a corner case.
227+
228+Again: *even Bitcoin* isn't that far from being able to build a
229+practical decentralized ledger without a blockchain. The same gossip
230+model that broadcasts nonconflicting transactions is not at all
231+incapable of reacting to conflicting transactions. We're not really
232+recommending that Bitcoin ditch the blockchain, but this gives us hope
233+with our fundamentally easier problem.
234+
235+## Various specific specialnesses
236+
237+What are the differences between the Urbit double-sell problem and the
238+Bitcoin double-spend problem? Is our special case really, truly,
239+special?
240+
241+In worse-is-better mode, we want every special-case advantage we can
242+get. Every difference we find is one we can use as a leverage point.
243+
244+### Minimal transaction interdependency
245+
246+The first major technical difference between the Urbit title history
247+and the Bitcoin blockchain is that Bitcoin and its relatives are
248+designed to secure a complete, interdependent, collective transaction
249+history. Except for the parent-child relationship, Urbit wills are
250+independent of each other.
251+
252+Bitcoin's use of UTXOs rather than balances reduces interdependency
253+somewhat. But the validity of each transaction is still dependent on
254+all the transactions that created its inputs. (Ethereum drops the
255+UTXO model, whose value in reducing dependencies may not match its
256+cost in complexity.)
257+
258+Urbit is not a payment system or a shared ledger. Bitcoins are
259+divisible, but Urbit ships aren't. An Urbit will containing its
260+parent chain is self-validating and has no dependencies. There is no
261+collective history. There is no Merkle tree. There is no chain, with
262+or without blocks.
263+
264+This simplification doesn't eliminate the double-sell problem in
265+theory, but makes it much more tractable in practice. The set of past
266+transactions (key transfers) that a new transaction could conflict
267+with is much smaller.
268+
269+### Trust hierarchy and governance structure
270+
271+The second major technical difference is that Bitcoin is a flat,
272+trustless, fully decentralized system. Urbit has a hierarchical,
273+polycentric governance system which is a trust hierarchy.
274+
275+Now and for quite some time, urbits default to automatically accepting
276+OS updates from their parent, making this trust effectively absolute.
277+This is not a novelty -- all "evergreen" browsers now work this way.
278+You trust your browser quite a bit, which means you trust your browser
279+vendor.
280+
281+And since few Bitcoin node operators code their own daemons, the same
282+dependency on a source distribution exists in Bitcoin. It's informal
283+and off-chain, though, which is not really a good thing. (Urbit is
284+like if Bitcoin Core hotpatched itself automatically by publishing
285+source-code diffs on the blockchain.)
286+
287+To be on the safe side, we should think of Urbit as a centralized
288+system until the OS update that turns off automatic updates is
289+shipped. This of course requires a certain measure of perfection,
290+which cannot in any way be predicted.
291+
292+After Arvo proper becomes "deciduous" (apps will of course always be
293+remotely hotpatched), child-to-parent trust should diminish to the
294+power of the parent to deny service to the child. This will always be
295+present without some major redesign.
296+
297+But it's opposed by the equal and opposite power of the child to
298+switch to a different parent, creating a competitive market in
299+governance. This should deliver service at the highest quality and
300+lowest price, fortifying trust with incentives. The power of a
301+restaurant owner to deny you service does not give the food industry
302+the power to starve you to death.
303+
304+And if no parent wants your business, with 2^16 stars who could accept
305+it, you probably shouldn't be on the network at all. Urbit, like a
306+real society, is designed to be a network of incentives that holds
307+everyone in productive tension. We want everyone to be free, and no
308+one to be unaccountable.
309+
310+For example: a parent which extends its will also needs to send
311+updated signatures to its children and former children. Either will
312+otherwise be unable to propagate their dossiers to ships whose
313+cabinets have the updated parent will, *even if some other parent
314+adopts them*.
315+
316+But Urbit is a collectively governed and updatable system. A star or
317+galaxy which denies service to its own children will (a) get a bad
318+reputation, (b) drive its children to other parents, and (c) force the
319+network to hardcode a ship-specific exception into the propagation
320+algorithm, overriding stale-key revocation.
321+
322+In theory, a parent can extort its children by denying service. In
323+practice, since Urbit is self-governing, not ungoverned, and its
324+government can right this wrong with a source-code patch, the
325+extortion won't work. So it won't happen. So the act of government
326+won't be necessary, and won't happen either.
327+
328+Of course a perfectly decentralized system would be freer. But worse,
329+as usual, is better. And since we are paying for this worseness,
330+imperfect decentralization, we'd like to get all the mileage out of it
331+that we can.
332+
333+Again, Urbit, not being a trustless system, does not have to solve the
334+trustless consensus problem -- although we still very much want to
335+minimize and bound the delegation of trust. This is an obvious
336+relaxation of Bitcoin's hard theoretical problem.
337+
338+### Routing: security through topology
339+
340+Please don't laugh. Security through routing topology isn't a joke --
341+unless firewalls are a joke. Cisco sells too many firewalls for them
342+to be a joke.
343+
344+In Bitcoin we may have all sorts of reasons to ask what ship holds
345+what coins. In Urbit, when we ask what key controls an ship, we are
346+either sending or receiving a message. Is our outgoing message safely
347+sealed to the current owner? Is our incoming message properly signed
348+by the current owner?
349+
350+When the parent also forwards all packets until a P2P link is
351+established (Urbit forwarding resolves to STUN if either party has
352+full cone NAT, TURN if not), we can design in all kinds of
353+interactions between routing and certificate management.
354+
355+In particular, a parent which is a single source of truth in some
356+sense is a very attractive design element. Again, we're paying for
357+our imperfect decentralization. We might as well use it.
358+
359+For example: I want to send a packet to ship X. Two parties, A and B,
360+are competing for ownership of X. A is the rightful owner. I am
361+secure against B if either of these conditions are true: my packet can
362+only be decoded by A, or my packet is not sent to B. Both would be
363+ideal, of course. Either will do.
364+
365+### Latency and friction tolerance
366+
367+Confirmed Bitcoin payments aren't especially low-latency by payment
368+standards. They're reasonably low-friction by payment standards. But
369+fees are certainly a thing.
370+
371+Urbit is digital real estate, not digital money. When we look at the
372+standards of latency and friction in actual real estate -- where a
373+transaction that takes a week is super-fast, a 5% payment overhead is
374+normal, and a third-party intermediary is standard -- we see a much,
375+much weaker set of requirements.
376+
377+Obviously a week and a 5% overhead are incredibly sucky levels of
378+latency and friction. The point is that even if this is the only
379+level of performance Urbit can achieve, that's not necessarily the end
380+of the world for Urbit.
381+
382+For instance: if we wait a week for a double-sell alert, and we don't
383+see one, either the gossip network is totally hosed (to a point we'd
384+surely find out about), or there wasn't a double-sell. This stuff is
385+just not that hard.
386+
387+(One little-made point about Bitcoin, by the way, is that Bitcoin
388+enables trust-free *payments*, but not trust-free *exchanges*. The
389+seller still needs to deliver the goods either before or after the
390+payment; in either case, there is trust. Only Ethereum can solve this
391+problem -- and only for digital exchanges.)
392+
393+## We can afford to be patient
394+
395+With a better-is-better, general-purpose solution, one size fits all
396+until the end of time. With a worse-is-better, special-case solution,
397+the details of that case are critical.
398+
399+But we actually don't know these details. One rule of the
400+worse-is-better school: never solve a problem before you have to. You
401+don't know all the details, and you need those details. (Of course,
402+with the better-is-better approach, the details don't matter at all.)
403+
404+Is there such a thing as a transitive adjective? Grammarians are
405+unclear. But technically, the adjective "secure" is always an error.
406+Nothing is "secure." "Secure against X" is the correct usage, where X
407+is some threat model.
408+
409+Urbit is an insecure test network with no sensitive data on it. This
410+will remain true for a while. No Urbit ship has ever been transferred
411+cryptographically. This will also remain the status quo for quite a
412+while.
413+
414+We *don't know the threat model*. We *can't solve the problem*. It's
415+as simple as that. The best Urbit can do, at the present time, is map
416+out the ballpark of solutions.
417+
418+And we have time to proceed patiently. One way to think about the
419+timing of cryptographic decentralization is to look at the DNS. It's
420+clear that now, a decentralized DNS with cryptographic domain-name
421+titles would be extremely desirable. It was never designed to happen.
422+It will probably never happen.
423+
424+But if we look at when cryptographic DNS with allodial title *should*
425+have happened, it's probably sometime in the mid-90s. Ie, after over a
426+decade of production use of the Domain Name System. In short: there's
427+no need to hurry here.
428+
429+## A map of the solution space
430+
431+But we still need a map of the worse-is-better ballpark. Let's look
432+at some options, from least secure to most.
433+
434+The essential question we can't answer, without a better sense of the
435+threat model, is how aggressively we need to propagate wills. We
436+especially are not sure about the threat model of malicious stars
437+(16-bit ships) and/or galaxies (8-bit ships).
438+
439+We need to know what kinds of double-sell attacks are plausible. Since
440+we don't know this yet, we need not one propagation design, but a
441+range of designs -- stretching all the way up to a full blockchain.
442+
443+### Acceptance model
444+
445+What doesn't change is the will *acceptance* model. A ship will
446+always accept a new will if (a) the new will extends the old will, (b)
447+the parent signature on the initial public key is at least as recent
448+as the ship's current version of that parent.
449+
450+The parent clause implies that when a parent extends its will, it
451+needs to send updated initial signatures to all the children it's
452+launched. Otherwise, their own future updates will not be accepted by
453+ships that have seen the parent's update.
454+
455+This is not a big deal in the world of parent responsibilities. But
456+like many such responsibilities, it lets a badly operated parent deny
457+service to its children.
458+
459+A ship accepts all wills, new or updated, and stores them permanently.
460+If this is abused trivially as a DoS vector, Urbit will mitigate it by
461+the usual technique of degrading service to non-scarce ship tiers.
462+Scarcity rules.
463+
464+### Level 0: passive propagation only
465+
466+The Ames protocol can piggyback a will on any packet. Each ship also
467+permanently records which other ships hold which version of its own
468+will. An update is automatically sent when needed.
469+
470+With level 0 propagation, all key exchanges in Ames are passive and
471+invisible to the programmer. When the sender has no will or symmetric
472+key for the receiver, the packet is signed, bundled with the will, and
473+sent as cleartext.
474+
475+Signed cleartext? If you have no will for a peer, you've never talked
476+to this ship before. Your first words to a stranger probably won't be
477+super secret. If this isn't true, the programmer has to send an
478+innocuous message first and wait for it to be acknowledged. Kicking a
479+rare corner case up to the programmer is a classic worse-is-better
480+move.
481+
482+With level 0 propagation only (the current implementation, because we
483+don't even care about key updates yet), an updated does not
484+effectively revoke the old key. If Alice updates to a new key,
485+Mallory steals the old key, then Mallory talks to Bob before Alice
486+does, Mallory can impersonate Alice.
487+
488+So not only does level 0 not solve double-sell, it's also quite
489+insecure. But it's still fine for now (because Urbit isn't really
490+decentralized yet).
491+
492+### Level 1: neighbor-only propagation with parent filtering
493+
494+The list of other ships which hold your will, essentially the list of
495+all ships you've talked to, is your list of *neighbors*. You
496+essentially have permanent, if perhaps very sporadic, conversations
497+with all your cryptographic neighbors. Symmetric keys time out fast,
498+but wills are held forever.
499+
500+By proactively pinging all your neighbors when you extend your will,
501+you prevent anyone who finds the old private key (ideally you update
502+when your old key is stale, not when it's actually been *stolen*) from
503+jumping into these conversations.
504+
505+What about non-neighbors? We add a second feature: a parent will only
506+forward a packet if its sender matches the current will. So updating
507+your parent (who is always your neighbor) transforms it
508+
509+There are two cases: where Alice's parent Trent is also Bob's parent,
510+and where Bob has a separate parent Walter. In the first case, the
511+expected packet flow is that Alice will send the packet to Trent, who
512+sends it to Bob. In the second case, Trent sends it to Walter, who
513+sends it to Bob.
514+
515+In either flow, Bob or Walter is sure that the packet came through
516+Trent. Since Trent is Alice's parent and hence Alice's neighbor, and
517+hence is updated, Trent will forward no packets from Mallory. So
518+Mallory has no way to reach Bob.
519+
520+Non-neighborly packets not matching this flow can't be accepted.
521+Suppose Mallory gets Bob's public IP address directly? If Bob gets a
522+non-forwarded packet marked "Alice," but Alice is not his neighbor,
523+that's super weird. He must reject the packet.
524+
525+Level 1 is nice because it provides a reasonable level of security,
526+but it's still extremely simple. Not only does it not use a
527+blockchain, it doesn't even need a gossip flood!
528+
529+And level 1 propagation also does a competent worse-is-better job on
530+the double-sell problem. Trent simply becomes the single source of
531+truth. The buyer of Alice needs to check that Trent, her parent, has
532+Alice's latest will from the buyer's perspective. If Alice's seller
533+propagates a different fork to other ships, this damages Alice's
534+ability to communicate, but produces no obvious gain for the seller.
535+
536+Level 1 puts a lot of security weight on the parent. If Trent
537+misbehaves, for example by not filtering Mallory's packets, Alice can
538+be impersonated. And its solution to the double-sell problem is
539+adequate, but not ideal. Level 1 is the worse-is-better sweet spot
540+for a young Urbit, but will probably give way to level 2.
541+
542+### Level 2: broadcast all wills
543+
544+We can't say definitively that level 1 will be inadequate. But
545+intuitively, it feels slightly weak. We will probably need a gossip
546+protocol.
547+
548+The only ships that can be transferred are galaxies, stars and
549+planets; if all updates of all wills of all these major ships are
550+broadcast, we hardly have a scaling problem like Bitcoin's. Even if
551+changing the lock on your front door counted as a real-estate
552+transaction, there are a lot fewer real-estate transactions than
553+credit-card payments.
554+
555+Moreover, conflict detection (double-sell alerts) is a natural part of
556+any gossip protocol. Gossip should be between stars, not directly
557+between planets; horizontal gossip, vertical distribution.
558+Star-to-planet distribution may still be limited.
559+
560+Especially since stars are somewhat trusted, ganking a gossip protocol
561+with random update edges is incredibly hard and requires a pretty
562+preposterous conspiracy. And flooding all planet wills does a lot to
563+keep stars honest. It would be incredibly stupid for a star to try a
564+double-sell attack against its own planets, but code is always more
565+powerful than mere incentives.
566+
567+Finally, the exit model isn't fully baked yet, but it will probably
568+have more aggressive propagation needs. What's clear is that an exit
569+statement is another type of will entry, involving a signature both
570+from the previous key and the new parent. You can only escape to
571+another ship of your parent's class that agrees to adopt you.
572+
573+In general, because escaping need to work for routing (without the
574+cooperation of the former parent), wills containing an exit statement
575+(a planet moving to a new star, or a star moving to a new galaxy)
576+exit) probably need to be flooded globally among all 8-bit galaxies
577+and 16-bit stars. Exits should be rare, so they shouldn't present
578+much of a scaling problem.
579+
580+### Level 3: full blockchain
581+
582+Obviously a blockchain can solve Urbit's problem. Probably the right
583+thing to do would be to use the Bitcoin blockchain, and confirm the
584+current Urbit key version with a "colored coin" of some kind. (This
585+should not be confused with the orthogonal idea of representing Urbit
586+*invitations* as colored coins.)
587+
588+This involves making all Urbit ships into Bitcoin SPV nodes at the
589+least. Not the worst fate in the world, but also a cost.
590+Alternatively, Urbit could have its own "altcoin" blockchain, with
591+actual Urbit-specific mining.
592+
593+Again, given that level 2 seems like it should work quite well, it's
594+hard to see why level 3 propagation would be needed. But we can't
595+really know at this point. There's a reason Urbit can update itself
596+over the air. That's why we say: Urbit (probably) doesn't need a
597+blockchain. If it does, we can retrofit it.
598+
599+## Retrospective note
600+
601+This essay has presented the Urbit cryptographic title as a low-power,
602+stripped-down, worse-is-better variation of Nakamoto consensus.
603+That's clearly the right way to introduce it in 2016.
604+
605+Actually, though, this part of Urbit was designed before the Bitcoin
606+paper was published. There were other academic approaches to
607+Byzantine consensus, but certainly none validated on a large scale as
608+Nakamoto consensus has been. It never seemed appropriate to treat
609+cryptographic titles to network addresses as a case of generalized
610+Byzantine consensus, and without the existence of Bitcoin there was no
611+temptation in this direction.
612+
613+In a sense the secret of PKIs is that PKIs are easy. What's hard is
614+adapting an existing identity and authorization structure into a PKI.
615+If you build the simplest possible PKI, then force the human processes
616+to fit themselves to the protocol instead of vice versa, you have a
617+much easier job. Similarly, if you are not adding PKI to an existing
618+network, but building the network and the PKI as one system, many
619+simplifications are possible. This is once again the classic
620+worse-is-better method.
main/blog/byte-0.mdnew file+267
@@ -0,0 +1,267 @@
1+---
2+date: ~2016.7.16
3+type: post
4+title: Urbyte 0: setup, nouns
5+author: The Urbit Team
6+layout: urbit,post
7+comments: true
8+hide: true
9+navmode: navbar
10+navdpad: false
11+navselect: blog
12+navpath: /
13+navhome: /
14+navclass: urbit
15+---
16+<br /><br />
17+This is the first post in our new Urbyte series.
18+
19+The Urbyte concept is that you, an ordinary programmer, not a
20+kernel hacker or a Haskell magician (not that there's anything
21+wrong with kernel hackers or Haskell magicians) can learn Hoon,
22+Urbit's programming language.
23+
24+We're going to test this daring idea by teaching you Hoon, in
25+easily digestible, byte-size chunks. Each post in the "Urbytes"
26+series is designed to take about 15 minutes to consume. (Maybe
27+10, if you're a kernel hacker or a Haskell magician.)
28+
29+We'll assume no knowledge of Urbit except for previous lessons.
30+Ideally you have some basic familiarity with programming and can
31+operate the Unix command line.
32+
33+We strongly encourage you to type in all the examples in each
34+Urbyte. Urbytes are short, but not meant to be skimmed.
35+
36+## Install
37+
38+First, [install Urbit](http://urbit.org/docs/using/install/) on any
39+Mac or Unix machine. On Windows, make a virtual Linux machine
40+using VirtualBox or a similar tool.
41+
42+While you're downloading, read about Urbit ships:
43+
44+## Ships
45+
46+In the Urbyte exercises you'll be running what we call a "fake
47+`~zod`." A string starting with `~`, like `~zod`, is what Urbit
48+calls a *ship*.
49+
50+A ship is one number which means four things: a network address,
51+a cryptographic identity, a memorable name, and a single virtual
52+computer. We write this number in a base-256 format, where each
53+digit is a syllable.
54+
55+`~zod` is ship zero, making it a very important ship indeed.
56+Your "fakezod" thinks it's `~zod`, but no one can talk to it.
57+
58+This is not the true Urbit experience, of course. But it's
59+ideal if all you want to do is play around with Hoon. If you
60+have a real urbit ship, of course, you can use that instead.
61+
62+## Create and restart
63+
64+To create a fakezod, run:
65+
66+```
67+$ urbit -FI ~zod -c myzod
68+```
69+
70+This will take about 30 seconds and spin out a bunch of boot
71+messages. At the end, you'll see:
72+
73+```
74+ames: on localhost, UDP 31337.
75+http: live (insecure, public) on 8080
76+http: live ("secure", public) on 8443
77+http: live (insecure, loopback) on 12321
78+~zod:dojo>
79+```
80+
81+To make sure everything is working, type `(add 2 2)` at the
82+prompt, then hit return. Your screen now shows:
83+
84+```
85+ames: on localhost, UDP 31337.
86+http: live (insecure, public) on 8080
87+http: live ("secure", public) on 8443
88+http: live (insecure, loopback) on 12321
89+> (add 2 2)
90+4
91+~zod:dojo>
92+```
93+
94+That's super awesome! You made your first noun! We'll make a
95+few more in a second. First, quit Urbit with `^D`:
96+
97+```
98+> (add 2 2)
99+4
100+~zod:dojo>
101+$
102+```
103+
104+You now have an Urbit image, or *pier*, in the `myzod` directory.
105+Right now the only thing in your pier is Urbit's system files:
106+
107+```
108+$ ls -a myzod
109+./ ../ .urb/
110+```
111+
112+Restarting uses almost the same command, but without `-c`. There
113+are also fewer startup messages.
114+
115+```
116+$ urbit -FI ~zod myzod
117+[...]
118+ames: on localhost, UDP 31337.
119+http: live (insecure, public) on 8080
120+http: live ("secure", public) on 8443
121+http: live (insecure, loopback) on 12321
122+~zod:dojo>
123+```
124+
125+Note also that Urbit is an ACID database. You can stop it
126+politely with `^D`, abruptly with `^Z`, even violently with `kill
127+-9`. None of these should cause Urbit to lose data. If you do
128+manage to corrupt a pier, it's a bug; please let us know.
129+
130+## Let's make a noun
131+
132+We already made a noun. Let's make another. (We won't show the
133+`~zod:dojo> ` prompt in future, just the echoed command.)
134+
135+```
136+~zod:dojo> 42
137+```
138+
139+You'll see:
140+
141+```
142+> 42
143+42
144+~zod:dojo>
145+```
146+
147+## Understanding nouns
148+
149+A value in Urbit is called a *noun*. A noun is either an *atom*
150+(an unsigned integer of any size) or a *cell* (an ordered pair of
151+any two nouns).
152+
153+In a sentence, a noun is a binary tree whose leaves are numbers.
154+If you know Lisp, a noun is a minimalist S-expression. A noun
155+can't have cycles (pointer loops). An implementation can use
156+dags (shared subtrees), but the programmer can't detect them
157+(there's no pointer equality test).
158+
159+The absence of graph cycles makes nouns much easier to save in
160+persistent storage, send over the network, manage memory for
161+(Urbit has no tracing garbage collector), etc.
162+
163+## Let's make some atoms
164+
165+So one kind of atom is a number. And the syntax (or at least one
166+syntax) for a number is a decimal, like `42`. Seems legit.
167+
168+But just to be sure we understand this decimal thing, let's make
169+some atoms:
170+
171+```
172+> 4
173+4
174+```
175+
176+Seems pretty straightforward.
177+
178+```
179+> 42
180+42
181+```
182+
183+Why not try it again to make sure?
184+
185+```
186+> 420
187+420
188+```
189+
190+Can we add another zero?
191+
192+```
193+~zod:dojo> 420
194+```
195+
196+Wat? When you tried to type that next zero and turn `420` into
197+`4200`, Urbit actually *deleted the `0` and beeped at you.*
198+
199+But isn't an atom an unsigned integer of any size? It is.
200+But the syntax is actually `4.200`:
201+
202+```
203+> 4.200
204+4.200
205+> (mul 42 100)
206+4.200
207+```
208+
209+Just think of it as "4,200", written the German way with a dot.
210+
211+Why? Long decimals are unreadable. Human beings know this, so
212+we group them in threes. For some historical reason, programming
213+languages never copied this marvelous usability innovation.
214+
215+English notation for decimals is more common than German
216+notation. Unfortunately, dot is URL-safe and comma isn't, and
217+it's nice to have a regular syntax for atoms that's URL-safe.
218+
219+As for why Urbit deleted your zero: it parses your command line
220+as you type, and rejects any characters after the parser stops.
221+
222+## Let's make some cells
223+
224+There's not much mystery about cells. A cell has a head and a
225+tail, and puts brackets around them:
226+
227+```
228+> [42 420]
229+[42 420]
230+```
231+
232+`42` is the *head* and `420` is the *tail* of the cell. Cell
233+syntax nests to the right:
234+
235+```
236+> [4 [42 420]]
237+[4 42 420]
238+
239+> [4 42 420]
240+[4 42 420]
241+
242+> [[4 42] 420]
243+[[4 42] 420]
244+```
245+
246+If you look at these nouns as binary trees, you see
247+
248+```
249+ [4 [42 420]] [[4 42] 420]
250+ . .
251+ / \ / \
252+ 4 . . 420
253+ / \ / \
254+ 42 420 4 42
255+```
256+
257+Each number is an atom; each dot is a cell.
258+
259+## Questions and/or exercises
260+
261+And we're done with our very first Urbyte! Here's a question to
262+think about. These exercises are never required, but they may
263+be fun and/or interesting.
264+
265+The question today is: what is the right way to represent common
266+data structures as nouns? A linked list? A string? A *signed*
267+integer? A table or associative array?
main/blog/byte-1.mdnew file+302
@@ -0,0 +1,302 @@
1+---
2+date: ~2016.7.14
3+type: post
4+title: Urbyte 1: atoms, auras, types
5+author: The Urbit Team
6+layout: urbit,post
7+comments: true
8+hide: true
9+navmode: navbar
10+navdpad: false
11+navselect: blog
12+navpath: /
13+navhome: /
14+navclass: urbit
15+---
16+<br /><br />
17+Thanks for reading Urbyte 0! In Urbyte 1, we'll look a little
18+more deeply at atoms and other simple noun types.
19+
20+## The atom question
21+
22+It seems limiting to have only one kind of atom, the unsigned
23+decimal. How are signed integers represented? Symbols?
24+Floating-point values? Dates? IP addresses? Even just hex?
25+
26+The Lisp S-expression, with cons cells and atoms, is very similar
27+to Hoon's nouns. But languages in the Lisp family mark each atom
28+with a dynamic tag that defines the atom's meaning. These tags
29+are effectively dynamic types.
30+
31+Hoon also tracks the meaning of atoms. But Hoon is statically
32+type, so an atom description is obviously a type. An atomic type
33+in Hoon is called an *aura*. The aura syntax is just an ASCII
34+string starting with `@`, like `@ud` for unsigned decimal.
35+
36+## Examples
37+
38+With the dojo command `? expression`, you tell the dojo to print
39+both the product of the expression, and the product's type.
40+
41+Let's type in some aura examples:
42+
43+### Unsigned integers
44+
45+```
46+> 42
47+42
48+
49+> ? 42
50+ @ud
51+42
52+
53+> ? 0x42
54+ @ux
55+0x42
56+
57+> ? `@ud`0x42
58+ @ud
59+66
60+
61+> ? (add 0x42 42)
62+ @
63+108
64+```
65+
66+So decimal is aura `@ud`, hexadecimal is `@ux`. Note the `@ud`
67+cast syntax, which lets us change the type of an atom, and
68+converting hex `0x42` to a decimal.
69+
70+An aura is soft type. The programmer can always insist on how
71+to interpret an atom. Hoon doesn't have "dependent types"; and
72+can't enforce any constraints the aura puts on the atom's value.
73+
74+### Signed integers
75+
76+```
77+> -7
78+-7
79+
80+> ? -7
81+ @sd
82+
83+> --7
84+--7
85+
86+> ? --7
87+ @sd
88+--7
89+
90+> ? (sum:si -7 --7)
91+ @s
92+--0
93+
94+> `@ud`-7
95+13
96+
97+> `@ud`--7
98+14
99+```
100+
101+`--7` means "positive 7." `+7` might have been better, but `+`
102+is not URL-safe.
103+
104+With unlimited atom width, traditional sign extension makes no
105+sense. The sign bit is the low bit. Even atoms are positive,
106+odd atoms are negative.
107+
108+Hoon needs `--` to distinguish positive signed numbers because
109+Hoon's type system doesn't feed back into its parser. The
110+expression itself needs to control whether it's `@sd` or `@ud`.
111+
112+And Hoon has no overloading or type detection; we need to use a
113+different function to add signed integers. We'll always choose
114+slightly clunkier syntax in exchange for much simpler semantics.
115+
116+### Symbols
117+
118+```
119+> ?
120+ @t
121+'Ürbit'
122+
123+> ? `@ux`'Ürbit'
124+ @ux
125+0x7469.6272.9cc3
126+
127+> ? `@ux`'foo'
128+ @ux
129+0x6f.6f66
130+```
131+
132+The `@t` aura is UTF-8 text, with the first byte low. Again,
133+Hoon doesn't actually have the power to check statically that
134+your UTF-8 is valid. Auras don't tell you what an atom is, just
135+what you want it to be.
136+
137+### Dates, floats, IP addresses...
138+
139+```
140+> ? now
141+ @da
142+~2016.7.11..01.32.04..2c0d
143+
144+> ? @~h6
145+ @dr
146+~h6
147+
148+> `@ux`~h6
149+ @ux
150+0x5460.0000.0000.0000.0000
151+
152+> ? (add ~h6 now)
153+ @
154+170.141.184.502.236.089.414.812.327.632.537.911.296
155+
156+> `@da`(add ~h6 now)
157+~2016.7.11..07.32.28..842f
158+
159+> ? .3.14
160+ @rs
161+.3.14
162+
163+> ? .127.0.0.1
164+ @if
165+.127.0.0.1
166+```
167+
168+`@da` is a 128-bit absolute date (64 bits for fractions of a
169+second, 64 bits for seconds); `@dr` is a relative date; `@rs` is
170+a single-precision IEEE real; `@if` is an IPv4 address. This is
171+not all Hoon's aura syntax, but it's a good range of examples.
172+
173+`\`@da\~(add ~h6 now)` is a good example of the use and limits of
174+the aura system. Unlike the signed-integer sum, offsetting a
175+date can use plain atomic addition. But the product of `add` is
176+a flavorless atom with no aura, so we have to cast it back to a
177+date aura.
178+
179+## Auras explained
180+
181+Does an aura represent the units of the atom, its syntax, its
182+semantics, its constraints?
183+
184+It can mean any or all of these. Programmers can even use their
185+own aura strings which nothing in Hoon recognizes. The set of
186+syntaxes that Hoon knows how to parse and print is fixed, but
187+there are plenty of reasons to introduce user-defined auras.
188+
189+If your program uses both fortnights and furlongs, the aura
190+system will not parse a fortnight syntax or print furlongs in the
191+dojo. But it will keep you from accidentally confusing the two,
192+which is probably most of what you want.
193+
194+Hoon does interpret the string in one way: auras specialize to
195+the right. For example, `@u` is any unsigned integer; `@ux` is
196+an unsigned integer that likes to be printed as hex.
197+
198+Hoon's typechecker lets you go up or down the specialization
199+ladder, but complains if you try to go across. For example,
200+`@u` can silently convert into `@ux` or `@ux` into `@u`, but
201+turning `@ud` into `@ux`, `@ux` into `@da`, etc, requires an
202+explicit cast.
203+
204+## Constants, symbols, loobeans and null
205+
206+An atom type isn't just an aura. There's another choice: the
207+type can be *general* and contain all atoms (like the types we've
208+seen above), or *constant* and contain just one atom.
209+
210+A type describes a set of values. While auras may informally
211+describe a format for which not all atoms are valid, any atom can
212+be cast into the aura whether valid or invalid.
213+
214+In a constant, the type contains just one atom. Hoon actually
215+enforces this restriction.
216+
217+The regular syntax for an atomic constant is just to put `%` on
218+front of the constant. Like:
219+
220+```
221+> ? 42
222+ @ud
223+42
224+
225+> ? %42
226+ $42
227+42
228+
229+> ? %~h6
230+ $~h6
231+~h6
232+```
233+
234+The `$` on the type indicates the constant. Note that constants
235+still have auras; `~h6` (6 hours) is still a `@dr`.
236+
237+## Common irregulars
238+
239+There are a few special forms worth learning early: symbols,
240+null, and loobeans:
241+
242+```
243+> ? %foo
244+ $foo
245+%foo
246+
247+> `@ux`%foo
248+0x6f.6f66
249+
250+> .n
251+.n
252+
253+> |
254+.n
255+
256+> .y
257+.y
258+
259+> &
260+.y
261+
262+> `@ud`&
263+0
264+
265+> ~
266+~
267+
268+> `@ud`~
269+0
270+```
271+
272+`%symbol` is a constant with aura `@tas` (text, ASCII, symbol).
273+The symbol rules, which are enforced by the Hoon parser but not
274+(as we've discussed) the type system: a symbol is lowercase with
275+infix hyphens ("kebab-case").
276+
277+Loobeans are not true and false, but yes (`.y` or `&`) and no
278+(`.n` or `|`). This is because *yes is zero and no is one*.
279+This makes sense at a certain mathematical level, though it
280+was probably a mistake.
281+
282+`~` is Hoon's nil: constant `0`, aura `@n`.
283+
284+## Cells and types
285+
286+Naturally, all these atoms fit neatly into cells, and retain
287+their types within the cells.
288+
289+```
290+> ? [~h6 [.3.14 %foo] 0xdead.beef]
291+ {@dr {@rs $foo} @ux}
292+[~h6 [.3.14 %foo] 0xdead.beef]
293+
294+> `*`[~h6 [.3.14 %foo] 0xdead.beef]
295+[398.449.671.992.126.314.905.600 [1.078.523.331 7.303.014] 3.735.928.559]
296+```
297+
298+Another fun trick is to cast any noun to the `*` type, ie,
299+generic noun. We then know nothing about the noun, so we print
300+all its atoms as decimals.
301+
302+That's all for today! Please wait for the next fun Urbyte.
main/blog/byte-2.mdnew file+70
@@ -0,0 +1,70 @@
1+---
2+date: ~2016.7.14
3+type: post
4+title: Urbyte 2: subject-oriented programming
5+author: Curtis Yarvin
6+layout: urbit,post
7+comments: true
8+hide: true
9+navmode: navbar
10+navdpad: false
11+navselect: blog
12+navpath: /
13+navhome: /
14+navclass: urbit
15+---
16+<br /><br />
17+
18+You're playing in the Urbit REPL/shell (`:dojo`). (If you
19+accidentally press `^X`, it will put you into the other default
20+app, `:talk`; press `^X` again to switch back.)
21+
22+The dojo actually has its own command language, which is a
23+superset of Hoon. Any input string which parses as a Hoon
24+expression is a dojo command that prints the product of the
25+expression. In the space of strings which don't parse as Hoon
26+expressions, the dojo has other commands.
27+
28+If this is just too abstract, think of the dojo as half Lisp
29+REPL, half Unix shell.
30+
31+### Setting dojo variables
32+
33+Like the Unix shell, the dojo has command variables. Type
34+
35+```
36+> =a 42
37+```
38+
39+You've set the dojo variable `a` to the value `42`. The dojo
40+puts this binding in the Hoon subject for expressions to use.
41+So you can write:
42+
43+```
44+> a
45+42
46+
47+> [4 a]
48+[4 42]
49+
50+> ? [4 a]
51+ {@ud @ud}
52+[4 42]
53+```
54+
55+To unbind `a`, just omit the value:
56+
57+```
58+> =a
59+```
60+
61+You removed `a` from the subject. (We can bind it again later, or
62+bind over it without removing it.)
63+
64+```
65+> [4 a]
66+-find.a
67+```
68+
69+Now it's
70+