@@ -0,0 +1,307 @@
2+ <img src="frontend/assets/img/logo.png" alt="%seax" width="510">
5+The [Urbit](https://urbit.org/) meta-search engine.
8+ <img src="docs/screenshot.png" alt="%seax showing results for "urbit" from %wiby and %urbit-dev" width="700">
13+It's time Urbit got some searching capabilities.
15+For most people, their computer is a bootloader to the closest search engine. Searching is a central part of everyone's computing workflow.
17+The `%seax` meta-search engine is able to aggregate results from open search APIs and bring them inside the comfort of your Urbit ship. No longer do you have to keep a second tab open.
19+Let's make searching fun again. :)
23+- [Features](#features)
24+- [Search sources](#search-sources)
25+- [Requirements](#requirements)
26+- [Installation](#installation)
28+- [How it works](#how-it-works)
30+- [Privacy and network traffic](#privacy-and-network-traffic)
31+- [Peers (disabled)](#peers-disabled)
32+- [Debugging](#debugging)
33+- [Adding a search source](#adding-a-search-source)
34+- [Development](#development)
35+- [Project layout](#project-layout)
36+- [Known issues](#known-issues)
43+- One search box for several search engines. Your ship sends the query to all of them and shows their results on one page.
44+- Results appear as each source answers, without waiting for the slowest.
45+- A page that more than one source returns is shown once and ranked higher.
46+- Click a source's name to see only its results.
47+- Runs on your own ship at `/apps/seax/`, with a tile in Landscape. Only you can search: visitors who aren't logged in are sent to the login page.
48+- Nothing runs in the background: %seax sends requests only when you search.
52+| Source | Shown as | What it is | Request your ship sends | Status (October 2026) |
54+| [wiby](https://wiby.me/) | `%wiby` | "Search Engine for the Classic Web", an index of simple, older-style pages | `GET https://wiby.me/json/?q=<query>`, wiby's public JSON API | Works |
55+| [Urbit docs](https://docs.urbit.org/) | `%urbit-dev` | The site search of docs.urbit.org, which is built on GitBook | `POST https://docs.urbit.org/~gitbook/search` with `{"query": "<query>", "scope": {"mode": "all"}}` | Works |
56+| [CrowdView](https://crowdview.ai/) | `%crowdview` | A search engine for forum discussions (Reddit, Hacker News, Stack Exchange and others) | `GET https://crowdview-next-js.onrender.com/api/search-v3?query=<query>`, the endpoint crowdview.ai's own page uses | Works, but has no results for many queries, such as "urbit" |
58+The query is URL-encoded in the GET requests, and sent as a JSON string to the Urbit docs, whose request also has a `content-type: application/json` header. Your ship adds no user agent, cookies or other headers: Vere sends only `Host`, and `Content-Length` when there's a body.
60+The page's sources row shows each source that is still loading or found something. A source that fails, or has no results, is hidden.
62+Until October 2026, %seax also searched [Yep](https://yep.com/). By then Yep blocked requests that don't come from a browser, so it returned no results, but every search still sent it your query. Its official API needs an API key and is paid after 1,000 requests.
66+- A ship running Urbit OS 408k, which needs Vere 4.4 or newer (4.6 is current). The desk declares `[%zuse 408]` in `zod/seax/sys.kelvin`. When your ship moves to a newer Urbit OS, it suspends %seax until you install a version of the desk that declares the new one. A desk upgraded from the 2023 version is the exception; see [Installation](#installation).
67+- To build the desk: [Nix](https://nixos.org/download/) with flakes enabled, on x86_64-linux, aarch64-linux or aarch64-darwin. Nix fetches Elm and the other build tools already built from cache.nixos.org, so the build only compiles the frontend and copies files.
71+[INSTRUCTIONS.md](INSTRUCTIONS.md) has the full steps: on a fake ship to try %seax out, on your own ship, upgrading the 2023 version, and removing %seax. In short:
73+1. Build the desk: `nix build` in a clone of this repository puts it in `result/`.
74+2. In your ship's dojo, create and mount a desk: `|new-desk %seax`, then `|mount %seax`.
75+3. Replace the files of the mounted desk, `<pier>/seax/`, with those in `result/`. Nix's files are read-only, so make the copies writable with `chmod -R u+w <pier>/seax`, or the next update can't replace them.
76+4. In the dojo: `|commit %seax`, then `|install our %seax`.
78+If you installed the 2023 version, which was made for Urbit OS 413k, it was suspended before your ship could update to 408k. Upgrade it the same way, without `|new-desk`. The new agent drops the old searches, the list of peers, and the copy of your Tlon notes if the old version made one. Because the desk dates from before Urbit OS 410k, your ship counts it as essential: a later Urbit OS update waits until %seax supports it, instead of suspending %seax. Run `|essential-desk %seax |` to have it suspended like other apps.
80+To remove %seax, run `|uninstall %seax`, which stops it and keeps its data. `|nuke %seax` then deletes the data.
84+Open `/apps/seax/` at your ship's web address, e.g. `http://localhost:8080/apps/seax/`, or click the %seax tile in Landscape. If you aren't logged in, you'll be asked for your `+code` first.
86+- Type a query, then press Enter or click **search**.
87+- The **Sources** row names each source that is still loading or has results. Sources that are loading pulse between grey and black, and can't be clicked until they finish.
88+- Click a source to show only its results. Click more sources to add theirs, and click a source again to remove it. With no source selected, all results are shown. The selection stays when you search again.
89+- Each result shows an emoji, the title and the address. The emoji is picked from a hash of the title and is only decoration. Titles are shown without HTML tags, and with HTML entities and `%` escapes decoded.
90+- Click a title to open the result in a new tab. The address under it isn't a link.
91+- While no source has results, the page says "No results found... :)". It also says so while the sources are still loading.
92+- The logo on the start page tilts as you move the mouse over it.
93+- The page's address doesn't change when you search, so you can't bookmark a search. The back button leaves the app, and reloading the page brings back the start page.
94+- The **[About]** link at the bottom leads to this README, and **[Code]** to this repository.
98+%seax is a Gall agent, [`zod/seax/app/seax.hoon`](zod/seax/app/seax.hoon), and an Elm app, [`frontend/`](frontend). The agent serves the built Elm app from the desk's `web/` directory.
102+ participant B as Browser (Elm app)
103+ participant E as Eyre
104+ participant S as %seax
105+ participant I as Iris
106+ participant W as wiby, Urbit docs, CrowdView
107+ B->>E: subscribe to /search/<query> on %seax
108+ E->>S: watch /search/<query>
109+ S-->>B: %flush: every source is loading
110+ S->>I: one HTTP request per source
111+ I->>W: requests from your ship
114+ S-->>B: %drain: what changed in the sources and ranked results
117+- **Serving the app.** The agent binds `/apps/seax` in Eyre and serves the files in the desk's `web/` directory to logged-in users. Everyone else is redirected to `/~/login`. The Landscape tile comes from `zod/seax/desk.docket-0`.
118+- **Searching.** The Elm app subscribes to `/search/<query>` on the agent through an Eyre channel, using [airlock](https://github.com/iko-soy/airlock). The agent accepts the subscription only from the ship itself. For a query that isn't open yet, it sends one HTTP request per source through Iris. Every new subscriber gets the current state right away.
119+- **Streaming.** Each time a source answers, the agent parses the answer, ranks all the results so far and sends the change. The first message is the whole state (`%flush`), and later ones are diffs of it (`%drain`), made by [`lib/sink.hoon`](zod/seax/lib/sink.hoon) and [`lib/noun-diff.hoon`](zod/seax/lib/noun-diff.hoon). airlock's `Ur.Sub.sink` applies them in the browser.
120+- **Sharing searches.** While a query has subscribers, its results stay in the agent's state. Another tab that searches for the same query gets them right away, without new requests. Queries have to match exactly, including case and spaces. When the last subscriber of a query leaves, the agent forgets the query. A new search in the same tab leaves the old query, but closing or reloading a tab doesn't: Eyre ends the tab's subscriptions when it closes the tab's channel, 12 hours after the tab disconnects. Results aren't refreshed while a query stays open.
121+- **Failures.** A source fails when its response has no body, or the body isn't the JSON the source's parser expects. The agent doesn't look at the HTTP status. When Vere can't make a request (DNS, connection or TLS errors, or a connection that stalls for 5 minutes), Iris returns status 504 with no body, so that source fails too. Iris follows redirects with status 301, 303 and 307; other redirects make the source fail. Failed requests aren't retried.
125+Each source has a weight in [`zod/seax/lib/engines.hoon`](zod/seax/lib/engines.hoon), currently `.2` (2.0) for all of them. [`zod/seax/lib/rank.hoon`](zod/seax/lib/rank.hoon) merges the results that have exactly the same URL. Each URL gets a score, which is the product of three numbers:
127+- the weights of the sources that returned it, multiplied together,
128+- the number of times it was returned,
129+- the sum of 1/position for each time, counting positions from 1.
131+Results are shown with the highest score first. For a URL that only one source returns, the score is the source's weight divided by the position: a first result scores 2, a second one 1, a third one 0.67. Since the weights are all the same, the first results of all sources come first, then their second results, and so on. Results with the same score come in no particular order.
133+URLs that differ only by a trailing slash or by `http`/`https` count as different URLs. A merged result keeps the title from one of its sources. See [Known issues](#known-issues) for a bug in the scores of merged results.
135+## Privacy and network traffic
137+- **Searches.** Your query goes from your ship to wiby.me, docs.urbit.org and crowdview-next-js.onrender.com, over HTTPS. Those services see the query and your ship's IP address.
138+- **Nothing else.** %seax makes requests only when you search for a query that isn't already open. It has no timers, and it doesn't contact other ships, except once: an upgrade from an older version calls off that version's pings to its peers that are still waiting for an answer. See [Peers (disabled)](#peers-disabled).
139+- **The page.** It loads nothing from other sites: the fonts, scripts and images all come from your ship. airlock normally pokes ~zod when it opens a channel, and [`frontend/airlock.patch`](frontend/airlock.patch) removes that poke. The bug is reported at [iko-soy/airlock#15](https://github.com/iko-soy/airlock/issues/15). Eyre's login page, which comes with Urbit, loads its fonts from rsms.me and storage.googleapis.com.
140+- **Stored data.** Search results stay in the agent's state while their query is open, which can be up to 12 hours after you close its tab. As with everything your ship does, the requests and responses are also recorded in your pier's event log. The agent also prints your queries and their results in the dojo.
141+- **Other ships.** Only you can search. %seax refuses other ships' pokes and searches, so they can't use it through your ship.
145+From August 2023 until this version, %seax kept a list of peers: other ships that might run %seax. The list started with ~zod and ~racfer-hattes. Every 30 seconds, %seax pinged its peers by subscribing to `/ping` on their %seax, and told them which peers had answered, so that ships running %seax could find each other.
147+Nothing ever used the list, and neither the code nor the commit messages say what it was for. The 2023 code didn't work well either: a peer was dropped after missing one ping, every other check emptied the list, and from October 2023 ships refused the peer lists that other ships sent them, because `+on-poke` couldn't read pokes from other ships.
149+This version disables the peer network:
151+- It doesn't ping other ships or send them anything, and it has no timers.
152+- An upgrade drops the list and calls off the pings that are still waiting for an answer.
153+- It refuses pings and peer lists from other ships.
157+%seax prints a lot to the dojo:
159+- the wire of every response it gets from Arvo, e.g. `/search-result/<query>/<source>` for each HTTP response,
160+- each source's parsed results, followed by the state of every source,
161+- the path of every search that closes,
162+- the stack trace of every crash while it handles a response, such as the late answers in [Known issues](#known-issues).
164+A few dojo commands help:
166+- `:seax +dbug` prints the agent's state: each open query, with its number of subscribers and the results of each source. A source that is still loading has no entry, `~` means that it failed, and `[~ ~]` that it found nothing.
167+- `:seax +dbug [%incoming ~]` lists the subscriptions to the agent, i.e. open searches.
168+- `:seax +dbug [%outgoing ~]` lists the agent's own subscriptions. It should print "no matching subscriptions", also right after an upgrade from an older version.
169+- `:seax +dbug [%state '~(key by search-subscriptions)']` evaluates Hoon against the state, here listing the open queries.
170+- `+vats %seax` shows whether the desk is running or suspended, e.g. after an Urbit OS update, and which Urbit OS version it declares in `/sys/kelvin`.
172+When Vere can't make a request, it prints `http: fail (<n>, 504): <reason>`, e.g. `unknown node or service` when a DNS lookup fails, and that source counts as failed.
174+## Adding a search source
176+1. Create `zod/seax/lib/engines/<name>.hoon`. It holds an `engine` core: `++request` builds the HTTP request for a query, and `++results` parses the response into a list of `[title url]`, or `~` if it can't. `++get` in [`zod/seax/sur/seax.hoon`](zod/seax/sur/seax.hoon) makes a GET request with the query appended to a URL. It escapes the query twice, because Vere unescapes the URL once before sending it. For example, for an API that returns `[{"title": ..., "url": ...}, ...]`:
186+ (get "https://example.com/search?q=" query)
188+ |= response=mime-data:iris
189+ ^- (unit (list search-result))
190+ ;< jon=json _biff (de:json:html q.data.response)
196+ (ar (ot ~[title+so url+so]))
200+2. In [`zod/seax/lib/engines.hoon`](zod/seax/lib/engines.hoon), import it (`/+ engines-<name>`) and add `[name=%<name> engine=engines-<name> weight=.2]` to the list. The weight is a floating-point literal: `.2` is 2.0.
201+3. Build and commit the desk again. Nix only sees files that git tracks, so `git add` the new file before building. The Elm app shows every source the agent has, so it doesn't need any change.
203+A source counts as failed if `++results` returns `~` or an empty list. Note that `ar` from `dejs-soft` gives `~` if even one result doesn't parse.
207+[INSTRUCTIONS.md](INSTRUCTIONS.md#development) explains how to work on the frontend. `dev/elm-live.sh` serves it at `http://localhost:8000` with Elm's debugger, rebuilds it when the code changes, and passes requests for `/~/` on to a running ship.
209+The flake has these outputs:
211+- `packages.<system>.desk`, the default: the desk, with the built frontend in `web/`.
212+- `packages.<system>.frontend`: the built frontend.
213+- `packages.<system>.fonts`: the Roboto Light web font, converted to WOFF2 from nixpkgs' Roboto, and its license.
214+- `packages.<system>.airlock-src`: airlock at the commit pinned in `flake.lock`, with `frontend/airlock.patch` applied.
215+- `devShells.<system>.default`: a shell with `elm` 0.19.2, `elm-live` and `elm2nix`.
217+To change the agent, edit `zod/seax`, rebuild the desk and copy it into your mounted desk, as in the installation steps, then `|commit %seax`. The agent reloads with its state. Nix only sees files that git tracks, so `git add` new files before building.
222+README.md, INSTRUCTIONS.md this file, and how to install and develop
223+LICENSE, NOTICE the MIT License, and the files from other projects
224+flake.nix, flake.lock the Nix build: desk, frontend, font and airlock
225+dev/elm-live.sh frontend development server, with a proxy to a ship
226+docs/screenshot.png the screenshot above
227+zod/seax/ the %seax desk
228+ app/seax.hoon the agent
229+ sur/seax.hoon the search-result and engine types, and ++get
230+ lib/engines.hoon the sources and their weights
231+ lib/engines/*.hoon one file per source
232+ lib/rank.hoon merges and ranks results
233+ lib/sink.hoon sends state as a %flush and then %drain diffs
234+ lib/noun-diff.hoon diffs of nouns, for lib/sink.hoon
235+ lib/dbug.hoon, lib/server.hoon from Urbit's base-dev files
236+ lib/docket.hoon, sur/docket.hoon for the Landscape tile
237+ lib/de-html.hoon, lib/extract-links.hoon, lib/extract-text.hoon, lib/words.hoon
238+ HTML and text helpers from the first, crawler
239+ version, which nothing uses now
240+ mar/ marks for the desk's files
241+ desk.docket-0 the tile: title, description, colour, image, URL,
242+ version, website and license
243+ desk.bill the agent to start: %seax
244+ sys.kelvin [%zuse 408]
245+frontend/ the Elm app
246+ src/Main.elm the whole app
247+ src/Route.elm, src/Pages/Search.elm unused leftovers of an early version;
248+ Pages/Search.elm doesn't compile
249+ index.html, script.js the page, and the glue between Elm and the event stream
250+ fetch-event-source.js event-stream client, from airlock's example
251+ assets/ CSS, the logo and tile images, the logo tilt script
252+ elm.json Elm 0.19.2 and its packages; airlock's code comes from airlock/src
253+ elm-srcs.nix, registry.dat pinned Elm packages, for the Nix build
254+ airlock.patch stops airlock from poking ~zod when it opens a channel
261+- **CrowdView often has no results,** and %seax shows a source without results the same way as a failed one: it's hidden.
262+- **One malformed result fails a whole source.** If any result in a source's response doesn't parse, the whole source counts as failed.
266+- **Scores of merged results are wrong.** `lib/rank.hoon` adds the 1/position terms of a merged URL, and compares scores, as integers instead of floating-point numbers. As a result:
267+ - URLs that 2 sources return always come before URLs that only 1 returns. Among themselves, they come roughly in order of the product of their two positions, smallest first, instead of by their scores, and those whose positions multiply to 4 or less tie at the top.
268+ - URLs that all 3 sources return come before those.
269+ - Single-source results are ranked correctly.
270+- **Late answers crash the agent.** If you search for something else before every source has answered, the late answers to the old query crash the agent's `+on-arvo`, and the dojo prints a stack trace for each one. Nothing else is affected. The agent never cancels its requests.
271+- **Restarting the ship during a search leaves sources loading.** When Vere restarts, Iris cancels the requests that are still open. The agent crashes on each cancellation, and those sources keep loading until the query closes, which can be 12 hours after you close its tab.
272+- **A source that stalls keeps loading** until Vere's HTTP client gives up after 5 minutes.
273+- **Results are never refreshed.** A query that is open in any tab, or in a tab closed less than 12 hours ago, gives the results it already has.
277+- **A `%` in a title can hide it.** Titles are `%`-decoded, and when that fails, as on a lone `%`, the title disappears and the result can't be clicked. Escapes that do decode are decoded even when they are part of the real title.
278+- **Narrow screens are cut off.** The logo and the search box have fixed widths, so on a phone the search box and the search button are cut off, and on the start page the logo too. Titles and addresses never wrap, so long ones are cut off as well. The page can't scroll sideways.
279+- **The source filter stays for the next search.** If the selected sources have no results for the new query, the list is empty without the "No results found" message, and a selected source that is hidden can't be unselected.
280+- **"Come back later" doesn't help.** The no-results message says "We'll try and find you something. Just come back later.", but nothing searches again.
281+- **Searching for the same query again does nothing,** so the page can't refresh its results.
282+- **Searches can't be bookmarked,** because the page's address doesn't change.
283+- **Smaller things:** only the title of a result is a link, although the whole result looks clickable; the logo tilts only on the start page; the page has no favicon; the search box isn't focused when the page opens; and the page logs every event from the ship to the browser console.
287+- **July 2023.** iko created the repository on 23 July and started %seax on 27 July, as a web crawler with a reverse word index. On 29 July it switched to searching other search engines, with Yep, CrowdView, wiby and Alexandria as sources, and from 31 July it sent each query to all of them. Streaming results through `%sink` and noun diffs, ranking and the first Elm frontend were in place by 2 August, and filtering by source and animations by 5 August.
288+- **August 2023.** iko added liveness checks between %seax ships on 9 August, and "bad peer discovery" (gossip) on 27 August.
289+- **October 2023.** Evangelos Lamprou added the logo and its tilt effect, improvements to the results page including the emoji, the Urbit developer docs as a source, and the first INSTRUCTIONS.md. He also added a `%get-notes` poke that copied the ship's Tlon notebook notes into the agent's state. The notes were never searched.
290+- **Upstream.** iko also contributed `%sink` and the noun-diff library to urbit/urbit in July 2023 ([#6712](https://github.com/urbit/urbit/pull/6712), [#6681](https://github.com/urbit/urbit/pull/6681)), where they live in `pkg/base-dev`.
291+- **2026.** %seax was ported to Urbit OS 408k:
292+ - It is built with a Nix flake, served from the ship, and has a Landscape tile.
293+ - The docs.urbit.org search replaced the old urbit.org developer search.
294+ - Alexandria, whose API is gone, and the Tlon notes were removed.
295+ - The peer network, which nothing used, was disabled.
296+ - Yep, which blocks requests that don't come from a browser, was dropped.
300+Created by [~racfer-hattes](https://iko.soy) and [~dopnyr-figbud](https://vagos.lamprou.xyz/).
302+- Results come from [wiby](https://wiby.me/), the [Urbit docs](https://docs.urbit.org/) and [CrowdView](https://crowdview.ai/).
303+- The frontend is written in [Elm](https://elm-lang.org/) with [airlock](https://github.com/iko-soy/airlock), [elm-ui](https://github.com/mdgriffith/elm-ui) and [elm-animator](https://github.com/mdgriffith/elm-animator), and includes the [Roboto](https://github.com/googlefonts/roboto-3-classic) font.
307+MIT, see [LICENSE](LICENSE). Files from other projects are listed in [NOTICE](NOTICE).