git.iko.soy/seax
on master help sign in

The Urbit search engine

git clone ssh://git.iko.soy/seax
ssh https

%seax

The Urbit meta-search engine.

%seax showing results for "urbit" from %wiby and %urbit-dev

About

It's time Urbit got some searching capabilities.

For most people, their computer is a bootloader to the closest search engine. Searching is a central part of everyone's computing workflow.

The %seax meta-search engine is able to aggregate results from open search APIs and bring them inside the comfort of your Urbit ship. No longer do you have to keep a second tab open.

Let's make searching fun again. :)

Contents

Features

Search sources

SourceShown asWhat it isRequest your ship sendsStatus (October 2026)
wiby%wiby"Search Engine for the Classic Web", an index of simple, older-style pagesGET https://wiby.me/json/?q=<query>, wiby's public JSON APIWorks
Urbit docs%urbit-devThe site search of docs.urbit.org, which is built on GitBookPOST https://docs.urbit.org/~gitbook/search with {"query": "<query>", "scope": {"mode": "all"}}Works
CrowdView%crowdviewA search engine for forum discussions (Reddit, Hacker News, Stack Exchange and others)GET https://crowdview-next-js.onrender.com/api/search-v3?query=<query>, the endpoint crowdview.ai's own page usesWorks, but has no results for many queries, such as "urbit"

The query is URL-encoded in the GET requests, and sent as a JSON string to the Urbit docs, whose request also has a content-type: application/json header. Your ship adds no user agent, cookies or other headers: Vere sends only Host, and Content-Length when there's a body.

The page's sources row shows each source that is still loading or found something. A source that fails, or has no results, is hidden.

Until October 2026, %seax also searched Yep. By then Yep blocked requests that don't come from a browser, so it returned no results, but every search still sent it your query. Its official API needs an API key and is paid after 1,000 requests.

Requirements

Installation

INSTRUCTIONS.md has the full steps: on a fake ship to try %seax out, on your own ship, upgrading the 2023 version, and removing %seax. In short:

  1. Build the desk: nix build in a clone of this repository puts it in result/.
  2. In your ship's dojo, create and mount a desk: |new-desk %seax, then |mount %seax.
  3. Replace the files of the mounted desk, <pier>/seax/, with those in result/. Nix's files are read-only, so make the copies writable with chmod -R u+w <pier>/seax, or the next update can't replace them.
  4. In the dojo: |commit %seax, then |install our %seax.

If you installed the 2023 version, which was made for Urbit OS 413k, it was suspended before your ship could update to 408k. Upgrade it the same way, without |new-desk. The new agent drops the old searches, the list of peers, and the copy of your Tlon notes if the old version made one. Because the desk dates from before Urbit OS 410k, your ship counts it as essential: a later Urbit OS update waits until %seax supports it, instead of suspending %seax. Run |essential-desk %seax | to have it suspended like other apps.

To remove %seax, run |uninstall %seax, which stops it and keeps its data. |nuke %seax then deletes the data.

Usage

Open /apps/seax/ at your ship's web address, e.g. http://localhost:8080/apps/seax/, or click the %seax tile in Landscape. If you aren't logged in, you'll be asked for your +code first.

How it works

%seax is a Gall agent, zod/seax/app/seax.hoon, and an Elm app, frontend/. The agent serves the built Elm app from the desk's web/ directory.

sequenceDiagram
    participant B as Browser (Elm app)
    participant E as Eyre
    participant S as %seax
    participant I as Iris
    participant W as wiby, Urbit docs, CrowdView
    B->>E: subscribe to /search/<query> on %seax
    E->>S: watch /search/<query>
    S-->>B: %flush: every source is loading
    S->>I: one HTTP request per source
    I->>W: requests from your ship
    W-->>I: responses
    I-->>S: a response
    S-->>B: %drain: what changed in the sources and ranked results

Ranking

Each source has a weight in zod/seax/lib/engines.hoon, currently .2 (2.0) for all of them. zod/seax/lib/rank.hoon merges the results that have exactly the same URL. Each URL gets a score, which is the product of three numbers:

Results are shown with the highest score first. For a URL that only one source returns, the score is the source's weight divided by the position: a first result scores 2, a second one 1, a third one 0.67. Since the weights are all the same, the first results of all sources come first, then their second results, and so on. Results with the same score come in no particular order.

URLs that differ only by a trailing slash or by http/https count as different URLs. A merged result keeps the title from one of its sources. See Known issues for a bug in the scores of merged results.

Privacy and network traffic

Peers (disabled)

From August 2023 until this version, %seax kept a list of peers: other ships that might run %seax. The list started with ~zod and ~racfer-hattes. Every 30 seconds, %seax pinged its peers by subscribing to /ping on their %seax, and told them which peers had answered, so that ships running %seax could find each other.

Nothing ever used the list, and neither the code nor the commit messages say what it was for. The 2023 code didn't work well either: a peer was dropped after missing one ping, every other check emptied the list, and from October 2023 ships refused the peer lists that other ships sent them, because +on-poke couldn't read pokes from other ships.

This version disables the peer network:

Debugging

%seax prints a lot to the dojo:

A few dojo commands help:

When Vere can't make a request, it prints http: fail (<n>, 504): <reason>, e.g. unknown node or service when a DNS lookup fails, and that source counts as failed.

Adding a search source

  1. Create zod/seax/lib/engines/<name>.hoon. It holds an engine core: ++request builds the HTTP request for a query, and ++results parses the response into a list of [title url], or ~ if it can't. ++get in zod/seax/sur/seax.hoon makes a GET request with the query appended to a URL. It escapes the query twice, because Vere unescapes the URL once before sending it. For example, for an API that returns [{"title": ..., "url": ...}, ...]:

    /-  *seax
^-  engine
=<
|%
++  request
  |=  query=tape
  ^-  request:http
  (get "https://example.com/search?q=" query)
++  results
  |=  response=mime-data:iris
  ^-  (unit (list search-result))
  ;<  jon=json  _biff  (de:json:html q.data.response)
  (apex jon)
--
|%
++  apex
  =,  dejs-soft:format
  (ar (ot ~[title+so url+so]))
--

  2. In zod/seax/lib/engines.hoon, import it (/+ engines-<name>) and add [name=%<name> engine=engines-<name> weight=.2] to the list. The weight is a floating-point literal: .2 is 2.0.

  3. Build and commit the desk again. Nix only sees files that git tracks, so git add the new file before building. The Elm app shows every source the agent has, so it doesn't need any change.

A source counts as failed if ++results returns ~ or an empty list. Note that ar from dejs-soft gives ~ if even one result doesn't parse.

Development

INSTRUCTIONS.md explains how to work on the frontend. dev/elm-live.sh serves it at http://localhost:8000 with Elm's debugger, rebuilds it when the code changes, and passes requests for /~/ on to a running ship.

The flake has these outputs:

To change the agent, edit zod/seax, rebuild the desk and copy it into your mounted desk, as in the installation steps, then |commit %seax. The agent reloads with its state. Nix only sees files that git tracks, so git add new files before building.

Project layout

README.md, INSTRUCTIONS.md  this file, and how to install and develop
LICENSE, NOTICE             the MIT License, and the files from other projects
flake.nix, flake.lock       the Nix build: desk, frontend, font and airlock
dev/elm-live.sh             frontend development server, with a proxy to a ship
docs/screenshot.png         the screenshot above
zod/seax/                   the %seax desk
  app/seax.hoon             the agent
  sur/seax.hoon             the search-result and engine types, and ++get
  lib/engines.hoon          the sources and their weights
  lib/engines/*.hoon        one file per source
  lib/rank.hoon             merges and ranks results
  lib/sink.hoon             sends state as a %flush and then %drain diffs
  lib/noun-diff.hoon        diffs of nouns, for lib/sink.hoon
  lib/dbug.hoon, lib/server.hoon         from Urbit's base-dev files
  lib/docket.hoon, sur/docket.hoon       for the Landscape tile
  lib/de-html.hoon, lib/extract-links.hoon, lib/extract-text.hoon, lib/words.hoon
                            HTML and text helpers from the first, crawler
                            version, which nothing uses now
  mar/                      marks for the desk's files
  desk.docket-0             the tile: title, description, colour, image, URL,
                            version, website and license
  desk.bill                 the agent to start: %seax
  sys.kelvin                [%zuse 408]
frontend/                   the Elm app
  src/Main.elm              the whole app
  src/Route.elm, src/Pages/Search.elm    unused leftovers of an early version;
                            Pages/Search.elm doesn't compile
  index.html, script.js     the page, and the glue between Elm and the event stream
  fetch-event-source.js     event-stream client, from airlock's example
  assets/                   CSS, the logo and tile images, the logo tilt script
  elm.json                  Elm 0.19.2 and its packages; airlock's code comes from airlock/src
  elm-srcs.nix, registry.dat             pinned Elm packages, for the Nix build
  airlock.patch             stops airlock from poking ~zod when it opens a channel

Known issues

The search sources:

The agent:

The page:

History

Credits

Created by ~racfer-hattes and ~dopnyr-figbud.

License

MIT, see LICENSE. Files from other projects are listed in NOTICE.