As a follow-up to #1834:
- Update the sublibrary graph generation script so it works with the no-longer-linear graph
- Regenerate the actual image
- Update the `.cabal` description to match
This is a rework of #1798 to facilitate a simpler web stack.
# Demo
View http://swarmgame.net/
NOTE: Requires IPv6
# Motivation
Hosting cost is a main motivation. Cost per month for an EC2 instance, RDS, and the requisite other services approaches >$50 per month. In contrast, the lowest-tier Lightsail instance is $3.50/month.
The deployment process is of course simplified.
An incidental benefit to using SQLite is reduced latency of web requests; we no longer need to fetch credentials from an AWS API to connect to Postgres.
## Changes
Major changes:
* Use `sqlite` instead of `postgres`
* Use Docker to build a statically-linked deployable binary, rather than deploying the app within a Docker image
Fortunately, the API of `sqlite-simple` is near-identical to that of `postgresql-simple`, so most of the code change there is just to rip out AWS-specific stuff and Postgres connection info. I have no hesitation to delete this code since if we ever want to use the previous stack again, we can just look at #1798.
Towards #1797
Hosts an online repository of scenarios, against which solutions may be submitted. This is the foundational layer that may support more structured "tournaments", scenario ranking, or other social activity.
# Demo
## Live server
http://swarmgame.net/list-games.html
One can use the [`submit.sh`](https://github.com/swarm-game/swarm/pull/1798/files#diff-450877e3442a0ec1c5cbe964808a263d67f1e680d3aa3c3bf9ae6f51eca682fb) script and see valid uploads reflected live on the website.
## Local testing
### Automated tests
These are database-agnostic.
scripts/run-tests.sh swarm:test:tournament-host
### Manual tests
These test database interactions. It requires first setting up a local Postgres server.
1. Start `tournament/scripts/demo/server-native.sh` in one console
2. Run `tournament/scripts/demo/client/test-cases/local/good-submit.sh` in another
# Features
* Upload and validates scenarios
* Download scenarios with solution redacted
* Submit, validate, execute, and score solutions
# Key components
* Servant server
* Hosted on AWS in a Docker container
* Stores to a Postgres database in Amazon RDS
* Shares some code with the integration tests for evaluating scenarios and solutions
The production database uses IAM to manage logins. The web app uses the AWS API to fetch a "token" which can be used to log in instead of a password. This avoids having to store a password on the server.
# TODO
- [ ] User authentication (GitHub OpenID?)
This is a random refactoring that does two things:
* Improves speed of `scripts/gen/render-sublibrary-dependencies.sh` script
* Refine imports of the `TestModel.hs` (renamed to `TestRepl.hs`) unit test
It appears that the only reason that the `swarm-unit` test suite imports the entire `swarm` sublibrary (hence the `brick` package), is due to `TestRepl`, which (now) imports `Swarm.TUI.Model.Repl`. Perhaps it would make sense to separate these at some point?
Prerequisite to #1798
## Changes
* Pass the final `TickNumber` count as a member of `Won` constructor so that it can be used in scoring submitted solutions
* Extract a helper function `codeMetricsFromSyntax` that can be reused by tournament server
* `ToJSON` instance for `ScenarioMetadata`
* New script `list-sublibraries.sh` to list the sublibraries defined in a package
Fixes#1598
## Demo
Illustrates immediate (same-tick) reaction to a change of a `watch`ed cell:
scripts/play.sh -i data/scenarios/Testing/1598-detect-entity-change.yaml --autoplay --speed 1
## Background
Robots can be **scheduled** to wakeup from a `watch` by the `wakeWatchingRobots` function.
Robots may be **actually awakened** by the `wakeUpRobotsDoneSleeping` function.
Previously, `wakeWatchingRobots` would only ever schedule wakeups at `currentTick + 1`. Then, once the next tick is reached, in `wakeUpRobotsDoneSleeping` only robots that had been scheduled to wake on precisely the **current tick** could actually be awakened.
But this made it impossible for one robot to cause another robot, which had been sleeping, to actually wake up within the same tick that scheduling of the wakeup is performed.
The importance of this comes into play for system robots that are `watch`ing a cell and intended to react instantaneously to player actions (usually by running code in an `instant` block).
During each "tick", every active robot gets exactly one "turn" to execute code. Given the following ID assignments:
| Robot ID | Robot name |
| --- | --- |
| `0` | `base` |
| `1` | `systemBot` |
the `systemBot` will always execute *after* the `base` robot in a given tick.
If the `systemBot` is `watch`ing a given cell, a modification of that cell by the `base` will schedule a wakeup of the `systemBot`. If the scheduled wakeup tick is `currentTick + 1`, then the `base` will have **another execution turn** before the `systemBot` gets an opportunity to react to the `base`'s action at the current tick, causing the `systemBot` to overlook actions that may be relevant to a goal condition.
## Solution
In contast, if we schedule the `systemBot` wakeup for `currentTick` instead of `currentTick + 1`, then it should have an opportunity to wake, run, and react to the `base`'s action before the `base`'s next turn.
But in the status quo, the selection of which robots to run in a given tick is made only once, before iteration over those robots begins. We need instead to update the robots-to-run pool dynamically, after each iteration on an individual robot. The most appropriate data structure for the iteration pool is probably a [Monotone priority queue](https://en.wikipedia.org/wiki/Monotone_priority_queue), but we approximate this with an `IntSet` of `RID`s and the [`minView`](https://hackage.haskell.org/package/containers-0.7/docs/Data-IntSet.html#v:minView) function.
Being able to alter the list of active robots in the middle of a given tick's robot iteration has required a change to the `runRobotIDs` function. Instead of using a `forM_` to iterate over a static list of `RIDs`, we have a custom iteration function `iterateRobots` to handle the dynamic set.
## Performance
Benchmarks were performed locally with `stack` and GHC 9.6.
I had tested the replacement of the `forM_` loop with `iterateRobots` in isolation, and it had zero effect on benchmarks. But some other trivial changes tested in isolation, such as the addition of docstrings, yielded a +6% increase in benchmarks. So there seems to be some instability in the ability of the compiler to perform certain optimizations.
With this PR, increases in benchmark times ranging from 7% to 11% were observed. The obvious culprit is that `wakeUpRobotsDoneSleeping` is now called N times per tick, rather than once per tick, where N is the number of robots that are initially (or become) active during that tick.
### Mitigating with an "awakened" set
In the common case (i.e. when there are *no* watching robots), the `wakeUpRobotsDoneSleeping` that is now executed `N` times per tick (instead of once per tick) incurs a `Map` lookup, which is `O(log M)` in the number of distinct future wakeup times across all robots.
However, those extra `N` invocations only exist to serve the purpose of `watch`ing robots that may need to wake up at the current tick. It may be more efficient to have a dedicated `Set` in the `GameState` for such robots that gets populated parallel to this insertion:
ad5c58917e/src/swarm-engine/Swarm/Game/State/Robot.hs (L387)
Then if this set remains `null`, we can avoid paying for an additional `O(N * log M)` operations entailed by the `Map` lookups into `internalWaitingRobots`.
#### Result
Indeed, storing awakened bots in a new `currentTickWakeableBots` set restored the benchmarks nearly to the baseline. Instead of the regressions in the 10-20% range observed before, now only a few benchmarks had at most 3-4% increases.
### CI regressions
In CI, (`snake`, `powerset`, and `word-search`) exceeded their timeout thresholds in GHC 9.2 and GHC 9.4. No regressions were observed with GHC 9.6. To accommodate the lowest common denominator, I bumped the thresholds for those three scenarios to get a passing build.
I don't believe it is worth further effort or investigation to optimize for GHC 9.4, since such efforts will be moot for GHC 9.6 and onward.
Test invocation:
cabal test swarm-integration --test-show-details streaming --test-options '--color always --pattern "word-search"'
| **Scenario** | GHC 9.4.8 | GHC 9.4.8 | GHC 9.6.4 | GHC 9.6.4 |
| --- | --- | --- | --- | --- |
| | Before | **After** | Before | **After**
| `snake` | 1.84s | **5.38s** | 1.62s | **1.67s** |
| `powerset` | 1.66s | **5.09s** | 1.56s | **1.66s** |
| `word-search` | 0.56s | **1.91s** | 0.44s | **0.48s** |
### Potential improvements
#### `TickNumber` as `Map` key
`waitingRobots` is of type `Map TickNumber [RID]`. Instead of `Map`, we could have a `newtype` that encapsulates a `IntMap` to make lookups by `TickNumber` more efficient.
Closes#1641
The `data/terrain.yaml` file is now the authoritative source of terrains, though `BlankT` is still a hard-coded special case.
I have not changed the underlying integer representation of terrain in the world function, which means that the `terrainIndexByName` Map in the `TerrainMap` record is needed for translating between `Int` and `TerrainType`.
# Demo
scripts/play.sh -i data/scenarios/Testing/1775-custom-terrain.yaml
# Changes
* There used to be a function called `integrateScenarioEntities` that combined the `EntityMap` stored in the `Scenario` record with the global entity map. However, the global entity map is accessible at parse time of the `Scenario`, so we do the combining there and only ever store the combined map in the `Scenario` record.
* JSON Schema for terrain
* Removed the distinction between "World" attributes and "Terrain" attributes
* Unit tests for scenario-defined terrain and related validations
* Validate existence of referenced terrain at scenario parse time
* Validate attributes referenced by terrains at parse time
Opens a live-reloading preview of the world in VS Code.
The renderer has been modified to optionally render a blank image instead of crashing upon invalid YAML.
## Prerequisites:
Install inotify tools:
sudo apt install inotify-tools
## Usage:
scripts/preview-world-vscode.sh data/scenarios/Fun/horton.yaml
Once the VS Code editor tabs are opened, one can press <kbd>CTRL</kbd> + <kbd>\\</kbd> (backslash) with the image selected to split the editor pane horizontally.
One may then navigate to the left-pane's copy of the image preview with <kbd>CTRL</kbd> + <kbd>PageUp</kbd>, and then <kbd>CTRL</kbd> + <kbd>w</kbd> will close the redundant image preview.
## Screenshot
In support of #1598 (for #1739).
Runs the `idle` benchmark for many more ticks and with many more robots to try to emphasize the effect of regressions and mitigate jitter.
Also factors out common code from the `benchmark-against-parent.sh` script.
Builds upon #1587.
Extract all symbol names that are not native to the current project and insert them into our own custom spell checking dictionary's "words" list.
The premise is that symbols that *are* native to our project should be spellchecked, but foreign symbols that constitute unrecognized dictionary words are presumably intentionally spelled that way.
# Convention
Manually-added words (i.e. for names in code that we've written for this project) will go into `.vscode/settings.json`.
The automatically generated word list from third-party packages goes into `cspell.json`.
# Usage
scripts/spellcheck/autopopulate-spellchecker.sh
Closes#845
**All changes are non-significant whitespace.** This can be verified with:
git show --ignore-all-space --ignore-blank-lines
in which the remaining changes are only elimination of manual word-wrap in descriptions.
Normalization is accomplished with this command:
scripts/normalize-all-scenarios.sh
This is an initial normalization pass that shall be a pre-requisite for #1713.
This seems to help with rebuild times, since we are specifically requesting to only build the `swarm` executable and not, say, the `swarm-docs` executable.
Both `execConst` and `stepCESK` are huge case statements. This refactor puts them in different modules.
Also many supporting functions for implementing commands in `execConst` are moved to their own modules.
Whereas `Step.hs` previously had `2931` lines, the new linecounts are:
| File | lines |
| --- | --- |
| `Step.hs` | 848 |
| `Arithmetic.hs` | 124 |
| `Const.hs` | 1687 |
| `Command.hs` | 417 |
The only tricky thing was that `execConst` is mutually recursive with `runCESK`. So to get them into different modules, I pass a wrapper of `runCESK` as an argument to `execConst`.
* add wave program and parametrise it to compare inlined/generic version
* use [`tasty-bench`](https://hackage.haskell.org/package/tasty-bench) library to show comparison
* move benchmarks to test folder as they can now share tasty code
* closes#1574
Using the recursive definition with ifs leads to a 3x slowdown:
```
wavesInlined
10: OK
361 ms ± 29 ms
20: OK
718 ms ± 35 ms
30: OK
1.066 s ± 28 ms
40: OK
1.437 s ± 37 ms
wavesWithDef
10: OK
1.052 s ± 51 ms, 2.92x
20: OK
2.117 s ± 34 ms, 2.95x
30: OK
3.144 s ± 80 ms, 2.95x
40: OK
4.191 s ± 91 ms, 2.92x
```
But if we just inline and simplify the code, we can remove the runtime overhead completely.
Closes#1436.
The schema `.json` files are now the authoritative source of truth for documentation.
Wrote a very simple parser for JsonSchema to extract the documentation from JSON.
Split the README.md into [static](c314cc50a1/data/scenarios/README.md) and [auto-generated](c314cc50a1/data/scenarios/doc-fragments/SCHEMA.md) parts.
Added a custom `"footers"` key to schema files to support inclusion of other markdown files for each object description.
# Schema doc regeneration
./scripts/regenerate-schema-docs.sh
In preparation for #1483. `LogEntry` started life as something specific to robot logs. It then evolved to be used in the system log as well (see #1039 and #652), but in a sort of hacky way. This PR refactors `LogEntry` to be more generic.
- Move `Swarm.Game.Log` -> `Swarm.Log` since it's not specific to gameplay.
- Rename `ErrorLevel` to `Severity`, add a new `Info` level, and add a top-level `leSeverity` field
- Rename `leRobotName` to just `leName`, since it was already being used to name both robots and system components anyway
- Move robot-specific fields (*e.g.* robot ID) into the new `RobotLogSource` type, and add `LogSource` to differentiate between robot and system logs
- Various other minor improvements and tweaks
Towards #872
Previously, the `GameState` record had `41` toplevel members. It now has `22`. Logical grouping of the fields makes it easier to peruse the code and paves the way to split `State.hs` into smaller modules. Some functions now may even be able to operate exclusively on subsets of the game state, rather than having to pass in `GameState` as a whole.
There is potential to go even farther, by extracting view-related and robot-related members to their own records, but I figured I'd pursue this refactoring incrementally.
Closes#1428
Since the authoritative validation of scenario files is actually performed by virtue of `swarm` parsing them, this CI job actually exists to ensure the JSON Schema descriptions are accurate. This is important for two purposes:
* Documentation is generated from the JSON Schema files (#1436)
* JSON Schema has integration with VS Code and other IDEs
# Testing
Verified that the schema checker action does indeed work by intentionally pushing an invalid scenario file in f789f81.
* use `Markdown.Document` as `entityDescription`
* add missing spaces in `chunksOf`
* fix code in `entities.yaml` (mostly types and few outdated snippets)
* add code markdown in craft tutorial
* use colours for types and entities
- closes#1408
- closes#1409
Closes#144.
This builds upon portals support (#1356)
# Demo
scripts/play.sh --scenario data/scenarios/Testing/144-subworlds/subworld-mapped-robots.yaml --autoplay --speed 2
[](https://asciinema.org/a/vC13dW8M1S8t2b1J4XkW80U1q)
# Future work
* Augment portal definitions with an optional "relative orientation" attribute, that can turn the player around when passing through the portal (#1379)
* Specify whether portal performs instant transportation or whether `move down` is required (#1368)
towards #1171
## New tests
Unit tests:
scripts/run-tests.sh --test-arguments '--pattern "Relative direction"'
Integration tests for `sniff` and `chirp` demonstrate how to home in on an item using distance and orientation, respectively.
## Efficiency
For the sake of execution time, I have capped the max "diameter" (`N`) of both commands to `200` cells. In the worst case (the entity is not present), `O(N^2)` cells are inspected, which manifests as a perceptible delay when the command is run. I came across the `getElemsInArea` function that seems to suggest that an `O(N * log N)` search may be possible. Is that the case?
Otherwise we may be able to add some new data structures to the game state for efficient entity location querying.
`stack` invocation with `--fast` to work around #1000
Example invocation:
./scripts/play.sh --scenario data/scenarios/Challenges/Ranching/gated-paddock.yaml --autoplay
Closes#936.
With this change, `stack build` should be invoked as normal, but to run tests one should invoke the new script, which will place compilation artifacts in a separate directory so as to not stomp on the previous `stack build` artifacts.
## Demo
```
stack build
./scripts/run-tests.sh
stack build
```
All perform incremental compilation rather than from-scratch.
Restyled.io uses fourmolu version `0.10.1.0`. It is not clear how it could be pegged to `0.4.0.0`. The latest I was able to install on my machine with `cabal install` was `0.9.0.0`.
fixes#888
