README.md · roboservant

git.iko.soy/roboservant

114 lines 4.9 kB history raw

1# roboservant

3Automatically fuzz your servant apis in a contextually-aware way.

5[![Stack CI](https://github.com/mwotton/roboservant/actions/workflows/ci.yml/badge.svg)](https://github.com/mwotton/roboservant/actions/workflows/ci.yml)

6[![Cabal CI](https://github.com/mwotton/roboservant/actions/workflows/cabal.yml/badge.svg)](https://github.com/mwotton/roboservant/actions/workflows/cabal.yml)

8## example

10see full example [here](EXAMPLE.md)

12## why?

14Servant gives us a lot of information about what a server can do. We

15use this information to generate arbitrarily long request/response

16sessions and verify properties that should hold over them.

18## how?

20In essence, ```fuzz @Api yourServer config``` will make a bunch of

21calls to your API, and record the results in a type-indexed

22dictionary. This means that they are now available for the

23prerequisites of other calls, so as you proceed, more and more api

24calls become possible.

26We explicitly do not try to come up with plausible values that haven't

27somehow come back from the API. That's straying into QC/Hedgehog

28territory: if you want that, come up with the values on that side, and

29set them as seeds in the configuration.

31### what does it mean to be "available"?

33In a simple API, you may make a call and get back a `Foo`, which will

34allow you to make another call that requires a `Foo`. In a more

35complicated app, it's likely that you'll send a request body that

36includes many subcomponents, and it's likely you'll get a response

37that needs to be broken down into pieces before it's useful.

39To cope with this, we have the typeclasses `BuildFrom` and

40`Breakdown`. You can write instances for them if you feel like it, and

41indeed it's currently required for recursive datatypes if you don't

42want the fuzzer to hang, but for the majority of your types it should

43be sufficient to derive them generically. (Sensible instances are

44provided for lists.)

46There are two basic strategies here. In some cases, you want to regard

47a type as indivisible: that's why we like newtypes, right? In this

48case, we can derive using the `Atom` strategy.

50``` haskell

51deriving via (Atom NewtypedKey) instance Breakdown NewtypedKey

52deriving via (Atom NewtypedKey) instance BuildFrom NewtypedKey

53```

55This is saying "A can neither be built from components or broken down

56for spare parts. Hands off!". This is a good strategy for key types,

57for instance.

59If instead it's a big complicated thing with lots of juicy

60subcomponents, we want to rip it apart using Generics and feast on

61its succulent headmeats:

63``` haskell

64deriving via (Compound Payload) instance Breakdown Payload

65deriving via (Compound Payload) instance BuildFrom Payload

66```

68### priming the pump

70Sometimes there are values we'd like to smuggle into the API that are

71not derivable from within the API itself: sometimes this is a warning

72sign that your API is incomplete, but it can be quite reasonable to

73require identifying credentials within an API and not provide a way to

74get them. It might also be reasonable to have some sample values that

75the user is expected to come up with.

77For those cases, override the `seed` in the `Config` with a

78list of seed values, suitably hashed:

80``` haskell

81defaultConfig { seed = [hashedDyn creds, hashedDyn userJwt]}

82```

84## why not servant-quickcheck?

86[servant-quickcheck](https://hackage.haskell.org/package/servant-quickcheck)

87is a great package and I've learned a lot from it. Unfortunately, as mentioned previously,

88there's a lot of the state space you just can't explore without context: modern webapps are

89full of pointer-like structures, whether they're URLs or database

90keys/uuids, and servant-quickcheck requires that you be able to generate

91these without context via Arbitrary.

93## limitations and future work

95Currently, the display of failing traces is pretty tragic, both in the

96formatting and in its non-minimality. This is pretty ticklish:

97arguably the right way to do this is to return a trace that we can

98also rerun, and let quickcheck or hedgehog a level up shrink it until

99it's satisfactorily short. In the interest of being useful earlier

100rather than later, I'm releasing v1.0 before I crack this particular

101nut. We do know which calls we made that led to the failing case, so

102we would want to show that distinction in a visible way: it's possible

103that other calls that don't have direct data dependencies were

104important, but we definitely know we need the direct data dependencies.

105

106The provenance stuff is a bit underbaked. It should at least pull a

107representation of the route chosen rather than just an integer index.

108

109It would also be nice to have a robust strategy for deriving recursive

110datatypes, or at least rejecting attempts to generate them that don't

111end in an infinite loop.

112

113Currently the `FlattenServer` instance for `:>` is quadratic. It would

114be nice to fix this but I lack the art.