@@ -1,101 +1,388 @@
11# effects-benchmarks
22
3−This is a set of memory and time benchmarks for the various Haskell effects libraries, including `fused-effects`, `polysemy`, `freer-simple`, and `extensible-effects`. Benchmarks for equivalent `mtl` and handwritten computations are included for comparison purposes. It uses Cabal's new Backpack feature to save a great deal of boilerplate code.
43
5−# Some results
4+## Results
65
7−On a 15-inch 2017 MacBook Pro (3.1GHz Intel i7, 16GB RAM), here are results for space benchmarks:
6+<details>
7+
8+<summary>Linux – Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz</summary>
89
910```
11+*** Time benchmarks ***
12+benchmarked Countdown/Put/fused-effects
13+time 13.54 ms (13.44 ms .. 13.67 ms)
14+ 0.999 R² (0.998 R² .. 1.000 R²)
15+mean 13.58 ms (13.54 ms .. 13.69 ms)
16+std dev 165.6 μs (94.38 μs .. 276.3 μs)
17+
18+benchmarked Countdown/Put/mtl
19+time 4.885 ms (4.777 ms .. 4.954 ms)
20+ 0.998 R² (0.993 R² .. 1.000 R²)
21+mean 4.995 ms (4.934 ms .. 5.227 ms)
22+std dev 333.6 μs (54.64 μs .. 696.6 μs)
23+variance introduced by outliers: 37% (moderately inflated)
24+
25+benchmarked Countdown/Put/polysemy
26+time 20.82 ms (20.58 ms .. 21.10 ms)
27+ 0.999 R² (0.997 R² .. 1.000 R²)
28+mean 20.82 ms (20.70 ms .. 21.00 ms)
29+std dev 359.1 μs (243.7 μs .. 578.8 μs)
30+
31+benchmarked Countdown/Put/freer-simple
32+time 20.18 ms (19.82 ms .. 20.56 ms)
33+ 0.998 R² (0.997 R² .. 0.999 R²)
34+mean 19.90 ms (19.56 ms .. 20.53 ms)
35+std dev 1.084 ms (524.8 μs .. 2.078 ms)
36+variance introduced by outliers: 22% (moderately inflated)
37+
38+benchmarked Countdown/Put/extensible-effects
39+time 22.01 ms (21.29 ms .. 22.72 ms)
40+ 0.997 R² (0.995 R² .. 0.999 R²)
41+mean 20.74 ms (19.73 ms .. 21.26 ms)
42+std dev 1.554 ms (611.3 μs .. 2.309 ms)
43+variance introduced by outliers: 30% (moderately inflated)
44+
45+benchmarked Countdown/Put/shallow
46+time 2.432 ms (2.390 ms .. 2.477 ms)
47+ 0.998 R² (0.997 R² .. 0.999 R²)
48+mean 2.439 ms (2.425 ms .. 2.456 ms)
49+std dev 47.42 μs (39.74 μs .. 62.97 μs)
50+
51+benchmarked Countdown/Put/effectful - static
52+time 555.5 μs (542.5 μs .. 567.2 μs)
53+ 0.995 R² (0.985 R² .. 1.000 R²)
54+mean 548.2 μs (544.0 μs .. 561.4 μs)
55+std dev 19.40 μs (5.420 μs .. 41.81 μs)
56+variance introduced by outliers: 12% (moderately inflated)
57+
58+benchmarked Countdown/Put/effectful - dynamic
59+time 2.729 ms (2.688 ms .. 2.777 ms)
60+ 0.998 R² (0.996 R² .. 0.999 R²)
61+mean 2.736 ms (2.722 ms .. 2.761 ms)
62+std dev 59.64 μs (42.50 μs .. 90.32 μs)
63+
64+benchmarked Countdown/Put+Exc/fused-effects
65+time 11.89 ms (11.76 ms .. 12.06 ms)
66+ 0.999 R² (0.999 R² .. 1.000 R²)
67+mean 11.93 ms (11.87 ms .. 12.05 ms)
68+std dev 199.3 μs (102.4 μs .. 324.7 μs)
69+
70+benchmarked Countdown/Put+Exc/mtl
71+time 5.196 ms (5.120 ms .. 5.299 ms)
72+ 0.998 R² (0.994 R² .. 0.999 R²)
73+mean 5.200 ms (5.169 ms .. 5.254 ms)
74+std dev 108.8 μs (69.46 μs .. 167.0 μs)
75+
76+benchmarked Countdown/Put+Exc/polysemy
77+time 29.24 ms (28.82 ms .. 29.79 ms)
78+ 0.999 R² (0.998 R² .. 1.000 R²)
79+mean 29.51 ms (29.26 ms .. 29.95 ms)
80+std dev 681.2 μs (452.7 μs .. 990.8 μs)
81+
82+benchmarked Countdown/Put+Exc/freer-simple
83+time 13.32 ms (12.65 ms .. 14.12 ms)
84+ 0.984 R² (0.974 R² .. 0.993 R²)
85+mean 11.50 ms (11.00 ms .. 11.99 ms)
86+std dev 1.209 ms (1.080 ms .. 1.435 ms)
87+variance introduced by outliers: 48% (moderately inflated)
88+
89+benchmarked Countdown/Put+Exc/extensible-effects
90+time 17.03 ms (15.01 ms .. 18.76 ms)
91+ 0.956 R² (0.928 R² .. 0.980 R²)
92+mean 13.62 ms (12.99 ms .. 14.50 ms)
93+std dev 1.750 ms (1.315 ms .. 2.083 ms)
94+variance introduced by outliers: 56% (severely inflated)
95+
96+benchmarked Countdown/Put+Exc/shallow
97+time 580.0 μs (475.1 μs .. 675.5 μs)
98+ 0.971 R² (0.958 R² .. 1.000 R²)
99+mean 573.0 μs (566.0 μs .. 584.6 μs)
100+std dev 14.90 μs (1.088 μs .. 19.80 μs)
101+
102+benchmarked Countdown/Put+Exc/effectful - static
103+time 588.1 μs (520.7 μs .. 679.1 μs)
104+ 0.974 R² (0.954 R² .. 0.998 R²)
105+mean 602.3 μs (596.0 μs .. 612.8 μs)
106+std dev 13.75 μs (5.999 μs .. 17.97 μs)
107+
108+benchmarked Countdown/Put+Exc/effectful - dynamic
109+time 2.769 ms (2.747 ms .. 2.806 ms)
110+ 0.997 R² (0.992 R² .. 1.000 R²)
111+mean 2.796 ms (2.783 ms .. 2.832 ms)
112+std dev 48.48 μs (21.03 μs .. 95.12 μs)
113+
114+benchmarked HTTP/fused-effects
115+time 1.937 ms (1.916 ms .. 1.976 ms)
116+ 0.997 R² (0.993 R² .. 1.000 R²)
117+mean 1.937 ms (1.928 ms .. 1.957 ms)
118+std dev 29.94 μs (11.47 μs .. 54.28 μs)
119+
120+benchmarked HTTP/polysemy
121+time 34.56 ms (33.19 ms .. 36.29 ms)
122+ 0.995 R² (0.992 R² .. 1.000 R²)
123+mean 33.67 ms (33.40 ms .. 34.39 ms)
124+std dev 841.1 μs (347.0 μs .. 1.492 ms)
125+
126+benchmarked HTTP/extensible-effects
127+time 19.92 ms (18.98 ms .. 21.52 ms)
128+ 0.983 R² (0.970 R² .. 0.991 R²)
129+mean 17.74 ms (16.85 ms .. 18.62 ms)
130+std dev 1.942 ms (1.676 ms .. 2.228 ms)
131+variance introduced by outliers: 48% (moderately inflated)
132+
133+benchmarked HTTP/Deep embedding
134+time 28.14 ms (25.55 ms .. 30.55 ms)
135+ 0.980 R² (0.966 R² .. 0.993 R²)
136+mean 24.73 ms (22.59 ms .. 26.22 ms)
137+std dev 3.694 ms (2.445 ms .. 4.726 ms)
138+variance introduced by outliers: 61% (severely inflated)
139+
140+benchmarking HTTP/Shallow embedding ... took 11.97 s, total 40147 iterations
141+benchmarked HTTP/Shallow embedding
142+time 22.38 μs (21.23 μs .. 23.27 μs)
143+ 0.994 R² (0.977 R² .. 0.999 R²)
144+mean 25.00 μs (24.76 μs .. 25.35 μs)
145+std dev 484.0 ns (352.7 ns .. 601.7 ns)
146+
147+benchmarked HTTP/freer-simple
148+time 19.35 ms (18.52 ms .. 20.60 ms)
149+ 0.980 R² (0.959 R² .. 0.992 R²)
150+mean 17.17 ms (16.27 ms .. 18.04 ms)
151+std dev 1.945 ms (1.653 ms .. 2.326 ms)
152+variance introduced by outliers: 45% (moderately inflated)
153+
154+benchmarking HTTP/effectful - static ... took 11.95 s, total 38223 iterations
155+benchmarked HTTP/effectful - static
156+time 25.03 μs (16.61 μs .. 35.70 μs)
157+ 0.838 R² (0.740 R² .. 0.999 R²)
158+mean 25.57 μs (24.80 μs .. 27.01 μs)
159+std dev 1.644 μs (121.0 ns .. 2.127 μs)
160+variance introduced by outliers: 18% (moderately inflated)
161+
162+benchmarked HTTP/effectful - dynamic
163+time 4.058 ms (3.662 ms .. 4.753 ms)
164+ 0.918 R² (0.847 R² .. 0.997 R²)
165+mean 3.886 ms (3.773 ms .. 4.122 ms)
166+std dev 397.4 μs (161.9 μs .. 614.5 μs)
167+variance introduced by outliers: 50% (severely inflated)
168+
169+*** Space benchmarks (these will take some time to run) ***
170+
10171Countdown
11172
12− Case Allocated
13− fused-effects 40
14− mtl 40
15− polysemy 1,304
16− freer-simple 1,096
17− extensible-effects 1,304
18− shallow 232
173+ Case Allocated GCs
174+ fused-effects 2,512 0
175+ mtl 792 0
176+ polysemy 4,184 0
177+ freer-simple 1,120 0
178+ extensible-effects 1,264 0
179+ shallow 232 0
180+ effectful - static 4,000,488 3
181+ effectful - dynamic 14,401,376 13
19182
20183Countdown + exc
21184
22− Case Allocated GCs
23− fused-effects 336
24− mtl 336
25− polysemy 792
26− freer-simple 864
27− extensible-effects 984
28− shallow 320
185+ Case Allocated GCs
186+ fused-effects 1,080 0
187+ mtl 608 0
188+ polysemy 4,144 0
189+ freer-simple 576 0
190+ extensible-effects 688 0
191+ shallow 0 0
192+ effectful - static 4,002,080 3
193+ effectful - dynamic 14,403,832 13
29194```
30195
31−Time (`State`-based countdown):
196+</details>
197+
198+<details>
199+
200+<summary>macOS – MacBook Pro (13-inch, 2018, Four Thunderbolt 3 Ports) – 2.3 GHz Quad-Core Intel Core i5</summary>
32201
33202```
203+*** Time benchmarks ***
34204benchmarked Countdown/Put/fused-effects
35−time 5.692 μs (5.570 μs .. 5.835 μs)
205+time 13.54 ms (13.44 ms .. 13.67 ms)
206+ 0.999 R² (0.998 R² .. 1.000 R²)
207+mean 13.58 ms (13.54 ms .. 13.69 ms)
208+std dev 165.6 μs (94.38 μs .. 276.3 μs)
36209
37210benchmarked Countdown/Put/mtl
38−time 5.500 μs (5.386 μs .. 5.719 μs)
39−variance introduced by outliers: 35% (moderately inflated)
211+time 4.885 ms (4.777 ms .. 4.954 ms)
212+ 0.998 R² (0.993 R² .. 1.000 R²)
213+mean 4.995 ms (4.934 ms .. 5.227 ms)
214+std dev 333.6 μs (54.64 μs .. 696.6 μs)
215+variance introduced by outliers: 37% (moderately inflated)
40216
41217benchmarked Countdown/Put/polysemy
42−time 1.903 ms (1.845 ms .. 1.954 ms)
43−variance introduced by outliers: 72% (severely inflated)
218+time 20.82 ms (20.58 ms .. 21.10 ms)
219+ 0.999 R² (0.997 R² .. 1.000 R²)
220+mean 20.82 ms (20.70 ms .. 21.00 ms)
221+std dev 359.1 μs (243.7 μs .. 578.8 μs)
44222
45223benchmarked Countdown/Put/freer-simple
46−time 1.324 ms (1.282 ms .. 1.361 ms)
224+time 20.18 ms (19.82 ms .. 20.56 ms)
225+ 0.998 R² (0.997 R² .. 0.999 R²)
226+mean 19.90 ms (19.56 ms .. 20.53 ms)
227+std dev 1.084 ms (524.8 μs .. 2.078 ms)
228+variance introduced by outliers: 22% (moderately inflated)
47229
48230benchmarked Countdown/Put/extensible-effects
49−time 1.383 ms (1.343 ms .. 1.422 ms)
231+time 22.01 ms (21.29 ms .. 22.72 ms)
232+ 0.997 R² (0.995 R² .. 0.999 R²)
233+mean 20.74 ms (19.73 ms .. 21.26 ms)
234+std dev 1.554 ms (611.3 μs .. 2.309 ms)
235+variance introduced by outliers: 30% (moderately inflated)
50236
51237benchmarked Countdown/Put/shallow
52−time 275.1 μs (270.1 μs .. 281.6 μs)
53−```
238+time 2.432 ms (2.390 ms .. 2.477 ms)
239+ 0.998 R² (0.997 R² .. 0.999 R²)
240+mean 2.439 ms (2.425 ms .. 2.456 ms)
241+std dev 47.42 μs (39.74 μs .. 62.97 μs)
54242
55−Time (`State` + `Error` countdown):
243+benchmarked Countdown/Put/effectful - static
244+time 555.5 μs (542.5 μs .. 567.2 μs)
245+ 0.995 R² (0.985 R² .. 1.000 R²)
246+mean 548.2 μs (544.0 μs .. 561.4 μs)
247+std dev 19.40 μs (5.420 μs .. 41.81 μs)
248+variance introduced by outliers: 12% (moderately inflated)
249+
250+benchmarked Countdown/Put/effectful - dynamic
251+time 2.729 ms (2.688 ms .. 2.777 ms)
252+ 0.998 R² (0.996 R² .. 0.999 R²)
253+mean 2.736 ms (2.722 ms .. 2.761 ms)
254+std dev 59.64 μs (42.50 μs .. 90.32 μs)
56255
57−```
58256benchmarked Countdown/Put+Exc/fused-effects
59−time 9.829 μs (9.645 μs .. 10.04 μs)
257+time 11.89 ms (11.76 ms .. 12.06 ms)
258+ 0.999 R² (0.999 R² .. 1.000 R²)
259+mean 11.93 ms (11.87 ms .. 12.05 ms)
260+std dev 199.3 μs (102.4 μs .. 324.7 μs)
60261
61262benchmarked Countdown/Put+Exc/mtl
62−time 9.968 μs (9.779 μs .. 10.15 μs)
263+time 5.196 ms (5.120 ms .. 5.299 ms)
264+ 0.998 R² (0.994 R² .. 0.999 R²)
265+mean 5.200 ms (5.169 ms .. 5.254 ms)
266+std dev 108.8 μs (69.46 μs .. 167.0 μs)
63267
64268benchmarked Countdown/Put+Exc/polysemy
65−time 418.1 μs (408.9 μs .. 427.1 μs)
269+time 29.24 ms (28.82 ms .. 29.79 ms)
270+ 0.999 R² (0.998 R² .. 1.000 R²)
271+mean 29.51 ms (29.26 ms .. 29.95 ms)
272+std dev 681.2 μs (452.7 μs .. 990.8 μs)
66273
67274benchmarked Countdown/Put+Exc/freer-simple
68−time 1.157 ms (1.120 ms .. 1.199 ms)
275+time 13.32 ms (12.65 ms .. 14.12 ms)
276+ 0.984 R² (0.974 R² .. 0.993 R²)
277+mean 11.50 ms (11.00 ms .. 11.99 ms)
278+std dev 1.209 ms (1.080 ms .. 1.435 ms)
279+variance introduced by outliers: 48% (moderately inflated)
69280
70281benchmarked Countdown/Put+Exc/extensible-effects
71−time 1.232 ms (1.212 ms .. 1.262 ms)
282+time 17.03 ms (15.01 ms .. 18.76 ms)
283+ 0.956 R² (0.928 R² .. 0.980 R²)
284+mean 13.62 ms (12.99 ms .. 14.50 ms)
285+std dev 1.750 ms (1.315 ms .. 2.083 ms)
286+variance introduced by outliers: 56% (severely inflated)
72287
73288benchmarked Countdown/Put+Exc/shallow
74−time 55.12 μs (54.34 μs .. 56.29 μs)
75−```
289+time 580.0 μs (475.1 μs .. 675.5 μs)
290+ 0.971 R² (0.958 R² .. 1.000 R²)
291+mean 573.0 μs (566.0 μs .. 584.6 μs)
292+std dev 14.90 μs (1.088 μs .. 19.80 μs)
76293
77−Time (custom HTTP DSL):
294+benchmarked Countdown/Put+Exc/effectful - static
295+time 588.1 μs (520.7 μs .. 679.1 μs)
296+ 0.974 R² (0.954 R² .. 0.998 R²)
297+mean 602.3 μs (596.0 μs .. 612.8 μs)
298+std dev 13.75 μs (5.999 μs .. 17.97 μs)
299+
300+benchmarked Countdown/Put+Exc/effectful - dynamic
301+time 2.769 ms (2.747 ms .. 2.806 ms)
302+ 0.997 R² (0.992 R² .. 1.000 R²)
303+mean 2.796 ms (2.783 ms .. 2.832 ms)
304+std dev 48.48 μs (21.03 μs .. 95.12 μs)
78305
79−```
80306benchmarked HTTP/fused-effects
81−time 8.311 μs (8.060 μs .. 8.592 μs)
307+time 1.937 ms (1.916 ms .. 1.976 ms)
308+ 0.997 R² (0.993 R² .. 1.000 R²)
309+mean 1.937 ms (1.928 ms .. 1.957 ms)
310+std dev 29.94 μs (11.47 μs .. 54.28 μs)
82311
83312benchmarked HTTP/polysemy
84−time 14.99 ms (14.26 ms .. 15.80 ms)
313+time 34.56 ms (33.19 ms .. 36.29 ms)
314+ 0.995 R² (0.992 R² .. 1.000 R²)
315+mean 33.67 ms (33.40 ms .. 34.39 ms)
316+std dev 841.1 μs (347.0 μs .. 1.492 ms)
85317
86318benchmarked HTTP/extensible-effects
87−time 1.612 ms (1.578 ms .. 1.654 ms)
319+time 19.92 ms (18.98 ms .. 21.52 ms)
320+ 0.983 R² (0.970 R² .. 0.991 R²)
321+mean 17.74 ms (16.85 ms .. 18.62 ms)
322+std dev 1.942 ms (1.676 ms .. 2.228 ms)
323+variance introduced by outliers: 48% (moderately inflated)
88324
89325benchmarked HTTP/Deep embedding
90−time 2.027 ms (1.876 ms .. 2.184 ms)
326+time 28.14 ms (25.55 ms .. 30.55 ms)
327+ 0.980 R² (0.966 R² .. 0.993 R²)
328+mean 24.73 ms (22.59 ms .. 26.22 ms)
329+std dev 3.694 ms (2.445 ms .. 4.726 ms)
330+variance introduced by outliers: 61% (severely inflated)
91331
332+benchmarking HTTP/Shallow embedding ... took 11.97 s, total 40147 iterations
92333benchmarked HTTP/Shallow embedding
93−time 15.94 μs (15.49 μs .. 16.46 μs)
334+time 22.38 μs (21.23 μs .. 23.27 μs)
335+ 0.994 R² (0.977 R² .. 0.999 R²)
336+mean 25.00 μs (24.76 μs .. 25.35 μs)
337+std dev 484.0 ns (352.7 ns .. 601.7 ns)
94338
95339benchmarked HTTP/freer-simple
96−time 1.478 ms (1.443 ms .. 1.515 ms)
97−```
340+time 19.35 ms (18.52 ms .. 20.60 ms)
341+ 0.980 R² (0.959 R² .. 0.992 R²)
342+mean 17.17 ms (16.27 ms .. 18.04 ms)
343+std dev 1.945 ms (1.653 ms .. 2.326 ms)
344+variance introduced by outliers: 45% (moderately inflated)
345+
346+benchmarking HTTP/effectful - static ... took 11.95 s, total 38223 iterations
347+benchmarked HTTP/effectful - static
348+time 25.03 μs (16.61 μs .. 35.70 μs)
349+ 0.838 R² (0.740 R² .. 0.999 R²)
350+mean 25.57 μs (24.80 μs .. 27.01 μs)
351+std dev 1.644 μs (121.0 ns .. 2.127 μs)
352+variance introduced by outliers: 18% (moderately inflated)
353+
354+benchmarked HTTP/effectful - dynamic
355+time 4.058 ms (3.662 ms .. 4.753 ms)
356+ 0.918 R² (0.847 R² .. 0.997 R²)
357+mean 3.886 ms (3.773 ms .. 4.122 ms)
358+std dev 397.4 μs (161.9 μs .. 614.5 μs)
359+variance introduced by outliers: 50% (severely inflated)
98360
99−# Future work
361+*** Space benchmarks (these will take some time to run) ***
362+
363+Countdown
364+
365+ Case Allocated GCs
366+ fused-effects 2,512 0
367+ mtl 792 0
368+ polysemy 4,184 0
369+ freer-simple 1,120 0
370+ extensible-effects 1,264 0
371+ shallow 232 0
372+ effectful - static 4,000,488 3
373+ effectful - dynamic 14,401,376 13
374+
375+Countdown + exc
376+
377+ Case Allocated GCs
378+ fused-effects 1,080 0
379+ mtl 608 0
380+ polysemy 4,144 0
381+ freer-simple 576 0
382+ extensible-effects 688 0
383+ shallow 0 0
384+ effectful - static 4,002,080 3
385+ effectful - dynamic 14,403,832 13
386+```
100387
101−I am going to make this emit pretty graphs, or at least graphable .csv files, someday.
388+</details>