eeb84b0 · streamly · git.iko.soy

git.iko.soy/streamly

Some tweaks for encodeUtf8 perf

* Use a fast path yield point for ascii encoding. With single yield point decode/encode on a 500MB file takes 3 sec, with two yield points it takes 1.4 sec. In both cases the pipeline is fully fused with the help of the plugin. * bump up the spec-constr-recusrive limit We still need our GHC plugin for full fusion of decode/encode pipeline.

commiteeb84b009b3565acafa24431a1e652ccfaac8475 authorHarendra Kumar <harendra.kumar@gmail.com> dateThu 31 Oct 2019, 19:18 (7 years ago) parentb86b65f

3 files changed+7−4

compare with master

unifiedsplit

docs/Build.md+1−1 src/Streamly/Streams/StreamD.hs+5−2 streamly.cabal+1−1

docs/Build.md+1−1

@@ -2,7 +2,7 @@

33Recommended GHC options are:

5− `-O2 -fspec-constr-recursive=10 -fmax-worker-args=16`

5+ `-O2 -fspec-constr-recursive=16 -fmax-worker-args=16`

77`-fspec-constr-recursive` is needed for better stream fusion by enabling

88the `SpecConstr` optimization in more cases.

src/Streamly/Streams/StreamD.hs+5−2

@@ -3815,6 +3815,10 @@ data WList = WCons !Word8 !WList | WNil

38153815

38163816data EncodeState s = EncodeState s !WList

38173817

3818+-- More yield points improve performance, but I am not sure if they can cause

3819+-- too much code bloat or some trouble with fusion. So keeping only two yield

3820+-- points for now, one for the ascii chars (fast path) and one for all other

3821+-- paths (slow path).

38183822{-# INLINE_NORMAL encodeUtf8 #-}

38193823encodeUtf8 :: Monad m => Stream m Char -> Stream m Word8

38203824encodeUtf8 (Stream step state) = Stream step' (EncodeState state WNil)

@@ -3828,8 +3832,7 @@ encodeUtf8 (Stream step state) = Stream step' (EncodeState state WNil)

38283832 case ord c of

38293833 x

38303834 | x <= 0x7F ->

3831− Skip

3832− (EncodeState s (WCons (fromIntegral x) WNil))

3835+ Yield (fromIntegral x) (EncodeState s WNil)

38333836 | x <= 0x7FF -> Skip (EncodeState s (ord2 c))

38343837 | x <= 0xFFFF ->

38353838 if isSurrogate c

streamly.cabal+1−1

@@ -205,7 +205,7 @@ common compile-options

205205

206206common optimization-options

207207 ghc-options: -O2

208− -fspec-constr-recursive=10

208+ -fspec-constr-recursive=16

209209 -fmax-worker-args=16

210210

211211common threading-options