fix(streaming): Data Truncation Issue in useHandleStream Function (#2079)

**Description**
This PR introduces modifications to the useHandleStream function in the
useHandleStream.ts file, aiming to resolve the issue of data truncation
leading to incomplete display of Q&A on the front end. By optimizing
data handling logic and enhancing error management, this change ensures
the integrity of data streams and accurate display on the front end.

**Changes Made**
Introduced a variable incompleteData to store and process incomplete
data fragments.
Implemented logic to concatenate incomplete data with new chunks and
parse them once complete.
Added error handling to gracefully manage the parsing of incomplete data
and data strings.
Improved the handling of the first data chunk, enhancing the coherence
of data processing.
**Motivation and Context**
The primary motivation for these changes is to address the issue of
incomplete front-end display, caused by data stream truncation under
unstable network conditions. This optimization is intended to enhance
the stability and reliability of the application when handling large
volumes of real-time data streams, ensuring that the user interface
correctly displays complete Q&A information.

**Checklist before requesting a review**
 My code follows the style guidelines of this project.
 I have performed a self-review of my own code.
 I have commented in hard-to-understand areas of my code.
 I have made corresponding changes to the documentation (if applicable).
I have added tests that prove my fix is effective or that my feature
works (if applicable).
 New and existing unit tests pass locally with my changes.
Any dependent changes have been merged and published in downstream
modules (if applicable).
**Screenshots (if appropriate)**

![image](https://github.com/StanGirard/quivr/assets/80630709/fb4f4c7f-5541-418c-8d9a-2cdc8ea641b9)

Co-authored-by: qcloud <ubuntu@localhost.localdomain>
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>

frontend/app/chat/[chatId]/hooks/useHandleStream.ts

@@ -12,11 +12,24 @@ export const useHandleStream = () => {
   ): Promise<void> => {
     const decoder = new TextDecoder("utf-8");
     let isFirstChunk = true;
+    let incompleteData = "";
 
     const handleStreamRecursively = async () => {
       const { done, value } = await reader.read();
 
       if (done) {
+
+	if (incompleteData !== "") {
+          // Try to parse any remaining incomplete data
+
+          try {
+            const parsedData = JSON.parse(incompleteData) as ChatMessage;
+            updateStreamingHistory(parsedData);
+          } catch (e) {
+            console.error("Error parsing incomplete data", e);
+          }
+        }
+
         return;
       }
 
@@ -25,15 +38,30 @@ export const useHandleStream = () => {
         onFirstChunk();
       }
 
-      const dataStrings = decoder
-        .decode(value)
+      // Concatenate incomplete data with new chunk
+      const rawData = incompleteData + decoder.decode(value, { stream: true });
+
+      console.log("Raw data before cleaning:", rawData);
+
+      const dataStrings = rawData
         .trim()
         .split("data: ")
         .filter(Boolean);
 
-      dataStrings.forEach((data) => {
-        const parsedData = JSON.parse(data) as ChatMessage;
-        updateStreamingHistory(parsedData);
+      dataStrings.forEach((data, index, array) => {
+        if (index === array.length - 1 && !data.endsWith("\n")) {
+          // Last item and does not end with a newline, save as incomplete
+          incompleteData = data;
+
+          return;
+        }
+
+        try {
+          const parsedData = JSON.parse(data) as ChatMessage;
+          updateStreamingHistory(parsedData);
+        } catch (e) {
+          console.error("Error parsing data string", e);
+        }
       });
 
       await handleStreamRecursively();
@@ -46,3 +74,4 @@ export const useHandleStream = () => {
     handleStream,
   };
 };
+