unlock event if event trigger info is not found in cache (#4213)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

CHANGELOG.md

@@ -21,6 +21,7 @@ The order and collapsed state of columns is now persisted across page navigation
 - docs: add One-Click Render deployment guide (close #3683) (#4209)
 - server: reserved keywords in column references break parser (fix #3597) #3927
 - server: fix postgres specific error message that exposed database type on invalid query parameters (#4294)
+- server: fix an edge case where some events wouldn't be processed because of internal erorrs (#4213)
 
 ## `v1.2.0-beta.3`
 

server/src-lib/Hasura/Events/Lib.hs

@@ -259,7 +259,15 @@ processEventQueue logger logenv httpMgr pool getSchemaCache EventEngineCtx{..} =
       let meti = getEventTriggerInfoFromEvent cache e
       case meti of
         Nothing -> do
+          --  This rare error can happen in the following known cases:
+          --  i) schema cache is not up-to-date (due to some bug, say during schema syncing across multiple instances)
+          --  ii) the event trigger is dropped when this event was just fetched
           logQErr $ err500 Unexpected "table or event-trigger not found in schema cache"
+          liftIO . runExceptT $ Q.runTx pool (Q.RepeatableRead, Just Q.ReadWrite) $ do
+            currentTime <- liftIO getCurrentTime
+            -- For such an event, we unlock the event and retry after a minute
+            setRetry e (addUTCTime 60 currentTime)
+          >>= flip onLeft logQErr
         Just eti -> do
           let webhook = T.unpack $ wciCachedValue $ etiWebhookInfo eti
               retryConf = etiRetryConf eti