kanaka/mal
1
1
Fork 0
mirror of https://github.com/kanaka/mal.git synced 2024-08-16 17:20:23 +03:00
Code Issues Projects Releases Wiki Activity

runtest: Treat test timeouts as fatal

Browse Source 
If a test times out, that means it hasn't returned a prompt within
reasonable time.  Continuing to type at it is unlikely to be helpful,
since the lack of a prompt indicates that the subprocess has hung, so
subsequent tests are likely to time out as well.

This is particularly annoying in step 5, where the line that usually
times out, "(def! res2 (sum2 10000 0))" has its output ignored, so a
timeout is treated as success, meaning that runtest proceeds to the next
test, which of course also times out.
This commit is contained in:
Ben Harris 2020-06-01 15:08:10 +01:00
parent 13404d9173
commit 9d7fb0fd8d
1 changed files with 7 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
runtest.py
View File

@ -264,6 +264,9 @@ fail_cnt = 0
soft_fail_cnt = 0
failures = []
class TestTimeout(Exception):
pass
while t.next():
if args.deferrable == False and t.deferrable:
log(t.deferrable)
@ -293,7 +296,10 @@ while t.next():
res = r.read_to_prompt(['\r\n[^\s()<>]+> ', '\n[^\s()<>]+> '],
timeout=args.test_timeout)
#print "%s,%s,%s" % (idx, repr(p.before), repr(p.after))
if (t.ret == "" and t.out == ""):
if (res == None):
log(" -> TIMEOUT (line %d)" % t.line_num)
raise TestTimeout("TIMEOUT (line %d)" % t.line_num)
elif (t.ret == "" and t.out == ""):
log(" -> SUCCESS (result ignored)")
pass_cnt += 1
elif (re.search(expects[0], res, re.S) or