We need to check below top level too, since there could be holes that we're happy to resolve by searching. The linearity test added illustrates a place where this is needed.
