This fix does not work on ghc-8.0, because I do not understand
the 8.0 API in this instance. Could be resolved by looking
at annotations, but that really should not be necessary.
Split up annotations by top-level elements in one
go, instead of doing the filtering per top-level
element (which necessarily makes things quadratic,
or rather O(n*m) with n top-level elements and m
size of annotation map). The fixed version should
be O(log n * m), and log n is negligible.