in malloc, refuse to use brk if it grows into stack

the linux/nommu fdpic ELF loader sets up the brk range to overlap
entirely with the main thread's stack (but growing from opposite
ends), so that the resulting failure mode for malloc is not to return
a null pointer but to start returning pointers to memory that overlaps
with the caller's stack. needless to say this extremely dangerous and
makes brk unusable.

since it's non-trivial to detect execution environments that might be
affected by this kernel bug, and since the severity of the bug makes
any sort of detection that might yield false-negatives unsafe, we
instead check the proximity of the brk to the stack pointer each time
the brk is to be expanded. both the main thread's stack (where the
real known risk lies) and the calling thread's stack are checked. an
arbitrary gap distance of 8 MB is imposed, chosen to be larger than
linux default main-thread stack reservation sizes and larger than any
reasonable stack configuration on nommu.

the effeciveness of this patch relies on an assumption that the amount
by which the brk is being grown is smaller than the gap limit, which
is always true for malloc's use of brk. reliance on this assumption is
why the check is being done in malloc-specific code and not in __brk.

src/malloc/malloc.c

@@ -152,6 +152,14 @@ void __dump_heap(int x)
 }
 #endif
 
+static int is_near_stack(uintptr_t b)
+{
+	const uintptr_t c = 8<<20;
+	uintptr_t a = (uintptr_t)libc.auxv;
+	uintptr_t d = (uintptr_t)&b;
+	return a-b<=c || d-b<=c;
+}
+
 static struct chunk *expand_heap(size_t n)
 {
 	static int init;
@@ -174,7 +182,7 @@ static struct chunk *expand_heap(size_t n)
 	new = mal.brk + n + SIZE_ALIGN + PAGE_SIZE - 1 & -PAGE_SIZE;
 	n = new - mal.brk;
 
-	if (__brk(new) != new) {
+	if (is_near_stack(mal.brk) || __brk(new) != new) {
 		size_t min = (size_t)PAGE_SIZE << mal.mmap_step/2;
 		n += -n & PAGE_SIZE-1;
 		if (n < min) n = min;