The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Protecting heap/stack

Is there any techniques or resources available that show how to protect your heap/stack from 3rd party library providers functions? For example their code having dangling pointers that could possibly write to a dword or word into your application memory. Or having access violations.

This is difficult if you're statically linking into your executable because its part of the same code base and application process. The other alternative is loading the library into a DLL into its own process.
Entity Send private email
Tuesday, September 18, 2007
I gotta ask the obvious: Why on earth would you be using such tools if they blast your data? I'd be looking for a new vendor if I were you.
Wednesday, September 19, 2007
I agree in regards to finding another vender and I would find another if this situation did present itself. But I'm interested to know if their exists any technique that you can use, to prevent dangling pointers from a 3rd party provider from touching or even attempting touching your application heap/stack. eg.. sand boxing the thing.

For example, you call a function in a 3rd party library and its void, and it takes no parameters. It just does simple calculations. But internally it self destructs, completely writing to random memory locations that may be valid/invalid. This could cause an access violation or read access violation or anything. The only logical conclusion is to terminate the application, even with exceptions.

So i was wondering, is their anyway to protect your memory from tampering from 3rd party library providers? Sort of how windows uses Guard pages to prevent writing above the current top of the stack.
Entity Send private email
Wednesday, September 19, 2007
The simplest option is to make a minimal interface to that untrusted library in a separate process which uses some form of cross process communication to get the results you need back to your process.

As the processes have separate address spaces one can't affect the other (short of a call to WriteProcessMemory() or similar).

If you're just worried about stack trashing then a new thread is probably sufficient as each thread has its own stack.
Wednesday, September 19, 2007
Even if you are call the library from another process, how do you detect corruption?
Wednesday, September 19, 2007
First, whether you use static or dynamic linking to third party libraries, they are still part of your address space, and wild pointers will be able to cause havoc. The only way to get the library in another address space is to write a separate program that uses the library and communicate between that program and your main program via IPC.

Second, guard pages don't really help anything, they just make people feel better about their stupid OS design. They may catch some errors, but can never catch all errors (unlike separate address spaces). A wild pointer is even worse than heap/stack collision, because the wild pointer can access any part of your address space at any time without touching the intervening addresses. Guard pages only work when you have progressive sequential access to the address space, so you can trigger a violation of the guard page before the heap and stack actually overlap.

Finally, as to detecting corruption, unless you have some very well defined interface protocol, you are probably looking at an avatar of the halting problem (i.e. it's not solvable). Seriously, if you can't rely on this third party library not to throw wild pointers and you can't even rely on it to return correct data, why the hell are you using it: what is it good for? I sure hope you didn't may money for this piece of crap.
Jeffrey Dutky Send private email
Wednesday, September 19, 2007
Assuming you're pretty sure the library is the problem, and you're pretty much stuck with it, valgrind ( ) may be able to help you (linux only, though).

If you're on Windows, the old BoundsChecker was the bomb (dont know how it's done since Compuware bought the company), or you can try Purify.
BillAtHRST Send private email
Wednesday, September 19, 2007

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz