[tor-dev] TBB Memory Allocator choice fingerprint implications

[Tom Ritter]

[Replying to both emails]

My hope is that most of this stems from my cursory work in replying
and a general misunderstanding.

I've seen people advocating for replacing the memory allocator in Tor
Browser since I started the effort five years ago here:
https://github.com/iSECPartners/publications/blob/master/reports/Tor%20Browser%20Bundle/Tor%20Browser%20Bundle%20-%20iSEC%20Deliverable%201.3.pdf
My replies were primarily focused on replying to the advocations I've
seen in the past, and the misconceptions I've seen repeated there
(like LD_PRELOAD).

On Wed, 21 Aug 2019 at 12:14, Daniel Micay <danielmicay at gmail.com> wrote:
>
> On Sat, Aug 17, 2019 at 09:17:40PM +0000, Tom Ritter wrote:
> > On Sat, 17 Aug 2019 at 15:06, procmem at riseup.net <procmem at riseup.net> wrote:
> > > Question for the Tor Browser experts. Do you know if it is possible to
> > > remotely fingerprint the browser based on the memory allocator it is
> > > using? (via JS or content rendering)
> >
> > Fingerprint what aspect of the browser/machine?
>
> Performance-based fingerprinting of the browser can easily differentiate
> between using a different malloc implementation. That can already obtain
> a lot of fingerprinting information about the hardware and OS so this
> may not actually matter much, but it's entirely possible.


Agreed.


> > > We are thinking of switching Tor Browser to use the minimalist and
> > > security oriented hardened_malloc written by Daniel Micay. Thanks.
> >
> > I wouldn't advise giving up partitioning for.... what exactly? What
> > features does this allocator have that 68's jemalloc doesn't?
>
> The hardened_malloc allocator heavily uses partitioning, and has a much
> stronger implementation than the very weak approach in mozjemalloc. It
> statically reserves memory regions for metadata and a dedicated region
> for each arena, with each size class receiving a dedicated sub-region
> within the arena. These sub-regions are placed within their own guard
> region and each have a high entropy random base. It never mixes address
> space between these regions or reuses the address space. This is much
> different than what you call 'partitioning' in mozjemalloc which does
> not really qualify. What you're talking about is mozjemalloc exposing an
> API for choosing the arena from the code, which can certainly be done
> with hardened_malloc too. However, in mozjemalloc, the address space for
> different arenas is mixed together and reused between them. It's really
> a stretch to call this partitioning, and it doesn't have the baseline
> separation of size classes either.
>
> People can read about hardened_malloc in the README:
>
> https://github.com/GrapheneOS/hardened_malloc/blob/master/README.md#hardened-malloc
>
> I don't know why you're making the misleading claim that people would
> need to give up partitioning.

My reply about giving up partitioning here (which I clarify in the
second email) is that using LD_PRELOAD will negate partitioning.

> It's also really a stretch to call what
> Mozilla is doing in mozjemalloc partitioning in the first place, so your
> claim is really quite backwards...

Let me start by asserting emphatically and agreeing that
hardened_malloc is a much stronger allocator than mozjemalloc or
jemalloc. PartitionAlloc is also for that matter.

Mozilla's partitioning has always been focused on preventing the
Use-After-Free problems that have plagued Firefox for years. Allocate
a DOM object, retain a pointer to it, free it, replace it with an
Arraybuffer, align the vtable entry, redirect execution. We allocate
ArrayBuffer contents and strings in separate arenas to avoid the
immediate reuse of these bytes.


On Wed, 21 Aug 2019 at 13:40, Daniel Micay <danielmicay at gmail.com> wrote:
>
> On Mon, Aug 19, 2019 at 04:09:36PM +0000, Tom Ritter wrote:
> > Okay I'm going to try and clear up a lot of misconceptions and stuff
> > here.  I don't own Firefox's memory allocator but I have worked in it,
> > recently, and am one of the people who are working on hardening it.
>
> This makes it clear why you're spreading misinformation. You're going
> out of your way to make false and misleading claims about mozjemalloc
> and hardened_malloc, particularly your bogus comparisons between them.

My comparisons were rushed and cursory. I apologize and I'll clarify
my conclusions at the end of the email.

> Bolting on a few weak implementations of hardening features to an
> allocator inherently very friendly to memory corruption exploitation
> does not make it anything close to being hardened allocator, sorry.

Fair.

> > Firefox's memory allocator is not jemalloc. It's probably better
> > referred to as mozjemalloc. We forked jemalloc and have been improving
> > it (at least from our perspective.) Any analysis of or comparison to
> > jemalloc is - at this point - outdated and should be redone from
> > scratch against mozjemalloc on mozilla-central.
>
> It's not particularly different and the comparison isn't outdated.

As above, the comparisons I were referring to were the five year
document I wrote, and past trac tickets and threads here.

> > LD_PRELOAD='/path/to/libhardened_malloc.so' /path/to/program will do
> > nothing or approximately nothing. mozjemalloc uses mmap and low level
> > allocation tools to create chunks of memory to be used by its internal
> > memory allocator. To successfully replace Firefox memory allocator you
> > should either use LD_PRELOAD _with_ a --disable-jemalloc build OR
> > Firefox's replace_malloc functionality:
> > https://searchfox.org/mozilla-central/source/memory/build/replace_malloc.h
>
> LD_PRELOAD is not how hardened_malloc is supposed to be used outside of
> testing it anyway. It's meant to be integrated in libc, in which case
> --disable-jemalloc would be enough, although it can also be integrated
> into a specific program. That doesn't sidestep the importance of doing
> other hardening in libc and the rest of the system though.

We agree on this.

> > Fingerprinting: It is most likely possible to be creative enough to
> > fingerprint what memory allocator is used. If we were to choose from
> > different allocators at runtime, I don't think that fingerprinting is
> > the worst thing open to us - it seems likely that any attacker who
> > does such a attack could also fingerprinting your CPU speed, RAM, and
> > your ASLR base addresses which depending on OS might not change until
> > reboot.
>
> They can obtain a lot more than just information about the hardware. A
> lot of the hardware and OS information that fingerprinting mitigations
> try to hide are leaked via performance measurements. It can also leak a
> lot of data from within the browser.

Also agreed. I wasn't mentioning things like the broad class of
pixel-stealing attacks, or the user activity-class of attacks. Which
is also the reason I advocate for getting Fuzzyfox into a usable
state.

> > The only reason I can think of to choose between allocators at runtime
> > is to introduce randomness into the allocation strategy. An attacker
> > relying on a blind overwrite may not be able to position their
> > overwrite reliably AND it has the cause the process to crash otherwise
> > they can just try again.
>
> The hardened_malloc design provides far more than randomization

Of course. But the randomization I'm referring to isn't the
randomization inside the allocator, it's the random choice of *which*
allocator to use.

> > Allocators can introduce randomness themselves, you don't need to
> > choose between allocators to do that.
>
> This is not something that can simply be bolted onto an existing
> allocator design with a good approach. It needs to be more heavily
> integrated into the design, and the same applies to an even greater
> extent to more important security features than weak fine-grained
> randomization.

I agree.


> > In virtually all browser exploits we have seen recently the attacker
> > creates exploitation primitives that allow partial memory read/write
> > and then full memory read/write. Randomness introduced is bypassed and
> > ineffective. I've seen a general trend away from randomness for this
> > purpose. The exception is when the attacker is heavily constrained -
> > like exploiting over IPC or in a network protocol. Not when the
> > attacker has a full Javascript execution environment available to
> > them.
> >
> > When exploiting a memory corruption vulnerability, you can target the
> > application's memory (meaning, target a DOM object or an ArrayBuffer)
> > or you can target the memory allocator's metadata. While allocator
> > metadata corruption was popular in the past, I haven't seen it used
> > recently.
>
> The importance of out-of-line metadata is far beyond simply preventing
> exploitation through the allocator metadata. It's crucial for a hardened
> allocator to have a reliable source of information about allocations
> without trusting data read from freed allocations or next to the memory
> allocations.

My understanding of this statement is that the metadata must be
protected against non-security-related corruption; if I'm
misunderstanding I'm happy to be educated.

> > Okay all that out of the way, let's talk about allocators.
> >
> > I skimmed https://github.com/GrapheneOS/hardened_malloc and it looks
> > like it has:

I can see how my skimming, and simplistic bullet points, could be
insulting to someone who has invested such a great deal of work into
their project. I'm sorry; it's too easy to forget that on the other
side of text is a person who deserves our consideration, and that on
the other side of code is a significant investment of effort.

> >  - support for arenas
>
> This is a completely dishonest and ridiculous misrepresentation of the
> design and security properties laid out in that README. People should
> read it for themselves and they'll see that your attempt at spinning
> misinformation about it is a complete joke.

I'd intended this as a simplified response to the expected points I'd
seen brought up in the past. I apologize if it came across as
intentionally dishonest.

> > mozjemalloc:
> >  - arenas (we call them partitions)
>
> Unlike hardened_malloc, they're mixed together and address space is
> reused between them rather than having strong isolation. The approach in
> hardened_malloc also partitions each size class. Calling the mozjemalloc
> arenas partitions as if it's an implementation of a security feature is
> a joke.

As I mentioned before, our implementation of partitions is focused on
preventing the immediate reuse of bytes to negate the common UAF
pattern we have seen for years. hardened_malloc's is clearly stronger.

> >  - randomization (support for, not enabled by default due to limited
> > utility, but improvements coming)
> >  - double free protection
> >  - zero-filling
> > In Progress:
> >  - we're actively working on guard regions
>
> As covered above, you're being misleading with each of these points, by
> portraying these things as something black and white that the allocator
> either has or doesn't have

That's a fair criticism.

> In particular, talking about randomization and guard regions as if this
> is a matter of having them or not having them is ridiculous. There is
> more to invalid free detection than double free detection and how well
> it works has a lot of variation. It can be deterministic detection like
> the hardened_malloc implementation, or probabilistic detection that an
> attacker could much more easily bypass. The reuse of freed allocations
> also matters a lot, since once it's handed out again, a free based on
> a past generation allocation won't be considered invalid, despite it
> being wrong and dangerous. This is why the design of memory allocation
> reuse and quarantines matters so much. The documentation on thread
> caching in the hardened_malloc README elaborates on why that's not
> compatible with a hardened allocator due to interfering with doing
> anything like this properly.

Also a fair assessment.

> > Future Work:
> >  - out of line metadata
>
> There's a huge variation in what this means. The hardened_malloc
> metadata is fully out-of-line in a dedicated region, with that address
> space never mixed / reused with anything else. The same applies to all
> the size class regions within arenas.

That's very strong protection - I hope we can integrate the same level
of security in the future.

> >  - MPK
>
> For what exactly?

Still TBD; but our initial thoughts are using it in the JIT Engine and
to effect XOM for certain use cases.


> > But the benefit gained by slapping in an LD_PRELOAD and calling it a
> > day is small to zero. Probably negative because you'll not utilize
> > partitions by default. You'd need a particurally constrained
> > vulnerability to actually prevent exploitation - it's more likely
> > you'll just cost the attacker another 2-8 hours of work.
>
> The claim that mozjemalloc has partitioning and hardened_malloc does not
> couldn't be further from the truth.

A misunderstanding: I didn't claim that; I claimed that using
LD_PRELOAD would not make use of partitions.

> I find it ridiculous how you attempt to
> attack the project with these lies to promote your own work, which is
> hardly comparable at all. It's not the same thing. Bolting on a few
> security features to an allocator design that's exploitation friendly
> from the ground up doesn't make it a hardened allocator. That's even
> more true when the implementations of those features are unnecessarily
> weak.

My intention was not to promote my own work; or negate yours. I'm
sorry it came across that way. As I said at the top of this email,
hardened_malloc is clearly a much stronger allocator that mozjemalloc.

I'll try to clear up my intention with my email here:

Replacing the allocator used by Tor Browser naively (LD_PRELOAD) would
be a net loss in security. I think (?) you'd agree. Replacing the
allocator used by Tor Browser correctly would grant a hardened
allocator but would not significantly affect how the past several
exploits written against Firefox would be written, and would not be a
significant impediment to exploit authors who are using the type of
vulnerabilities we have seen exploited of late.

It would certainly affect the exploitation of more limited
vulnerabilities (fixed-byte over-reads/writes).

> > Out of line metadata is on-the-surface-attractive but... that tends to
> > only help when you have a off-by-one/four write and you corrupt
> > metadata state because it's the only thing you *can* do. With out of
> > line metadata, you can just corrupt a real object and effect a
> > different type of corruption. I'm pretty skeptical of the benefit at
> > this point, although I could be convinced. We don't see metadata
> > corruption attacks anymore - but I'm not sure if it's because we find
> > better exploit primitives or better vulnerabilities.
>
> Out-of-line metadata is not simply about preventing attacks on the
> metadata itself. It provides much more than that. I'm not sure why you
> think your ignorant opinions on these topics matter, when you so clearly
> don't know what you're talking about at all.

Unfair! :)

> > In particular, if you wanted to pursue hardened_malloc you would need
> > to use replace_malloc and wire up the partitions correctly.
> > Randomization will almost certainly not help (and will hurt
> > performance)*.
>
> The hardened_malloc design is focused on reliable, deterministic memory
> corruption mitigations. Randomization is used where possible, and in a
> way that has a low impact on performance. The high entropy base
> randomization for each size class region within arenas has no
> significant impact on performance. The randomization for large
> allocation (> 128k) guard regions and that quarantine has no substantial
> impact on performance. The impact from slot randomization and the slab
> allocation quarantine is measurable but not high, and the slab
> quarantine has no substantial impact.

My statement on performance was based on my experiments enabling our
limited randomization in the Content Process (numbers lost, but they
came from https://bugzilla.mozilla.org/show_bug.cgi?id=1376408#c18).
It really hurt DOM node traversal. I'll concede it's possible that
hardened_malloc may have less any performance cost here - I haven't
tested it. But... I'm skeptical.

(You seem to agree that randomization is not a strong security feature?)

> > MPK sounds nice but you have to use it correctly (which
> > requires application code changes), you have to ensure there are no
> > MPK gadgets, and oh wait no one can use it because it's only available
> > in Linux on server CPUs. =(
>
> Using MPK is not one of the major features of hardened_malloc and the
> usage doesn't rely on not having MPK gadgets. This is explicitly
> documented in the README. It's the only optional security feature that's
> not enabled by default. It should be pointed out that most of security
> offered by hardened_malloc is not a feature that can be turned on or off
> because it's the design itself that's hardened, not the fact that it has
> some optional security features bolted onto it.

Fair! (Although I'm confused about your comment saying you can provide
protection without eliminating MPK gadgets.)

> > In conclusion, while it's possible hardened_malloc could provide some
> > small security increase over mozjemalloc, the gap is much smaller than
> > it was when I advocated for allocator improvements 5 years ago, the
> > effort is definitely non-trivial, and the gap is closing.
>
> No, you're just making false attacks and misleading comparisons / spin
> to promote your own work, which is trash.

=(

> You're being incredibly
> dishonest and unethical.

I'd hope that the principal of charity would lead to the conclusion
that instead I was being simplistic and ignorant.

> You didn't even bother to inform yourself about
> hardened_malloc by actually reading through the documentation. Instead,
> you just jump to conclusions and present yourself as an expert on topics
> you are clearly incredibly ignorant about. You really don't know what
> you're talking about, and your post on this mailing list is offensive.
> Your post as a whole is nonsense, and your conclusion is bogus.

Which conclusion? The one I didn't make (but certainly implied) that
mozjemalloc is comparable security-wise to hardened_malloc? I agree,
such a conclusion is bogus.

Or the one that replacing the allocator will not have a significant
effect on the exploits written for the past couple years of Firefox
expoits?

This seems to be the central misunderstanding, and why I agree that
you are entitled to take offense.


> [ending statements]

I don't have any knowledge of, or context about, your past
interactions with Mozilla. We all wear many hats; but this discussion
isn't about Mozilla, Firefox, or replacing Firefox's memory allocator
- it's about Tor Browser. If you'd like to escalate your concerns,
they can be forwarded to Tor's Community Council; that information is
available at https://trac.torproject.org/projects/tor/wiki/org/CommunityCouncil

-tom