[tor-dev] prop224: HSDir caches question with OOM

[David Goulet]

Hi!

(For the following, I'm only talking about HS directory.)

Here is my conundrum. I was working on plugging the new cache for proposal 224
into our OOM. I stumbled upon a fun problem.

We plan to have both current and new HS protocol in parallel. This means two
caches for both version 2 (current) and 3 (prop224). So far so good.

However, here is the problem. Now, our OOM handlers is called in:
cell_queues_check_size(). The heuristic is basically:

      /* If we're spending over 20% of the memory limit on hidden service
       * descriptors, free them until we're down to 10%.  */

Considering we have _two_ caches now, we somehow have to cleanup both of them
"evenly" that is if we have to remove 20kb we either remove 10kb on each (if
possible) or we actually clean them by entry expiry time instead regardless of
which version.

I think the latter is what we want but then it complicates things a bit since
both cache contains _different_ data struct. and indexed with different keys
(DIGEST vs DIGEST256). One option here is to go over all entries in both cache,
create temporary objects and put them in a sorted list (by expiry time). Those
"temp object" would contain something like: 1) cache version, 2) expiry time
and 3) index key. Once done, we go over that list and delete entries until we
reach our number of bytes we need to remove.

This could work since we would do that when we are at 20% memory but still it
requires allocating some small objects to _clear_ our memory... Someone has
maybe something _better_ to propose? :)

(keep in mind that minimal change to the current HS code is preferable. I know
that unifying both caches somehow would help but that's lots more work! and
the new cache for prop224 will be able to support versionning for which our
current one doesn't, it's baked for v2.)

I'm asking because this will require some refactoring for which I want to make
sure we are in agreement on how to proceed and it doesn't end up not the right
solution.

Basically my goal is to move the OOM handler for both v2 and v3 to one single
function called hs_cache_handle_oom() and from which we'll do the gymnastic
describe above. We'll stop calling rend_cache_clean_v2_descs_as_dir()
explicitely after that in cell_queues_check_size().

Cheers!
David
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 603 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20160415/76b656d9/attachment.sig>