[metrics-team] Cache advice for Onionoo Node.js API client

[Luke Childs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello everyone,

I’m Luke, I spoke to some of you about Tor Explorer (
https://github.com/lukechilds/tor-explorer) on IRC the other day.

I’ve separated the Onionoo bits from Tor Explorer out into a reusable
Node.js module (https://github.com/lukechilds/onionoo-node-client). This
will allow other people to easily include this module
in their Node.js projects and use the Onionoo API with minimal effort.

I’m just cleaning it up and making sure everything works properly before
publishing v1. You can track progress here if you’re interested:
https://github.com/lukechilds/onionoo-node-client/milestone/1

I’m slightly confused about how to handle caching and was wondering if any
Onionoo devs (or anyone else) could provide some guidance on a couple of
questions I have.


1. Cache strategy

I’m currently just saving the responses in memory for however long the
Cache-Control max-age header value says, after the max-age has expired a
new request will be made. However I read this in the
docs:

> Clients should make use of the "Last-Modified" header of responses and
include that timestamp in a "If-Modified-Since" header of subsequent
requests.

This suggests I should be storing the responses indefinitely. After the
max-age has expired I should do a new request with If-Modified-Since and if
I get a 304, return my already cached value (and
possibly update the max-age in my cache?). Just double checking, is that
correct?


2. How to calculate max-age

Regarding max-age, what is the correct value I should be using here? I was
just using the Cache-Control max-age header value directly, however I
noticed this often stays the same but the age
header increments. Possibly varnish caching the max-age header from
Onionoo? So how should I reliably calculate the cache lifetime by the time
I get the response?

I think this would offset the time Varnish has already been holding it for:

max-age - age = cache-lifetime-ms

but maybe this would be more accurate to compensate for network latency too:

date + max-age = valid-until-date
valid-until-date - now = cache-lifetime-ms


If anyone could offer some advice on these two issues that would be greatly
appreciated

Cheers,
Luke

P.S I find GitHub issues to be quite efficient for discussing this sort of
thing. If you have a GitHub account, feel free to post replies in the cache
issue:
https://github.com/lukechilds/onionoo-node-client/issues/2. If you’d rather
not use GitHub, no problem :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJYVkR7AAoJEJPWEqnWHCga+MQH/R2eH1OzvUgTKt8JV4Ve4hUX
hBLzLdgzRLSqpcYQXvAMHLT9Yo3fsCs1EtSEdrL6wYmcB0kK6Ql3CtKJA4ejy0kM
Aju9769/U+g5yta7sYNdkO6YYXxYKrhXpjFMslvI4s6z7Z8wtk1JRFHEG9h5gj/J
BjNgq/lqJFcNGE67chrf1BsD4FXUxXdM14uz1QEyt8M0GjvABUgMPSNHXIkN6bmX
U0DkT/hFSIRLNh6VzxVpS2goKt0X7E4eXQE2KDHKSsb7imvRU9y4mUA/DgiG7J1P
/K9gzTwF1EHKhGHE8UzthGv9isDpjuXYcGYxjP/DLoFnpB9nZkpUi3671l7oYwI=
=OEw1
-----END PGP SIGNATURE-----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/metrics-team/attachments/20161218/d9f64d2b/attachment.html>