FreeBSD tweak may yield better performance

[Scott Bennett]

     FreeBSD 7.2 and later offer a little-advertised kernel facility that
benefits processes with "large" virtual memory requirements, in particular,
"large" working sets.  What it does is to monitor process memory requirements
and to promote processes whose memory requirements become "large" enough to
use 4 MB pages instead of 4 KB pages.  The benefit to I/O bound processes
will typically be very difficult to detect, but for many CPU-bound processes,
the performance kick can be rather dramatic.  For example, my full disk backups
typically were taking in excess of 8 hours with the feature disabled, but
with the feature active, the backups, the output of which gets compressed
by piping it through gzip(1), finished in 4 hours and 15 minutes.  For this
particular task, it was like getting a new computer.
     Very high-data-rate tor relays apparently do use a lot of CPU time, so
it occurred to me that such systems might also benefit from enabling the
so-called "superpages" facility.  I am currently only able to run a tor relay
as a "stealth" relay to avoid detection and punishment by my ISP :-(, so I
cannot run the experiment myself.  If someone running a high-data-rate relay
under FreeBSD 7.2 or later would like to try it, here's what you need to do.
Basically, just add the following line to /boot/loader.conf and then reboot
your system.

vm.pmap.pg_ps_enabled=1

This sysctl variable can only be changed at boot time, and its default value
is 0 (i.e., not enabled), so if you are interested in trying it, keep in mind
that a reboot will be necessary to activate promotion to superpages.  Note
that contention for real memory will cause this facility to demote superpage
processes back to 4 KB pages as required to meet systemwide memory needs.
     In case it isn't obvious where the performance enhancement comes from,
consider that most/all modern CPUs from Intel and AMD have TLBs that hold
a maximum of 128 entries, each one containing a mapping of a virtual memory
page to a real page frame's base address.  That means that the TLB can cache
translated addresses covering a maximum of 512 KB when 4 KB pages are in use.
If a process's working set exceeds that size, then there will be frequent TLB
misses, each of which causes a processor stall while that DAT circuitry goes
through the full address translation process, complete with multiple fetches
from memory.  This seems especially wasteful when one considers that the data
or instructions needed by the processor might actually already be in L1 cache
lines, but inaccessible without a fresh, full address translation. :-(  By
contrast, a process that has been promoted to superpage usage has had its
TLB entry requirements reduced by as much as a factor of 1024, so the entire
process might need only a few TLB entries, after which no further address
translations need be done, except when context switches have caused TLB
entries to be purged.
     Anyway, if someone running a high-data-rate tor relay on a FreeBSD
system of release 7.2 or later would like to try the above tweak and report
back to the list any observed changes in performance, I would be grateful
for the information.  torstatus.kgprog.com currently shows 9 relays with
peak traffic rates greater than 1 MB/s that run on FreeBSD systems, although
the release level is not shown.  Those very active relays are the ones I
would expect to see get the greatest benefit from using superpages.


                                  Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet:       bennett at cs.niu.edu                              *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."                                               *
*    -- Gov. John Hancock, New York Journal, 28 January 1790         *
**********************************************************************
***********************************************************************
To unsubscribe, send an e-mail to majordomo at torproject.org with
unsubscribe or-talk    in the body. http://archives.seul.org/or/talk/