[tor-commits] [doctor/master] Free memory prior to sending email

[atagar at torproject.org]

commit 53277867865a0192dc30d29c35e9558d85082421
Author: Damian Johnson <atagar at torproject.org>
Date:   Sat Oct 11 12:36:49 2014 -0700

    Free memory prior to sending email
    
    Frequently we run into OOM issues when sending a notification email at the end
    of our run...
    
      Traceback (most recent call last):
        File "/srv/doctor.torproject.org/doctor/consensus_health_checker.py", line 705, in <module>
          util.send("Script Error", body_text = msg, destination = util.ERROR_ADDRESS)
        File "/srv/doctor.torproject.org/doctor/util.py", line 97, in send
          stderr = subprocess.PIPE,
        File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
          errread, errwrite)
        File "/usr/lib/python2.7/subprocess.py", line 1153, in _execute_child
          self.pid = os.fork()
      OSError: [Errno 12] Cannot allocate memory
    
    This is because we call subprocess.Popen which forks our process, doubling the
    amount of memory we use. Sending an email obviously doesn't require much, but
    since we're using hundreds of megs for all the consensuses we download that's
    what is allocated.
    
      http://stackoverflow.com/a/13329386/1067192
    
    Attempting to address this by freeing descriptor content prior to forking our
    process.
---
 consensus_health_checker.py |   10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/consensus_health_checker.py b/consensus_health_checker.py
index 54c50da..8254dc5 100755
--- a/consensus_health_checker.py
+++ b/consensus_health_checker.py
@@ -7,6 +7,7 @@ Performs a variety of checks against the present votes and consensus.
 """
 
 import datetime
+import gc
 import time
 import traceback
 
@@ -219,6 +220,15 @@ def main():
     for issue in issues:
       rate_limit_notice(issue)
 
+    # Reclaim memory of the consensus documents. This is ebecause sending an
+    # email forks our process, doubling memory usage. This can easily be a
+    # trigger of an OOM if we're still gobbling tons of memory for the
+    # descriptor content.
+
+    del consensuses
+    del votes
+    gc.collect()
+
     if TEST_RUN:
       print '\n'.join(map(str, issues))
     else: