[metrics-bugs] #26193 [Metrics/CollecTor]: Tarballs are not compressed in a run following an aborted run

[Tor Bug Tracker & Wiki]

#26193: Tarballs are not compressed in a run following an aborted run
-----------------------------------+----------------------
     Reporter:  karsten            |      Owner:  karsten
         Type:  defect             |     Status:  assigned
     Priority:  Medium             |  Milestone:
    Component:  Metrics/CollecTor  |    Version:
     Severity:  Normal             |   Keywords:
Actual Points:                     |  Parent ID:
       Points:                     |   Reviewer:
      Sponsor:                     |
-----------------------------------+----------------------
 Today I wondered why some of our 2018-05 tarballs are over 3 days old even
 though we're creating new tarballs every 2 days. And I wondered even more
 after finding a seemingly completed run from yesterday. Here's what
 happened:

  - The May 19 run succeeded without issues.
  - The May 21 run was interrupted, probably due to a host reboot.
 Apparently, it finished compressing tarballs except for bridge extra-info
 descriptors, but it did not move any of them in place.
  - The May 23 run went through, but it did not compress any tarballs
 except for bridge extra-info descriptors.

 I think this is related to
 [https://gitweb.torproject.org/collector.git/tree/src/main/resources
 /create-tarballs.sh#n102 line 102 in the script]:

 {{{
 for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
   echo `date` "Creating" ${TARBALLS[$i]}'.tar'
   tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
   if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then           # <- this one
     echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
     xz -9e ${TARBALLS[$i]}.tar
   fi
 done
 }}}

 Explanation: there were still compressed files in the working directory
 from May 21, and as a result we did not attempt to compress new tarballs
 from May 23.

 Suggested fix: remove that `if` statement and compress the tarball
 regardless of whether a previous compressed tarball exists in the working
 directory using `xz -9e -f`:

 {{{
 diff --git a/src/main/resources/create-tarballs.sh b/src/main/resources
 /create-tarballs.sh
 index cd16b2dc..d247c520 100755
 --- a/src/main/resources/create-tarballs.sh
 +++ b/src/main/resources/create-tarballs.sh
 @@ -99,10 +99,8 @@ done
  for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
    echo `date` "Creating" ${TARBALLS[$i]}'.tar'
    tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
 -  if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then
 -    echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
 -    xz -9e ${TARBALLS[$i]}.tar
 -  fi
 +  echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
 +  xz -9e -f ${TARBALLS[$i]}.tar
  done

  cd $OUTDIR/webstats/
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/26193>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online