[tor-bugs] #20717 [Core Tor/Tor]: Hashing api should return negative values for errors

[Tor Bug Tracker & Wiki]

#20717: Hashing api should return negative values for errors
------------------------------+--------------------------------
     Reporter:  chelseakomlo  |      Owner:
         Type:  defect        |     Status:  new
     Priority:  Medium        |  Milestone:  Tor: 0.3.0.x-final
    Component:  Core Tor/Tor  |    Version:
     Severity:  Normal        |   Keywords:  029-backport
Actual Points:                |  Parent ID:
       Points:                |   Reviewer:
      Sponsor:                |
------------------------------+--------------------------------
 In crypto.c, the hashing API currently returns 1 for error, 0 on success.
 Specifically in: crypto_digest, crypto_digest256, and crypto_digest512.

 This is often misinterpreted in calling functions. For example,
 crypto_pk_get_digest and crypto_pk_private_sign_digest both incorrectly
 assume negative error return values.

 The fix for this should refactor to return -1 on error, and change calling
 functions that currently expect 1 on error, such as
 crypto_expand_key_material_TAP and crypto_pk_get_hashed_fingerprint.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20717>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online