Fix error-checking compiles for mutex

Fixes: #14229
Reviewed-by: NKurt Roeckx <kurt@roeckx.be>
Reviewed-by: NPaul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14264)

INSTALL.md

@@ -1666,6 +1666,13 @@ most UNIX/Linux systems), and Windows threads.  No other threading models are
 supported.  If your platform does not provide pthreads or Windows threads then
 you should use `Configure` with the `no-threads` option.
 
+For pthreads, all locks are non-recursive. In addition, in a debug build,
+the mutex attribute `PTHREAD_MUTEX_ERRORCHECK` is used. If this is not
+available on your platform, you might have to add
+`-DOPENSSL_NO_MUTEX_ERRORCHECK` to your `Configure` invocation.
+(On Linux `PTHREAD_MUTEX_ERRORCHECK` is an enum value, so a built-in
+ifdef test cannot be used.)
+
 Notes on shared libraries
 -------------------------
 

crypto/threads_pthread.c

@@ -55,7 +55,7 @@ CRYPTO_RWLOCK *CRYPTO_THREAD_lock_new(void)
      * We don't use recursive mutexes, but try to catch errors if we do.
      */
     pthread_mutexattr_init(&attr);
-#  if defined(NDEBUG) && defined(PTHREAD_MUTEX_ERRORCHECK)
+#  if !defined(NDEBUG) && !defined(OPENSSL_NO_MUTEX_ERRORCHECK)
     pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ERRORCHECK);
 # else
     pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_NORMAL);