ASN.1: Reject invalid definite long form length values in DER encoding

The definite long form for the length is allowed only for cases where
the definite short form cannot be used, i.e., if the length is 128 or
greater. This was not previously enforced and as such, multiple
different encoding options for the same length could have been accepted.

Perform more strict checks to reject invalid cases for the definite long
form for the length. This is needed for a compliant implementation and
this is especially important for the case of verifying DER encoded
signatures to prevent potential forging attacks.

Signed-off-by: Jouni Malinen <j@w1.fi>

src/tls/asn1.c

@@ -231,6 +231,11 @@ int asn1_get_next(const u8 *buf, size_t len, struct asn1_hdr *hdr)
 		}
 		tmp &= 0x7f; /* number of subsequent octets */
 		hdr->length = 0;
+		if (tmp == 0 || pos == end || *pos == 0) {
+			wpa_printf(MSG_DEBUG,
+				   "ASN.1: Definite long form of the length does not start with a nonzero value");
+			return -1;
+		}
 		if (tmp > 4) {
 			wpa_printf(MSG_DEBUG, "ASN.1: Too long length field");
 			return -1;
@@ -243,6 +248,11 @@ int asn1_get_next(const u8 *buf, size_t len, struct asn1_hdr *hdr)
 			}
 			hdr->length = (hdr->length << 8) | *pos++;
 		}
+		if (hdr->length < 128) {
+			wpa_printf(MSG_DEBUG,
+				   "ASN.1: Definite long form of the length used with too short length");
+			return -1;
+		}
 	} else {
 		/* Short form - length 0..127 in one octet */
 		hdr->length = tmp;