Analysis of DTLS Implementations Using Protocol State Fuzzing

Abstract

Recent years have witnessed an increasing number of proto-cols relying on UDP. Compared to TCP, UDP offers perfor-mance advantages such as simplicity and lower latency. Thishas motivated its adoption in Voice over IP, tunneling techno-logies, IoT, and novel Web protocols. To protect sensitive dataexchange in these scenarios, the DTLS protocol has been de-veloped as a cryptographic variation of TLS. DTLS’s mainchallenge is to support the stateless and unreliable transport ofUDP. This has forced protocol designers to make choices thataffect the complexity of DTLS, and to incorporate featuresthat need not be addressed in the numerous TLS analyses.We present the first comprehensive analysis of DTLS im-plementations using protocol state fuzzing. To that end, we ex-tend TLS-Attacker, an open source framework for analyzingTLS implementations, with support for DTLS tailored to thestateless and unreliable nature of the underlying UDP layer.We build a framework for applying protocol state fuzzing onDTLS servers, and use it to learn state machine models forthirteen DTLS implementations. Analysis of the learned statemodels reveals four serious security vulnerabilities, includinga full client authentication bypass in the latest JSSE version,as well as several functional bugs and non-conformance is-sues. It also uncovers considerable differences between themodels, confirming the complexity of DTLS state machines.