[1910.11015] Taxonomy of Real Faults in Deep Learning Systems
In our future work we plan to use the presented taxonomy as a guidance to improve DL systems testing and as a source for the definition of novel mutation operators.
Abstract: The growing application of deep neural networks in safety-critical domains
makes the analysis of faults that occur in such systems of enormous importance.
In this paper we introduce a large taxonomy of faults in deep learning (DL)
systems. We have manually analysed 1059 artefacts gathered from GitHub commits
and issues of projects that use the most popular DL frameworks (TensorFlow,
Keras and PyTorch) and from related Stack Overflow posts. Structured interviews
with 20 researchers and practitioners describing the problems they have
encountered in their experience have enriched our taxonomy with a variety of
additional faults that did not emerge from the other two sources. Our final
taxonomy was validated with a survey involving an additional set of 21
developers, confirming that almost all fault categories (13/15) were
experienced by at least 50% of the survey participants.