Tarik Crnovrsanin , Calvin Yu , Dane Hankamer, and Cody DunneAbstract:Unmanned aerial vehicles are becoming common and have many productive uses. However, their increased prevalence raises safety concerns—how can we protect restricted airspace? Knowing the type of unmanned aerial vehicle can go a long way in determining any potential risks it carries. For instance, fixed-wing craft can carry more weight over longer distances, thus potentially posing a more significant threat. This paper presents a machine learning model for classifying unmanned aerial vehicles as quadrotor, hexarotor, or fixedwing. Our approach effectively applies a Long-Short Term Memory (LSTM) neural network for the purpose of time series classification. We performed experiments to test the effects ofchanging the timestamp sampling method and addressing the imbalance in the class distribution. Through these experiments, we identified the top-performing sampling and class imbalance fixing methods. Averaging the macro f-scores across 10 folds of data, we found that the majority quadrotor class was predicted well (98.16%), and, despite an extreme class imbalance, the model could also predicted a majority of fixed-wing flights correctly (73.15%). Hexarotor instances were often misclassified as quadrotors due to the similarity of multirotors in general (42.15%). However, results remained relatively stable acrosscertain methods, which prompted us to analyze and report on their tradeoffs. The supplemental material for this paper, including the code and data for running all the experiments and generating the results tables, is available at https://osf.io/mnsgk/. Index Terms—UAV, Classification, LSTM, Time Series
(cid:18)(cid:26)(cid:31)(cid:21)(cid:27)(cid:23)(cid:26)(cid:29)(cid:17)(cid:18)(cid:26)(cid:31)(cid:21)(cid:27)(cid:23)(cid:26)(cid:29)(cid:16)(cid:31)(cid:30)(cid:29)(cid:28)(cid:27)(cid:26)(cid:25)(cid:24)(cid:15)(cid:29)(cid:25)(cid:26)(cid:14)(cid:29)(cid:31)(cid:28)(cid:13)(cid:12)(cid:29)(cid:28)(cid:31)(cid:30)(cid:27)(cid:26)(cid:25) (cid:11)(cid:29)(cid:25)(cid:26)(cid:14) 20s40s60s80s0s10s30s50s70s90s (b) Fixed Window Average Sampling Fig. 3: The two figures illustrate how fixed window average and average sampling work on flights with different flight times. For average sampling, the amount of data averaged changes across flights, with shorter flights averaging a smaller portion of the data. Fixed window average averages the same amount of data across flights but varies the gaps. For each flight, the algorithm uses the longest feature for sampling. For any shorter feature, we sample partially or not at all to guarantee that the sampling is aligned within a flight.
id: 08cc7ef2657a718fd711c9133389165b - page: 7
Each flight has a local minimum and maximum as determined by the first and last timestamps of every features. Since features may have different ranges, there will likely be empty space for shorter ranges. However, we take the minimum and maximum and can divide each sample into an even number of intervals. With the complexities involved in flight data, there is high potential in abstracting away details in a particular flight. For these reasons, we see an opportunity in exploring more timestamp sampling methods based on our domain knowledge of flight patterns.
id: 7c1fee3db96dd6fb37e4aadde73b3258 - page: 7
2) Fixed Window Average Sampling: Windowed averaging reduces and normalizes the amount of information compared across different flights. For example, as shown in Figure 3b, we can capture an 8 second window, which abstracts less data than averaging over an entire interval and acquires more data than sampling a single point. Averaging over the same duration means that the same durations are compared across different flights unlike the averaging approach which may average over more or fewer points depending on the length of the flights. Figure 3b shows that since we are using the same window size for each flight, the amount of space between the end of each window and the beginning of the next window may vary (i.e. 12 seconds vs. 7 seconds). This method introduces a new parameter to tune, the window duration, which will be discussed further in the experiments section. Selecting this value involves analyzing the overall duration of the flights for each class.
id: bc9f8eaf8cd390e3ff13f74a087b7491 - page: 7
V. TRAINING AND EVALUATION We performed a standard neural network training procedure depending on a set number of epochs and batch size. The standard classification loss, cross-entropy, was used alongside the Adam optimizer during gradient descent. To obtain consistent results, we ran experiments using different training configurations until the results stabilized and provided consistent results. Because of the heavy imbalance in the dataset and relatively quick training times, we also employed k-fold cross-validation. We average the results across all folds and compute standard deviation to determine the consistency of the results. Each fold produces a precision, recall, and fscore for the number of quadrotors, hexarotors, and fixedwing UAVs identified correctly. Although not necessary, averaging results across multiple folds of data ensures a better representation of performance and robustness in the few instances of hexarotor and fixed-wing flights.
id: c0ebb031b9d34473a4333c5a805b8f02 - page: 7