Update February 2023: A third dataset has been posted to the CSP Blog: CSPB.ML.2023. It features cochannel signals.
Update January 2023: I’m going to put Challenger results in the Comments. I’ve received a Challenger’s decisions and scored them in January 2023. See below.
In this post I provide a second dataset for the Machine-Learning Challenge I issued in 2018 (CSPB.ML.2018). This dataset is similar to the original dataset, but possesses a key difference in that the probability distribution of the carrier-frequency offset parameter, viewed as a random variable, is not the same, but is still realistic.
Blog Note: By WordPress’ count, this is the 100th post on the CSP Blog. Together with a handful of pages (like My Papers and The Literature), these hundred posts have resulted in about 250,000 page views. That’s an average of 2,500 page views per post. However, the variance of the per-post pageviews is quite large. The most popular is The Spectral Correlation Function (> 16,000) while the post More on Pure and Impure Sinewaves, from the same era, has only 316 views. A big Thanks to all my readers!!
When I process the generalized challenge (GC) dataset, I obtain results that are nearly the same as when I process the challenge (C) dataset. This is because there is no training in my cyclostationary signal processing approach to modulation recognition and parameter estimation, and there is no prior information that is supplied to the algorithm, so it doesn’t matter that things have changed with the signals’ non-constellation probability distributions. The BPSK signals in the GC dataset are just like the BPSK signals in the C dataset as far as spectral correlation, spectral coherence, cyclic temporal cumulants, etc., are concerned. The signal-processing approach can recognize all BPSK signals. A BPSK signal just needs to conform to the (textbook) BPSK signal model to be recognized; particular numerical values of the modulation parameters are not important, and they will be accurately estimated along the way.
Carrier-Frequency Offset Results
The primary focus of the original Challenge was carrier-frequency-offset (CFO) estimation. So let’s start with the CFO results for the GC dataset in Figure 1.
Compare this result with Figure 1 from the Challenge post, reproduced here for your convenience,
So the CSP-based approach to modulation recognition (My Papers [25,26,28]) possesses a high degree of generalizability with regard to estimation of the carrier-frequency offset parameter. Let’s now turn to the modulation-classification results.
Modulation Classification Results
First the full confusion matrices for the eight-class problem embodied by the Challenge and Generalized Challenge datasets. Figures 3-5 show the results for three processing block lengths: 32,768, 16,384, and 8,192 samples, respectively.
The conclusion is that the performance of the cyclic-cumulant-based modulation-classification algorithm I outline in My Papers [25,26,28] is insensitive to the change in carrier-frequency offset. The question is: Can we construct a neural network that is trained on the Challenge set, but that performs equally well on the Generalized Challenge set (and the performance is good)? So far, when training a convolutional neural network using the Challenge dataset I/Q samples, my colleagues and I have not been able to get good generalization performance–the network fails on the Generalized Challenge dataset.
I hasten to add that this isn’t some big revelation coming out of the CSP Blog. The dataset-shift problem, also known as the generalization problem and other names, is well-known in machine learning.
One thing that has occurred to me over and over while pondering all this generalization stuff with neural networks is: Why do we continue to stick with convolutional layers? I get that convolutional layers are appropriate for image processing (at least I think I get that because linear processing of images can get you a long way), but here, in RF signal processing, the characteristics of the class are embedded throughout the waveform samples through the structure of many involved random variables. We’re not attempting to identify and locate a picture of a cat no matter where it is in an image. The ‘cat’ (modulation type) is distributed throughout the entire ‘image’ (data sample vector). We don’t need to localize anything. We don’t need to grasp edges or boundaries or shapes. (See the Comments below for more on this line of thinking.)
The fundamental probability structure of an RF signal can be brought to light by subjecting the data samples to nonlinearities–that’s all I’m doing when I apply the arithmetic of CSP, whether it is the traditional second-order cyclic autocorrelation or the higher-order cyclic temporal cumulants. Why not have explicit nonlinear layers like squarers in our networks?
The format of the signal files is the same as in the Challenge dataset–go look over there for details. The difference here is that I’m not providing the true parameters and signal-type labels. The idea is that a neural network is trained on some other dataset (perhaps the Challenge dataset, perhaps some other dataset, whatever), and then applied to the Generalized Challenge dataset. So if you take me up on this part of the challenge, you’ll have to submit your machine’s answers and trust me to give you an accurate score.
Since there is no possibility of training here, I don’t think I need to provide as many examples of each class as I did in the Challenge dataset. I created the same number (112,000), but for now I’m just posting 20,000 in five batches of 4,000 files each.