This installment of the Signal Processing Toolkit series of CSP Blog posts deals with the ubiquitous signal-processing operation known as convolution. We originally came across it in the context of linear time-invariant systems. In this post, we focus on the mechanics of computing convolutions and discuss their utility in signal processing and CSP.
Review of Convolution Mathematics
The convolution integral, which is used for continuous-time signals, is defined as
while the convolution sum is used for discrete-time signals
Convolution is intimately involved in the study and use of linear time-invariant systems, or filters, examples of which include lowpass, bandpass, bandstop, and highpass filters. A linear time-invariant continuous-time system is characterized by its impulse-response function, typically denoted by , which is equal to the output of the system when the input is an impulse function (Dirac’s delta function) applied at time . For linear shift-invariant discrete-time systems, the impulse-response function is the output of the system when the input is a discrete-time impulse (Kronecker’s delta function). The impulse-response function is typically denoted by the symbol in either case. For continuous-time systems with arbitrary input , the output is given by (1) or, equivalently, (2). For discrete-time linear shift-invariant systems with arbitrary input , the output is given by (3) or, equivalently, (4).
Convolution is typically difficult for new signal-processing students to comprehend and to compute with pencil-and-paper. And it can be difficult for everybody if and/or are complicated functions. So in this post we’ll go over a couple examples, illustrating how to compute convolutions on paper as well as how to compute them using MATLAB. At the end of the post, we’ll sketch how convolution is used in CSP.
The Convolution Theorem
An important aid in computing convolutions as well as in various kinds of analysis involving linear systems is the convolution theorem. This theorem says that the Fourier transform of a convolution (say, the Fourier transform of in (1)) is equal to the product of Fourier transforms for the signals undergoing the convolution. In mathematical terms, using our CSP-Blog notation,
Similarly, if we have a convolution in the frequency domain, the inverse transform of that convolution is the product of the inverse transforms of the frequency functions undergoing convolution,
These results are summed up by saying that “convolution in one domain is multiplication in the other.”
Convolution Example “By Hand”
The canonical example of convolution is convolving a rectangle with itself, so we’ll be looking at that example in detail, through a variety of lenses. In this first look, we’ll simply go through the calculations required to fully evaluate the convolution integral.
We want to convolve a rectangle with itself. A rectangle with unit height and unit width is usually denoted by the function,
There are alternative definitions for that deal in different ways with the value of the function at . These won’t matter to us here at the CSP Blog because we are dealing with simple integrable functions, and the convolution result is usually continuous. But if you see some kind of edge problem in your convolution work, let everybody know in the Comments below.
A rectangle with height , width , and center at is then easily expressed in terms of the function
as shown in Figure 1. So our present question is: What is ? The convolution integral in this case is
From (13), we need to form the integrand product for each . This means we need to understand the function . The function is a time-reversed version of , and since is symmetric about , in the case of our centered rectangle function.
The presence of the variable in then shifts this rectangle by units of time. Let’s say is large in magnitude and negative. Then the shifted rectangle will have it center at , which is a large and negative number. In other words, when is large and negative, the rectangle represented by is shifted far to the left on the axis. Similarly when is large and positive, the rectangle gets shifted to the right.
The way we proceed is to start evaluating the convolution integral for a large negative , then increase until it becomes large and positive. This can be visualized as first flipping the signal (which accounts for the negative sign in ), shifting it far to the left (start with large negative ), then sliding the flipped shifted signal to the right, computing the value of for each , and using that to evaluate the convolution integral for that value of . This is illustrated with a MATLAB-based video below. But for now, we continue with the pencil-and-paper analysis.
For , the flipped and shifted rectangle does not overlap with the signal , so that the product is zero, and therefore the convolution integral is zero. Or,
Let’s keep sliding to the right by increasing . When , the right edge of is coincident with the left edge of . As increases past , the two rectangles overlap. When we multiply by the resulting function of is non-zero only for in the overlap region.
The value of the convolution integral is therefore the area under the curve for values of that lie within the overlap region. When , the overlap is complete. For this case of a simple rectangular overlap region, we can do this integral in our heads, but let’s write it out for the sake of really focusing on the details.
For , then, we have
To check this result, note that for , , as it should because that is the case of a single point of overlap. For , we have , which is simply the area of the rectangle , which checks because at , the two rectangles are coincident.
As increases past zero, the left edge of slides into the interior of , and the right edge slides past , so the overlap between the two rectangles begins to decrease. When , the overlap region becomes a single point, and the convolution integral is again zero,
Finally, when , the two rectangles do not overlap, and the convolution is zero for all these values of . Summing up the different results corresponding to different intervals for , we have
Graphing shows that the function is a rectangle with height and width , as shown in Figure 8.
You can likely see that if we were to convolve two rectangles with the same widths but different heights and , the result would be a triangle with width and height .
Convolution Examples “By MATLAB”
So some convolutions can be performed by straightforward application of the convolution integral (or sum in discrete time), as we’ve just seen. For many, though, that method is either tedious or impractical, and so we resort to numerical evaluation using our own code or using the convolution functions provided by a sophisticated software package like MATLAB or a programming language like python. This software can be used to do the time-domain convolution directly and it can be used to leverage the convolution theorem to perform the convolution.
Here we’ll focus on MATLAB, as we usually do at the CSP Blog. Much can be learned by writing your own generic convolver, so I do recommend you do that. But for the present post, we’ll explore how to use standard MATLAB commands to perform convolutions. Along the way, the examples I provide should also develop your intuition about what convolution ‘does’ or ‘means.’
The basic tools for convolution in MATLAB are conv.m and filter.m To use the convolution theorem, we need the Fourier transform, which is fft.m, and the inverse Fourier transform, which is ifft.m, as well as simple pointwise vector (signal) multiplication. The following examples use conv.m when directly performing convolution, but they could easily be converted to use filter.m.
The MATLAB script used to create the following examples can be downloaded here.
Convolution of a Rectangle with Itself
Let’s first revisit the “by hand” example above, in which we convolved a rectangle with itself. Here we generate a rectangle as a vector in MATLAB, then convolve it with itself using conv.m. The rectangle has width samples and is shown in Figure 9. The result of using conv.m on that rectangle is also shown in Figure 9. From our “by hand” analysis above, we expect a rectangle with width samples and height , which expectations are confirmed.
Convolution of a Rectangle with Itself Twice
Continuing on, let’s use conv.m to convolve a rectangle with itself, then convolve that result with the rectangle. If the rectangle is denoted by , we are attempting to find . The result is shown in Figure 10. Notice that the result of the convolution gets wider each time, so that $s(t) \otimes s(t)$ is twice as wide as , and is wider than by the width of . This is a general property of convolving with a ‘pulse-like function’ such as : the output of the convolution is smoother than the input. Whereas contains two step-function discontinuities, and so is not particularly smooth, is continuous, but has a discontinuous derivative (at , , and ). The function is, by visual inspection, continuous, and has continuous first derivative. The convolution outputs are smoother (fewer discontinuities) than the convolution inputs. So sometimes convolution with a pulse-like function (e.g., a rectangle) is referred to as a smoothing operation.
Convolution of a Triangle with Itself
Now let’s tackle a more difficult convolution two ways. The first way is by using conv.m and the second way is by using the convolution theorem. The signal is the triangle in Figure 10, which results from convolving the rectangle with itself. We wish to determine the convolution of the triangle with itself. We can easily apply conv.m to get the top plot in Figure 11. Note that this function appears similar to the bottom plot of Figure 10, which is the convolution of the rectangle with itself twice, but is a bit smoother still, since it is actually the convolution of the rectangle with itself three times, or the convolution of the triangle with itself once.
The convolution theorem says we can also obtain the convolution of the triangle with itself by Fourier transforming the triangle, multiplying that transform by itself, and then inverse transforming the result. The Fourier transform of the triangle is shown in the middle plot of Figure 11, and the output of our application of the convolution theorem is the final plot. We see that the top and bottom plots of Figure 11 match, as they should.
Convolution of a Modulated Impulse Train with Various Pulse-Shaping Functions
In this sequence of examples, I hope to strengthen your intuition about the effects and uses of convolution in the RF communication context. We’ll focus on convolving a modulated impulse train with various pulse-like functions, which is a good model for how a baseband RF communication signal is actually generated in practice.
A modulated impulse train is a set of impulses (here Kronecker delta functions) separated by a fixed amount, say , and each multipled by a random number called a symbol. We’ll restrict our symbols to the binary set . A typical binary modulated impulse train with impulse (symbol) separation is shown in the top plot of Figure 12.
The middle plot of Figure 12 shows the modulated pulse train after convolving it with a rectangular pulse with width much less than . Imagine flipping that rectangle, shifting it toward negative infinity, then sliding it along the time axis to the right. At each time instant (slide value), multiply the pulse and the impulse train together and find the area under the resulting curve. You’ll see that what should happen is that the narrow rectangle is replicated at the location of each impulse, but it is multiplied by the random value associated with the impulse. And that’s exactly what we see in the middle plot. The bottom plot in Figure 12 show the same process, but now the rectangle is the same width as the separation between impulses, so there is no gap between adjacent replications of the pulse. So we can create a pulse-amplitude-modulated (PAM) signal with any pulse function we wish by starting with the modulated impulse train and then simply convolving it with the desired pulse. If we multiply the signal in the bottom plot of Figure 12 by a complex-valued sine wave we obtain the textbook rectangular-pulse BPSK signal that is the signal used throughout the CSP Blog to illustrate and unify all discussed aspects of cyclostationarity.
In Figure 13 we perform a variation of the experiment of Figure 12, replacing the rectangular pulses with half-period sine functions. The middle plot shows the case of a relatively (with respect to ) narrow half-cosine and the bottom plot shows the case where the width of the half period is equal to the separation between the impulses. The bottom plot is exactly what the minimum shift keyed (MSK) signal uses as its inphase component; it uses a delayed version of the signal in the bottom plot with independently chosen symbols for the quadrature component. The delay is half the symbol period, , as in all conventional offset quadrature phase-shift keyed (OQPSK) signals, such as OQPSK, MSK, and GMSK.
Finally, in Figure 14 we repeat the impulse-train experiment with a highly practical pulse: the square-root raised-cosine (SRRC) pulse. Keeping with the method employed for Figures 11 and 12, we show two cases: a narrow-in-time version of the SRRC pulse (middle) and an SRRC pulse designed for the symbol rate implied by the separation of the impulses in the top plot. The bottom signal is a good model for a perfectly downconverted noise-free BPSK signal that uses SRRC pulses.
Convolution of a Modulated Rect-Pulse Train with a Multipath Channel
In this final “by MATLAB” convolution example, we look at the effect of a multipath channel on a simple binary rectangular-pulse signal. I choose a pulse width that is smaller than the symbol interval so we can more clearly see the effect of the channel’s impulse response on the signal. The top plot of Figure 15 shows the undisturbed binary signal. For this signal, the symbol rate is but the pulse width is . So the pulses stand out clearly.
The impulse response of the multipath channel is shown in the middle plot of Figure 15. Since it is complex-valued, I show the real and imaginary parts separately. It is a simple channel with non-zero magnitudes only for three distinct delays and samples.
The signal obtained by convolving the binary signal in the top plot with the channel impulse response in the middle plot is shown in the bottom plot. Because the pulse width is relatively narrow, we can clearly see the effects of the channel on one symbol before the next one arrives.
Convolution Example “By Video”
In this section we present several videos that show convolution-in-action. Each frame of a video shows the two functions undergoing integration in the convolution integral, their product, and the result of the integration itself. That is, we consider convolutions between and given by
and we plot and , , and on separate axes. Each video frame corresponds to a different value for . That is, these videos show the ‘flip-slide-integrate’ operations inherent in convolution.
The first video is shown in Video 1, and it corresponds to convolving a unit-height rectangle with width with another rectangle , which has height and width . From our previous work, we know the result show be a triangle with width and height .
In the second convolution video we convolve two triangles with width (base) of samples and two different heights. This is shown in Video 2.
Finally, in Video 3 we show the effect of a smoothing operation on a noisy sine wave. The noisy sine wave is convolved with a rectangle with unit area. The width of the rectangle is smaller than half a period of the sine wave. As can be seen by the convolution result , the effect of the convolution is to remove some of the noise experienced by the sine wave, more clearly revealing it in a plot. This convolution operation is also a low-pass filter because if you think of the rectangle as the impulse response of an LTI system, then the corresponding transfer function is a sinc function centered at the origin of the frequency axis. Only spectral components that lie within the main lobe of that sinc function are passed with significant amplitude by the linear system, thereby attenuating the noise components that lie outside of the mainlobe. We’ll talk more about low-pass, band-pass, and high-pass filters in a future Signal Processing ToolKit post.
A Weird and Kind-of-Cool Convolution Theorem Result
Let’s reconsider the humble unit-height rectangle function with width , which is denoted by . We’ve already seen that the Fourier transform of the rectangle is the sinc function:
as shown in Figure 16. Since the rectangle has unit height, when we multiply it by itself, it is unchanged,
This immediately implies that the Fourier transform of the squared unit-height rectangle is also the sinc function,
But by the convolution theorem, the Fourier transform of the product of the two rectangles is the convolution of their individual Fourier transforms,
This implies that the sinc function is invariant under convolution–a sinc convolved with a sinc is a sinc,
Since , the convolution of sincs is also just the original sinc. In Figure 17 we plot the Fourier transform of the function and also the convolution of that result with itself, and they are indeed identical.
The result (24) also implies that the sinc is equal to itself convolved with itself times,
Here is a movie that shows the sinc function being convolved with itself, resulting in itself:
I can think of a few more signals that possess this property.
Convolution in Probability Theory
Convolution comes up in many contexts, but an important one for us, as statistical signal processors, is in probability theory. We’ll look at random variables and random processes in detail in future Signal Processing ToolKit posts, but for now I note that if we add together two independent random variables, the probability density function of the sum is the convolution of the individual probability density functions.
That is, let the random variable have probability density function and the random variable have probability density function . If the two random variables are statistically independent (sometimes simply referred to as independent), then their joint probability density function is the product of their individual (‘marginal‘) density functions,
If we define the random variable by
then the density for is given by (The Literature [R149])
Convolution also figures prominently in modern machine learning in the convolutional neural networks used to perform various recognition and classification tasks. The ‘convolutional layers’ in the multi-layer deep-learning neural networks perform two-dimensional convolution, a subject we did not cover in this post, but which is a straightforward generalization of the one-dimensional convolution we typically focus on in signal processing. (Image processing, which has greatly benefited from machine learning, employs two-dimensional convolution for various image-processing tasks because the basic data structure of interest is a two-dimensional signal.)
Significance of Convolution and the Convolution Theorem in CSP
Convolution is ubiquitous. The most obvious example for us is the frequency-smoothing method (FSM) of spectral correlation function (SCF) estimation. The FSM estimates the SCF by first calculating the cyclic periodogram for seconds of input ,
After calculating the cyclic periodogram, the FSM then smooths it by convolving it with a pulse-like function (such as a rectangle!), which reduces the variability of the estimate and pushes it closer to the underlying theoretical SCF for the signal(s) in the data ,
In addition, any time we talk about narrowband spectral components of a signal, we are really talking about the application of a linear time-invariant system (‘filter’) to some data, and we have established that the output of an LTI system is the convolution of the impulse-response of the system with the input signal. So the narrowband spectral components that we extract and correlate to find spectral correlation are fundamentally connected to convolution.