The Altar of Optimality

When I was a wee engineer, I’d sometimes clash with other engineers that sneered at technical approaches that didn’t set up a linear-algebraic optimization problem as the first step. Never mind that I’ve been relentlessly focused on single-sensor problems, rather than array-processing problems, and so the naturalness of the linear-algebraic mathematical setting was debatable–however there were still ways to fashion matrices and compute those lovely eigenvalues. The real issue wasn’t the dimensionality of the data model, it was that I didn’t have a handy crank I could turn and pop out a provably optimal solution to the posed problem. Therefore I could be safely ignored. And if nobody could actually write down an optimization problem for, say, general radio-frequency scene analysis, then that problem just wasn’t worth pursuing.

Those critical engineers worship at the altar of optimality. Time for another rant.

Optimality Indoctrination

It starts early on, in calculus. We learn to find the extremal points of a differentiable function soon after we learn about differentiation. Find the derivative! Set it equal to zero! Use algebra to solve! Lather, rinse, repeat.

Then in engineering courses, we develop physical models that produce functions, and we then apply the calculus tools to find their extremal values. These functions, unlike most in calculus classes, have meanings connected to more-or-less real-world problems. Now when we find the maximum of some function, we’re finding the maximum velocity, or when we find the minimum of another function, we’re finding the minimum cost. The optimal cost!

And we keep doing this in school, over and over. What’s the optimal solution to some posed problem? Find the mathematical model, turn the calculus crank, draw a box around the solution.

And sometimes this is actually workable in engineering settings. We may have a subproblem involving, say, finding an optimal filter. That is, we have a good description of the input to the filter (say, a deterministic Fourier-series representation, or a stochastic-process model), some constraint on the form of the filter (say, finite-impulse response), and an idea of the desired output (say, maximum signal-to-noise ratio). Then we may be able to find the best (optimal) such filter by applying our mathematical tools.

Are There Optimal Engineering Solutions when the System is Complex?

Simple systems are often functions of a single variable. To find the best solution, you might need only find the extremal values of such a function, and calculus is your friend.

When the system is even a little bit complex, it is not a function of a single variable. It is often an unknown function of many variables, some of which are known. Then how do we find the optimum solution? In some cases it is possible to write down a cost function, or error expression, or SNR formula, etc., that is a function of all the identified variables. Then it can be extremized (maximized or minimized, as the situation warrants) using mathematics or by numerical evaluation/search. The function one writes down is often called the objective function.

In many cases, though, there isn’t a single objective function, or it can’t be determined. So in these cases, is there actually an optimal solution or design? ‘Optimal’ is relative to the chosen objective function, and if there isn’t one, then, well, no there isn’t an optimal answer. What is there, though? Trade-offs. That’s what there is, and that’s what engineering is all about, ‘making trades’ as engineers say.

For example, what is the objective function to maximize for designing an electric car? What is the objective function for a fighter jet? Or a C compiler? What is the optimal car, jet, or compiler?

The Ultimate Driving Machine

I’ve always been amused by the BMW advertising tagline: “BMW: The Ultimate Driving Machine.” But when I first heard it, I had already seen BMW models change multiple times in the past, and I knew that for sure there would be another set of different models next year. At any given time, though, BMW professes to have created the ultimate car. Er … driving machine. Anyway, for what possible meaning of ‘ultimate’ could this even be remotely true? None. There is no ultimate driving machine. There is no optimal car. There are just a whole bunch of tradeoffs that car designers can make in pursuit of a product that is appealing to enough paying customers that the company can make a profit by selling lots of them.

If BMW succeeded in creating the ultimate (optimal) driving machine, that would be the last model they produced. Why create anymore? You’ve got the ultimate. Ultimate means final.

For BMW, the tradeoffs skew toward high horsepower (large values of acceleration) and luxurious interiors (leather, wood paneling, great sound, etc.). But for the Toyota Prius, the tradeoffs are quite different. Gas mileage and low carbon emissions are at the top of this list, not zero-to-sixty in six seconds.

Connection to Historical RF Signal Processing Trends

Over the past three decades, I’ve noticed supplicants worshiping at the altar, clutching their likelihood-ratio rosaries and painting the world in drab colors of Guassianity ocher, statistical independence ecru, and textbook-signal gray. They’ll twist their signal models any which way they can to appease the Gods of Optimality, even if the model becomes so divorced from reality or from any actual problem needing solving that it should be discarded like so much dusty doctrine. But! But the Holy Crank can be turned and the Optimal Solution can thereby be found, praise be to the Seven.

So we ended up with papers that present some optimal parameter estimator for some textbook signal in white Gaussian noise, and then the next year a paper with the exact same signal setup except now the noise is exponential, and the optimal estimator (slightly different, like the predictions of the Second Coming by two slightly different numerological cranks) presented with a flourish and deep bow. The following year, another sect emerges, redoing the same estimator but now for a mixture model of the noise. And on and on …

For us heathens out in the fields, the optimal estimators and detectors were mostly useless. Not all! The Viterbi Algorithm is not useless. But that’s coding, really, not so much statistical signal processing. Our problem out in the laity was that the data that we use as input to the holy optimals did not quite correspond to the assumptions that let that oh-so-blessed crank turn. So they didn’t work well. Commonly this is referred to as an algorithm being ‘brittle.’ To complicate matters, the optimals were often too computationally expensive to use in many applications (like some CSP, I am oath-bound to admit) and so had to be approximated, sometimes rather roughly.

Connection to Modern RF Signal Processing Trends

In modern times, the form of worship has changed. We still want to minimize or maximize something, but we don’t want to do it mathematically. We want to get a computer to do it for us. Don’t bother me with mathematical models of signals and systems, for those things are of the world, and we seek otherworldly performance. We gather and present our oblations, dutifully and reverently (if not carefully) labeled and packaged. Then we Run the Machine to minimize the labeling error. And, lo! The error is indeed minimized.

By the munificence of the Gods, the error is minimized whether the labels and oblations are in correspondence or not! And we need not concern ourselves with the details, for the sanctitude of the Machine will prevail.

But, like the pious optimality worshipers of old, the new clergy keep producing Machines that minimize the error but fail when used by the laity out in the field. We have brittleness here too, and we know not how to soften it. We know not how to make general our all-too-specific optimal Machine.

A schism is at hand! On the one side are those that strive mightily to improve, polish, tweak, and tinker with the Machine’s holiest of parts. If only we can read the mind of God, they think, we will find the ultimate machine that will provide the final optimality. On the other are those that want to rend the Machine, tearing at its very fabric and structure, until it becomes general, useful, and tame. Although perhaps no longer provably optimal.

End of Rant.