7
$\begingroup$

This is in regards to a recent question: DSP Benchmarks for ARM Cortex-M4 processor, where the OP asks where he can find benchmarks for his hardware. Do they add value to the site?

My concerns are:

  1. Questions on where to find a benchmark only serve to direct people away from this site, diminishing our potential to be a good resource of answers, not mere links.
  2. Such questions will keep on coming, one for each platform and more every time there is a newer model. Although an objective question, I fear that this will be DSP's equivalent of "How cool is the new Macbook Pro?"...

That said, I very much see the value in benchmarks as it's critical to designing systems. I think that it would be more beneficial for us as a site if we did the benchmarks ourselves! So rather than having questions that ask where to find information (which I propose be declared "not constructive"), we should strive to become the source for benchmarks for different products.

I do not yet know of a good way to go about that, but that can be hashed out in a different meta post. For now, I'd appreciate the community's input in deciding the fate of such questions.

$\endgroup$
4
$\begingroup$

I think that for practitioners who are tasked with actually implementing DSP techniques in a product, attainable performance of hardware targeted toward signal processing are a very relevant topic. The real problem with benchmarks, however, is one of relevance toward your application. By their nature, performance benchmark data is ruled by the marketing departments of IC manufacturers; the whole purpose of publishing the data is to increase the number of units sold. Because of this, you usually see synthetic benchmarks like Dhrystone MIPS or unattainable theoretical throughput numbers like "total number of MACs per second." These are of dubious value when it comes to actually evaluating a processor to see if it has the beef to handle your application.

In my experience, the best way to see if a particular part is a good fit for your application is to dig in yourself and evaluate it in some depth. This might consist of analysis of the architecture's instruction set, peripherals, etc. to estimate the number of cycles that your algorithms and I/O will require. Even better, most manufacturers have embraced the fact that making low-cost development hardware available is a good way to promote your platform; nothing beats getting ahold of the real article to actually verify that your requirements can be met. DSP platforms of all kinds, from microcontrollers to large microprocessors to FPGAs, are becoming more and more amenable to rapid prototyping early in the design phase.

And that is where I think the goal of making dsp.SE a place for publishing benchmarks becomes difficult. There is an almost incomprehensible number of combinations of hardware platforms, design constraints, and target algorithms; it's simply not practical to assume that you will be able to service everyone's performance estimation needs. Sure, there might be more common questions like "what embedded architecture is best if I need to evaluate as many 4096-point FFTs per second as possible?", but I would expect those sorts of pointed questions to be rare (but I would still say they are on-topic). More commonly, I would expect a software radio designer, for example, who is trying to determine whether his OFDM receiver can be implemented in real-time on a specific processor. I can't imagine a general-purpose benchmark that would give him sufficient faith to make a hardware choice; instead, good-old-fashioned analysis of the details is needed.

In summary: I have no problem with hardware-performance questions. They are a fact of life for those implementing DSP for a living. I don't think dsp.SE can practically be a storehouse of authoritative benchmark data. I also don't see this as a problem; sure, it points people away from the site, but so do many answers that reference Wikipedia or cite other sources. People who ask questions here shouldn't really care whether all of the answer is posted here; if they get a quality answer, then they will come back next time they have a problem.

$\endgroup$
  • $\begingroup$ Your point is well taken. Not having any experience with implementing in hardware, I wasn't aware that these benchmark data were dominated by the companies. My concern still stands though... as you said, there is an incomprehensible number of combinations of hardware platforms & constraints, so I'm not sure if questions on performance details for those would fall under "too localized". I feel that questions like "How should I go about evaluating the performance" are of much more value than "where can I find X" questions. $\endgroup$ – Lorem Ipsum Sep 23 '11 at 4:49
  • 2
    $\begingroup$ Having asked the original question, I will add a little background. I have an existing product that uses an old and relatively expensive DSP. I have been looking at alternative processors including the Renesas MX and ARM M4 doth of which have MACC type instructions. There are benchmark figures for the supplied Renesas libraries that are of the form the x-point FFT at Y MHz takes Z ms, The FIR filter with W coefficients takes Vms etc. I was hoping that there were figures somewhere for the ARM library. $\endgroup$ – uɐɪ Sep 23 '11 at 7:30
  • $\begingroup$ Attempting to evaluate the performance of a number of target platforms myself will be extremely expensive in tools (different compiler for each target), development boards and implementation time. $\endgroup$ – uɐɪ Sep 23 '11 at 7:33
  • $\begingroup$ @Ian Could you add some of that background to the original question? I think it's interesting, and it sparked some ideas for sources of answers that I didn't have reading the question before. $\endgroup$ – datageist Sep 23 '11 at 9:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .