[llvm-dev] Testing compiler reliability using Csmith (original) (raw)

Matt P. Dziubinski via llvm-dev llvm-dev at lists.llvm.org
Wed Dec 12 08:54:11 PST 2018

You may also be interested in the following resources on compilers correctness (articles, software, and talks -- from the general topics to the ones specifically focused on testing, validation, and verification):

https://github.com/MattPD/cpplinks/blob/master/compilers.correctness.md

Best regards,

Matt P. Dziubinski

On 12/7/2018 20:19, Philip Reames via llvm-dev wrote:

There's also a couple other active fuzzing efforts running concurrently:

* OSSFuzz is continuously running a set of codegen and individual pass fuzzers.  A bit more information can be found here: http://llvm.org/docs/FuzzingLLVM.html#continuously-running-and-finding-bugs * We (Azul Systems) have a continuously running Java fuzzer which exercises LLVM through our Falcon compiler, and regularly finds regressions and occasional deep long lurking issues.  We don't have a public bug tracker for this, but a sizable portion of our upstream bug fixing activity is driven by the output of this tool. Philip On 12/7/18 11:04 AM, Zhendong Su via llvm-dev wrote: Thanks, Vedant.  Yes, we have done a lot of testing of Clang/LLVM (and GCC) in the past several years (more details at https://people.inf.ethz.ch/suz/emi/index.html):

[GCC/LLVM bugs: 1,602 (total) / 1,007 (fixed)] [Reports: GCC (link1 <http://tinyurl.com/ojzdt78>, link2 <http://tinyurl.com/oxlkmjc>, link3 <http://tinyurl.com/jd3jnl3>, link4 <http://tinyurl.com/zs2826l>, link5 <https://tinyurl.com/yb7okzqd>), LLVM (link1 <http://tinyurl.com/qj8x5ry>, link2 <http://tinyurl.com/nc9celr>, link3 <http://tinyurl.com/qy58smb>, link4 <http://tinyurl.com/z7y8rwo>, link5 <https://tinyurl.com/y7wdf5b4>)] Best, Zhendong -------- Zhendong Su Professor, Department of Computer Science, ETH Zurich Office: CAB E 64.1, Tel: +41 44 633 77 72 https://people.inf.ethz.ch/suz/ -------- On Fri, Dec 7, 2018 at 7:42 PM Vedant Kumar <vsk at apple.com_ _<mailto:vsk at apple.com>> wrote: Hello, + Zhendong, who’s done a lot of work on automated testing of llvm.

On Dec 6, 2018, at 10:15 PM, Radu Ometita via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:

Hello everyone! We are working on writing a paper about testing the reliability of C compilers by using Csmith (a random C99 program generator). A previous testing effort, using Csmith, found 202 LLVM bugs, which represented 2% of all reported bugs at that time (PDF: https://www.flux.utah.edu/download?uid=114): . However, after this paper was published we are unaware of any further testing using Csmith, and we would like to ask you if you are aware of any such efforts or further results. Just speaking for myself here, I use Csmith as part of my pre-commit testing. vedant

Best regards, Radu Ometita, Functional compilers engineer @IOHK

-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181212/52a14ac5/attachment.html>

More information about the llvm-dev mailing list