ARM IDE

Any constructs used in common libraries automatically fall under the label of commonly used and well-tested features. Incorrect behaviour (either because the compiler writers did not properly interpret the standards, or because the compiler does not implement their interpretation) will be quickly spotted and handled by the build test suites.

My point is that there is no need for Plum Hall validation for common features and common language constructs - the compiler's standard test suites should cover all these features. Don't get me wrong - I am still glad that my compiler suppliers use Plum Hall to test and improve the tools. More testing, especially testing in different and independent ways, is always good. But I don't see any benefit for *me* to know the details of the Plum Hall validation tests.

Add to this mix, you have the problem that most embedded compilers have some non-standard additions or extensions that are essential for their use and effectively invalidate independent tests. If a compiler has an extra "flash" keyword, for example, then you must either test with the keyword disabled (which means testing with different settings from when you use the compiler), or with the keyword enabled (which means the compiler is no longer compliant, as you can't use "flash" as an identifier). Even if there is a middle ground such as "__flash" as the keyword, the Plum Hall tests will not cover this feature, which will be heavily used in real programs, meaning that their worth as an independent test tool is greatly diminished.

Well-written libraries will function identically independently of the sign of plain "char".

But it is certainly important to remember that there are parts of the C standards that are not fully specified, and are implementation dependent. Plum Hall cannot validate these (although perhaps it can report on them in some way) - with multiple correct answers, there is no pass/fail test. So source code that is dependent on these features may work on one Plum Hall validated compiler, and fail on another. The only way to get around this is to avoid using constructs that have such dependencies (for example, using "signed char" or "unsigned char" explicitly when it is relevant). That's much the same thing as avoiding obscure (but standards-compliant) language constructs that are unlikely to be well reviewed and tested by the compiler's standard test suites.

Perhaps I'm not making myself clear. As a compiler user, or car driver, I am concerned that the tools work as I expect them to do, when *I* use them. If I don't drive in temperatures under -25 C, then I am not concerned about how the car reacts in those circumstances. That's all there is to it - you can't start adding fantasies about how I *might* theoretically be able to go outside my assumptions. A car that explodes when you start the engine at -26 C is still within my required specifications. A C++ compiler that generates subtle bugs for a six-layer deep class hierarchy with multiple virtual inheritance and overloaded virtual "friend" operators is also within my required specifications for a compiler - it's not code that I would use or consider safe.

Again, you are misunderstanding me. I'm in favour of compiler manufacturers using Plum Hall (as CodeSourcery and other gcc testers do), and doing as much testing as reasonably practical - I just don't care about validations or certifications that are not relevant in my use of the tools. A car manufacturer will use the same tests of the UK and Germany (although they might distinguish between Norway and Oman markets), but I don't care about the results outside my area of interest.

I was thinking of the host here (as you already talked about different targets). That's what's relevant when you decide that it is the compiler binary that's important.

There are dozens of variants of XP (different service packs, different languages, different choice of additional software that can give different versions of system libraries).

And what about compilers that run under different OS's? There are plenty of commercial tools that run under a variety of *nix's.

Perhaps the compilers you sell are limited in this way. Other tools may come with multiple libraries, or multiple variants of their libraries (perhaps maths libraries designed for small size, high speed, or high accuracy). If you add in support for different operating systems, you get a whole new set of library issues - even the basic C libraries can come in variants for different OS's, and with or without support for things like multi-threading.

I think it is fair to say that if you want a compiler that can claim to be third-party "validated" or "certified" in some way, you are talking about a *very* restricted set of circumstances - bare-bones with no operating system, a specific small library, specific target processors, specific compiler settings, and specific host environments (including OS, processor, and installed additional software).

It's like Windows NT's famous "C2" security levels - they are only valid on a machine that is so locked-down that it is barely usable, and don't apply in the real world.

If my company felt that Plum Hall validation was relevant to our work, there would be no possible choice except to buy the test suite and run it ourselves on our workstations. Of course, it would be useful to know that our compiler suppliers had run the tests themselves - then we would know what to expect. But only our own results would have any real weight.

Validating a software design process and a software project for SIL is

*always* difficult. I haven't dealt with higher SIL levels, but I did work on a project with lower SIL levels (the software was written in assembly) - continuous and extensive functional testing and a design and development methodology that avoids or detects flaws, are the key elements. Choice of tools is obviously important - but tool validation by Plum Hall is neither necessary nor sufficient.

Don't worry too much about it - it was just a random example.

So can the binary if you try hard enough (or have a virus on your windows machine, as many do).

Clearly any testing or validation on a source code bundle will only be valid as long as the source code is not modified.

Reply to
David Brown
Loading thread data ...

Not all are implemented or commonly used. Complex maths?

Only if you do the full tests. And document all the unspecified, undefined and implementation specific behaviour the standard permits/

Not at all. Also the Plum-Hall and Perennial are independent tests. Writing your own tests proves little.

Correct. There is generally no need for users doing no safety critical or mission critical work to know the test results. You just need to know that the compiler manufacturer is taking all reasonable steps to go a good compiler.

When you get to Safety Critical the burden of checking is on you not the tool company to check suitability.

Certainly not.

Plum-Hall and Perennial test the standard C. Then you document the differences between the compiler and the Standard C. There are also other test suites internal to the compiler company such as build tests etc that you look at.

Validating a compiler usually requires an NDA with the compiler manufacturer to look at their other tests, development and build procedures.

This is not required for the majority but it is for Safety Critical. If there is an error in the system people get killed and you end up in court.

They should... but is [plain] char singed or unsigned ? It's not just the libraries. It is YOUR code as well. Do you pass a signed char or unsigned char to the libraries?

This is a problem if you use MISRA-C and or static analysis. It *may* also change if you change libraries.

Yes.

Quite.

Yes. There is more to it than just the one test suite.

Yes. However the libraries use [plain] char. This is something we have debated several times in the MISRA meetings.

The problem is the ISO -C teams do not want to enforce it one way or the other as it will break many compilers and much source code.

I agree. Best to avoid some things.

No if specified it will be well tested.

Yes but there are several thousand "I"s who all want to use the compiler differently

But what about those who do and never drive it in conditions over +2? The problem is (and I have see this many times) is what is "normal" for one users is abnormal for another. There is no "normal" that 95% of users agree on.

Otherwise we would ditch the hosted versions of C. The vast majority of C development is on embedded systems.....

You don't need them. Only those doing safety critical work do.

You don't but others do. The compiler company needs to test to all users. The validation for critical use also has to test the whole tool

You can only test the binary. The source is as everyone points out modifiable by the user.

Standard compiler system libraries?

Then you specify which you test on.

Then you test all of them if that is what is supplied. BTW we do I misunderstood what you meant.

Yes. That is why compiler Validation is not a simple thing.

No. Not VERY restricted. For example we test the IAR compilers. They run on Windows. We use XP for that. We test with all variants of their standard library. Usually on multiple targets. This can take a couple of days.

No idea.

For SIL4 you need to do that. For SIL0-3 you can use independent validation on a reference platform similar to the one you use.

Yes.

Yes. Not simple either. For the tools you are not validating the process used to build the tools just that for the following set of legal inputs you get the following legal outputs. However looking at the compiler manufactures system does increase confidence and due diligence.

Plum Hall does not do assembler testing AFAIK Neither does Perennial

Of course I will worry about it... I am curious. Not you have mentioned it :-)

--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills  Staffs  England     /\/\/\/\/
 Click to see the full signature
Reply to
Chris H

The compiler business is small and it is generally quite possible to contact the developers directly. Support staff deals with the day to day stuff but developers are generally very much involved in dealing with real issues. Unlike open source it is generally easy to determine who actually is responsible for specific design and implementation decisions.

"big-name test suites" offer organized methodical language testing to a specific standard. There are no similar test FOSS test suites available that I know of. GCC's regression tests the last time I looked were a small scatted collection of random code fragments.

One of the biggest problems facing compiler testing is the ability to deal with testing multiple code generation sequences that come from syntactically identical sources. Code generation for example that deals with variable placement context for example. Outside commercial tools very little serious work has been done to address this problem.

Regards

-- Walter Banks Byte Craft Limited

formatting link

Reply to
Walter Banks

In message , Walter Banks writes

Quite so I can directly get in touch with all the developers and project managers for any of the compilers I sell. Then again I have NDAs with them as a distributor. So far I have not had any legitimate question from a user that we have not been able to answer.

More to the point with the complete version control they have they can tell me EXACTLY who did which source file when and the full history. Also they have all the supporting documentation. I doubt you can do this with any version of GCC

Quite. The Big Name test suites are developed to a standard as high as safety critical software. Full histories and documentation with formal procedures.

Quite. The problem is that whilst the big name suites have some provenance, as the authors are also on the C and C++ standards bodies and helped shape the standard they are writing a test suite fro the GCC test suite does not have that similar standing.

--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills  Staffs  England     /\/\/\/\/
 Click to see the full signature
Reply to
Chris H

Of course it does. Neither Plum Hall nor Perennial are perfect. During the last five years, I've found two or three code generation bugs in a commercial compiler claiming to be verified by those two, with embarrassingly simple pieces of code (like a backwards for loop). Of course I hope my failing simple code snippets were now added to the compiler maker's test suite, to make sure this doesn't happen again.

Ironically, my score on gcc bugs is just one in 10 years. No, I don't want to draw any statistically significant conclusion from that, but I wouldn't say a compiler is significantly better than another just because the first's test suite has a "Plum Hall" label on it.

Define Safety Critical.

We're doing automotive stuff, and view almost everything as safety-critical. No people are instantly killed if the MPEG decoder in your car stereo crashes, the RDS interpreter tunes to a wrong frequency, or the display goes black. But it may irritate the driver (especially if the bug is accompanied with loud noise), distract him from traffic, and

*boom*. Still we hope to be able to avoid having to validate the compiler. We're doing classic testing, with all sorts of black-box and white-box tests, to get the chance of a software failure low enough. After all, software isn't the only thing that can fail. Compiler and software can be perfect if the CPU has a bug, or a resistor falls off the PCB.

Having a validated compiler just increases the probability that the problem was not introduced in the ".c"-to-".o" conversion, but does not eliminate it.

Of course a compiler will use things like kernel32.dll, which might behave differently on different versions of XP. One advance beta version of the linker we use behaved differently on XP and 2000.

Stefan

Reply to
Stefan Reuther

Well, you *think* you've answered their questions satisfactorily. Maybe they just gave up trying to explain the problem.

Multi-level support has its disadvantages when you want to go into deep details. Like, you want to know whether __mulli3 has been designed with CPU erratum #1009823 and application note #512 in mind, and whether you can safely call it from inline assembly (completely made-up example). If all you have is a first-level support contact, you'll have a hard time trying to explain them what you want.

Here, some phone companies implement the last mile using VoIP and DSL. Try calling them to find out whether they use G.711 or G.729. With one, fax will work, with the other, it will not. Now explain that to a support agent, who is trained to tell people to check that their devices are correctly plugged in.

As far as I know, most open-source projects (including gcc) have public, world-readable version control systems.

But if I have a problem I don't care who *did* X, I want to know who can

*fix* X. This isn't in the VCS.

Stefan

Reply to
Stefan Reuther

No.

Why?

--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills  Staffs  England     /\/\/\/\/
 Click to see the full signature
Reply to
Chris H

An end-user does not normally have these NDAs or this sort of influence with the compiler companies - getting developer contact can be hard or impossible if you are a small company with only one or a few licenses. Going through a distributor (such as yourself) is one way to get contact with the experts - assuming that all steps in the chain are happy to help.

Of course, that's a generalisation and there are plenty of exceptions. I'm sure Bytecraft will give excellent support directly from the developers, if we can judge by Walter Banks' contributions to this public newsgroup. I have also had excellent technical support from ImageCraft - not exactly a big player, but famous for their support.

On the other hand, I have had technical issues with a big name compiler in which I found a bug in their library. I even sent them a fix for it. I was basically told that they don't care about the problem, and that they were not going to fix it.

Technical support quality is definitely a widely varying priority for different suppliers - part of the job of a distributor is to know and recommend suppliers on such criteria.

I don't need to ask - I can simply look at the publicly available source code repository:

formatting link

Reply to
David Brown

Yes. That is how it works. However the problem is that some questions (and I have seen these questions sent simultaneously to several distis and the compiler company) are answered on page 2 of the manual.

In many cases the person asking the question has very limited experience and the language and the target and is asking the wrong question completely. This is why compiler companies use first line support and distis to weed out the idiots, simpletons and those who seem unable to read the manual

The other point is I can discuss the problem with the end user and the developer in more depth having an NDA in place. It helps I am a qualified and experienced SW Engineer.

The problem is that once a user has the developers email address they thing the have a personal friend and help line for life. The first port of call (before the manual) is the developer.

Where I get any support from inside a company I always remove any direct email addresses to the developers before passing on the information.

Getting direct access tot he developers doesn't happen in any other industry.

BTW can the users of your companies products talk DIRECTLY to you about the software functionality of your companies products?

Where required they do.

Quite so.

Interesting. Love to know who that was....

Distributors recommend specific tools on many criteria. (btw there is no universal "best" compiler for anything.... well that's not strictly true. For some of the arcane MCU there are some times "the best" but in other cases there are many factors to weigh up. None of the factors are size of margin before you ask!

For EVERY version of GCC?

You know what all the code does and why it is there?

--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills  Staffs  England     /\/\/\/\/
 Click to see the full signature
Reply to
Chris H

How do you know?

Granted, developer support is a *lot* more helpful than phone company support, but I have already got back bug reports for formal reasons without anyone having looked at the problem. One that I remember is that a filesystem library was unmarshalling stuff using 'char', not 'unsigned char'. Everyone sees that this is wrong by looking at the code; I sent file name and line number. The answer I got back was "please send a self-contained test case including an image of the failing file system". Had I been talking to a developer familiar with the code, not a support guy, that would probably have been fixed that in a second.

(Various events led to the situation that we don't use that library; hence I have not dug much further.)

Stefan

Reply to
Stefan Reuther

... snip ...

How do you compensate for variations in parameter passing, calling conventions, etc. in various libraries? Do you mean you have the common library source, and compile it under each system?

--
 [mail]: Chuck F (cbfalconer at maineline dot net) 
 [page]: 
 Click to see the full signature
Reply to
CBFalconer

We don't compile the library or the compiler. We don't start with the source.

We test the library supplied with the compiler. We test on windows XP, this is the host system for these compilers. They only run on windows.

--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills  Staffs  England     /\/\/\/\/
 Click to see the full signature
Reply to
Chris H

What "phone company"?

This is a problem I mentioned before there are three types of Char. the sign of plain char is implementation defined. The standard library uses plain char quite a bit

Absolute bollax. When talking to the internal support team or the developer the first thing they ask for is "please send a self-contained test case including an image of the failing file system". They want to see an example because a lot of users do not understand the compiler, the language or what is actually happening.

Statistically I have found it worse amongst those who say "If I had the source I could fix it" which is why I am very unconvinced when people say having the course of a compiler is useful

The other thing is the disti and the compiler support team may have seen this problem many times before.... and know the solution. It is not always (or indeed often) a compiler bug.

As for talking to "the developer" the support teams are often the same as the developer and the rotate tasks and have a spell on the support desk. In other places the support team is far better at handling support problems than the developers. It is a different sort of game.

The support team are usually qualified and experienced engineers with quite often far more experience than the people who call in. Especially when it comes to the language and compilers.

As for "it would have been fixed in a second" what is not true. You can just change something in a compiler. You need to look very carefully at what else it effects and do full regression tests (for the build) and full tests (Plum hall, maths, language extensions, corner cases etc etc but only AFTER you have looked to see if you need to add new tests. And done all the documentation

Some compilers where you have the source and can "fix things in a second" are a complete bag of nails and the source is a complete nightmare.

Like many you see the problem and the solution far too simplistically.

--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills  Staffs  England     /\/\/\/\/
 Click to see the full signature
Reply to
Chris H

You have just proven my point. First-level support assumes the user is too stupid to use the product, and users with a clue have a hard time to convince them that they actually know what they're talking about.

In this particular case, the problem was in CD multisession handling, thus "sending an image" would have meant to burn a CD and shipping it over the ocean, along with our hardware because the reference hardware has no CD drive. Expensive in time and money.

The problematic code was about this: char buffer[4]; fetch_buffer_somewhere(buffer); address = buffer[0] + (buffer[1]

Reply to
Stefan Reuther

In message , Stefan Reuther writes

The first line support I know are as capable and experienced as the developers. There is usually little difference between the two technically. The difference tends to be the developers are less tolerant than the support guys. The support guys will also be able to answer most questions faster because they have been asked them more than once.

No more than talking to the developers. What makes you so special and different to anyone else who calls?

I think you have just proved my point. If you can't describe the problem and encapsulate it without having to send the hardware and a complete CD a compiler developer is less likely to want to talk to you than a support person.

Assume makes an ass out of you and me...

Far be it for me to disagree with you

--
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
\/\/\/\/\ Chris Hills  Staffs  England     /\/\/\/\/
 Click to see the full signature
Reply to
Chris H

Alright. I misinterpreted your "with a single standard library. We test that."

--
 [mail]: Chuck F (cbfalconer at maineline dot net) 
 [page]: 
 Click to see the full signature
Reply to
CBFalconer

Well, as the saying goes: At court and on the open sea, we're all in God's hand.

You are liable if you are *found* negligent in a court of law. Now in an ideal world people would only be found negligent if they actually did neglect significant parts of their duties. But ours is, of course, not an ideal world.

On a similar note, it's impossible to validate any chip with more than about 60 pins, simply on the grounds that even a test of all possible combinations of static binary inputs to its pins (not to mention analog inputs, voltage thresholds, timing, sequence, ...) would never finish during the earth's life expectancy.

Reply to
Hans-Bernhard Bröker

You do realise he is talking about the source to the *library* here, not the compiler? An experienced developer of a high level is going to be just as capable of understanding, debugging and correcting a toolkit's library as the library developers are - assuming there is decent information available about things like calling and naming conventions in the library. They are not going to be as skilled at specific tasks such as writing the library code in a way that it compiles properly for all its targets, and perhaps they are not going to write code as optimally or portably. And they are certainly not going to be able to run all the tests and qualifications.

But in situations like this one (and cases I have seen myself), the user is perfectly capable of finding and correcting a bug in the library source code if he has access to it. The fact is that some (by no means all) compiler support lines cannot deal appropriately with this situation. They either assume that the user is the problem, or they follow rigid and inappropriate routines such as asking for complete examples of the problem.

I agree that having the source to the *compiler* is not helpful to most users when it comes to finding or fixing bugs (it can be useful for other things, such as for changing development platform, archiving tools, or as a safety net in case the compiler developers stop development on the tools). But no one is talking about compiler source code here.

Reply to
David Brown

No. Plum Hall and Perennial are language conformance tests. The kind of feature you are describing is implementation specific and not covered.

Plum Hall and Perennial both are well respected as independent test suites that validate language implementation against the standards. Both companies are part if WG-14 and offer an independent view of how the standard should be interpreted.

There is a lot of confusion about the meaning of successfully running these test suites. Compiler companies run these as one of several standard and private tests to validate and release compilers.

Regards

-- Walter Banks Byte Craft Limited

formatting link

Reply to
Walter Banks

I understand all about these support issues, and the reasons why it is often undesirable to allow users direct contact with developers. I have also been on the other side of the equation. Yes, we *do* have end users who talk directly to myself and other software developers - not many, but some experienced and technically competent end users have direct contact. It can be of great benefit to both parties. I've also had occasional direct contact with developers of hardware parts we buy - again, it benefits both sides.

The important thing here is that we are talking about expert users - not average users, and certainly not the "idiots, simpletons, and those who seem unable to read the manual".

For almost all open source projects, many low-cost tools, and a fair number of high-cost tools, a major source of support is from public web forums, newsgroups, or mailing lists. A good public support forum is frequented by the tools developers, as well as expert users who enjoy using the tool, and are happy to help other users. Clearly there are times when conventional technical support, either directly to the suppliers or via a distributor, is better suited - for example, you can't sign an NDA with a public forum.

Incidentally, the support I have had from ImageCraft has been mostly on their mailing list, with many of the questions answered directly by the developer. Many other people's questions have been answered by me, and other "high level" users.

I'd prefer not to name names (especially as I can't remember which of two possible suppliers it was...).

I have always been a fan of choosing the best tool for the job, and that choice depends on many factors - technical support is only one of them.

The quality of the logs and comments probably varies somewhat, as it does in any project from any developers. But the history goes back a fair way - look at .

Of course, that particular source code repository is for the main FSF gcc tree. There are a great many other gcc development trees for different targets, experimental branches, or alternative development branches. Some of these will be directly open to the public, others will be internal to their developers (the GPL license requires you to make the source code available along with binary releases, but makes no requirements on the source history). If you are getting your gcc binaries ready-made (such as from CodeSourcery) rather than checking out and building your own copy, you might want to ask your particular supplier about their source code tracking and logging. Of course, CodeSourcery commit most of their changes back to the main FSF tree, so you can look them up there.

Reply to
David Brown

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.