How good is peer review really?

In the refereeing I've done, and the refereeing I'm aware of there's always a trail of personal acquaintance between the editor and the referees. I ca n see how editors under pressure to find referees might cut corners, but th ey shouldn't cut so deeply that they could send out stuff to be reviewed to fake addresses purporting to be those of real people.

--
Bill Sloman, Sydney

"Bill Sloman"

" In the refereeing I've done, and the refereeing I'm aware of there's always a trail of personal acquaintance between the editor and the referees. I can see how editors under pressure to find referees might cut corners, but they shouldn't cut so deeply that they could send out stuff to be reviewed to fake addresses purporting to be those of real people."

** Crikey - ya might as well get usenet posters to do the reviewing !!

No shortage of folk willing to take a peer at stuff there ......

.... Phil

Peer-review is still somewhat subjective. One of my earlier papers was rejected the first time, because one of the reviewers was is a who's who at UCLA, and our paper went against his cherished ideas and beliefs. The funny thing that my the then adviser found out later was that another reviewer was a past student of the first guy. The same paper, with some minor modifications was published at a later conference. Another time, our paper was accepted by 2/3 votes the first time, but the rejecting reviewer raised points completely irrelevant to our work. It appeared that the rejecting reviewer did not have good grasp of our main topic of work,

It is always worth remembering that as a rough heuristic about 10% of everything published in peer reviewed journals is subsequently shown to be incorrect by later researchers. Once it has been published then other researchers will try to reproduce the observation and/or write irate letters to the editor pointing out it is complete bollocks. More usually there are comments suggesting alternative interpretations and pointing out any ambiguities or minor flaws in the reasoning.

That is the beauty of the scientific method. Mistakes are always found eventually because nature is the final arbiter in any dispute.

BTW It doesn't sound like a very scholarly journal if they accept "expert" peer reviewers with no substantial track record in the field (or in this case just random alias email addresses)!

There can be really serious travesties where a fantastically important paper gets refused publication because it is so far ahead of its time that no reviewer would give it the time of day. The unfortunate Russian chemist Boris P. Belousov was one such victim in the 1950's and sadly it broke him. He didn't live long enough to see his reaction widely popularised after it was rediscovered by Anatol Zhabotinsky in the

1960's published 1968 and picked up in the West. The full detailed mechanism of his flip flop chemical clock was not fully understood until 1972 - a feat which required a collaboration of geniuses!

There are only a handful of such periodic clock reactions known.

--
Regards, 
Martin Brown

XXXX my comments. Martin, you raise some interesting points...

XXXX OK, but how do you know that it's not 60% incorrect, but only 10% caught?

XXXX I respectfully disagree about the scientific method. Back when Sir Bacon 'invented' it, 1620, the world was a much simpler place, and science was concerned with simple cause-and-effect, such as the nature of light refraction. Today, most scientists are still using a method invented in 1620!

So, in a sense (poetic license alert!), current-day scientists are half-dead zombies who get wheeled out to prop up various out-moded ideas which are often money-grabs. If you look carefully at scientists, you can see the small wheels down near the floor, and also the slots in their heads where synchronized memory cards are plugged in.

I jest, of course, but the idealism that animated Bacon to "find out what is true," does not work for complex systems and current societies. The Bacon system is called "logical positivism," and has been out of date or about 50 years now. Some current theories in philosophy of science say that scientists just copy each other mindlessly, until there is a crisis of "cognitive dissonance"due to new ideas. (Kuhn, Struct Scient Revolutions, ca. 1965.) He popularized the phrase "paradigm shift."

But I myself don't think that scientific revolutions have to happen, especially for complex systemic questions.

Today, science cannot explain 95% of the universe. (dark matter) In effect, the certainty of science as a generalized knowledge system is shown to be a hubric fraud, and it's priests just money changers at the temple. Or have they not told you, "Well, I am authority, but I can't explain near 100% of what is going on?"

huh?

everything published in peer reviewed journals is subsequently shown to be incorrect by later researchers. Once it has been published then other res earchers will try to reproduce the observation and/or write irate letters to the editor pointing out it is complete bollocks.

I've written a few of them, and seen some of the published.

d pointing out any ambiguities or minor flaws in the reasoning.

ught?

The good stuff gets followed up, cited and expanded on, but that's likely t o be closer to 10% of what gets published. The median scientific author onl y publishes one paper, and I suspect that most papers never get cited - not because they are wrong, but because they aren't all that interesting, or w eren't taking the field in a direction that turned out to be productive.

eventually because nature is the final arbiter in any dispute.

rancis Bacon 'invented' it, 1620, the world was a much simpler place, and s cience was concerned with simple cause-and-effect, such as the nature of li ght refraction.

Bacon lived from 1561 to 1626, and refraction was discovered by Ibn Sahl in 984 AD.

Bacon wrote quite a bit about empiricism, and his ideas were influential, b ut modern science is more about documenting observations and publishing the m in peer-reviewed journals. "Inventions" like the Royal Society came quite

avans", and the British "Philosophical Transactions of the Royal Society" - didn't start publishing until 1665, and they've been evolving ever since.

Ignorant clown.

ead zombies who get wheeled out to prop up various out-moded ideas which a re often money-grabs. If you look carefully at scientists, you can see the small wheels down near the floor, and also the slots in their heads where synchronized memory cards are plugged in.

Funny. I've met quite a few scientists, and these features haven't been obv ious.

is true," does not work for complex systems and current societies. The Ba con system is called "logical positivism,"

Actually "logical positivism" dates from the 1920's. Bacon wouldn't have be en silly enough to endorse such an absurd creed, and Popper's doctrine of " falsifiability" probably has more adherents today, though most scientists s eem to be happy to define science as "what we do", and Michael Polyani's "p ersonal knowledge" formalises this point of view in a way that makes a lot of sense to me.

So Haitic got the death date for logical positivism more or less right, tho ugh he was hopelessly mistaken about about it's origins.

py each other mindlessly,

Something of a misrepresentation. Accepted theories tend to stay accepted u ntil there's evidence that makes it necessary to set them aside. There's no thing "mindless" about using tools that work, as long as they do work.

Struct Scient Revolutions, ca. 1965.) He popularized the phrase "paradigm shift."

Typically, there's an influx of new data that can't be fitted into the old structures. The intellectual revolution that made Einstein famous grew out of the wealth of new data coming from spectroscopy. Planck didn't invent qu antised radiation to explain spectral lines, and it took Einstein's insight to go from the wavelength dependence of the photo-electric effect to the p erception that electromagnetic radiation really was quantised, rather than just acting as if it was.

cially for complex systemic questions.

It can take a while.

t, the certainty of science as a generalized knowledge system is shown to b e a hubric fraud, and it's priests just money changers at the temple. Or h ave they not told you, "Well, I am an authority, but I can't explain near 1

00% of what is going on?"

Sciences isn't a "generalised knowledge system" but a mechanism for pulling together the knowledge we have and for creating a framework that can accom modate new knowledge and reconcile it with what we already know.

There's not a lot of hubris involved. The fact that we know that there's a lot of "dark matter" in outer space - or something that makes ordinary matt er behave as if there twenty times as much matter around as we can see - pu ts us ahead of the last generation of astronomers, who got excited by the i dea that there were other galaxies around, and that they were receding from us. We now have a lot more to know we are ignorant about ...

--
Bill Sloman, Sydney

Any system conceived by humans can be defrauded by humans.

I think that peer review is a lot like democracy, in that it's a crappy way to do things, but it's better than all of the alternatives.

And in this case, it sounds like the problem isn't peer review in general

-- it's peer review done sloppily and without the normal checks and balances. It's not quite as bad as the "democracy" in Soviet Russia where the voting was done by putting your name on a slip of paper and dropping it into a box with the candidate's name on it, under the watchful eyes of the local police, but I can certainly see ways to make it better.

--

Tim Wescott 
Wescott Design Services 
http://www.wescottdesign.com

Reading comprehension problems again, Joerg? The article has nothing to do with the peer review process, it has to do with a fraud perpetrated by a si ngle individual in Taiwan. You remind me of the class dumbass always asking stupid and only peripherally related questions to mkae it look like he's p articipating.

Peer review is unbeatably good. In this case, it was the peers that noticed the conspiracy, after all. By contrast, the Catholic heresy trial against Galileo was carried out by folk who didn't know enough to ask the INTERESTING question, 'tell us about your telescope'. There ensued a few centuries of the church being cast as anti-science, and they found a way to recant (basically, pardoned/exonerted a convicted heretic).

Review by non-peers: very slow and inefficient.

For the interested, you can get Galilean telescope plans and parts here:

to

other

usually

caught?

Bacon

refraction.

half-dead

often

wheels

synchronized

what is

Bacon

50

scientists

1965.) He

especially

effect, the

hubric

not

is going

Russian

it

until

journal-retracts-60-articles-smashes-peer-review-ring/

general

You are catching on. Now add the creation of a group of new "journals" with a nice coterie of aligned editors and reviewers and you get AGW.

?-)

-

l

-- it's peer review done sloppily and without the normal checks and balances. It's not quite as bad as the "democracy" in Soviet Russia where the voting was done by putting your name on a slip of paper and drop ping it into a box with the candidate's name on it, under the watchful eyes of the local police, but I can certainly see ways to make it better.

with a nice coterie of aligned editors and reviewers and you get AGW.

Don't be silly. The top AGW papers get published in Nature and the Proceedi ngs of the National Academy of Science (PNAS).

Since the members of the US Academy of Science get roped in to edit stuff f or PNAS, your "coterie of aligned editors" is drawn from those scientists t hat have been recognised as the best in the country - not a great basis for a conspiracy theory."Nature" is just a top journal across a number of fiel ds - it seems unlikely that the pool of referees its editors use for AGW pa pers is any less good than the ones they have lined up for other areas.

But conspiracy theory enthusiasts aren't confined by trivial concepts like plausibility.

--
Bill Sloman, Sydney

It is better than that. Most scientific journals make a real effort to have reviewers that do fair reviews and helpful criticism of papers. You could certainly argue there are far to many low grade journals now.

The Land of the Hanging Chad and disenfranchised voter can hardly criticise other democracies for their dodgy voting practices.

Or so *you* would like to believe.

It is much closer to the mark to say that there are a bunch of rabid vocal right wingers with close connections to big tobacco that are spreading lies about the entire scientific community to try and persuade the general public to ignore the long term risks of AGW.

Basically the frog in a pan of luke warm water on the stove has been tricked by these fossil fuel lobby conmen to turn up the the gas.

The only way that Americans will ever take AGW seriously is when the entire mid-West is reduced to a permanent dust bowl or major cities are regularly underwater. New York, New Orleans and Florida Keys are good candidates although it would be much more fitting if Houston got trashed. ISTR Houston last took a really big storm hit in about 2001.

--
Regards, 
Martin Brown

Actually they did but that knowledge was not suitable for public consumption. The Vatican Observatory was founded in 1578 predating Galileos trial by more than 50 years. They were no slouches either and were doing cutting edge research. The Jesuits had made their own telescopes by 1611 and had repeated his observations - the problem was with the suits and The Pope higher up the chain of command...

It is more nuanced than the usual popularisation would have you believe. The only certainty is that where scientific evidence and The Holy Bible do not agree the zealots insist on the latter being right. (and no amount of evidence will ever change their minds)

A high proportion of Americans are still struggling to come to terms with true geological timescales and prefer Bishop Ushers 6000 years :(

Yes. It has only taken the Catholic church a mere 359 years to finally admit that Galileo was in fact right and the Earth did orbit the Sun!

It is a curious fact that although the Catholic church never publicly admitted to having heretic scientific knowledge the Jesuits used it. They included scientists and engineers of the highest order and thanks to them trying (and for a while succeeding) to convert the Chinese emperor to Christianity we have detailed diagrams (wood block prints) of medieval technologies like cannon boring and the making of various scientific instruments. Ferdinand Verbeist beat the top Chinese astronomers in a deadly game of chicken predicting the future eclipse calendar armed with the latest (heretical) European knowledge.

The Jesuit observatory at the Vatican is still capable of good work. Their library contains a fabulous collection of ancient texts.

If you don't mind an upside down image and some chromatic aberration and large magnifying glass and a small eyeglass or loupe will do at a pinch. Cardboard tube to mount it.

Should be good enough to see Jupiters Moons and phases of Venus. (cheap opera glasses would be even better)

--
Regards, 
Martin Brown