Re: My Vintage Dream PC

in the mid-70s i started making comments about i/o slowing down significantly ... part of being able to see this was possibly having started (when I was undergraduate in the 60s) doing dynamic adaptive resource management ... and something I called "scheduling to the bottleneck". a recent reference (also mentions "re-releasing" a "resource management" product in the mid-70s, SHARE having called for making the cp67 "wheeler scheduler" available for vm370):

there is reference to comparison that I did in the early 80s, between a "current" system and a nearly 15yr early system doing essentially the same type of workload. My comment was that the relative system thruput of disks had declined by an order of magnitude of the period.

some disk division executives took exception and assigned the performance group to refute the states ... but after a couple week they came back and effectively said that i had slightly understated the problem. the issue was that processor power had increased appox. 50 times ... but disk thruput had increased only by 3-5 times (resulting in net relative system thruput decline of a factor of 10 times).

the performance group turned the study into a SHARE report recommending disk configuration suggestions to improve system thruput ... references to presentation B874 at SHARE 63, 8/18/84:

a little topic drift recent reference/post getting to play disk engineer:

As I've mentioned regarding relational databases ... the amount of real storage started to dramatically increase in the late 70s ... and systems started to leverage the additional real memory for caching and other techniques as method for compensating for disk thruput bottleneck.

in the 70s ... there was a little contention between the '60s database product group in STL (bldg 90) and the system/r (original relational/sql) group ... misc. posts mentioning system/r

with the older style database group claiming that the "implicit" index (for locate a record) in rdbms doubled the physical disk storage of typical database and significantly increased the number of disk i/os (as part of reading the index to find a record location). The system/r group pointed at that the physical record pointers that were part of the data significantly increased the manual management of "60s" databases.

going into the 80s ... the disk space significantly increased & price/bit significantly decreased (mitigating rdbms disk space penalty), available real memory significantly increased (allowing rdbms indexes to be cached, significantly reducing the disk i/o penalty), and DBMS people skill became relatively scarce and cost significantly increased. All of this shifted various trade-offs vis-a-vis 60s DBMS and RDBMS.

Note however, there is still quite a bit of use of 60s DBMS technology ... especially in various large financial and/or business critical operations. a few recent references:

above also mentions that when Jim left for Tandem, he was handing a lot of stuff off to me ... including consulting with STL 60s DBMS group and talking to customers about System/R ... a couple old email references:

note measuring access latency in number of processor cycles ... number of processor cycle latency to access real memory today is compareable to

High reliability = 2nd CPU is back up, not slave.

Andrew Swallow

Well, we've never had to do a full restore. I do occasionally dig into The Cave and pull an old CD or DVD backup, like when I'm working at home and want to see an older design, or the rare occasion when a file is missing or otherwise wrong in the M:\\LIB directory. The files are always there, no problems so far.

They're just files burned into DVDs. Not a lot can go wrong. Doing zero-based weekly backups onto write-once media gives us a lot of redundancy.

John

We burn and verify. Of the ones I've pulled from storage, some many years old, I've had no read errors. And the backup redundancy - a full set burned every week - is huge.

WHAT? These are ARCHIVE BACKUPS. Why would any sane person shred them? We KEEP them.

They're not cheaper unless you reuse them, and that has the propagating-corruption problem as noted.

Fee? The guy who burns the backups works for us, and it doesn't take much of his time to do it. The Cave is under my garage. One other employee takes an occasional set home for storage. And I take a set up to Truckee now and then.

What sort of backup procedures do you do?

John

It works and is very, very unlikely to allow us to lose anything.

Yes, but it over-writes existing copies of those files, no?

"Corrupted" can mean missing, or changed for some reason. We may not know that happened for years. Sometimes somebody will, for example, change a manual or a parts list, and later we want to see the older versions. A rolling backup would lose the old versions. We've also had a few occasions when a file just went missing, but both the original release medium, floppy or whatever, and the weekly backups are available in storage.

All burns done before the file was corrupted will be correct, and ones after the corruption date are bad. But we have them all.

Newly-released files are inherently tested, because they are what manufacturing uses to make stuff. It;s the older infrequently-used stuff that doesn't get tested often, but by that time we have tens, maybe hundreds of good archive copies.

Why is keeping baggies of DVDs, in boxes, in a few secure locations, a risk?

Can't do this kind of stuff in large, heavy

I don't do large or heavy.

My son-in-law does nuke weapons stuff, and works with explosives. I commented that that must be fun and he said, no: by the time you deal with the management, range safety, security, blah blah blah, you get nowhere near the shot, and there's no fun at all. When we want to blow something up, we go into the lab, or onto the roof, and blow it up.

Thank Goodness I don't have a real company. We just have fun and make money building electronics.

John

Those words, "hundreds of releases of the design", give me cold shudders. We use a military drawing control system, and any product - hardware, software, mixed - is formally released as a set of standalone, numbered documents with rev letters, all under a master bill of materials, itself numbered and rev lettered. Only manufacturing makes and ships things, and only from the formally released files.

The idea of casually shipping rev 2.3.04b is a nightmare.

Also

How do you know that you'll be able to run the VCS ten years from now, and that all the files will be intact? How do you know which customers are running which of the hundreds of possible versions of one product?

Who manages the VCS?

More important, of the copies of your things that have ever been shipped to customers, what fraction had one or more bugs?

Each of our formal letter release packages has a README file that notes the reason for and nature of all changes. If a bug was fixed, it's documented there. Releasing another package is a big public deal, and if it was because of a bug, everybody knows it. It seldom is.

John

Oh, yes. Compare to the cost of having someone write them, I suppose you _could_ give the job to an intern. And on the data is much easier to verify latter if it's on a hard disk. Even USB drives can be bought for around $100 a Terabyte and I AFAIK the data retention is better and naked drives would be better.

I'd checksum them though. That way one could do a quick check now and again. I remember on company that lost it's data even though it was backed up to two drives, because the writing machine had a bad SCSI board.

Chez Watt?

It's either zero or it's non zero. If you a one in a million guy in China there are a thousand people just like you.

Few real companies around, I suppose.

It's shocking how many companies can't find source files, or can no longer assemble the tools to edit and regen products. I've shipped over 3000 temperature controllers because a biggish British company lost the business; they refused to modify some code, because they couldn't modify the code. Well, their thermocouple front-end sucked, too.

John

Proper file maintenance and backup won't (probably) affect *this* quarter or the next, so it gets put under the VP of blue sky which is a position generated mainly to satisfy equal opportunity laws.

The Silicon Valley culture - use the trendiest tools and languages, job-hop often, product lifetimes measured in months - doesn't encourage long-term planning.

John

There is no such thing as a write once hard drive, idiot.

That stupidity is pretty damned pathetic coming from someone claiming to be an engineer.

No, it does not. You are supposed to check your system for errors BEFORE performing the backup, ass.

If you performed it correctly NO files will be corrupt. If they were written as a valid file, and the app put corrupt data in there, that is another thing entirely, and that will still propagate onto any backup media or methodology you are planning on using.

You are obviously clueless about it, and not caring to reduce the cost of your belabored method of choice.

Also, if you had any brains at all, you would simply create a RAID solution with bit striped, hot swap drives in place, and then you only need to do an incremental off site-over-the-net backup each day to another HARD DRIVE device at the remote location. Your in house data is

That is where things are right now. You go out and get an ultra SCSI SAS solution that incorporates a RAID array, and a chassis with hot swap drive bays and power supply bays.

Once a file is (successfully) written, it is impossible for it to become corrupted short of the entire bank of drives taking a dump all at once. Even then, it is technically recoverable, just a bit more costly to do so.

The data is more secure (absolutely secure, in fact), and it reads and writes faster by way of both the interface as well as the array schema.

SAS is the way to go. You can even get 15,000 rpm mini drives and make small arrays that you could slip in and out of a system as an entire array. No need for drive bays with those little critters.

So what?! The whole idea is to write the CURRENT data set, dingledorf!

Especially thankful that it is not "large or heavy" as well.

Buy some hard drives and an NAS chassis John. It is cheaper, faster, more reliable, and you can stop pissing and moaning about your engineers' private directory sizes.

Deviation approvals are a pain in the ass too.

That is why full revs are utilized often, and also why the number climbs quickly until the final design cycles have been run through.

Mid rev shipments MUST show as deviant from the final design or you can experience serious field problems that in some cases could cause serious failure or repair mode problems at inopportune times.

Full write with full verify against original data.

We never ship "mid rev" anything. Only formally released things are shipped, and releases advance by rev letter. And we know exactly what hardware and firmware every customer has, on every serial numbered product. If we do find a bug, or make an improvement, everybody gets notified.

Bug fixes are free, forever.

John