software situation

Don Y snipped-for-privacy@foo.invalid wrote: |------------------------------------------------------------------------------| |"The fallacy of RAID is that the larger the array, the more likely | |a *second* fault manifests while attempting to recover from the | |first." | |------------------------------------------------------------------------------|

Hmm. How so? I had not heard this before. (I never yet am responsible for a RAID, but Don Y incites me to be skeptical about RAID boasts should I ever become responsible for a RAID.)

|------------------------------------------------------------------------------| |"It's like having gold speaker wires -- something to brag | |about that has no real value in most cases." | |------------------------------------------------------------------------------|

But redundancies are really valuable in most cases. I make backups. I might not use a RAID, but I am not going to give up on backups!

|------------------------------------------------------------------------------| |"There was a 128 byte structure buried in the design implemented triply | |redundantly. *AS* three structures having identical values located in | |contiguous blocks of memory. | | | |The number of ways that such an approach could fail that could not be | |reliably detected (e.g., address failure, device failure, etc.) was laughable.| |And, the recovery algorithm was poorly conceived (what if no two copies | |agree COMPLETELY? Do you install a default value? What should that value | |be? If predictable, then this is an attack vector...)" | |------------------------------------------------------------------------------|

Oscar Portela Arjona and I worked on software for a satellite platform (important because if the platform is faulty then no payload can succeed) and for a payload of this mission (which in a sense is less important). We did not design a payload-computing layout. This payload-computing layout was designed to be redundant due to the motivation that in space they use redundancy for critical systems to be safe. So Oscar Portela Arjona insightfully contrasted this uncritical payload's redundant computers with this critical platform's single (i.e. non-redundant) computer. Oscar Portela Arjona was so angry at mismanagers that Oscar Portela Arjona quit.

|------------------------------------------------------------------------------| |"This is the same mentality behind folks adopting RAID or ZFS needlessly. | |They don't think their decision through and, instead, convince themselves | |that they have taken concrete measures to improve the reliability | |of their data! (how often should you do a patrol read? how do you | |respond to errors detected/corrected? what criteria do you use to | |retire media? do you support a hot spare? how many??)" | |------------------------------------------------------------------------------|

I tried ZFS (via an open-Solaris distribution); UFS (via e.g. FreeBSD); ext (via Linux); Minix; and even FAT (via FreeDOS, avoiding installing a never installed copy of Windows XP still in its shrinkwrapping) in 2008. Only ZFS (Solaris) thereof became corrupted: and it became corrupted (and unbootable) within only a few days of testing! (S.

The reverse is also true for folks looking for careers in hardware design. If you don't know how the software will interface with "your" hardware, the answer is likely: "poorly".

And, to all, "coding" isn't software design in much the same way assembling a prototype isn't hardware design.

Divide-and-conquer, information hiding, one-page "programs" all suggest an OS should cater to small, "decomposed" problems executing in *true* parallelism (the multitasking illusion doesn't work in the era of multiple cores/hardware threads, distributed systems, etc.

To these criteria, I've added "accountability" as you want to be able to wrap a virtual "box" around any set of actors and pretend THAT is a product with real world constraints. E.g., how do you ensure a task doesn't disproportionately (ab)use resources meant to be shared with other co-operating tasks? (And, what do you do if/when it does??)

... and comment lines in the source code do _not_ constitute "documentation".

It takes a long time to rebuild an array. And, while you are doing so, the array is operating in degraded mode -- there is no redundancy provided for the data present. The unrecoverable error rate, while considered miniscule, becomes a certainty as the size of the array grows (most of my data farm are 64T boxes).

To rebuild the array, you have to read EVERY bit, compute the parity and rewrite, as needed. All the while, hoping nothing hiccups. (and, this assumes you've done this recently enough that you are awae of the gotchas that you may encounter as you can't hit "PAUSE" and come back to it when you've had time for research).

To *detect* corruption, you have to do this periodically (patrol reads) which competes with your use of the medium.

I've approached this differently. I store plain copies of everything along with their hashes (in a separate database). When I spin up a box, I have a daemon that runs on said box which queries the DBMS to determine what should be checked "now" (how far has the patrol read on THIS collection of drives progressed?). It computes the hashes of the files in question and compares to the hashes stored in the DBMS, alerting me to any files whose hashes have been corrupted.

It then reports the locations of alternate copies of those offenders (which might be on "cold" spindles). E.g., a file may exist under different names in different places on the same spindle, different spindle or in a different box. C:\2025\taxes\federal D:\tax_returns\federal\2025 C:\mystuff\business_records.rar

It requires automation for mainstream use but provides far more practical redundancy for a homelab where the goal is to just KEEP stuff and accesses are infrequent (e.g., if a file has been corrupted, is it because the volume should be replaced? if so, then you'll want to be able to find copies of everything ON that volume, along with a description of the filesystem structure -- which may no longer be visible!)

Of course! See above. I no longer use "backup software" because they all want to wrap the backed up data in some proprietary container. Instead, just "throw" a copy of <whatever> off to <somewhere> and let something track where it is and what it duplicates.

ZFS requires a fair bit of resources to work properly.

One thing folks tend to forget is the *hardware* can fail. Now, how do you "recover" your data?

With simple volumes, I can move a drive from one machine to another, as may be appropriate, and still have access to the drive's contents. No special routines to remember to bring it online for access/recovery -- it's just a disk.

ECMMWF has access to an AI weather forecaster that they refuse to use beyond in-house study. They continue their conventional weather forecasting operations as usual without it.

This is unlike many other applications, where people use AI without a clue as to how the AI is arriving at its results.

People start paying attention when extreme weather is involved. The forecasting is accurate enough these days to give people plenty of time to prepare. U.S. East and Gulf coasts rely on it.

Ross Finlayson snipped-for-privacy@GMail.com wrote: |----------------------------------------------------------------| |"The "since" and "until" are often among the most suprising | |or unintuitive keywords represented in language, since | |they're as of temporal modality, vis-a-vis "until" and "while"."| |----------------------------------------------------------------|

A then supervisor had drafted a proposal in English to be read by persons who do not natively read English. He had written "Since" for "Because". I have proposed replacing "Since" by "Because" but he insists on using "Since" for "Because" while professing that they are sufficiently skilled at English. (S.

Eschew comments as they are yet another thing that can get out-of-sync with the code. People are lazy; given the choice of reading the comments and the code, the comments are likely easier and exist at a higher level of abstraction (with which one would have to infuse the code). But, the *code* is what the compiler reads!

Comments should describe the *design* and the skillset of the developer should be able to suss-out the implementation and its compliance with said design.

No. He probably didn't say or write that.

I bought one of the original IBM PC 5150 computers and spent a few years dealing with the limitations of the architecture, associated memory map and subsequent band aids such as Lotus-Intel-Microsoft expanded memory and expanded memory:

Original IBM PC memory amp:

Later additions (extended memory, expanded paged memory, and expanded video memory):

Without the various expanded and extended band-aids, the original IBM-PC was limited to 640K for user program memory. The remaining

IBM designed the original PC hardware and memory map. Bill Gates and Microsoft were originally hired to port the Microsoft Basic interpreter to the new hardware after IBM discovered that they didn't own any software or programming languages for their new computer. If Bill Gates had said something like the quote, he would have said

Bill Gates was famous at Microsoft for program reviews which demanded smaller and more efficient code. He knew that program memory and computation space were limited and did his best to work within the hardware limitations. For example:

There have been numerous discussions on this topic since 1980 (the year the IBM PC was introduced. You should be able to find something relevant with any search engine. For example, I found this article to be rather interesting:

Don Y snipped-for-privacy@foo.invalid wrote: |-----------------------------------------------------------------------| |"> I tried ZFS (via an open-Solaris distribution); UFS (via | |> e.g. FreeBSD); ext (via Linux); Minix; and even FAT (via FreeDOS, | |> avoiding installing a never installed copy of Windows XP still in its| |> shrinkwrapping) in 2008. Only ZFS (Solaris) thereof became corrupted:| |> and it became corrupted (and unbootable) within only a few days of | |> testing! | | | |ZFS requires a fair bit of resources to work properly. | | | |One thing folks tend to forget is the *hardware* can fail." | |-----------------------------------------------------------------------|

That hardware never fails. In particular it was a new computer which was switched on for its 1st time in Summer 2008. Its ZFS installation failed before 2009.

Most of my (FAT,-non-ZFS)-USB-flash-drives misadventures do not actually involve the data on these drives becoming destroyed, but instead their File-Allocation Tables becoming ruined (apparently because of USB power problems, possibly accompanied by operating-systems bugs). So files are on them, but directories' layouts are flattened and files are renamed to unhelpful names. So in theory I could recover files, but I save many intermediary drafts, which would be a good policy for tracking and accountability, but I can not easily determine which files amongst thousands are the best versions.

Thanks for answering about RAID. (S.

Recall that CP/M machines were common when the PC was first introduced. And, had memory constrained to 64KB -- an order of magnitude less.

With comparable performance (given the PC's 8086 was running at ~5MHz!).

The Reading Machine (predated the PC by several years) had 16K words of memory and no possibility of secondary storage (disk) -- beyond what was used for IPL (in case the core got corrupted). The bootstrap was sixteen (16) words in a tiny bipolar ROM.

Arcade pieces (1981) were in the 16-24KB range in terms of complexity and had to generate video and audio in near-real-time. (something that would have been laughable with a PC).

The fact that you don't NEED lots of resources to achieve meaningful work is a red herring; you add resources to make implementations more robust or maintainable. Tweaking the boot loader on the Reading Machine required a disproportionate amount of effort as you couldn't add a *17th* word! It's a lot easier -- and more reliable -- to be able to load a second image of a bootstrap and *verify* that before relying on it (as anyone who had to flash from RAM learned when stuck with a "bad reflash" bricking their device!)

It's amusing how much has been "wasted" in efforts to save a few pennies in most products!

The point is, you need the electronics and software to remain operational in order to be able to recover the data on the disk. In practical terms, it means you need a redundant machine to have any hope of accessing or recovering data if the first machine dies (or, if the HBA on the first machine develops problems).

My San resides on a bunch of similar (but not strictly identical) servers, chosen because they each support multiple spindles. Sadly, the HBAs in them differ in their native capabilities. Some can only create RAID volumes; JBOD isn't an option.

Creating a RAID volume (even RAID0) on device X gives you no guarantees that it can be accessed on device Y -- even if the two devices are "the same".

OTOH, if the HBA can be configured to operate as JBOD, then you can freely swap physical media between machines (or, between "ports" on a single machine).

I've been systematically converting every box to this sort of disk interface to make "moving data" nothing more complex than ejecting a volume and plugging it in, elsewhere.

The COTS NAS boxes are lined up for obsolescence each time I rescue another multi-spindle box that can host its volumes. (cuz I can't dick with closed software and the constraints it puts on the volumes it supports!)

Of course, I have this luxury because I'm not pushing for performance and don't have to support "other users". OTOH, it's nice not to have to waste shelf space on text books or file cabinets full of records...

Oops. The ECMWF was founded in 1975. I believe that forecasts were widely distributed starting in about 1992.

Key Historical Milestones

Nov 1, 1975: The Convention formally comes into force, establishing the Centre.

1992: Launch of operational ensemble forecasting (EPS) to predict weather uncertainty.

2020s: Post-Brexit, operations shifted to include Bonn, Germany, and the headquarters initiated plans to move to the University of Reading's Whiteknights Park campus.

Jeff Liebermann snipped-for-privacy@cruzio.com wrote: |------------------------------------------------------------| |"

Ah, here the idea of "co-operative scheduler" (vis-a-vis "pre-emptive scheduler") has that there's a notion of the model of an o.s. (scheduler, allocator) of co-operation vis-a-vis "the re-routine", which is a sort of idea like "co-routine", where basically everything is non-blocking by design and convention, and instead of a co-routine stack is a sort of memo-ized monad, then about matters of the scheduling like "I cut you pick", "straw-pulling", and "hot potato", with anti-gaming built in to the algorithm, device drivers are provided as "generic universal drivers", then that user-space gets a usual "quotas/limits" and while a contrived user-space program may actually run a hot inner loop, otherwise the deadlock/starvation and other issues in concurrency are to be figured out, for the allocator/scheduler.

I.e., the usual idea of the "co-operative" lives inside the kernel, user-space is nominally adversarial and the network is nominally un-trusted. System calls it's figured are implemented as of a "co-operative" implementation.

It's mostly as of a "design" while though I put it through the wringer as it were of some "large, competent, conscientious, co-operative reasoners" or a "bot panel", I can post a link or reference or all the text of them.

The CEOs are now starting to lay themselves off.

Most CEOs are using the embrace of artificial intelligence as a cover to lay off staff and cut payroll costs in the name of “efficiency.” But a couple are using it as an excuse to lay themselves off.

I assume tasks (processes) run without interruption until they need an unavailable resource, at which point, they block.

But, *other* tasks are also doing so, concurrently.

As such, EVERY time a resource is released, the scheduler (theoretically) runs. So, a task that causes a resource to be made available for another blocking task can be immediately preempted by that blocked task now being "ready" to run.

The distinction is important because tasks can reside in different cores as well as on different nodes. So, even if a task spins in a tight loop, not altering the availability of any resources, it can still be preempted by the actions of some other executing task.

Treating the design of the OS in a similar fashion, I can transfer "ownership" of specific objects to whichever tasks (servers) I consider appropriate. Dynamically.

E.g., when physical memory is free'd, I give it to a task that scrubs it (so the next user of said memory never sees any "data" that may have occupied those memory locations by a previous "user") and verifies its functionality. When some task NEEDS additional memory, it blocks waiting on the availability of such memory -- which causes this "scrubber" to make available pages that it deems as "clean and functional".

Chopping responsibilities up like this makes it easier to "get it right" -- at the expense of some performance (each of these interactions have to cross protection domains so the interactions aren't as lightweight as a simple function call in a monolithic kernel).

Processors are cheap. Memory is cheap. Developer time and latent bugs are costly (figure you have to spend a man-week looking into a suspected bug. If you're making 10,000 units, EACH such distraction can justify an additional $1 in hardware costs, without factoring in the externalities of cost to users, reputation, etc.)

The network is not a named resource. It is used by the OS to exchange messages with other kernel instances running on other nodes. So, when a task does:

object=>method(arguments)

it doesn't need to know if the referenced object is local or remote. The kernels handle location independence.

Traffic is encrypted with different keys for each node. So, discovering a key (e.g., by attacking a specific node) only gives you access to the traffic for that node.

This also allows for:

object=>move(new_server)

to force the object to be managed by another server (for that particular class of object) which will likely cause the object instance to "physically" move to whichever node on which that server is executing.

So, I can move every object off of a particular node -- or, onto a specific node!

In this way, I can bring hardware and software on/off-line on demand to adapt to changes in needs and available resources. E.g, if I'm running on battery (backup) power, I can shut down individual nodes to reduce power consumption after migrating their current responsibilities to other nodes or outright killing them off -- after checkpointing their progress. If I have some new

Objects are capability-based. So, you can only access objects for which you currently *have* a capability and only to the extent permitted by said capability.

Capabilities are un-named and managed in the kernel(s) so can't be counterfeited. You can *know* that a particular object exists (e.g., TheFrontDoor, TheGunSafe, TheBankAccount) but can't do anything to/with it because you likely haven't been given access to it and can't GET access to it as there isn't a central name registry that you could hack!

It seems fairly obvious that devices can no longer act as islands in the

And, rather than the heavyweight client-server interface where things interact in a generic, high-level manner (CORBA-ish), it seems much more practical to let them interact in a manner that is more natural to their designs. Do you want to have to standardize on every such interaction with an industry-wide "committee" arguing about how many humps the horse should have on its back? Or, do you want to make product that solves problems while your competitors are trying to define a level playing field??

I myself mistakenly professed: |---------------------------------------------------------------------| |"That hardware never fails. In particular it was a new computer which| |was switched on for its 1st time in Summer 2008. Its ZFS installation| |failed before 2009." | |---------------------------------------------------------------------|

Uh oh. I am making a confession that I am remembering today that actually that hardware did suffer only 1 failure - but only a keyboard which we replaced in circa 2012. Sorry! (S.