KiCAD Feedback Requested

There is not much to economize. Memory is cheap.

In reality you don't want to upgrade a library without testing whether the software still works. Rebuilding the executable is an insignificant amount of work compared to regression testing.

That's exactly how software bloat comes about :-)

Yeah, I just bought 10 GB of ECC server RAM for $21.39.

Exactly. Memory is cheap -- until you really *need* it for something and don't *have* it.

Do I want 20 copies of a static linked libc (portions thereof) sitting in memory all the time? One instance of printf() in this program, another instance of printf() in this OTHER program, etc. With a static linked executable, the only remedy the user has is to kill programs that are using resources that some *other* program could better use.

What about even *bigger* libraries (like the bloat for GUI's)? Do you want each program designed to run in that GUI environment to drag in its own copy of the libraries that it needs? Have you watched to see just how many dependencies there are in these libraries? Here's just the *package* dependencies for qt:

qt3 (C++ X GUI ToolKit) mng (Multiple-image Network Graphics reference library) lcms (Little Color Management System -- a color management library) tiff (Library and tools for reading and writing TIFF data files) jpeg (we'll see, below, what that needs) xz (XZ compression utilities) jpeg jpeg png (Library for manipulating PNG images) jpeg (IJGs jpeg compression utilities)

Note that each package can also draw on the "standard" libraries so you're talking about a *lot* of code!

Do you really want each qt-based application to carry "all" of this in *with* it? Even if it only uses a "small part", the chances are that it uses many of the *SAME* small parts as the other qt-based applications that you have running! -- the same code to implement menus, buttons, lists, checkboxes, etc.

OTOH, with dynamic links, you can exploit sharing. *And* get the benefits of "upgradability" *and* "upgrade prevention" just by manipulating file names, LD_LIBRARY_PATH, etc.

Want to prevent kicad from using a NEW version of libc? Build KiCAD with libc references renamed to libc_KiCAD and put a link between libc_KiCAD and the version of libc that you want to *wire* to KiCAD. If a new libc library is released, save the old one as libc_version_X (so the new one doesn't overwrite it) and update the libc_KiCAD link to point to this, instead.

The other alternative is to wrap kicad and set LD_LIBRARY_PATH to prepend the location of kicad's libraries (using the same original generic names!) so it finds them *before* it finds the "more recent" versions in the normal hiding places.

I.e., you can make dynamically linked executables behave as if statically linked (aside from the extra startup/run costs). OTOH, you can't make statically linked executables behave like dynamically linked, period.

Want to see if the new version of libc fixes a problem you are having in some piece of code? You'll have to rebuild the executable before you can even test it (if statically linked).

Do the same in a shared library environment? Just move the "old" library out of the way (rename it) and see how things work with the new library taking its place!

If you get sloppy with resources, you later discover you've shot yourself in the foot: "Gee, why can't I run my CAD program on my Linux-enabled cell phone? Why does it crash every time I try to create another sheet?"

Then, you find yourself having to develop another version of your application for that *different* memory environment.

(Turn off your swap and see how much you can do before you get a panic -- then you'll have a better idea of how "cheap" memory is! Buy *more*!! :> )

It is often informative to drag out older versions of software and run them on modern hardware. *Then* you see just how bloated and sluggish things have really become! Fire up the "new and improved" version and see how much slower it runs. Or, how much more resources it demands. And remind yourself why, exactly, you made this upgrade??

I *just* updated my Windows machines to XP. I've never been accused of "keeping up with fashion" :>

The UN*X philosophy builds on smaller "programs" to provide greater functionality. So, some tool that you use might be employed in some *other* "meta tool". An exploit of that meta tool could exploit a defect in that tool that you *thought* was only used "by you".

The same is more true of libraries. Look at how often DLL's get updated on Windows machines...

The point is that dynamic/shared libraries can be made to behave

At run-time, there is a small overhead to each indirect reference.

In return, you can reduce the amount of disk space required to store the executables+libraries as well as memory footprint. And, you can "plug in" a new library -- even if only to see how it alters the performance of your application prior to deciding whether or not to adopt it.

Windows' problem with DLL's is they don't allow them to peacefully coexist at different version levels. And, their library search rules make it hard for you to "game the system" to get the behavior you want.

You can easily test to see where things fall apart! Turn off your swap partition. If you can use the machine in EVERY situation that you would normally, then you've got plenty of RAM for your needs. OTOH, if some application eventually needs to swap out a page for some reason and finds "no place to put it"... :>

My data needs tend to be *big*. I.e., I set the per process memory resource hard and soft limits at ~2G. Even then, I often have to watch the sizes of running processes to see if they will bump into this limit before finishing their work (though it feels sort of like watching a kettle boil... doesn't do anything to *alter*/improve the process! :< )

I haven't repeated the initial exercise for several release cycles. When I originally did it, the results were very obvious (savings). So much so, that I spent ~6 months trying to shoehorn a similar feature into the OS I was developing at the time (since embedded devices have far less resource flexibility than laptops/desktops/servers).

A computer that won't run the software that I use (which I don't always have the source for) is not a very useful device.

Cheers

Phil Hobbs

How does any of that apply? All I've said is that you can get the "immutable" aspects of statically linked libraries

If you are using *purchased* software, then you are at the mercy of your vendor! But, you can still bend the dynamic executable to fit your constraints (as above) whereas the statically linked executable leaves you no choice.

I just dragged out my old Z80 based CP/M system to get it ready for a museum donation. This is the system I built in the late 1970's and early

Turbo Pascal uses 30K for the entire IDE - full screen (character mode) editor, compiler, in memory compile/execute. The error message text is an option to save a little memory. When compiling or executing the program in memory it will bring up the editor to the failing line on error. Oh and you can do paging - it was called overlays and you had to figure out the dependencies and ordering yourself and set it up. (I saw this again a couple of years ago for programming SPEs on the Cell Processor - the new kids were clueless.)

I now takes multiple mega(giga?)bytes to do the same thing. Of course now the display is graphical, in color, and you can use a mouse instead of the arrow keys. Interestingly the modern system is only a few times faster - not the orders of magnitude difference in speed between an 8 bit 4 MHz and a 32/64 bit 2-4 GHz processor (not to mention floppies and

Like what? I run huge compilations (10 parallel builds) and I never see those use more than 1GB on my Linux machine. My XP machine has 2GB of memory and swap disabled. No problems at all.

Thats something you really don't want to do. Do you have any idea how much work is involved in creating a Linux distribution for just finding out which libraries are compatible with certain versions of software?

Now imagine the mess you step into if you want to distribute a piece of proprietary software which should run on various Linux versions. An elegant technical solution is not always a good solution from a user's perspective. 'Flexibility' often translates into 'many things can go wrong'.

It's common for libraries to be incompatible with each other, though, and anyway that's a whole lot of work just to keep a stupid executable executing. Static linking for my stuff.

Cheers

Phil Hobbs

It seems as though the OS could do some low-priority background processing, computing CRCs or similar of code data on a page-by-page basis, storing them in a table somewhere, noticing when the same CRCs show up, and then verifying in such cases that the pages are really identical, at which point one copy can be purged and just have its virtual memory re-mapped to point to the other copy? The computed CRCs could also be cached so that this could all occur at load/page fault time instead for future runs.

I realize that this requires libraries to load at the beginning of a page, or at least there to some some mechanism that denotes the beginning and end of a given subroutine, but that requirement doesn't seem too onerous. In fact, I'm thinking this information might already be in the executable, since at load time you need to go in and perform absolute address fix-ups?

Try running a database service.

I wasn't advocating building an entire distribution! I was pointing out how you can ensure that a *particular* dynamic linked application that relies on a *particular* version of a dynamic library (i.e., it breaks when you symlink libWHATEVER to libWHATEVER.versionY instead of libWHATEVER.versionX -- to which it had previously been linked) can be made to continue to operate when THE REST OF THE DISTRIBUTION has been upgraded to expect libWHATEVER to now be at .versionY

For example, my /usr/lib/libc.so (i.e., the "nominal" libc shared library that "everything" uses) is symlinked to ./libc.so.12 (i.e., version *12* of the library). libc.so.12 is, in turn, symlinked to ./libc.so.12.128.2 !! Hard links could have been used but symlinks makes it *really* obvious how each name resolves! :>

So, an application that relies on the "system/nominal" libc actually gets libc.so.12.128.2 when it is loaded. If an application is found NOT to work with version 12 but *did* work with version 11 (before some "system update"), /usr/lib/libc.so.11 is still available as

I.e., the update doesn't overwrite the existing "libc" -- it just installs a new libc WITH A UNIQUE NAME and arranges for the nominal "libc" to *point* to that new libc. Windows takes the heavy-handed approach of *clobbering* old versions of DLL's pretty regularly. That;s fine if you can be guaranteed that every *existing* executable will continue to work with the new DLL. That doesn't seem to be the case, in practice!

The (BSD) executable that worked when libc was symlinked to libc.so.11 can still be made operable even though libc is now symlinked to libc.so.12 by creating a link UNIQUE FOR THAT EXECUTABLE so that

If this is "hard" for Linux to do, then I suggest that's a problem with Linux as it is a normal way of doing things for the *BSD's...

The above works perfectly! If your proprietary software is designed to use the published specification of libc, then it will always want to use "libc.so". Any changes to libc.so (i.e., as it is upgraded to point to libc.so.12 instead of libc.so.11) will SILENTLY AND WITHOUT ANY USER INTERVENTION ensure that you have the "most correct" version of the library. WITHOUT you having to release a new version of *your* proprietary software!

E.g., perhaps its a new graphics widget library that allows some external tool to be used to modify widgets within *your* layouts. Providing a new functionality to your software WITHOUT you having to write any new code! If you had statically linked your executable to the previous library, you would find that every other application built on this library had this ability... EXCEPT yours!

If, however, your software exploited something "undocumented" in the library... or, implemented a workaround for some *problem* in the library (i.e., before the library had been formally upgraded) *and* that workaround proved to be incompatible with how the library was eventually "fixed" (possibly because you failed to design your workaround to deal with this possibility!), then the library's upgrade could break *your* application. Because *you* violated the contract that was published for the library (just because the library is "broken" doesn't give you the liberty to redefine how it *should* work! If you want to do that, create your own library :> ).

So, your executable would either bind (dynamically) to the libc.so.11 version of the library EXPLICITLY (or, libMINE.so if you prefer). Or, users would make that adjustment using the mechanisms I outlined.

The only time static libraries make sense (desktop) is when the *dynamic* libraries might not be available/accessible.

E.g., /usr/lib in many installations is a separate file system. So, if the disk "has issues" and that file system is trashed, you can't rely on having access to any of the libraries stored on it. Indeed, the filesystem may not even be *mountable*!

So, the tools that you would typically use to repair it need to reside somewhere that is less likely to be trashed (e.g., some portion of the root partition -- if that gets trashed, you're screwed, regardless!) *and* have the required libraries embedded in their executable images (static linked).

Ever wonder why there are /bin, /sbin, /stand, /usr/bin, /usr/sbin, etc. instead of just one (or two?) places for everything? E.g., like \WINDOWS and \Program Files? :>

The frame is hardcoded in every CAD package I've encountered. Some allow to adjust the markings and size but thats about it. I don't understand the big fuzz about a border around a diagram. Just specify its supposed to be the way it is when asked.

I actually still have several boxes of (1.2M?) 8" floppies (though I've since lost all the hard-sectored ones :< ) I have an old "Megaboard" -- 512MB of RAM, 3? DARTs, etc. and one surviving 8" floppy drive (with a "door lock" so I could cache floppy accesses in that huge RAM and write them back at my leisure without fear of the user opening the door and ejecting the floppy).

Exactly. This is what the "memory is cheap" (MIPS's are cheap) argument gets you. The advantage all falls to the software developer instead of the *user*. His job gets easier, the user sees little

DASH (FutureNet) was written in ASM many years ago (early 80's?). Ran very well on machines of that era. Sometime later, it seems to have been rewritten in (?). On identical hardware, it was ~half as responsive. I.e., you *had* to upgrade the hardware just to keep the performance constant. But, there were no noticeable improvements to the product.

Look at how that "extra" memory and speed gets used nowadays... glitzier user interfaces with animated icons, etc. But, how much more *productive* is the user? (granted, many things are "gray-matter limited"... but many things are *not*!)

You could boot a CP/M machine OFF A FLOPPY in less time than a 3GHz PC boots off a 15K RPM hard disk! (apples oranges)

I was heartbroken when I had to give up my Brief editor -- because it was just too *fast* on modern machines! Hit the down arrow key and you're at the end of file (thousands of lines later) before you can release it (and you *saw* each of those lines scroll by in the process!).

It's your memory/MIPS... if you want to watch pieces of paper crumple as they fall into a trash can... so be it! :>

Mnm... at least in ORCAD Capture and Pulsonix there's nothing too particularly special about title frames; you can absolutely go off and draw new ones if you feel like it, adding or removing fields, adding your company logo, changing dimensions and title block size, etc. It's also straightforward to replace the current frame with a new one.

I kinda agree, but I haven't worked in the environments that, e.g., Joerg has where for various regulatory reasons you have to be very strict about document control.

(Where I work these days... we do have proper document control and all, and what the engineers learn quickly is that you leave your project status at the "prototype" status until the thing is 99.99% working, since the minute you change it to "released" even trivial changes like component values then require a sign-off from multiple people... i.e., things start moving a lot more slowly!)

Well, if you want to compare *relatively new* users of, say, WordPerfect

The market, though, seems to be dominated by people who aren't interested in being experienced/expert users, so from that point of view overall productivity probably has increased a lot on average. It's actually pretty nice to know that just about anyone can now use, e.g., a travel-planning application, a banking application, a word processor, etc. with absolutely zero formal training these days.

There are a few productivity improvements that all that extra memory and speeds get you completely ignoring the glitz, though: Near-instantaneous searching of all the files on your PC (ala Google Desktop) is pretty nice, but requires lots of disk space and processing to make it work well. Voice input on smartphones is becoming surprisingly usable, and that's no lightweight CPU task either. Even just playing back, say,

---Joel

The OS could compute the CRC as the page is loaded -- assuming the page is immutable.

But, it's an expensive solution to what can otherwise be simply implemented (by manipulating the namespace of files). I.e., the CRC means every byte has to be examined (instead of just copying them at bus speeds). Then, you have to keep track of which CRC's pertain to which pages (in a table... how big should the table be before it becomes one of diminishing returns?). And, the CRC doesn't unambiguously indicate matches -- so, you have to be able to remember "page X, CRC Y does not match page Z, CRC Y" (so encountering ANOTHER page with a CRC Y causes you to check it against *both* page X and page Z)

The OS can't imbue semantic value to the CRC. OK, these pages appear to be shareable (and immutable). So, it can map both of the *logical* pages to a single *physical* page but, this solution requires virtual memory (this is how I do it in my OS but it is done explicitly instead of by trying to "examine" pages for possible matches).

But, it (OS) can't know how that page is being used -- will it persist for some amount of time? Or, disappear? What if it is not immutable? etc.

I implement shared memory using a hack on this called "Copy on Write" (CoW). Basically, if you and I want to share some memory (like a copy of some code or some big piece of data), the OS marks the memory as read-only, maps it into *both* logical address spaces (of course, it usually already exists in *one* of those address spaces! And, two can just as easily become three, etc.) -- possibly at *different* logical addresses. Then, lets the "users" access it.

If any one of them WRITES to the single, *shared* physical memory, a trap occurs. The OS sees that this "user" has a different idea of what the memory should look like, *now*. So, it creates a *copy* of the shared page just for that "user", marks the page as "writeable" (since the user is the ONLY one with access to this unique instance of the page) and lets the "write" happen.

Meanwhile, the users still sharing the UNMODIFIED page continue on their merry way (until one of *them* decides to modify their "virtual copy" of the page -- requiring a similar *physical* copy operation).

While it seems like a lot of "mechanism", it can give you a lot of bang for the buck!

If, for example, you are sharing the CODE in a library, then the same mechanism applies -- except that no one EVER modifies any instance of the page... because it truly *is* "READ ONLY"/immutable. A write triggers a fatal fault for the "user" attempting it!

And, starting a child process via "fork()" makes it relatively easy to create a whole new "virtual copy" of the memory image -- while *defering* the issues of allowing the child process to alter it's copy of the image.

It's also a win as it lets big pieces of data move quickly between "users" (of course, "user" is really "task"/"process") without lots of physical copying (think of processing video, for example).

I.e., this is an example of something I mentioned elsewhere, recently, wrt treating things that are conceptually similar with identical code -- instead of handling each with a different mechanism. Build a sharing mechanism and then see how it can solve

Tell the OS *up front* that you are going to share something (a library) and let it implement the mechanisms to do that sharing for you.

Yes, one of my customers has started to work just like that. Same goes for "projects". Spend 2 years planning for them - doing what any normal person would call "project work". R&D, 3D models, costings, tests. So that when "officially" started they go smoothly and to schedule.

ctly

rary

on

Too bad. The 'cure' of introducing artificial dependencies in the form of shared libraries is worse than just about any disease.

The widespread use of shared libraries has kneecapped Linux in the desktop marketplace, and in the days of "DLL hell," it almost took down Windows as well. (It might well have done so, in fact, if Linux hadn't been even more of a mess.)

-- john