Regarding calculation of free memory

Ahhh, now I know why I've never had any problems replacing malloc on Linux, Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)

--
David Gay
dgay@acm.org
Reply to
David Gay
Loading thread data ...

In article , David Gay writes: |> |> > |> > No. Replacing malloc is not supported in C, and often breaks |> > |> > the nastier vendor and third-party libraries. The reason is that |> > |> > the suite of malloc-related functions is not independent, and they |> > |> > often use non-standard interfaces. |> > |> |> > |> There are still systems where you can reasonably do so - shipping |> > |> more than one possible malloc to use is a good hint for example. |> > |> > It's a hint that the VENDOR can do it - but not that the application |> > programmer can. Yes, an experienced hacker can, which was the basis |> > of my remark that I can, but I can't advise users to. |> |> Ahhh, now I know why I've never had any problems replacing malloc on Linux, |> Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?)

Or perhaps you just have a slightly narrower range of experience than I do?

I have seen failures on half a dozen Unices, including Solaris and Linux. Of course, as I said, it also depends on what other software you are using in your application.

Regards, Nick Maclaren.

Reply to
Nick Maclaren

Well, frankly, for the Linux case, you can go and look at the source code to check if there's any extra magic entry points or other dependencies. For the others, it definitely pays to check linker output files to make sure you didn't, e.g., accidentally end up with two copies of malloc. Yes, you'll have to repeat such checks if you change libc version - that's a price you have to pay. And if you're going to say "but I've run into problems", I'd like to get a specific description, rather than some nebulous "It happened to me somewhere, sometime, somehow".

--
David Gay
dgay@acm.org
Reply to
David Gay

As I said, this is (usually) within the ability of an experienced hacker. Do you SERIOUSLY think that it is reasonable for me to tell a Fortran programmer to find out what version of malloc a system is running (when he did not install it) and to inspect both the source code of that malloc and its replacement?

As far as Solaris etc. goes, you first have to find out which version (or versions) of malloc the application is using, INCLUDING those called by dynamic libraries. That is unusually hard under Solaris, because ldd doesn't do what you apparently thinks that it does, and a fair amount of reverse engineering of the behaviour of the linker and loader is needed. Then you have to extract the relevant symbols, see which library is using what, and decide if there is a clash.

Oh, and for non-standard symbols, you may have to reverse engineer the binary to see what they do.

Regards, Nick Maclaren.

Reply to
Nick Maclaren

How strange. I use it. In C.

How strange. I use my own malloc(). In C.

Why not? Gives you complete control. You can tune your malloc() to your application.

--
  ... Hank

http://home.earthlink.net/~horedson
 Click to see the full signature
Reply to
Hank Oredson

My original comments re sbrk() and malloc() were not about Unix. And even with various Unices, they still apply.

If your application can be smart about it's use of memory, that can be a huge win. On any OS. Including Unix flavored ones.

--
  ... Hank

http://home.earthlink.net/~horedson
 Click to see the full signature
Reply to
Hank Oredson

Yes. Any programmer should know her tools well. That includes the details of how any libriaries work, and knowledge of other libraries available to provide similar support functions. NTRAN comes to mind ...

Nonesense.

--
  ... Hank

http://home.earthlink.net/~horedson
 Click to see the full signature
Reply to
Hank Oredson

That is only if people insist on trying to re-use free'd memory in unsupported ways or otherwise abuse a quite simple interface.

The C malloc() interface is quite simple: You get memory if the overlying system lords decide to grant it, and you can do whatever you want with it within reason (so long as you don't make assumptions about its underlying byte/word length/order, etc.) but you can't make any assumptions whatsoever about being able to get it back if your free() it or about having it be visible in any predictable way from anyplace outside the code segment that allocated it. By the way, the system lords may lie to you, just as your bank is likely to do if you try to withdraw a large amount of cash all at once, because they already loaned it out to someone else.

Replacing malloc() is quite supported in C, just as is implementing your own version of sqrt(). That's obviously the point of any reasonable programming language. It's a way to convince a stupid CPU to do work you want done.

You just can't make any underlying assumptions about the implementation of the vendor function or your ability to interoperate with it.

The bottom line is that the computer does what you tell it to do, not necessarily what you want. Shouting at the deaf is seldom effective.

This endless bickering about malloc() never ceases to amaze me. Programmers simply won't accept that you can't really shove fifty pounds of shit into a twenty pound sack. They also can't accept that someone else may have stolen the booty before they got there, or even in between the time that they measured the bag and started shoveling.

It's Chinatown, Jake.

Reply to
Colonel Forbin

No self respecting Fortran programmer is going to believe seductive lies from an operating system or a program library.

Reply to
Colonel Forbin

Nick's point is that it is not a part of the C standard. My Linux manpage for sbrk() says to #include and under the "CONFORMING TO" heading says this:

brk and sbrk are not defined in the C Standard and are deliberately excluded from the POSIX.1 standard (see paragraphs B.1.1.1.3 and B.8.3.3).

Of course you *can* call it from C, but you can't expect it to be there on all systems, especially "bare metal" embedded systems.

--
Michael N. Moran           (h) 770 516 7918
5009 Old Field Ct.         (c) 678 521 5460
 Click to see the full signature
Reply to
Michael N. Moran

So you can in fact in the Solaris case too these days, though that hasn't really been needed so far AFAICT.

--
	Sander

+++ Out of cheese error +++
Reply to
Sander Vesik

Ahem. Sorry, Nick, but at the very least in case of Solaris, it is very easy to find out what exactly it is that the dynamic linker does and comapre the results of different runs. This is certainly not advanced hacker level.

--
	Sander

+++ Out of cheese error +++
Reply to
Sander Vesik

Who cares? When programming, one uses the tools available. Nitpicking "standard" vs. "POSIX" vs. etc. is just a waste of time. If we need better standards, write them. But in any case I was not thinking of C at all, and particularly not of Unix / Linux in my original comments. I use the "C hacker" way of talking about things because there are not a lot of folks who will understand when I say "Just use ER MCOR$, it will work the same in FOR and FTN." The point is that the application can manage memory better than the OS can, and there is always some way for the application to get hold of that memory to manage.

So what?

The programmer must know her tools. If she is writing a banking backroom system, she will not be interested in porting it to a Blackberry, or an access point.

--
  ... Hank

http://home.earthlink.net/~horedson
 Click to see the full signature
Reply to
Hank Oredson

Oh, really? It is (relatively) easy to find out what is loaded, though even that needs fairly advanced knowledge. But finding out which library and which symbol in that library is used to satisfy another symbol in another library is not so easy. I had a problem recently that came down to this. Nasty.

Solaris is not as extreme as Tru64, but you can still get multiple different objects of the same name in the same executable (or, at least, could until recently), and finding out whether you have and which one is used for what is definitely hard. In particular, there is no PRECISE specification of any Unix linker that I know of, let alone any on the loader :-(

I still have a program somewhere that I wrote to investigate this area, and I was flabberghasted by what I found. I was expecting oddities, but not on the scale that I found them.

Regards, Nick Maclaren.

Reply to
Nick Maclaren

Yes. And you can't expect it to work, and be used, the same way, either, which is a more serious problem. But you and I are talking about programs where people care about portability and RAS, both between systems and over time - and the happy hackers don't even understand the concept. Some of them will learn, as new versions of even their favourite system causes their code to break; others will not learn from experience.

Regards, Nick Maclaren.

Reply to
Nick Maclaren

You can certainly do so in Solaris (where we do ship far too many: libc malloc, BSD malloc, SysV -lmalloc, mmap -lmapmalloc, -lmtmalloc, watchmalloc (for debugging) and libumem).

And you can substitute your own as long as you make sure you implement *all* routines our malloc libraries commonly export (mallocs often define some of their functions in terms of the others; but some implement foo() in terms of bar() and others vice versa and this tend to lead to nasty problems if you don't roll a complete implementation)

Casper

--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
 Click to see the full signature
Reply to
Casper H.S. Dik

The Solaris runtime linker allows you to do a complete trace of all symbol references and which particular symbol definition they are bound to.

Prior to Solaris 10 where some of the system libraries existed in static form, the problem was intractable as malloc implementations may have found their way into the exectubale in static form; mixing malloc implementations in perilous, at best.

Casper

--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
 Click to see the full signature
Reply to
Casper H.S. Dik

In article , Casper H.S. Dik writes: |> snipped-for-privacy@cus.cam.ac.uk (Nick Maclaren) writes: |> |> >I still have a program somewhere that I wrote to investigate this |> >area, and I was flabberghasted by what I found. I was expecting |> >oddities, but not on the scale that I found them. |> |> The Solaris runtime linker allows you to do a complete trace of all |> symbol references and which particular symbol definition they are |> bound to.

That wasn't the issue I was referring to. It was worst on Tru64, but I can't remember if it was serious on Solaris or it still exists. It was the situation where reference A (= fred) could not be matched by object B (= fred) unless object C (= fred) was included first (perhaps by a reference to object D (= joe)). Seriously.

The most likely cause of this was incompatible references, which could be 'mated' if there were a Fortran COMMON block of the same name.

|> Prior to Solaris 10 where some of the system libraries existed in |> static form, the problem was intractable as malloc implementations |> may have found their way into the exectubale in static form; mixing |> malloc implementations in perilous, at best.

Exactly.

Regards, Nick Maclaren.

Reply to
Nick Maclaren

In article , Casper H.S. Dik writes: |> |> >Ahhh, now I know why I've never had any problems replacing malloc on Linux, |> >Solaris or the Amiga's OS. I'm a VENDOR! (or maybe not?) |> |> You can certainly do so in Solaris (where we do ship far too many: |> libc malloc, BSD malloc, SysV -lmalloc, mmap -lmapmalloc, -lmtmalloc, |> watchmalloc (for debugging) and libumem). |> |> And you can substitute your own as long as you make sure you |> implement *all* routines our malloc libraries commonly export |> (mallocs often define some of their functions in terms of the others; |> but some implement foo() in terms of bar() and others vice versa and |> this tend to lead to nasty problems if you don't roll a complete implementation)

cd /usr/lib for i in *alloc*.so do echo +++ $i +++ nm $i | grep GLOB | grep -v UNDEF | sed 's/^.*|//' | grep -v '^_' | \ sort -u | xargs echo done

+++ libbsdmalloc.so +++ SUNW_1.1 free malloc realloc +++ libmalloc.so +++ SUNW_1.1 calloc free malloc realloc +++ libmapmalloc.so +++ SUNW_0.7 SUNW_1.1 SUNWprivate_1.1 calloc cfree free mallinfo malloc mallopt memalign realloc valloc +++ libmtmalloc.so +++ SUNW_1.1 SUNW_1.1.1 SUNWprivate_1.1 calloc free malloc mallocctl memalign realloc valloc

I can't remember which system it was that had an undocumented call that I had to reverse engineer from the pseudo-assembler.

Regards, Nick Maclaren.

Reply to
Nick Maclaren

Well ... there *is* static allocation. Then subystems can their own private memory allocations.

And, indeed, in systems that must operate indefinitely, one must not use libraries with such dependencies because any global general purpose heap (malloc/free) will fragment unless all blocks are the same size.

Frequently, the same systems that have such requirements also have limited memory resources and so using a global malloc with as single fixed size block is unacceptable because of the wasted memory in small allocations.

As a side note, its not the malloc that fragments heaps, its the continuous malloc/free cycle over time. For this reason, systems that must operate continuously for an indefinite period of time

*can* use malloc/new at initialization ... as long as they don't use free/delete ...
--
Michael N. Moran           (h) 770 516 7918
5009 Old Field Ct.         (c) 678 521 5460
 Click to see the full signature
Reply to
Michael N. Moran

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.