Static allocation failure?

Hi - First time posting. I have a quick question that hopefully is general enough that someone can help me out.

Lets say I have an array that I declare statically in a function, and I want it to be around 2MB of space, but if I make it that large, the OS I am using never comes up correctly on the board. However, if I make it smaller (say, 1.5 MB), I can load it on the board, and the whole image comes up just fine.

Ideas? I am certainly not out of memory... the board has around 128MB and I'm not even close. I am using Integrity for my RTOS.

Thanks! M

Reply to
mikewmedved
Loading thread data ...

What happens if you declare it globally?

Reply to
larwe

No, but you are probably out of memory allocated to stack space! Actually I'm suprised 1.5 MB works.

A quick google search found this informative comment:

"However, we tend to discourage users from declaring their arrays statically in this way, for two reasons. First, this allocates the array on the stack, which has a very limited size on most operating systems (declaring an array with more than a few thousand elements will often cause a crash)." (from

formatting link

I don't think the answer is increasing your stack space, but putting the array someplace else.

Reply to
cs_posting

I looked at your site and think I understand why you thought stack space would help... they say: "Multi-dimensional arrays declared statically (that is, at compile time, not necessarily with the static keyword..." That would be like an array declared in a function that is scoped in the function... which of course gets put on the stack.

I am talking about an array that is declared with the static keyword, like:

void foo() { static int myArray[512000]; // some big number }

This would certainly not go on the stack.

In fact, when I increased the heap size, it turns out that it works... which is totally confusing. The heap is supposed to be only for dynamic allocation...

Reply to
mikewmedved

Interesting that you should use the word 'supposed'. One meaning is "required by or as if by authority", while, interestingly enough, another is "held as an opinion : BELIEVED; also : mistakenly believed" (from Meriam-Webster online;

formatting link

While the first is, I believe, true, I wouldn't put it past any compiler writer to put your big array on the heap sometime before main() is run and keep it there. Yes, it _should_ be statically allocated in RAM -- but _is_ it statically allocated in RAM? For that matter, does ANSI-C say it has to be statically allocated, or just that it doesn't change for the lifetime of the program -- which on a desktop would certainly allow allocation off of the heap?

It may be that if you declare it as static within the file, i.e.

static arrayForFoo[512000]; void foo() { ... }

it may go into static RAM -- or it may be that your compiler writer just puts _every_ array onto the heap, for whatever reason.

Good luck.

--
Tim Wescott
Wescott Design Services
 Click to see the full signature
Reply to
Tim Wescott

Even wierder... it turns out that giving the address space more heap allowed everything to come up on the board succesfully, however, the initial task for the address space is completely blown away.

Even after moving it out of the function as you described above, the same behavior continues.

Wierd, wierd, wierd.

M
Reply to
mikewmedved

All a very good lead in to...

So your next step should be to look at you linker map file and compare that against what your intended memory map is. You might be surprised to find that your linker is not locating things where you wanted or expected.

Be careful, some linkers don't name static data objects declared within the scope of a function in the map file. You might have to scope it global to overcome that. It might even change the linker location of the object! It'll take some minor sleuthing.

JJS

Reply to
johnspeth

Come to think of it, a globally allocated array couldn't be instantiated on the heap without some really weird stuff going on -- the linker has to know where it is, which pretty much forces it to be statically allocated. A perverse tool may still put

static int bar[42];

on the heap, but it'd have to be _really_ perverse.

I notice that you're making a really huge array -- I've had troubles before with some tool chains (Borland 3.x or maybe 2.x) that couldn't deal with arrays bigger than 65535 bytes* -- could your tool chain be lacking imagination in the big-array department?

  • That was because of a segment addressing difficulty on the old 80x86, but still.
--
Tim Wescott
Wescott Design Services
 Click to see the full signature
Reply to
Tim Wescott

I guess I'll show my lack of knowledge... I know the array is located in the .bss section from the linker map file, but I don't necessarily know what this means. Does that mean the actual memory will come from this area, or is just the symbol there, or what?

Reply to
mikewmedved

Here is another interesting point... in this address space a static array already exists of the same size, so it definitly isnt a 65K limitation. This is using GHS' Multi, btw (definitly not a desktop app).

If I reduce the size of the other array, and leave mine around 2MB, it seems to work. However, if I have both at 2MB it blows up. Both show up in the .bss section regardless of whether they are globally static or declared statically in the function itself.

If you've every used Multi, it does have a zillion settings for the toolchain, although nothing which immediately jumps out as a solution to my problem.

Reply to
mikewmedved

You should check with the RTOS manufacturers. Have you tried not assigning the memory until initialization time, and then doing it via malloc? You can let the function do the assignment at first call, and do nothing else, by something like:

whatever yourfunction(yourparameters) {

static yourtype *yourarray; /* will normally initialize */ /* to NULL, but this needs */ /* to be checked on your */ /* system, not guaranteed */ if (!yourarray) { yourarray = malloc(SIZENEEDED); if (!yourarray) /* aborteverything */; } else { /* your normal function coding but using */ /* *yourarray where you used to use yourarray */ } } /* your function */

That way you still restrict the scope of the array to the function.

If the auto initialization to all bits zero of yourarray does not produce a NULL on your system you can add a static integer first time flag, which will be initialized to zero.

--
"If you want to post a followup via groups.google.com, don't use
 the broken "Reply" link at the bottom of the article.  Click on 
 Click to see the full signature
Reply to
CBFalconer

There's no law requiring that BSS and heap must be separate regions. The BSS can be, and I suspect rather regularly is, nothing else but the bottom of the heap. Don't mix up the memory region called "heap" with the malloc() function that manages handing you pieces of it. Just because a static array resides in the heap doesn't mean the runtime has to call malloc() on the first entry to that function to set it up.

Some context may be necessary to understand this:

The collection of all objects of static storage duration with (default) zero initializers in a C program form what is usually referred to as the .bss or BSS segment. It's entirely up to the startup code and the operating system (if any) to figure out how to actually set up this memory region. Under the assumption that the architecture in question has all-bits zero for all zeroes of atomic types, there's really nothing to prevent the runtime startup code from doing an equivalent of

BSS_start = _internal_malloc(BSS_size); if (! BSS_start) panic(); memset(BSS_start, 0, BSS_size);

to set up the BSS. On a Unix system, I would expect exactly this to happen. The _internal_malloc() there is the system call sbrk().

I.e. it's about the most natural thing in the world for BSS to end up sitting right in the middle between the DATA segment and the heap, as far as addressing is concerned, while functionally a part of the heap.

--
Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de)
Even if all the snow were burnt, ashes would remain.
Reply to
Hans-Bernhard Broeker

Back on Jan 13th, I wrote something here that might have been useful for you. A compiler and linker toolset for standard programming usually uses a mental concept for a program that follows this template:

Sect Description Access NV? Size ================================================================= CODE Generated code Execute Yes Fixed/static CONST Constants Read Yes Fixed/static INIT Initialized data Read/Write Yes Fixed/static BSS Uninitialized data Read/Write No Fixed/static HEAP Heap Read/Write No Variable, up STACK Stack Read/Write No Variable, down

The "NV" above means "non-volatile." And by this, I mean that somehow the data for these things must be retained somewhere when the power of the device is off. On PCs, this could be the hard disk that the operating system uses to store the programs. But it needs to be somewhere. Those sections marked "no" do _not_ require any non-volatile storage and they can be allocated from memory without any particular values being required there to start.

The BSS term is an old one, I think dating back to IBM 360 BAL (basic assembly language) days. It means "reserved storage" without the need for initializing it to anything in particular. C, of course, requires initialization of most things to a semantic zero. But this can be done without any real intelligence to speak of and can usually be applied to BSS storage as a whole. This is a little different from straight assembly coding where these sections really do NOT get initialized, at all. But it's not much different and I think C compilers can legitimately call it BSS, even if their startup code does stick something there to begin with. It isn't coming from non-volatile storage. Which is the main point, even if the details aren't exactly matching in practice.

So. Just imagine that the compiler, when compiling, has three places to which it may denote an instance of a variable as belonging -- INIT, BSS, and STACK. When you define a variable instance as 'static' within a function or at module level, otherwise, the compiler will assign it either to INIT or BSS. If you specify a set of initializers to an array or a single initializer value to a scalar, then the compiler knows that it must be placed in the INIT section. If you don't define it with initializers, then the compiler is free to place it into the BSS section, where it will be semantically set to zero when the program starts and before main() is called. If you define the variable without static (or using auto) within a function, then the compiler will assign it to STACK and base it off of the current stack frame pointer for the function. The function will include a prologue and an epilogue section of code to manage this stack frame concept and to allocate any needed local variable space at that time. This space, though, is not guaranteed to be a semantic zero and will be garbage unless you include initializers. If you include the initializers, the compiler is forced to then generate inline code to initialize those values on the stack at run time.

Does that help?

Jon

Reply to
Jonathan Kirwan

It is a very good explanation, with the exception that it doesn't really help me fix my problem. With my two large arrays, I can see the .bss section increasing or decreasing when I play with their sizes, but that doesn't help me understand why if I make my array bigger I have to adjust the heap size. Ande even assuming the OS allocates things in the BSS with memory from the heap, if I set my heap size way larger than the BSS section I still get funky run time behavior.

Reply to
mikewmedved

For one of two reaons:

1) your runtime system sets up the BSS as the start of what, as far as the OS is concerned, is your app's heap. I.e. the app is *in* the heap.

2) You're running out of memory. If the BSS is not part of what you get to configure as "heap size", then BSS and heap size together may request more memory than your system actually has. Arbitrarily crazy results may ensue, until you reduce the heap size enough to fit both in again.

Which of these it is can be determined by which direction you have to change the heap in, after enlarging the BSS.

--
Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de)
Even if all the snow were burnt, ashes would remain.
Reply to
Hans-Bernhard Broeker

I don't know your compiler or situation, so I can only speak generally and hope that _you_ can make something of it.

One thing I can suggest that you consider is this.. let's return to the picture:

Sect Description Access NV? Size ================================================================= CODE Generated code Execute Yes Fixed/static CONST Constants Read Yes Fixed/static INIT Initialized data Read/Write Yes Fixed/static BSS Uninitialized data Read/Write No Fixed/static HEAP Heap Read/Write No Variable, up STACK Stack Read/Write No Variable, down

Now, let's say that a linker wants to _know_ what the absolute size of the total program is. How? Does it start with an a-priori idea, something it gets from some external specifier? Or does it simply add up the totals somehow and use that?

The compiler generates a series of fragments which the linker then accumulates into each section. Once that is done, the linker will know exactly the size of CODE, CONST, INIT, and BSS. The linker will have absolutely _no_ information about how big HEAP and STACK are, though. How big does the stack need to be? Well, that depends on how routines are called, which calls which, how nested that might get (remember recursion?), etc. How big does the HEAP need to be? Well, that depends, too.

So what's a linker to do? Where does it set the HEAP and STACK pointers, to start??

Well, the HEAP is easy. It just starts at the end of the CODE, CONST, INIT, and BSS sections pasted together as a whole. So the heap "grows up" towards the top of the stack. But we still have no idea how big it should be.

What of the STACK?? The STACK grows down. So where to place the start of the STACK area? This is arbitrary. Any choice made can lead to problems. One possibility is that somewhere, perhaps in a linker specification file somewhere, there is an estimate of the STACK size. This may be arbitrary. Or it can be something you easily set in your compiler tool and which is then passed to the linker.

So, let's say that the linker sets up the STACK at some fixed spot. Not at the end of all that 128MB of RAM, as you might tend to assume it does. But somewhere else. Somewhere much, much closer to the code itself. Then when you make your BSS region larger with a larger static variable, you push up the HEAP start. This means that a problem will occur sooner if the stack pushes down into the up-growing heap. There is less memory for both. Which exactly kills your system will depend. But if you don't have enough combined memory for both during run-time, it will definitely show up sometime.

I think you need to closely examine what is going on with your linker and where it places the starting stack pointer, etc.

I had hoped the information would help be an guide in looking. That's all. I can't solve your problem for you as I have no way of testing your situation here. That's your job.

Jon

Reply to
Jonathan Kirwan

Something's not quite right there - "INIT" is read/write and non-volatile? What you are missing is a "DATA" section, which is read/write and which is initialised as a copy of the "INIT" section by the C startup code (which also zeros out BSS) before calling main(). Depending on the sort of system you have, INIT and DATA may be separate, or they may be combined - for example, if you have a bootloader that copies the program from flash/disk to ram before executing, the sections will be mapped to the same address (the bootloader copying handles the initialisation). However, if your main program runs directly from flash, then INIT and DATA are at distinct addresses (one is in flash, the other is a copy in ram).

As for the history of BSS, apparently it stands for Block Started by Symbol:

formatting link

Reply to
David Brown

Most of the deeply embedded systems I've seen allocate the bss segment at link time (or, if they're split, at locate time); once the hex file is generated the location of the bss segment is fixed. If there is a heap it is often given whatever memory is left over after the bss segment is filled, and it's size and location is also fixed at link/locate time.

Yes, in a Unix system the bss has to come from the OS's heap -- but that's a different world than many embedded systems come from.

--
Tim Wescott
Wescott Design Services
 Click to see the full signature
Reply to
Tim Wescott
[Please, if you don't refer to the majority of the text of the article you're replying, snip it. Fixed.]

Yep. But the OP of this thread doesn't seem to be in that kind of situation. He is apparently running some kind of OS his program has to cooperate with it.

--
Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de)
Even if all the snow were burnt, ashes would remain.
Reply to
Hans-Bernhard Broeker

Yes. What I tried to explain is that the section must have its contents stored in non-volatile storage somewhere. You are probably thinking of flash; but I'm not. If flash is used for the non-volatile storage, which is okay, then it must be copied into read-write RAM at startup. But I'm not conflating the concept of flash with NV. Flash is NV, but not all NV is flash.

No, I wasn't missing anything. I've written operating systems, many times over a period of 30 years or so. Been there, done that. See above.

Thanks. That's in the deep back of my memory, as I actually did read it. But long since forgot.

Jon

Reply to
Jonathan Kirwan

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.