Memleak in basic system

This suggests there are "non-working hours" (?). Does it incur losses at those times? (Or, is it actually NOT working/running?)

What's it *doing* during the working hours? What *might* be calling for additional memory in that time?

Is there a persistent store or are you logging to a memory disk, etc.?

Where does proc/meminfo show INCREASING memory usage?

When you kill(8) off "your" processes (i.e., anything that was not part of the standard "system"), is the memory recovered correctly by the kernel? Said another way, if you created a cron(8) task to kill off your processes every hour and restart them immediately thereafter, would the problem "go away" (i.e., be limited to a maximum, non-accumulating loss of ~2MB)?

Once you know which process(es) are responsible for the loss, you can explore them in greater detail.

Can't you look at memory usage on a task-by-task basis with ps? How about periodically running it, and looking for a task that's blowing up?

Not arm here, but old x86, so maybe not helpful:

this box (512 Mb ram) has MemFree: 13740 kB file server (128 Mb ram) has MemFree: 2248kB

It's the linux vm caching all sorts of stuff in ram. sooner or later forks will fail, or modules might not load. (basically anything that wants a bigger chunk of cont. memory)

echo 3 > /proc/sys/vm/drop_caches

frees some memory. see if that helps. (use periodically)

...There's another tunable (/proc/sys/vm/user_reserve_kbytes I think) which is supposed to help with that. but in the past I had tried that and module loading would still sometimes fail, and the box would go into swap-storms all the time. Idk... perhaps they fixed that by now...

There is no point in doing that. Kernel will automatically drop caches if processes need it.

Bye Jack

What /exactly/ are you monitoring from /proc/meminfo? If you are looking at MemFree, then you can expect it to go down regularly - once a system has been used for a while, you don't want MemFree to be more than about 10% of the systems memory. Remember, Linux uses free memory for disk cache. It will clear out old disk cache if it needs the memory for something else, but if the memory is not being used for processes, then it is always best to store file data in the spare ram.

So if your system is doing nothing but writing logs to the disk, then it will use steadily more memory for disk caching of the log files. It may not be particularly useful to have the log files in cache, but it is more useful than having nothing at all in memory.

Your key figure for the memory in use by processes (and therefore the memory that might be leaking), is MemFree - Buffers - Cached.

It means not between 8 in the morning and 7 in the evening.

We are saving the MemFree on a monitoring server.

Actually, the box is doing nothing, so there is pretty less to kill. There is an ssh server on which we regularly connect to get /proc/meminfo. The contents of MemFree is added to our monitoring system. After monitoring MemFree for two days on two different systems, this is what we got:

Hm, didn't know ps could show me the used memory... Been searching, but I only found a way to show the percentage of memory. I don't think that is accurate enough to see much difference.

I could use the drop_caches part when monitoring but according to top the caches are pretty stable. I don't think I'm trapped by the kernel cache.

But I consider it as a good idea. It could have happened I didn't take the cache in count.

True, I know it's not a good idea to monitor MemFree, it is influenced by tmpfs and caches. But I also wached the contents of my tmpfs and caches and they almost don't change. On the system there's my ssh server listening and logging to the busybox version of syslog which logs to memory.

It logs to memory but as far as I understand, it just allocates its circular buffer at start and doesn't allocate more memory while running

Do you have any suggestion on how to find out how much is used per process?

No, it is not a good idea to drop the caches manually - it is extremely rare that this is useful outside of disk benchmarking. It /is/ a good idea to take them into account when monitoring memory, of course.

Maybe look at /proc//status? The VmPeak and VmSize lines would be of particular interest.

top can do that. it's the VIRT top f{upupupup}s{ESC}

Can you run top, which shows individual process memory dynamically ?...

Regards,

Chris

Run the application on a desktop computer and use valgrind, LeakSanitizer or any of the other monitoring tools available.

-a

All,

Thanks for the suggestions. It looks like it was just some normal linux actions which were going on. Letting the system run for a week showed us no memleak. MemFree just goes a little up and down and we were worrying to early. So I was just panicking a little to early. Now can add our normal programs and try to find out which one is leaking. We got issues on running systems so that's why I started to look at my basic system in the first place.

Kind regards,

wimpunk.

Get yourself a good book on Linux kernel and read the chapters about memory management, so you'll understand. Unused RAM is excess RAM.

One possible book is: Understanding the Linux Kernel.

Nah, I just needed more coffee and so I would watch the correct values. I made the wrong associations and thought the problem was bigger because of the scaling of the monitoring system. Although, thanks for the suggestion about the book. Next thing on my shortlist is fixing a bug in a usb wifi kernel module.

For that, there is another book, Linux Device Drivers by Corbet et al.