memory problem related to squashfs

Hi,

I am using 2.6.8.1 with squashfs 2.0 in my device. Have found a problem related to squashfs? Any one knows how to solve this?

If I open/read a file for the first time, I will see a used memory increase.

After the first time, the used memory will become unchanged.

This used memory seems not drop back.

After enable the trace in squashfs. I found there are squashfs logmsg in the first time file read. The second/third time does not have these output.

It seems the file will be stored in the memory in the first time read.

This can be test easily with "cat filename". We also has a RAM fs. cat RAM fs files does not cause used memory increase.

Is this caused by squashfs or by other linux config? Is there a way disable it?

Thanks!

Yong

4.html is a 121 bytes html file.

# sysinfo Number of processes: 32

12:39am up 39 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.10 total used free shared buffers Mem: 30428 14956 15472 0 1180 Swap: 0 0 0 Total: 30428 14956 15472 # # handle_request filename 4.html handle_request output filename /webs/4.html

SQUASHFS: Entered squashfs_readpage, page index 0, start block 3b47c SQUASHFS: Entered squashfs_get_cached_block [28ad65:136a] SQUASHFS: Block @ 0x3b47c, compressed size 105 SQUASHFS: Calling decompress_lzma_7z: buffer_in size=105, buffer_out size=65536 SQUASHFS: Done with decompress_lzma_7z: output size=65536 inode.c read_data 65536 SQUASHFS: bytes 65536, i 0, byte_offset 0, available_bytes 4096 SQUASHFS: bytes 65536, i 1, byte_offset 4096, available_bytes 4096 SQUASHFS: bytes 65536, i 2, byte_offset 8192, available_bytes 4096 SQUASHFS: bytes 65536, i 3, byte_offset 12288, available_bytes 4096 SQUASHFS: bytes 65536, i 4, byte_offset 16384, available_bytes 4096 SQUASHFS: bytes 65536, i 5, byte_offset 20480, available_bytes 4096 SQUASHFS: bytes 65536, i 6, byte_offset 24576, available_bytes 4096 SQUASHFS: bytes 65536, i 7, byte_offset 28672, available_bytes 4096 SQUASHFS: bytes 65536, i 8, byte_offset 32768, available_bytes 4096 SQUASHFS: bytes 65536, i 9, byte_offset 36864, available_bytes 4096 SQUASHFS: bytes 65536, i 10, byte_offset 40960, available_bytes 4096 SQUASHFS: bytes 65536, i 11, byte_offset 45056, available_bytes 4096 SQUASHFS: bytes 65536, i 12, byte_offset 49152, available_bytes 4096 SQUASHFS: bytes 65536, i 13, byte_offset 53248, available_bytes 4096 SQUASHFS: bytes 65536, i 14, byte_offset 57344, available_bytes 4096 SQUASHFS: bytes 65536, i 15, byte_offset 61440, available_bytes 4096

# sysinfo Number of processes: 32

12:39am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.09 total used free shared buffers Mem: 30428 15052 15376 0 1180 Swap: 0 0 0 Total: 30428 15052 15376 # sysinfo Number of processes: 32 12:40am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.09 total used free shared buffers Mem: 30428 15052 15376 0 1180 Swap: 0 0 0 Total: 30428 15052 15376 # handle_request filename 4.html handle_request output filename /webs/4.html

# sysinfo Number of processes: 32

12:40am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.08 total used free shared buffers Mem: 30428 15060 15368 0 1180 Swap: 0 0 0 Total: 30428 15060 15368 # sysinfo Number of processes: 32 12:40am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.08 total used free shared buffers Mem: 30428 15052 15376 0 1180 Swap: 0 0 0 Total: 30428 15052 15376 #
Reply to
yozhang
Loading thread data ...

Hi,

I am using 2.6.8.1 with squashfs 2.0 in my device. Have found a problem related to squashfs? Any one knows how to solve this?

If I open/read a file for the first time, I will see a used memory increase.

After the first time, the used memory will become unchanged.

This used memory seems not drop back.

After enable the trace in squashfs. I found there are squashfs logmsg in the first time file read. The second/third time does not have these output.

It seems the file will be stored in the memory in the first time read.

This can be test easily with "cat filename". We also has a RAM fs. cat RAM fs files does not cause used memory increase.

Is this caused by squashfs or by other linux config? Is there a way disable it?

Thanks!

Yong

4.html is a 121 bytes html file.

# sysinfo Number of processes: 32

12:39am up 39 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.10 total used free shared buffers Mem: 30428 14956 15472 0 1180 Swap: 0 0 0 Total: 30428 14956 15472 # # handle_request filename 4.html handle_request output filename /webs/4.html

SQUASHFS: Entered squashfs_readpage, page index 0, start block 3b47c SQUASHFS: Entered squashfs_get_cached_block [28ad65:136a] SQUASHFS: Block @ 0x3b47c, compressed size 105 SQUASHFS: Calling decompress_lzma_7z: buffer_in size=105, buffer_out size=65536 SQUASHFS: Done with decompress_lzma_7z: output size=65536 inode.c read_data 65536 SQUASHFS: bytes 65536, i 0, byte_offset 0, available_bytes 4096 SQUASHFS: bytes 65536, i 1, byte_offset 4096, available_bytes 4096 SQUASHFS: bytes 65536, i 2, byte_offset 8192, available_bytes 4096 SQUASHFS: bytes 65536, i 3, byte_offset 12288, available_bytes 4096 SQUASHFS: bytes 65536, i 4, byte_offset 16384, available_bytes 4096 SQUASHFS: bytes 65536, i 5, byte_offset 20480, available_bytes 4096 SQUASHFS: bytes 65536, i 6, byte_offset 24576, available_bytes 4096 SQUASHFS: bytes 65536, i 7, byte_offset 28672, available_bytes 4096 SQUASHFS: bytes 65536, i 8, byte_offset 32768, available_bytes 4096 SQUASHFS: bytes 65536, i 9, byte_offset 36864, available_bytes 4096 SQUASHFS: bytes 65536, i 10, byte_offset 40960, available_bytes 4096 SQUASHFS: bytes 65536, i 11, byte_offset 45056, available_bytes 4096 SQUASHFS: bytes 65536, i 12, byte_offset 49152, available_bytes 4096 SQUASHFS: bytes 65536, i 13, byte_offset 53248, available_bytes 4096 SQUASHFS: bytes 65536, i 14, byte_offset 57344, available_bytes 4096 SQUASHFS: bytes 65536, i 15, byte_offset 61440, available_bytes 4096

# sysinfo Number of processes: 32

12:39am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.09 total used free shared buffers Mem: 30428 15052 15376 0 1180 Swap: 0 0 0 Total: 30428 15052 15376 # sysinfo Number of processes: 32 12:40am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.09 total used free shared buffers Mem: 30428 15052 15376 0 1180 Swap: 0 0 0 Total: 30428 15052 15376 # handle_request filename 4.html handle_request output filename /webs/4.html

# sysinfo Number of processes: 32

12:40am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.08 total used free shared buffers Mem: 30428 15060 15368 0 1180 Swap: 0 0 0 Total: 30428 15060 15368 # sysinfo Number of processes: 32 12:40am up 40 min, load average: 1 min:0.00, 5 min:0.00, 15 min:0.08 total used free shared buffers Mem: 30428 15052 15376 0 1180 Swap: 0 0 0 Total: 30428 15052 15376 #
Reply to
yozhang

You do not have a problem. Linux always tries to use your available RAM as a disk cache. As time goes on and you keep reading uncached parts of the disk, you'll see your free memory go down to nearly zero. However when programs begin demanding memory the disk cache will be flushed and released.

You can usually measure the size of the cache, eg cat /proc/meminfo.

It is always quite tricky to tell exactly how much memory a program is actually using. Linux does not even load an entire binary into memory at once, it only loads the parts which are demanded for execution; shared libraries only occupy memory once but they are mapped into the address spaces of all the programs which use them.

Reply to
Geronimo W. Christ Esq

This is NOT a Squashfs problem.

The used memory increase is because of file caching. This is normal behaviour for Linux. Linux has three caches which are involved in filesystem reading: the page cache, the buffer cache and the dentry (directory entry) cache. As files are read, Linux saves the information in these caches.

These caches use unallocated free memory. The memory used for the caches doesn't become unavailable for program use, in low-memory situations Linux will shrink these caches releasing the memory.

The used memory will be returned when Linux needs it for other uses.

Linux RAM filesystems (i.e. tmpfs) store their data inside the caches (locked so it cannot be flushed). Reading such filesystems do not pull data into the caches because it is already there.

The logging data is produced when Squashfs is entered to read filesystem (page) data. In the second and third cases the file read is satisfied from the page cache, and so Squashfs isn't re-entered to read the data, and it therefore doesn't generate any additional logging data.

This is entirely normal behaviour.

As previously mentioned, this is normal Linux behaviour. It is such a central part of the design of Linux that it is very hard to disable it. As the data used for the caches is returned when required, there is no reason why you would want to disable it either. Phillip Lougher

Reply to
phillip.lougher

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.