memory problem related to squashfs

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Hi,

I am using 2.6.8.1 with squashfs 2.0 in my device. Have found a problem
related to squashfs? Any one knows how to solve this?

If I open/read a file for the first time, I will see a used memory
increase.

After the first time, the used memory will become unchanged.

This used memory seems not drop back.

After enable the trace in squashfs. I found there are squashfs logmsg
in the first time file read. The second/third time does not have these
output.

It seems the file will be stored in the memory in the first time read.

This can be test easily with "cat filename". We also has a
RAM fs. cat RAM fs files does not cause used memory increase.

Is this caused by squashfs or by other linux config? Is there a way
disable it?

Thanks!

Yong

4.html is a 121 bytes html file.

# sysinfo
Number of processes: 32
12:39am up 39 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.10
total used free shared buffers
Mem: 30428 14956 15472 0 1180
Swap: 0 0 0
Total: 30428 14956 15472
#
# handle_request filename 4.html
handle_request output filename /webs/4.html

SQUASHFS: Entered squashfs_readpage, page index 0, start block 3b47c
SQUASHFS: Entered squashfs_get_cached_block [28ad65:136a]
SQUASHFS: Block @ 0x3b47c, compressed size 105
SQUASHFS: Calling decompress_lzma_7z: buffer_in size10%5, buffer_out
size65%536
SQUASHFS: Done with decompress_lzma_7z: output size65%536
inode.c read_data 65536
SQUASHFS: bytes 65536, i 0, byte_offset 0, available_bytes 4096
SQUASHFS: bytes 65536, i 1, byte_offset 4096, available_bytes 4096
SQUASHFS: bytes 65536, i 2, byte_offset 8192, available_bytes 4096
SQUASHFS: bytes 65536, i 3, byte_offset 12288, available_bytes 4096
SQUASHFS: bytes 65536, i 4, byte_offset 16384, available_bytes 4096
SQUASHFS: bytes 65536, i 5, byte_offset 20480, available_bytes 4096
SQUASHFS: bytes 65536, i 6, byte_offset 24576, available_bytes 4096
SQUASHFS: bytes 65536, i 7, byte_offset 28672, available_bytes 4096
SQUASHFS: bytes 65536, i 8, byte_offset 32768, available_bytes 4096
SQUASHFS: bytes 65536, i 9, byte_offset 36864, available_bytes 4096
SQUASHFS: bytes 65536, i 10, byte_offset 40960, available_bytes 4096
SQUASHFS: bytes 65536, i 11, byte_offset 45056, available_bytes 4096
SQUASHFS: bytes 65536, i 12, byte_offset 49152, available_bytes 4096
SQUASHFS: bytes 65536, i 13, byte_offset 53248, available_bytes 4096
SQUASHFS: bytes 65536, i 14, byte_offset 57344, available_bytes 4096
SQUASHFS: bytes 65536, i 15, byte_offset 61440, available_bytes 4096

# sysinfo
Number of processes: 32
12:39am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.09
total used free shared buffers
Mem: 30428 15052 15376 0 1180
Swap: 0 0 0
Total: 30428 15052 15376
# sysinfo
Number of processes: 32
12:40am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.09
total used free shared buffers
Mem: 30428 15052 15376 0 1180
Swap: 0 0 0
Total: 30428 15052 15376
# handle_request filename 4.html
handle_request output filename /webs/4.html

# sysinfo
Number of processes: 32
12:40am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.08
total used free shared buffers
Mem: 30428 15060 15368 0 1180
Swap: 0 0 0
Total: 30428 15060 15368
# sysinfo
Number of processes: 32
12:40am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.08
total used free shared buffers
Mem: 30428 15052 15376 0 1180
Swap: 0 0 0
Total: 30428 15052 15376
#


Re: memory problem related to squashfs
Hi,

I am using 2.6.8.1 with squashfs 2.0 in my device. Have found a problem
related to squashfs? Any one knows how to solve this?

If I open/read a file for the first time, I will see a used memory
increase.

After the first time, the used memory will become unchanged.

This used memory seems not drop back.

After enable the trace in squashfs. I found there are squashfs logmsg
in the first time file read. The second/third time does not have these
output.

It seems the file will be stored in the memory in the first time read.

This can be test easily with "cat filename". We also has a
RAM fs. cat RAM fs files does not cause used memory increase.

Is this caused by squashfs or by other linux config? Is there a way
disable it?

Thanks!

Yong

4.html is a 121 bytes html file.

# sysinfo
Number of processes: 32
12:39am up 39 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.10
total used free shared buffers
Mem: 30428 14956 15472 0 1180
Swap: 0 0 0
Total: 30428 14956 15472
#
# handle_request filename 4.html
handle_request output filename /webs/4.html

SQUASHFS: Entered squashfs_readpage, page index 0, start block 3b47c
SQUASHFS: Entered squashfs_get_cached_block [28ad65:136a]
SQUASHFS: Block @ 0x3b47c, compressed size 105
SQUASHFS: Calling decompress_lzma_7z: buffer_in size10%5, buffer_out
size65%536
SQUASHFS: Done with decompress_lzma_7z: output size65%536
inode.c read_data 65536
SQUASHFS: bytes 65536, i 0, byte_offset 0, available_bytes 4096
SQUASHFS: bytes 65536, i 1, byte_offset 4096, available_bytes 4096
SQUASHFS: bytes 65536, i 2, byte_offset 8192, available_bytes 4096
SQUASHFS: bytes 65536, i 3, byte_offset 12288, available_bytes 4096
SQUASHFS: bytes 65536, i 4, byte_offset 16384, available_bytes 4096
SQUASHFS: bytes 65536, i 5, byte_offset 20480, available_bytes 4096
SQUASHFS: bytes 65536, i 6, byte_offset 24576, available_bytes 4096
SQUASHFS: bytes 65536, i 7, byte_offset 28672, available_bytes 4096
SQUASHFS: bytes 65536, i 8, byte_offset 32768, available_bytes 4096
SQUASHFS: bytes 65536, i 9, byte_offset 36864, available_bytes 4096
SQUASHFS: bytes 65536, i 10, byte_offset 40960, available_bytes 4096
SQUASHFS: bytes 65536, i 11, byte_offset 45056, available_bytes 4096
SQUASHFS: bytes 65536, i 12, byte_offset 49152, available_bytes 4096
SQUASHFS: bytes 65536, i 13, byte_offset 53248, available_bytes 4096
SQUASHFS: bytes 65536, i 14, byte_offset 57344, available_bytes 4096
SQUASHFS: bytes 65536, i 15, byte_offset 61440, available_bytes 4096

# sysinfo
Number of processes: 32
12:39am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.09
total used free shared buffers
Mem: 30428 15052 15376 0 1180
Swap: 0 0 0
Total: 30428 15052 15376
# sysinfo
Number of processes: 32
12:40am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.09
total used free shared buffers
Mem: 30428 15052 15376 0 1180
Swap: 0 0 0
Total: 30428 15052 15376
# handle_request filename 4.html
handle_request output filename /webs/4.html

# sysinfo
Number of processes: 32
12:40am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.08
total used free shared buffers
Mem: 30428 15060 15368 0 1180
Swap: 0 0 0
Total: 30428 15060 15368
# sysinfo
Number of processes: 32
12:40am up 40 min,
load average: 1 min:0.00, 5 min:0.00, 15 min:0.08
total used free shared buffers
Mem: 30428 15052 15376 0 1180
Swap: 0 0 0
Total: 30428 15052 15376
#


Re: memory problem related to squashfs
Quoted text here. Click to load it

You do not have a problem. Linux always tries to use your available RAM
as a disk cache. As time goes on and you keep reading uncached parts of
the disk, you'll see your free memory go down to nearly zero. However
when programs begin demanding memory the disk cache will be flushed and
released.

You can usually measure the size of the cache, eg cat /proc/meminfo.

It is always quite tricky to tell exactly how much memory a program is
actually using. Linux does not even load an entire binary into memory at
once, it only loads the parts which are demanded for execution; shared
libraries only occupy memory once but they are mapped into the address
spaces of all the programs which use them.

Re: memory problem related to squashfs
Quoted text here. Click to load it

This is NOT a Squashfs problem.

The used memory increase is because of file caching. This is normal
behaviour for Linux. Linux has three caches which are involved in
filesystem reading: the page cache, the buffer cache and the dentry
(directory entry) cache. As files are read, Linux saves the information
in these caches.

These caches use unallocated free memory. The memory used for the
caches doesn't become unavailable for program use, in low-memory
situations Linux will shrink these caches releasing the memory.

Quoted text here. Click to load it

The used memory will be returned when Linux needs it for other uses.

Quoted text here. Click to load it

Linux RAM filesystems (i.e. tmpfs) store their data inside the caches
(locked so it cannot be flushed). Reading such filesystems do not pull
data into the caches because it is already there.

Quoted text here. Click to load it

The logging data is produced when Squashfs is entered to read
filesystem (page) data. In the second and third cases the file read is
satisfied from the page cache, and so Squashfs isn't re-entered to read
the data, and it therefore doesn't generate any additional logging
data.

This is entirely normal behaviour.

Quoted text here. Click to load it

As previously mentioned, this is normal Linux behaviour. It is such a
central part of the design of Linux that it is very hard to disable it.
As the data used for the caches is returned when required, there is no
reason why you would want to disable it either.
 
Phillip Lougher


Site Timeline