Thread

The only thing the host can tell us about the excessive memory usage by our website is that they are seeing a lot of requests on the cache/db_cache folder. Apparently the website is requesting lots of cache files in this folder that don’t exist.

I have just turned off query caching to see if this helps. But why would EE be looking for files in the db_cache folder that don’t exist?

I don’t think it would, by default. That said, and bear in mind that I am not a server admin, I don’t think this would adequately explain the server load they are reporting.

Still, let us know if turning off caching changes things for you; it just might.

Turning off Database Caching has only had a tiny effect on the memory usage. The host is still seeing huge memory usage by our website.

I still think it’s a caching problem. Just after I empty the cache or update a template the cache folder is empty and our mySQL database is about 65 Mb. After about 8 hours of not doing anything in CP the cache folder has grown to over 400 Mb and the database has grown to 250 Mb.

Is that normal? We have only about 600 to 700 unique visitors per month, we have 650 membes and about 5,800 weblog entries. How can the database grow four times in size from caching when database caching is turned off?

Thanks.

Can anyone help with this? I just checked the db_cache folder and there are more than 27,000 files in there. Also our database has 150 Mb of cache data in it right now.

How can this be possible when SQL Query Caching in Database Settings is not enabled? Is there another place where this can be turned off?

Thanks.

Hi eeart,

Do you have CP Home > Admin > Weblog Administration > Global Weblog Preferences > Cache Dynamic Weblog Queries? switched on?

Cheers

Greg

Yes, that was on. I’ve switched it off now. Hopefully this will fix my problem. Thank you.

Let us know how it goes, please 😊

Turning off caching has not resolved the memory issue. Our host is actually seeing larger processes of around 250 Mb now. :(

I’ve started to look into analyzing this via shell, but I am just a newbie at this. I used the Linux “top” command to get a realtime view of all my processes on the server and when I load our homepage all of a sudden there’s around 15 processes for “apache2” with a virtual size of 250 Mb each.

I just made a dump of the database and it was about 200 Mb again. Then I went into CP and cleared the “Cached Relationship Entries” .. now the database dump is just 47 Mb. Is it normal that these cached relationships would use up so much space in the database?

Now I’ve cleared the cache completely and I still see many 250 Mb processes. So I guess it’s not the large cache that is causing the problems.

I just turned on debugging in EE and the homepage is using 109 SQL queries (!) and it says that it used 12 Mb of memory, even though I can see in shell that it uses many processes of 250 Mb each. When loading the page a second time the number of SQL queries has gone down to 44, so I guess that a lot of them are still being cached.

What is going on? How can I debug this further?

eart-
I am going to see if our dev team can provide any additional insight. I will pass on any info from them, as soon as I hear something.

Thanks!
Adam

Here is an example of some processes from one of the web servers in our web clusters.

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
 7177 <userid>   15   0  294m  23m 3536 S  9.9  0.3   0:01.48 httpd              
 6637 <userid>   15   0  289m  19m 3332 S  5.9  0.2   0:04.16 httpd              
 5461 <userid>   15   0  289m  19m 3416 S  4.3  0.2   0:03.55 httpd              
 3125 <userid>   15   0  294m  23m 3244 S  3.0  0.3   0:03.78 httpd

2 of those processes above are using 19megs of ram and two are using 23megs of ram as indicated in the RES column.

You should not be worried about VIRT memory usage you should be looking more at RES or “resident memory” usage.  This will vary from server to server based on apache modules that are loaded/being used as well, but in my opinion you should generally see RES in the 15m - 30m size range for most EE sites, though could be larger depending on certain processes like file uploads and image processing.  It would be extremely rare to see the RES memory size grow very much larger than the php memory_limit size set in php.ini.

A good description of the VIRT RES and SHR memory columns in top:

VIRT column (virtual memory size of the process) means how much memory a particular application has requested, also it counts swapped to the disk memory pages. But this field does NOT mean that this process actually uses that amount of memory, to use it for real this memory has to be initiated for example using memset. Generally speaking you can disregard this column.

RES column (resident set size, the non-swapped physical memory that a task has used is the most important one – it actually shows how much physical RAM is allocated for a process.

SHR column says how much memory a particular application is sharing with other applications. E.g., most applications in Linux depend on libc, and as every application links with it and uses its functions then this library memory space is mapped to all process simultaneously to conserve memory.

Only thing I can say is that if your hosting provider is basing “memory usage” on the VIRT memory size rather than the RES memory size I would recommend finding a new hosting provider quickly.

Also a quick way to make sure your db and sql query caches are really off in EE is to directly edit your config.php file in your “system” folder.  Make a copy of your config.php file before you edit it directly, but right before the final ?> put the following two lines:

$conf['enable_db_caching'] = "n";
$conf['enable_sql_caching'] = "n";

Then go into your “cache” directory inside of your “system” folder and manually delete the db_cache and sql_cache directories.

While at some point the db_cache and sql_cache directories will reappear, they should stay empty.

You should not be worried about VIRT memory usage you should be looking more at RES or “resident memory” usage.

Thank you Nevin. You’re right, I was looking at the VIRT column, but the RES column is not much less. The processes in the RES column are between 194m and 204m.

It would be extremely rare to see the RES memory size grow very much larger than the php memory_limit size set in php.ini.

The memory_limit of PHP is 64 Mb. Then how can there be apache2 processes of 200 Mb? Does that mean it’s not EE that is causing these problems but something else?

By the way, when I load a bunch of pages in EE’s Control Panel then I also see lots of processes of approximately 200m appear. There’s something fundamentally wrong somewhere.

Have you tried creating a simple PHP page to access a fresh DB, and run some queries? What is the PHP/Apache Memory usage then?

I have a feeling this is server/PHP related, and not necessarily exclusively ExpressionEngine.