Thread

We are working on a new fairly large EE site built in 1.7.  We have roughly 250 weblogs, and over 30 user groups managed by Publish Plus.  We’ve run into some issues with the memory of the setting column for extensions.  We seem to have fixed the issue by changing the column from TEXT to LONGTEXT.  We had one of our developers, who normally doesn’t work with EE, look into the problem to see if there was a way to accommodate the site as it continues to grow so we don’t run into these issues.  He let us know that the problem doesn’t stem from the individual extensions, but how extensions are handled by EE in general.  This is what he wrote back:

“I looked through the code and it seems as if the getting and setting of settings are done in one place.  Everything is loaded into memory at once and saved at once.  Thus, there are no hooks to optimize this process.  EE simply asks an extension to get its data in two places: (1) when the extension is first loaded and (2) after the Settings Form is saved.  Also, it only saves settings in one place: when the Settings Form is saved.  Thus, the time it takes to read the data from and write the data to the database isn’t as important since it’s done very infrequently.  However, this is not a very good design.  Since all of the settings are loaded at once (and never unloaded while the extension is running) they are in memory at all times.  You can imagine that a lot of settings spread over dozens of installed extensions will lead to some memory issues on the server.  I doubt your application will become large enough that you won’t be able to increase the memory to accommodate this (and you may not even need to do that) but this is a terrible design for scalability and another example of how EE wasn’t written for large enterprise applications. There is no way to change this for an extension, as you don’t have control over when the code to get and save extension settings is called.  That is all done by internal EE extension handling code.

So in short, leave it as it is because there’s no other choice in EE.”

I’m wondering if how the settings are saved has changed in EE 2.0 for better optimization for larger sites?  Or is this true that EE truly isn’t able to handle large scale enterprise sites?

I’ve encountered this in one instance. Our initial solution was to change the database from TEXT to MEDIUMTEXT. That worked in terms of fitting the size of the string. It also didn’t have any negative impact on the front-end of our site. What it did do, however, is render one of our extension settings page unusable because EE couldn’t crunch the data (more specifically, the server couldn’t due to PHP memory limits - even at 256mb).

Thankfully, the culprit of the long settings was an add-on that could be extended and customized. I ended up using another db table just for those settings alone and all worked just fine.

I’m wondering if how the settings are saved has changed in EE 2.0 for better optimization for larger sites?  Or is this true that EE truly isn’t able to handle large scale enterprise sites?

Settings are handled the same was in EE 2.x (for now)

To follow up on this, one option is to contact the 3rd party developer and ask them if they can change how they store settings. It is technically possible for a 3rd party add-on to create and use its own tables for settings. Some actually already do this.

The Extension Settings table wasn’t built to handle intricate settings since extensions are typically pretty simple.

How many extensions are you using?  I’m not sure I get why this can be a problem unless the way you are using EE has gone seriously weird. 

There are a fairly large number of big sites using EE and while there’s been scale issues in certain areas I’ve never once had them be memory related in this way…

I also don’t quite understand how changing a database column from TEXT to LONG TEXT solved a memory issue with extensions…  LONG TEXT would use MORE memory since it allows you to store more data in a column than TEXT. 

I’ve had to adjust from TEXT to LONG TEXT in a few very unusual situations and I don’t think I’d call that a design issue in EE so much.  EE quite intelligently uses TEXT by default as its really all it’s going to need 99% in those other 1% of the time situations you can bump it up.

Jamie

No piece of software (or most anything else - any tool) can be designed to be all things to all people. It is my guess that “large scale enterprise sites” are not the target market for EE.

It would be my guess that large installations using (free or inexpensive) CMS software do a LOT of tweaking in order to serve the masses - maybe one example being Huffington Post (Wordpress, I think) with 24 million visitors per month. I can certainly guess that is not a default installation!

Of course, defining “large scale” is difficult in itself.
Based on the various lists of “top EE sites” (which I definitely think are not accurate - but yet give some idea of the basics, any site which would be in the top 15,000 on quantcast might qualify. One popular site is paidcontent.org, which also seems to run sister sites on the same EE installation.
http://www.quantcast.com/paidcontent.org

Our site, which runs mostly EE, is similar in traffic to sites like Nielsen.com, BMWUSA.com and many others which would be classified as pretty serious enterprises.

I think, in this case what you are running into is one of the millions of possible uses for EE and all the various extensions and add-ons which may not play well together without tweaking. This is probably true in a lot of cases - installations could be optimized for particulars uses…

One problem with this is that you have to pretty much decide whether to stay with 1.7 for a couple years before investing too much time and energy in customizing…

Anyway, just ramblings.

We know that NSM Publish Plus is causing the issue.  We’ve already talked with Leeevi about it.  He said the only thing we could do was to increase the column to long text. 

With the column shown as short text we ran into an issue when we had approx. 250 weblogs with about 10 different member groups.  Changing the column to long text has fixed the issue but we know that’s only a short term issue.