How to: query module pagination with sub-queries or aggregate functions
Posted: 10 July 2009 02:20 PM   [ Ignore ]  
Lab Technician
Avatar
RankRankRankRank
Total Posts:  1027
Joined  12-18-2008

as an update to this thread: as of EE 1.6.8 build 20091201, the code from my last post to this thread is now incorporated into EE so it is not necessary to maintain this hack (or fear upgrading to preserve it).

/happy

================ORIGINAL POST===================

The line we’ve heard time and time again (<- 4 separate links) is that the query module doesn’t support pagination if you have subqueries or grouping in your SQL statement.

As I have mentioned before, it seems to be this way because EE runs the query twice if you’re paginating results, except it replaces the SELECT fields with COUNT(*) to get the total number of results so it knows how many links to create.  I have also mentioned before that this is asinine because the num_rows variable exists whenever you run a query through the $DB->query function.

I have finally butted heads with this issue for the last time, and am putting my money where my mouth is. I propose the following changes as a hack/fix (feel free to incorporate this into the next version, if it passes muster) to this limitation:

mod.query.php:

line 97:

// Edit by Ty Wangsness, allows pagination to work with more complex queries
// $query = $DB->query(preg_replace("/SELECT(.*?)\s+FROM\s+/is", 'SELECT COUNT(*) AS count FROM ', $sql));
$query = $DB->query($sql);

line 99:

// if ($query->row['count'] == 0)
if($query->num_rows == 0)

line 117:

// $this->total_rows = $query->row['count'];
$this->total_rows = $query->num_rows;

It appears to work as expected in my limited tests (read: it works for me, YMMV). Hopefully we can put this issue to rest.

-Ty

 Signature 

EE Pro Network
eMarketSouth - web design & development, custom EE plugins & extensions, SEO, and more

Profile
 
 
Posted: 13 July 2009 07:14 AM   [ Ignore ]   [ # 1 ]  
Lab Technician
Avatar
RankRankRankRank
Total Posts:  1027
Joined  12-18-2008

over the weekend it bothered me that I was running the same query twice for no good reason, so this morning I modified the file yet again to have a class variable $data initially set to FALSE, changed all other instances of $query to $this->data, and added a bit of code toward the end of the pagination:

$sql .= " LIMIT ".$this->p_page.', '.$this->p_limit;
$this->data = FALSE; // <- added this line
if(!$this->data) {
    $this
->data = $DB->query($sql); // <- roughly line 188ish, added conditional around it
}

basically if the query doesn’t need paginating, the query module doesn’t have to run it twice.  if the query is using the pagination then it’ll get run again with a limit the second time.  this isn’t quite optimal because all the necessary data is already stored in the results, but oh well.

This change doesn’t appear to break anything, and as I’ve used the query module quite extensively on our key to savannah site, I’m pretty confident in the change.

noticed a decent number of views on this thread… nobody has anything to say though?  I know I’m not the only one that has had issues with getting pagination to work with non-trivial queries before.  I realize that this has the potential downside of having a large amount of data returned for no reason other than to count the results, but for my purposes it isn’t as big a deal as not having the pagination at all.

 Signature 

EE Pro Network
eMarketSouth - web design & development, custom EE plugins & extensions, SEO, and more

Profile
 
 
Posted: 15 July 2009 08:51 AM   [ Ignore ]   [ # 2 ]  
Lab Technician
Avatar
RankRankRankRank
Total Posts:  1027
Joined  12-18-2008

well I wasn’t happy with the hacks I’d come up with so far because the possibility of returning the entirety of a large dataset in my mind prevented those methods from being safe for general purpose use… so back to the drawing board and came up with this one:

line 97 of mod.query.php:

// $query = $DB->query(preg_replace("/SELECT(.*?)\s+FROM\s+/is", 'SELECT COUNT(*) AS count FROM ', $sql));
$query = $DB->query("SELECT COUNT(*) AS count FROM ($sql) AS query");

instead of using a regular expression to replace the SELECTed fields with COUNT(*) (which obliterates subqueries and is useless with grouping), just make the original query into a subquery and perform the COUNT(*) on the resulting table.  MySQL can’t optimize this *quite* as well as the original… but it won’t break the query, and it’s a lot better than returning all the data for php to count the rows.  In my testing it works very well, and I think it’s about the best we can do for paginating any query until the SQL_CALC_FOUND_ROWS -> FOUND_ROWS() race condition has been solved.

 Signature 

EE Pro Network
eMarketSouth - web design & development, custom EE plugins & extensions, SEO, and more

Profile
 
 
   
 
 
Post Marker Legend
New Topic New posts Hot Topic Hot Topic with new posts New Poll New Poll Moved Topic Moved Topic Sticky Topic Sticky topic
Old Topic No new posts Hot Old Topic Hot Topic with no new posts Old Poll Old Poll Closed Topic Closed Topic Announcement Announcements
Theme
Change Theme
Visitor Statistics
The most visitors ever was 1743, on December 02, 2009 02:47 PM
Total Registered Members: 115005 Total Logged-in Users: 99
Total Topics: 122438 Total Anonymous Users: 60
Total Replies: 647304 Total Guests: 525
Total Posts: 769742    
Members ( View Memberlist )