8 of 14
8
Plugin: CSVGrab
Posted: 08 May 2008 01:31 PM   [ Ignore ]   [ # 127 ]  
Lab Assistant
RankRank
Total Posts:  124
Joined  04-02-2008

I would also like to vote for the first post update.

I installed the first post plugin (it didn’t work).
I then had to read through these pages and then found the newer version of the plugin.

Once I installed the correct version everything worked great! I was only importing 150 articles which was more than I wanted to do manually.  I’ll defiantly be using this plugin often. Thank you!

First post would have saved me 1/2 hour.

 Signature 

Using EE since Apr08!

Come chat me up in IRC: #expressionEngine (irc.freenode.net)

My Version 1.6.3 & Build:  20080319

Profile
 
 
Posted: 13 May 2008 10:41 PM   [ Ignore ]   [ # 128 ]  
Research Assistant
Avatar
RankRankRank
Total Posts:  422
Joined  12-15-2005

This worked well for me for about 100,000 entries now, but it’s decided to start grinding again, to the tune of about one entry every two seconds. If I take off the unique setting it zips through pretty well—but instead of inserting duplicates it actually just inserts 1 of the dupe entries. For example, I have several branches of Joe’s Library, but it checks all occurrences and only inserts one of them. I tried a code snippet in an earlier post, but it just borks the script entirely… anyone out there with similar issues?

 Signature 

System Info: Production

EE Version:  1.66
Build
:       20081114
PHP Version
: 5.2.5

Profile
 
 
Posted: 13 May 2008 11:26 PM   [ Ignore ]   [ # 129 ]  
Lab Assistant
Avatar
RankRank
Total Posts:  140
Joined  05-13-2006

I only went as far as 50,000 total entries but the plugin still worked for me. When you try to insert too many records at a time, the duplicate entries occur, so if you can limit it to about 3,000 per run, everything should process smoothly.

 Signature 

Subvert Marketing Inc. - Whitehorse, Yukon, Canada

Profile
 
 
Posted: 13 May 2008 11:54 PM   [ Ignore ]   [ # 130 ]  
Research Assistant
Avatar
RankRankRank
Total Posts:  422
Joined  12-15-2005

OK, wow. First, I had already been trying to import a series of smaller files that I had split apart but to the same effect. However, I went into the database itself and modified the datatype of the field I needed to specify as unique (which was a uid in the csv) to a ‘char’ with a length limit of 10. I then created an index on that column. Bang. I’ve imported 30,000 entries now in the time between your response and this one!

 Signature 

System Info: Production

EE Version:  1.66
Build
:       20081114
PHP Version
: 5.2.5

Profile
 
 
Posted: 16 May 2008 03:38 AM   [ Ignore ]   [ # 131 ]  
Lab Assistant
RankRank
Total Posts:  165
Joined  12-21-2005

Firstly, apologies for neglecting this thread recently. It seemed to have slipped off my “Notify me by email when someone posts” list…

I now (finally) have a new version of CSVGrab that is multiple sites compatible - sorry for the delay in posting. An additional parameter, site_id=”“, has been added to select which site to import the data into.

Also, from now on, rather than posting the plugin in the forum, it now has its own page where the most recent version can be downloaded.

Please see the CSVGrab page on my site for the latest version.

Finally, I’d like to gather some opinions: the usage of the plugin has always been cumbersome and has become increasingly so as features have been added. What would the interest be in a CSVGrab module where the configuration is done using a wizard-style, step-by-step process?

 Signature 

Andrew Weaver — brandnewbox.co.uk


Member of EE Professional Network · More plugins, extensions and modules · Documentation

Profile
 
 
Posted: 16 May 2008 09:30 AM   [ Ignore ]   [ # 132 ]  
Lab Assistant
RankRank
Total Posts:  124
Joined  04-02-2008

IMHO a wizard style step-by-step process would be user friendly for users who don’t often use the ‘grabber’; however, for the more technical user a step-by-step process may take more time.

Thanks for the update Andrew.

 Signature 

Using EE since Apr08!

Come chat me up in IRC: #expressionEngine (irc.freenode.net)

My Version 1.6.3 & Build:  20080319

Profile
 
 
Posted: 20 May 2008 08:38 AM   [ Ignore ]   [ # 133 ]  
Lab Assistant
Avatar
RankRank
Total Posts:  256
Joined  01-23-2006

An interface would be nice for this. Perhaps one that has the file upload right in it. This might help offload this work to non-tech users who want to import data. Currently that would be a little cumbersome.

More so though, it would be great to have an expiration date added. Would be nice to have hours and minutes, not just dates as entry and expiration dates.

Thanks for the plugin though, I use it about once every month!

 Signature 

jeremy douglas media design | Twitter

Profile
 
 
Posted: 28 May 2008 11:34 AM   [ Ignore ]   [ # 134 ]  
Grad Student
Rank
Total Posts:  64
Joined  01-05-2008

I put the newest version (0.2.1) on my site and now I’m getting this error:
Warning: fgetcsv() [function.fgetcsv]: enclosure must be a character in /home/httpd/vhosts/.../pi.csvgrab.php on line 202
I was using 0.1 and it worked great, the only reason I changed it was because I wanted to use the “unique=” and it didn’t seem to work, so I thought maybe that feature wasn’t in the older version.
Here is the template I’m calling it with:
{exp:csvgrab url=“http://...csv”
weblog=“3”
title=“1”
skip=“1”
delimiter=”,”
encloser=”“
category_field=“4”
category_group=“3”
use=“2|3”
fields=“food-details|food-price”
unique=“title,food-details,food-price” }

Like I said, this template works on the same csv with the old version.  I didn’t change anything but the csvgrab version and now it’s not working.  Any Ideas??

Profile
 
 
Posted: 28 May 2008 01:14 PM   [ Ignore ]   [ # 135 ]  
Lab Assistant
RankRank
Total Posts:  165
Joined  12-21-2005

Try adding an encloser=“QUOTE” parameter anyway. The value is treated as optional by the PHP function.

Andrew

 Signature 

Andrew Weaver — brandnewbox.co.uk


Member of EE Professional Network · More plugins, extensions and modules · Documentation

Profile
 
 
Posted: 28 May 2008 01:35 PM   [ Ignore ]   [ # 136 ]  
Grad Student
Rank
Total Posts:  64
Joined  01-05-2008

Yep, that fixed it, thanks alot for the quick response.

Profile
 
 
Posted: 28 May 2008 01:39 PM   [ Ignore ]   [ # 137 ]  
Moderator
Avatar
RankRankRankRankRankRankRank
Total Posts:  12392
Joined  04-29-2002

Andrew, what about a single quote? Is that possible?

 Signature 

Quick Reference - EE Trial Options - EE Wiki - Docs for updating a build

Profile
MSG
 
 
Posted: 29 May 2008 03:32 AM   [ Ignore ]   [ # 138 ]  
Lab Assistant
RankRank
Total Posts:  165
Joined  12-21-2005

I think you can just use a single quote in the encloser (eg, encloser=”’”) but let me know if this does not work.

Andrew

 Signature 

Andrew Weaver — brandnewbox.co.uk


Member of EE Professional Network · More plugins, extensions and modules · Documentation

Profile
 
 
Posted: 05 June 2008 09:46 AM   [ Ignore ]   [ # 139 ]  
Grad Student
Avatar
Rank
Total Posts:  35
Joined  11-27-2007

Unfortunately, I am unable to get CSVGrab working. It looks like some sort of fopen security issue on the server. Here’s an excerpt from the Apache logs:

[Thu Jun 05 10:00:00 2008] [error] [client 12.34.56.789] PHP
Warning
:  fopen(http://255.123.456.78/testing.csv) [<a
href='function.fopen'>function.fopen</a>]: failed to open stream: A
connection attempt failed because the connected party did not properly
respond after a period of time
, or established connection failed
because connected host has failed to respond
.\r\n in
D
:\\folder\\subfolder\\plugins\\pi.csvgrab.php on line 192

Anyone with experience on same issue?

[UPDATE Jun 09]

After further experimentation, it seems that this error and subsequent failures may be due to PHP version 5 versus 4. Everything works fine under PHP4, but fails under PHP5. All other variables the same. Only change being version of PHP running.

Profile
 
 
Posted: 27 June 2008 04:51 PM   [ Ignore ]   [ # 140 ]  
Lab Assistant
Avatar
RankRank
Total Posts:  185
Joined  08-03-2007

Can anyone tell me what this means? 

Fatal error: Call to undefined function mb_convert_encoding() in /home/[path removed]/plugins/pi.csvgrab.php on line 231

I see this error when I attempt to view the go page to “activate” the plugin.

I can’t get CSVGrab working.

 Signature 

Paul Burton | 16toads Design

Profile
 
 
Posted: 27 June 2008 05:02 PM   [ Ignore ]   [ # 141 ]  
Moderator
Avatar
RankRankRankRankRankRankRank
Total Posts:  15380
Joined  05-15-2004

I think it indicates missing Unicode support (mb = multibyte) on your host. Ask them about it.

 Signature 

Everything will be good in the end. If it’s not good, it’s not the end.

Profile
MSG
 
 
Posted: 27 June 2008 08:47 PM   [ Ignore ]   [ # 142 ]  
Lab Assistant
Avatar
RankRank
Total Posts:  185
Joined  08-03-2007

Got it sorted out with my host and it seems to be functioning.

However, my test import is importing the custom fields from the CSV row 1 and nothing else in any of the data rows below.

Everything is mapped correctly.

category, title, name, professional_exp, state
party
, president, mr. jones, lawyer, iowa

Instead of pulling ‘president’ into the ‘title’ field in the weblog, it’s pulling ‘title’ into the ‘title’ field and so on.

 Signature 

Paul Burton | 16toads Design

Profile
 
 
Posted: 27 June 2008 09:00 PM   [ Ignore ]   [ # 143 ]  
Grad Student
Rank
Total Posts:  67
Joined  10-18-2002

Could you post your template?

Profile
 
 
Posted: 28 June 2008 06:06 AM   [ Ignore ]   [ # 144 ]  
Lab Assistant
Avatar
RankRank
Total Posts:  185
Joined  08-03-2007

{exp:csvgrab url="http://[url]/csv/officials-shortFormtest.csv"
    
delimiter=","
    
encloser="QUOTE"
    
weblog="17"
    
category_group="9"
    
title="2"
    
category_field="1"
    
use="3|4|5|6|7|8|9"
    
fields="official-name|state|current-office|family|prof-experience|congress-committees|campaign-address"
    
unique="official-name"
}

So, I’m not sure what is going on at this point.  I resaved the CSV file and all of the sudden the test worked and all columns were imported into their correct fields in the weblog.  I changed the title and official_name field for a second import test and it didn’t work - just a white screen on the go page.

Update:  I think I’ve got this figured out.  Turns out exporting a CSV from Excel didn’t work - CSVGrab didn’t like the file.  I had to convert the XLS doc to a Numbers format then export to CSV from Numbers. Strange, but it worked.  Great Plugin.

 Signature 

Paul Burton | 16toads Design

Profile
 
 
   
8 of 14
8
 
Post Marker Legend
New Topic New posts Hot Topic Hot Topic with new posts New Poll New Poll Moved Topic Moved Topic Sticky Topic Sticky topic
Old Topic No new posts Hot Old Topic Hot Topic with no new posts Old Poll Old Poll Closed Topic Closed Topic Announcement Announcements
Theme
Change Theme
Visitor Statistics
The most visitors ever was 1149, on July 16, 2007 09:33 AM
Total Registered Members: 64944 Total Logged-in Users: 63
Total Topics: 81920 Total Anonymous Users: 38
Total Replies: 440347 Total Guests: 279
Total Posts: 522267    
Members ( View Memberlist )