Plugin: FeedGrab | ExpressionEngine Community Forums

9 of 11

« First

Plugin: FeedGrab

J. Hull

Posted: 05 July 2008 04:34 PM

[ Ignore ] [ # 145 ]

Grad Student

Total Posts: 79

Joined 02-06-2007

OK - tried this for the first time importing my Tweets into my personal EE blog space.

First time it choked because of some weirdness I had with the new weblog/new custom fields. I deleted both and started over from scratch.

Second time - worked great! Imported in the 20 posts.

Although I just discovered EE’s 100 char. limit in the Title field. Doh! This results in many tweets being truncated (especially with the inclusion of the username, see below). Probably best to add another field for the tweet itself for anyone trying to do this too.

Two things I need to figure out now:

Twitter’s pubDate is not adjusted for my timezone, i.e. the date saved with my entry is 7 hours ahead of what it should be. Is there a way to modify this at the time of grabbing the RSS entry?

Secondly, the problem with the username affixed to the beginning of the title. I found this Trimmer plugin that can do it when I output the data, but again is there a way to parse this during the time of the RSS grab?

And I’m with klick above - need to be able to access that lower level data in FriendFeed AND be able to extract the two different titles from Google Reader shared items which currently look like this:

<link rel="alternate" href="http://www.kottke.org/remainder/08/06/15936.html" type="text/html"/>
<link rel="related" href="http://www.kottke.org/" title="kottke.org"/> 

Obviously the “alternate” one would be the one I’d like to use as the Title. “related” would be good for a “via” link (like the Google Reader Shared page).

Profile

allgood2

Posted: 05 July 2008 06:36 PM

[ Ignore ] [ # 146 ]

Research Assistant

Total Posts: 779

Joined 12-01-2002

There is an offset function in FeedGrab that allows you to modify date; but the one person I know who used it, was getting duplicate entries. Though admittedly, they were making other modifications as well. So I’d give it a try and see how it works for you. Make sure you have the most current version. I don’t use FriendFeed, so no comments on best method for that.

Profile

J. Hull

Posted: 06 July 2008 05:43 PM

[ Ignore ] [ # 147 ]

Grad Student

Total Posts: 79

Joined 02-06-2007

Hopefully there’s still work being done on this because I’ve got lots of uses for it in the coming months!

Added the offset and it works great (value is in seconds, for West Coasters that’s -25,200). I can see there being a problem when DST switches. Probably not worth the effort to incorporate that into plugin (easy change twice a year).

I just tried running it again with the unique value set and it worked great - no duplicate entries, I did get this error code though:

Notice: Undefined property: MagpieRSS::$last_modified in .../plugins/pi.feedgrab.php on line 1711 

Hope this helps.

Profile

J. Hull

Posted: 06 July 2008 07:05 PM

[ Ignore ] [ # 148 ]

Grad Student

Total Posts: 79

Joined 02-06-2007

REMOVED - not pertinent to this thread

Profile

J. Hull

Posted: 08 July 2008 06:10 PM

[ Ignore ] [ # 149 ]

Grad Student

Total Posts: 79

Joined 02-06-2007

OK - trying to break down the FriendFeed RSS using FeedGrab.

Like klick above I can get title, pubDate, and the link. But what I am really interested in is grabbing my comment within that entry. With the RSS feed, my comments feed is formatted like this:

<ff:comments>
  <ff:comment>
    <ff:date>2008-07-08T18:39:53Z</ff:date>
    −<ff:body>
       This is where my comment is!
     </ff:body>
    −<ff:user>
       <ff:id>This is where my USER ID is</ff:id>
       <ff:name>ny name</ff:name>
       <ff:nickname>myname</ff:nickname>
       <ff:profileUrl>http://friendfeed.com/myname</ff:profileUrl>
     </ff:user>
   </ff:comment>
</ff:comments> 

The problem is this might be within a list of 30-40 comments depending on the entry. Is there someway, using this plugin or XMLGrab, where I can check my USER ID, and if it is me, put the <body> of that content into my weblog? Can I use the @ to access this info?

Profile

scottb

Posted: 09 July 2008 10:11 AM

[ Ignore ] [ # 150 ]

Lab Assistant

Total Posts: 161

Joined 11-28-2007

I am researching the possibility of building a portal for an organization. The site would grab about 200 different RSS feeds daily. Some of them will need to be updated hourly. Drupal has extensive aggregator functions, but I would prefer to build this site in EE.

I can’t use Magpie because the pages strain under pulling multiple feeds, but FeedGrab may be able to do what I want. One problem I need to overcome is the automation of grabbing feeds. I have seen in another post a reference to using the EE cron plugin with FeedGrab. Based on what I read, my code might look something like this:

{exp:cron minute=“30” hour=”*” day=”*” month=”*” plugin=“feedgrab:FeedGrab”}

{exp:feedgrab url=“path_to_rss_feed”
weblog=“id_number”
title=“title”
date=“pubDate”
use=“link|description”
fields=“rss_url|rss_body” }

{/exp:cron}

The cron runs once an hour every day and every month. I have the impression from the cron documentation that I need to define both the name of the plugin and the function that needs to be called.

Unfortunately, this doesn’t work, and other variations don’t work, either. Can anyone offer suggestions?

Signature

Promise Media

Profile

Mike Essl

Posted: 09 July 2008 11:26 PM

[ Ignore ] [ # 151 ]

Grad Student

Total Posts: 60

Joined 04-17-2002

Hello, I’m getting duplicate entries, but only on the 3 newest entries:

Here is my code:

{exp:feedgrab url="http://api.flickr.com/services/feeds/photos_public.gne?id=96729633@N00&tags=unexpectedlyquitcom&lang=en-us&format=rss_200" 
weblog="1" 
title="title"
date="pubDate"
use="media:content@url|media:content@height|media:content@width|link" 
fields="flickrimageurl|flickrimageheight|flickrimagewidth|flickrurl"
unique="date,flickrurl”} 

I thought adding the unique setting would fix it but no dice. Even weirder is that it doesn’t always do it.

You can see the site here:
http://www.unexpectedlyquit.com

Profile

J. Hull

Posted: 16 July 2008 03:43 PM

[ Ignore ] [ # 152 ]

Grad Student

Total Posts: 79

Joined 02-06-2007

Great news!

At least for anyone interested in grabbing lower-level information from RSS feeds (like klick and me above).

I figured out a way for FeedGrab to be able to grab this information by using two Yahoo Pipes. For my purposes I was able to grab out my FriendFeed comments and feed them into my database.

The whole story, with links to the Pipes so you can mash your own, can be found here:

Integrating My FriendFeed Comments Into My Personal Blog

Hope this saves someone time in the future.

Profile

SSM

Posted: 19 July 2008 10:15 AM

[ Ignore ] [ # 153 ]

Summer Student

Total Posts: 24

Joined 07-12-2008

what if you don’t want all those RSS feeds stored in a weblog? I want to import my message board feed, but I have no desire to store thousands of message board thread titles. I just want the last 5 threads on my front page.

Is there a solution for this?

Profile

J. Hull

Posted: 19 July 2008 10:51 AM

[ Ignore ] [ # 154 ]

Grad Student

Total Posts: 79

Joined 02-06-2007

I think you would use Magpie for that (built into EE if I’m not mistaken).

I think the difference is:

FeedGrab is for taking the information in RSS feeds and putting them in your weblog.
Magpie is for simply displaying RSS feeds within your template (no information copied over).

I could be wrong, but that is my understanding.

Profile

supersusie

Posted: 22 July 2008 11:53 PM

[ Ignore ] [ # 155 ]

Research Assistant

Total Posts: 647

Joined 05-16-2004

I’m getting duplicate entries sometimes, like Mike Essl above. Any suggestions on how to debug?

TTFN
Travis

Signature

Fight spam better with “Defensio for EE,” a free module

Hop Studios Internet Consulting
http://www.hopstudios.com/

Profile

Andrew Weaver

Posted: 23 July 2008 02:40 AM

[ Ignore ] [ # 156 ]

Lab Assistant

Total Posts: 165

Joined 12-21-2005

I’m not sure why you are getting duplicates. This is how I import flickr feeds:

{exp:feedgrab
    url="http://api.flickr.com/services/feeds/photos_public.gne?id=25509357@N00&format=rss_200" 
    weblog="1" 
    title="title"
    date="dc:date.Taken"
    use="link|media:content@url|media:thumbnail@url|description|guid" 
    fields="flickr_link|flickr_image|flickr_thumbnail|flickr_description|flickr_guid" 
    unique="flickr_guid"
    category_field="media:category"
    category_group="2"
    category_delimiter="SPACE" 
} 

I use the guid field as the unique value. Let me know if this helps.

Signature

Andrew Weaver — brandnewbox.co.uk

Member of EE Professional Network · More plugins, extensions and modules · Documentation

Profile

J. Hull

Posted: 23 July 2008 12:13 PM

[ Ignore ] [ # 157 ]

Grad Student

Total Posts: 79

Joined 02-06-2007

Just so everyone knows, now that the excitement has died down with my whole Yahoo Pipes solution in combination with FeedGrab - it’s not really working out as well as I thought. There is some caching issue with Yahoo Pipes that isn’t allowing the feeds to be updated. Consequently it doesn’t work so well.

I’m hoping XMLGrab will be the solution, if not I’ll have to write my own plugin (which I’ve never done!)

Profile

Justin (Hop Studios)

Posted: 23 July 2008 07:15 PM

[ Ignore ] [ # 158 ]

Summer Student

Total Posts: 29

Joined 05-02-2007

Regarding Travis’s duplicate issue above (http://expressionengine.com/forums/viewthread/37598/P144/#433002):

It appears that the text of values inserted into the database is urlencoded. If you’re using a URL value as your unique identifier, and if the URL includes certain characters, is_entry_unique() will always return true, even if that URL is already present in the database (”%2b” does not equal “+”, for example).

We added a urldecode to the is_entry_unique() function (at appx. line 586-588):

// MODIFIED BY JUSTIN CRAWFORD
//$sql .= " AND " . $name . "=\"" . $DB->escape_str( $post[ $value ] ) . "\"";
$sql .= " AND " . $name . "=\"" . $DB->escape_str( urldecode( $post[ $value ] ) ) . "\""; 

-Justin

Signature

Hop Studios Internet Consulting
http://www.hopstudios.com/

Profile

SSM

Posted: 24 July 2008 11:57 AM

[ Ignore ] [ # 159 ]

Summer Student

Total Posts: 24

Joined 07-12-2008

I have 36,000 RSS entries I would like to import. Any advice on how to approach this?

Profile

scottb

Posted: 25 July 2008 11:15 AM

[ Ignore ] [ # 160 ]

Lab Assistant

Total Posts: 161

Joined 11-28-2007

Update: You can ignore the following post. I found what I needed to fix it.

I built out test pages on our existing site (version 1.6.3), and everything worked great. Then I copied the pages over to a new site (v1.6.4) along with the plugin. Suddenly on the new site I’m getting url output errors.

This code:

<a class=“rss_head” target=“_blank” href=”{rss_url}”>{title}</a><br />
<div class=“rss_description”>{rss_body}</div>

Works correctly on the first site. But on the second site, {rss_url} is outputting a complete path including the <a > so that I end up with two <a >s.

I have looked over the admin options for setting url output but don’t see anything that would indicate why I’m getting the errors. Any suggestions?

Signature

Promise Media

Profile

klick

Posted: 03 August 2008 04:46 AM

[ Ignore ] [ # 161 ]

Grad Student

Total Posts: 59

Joined 06-28-2007

J. Hull - 05 July 2008 04:34 PM
And I’m with klick above - need to be able to access that lower level data in FriendFeed AND be able to extract the two different titles from Google Reader shared items which currently look like this:

still haven’t figuered that out in detail.

plus having another problem:
my delicious feed keeps sticking empty entries (all with timestamp 0100)
between the other weblog entries when updating the “plugin call” template.

it also echoes this error code:

Notice: strtotime() [function.strtotime]: Called with empty time parameter in …/xx/plugins/pi.xmlgrab.php on line 489 

this causes my feeds to break.
if anyone has got a clue what’s happening let me know

ah and the twitter stream has some character encoding problems.
i’m on that.

besides that: wonderful plugin! thanks!

best
klick

Profile

chabbs

Posted: 05 August 2008 06:07 PM

[ Ignore ] [ # 162 ]

Grad Student

Total Posts: 35

Joined 07-08-2005

Does this look right?

{exp:cron minute="30" hour="*" day="*" month="*" plugin="feedgrab:FeedGrab"}
{exp:feedgrab url="my-rss" 
                          weblog="70" 
                          title="title"
                          date="pubDate"
                          use="link|description" 
                          fields="music-url|music-body" }
{/exp:cron} 

I’ve been trying to figure out how the cron plugin works with FeedGrab for the last 3 hours and I haven’t been able to find an answer. Can someone please tell me if this is right because it doesn’t seem to work?

Thanks for any help.

Profile

9 of 11

« First

‹‹ Plugin: Reeposition [Was MoveIt] Widon’t ››

Username		Remember Me?
Password		forgot password?

Post Marker Legend
	New posts	Hot Topic with new posts	New Poll	Moved Topic	Sticky topic
	No new posts	Hot Topic with no new posts	Old Poll	Closed Topic	Announcements

Visitor Statistics
The most visitors ever was 1149, on July 16, 2007 09:33 AM
Total Registered Members:	64914	Total Logged-in Users:	27
Total Topics:	81869	Total Anonymous Users:	26
Total Replies:	440138	Total Guests:	163
Total Posts:	522007