Thread

Andrew Weaver is updating his FeedGrab, CSVGrab, and XMLGrab into a new module, DataGrab, for EE 2.0. I tested it with Tumblr and it works, but not as elegantly as I would like. The XML that Tumblr produces really needs it’s own plugin/module to work with EE since there’s different types of posts, eg. text, photos, videos, etc.

DataGrab assumes that each entry is the same. I imported my Tumblr XML individually by type using the address http://your.tumblr.com/api/read?type=, but this was only a workaround and I don’t really recommend it.

Most obvious flaws of using this module with Tumblr are:
* No way of differentiating between post types.
* Data is collected from the most recent post and then assumes all others have the same attributes. ex. If your most recent text post doesn’t have a title, then the XML attribute [regular-title] will not show up.
* The import is saved, but must be manually refreshed.

DataGrab is a nice module to have if you plan to import identical XML data once or weekly and so on. However, for my intention of grabbing my Tumblr posts it doesn’t quite work. Hopefully someone more knowledgeable than I can create a plugin for this task.

http://brandnewbox.co.uk/articles/details/datagrab_for_ee2.0

Darn, that’s disappointing. Importing Tumblr is exactly what I’m planning on doing with DataGrab.

It sounds like your workaround by type was able to grab different feeds though.

What are the limitations of this method?

DataGrab successfully grabs the feeds, but then you’re left with seven saved imports that you have to manually refresh. The biggest problem is that most post types don’t have titles, and even those that do the title is optional. You’ll end up with many channel entries with blank titles. However, you could use the post’s slug, ex. http://your.tumblr.com/post/123456789/this-text-is-the-slug, but alas, you’re left with a URL-friendly formatted title.

Don’t get me wrong, DataGrab can potentially do what you want, but I’d really like to see something more custom-made. Perhaps Andrew could create a TumblrGrab. Most of the work has already been done. It just needs tweaked to recognize the differences in the Tumblr posts and to auto-refresh.

Oh, I see. I take it you are using the post’s slug as an entry title for more than just internal organization? i.e. the title is visibly displayed on your site? What about adding the title yourself? Is there a way to add a custom title using DataGrab?

My plan is to import a local copy of Tumblr’s posts into separate weblogs/channels for each type anyway. I want to recreate Tumblr’s types in my own EE-powered DB, so using different feeds is actually a part of the idea. I wasn’t planning on using the title for anything but internal organization in EE.

Auto-refresh would be nice—could a cron job be run to automatically import the data? Ideally, I’d like to import it into a queue for approval / dismissal… That could be done with a custom status, right?

Thanks for sharing info on your setup! Very helpful to know what can and cannot be done with DataGrab.

There’s currently no way to add anything custom at all using DataGrab, at least not as I could tell. Everything is handled with non-editable drop-downs to select your data. Although, now that I think about it, perhaps I could live with having the slug as the title. I could use an if statement to only show the titles of text/regular posts.

I set up Categories using the types of posts offered by Tumblr: text, quote, photo, link, chat, video, and audio. Is there a reason you’re wanting to have separate channels for them?

Oh! I was under the impression it would be possible to import data using DataGrab directly into a customized EE entry… I guess I should try the plugin to see how it actually works 😊

Categories is a snazzy idea! I’m going to think about the implications of that… So you are able to import the data as entries into a single EE channel, and assign a category based on which feed the data came from?

I was going to use different channels for each feed type because each Tumblr types really uses different fields. There are some overlaps, like “caption”, but even so, I would like have a “photo caption” kept distinct from a “link caption”, let’s say… I also want to import data from Delicious into my Link channel, photos from Flickr into my Photos feed, etc.

The general idea is to create a centralized hub for different types of media, kept distinct by type, regardless of source. I’m planning on both entering new entries manually, and importing feeds to automatically populate new entries. That way, all entry types will be entered in the EE DB in their respective channel, independently of which feed they came from originally, or are hosted. My design will pull from these channels in various ways to present the content based on type.

Is there a potential limitation or headache to using several channels to organize the data this way?

I’ll explain a little better of what I did. It seems we’re trying to accomplish the same goal. I’ve played around a little more and I’m pretty satisfied with my results thus far. I’m still hoping for an automatic refresh using cron, however.

1. Created a Channel, Categories Group, and Custom Field Group, all named Tumblelog.

2. Created seven categories for each type of Tumblr posts: text, quote, photo, link, chat, video, and audio.

3. Created custom fields for each Tumblr post attribute. regular-title, regular-body, link-text, etc. The list can be found here: http://www.tumblr.com/docs/en/api . I named each field the same as the corresponding attribute. I used textareas for regular-body, quote-text, conversation-text, video player, and audio player. The rest I used text inputs, but increased the size limit just in case of a really long URL.

4. Used DataGrab to import the xml. For Filename put: http://your.tumblr.com/api/read?type=. For Path put: /tumblr/posts/post. For each import, add the post type in the URL after ?type= in the Filename. ex. http://your.tumblr.com/api/read?type=text for text posts, etc.

5. During the import if you named your custom fields the same as the Tumblr attribute, then all you need to do is match them. For the Title I used the slug. If you use Categories, use the type attribute and be sure to select your Category Group in the dropdown, or else you’ll have to manually choose them later. I forgot to do this with my first import.

6. That’s it.

This in no way is the definitive guide of properly importing your Tumblr posts using DataGrab. I’m far from being an expert and if anyone has better suggestions, I’d love to hear them.

Hi Shawn,

Thanks for mentioning DataGrab in your posts.

I developed the DataGrab module so that it should be relatively easy to extend it to import other kinds of data and I’m looking for a few examples to develop and maybe use as documentation.

Would you be interested in a specific Tumblr import? I can see how the existing XML import type is too simplistic, so this could be an interesting example.

Andrew

Andrew,

for some time now I’ve had a personal project on hold where I want to merge 2-3 EE core sites I own and then move them into a MSM run site on a personal license. Would this feed grab module be able to help with that. The sites have well over 500 entries in multiple weblog/channels…

Andrew, I think having import options for specific sites, especially Tumblr, would be great. Unlike men, not all XML is created equal. Maybe you could create tiny plugins for this. Users could then download for the sites they need. Thanks.

Hi Shawn,

Thanks for your reply. Realistically, it will be a couple of weeks before I can start on this, but a Tumblr import seems like it might be a good ‘plugin’. Can I ask for your help when for requirements and testing?

Andrew

Hi Sean,

DataGrab could partially help. You’d still need to set up the old sites to export the data in XML/CSV format, and you’d need to set up the channels and custom field etc in the new site.

DataGrab would then be able to import the data into the new sites (with the caveat that it does not currently handle comments).

Andrew

Andrew,

for some time now I’ve had a personal project on hold where I want to merge 2-3 EE core sites I own and then move them into a MSM run site on a personal license. Would this feed grab module be able to help with that. The sites have well over 500 entries in multiple weblog/channels…

Andrew, I’ll gladly help test when it’s ready. Just shoot me an email. Thanks again for your interest. I’m sure many others will appreciate it as well.

Hi Andrew,

I would gladly test a Tumblr import plugin as well! That would be fantastically useful.

Please let me know if you need any help!