Thread

Hi there,

I’m developing a series of EE websites, and part of the requirements is to integrate with various data sources - e.g. import data via an XML feed, import from a MySQL db, and extract channel data & update a system.

I’d be quite happy to write this all by hand and work directly with the database (“INSERT INTO exp_channel_data ...”), but my CEO has concerns with tightly coupling our import process to EE tables in case the structure & layout of EE db tables change in the future - the preference would be to abstract the functionality away from directly issuing insert/update statements.  So, in a way, I’m wanting to use a sort of Stand-Alone Entry Form type of scenario that abstracts me from working directly with the tables.

Any suggestions on how this will best be achieved for inserting/updating data?

For extracting data from EE, my plan is to create a series of pages that use standard EE syntax to display channel entries, however, I’ll output the result as XML so that it can be read by an external system - this way, I dont have to issue SELECT statements directly against the db & can rather rely on EE to interact with the db.

What’s the best way for me to future-proof my web-apps?

Regards
Donald

Donald,

For importing data into EE from a variety of formats I would recommend the excellent DataGrab

For extracting data from EE, my plan is to create a series of pages that use standard EE syntax to display channel entries, however, I’ll output the result as XML so that it can be read by an external system - this way, I dont have to issue SELECT statements directly against the db & can rather rely on EE to interact with the db.

Yes that’s perfect if XML is your requirement. I use it for grabbing data for Flash all the time. There is even an XML tempalte type to use.

Does that help?

Hi John,

I’ve already purchased DataGrab and have spent some time working with it, however, it lacks some functionality that I require.  For example, when it does the import, it matches imported entries to existing ones based on the Title, however, I need to match on a different field.  And, the function to import entries that are related to others also doesnt quite meet my needs - e.g. Import ‘Hotels’, and then import ‘Hotel-Rates’ and then the availability periods for each of the Rates, etc, with the data possibly coming from a single XML feed.  It might work if I could freeze the titles on my side and then parse the feed multiple times, but expecting titles to never change on my side is optimistic, and if a perfect match isnt found then I’ll end up with duplicates - and, titles could change in the Source system which would also lead to dups - and, using some sort of ID as a Title will make maintenance of the entries complicated and prone to error for the end user. 

I did contact Andrew, the developer of DataGrab, but the release date of an update that might/might-not address some of my needs isnt firm ... its how these things go!

I also looked into using feed parser components (including Andrew’s one), but ran into problems whereby I needed to have certain functionality that was not supported by the feed parsers, and a module I created with the functionality was being called before values were being parsed, and so I was ending up with EE tags like {example-field} instead of actual values.

From your response, in terms of pointing me to DataGrab, I take it that I’ll either need to tweak DataGrab to the hilt, or write my own import process with the additional functionality - but, from a vanilla Expression Engine point of view there isnt much to go on?  Right?

Regards
Donald

Hi Donald,

Welcome to the forums!

I’m not sure what you mean when you suggest that there isn’t much to go on from a vanilla ExpressionEngine point of view. Can you clarify that?

It does seem as though you have very specific requirements that mean you would probably be better off developing your own modules within EE to manage this data.

Cheers

Greg

Hi Greg - ok, I’ll go the custom route and develop my own modules.

Many thanks

Donald

Sounds good, Donald. If you run into any issues, feel free to post to our Development and Programming forum.