Thread

What would be the best way to display data from varying sources and have them be sortable via time/date?

We need to have one page that lists items from 3 different data sources:

1. data from a 3rd party database with entries updated every 5 minutes.

2. twitter feed (*this one is currently optional but needs integrating eventually)

3. Expression Engine specific channel entries.

And have them be sorted via date & time.

At first, we’ve considered using Solspace’s Importer, and run a cron job at a high frequency (maybe once every 5-10 minutes) and import items #1 and #2 into channel #3, and they’d (presumably) all be organized within EE’s date/time in one nice place and easily sortable via EE’s channel module capabilities. However, that seems like a major performance risk. Plus, the database in #1 is mammoth, and wouldn’t Solspace’s Importer always be attempting to import the entire database each time? Can it be limited to only importing new entries?

We don’t even have a clear idea what would be the best approach at a very high-level before even planning the details of a solution.

Any suggestions for the approach we should take?

I am not sure what would be best for performance, i guess having a cron job running so frequently may have an effect.
I know that Importer by Solspace can check for duplicates, you would do these by having an identifier which can check against the current entries.

I use Importer to grab my twitter feed into a channel every 30mins.