Huge database

I manually entered the cities in the db. I’ve populated the exp_channel_titles and exp_channel_data and now when I try to go to edit php memory (256MB) gets exhausted…..

Does EE make a query like “Select * from exp_channel_titles” without LIMIT?

If you’re just storing city names I would recommend creating a unique table (or more) just to store the data. That way you can limit that table’s schema to just support the necessary field length and not waste space in the channel_titles and channel_data.

Does EE make a query like “Select * from exp_channel_titles” without LIMIT?

Not that I’m aware. You can enable the output profiler to see each query getting run though. You can find that under Admin > System Preferences > Output and Debugging Preferences.

Thnx for the answer. If I do this I need to make a custom fieldtype in order to show the data in the channel edit?

That’s correct. And since it’s a lot of data you can get pretty creative in how you design the interface through which the data is selected and displayed.

With this approach you can also easily create a template tag for select options etc.

Hi w84me,

Did you complete this project by manually adding 3 million entries into exp_channel_data and exp_channel_titles ? Or, did you use something like Data Grab?

We need to import 500k entries into EE. We are having problems with Data Grab (won’t do more than 300 at a time) and we are thinking of editing the database manually.

Any advice?

Thanks!

I’ve entered the data manually from command line. I haven’t tried Data Grab but raw SQL is always faster and more reliable.

We had to populate a database with 400K entries. Luckily we had the data available as a CSV file so I used the MySQL INFILE to load it up. One issue we encountered was making allocating enough disk space.