Developement & Source Code Management (SCM) with ExpressionEngine

Nicholas,

EE is an odd beast indeed. It’s pretty annoying that the software alters the database schema in various ways.

We use multiple environments and SCM (Git in our case) in all of our projects. Using a plump config.php file we’re able to keep paths in one place and not requiring updating in each environment.

As for the database changes to things like new templates, new channels and custom fields etc - there currently is no golden bullet. I can, however, tell you what we (and many others) do.

For series of channel additions, templates additions etc we simply migrate the data using Navicat’s data transfer. For custom field additions (since it modifies the exp_channel_data table) we manually create the fields in each environment. It’s annoying, but it’s the most efficient solution we’ve found from project to project.

Essentially we’ll open up each environment control panel in our browser tabs and flip between tabs copy/pasting the new field information in the same order so that the exp_channel_data tables are all identical.

The word on the street is that some add-on developers are working on a solution that should help this. I imagine it will involve some query logging and environment-aware settings. Until then, we’re pretty sold on our approach. It’s the best we’ve found for our work flow.

Feel free to ask more specific questions if I’ve left out some details you’re interested in.

Erik

Erik, thank you for your reply. It was very helpful.

Once I understand the EE database schema better, it sounds easy enough then to export an sql patch to update the channels/templates etc on the production server.

Looks like it is a matter of documenting the procedure as the changes go in and manually adding them via admin as suggested.

Thanks again for your assistance with this. Much appreciated.

Nicholas

Glad to help, Nicholas 😊

Looks like it is a matter of documenting the procedure as the changes go in and manually adding them via admin as suggested.

I’m not sure if this is what you mean - but for what it’s worth we (at one point) tried documenting our changes locally and then executing them in bulk (manually) in the other environments. (This is as opposed to making each change at the same point in time across each environment.) We found that to be more prone to user error than we liked.

One issue with order of channel creation can cause a lot of havoc in your system if you’re not careful. So, just be careful with that approach.

+1 thanks for the good info, erik.

ezanol,

Glad to help. Let me know if you have any questions about our process (not that it’s perfect though)

A process has emerged at my workplace with an uncanny similarity to Erik’s process. We use git for version control, a method to relative-ize the config.php, and Navicat to migrate databases.

Another thing I’ll mention is to communicate to the client SOME of the details of the development process. I routinely ask clients to stay out of the CP while I’m developing. This effectively “freezes” the production DB so I can copy it back and forth during development without fear of things getting out of sync. Note: this only works on smaller projects that have no Forums, commenting, or other forms of user participation. Smarter clients are even allowed access to a testing server to approve of changes before deploying to production. Though technical, the right clients will appreciate the advantages of the process. It also forces you to keep development concise and on schedule. There are many advantages to the process, though it is sometimes laborious.

Another thing I’ll mention is to communicate to the client SOME of the details of the development process. I routinely ask clients to stay out of the CP while I’m developing. This effectively “freezes” the production DB so I can copy it back and forth during development without fear of things getting out of sync.

We were originally doing the same thing. We would set a scheduled maintenance time where we’d request they not log in to the CP. We had one incident where they forgot and it messed a few things up because of it. That’s when we whipped up a small add-on called CP Pad Lock that effectively locked all member groups out of the CP but left the system on.

Technically you can achieve this by modifying permissions in each member group but when you have multiple groups accessing the CP a single “flip of the switch” is much easier.

As you alluded, the challenge is when there is ANY front-end user input like commenting, forums, SAEF or ecommerce purchases. We, too, use Navicat for data transfer and for each project we save a “profile” for data transfer letting us ignore tables we should never touch on production. It helps, but certainly isn’t perfect.

For sourcecode management i save a DB dump to my SVN repository. Not very granular, but it works. Your table-specific dumps sound better, but only if you definitely know what’s changed in yr db. How do people SVN databases??

As far as updating different environments, i use the (free) Reelocate addon…

http://devot-ee.com/add-ons/reelocate/

…then export the database to the new environment. Then ftp all changed files to your new environment server. Or, if you can, svn-update on your new server, that’s pleasingly effortless 😊

Theoretically if one were to use flat files instead of making changes to the db for templates, the only changes in the db that would need to be tracked are changes to ee settings? Then dumping the sql files and staging/committing those changes between edits.

Has anyone ever tracked changes to EE’s db via GIT or SVN? It sounds like everyone tracks changes via GIT and then manually copies/pastes their db changes.

I’ve read the below pdf posted from another article, but again, he mentions there being no silver bullet to the problem of tracking changes to EE’s db.

PDF: http://gititon.masugadesign.com/gititon_eeci2010.pdf

@chichilatte: Is my proposed method the same as what you’ve mentioned? Use flat files > Dump SQL > Version control > sync db either using dump or some type of addon to migrate the db to another location.

I’m reading this post with a level of growing trepidation. I started hunting for documentation about how the regular EE system upgrades affect the database schema, but this appears to indicate that schema updates also happen during normal use.

Is this documented anywhere?

Or, is this a case of records being added to the database with the schema staying the same?

Background: I’m working on deploying EE inside Docker in Elastic Beanstalk and am attempting to determine how best to deploy the database.