Garbled emails and overriding Email class for multibyte characters
Development and Programming
Emails being sent from EE often have garbled (mojibake) text in the email “from”, “reply to”, “subject”, attachment filename, and “body” when there are, or are more than a few multibyte characters in them. Here’s an example:
From: =e3=83=8b=e3=82=b3=e3=83=a9=e3=83=9c=e3=82=bf=e3=83=aa=e3=81 Subject: =e3=80=90=e3=82=af=e3=83=ac=e3=83=91=e3=82=b9
The above has email settings set to utf-8. In Japan, systems are old enough that 7bit encoding, such as iso-2022-jp, must be used to assure minimum garbled text. Changing the encoding in the CP’s Email settings only changes the type of garbled text I get :(
Up to now, partly fixing email for Japanese has required making changes to core EE files (EE 1.x) and CI Files (Email.php: EE 2.x). For example, some data for email elements, such as “from”, need to be passed through the mb_convert_encoding function to show multibyte characters correctly. This is what had to be done in EE 1.x, and what appears to have to be done in EE 2.x. (Eg: See this thread in Japanese on changing core files for EE 1.x)
I was wondering if there was a way to override the CI_Email Class without having to play in core files, which ideally I shouldn’t be doing anyway? If this is possible, I know that the EE_Email Class already extends the CI_Email class, so which class should I be overriding? Can either/both be overridden?
In any case, some email elements need to be passed through multibyte functions to avoid garbled text, so instead of overriding/hacking core files, a fix in the next EE version would be another welcome solution 😊
Moved to Development and Programming by Moderator
By the way, Japanese CI developers often seem to simply extend the CI_Email class in their applications to solve some (but not all) garbled text problems. How would this work in the context of ExpressionEngine, which is the application that already extends the CI_Email class?
HI Nicolas,
ExpressionEngine 2 is UTF-8 only.
You can extend the ExpressionEngine Mail class. If you wish to do so we can move this thread to the codeshare corner where our community can assist.
I would also recommend a feature request for the multibyte suggestion.
Regards, Barry
EE2 itself is UTF-8, yes, and I am thankful for it 😊 . But email is another beast, with its own encoding problems.
The garbled text problem may be similar to what is being asked in this thread. Special characters in emails are being displayed incorrectly when sent from EE. Doesn’t look like it’s been solved there either.
An CI or EE Email class override could solve some of these problems, but I haven’t figured out how to make this work yet. If you think the Codeshare Corner is where this should be, please go ahead and move the thread. However, I also think this might fall under technical support. 😊
If overriding isn’t going to work, then I’ll file a feature request. This needs to be fixed.
Hi, Nicolas. I am going to move this, but not to the CodeShare Corner. Instead, I’m moving it to the Development forum, since you mentioned extending CI or EE email classes.
Several months late, but what the hell… 😊
Nick, if I can be of help, please let me know - because this monster is still causing me major headaches. The problem, as you are getting to is the CI core. This same exact problem occurs in HalogyCMS as well and I’m about at wits end trying to get around it. I mean, its IDENTICAL output from Halogy as well. Garbled up Subject, Title, and From fields.
I really have no idea what the side effects of this move are, but I think this is interesting…
Go to codeigniter/system/libraries/Email.php (I’m on EE2.1.3 build 20110411 so the line numbers might only be appropriate there…)
comment out the _prep_q_encoding lines (Email.php line 200 & line 363) -you should find at least two of em - one is in the else statement in //prepare the display name and the other is in the function for setting the subject.
and now my from and subject fields are no longer garbled up.
If you see this post, please let me know your thoughts.
Cheers from Tokyo.
Thanks for the tip, I’ll try it out. Are attachments also working correctly, for example with filenames in Japanese only?
Based on what was mentioned elsewhere on Japanese CI sites, I would usually make some other modifications to the core files (change encoding, add PHP mb_ functions), in particular in codeigniter/system/libraries/Email.php (as you mentioned). However, I would always get something garbled, either the attachment name, subject, sender name… you name it.
After discussions at CICON Japan 2011 earlier this year, it seems we’re not alone with this problem. Most agreed that CI has problems with Japanese (maybe even multibyte in general) text in emails. Some even say that’s why CI isn’t growing as fast as it could in Asia. The other side of that is that Japan, in some cases, is lagging behind when it comes to web standards (they are still the fourth biggest IE6 users in the world as of this writing!).
From what I gathered in Japanese CI forums and sites, this CI core Email.php file needs to be extended to non-destructively modify CI emailing settings and functions. EE already extends the CI Email class, but can this be extended again to add what we need, without playing in core files?
Hope all is well for you in Tokyo, hope to visit again soon ^_^
I was looking for the mb_encoding bits as well earlier, but I think there have been many changes since EE1, so many of the old Japanese threads are no longer so easy to follow.
However, I’ve seen several mentions of problems related to the use of prep_q_encoding so it was a shot in the dark. But so far, it works. I’ve yet to test it for direct mailing to mobile phones and windows machines (you know bout IE6 and all that…)
Like you mentioned before - hacking core files sucks. Its dangerous and easily broken upon the next upgrade that comes down the pipe. The good thing is that what I did above only required two hash characters, and as such, is not a difficult task to repeat if needed. In fact, I think I’ll go try that out in HalogyCMS as well and see how it acts.
Sounds good, let me know how this works out. Gambatte!
a further follow up - looks like my previous post only works reliably with apple mail - and then, only on the desktop. In iOS the from and subject fields are still horribly mangled. I should have guessed it wouldnt be THAT easy…
Back to the drawing board!
Nick,
I have a new edit for you to try out. I came across an alternate hack for the prep q encoding method that is used in CI’s email.php.
I replaced the existing Prep Q Encoding with the version found here: http://pastie.org/private/hvfpdcai6kw4j69yc0ktg
and the subject line is no longer garbled up with =38=82=23= blah blah blah…
The only strange thing is that this seems to insert a space after every 7 or 8 multibyte characters. I think that this has to do with the 76 character limit for $wrapchars, but I’m not exactly sure about that.
If you give this a try, please post and let me know your findings.
Cheers,
Matthew.