Help with wrong characters being displayed

[Paddy Murphy]

Hi,

I have numerous characters coming through to my forum via RSS Feed Manager that are wrong.

For example;

' comes through as ' (I actually fixed this one by placing ' in the Replacement Variables)

But please help with these;

£ comes through as ?
“£ comes through as ??
“ comes through as ?

They're just ones I've noticed.

Is there a work around for this issue?

Thank you

[Lynne]

If you look at the feed and see the page source, is there an encoding listed? And is it the same encoding used on your site?

[Paddy Murphy]

Hi Lynne,

Thanks for the reply.

I went into great detail about this issue on the vBulletin forum here (http://www.vbulletin.com/forum/forum/vbulletin-4/vbulletin-4-questions-problems-and-troubleshooting/4258259-is-coming-through-as-39-via-feed-source).

Would you mind taking a look there to see the issues I've been having please? I hope it will give you a better insight to my issue, if not please let me know what other information you need. I've since gotten new Google Alert Feed url's.

Here's one for example - https://www.google.ie/alerts/feeds/00705490796826983622/16708823403621179732

I'm then putting these feeds through feedburner and they are being posted here (http://www.forum.norml.ie/forumdisplay.php?20-Ireland-News-Articles) on my forum.

Regards

[Lynne]

My 4.2.2 forum doesn't like that feed at all. I then went to the google alerts page and got a new RSS feed URL, and my site doesn't like it either. So, it may just be that those feeds can't be read by vbulletin.

[Paddy Murphy]

I don't use that feed directly. I put that feed through feedburner to give me this - http://feeds.feedburner.com/GoogleAlert-Cannabis

I just noticed, the error characters in this link (shown as squares) are the ones that are showing on my forum as ? (a question mark).

[Lynne]

So it isn't just the characters from the feed showing up wrong, it is those particular characters being used anywhere on your site that are showing up wrong?

In Language Manager > edit main language > what is your Language Code and HTML Character Set?

And in MySQL, what is the collation of the vbulletin tables?

[Paddy Murphy]

Well in this feed that I'm using for example (http://feeds.feedburner.com/GoogleAlert-Cannabis) some characters like the € (Euro symbol) is shown as a square. And when it then gets posted to my forum via RSS manager, the square is displayed as a ? (question mark). This happens with other characters to like " also.

Language Code = en

HTML Character Set = ISO-8859-1

In phpMyAdmin most of my database tables are set to latin1_swedish_ci

I changed some of the tables (post, thread etc) to utf8_general_ci to see if this would fix the issue but it did not.

[Lynne]

If it is showing incorrectly in the feed, then it certainly isn't going to correct itself in a post on your forum.

I would strongly recommend NOT mixing your table collations like that. You are going to cause yourself problems (not to mention that for utf8_general_ci, it is best to have the HTML Character Set be UTF-8).

[Paddy Murphy]

If it is showing incorrectly in the feed, then it certainly isn't going to correct itself in a post on your forum.

Ok. I think it's time to seek an alternative to Google Alerts. If I use the Google Alert (https://www.google.ie/alerts/feeds/00705490796826983622/7168298089616524864) directly I get this error;

XML Error: not well-formed (invalid token) at line 1

I would strongly recommend NOT mixing your table collations like that. You are going to cause yourself problems (not to mention that for utf8_general_ci, it is best to have the HTML Character Set be UTF-8).

Ok. How do I change them all to utf8_general_ci ?

Thanks

[Lynne]

I thought you had already changed some table collations?

Step 6 here explains it - http://www.vbulletin.com/forum/forum/vbulletin-3-8/vbulletin-3-8-questions-problems-and-troubleshooting/vbulletin-quick-tips-and-customizations/282710-language-charset-problems-things-to-check?t=275958&highlight=collation

NOTE: It is very very very important to make full database backups before proceeding if you are ever manually making any changes to your database.

[Paddy Murphy]

Yes I've changed about 20 of the tables. But there's over 200 more, I'd prefer not to do each one by one. Is there a way I can change all the tales in one go? A query ?

I know how to back up my forum database :-)

[Lynne]

I would think you would need to write a script if you want to do all the tables in one go.

[Paddy Murphy]

I would think you would need to write a script if you want to do all the tables in one go.
Ok. Is there a guide to do that?

[ozzy47]

If you know PHP programming, you should be able to knock something up.

[Paddy Murphy]

If you know PHP programming, you should be able to knock something up.
I don't. Can you point me to some guide or information on doing this please?

[Lynne]

You want a guide on how to write PHP code? I love w3schools for reference.

http://www.w3schools.com/php/default.asp

[Paddy Murphy]

Ok, all my database tables are now set to utf8_general_ci

I thought this would solve my problem but it has not.

When I set the HTML Character Set to UTF-8 I get squares for some of the characters (see attachment)

When I set the HTML Character Set to ISO-8859-1 I get � for some of the characters (see attachment)

See attachment for original characters in source article.

[kh99]

What are you using now for a feed? I tried a couple of the links you posted earlier but they didn't seem to be working.

[Paddy Murphy]

What are you using now for a feed? I tried a couple of the links you posted earlier but they didn't seem to be working.
Hi,

http://feeds.feedburner.com/google/qYjc

Thanks

[kh99]

Hi,

http://feeds.feedburner.com/google/qYjc

Thanks

Hmm...for some reason that seems to post only the article titles with a "More..." link, and all the titles seem to look OK.

[Paddy Murphy]

I'm using;

{feed:content:encoded}

Click here to read the full story... ({feed:link})

For the Body Template.

I should point out that's not just feeds I having this issue with. Since I changed the HTML Character Set to UTF-8 from ISO-8859-1 it is also effecting usernames that contain a ?

See - http://www.forum.norml.ie/member.php?11

The username should be Ois?n Mac Giolla Bhr?de but it is being displayed as Ois�n Mac Giolla Bhr�de

[kh99]

OK, well, the only problem I saw in that feed is "It& # 39;s" that's supposed to be "It's". But when I look at the feed xml I see this:
It's time to reconsider the regulation of marijuana and <b>hemp</b>.

An if you imagine decoding the characters, the &lt;b&gt;hemp&lt;/b&gt; later in the line becomes <b>hemp</b>, which correctly makes the word hemp bold in the post. But if you decode It&amp;#39;s you get It& # 39;s, which is what you see in the post. So it seems to me it's a problem in the feed, and I don't really see a solution to it (well, except maybe if you hacked the code to decode the post twice, but I'm not sure that wouldn't cause other issues).

Edit: BTW, I added some spaces in the above to prevent the codes from being interpreted.

[Paddy Murphy]

I was able to fix the apostrophe issue with a replacement variable - https://vborg.vbsupport.ru/external/2015/02/16.jpg

I don't mind the bold words so much. It's the ? & ? characters being displayed as boxes that I'd like to sort out.

[kh99]

Oh, OK, sorry. I remember seeing something earlier about replacement variables but I didn't pay enough attention.

I looked in to the other issue a little when you posted on vbulletin.com and I found that the feed seemed to be an ISO-8859-1 encoded document, with an xml tag that said it was UTF-8, and I think the rss feed poster code wasn't prepared to handle that. I would look in to it more but I didn't see those chars in the feed I was looking at.

[Lynne]

Try changing this in your config.php file and uncomment this line.

// $config['Mysqli']['charset'] = 'utf8';
so it looks like:
$config['Mysqli']['charset'] = 'utf8';

[Paddy Murphy]

ozzy47, kh99 & Lynne thank you very much for your help with this issue.

I'm not sure why but for the last 4/5 days everything seems to be running smoothly. Maybe the database took a day or two to recognise the that the tables were changed to utf8_general_ci. So feeds are good.

The characters in the usernames were ok after I edited them from the box to the correct character.

Thanks again :)

[ozzy47]

Well glad to hear all is well now. :)