this is easy to test when copy paste data from one resource to another.
I'll test the fix proposed by Fat-Zer next and report back.
If successful we can patch and push.

regards

I didn't have to do anything to test the proposed patch.
This is another example for theoretical work, that does not work :)

When recompiled and deployed the new tdelib with the fix, after restarting when the addressbook was open it did show double coded characters for all contacts.

Worse it saved couple of copies of the address book in the famous __[n] style

It turns out the function was called when opening the address book and mangled all non ASCII chars.

I've noticed another possible issue with there in the code. When I have some better fix I'll be back.

Thanks to Fat-Zer however for pointing the relevant place

regards

If I understand correctly, we need to push Fat-Zer patch as it is, right?

diff --git a/tdeabc/vcardparser/vcardparser.cpp
b/tdeabc/vcardparser/vcardparser.cpp
index 7ac07ce..db33263 100644
--- a/tdeabc/vcardparser/vcardparser.cpp
+++ b/tdeabc/vcardparser/vcardparser.cpp
@@ -152,7 +152,7 @@ VCard::List VCardParser::parseVCards( const TQString& text )
             KCodecs::quotedPrintableDecode( input, output );
           }
         } else {
-          output = TQCString(value.latin1());
+          output = TQCString(value.utf8());
         }

         if ( params.findIndex( "charset" ) != -1 ) { // have to
convert the data

By comment 2, I understood that the proposed patch at the moment brings more complications => double encoding of characters.

ah... this morning I must have been sleeping. I understood that exact opposite! :-(
So we wait until deloptes comes up with a working patch. Sorry for the noise.

Created attachment 2658 [details]
patch for tdelibs-tdeabc utf8 support


Fixed issues with non Utf8 support in the addressbook 
   vcardparser
   tools
   vcardformatplugin
   converter.createVCards
   migrated testing support from the Makefaile.am in to the CMake
   additional test programs
   resourcefile - improved the backup read/write
   
The patch by Fat-Zer also included.

Created attachment 2659 [details]
Patch by Slavek to add testing into tdeabc

This patch is prereq for 2658 - added here for completeness

From Emanoil comments on email:

Proper way to apply is:
1. Slavek patch from comment 7
2. my patch from comment 6

I will try to test it on my systems before push.

(In reply to Slávek Banko from comment #9)
> I will try to test it on my systems before push.

I have the first test result: virtually all of my contacts are now broken - it seems to double utf8 transcoding.

Detailed information: I have contacts in sync via CardDAV to our SOGo server. Fortunately broken contacts were not updated back to the SOGo server == contacts on my phone was not destroyed.

>I have the first test result: virtually all of my contacts are now broken - it >seems to double utf8 transcoding.
Slavek,
perhaps you can provide some test sample for Emanoil, so it can work on a fix and verify against his test cases and yours. Otherwise we could see this scenario repeat over and over :-)

This was the case for me as well before digging deeper into it.
It might be you need to update the Dav code - if it was relying on the broken tdeabc and one has "improved" only on the client side. I would guess that it is converting to utf before it goes to the Cal/CardDav.
It could be that I added somewhere something wrong.
The tests are really good for that, but they also rely on assumptions.

Let me know if I can help or learn something.

regards

The problem is most likely caused in

 tdepim/tderesources/carddav/resource.cpp

by

 Addressee::List addressees = conv.parseVCards( data.utf8() );

at line 430, 459

While it is defined as
 static VCard::List parseVCards( const TQString& text );

so it should be simply

 Addressee::List addressees = conv.parseVCards( data );

utf8() returns TQCString and this breaks it I guess.

You can test this by changing it in testread.cpp and running the test.sh

It is worth checking other resources that use parseVCards. This was part of the big problem.

regards

Created attachment 2660 [details]
tdepim : Fix utf8 handling in CardDAV

Emanoil, excellent analysis - patch created by your remarks works perfectly! CardDAV is now back in order.

Created attachment 2661 [details]
tdepim : Fix utf8 handling in tdeabc resource cache

Additional patch related to CardDAV and generally all external resources - fixed utf8 handling for reading vCards from local cache.

Glad to hear! I think we did a very good work, looking into the patches it all makes sense.
You can not imagine how glad I am to have cleaned up all this. I've been having issues since the time of opensync ~ 2005. Fortunately back then I had only phones using iso8859-15, but since couple of years it became a real pain and this was the only functionality I was desperately missing in TDE. I'm just glad that this year I am not overloaded as before. Lets hope this keeps constant.

Thank you all for helping and guiding.

regards

Created attachment 2662 [details]
tdepim : Fix utf8 handling in kaddressbook thumbnailcreator

The last patch related to parseVCards - for thumbnails and previews of vCard files.