Bug 1009

Running GDB on kio_imap4 gives something like that: basically, it is not dead, it is busy inside the "addquotes" function in libkmime. It looks like there are several calls to QT functions that eats a lot of CPU:

#0  0x0000003807b5bac1 in __strlen_sse2_pminub () from /lib64/libc.so.6
#1  0x00007fb5bed9d3c4 in qstrlen (
    str=0x21d30d0 "\\\"\\\\\\\"\\\\\\\\\\\\\\\"", '\\' <repeats 15 times>, "\"", '\\' <repeats 31 times>, "\"", '\\' <repeats 63 times>, "\"", '\\' <repeats 74 times>...) at /usr/lib64/qt-3.3/include/qcstring.h:61
#2  length (this=0x218ee00) at /usr/lib64/qt-3.3/include/qcstring.h:264
#3  KMime::addQuotes (str=..., forceQuotes=false) at /usr/src/debug/kdepim/libkmime/kmime_util.cpp:474
#4  0x00007fb5beff6233 in mailAddress::getStr (this=0x218ede0) at /usr/src/debug/kdepim/kioslaves/imap4/mailaddress.cc:206
#5  0x00007fb5bf025a4d in mailHeader::getAddressStr (aList=0x2191678) at /usr/src/debug/kdepim/kioslaves/imap4/mailheader.cc:197
#6  0x00007fb5bf02615c in mailHeader::outputHeader (this=0x21914c0, useIO=...) at /usr/src/debug/kdepim/kioslaves/imap4/mailheader.cc:136
#7  0x00007fb5beff8646 in mimeHeader::outputPart (this=0x21914c0, useIO=...) at /usr/src/debug/kdepim/kioslaves/imap4/mimeheader.cc:439
#8  0x00007fb5bf0118d5 in IMAP4Protocol::get (this=0x2115920, _url=<optimized out>) at /usr/src/debug/kdepim/kioslaves/imap4/imap4.cc:370
#9  0x00007fb5c732180c in KIO::SlaveBase::dispatch (this=0x2115920, command=<optimized out>, data=...)
    at /usr/src/debug/kdelibs/kio/kio/slavebase.cpp:1048
#10 0x00007fb5beffe134 in IMAP4Protocol::dispatch (this=0x2115920, command=67, data=...) at /usr/src/debug/kdepim/kioslaves/imap4/imap4.cc:1826
#11 0x00007fb5c731ed4d in KIO::SlaveBase::dispatchLoop (this=0x2115920) at /usr/src/debug/kdelibs/kio/kio/slavebase.cpp:301
#12 0x00007fb5befff51f in kdemain (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/kdepim/kioslaves/imap4/imap4.cc:135
#13 0x0000000000408c35 in launch (argc=4, _name=<optimized out>, args=<optimized out>, cwd=0x0, envc=<optimized out>, envs=<optimized out>,
    reset_env=false, tty=0x0, avoid_loops=false, startup_id_str=0x40bac7 "0") at /usr/src/debug/kdelibs/kinit/kinit.cpp:673
#14 0x0000000000409b72 in handle_launcher_request (sock=9) at /usr/src/debug/kdelibs/kinit/kinit.cpp:1240
#15 0x000000000040a397 in handle_requests (waitForPid=0) at /usr/src/debug/kdelibs/kinit/kinit.cpp:1443
#16 0x0000000000405f90 in main (argc=5, argv=0x7fff5ed2af58, envp=0x7fff5ed2af88) at /usr/src/debug/kdelibs/kinit/kinit.cpp:1908

Look at the "addQUotes" code in TDE upstream:
http://git.trinitydesktop.org/cgit/tdepim/tree/libkmime/kmime_util.cpp

WTF, this little loop is parsing a string caracter by caracter just to add backslashes in front of other caracters !!! No surprise it eats the CPU if the string is very long !


Now look at the "addQuotes" in KDE 4 upstream (by the way, I believe that kio_imap4 is abandonned starting from KDE 4.7):
https://projects.kde.org/projects/kde/kdepimlibs/repository/revisions/master/entry/kmime/kmime_util.cpp

There's some idea here: the "needQuotes" boolean is now set using a TQRegExp.
Sadly, this regexp is evaluated inside the caracter-parsing loop, which means the Regexp is evaluated on the whole string once per caracter !! Wow, this is even worse than in TDE 3.5.13 !

Created attachment 633 [details]
kdepim 3.5.13 : optimize the "addQuotes" function to avoid insanely long response time in large Imap folders

The attached patch is a total rewrite of the "addQuotes" function which use QT optimized functions.
I just tried it and it seems that now, big folders open very fast (if not instantly).
Please test it intensively before pushing it, since it is not a trivial change.

I really want to have a STABLE imap4 support in an Email client !

I have a large IMAP folder, I will test it.
Release 3.5.13.1 yet wait...

It should interest Serghei as well, look at the Opensuse-kde3 mailing list, it has the same problem:
http://lists.opensuse.org/opensuse-kde3/2012-03/msg00010.html

There are also "removeQuots" functions at the same place that should be rewritten in the same way.

(Odpověď na komentář #4)
> It should interest Serghei as well, look at the Opensuse-kde3 mailing list, it
> has the same problem:
> http://lists.opensuse.org/opensuse-kde3/2012-03/msg00010.html
> 
> There are also "removeQuots" functions at the same place that should be
> rewritten in the same way.

Francois, you prepare a patch for the function removeQuots? Or I do not have to wait and can push to the GIT patch only for addQuotes.

I tried the patch for addQuotes on mailbox with 60000 messages and there were no problems (although the CPU was still a lot of work - but I expected it).

Hello, thanks for testing.

Please wait a little, I'm currently fixing the removeQuots functions and some other random segmentation faults in kio_imap4.
I'll tell you when it's 100% stable on my computer, then you can push to git.

Created attachment 634 [details]
kdepim 3.5.13 : optimize the "addQuotes" and "removeQuots" functions to avoid insanely long response time in large Imap folders

Updated patch for "addQuotes" and "removeQuots" functions.

Created attachment 635 [details]
kdepim 3.5.13 : fix segfault in IMAP folder parsing

The attached patch fixes some segmentation faults in IMAP folder parsing.
I don't know if it's a consquence, but sometimes, some messages are incorrectly parsed and appear with subjet "unknown", sender "unknown" and date "unknown". Clicking on the message makes it appear correctly, so it's not a major problem ..
At least, kio_imap4 does not crash anymore when parsing my large IMAP folders.

OK you can try then push the attached 2 patches, they are sufficient to make Kmail usable on my IMAP server.

I think it's the first time in history that Kmail has an usable Imap support :)

(Odpověď na komentář #9)
> OK you can try then push the attached 2 patches, they are sufficient to make
> Kmail usable on my IMAP server.
> 
> I think it's the first time in history that Kmail has an usable Imap support :)

Thank you. I'm going to test. I still have problems with set properties on imap folders that contain non-ASCII characters.

François, looks good, thank you.

Fixed in GIT hash b0c92b30 and 9b43d6e1.