Bug 699

Further info: I thought the problem might be associated with tree view, which I use. Setting the view to icons resulted in the same slow response to list /usr/lib.

Looks like this has been common problem, but possibly fixed in KDE4:

http://aseigo.blogspot.com/2011/08/shout-out-to-dolphin.html

The xfe file manager is lightning fast. :(

FYI: KDE4 has this improvement in both Dolphin and Konqueror. Very nice not to
have to wait a long time. Perhaps some code snippets can be massaged from KDE4.

I would like to see this report receive attention in R14.0.1. I'm embarrassed to show other people my desktop when browsing large directories. :-)

Much related, konqueror does not use the linux kernel cache. For example, in konsole run ls -R /usr/lib then open konqueror to the /usr/lib. Konqueror will take a very long time just ro display the top level of /usr/lib.

Of late I have been required to test other desktop environments (job related), which includes testing other file managers. While most other file managers are feature-impotent compared to konqueror, all of them put konqueror to shame with respect to listing large directories.

The king of the hill remains xfe with respect to large directories. If xfe had support for tabs I would switch to xfe. I find the slowness in konqueror with respect to large directories to be embarrasing. This singular paper cut creates an awful impression of Trinity, despite so many other wonderful improvements.

On my system xfe lists /usr/lib in less than 2 seconds, whereas konqueror takes almost a half minute.

I hope this bug becomes a priority post R14.0.0.

After playing with bindfs in debug mode I think I understand why this is happening.  You will probably notice the worst Konqueror speed when loading a directory containing mostly subdirectories.

As you stated above Konqueror is extremely feature-rich.  Unfortunately one feature in particular (directory icons) needs to check for and read a .directory file for each and every subdirectory; this massively inflates the file system operation count and likely causes this bug.  There is no getting around the total load time requirements before all Konqueror features will become operational, but there might be a way to make the load sequence more palatable by showing a basic directory list as soon as possible after directory load start.

Ideally the file tdeioslaves should return a directory list to the parent application (e.g. Konqueror), then go back and asynchronously read metadata (such as the .directory files), updating the displayed icons only if needed.  It looks like an attempt was made to fix this in the following KDE 4 bug report and commit:
https://bugs.kde.org/show_bug.cgi?id=178678
https://projects.kde.org/projects/kde/kdelibs/repository/revisions/6369b556ae9beef6888699d23b91326bac950ba4/diff

I don't know how effective it actually was, and directly backporting it alost certainly won't work, but the idea seems sound enough to try for a proper implementation in TDE.

As mentioned, KDE4 dolphin and konqueror no longer have the problem. The xfe file manager does not have the problem and is faster than KDE4 dolphin and konqueror. The MATE Caja file manager is faster than TDE konqueror although not as fast as xfe. TDE konqueror does not use the linux caching system, which seems a shame (comment 4). Studying the code base of those file managers would help.

(In reply to Darrell from comment #7)
> As mentioned, KDE4 dolphin and konqueror no longer have the problem. The xfe
> file manager does not have the problem and is faster than KDE4 dolphin and
> konqueror. The MATE Caja file manager is faster than TDE konqueror although
> not as fast as xfe. TDE konqueror does not use the linux caching system,
> which seems a shame (comment 4). Studying the code base of those file
> managers would help.

KDE4 Dolphin and Konqueror do not have the large feature set of TDE Konqueror, therefore the comparison may or many not have any relevance.  As far as the Linux cache is concerned, it should be transparent.  Do you have a link to material stating that we have to add support for it to our code for it to work?

It would seem far more likely is that the extra file accesses are uncached (due to either overrunning/polluting the cache or repeatedly accessing files that do not exist), and that KDE4 either removed the offending feature or made it asynchronous so that you do not notice the slower load.

Tim

>KDE4 Dolphin and Konqueror do not have the large feature set of TDE Konqueror,
>therefore the comparison may or many not have any relevance.
I was referring to comment 1 and comment 2. The KDE devs did something to improve performance and the improvements are noticeable when using KDE dolphin/konqueror. Looking at their patches might help TDE konqueror and won't hurt.

>As far as the Linux cache is concerned, it should be transparent.
>Do you have a link to material stating that we have to add support
>for it to our code for it to work?
My focus is comment 4. I have no idea whether file managers should use kernel caching, but seems odd that I can perform an ls in konsole and then immediately in konqueror not observe an almost instantaneous display of the same directory. Seems at least TDE apps should "talk to one another." Perhaps I expect too much.

The KDE 4 patches were a red herring--most of them were to fix regressions introduced in KDE 4 itself and had no bearing on TDE.

The performance issues in TDE all stemmed from Konqueror's views loading detailed item metadata outside of the dedicated (delayed) metadata loading system.  It seems I put the last nail in the performance coffin quite accidentally when I added media item free space display support; that action made the only unaffected view (icon mode) just as slow as the rest of the views.  Most likely the original coders over time made the same mistake, leading to extreme performance regressions by subverting the core Konqueror design.

Konqueror now loads lightning fast in all modes except Info List View; this last mode unfortunately requires that all metadata be loaded before showing anything so that it can decide which info columns to display.

For future analysis, the most reliable test method I could come up with is:
sync; echo 2 > /proc/sys/vm/drop_caches
time /opt/trinity/bin/konqueror /usr/lib/
then close Konqueror as soon as it becomes responsive/lists the directory.  The 2 echoed to drop_caches drops the inode/dentry cache and friends, which reliably caused Konqueror to take forever to start up before these patches went in.

Fixed in GIT hashes a21cb20 (tdelibs) and 3e7ad37 (tdebase).

Please test and report back...

>Please test and report back.
My initial testing with tree view:

In each test I flushed the kernel cache and TDE cache, then started TDE.

I opened konq and in the Location bar typed /usr/lib. Much faster response.

Then I noticed when moving up or down one directory level, Konq seemed to freeze for a long time, but eventually displayed the selected directory. Repeated testing indicates the delay is about 20-25 seconds (T400 laptop, Core Duo, 2GB). Same result using the Alt-Up arrow shortcut.

Doesn't seem to matter which direction I moved in the directory. The first instance of moving up or down results in the delay. There is no refreshing or progress bar during the delay, thus the appearance of a freeze.

How I flush the kernel cache:

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

How I flush the TDE cache:

rm -f $HOME/.trinity/cache-${BOXNAME}/tdesycoca*

(In reply to Darrell from comment #11)
> >Please test and report back.
> My initial testing with tree view:
> 
> In each test I flushed the kernel cache and TDE cache, then started TDE.
> 
> I opened konq and in the Location bar typed /usr/lib. Much faster response.

Great!

> Then I noticed when moving up or down one directory level, Konq seemed to
> freeze for a long time, but eventually displayed the selected directory.
> Repeated testing indicates the delay is about 20-25 seconds (T400 laptop,
> Core Duo, 2GB). Same result using the Alt-Up arrow shortcut.

Just tested this in icon view mode; no hang on my system.  What view mode are you testing with?

> Doesn't seem to matter which direction I moved in the directory. The first
> instance of moving up or down results in the delay. There is no refreshing
> or progress bar during the delay, thus the appearance of a freeze.
> 
> How I flush the kernel cache:
> 
> free && sync && echo 3 > /proc/sys/vm/drop_caches && free

That flushes all application code as well--probably not the best thing to be doing unless you like staring at frozen applications as they page back from disk. ;-)

Looks like we're close to fixing this at any rate.  Thanks for testing!

Tim

>What view mode are you testing with?
First sentence of comment 11: tree view.

>That flushes all application code as well--probably not the best thing to be
>doing unless you like staring at frozen applications as they page back from
>disk.
As I replicated the delay consistently, I tested with a fresh default profile. I could not replicate the delay. Every single test I first flushed the kernel cache before starting TDE. Not being able to replicate the results indicated something in the overall konq configuration. After about a half-hour of trial and error guessing, I narrowed the problem to the konq left-side navigator panel (toggle with F9).

When the navigator panel is selected/configured to show the Root Folder (the folder icon on the nav panel toolbar) I then see the delays. Even in Icon view.

When the nav panel is toggled off, or any of the other nav panel toolbar options are selected, there is no delay. The delay always occurs when Root Folder is selected. The delay is caused by the nav panel drilling to the same directory location as the file panel in order to remain synchronized.

I'll guess the problem with the nav panel is updating with the same flawed code resolved in the file panel.

To view the delay:

* Open konq.
* Enable the nav panel (F9 is a toggle shortcut).
* Select a nav panel option other than Root Folder.
* In the konq Location bar, type a directory several levels deep. I chose /usr/lib/firefox/browser.
* Use the keyboard shortcut (Alt-Up) or konq toolbar to drill up in the directory tree.
* Notice no delays. I moved up two levels to /usr/lib.
* In the nav panel, select the Root Folder option.
* Attempt to drill back down to /usr/lib/firefox/browser.
* Notice the delay as the nav panel attempts to synchronize.

>Looks like we're close to fixing this at any rate.
Yes, seems the file panel is resolved. Now the nav panel. :)

Confirmed Darrell's observation.
The list panel has no delay (/usr/bin now opens in fraction of seconds compared to 50 seconds), but the treeview on the left shows some delay.
To replicate follow Darrell's steps.

(In reply to Darrell from comment #13)
> >What view mode are you testing with?
> First sentence of comment 11: tree view.

Sorry about that; I didn't see it the first time due to its proximity to the reply header. :-)

(In reply to Michele Calgaro from comment #14)
> Confirmed Darrell's observation.
> The list panel has no delay (/usr/bin now opens in fraction of seconds
> compared to 50 seconds), but the treeview on the left shows some delay.
> To replicate follow Darrell's steps.

I'll get on this as soon as I wrap up the kaboodle stuff I am working on.

Note to us: before closing this report as resolved, ensure the release notes contain a reference about improved performance with large directories. :)

>I'll get on this as soon as I wrap up the kaboodle stuff I am working on.
Looks like the libmagic mime changes need priority before this. The recent commits are causing FTBFS errors.

>Note to us: before closing this report as resolved, ensure the release notes >contain a reference about improved performance with large directories. :)
Already added yesterday, listed as second item in the list :)

(In reply to Michele Calgaro from comment #14)
> Confirmed Darrell's observation.
> The list panel has no delay (/usr/bin now opens in fraction of seconds
> compared to 50 seconds), but the treeview on the left shows some delay.
> To replicate follow Darrell's steps.

OK, the dirtree module stall should be fixed in GIT hashes 54578ce (tdelibs) and c6c4c0e (tdebase).

Good work in noticing the problem; I don't normally use the root view mode and didn't think that the dirtree module would be affected at all as there is no reason for it to be making mime magic calls.

Looks good here.

Looks good here. Thank you.