Here is the patch I submit, with comments slightly changed.

Description: unset the limit to line length in plain text files
 The 'TEXT_MAXLINELEN 300' constant is removed, as well as the test for long
 lines, in the 'textmagic' function, in kio/kio/kmimemagic.cpp.
 It made konqueror identify configuration files with long lines as files from
 unknown type.
 .
 The bug is still present in the KDE4 code. It has already been reported
 (http://bugs.kde.org/143100), but the report was closed.
 .
 This bugfix doesn't solve other errors in file detection (debian/rules files
 not detected as "make files", wrong reports of "Java code", "Objective-C
 code", "diff"...). (http://bugs.kde.org/116272)
 .
 Thanks to Darrell Anderson, Werner Joss and Richard J.M. Hill.
   [trinity-devel] kmailrc file type unknown
   [trinity-devel] Kmailrc File Type.

Created attachment 140 [details]
kdelibs-maxlinelength-1.patch

More infor:

Some ~/.trinity/share/config/*rc files are listed in Konqueror as Desktop Config Files rather than Plain Text Document.

~/.trinity/share/config/*rc files detected as desktop files are in fact really desktop files, at least in my folder.

(I opened another bug report for all other mime recognition problems here:
http://bugs.trinitydesktop.org/show_bug.cgi?id=661)

I was wrong concerning the files detected as desktop files:
They are not desktop files. They only begin like desktop files.

Created attachment 142 [details]
test301rc

test file recognized as plain text

Created attachment 143 [details]
test302rc

test file wrongly identified as from unknown type

Created attachment 144 [details]
kdelibs-maxlinelength-2.patch

I only removed a comment about kde4 in the comment of the patch.

I thought the code was still in KDE4 but it was a mistake, kmimemagic.cpp isn't in KDE4 anymore. KDE4 uses share-mime-info to detect file types.

This should be fixed properly via the use of the shared-mime-info database.

Unmarking as PATCHAVAIL

How do we fix this with the shared-mime-info database? What do we fix? Do we need to submit patches to upstream maintainers?

Is the 300 character string length something we stll need to patch in Trinity or is that a shared-mime-info database problem too?

I tested the max line length patch attached to this bug report. The patch succeeds with helping some files list as a text mime type rather than unknown. Regarding the original problem report with kmailrc, on my system this file now lists as a text file mime type.

Even if the shared-mime-info database needs to be updated, this patch nonetheless seems to help.

Patch in attachment 144 [details] pushed in GIT commit b61f0c47.

Let's push this bug report and bug 661 and bug 1213 to receive immediate attention in post R14.0.0. The mimetype problem is a humorous at times. I have three practically identical systems running Slackware 14.0 (all fully patched) and Trinity pre R14. On one system my .xsession-error file is identified as a Usenet News Message, on another as a C Source File, and the third as a Plain Text File.

I rewrote the mime type detection system in GIT hash 8066e87 (tdelibs).  KMimeMagic now uses libmagic and the system mime type detection file to determine what type unknown files are; it should be far more accurate than the ancient version of the same library that was integrated into TDE around 15 years ago.

I'm going to mark this as RESOLVED FIXED; if there are problems with mime type *detection* (not display and/or handling of the detected type) they will need to be filed upstream against libmagic.

I now see a bunch of Unknown file types in Konqueror. All of the files previously appeared in Konqueror as plain text files. For example, open Konqueror to $TDEHOME/share/config.

The files appear as Unknown with a fresh profile too.

> I now see a bunch of Unknown file types in Konqueror. All of the files
> previously appeared in Konqueror as plain text files. For example, open 
> Konqueror to $TDEHOME/share/config.
Darrell, is the extension of such files actually known to the system?
I don't know what TDE used to do with unknown files before the change to libmagic, but if the system does not actually know what a file is, I would prefer it displays "Unknown type" instead of "Plain text".

>is the extension of such files actually known to the system?
How does one learn that?

>I don't know what TDE used to do with unknown files before the change to
>libmagic, but if the system does not actually know what a file is, I would
>prefer it displays "Unknown type" instead of "Plain text".
What do you see when you view $TDEHOME/share/config? Plain text files or Unknown files?

> >is the extension of such files actually known to the system?
> How does one learn that?
Konqueror -> Settings -> Configure Konqueror -> File associations

> >I don't know what TDE used to do with unknown files before the change to
> >libmagic, but if the system does not actually know what a file is, I would
> >prefer it displays "Unknown type" instead of "Plain text".
> What do you see when you view $TDEHOME/share/config? Plain text files or
> Unknown files?
I see a long list of Unknown files, as you see. But all those files have no extension at all, so the system can not determine their mime types. If I add some extensions to the files, the mime type if known, is correctly detected. If unknown, it remains Unknown.
To me this behavior is consistent: if the system does not know the type, it simply tells you that it doesn't know it. I guess TDE used to display "Plain text" instead, which is ok if the file is actually a text file, but would be misleading is it was for example a binary one.

IMO, this bug could be re-closed. Just my 2 cents.

I tested Xfe, KDE4 dolphin, Cinnamon Nemo, MATE Caja. All show $TDEHOME/share/config files as something intelligent other than "Unknown," such as Document, Text, Plain Text. A cursory browse of other files with those file managers indicates TDE Konqueror now is the exception with identifying files.

Look at bin and lib directories. Almost all files are now listed as type Unknown.

(In reply to Darrell from comment #21)
> Look at bin and lib directories. Almost all files are now listed as type
> Unknown.

I think this is a secondary problem.  The old TDE-specific mime magic database needs to be removed from the system otherwise libmagic tries to load it and nothing gets detected properly due to the old format--the bin and lib problem you describe is characteristic of this problem.

See if you have a magic file in your trinity/share/mimelnk directory and remove it if present, then try again.

/opt/trinity/share/mimelnk/magic and ~/.trinity/share/mimelnk/magic do not exist on my updated test system. TDE caches are flushed/deleted whenever I update TDE files.

In Slackware 14.1, libmagic is installed with the file 5.14 package.

If you are not seeing the problem then something else on my system is causing libmagic to fail. Ideas?

(In reply to Darrell from comment #23)
> /opt/trinity/share/mimelnk/magic and ~/.trinity/share/mimelnk/magic do not
> exist on my updated test system. TDE caches are flushed/deleted whenever I
> update TDE files.
> 
> In Slackware 14.1, libmagic is installed with the file 5.14 package.
> 
> If you are not seeing the problem then something else on my system is
> causing libmagic to fail. Ideas?

I am definitely not seeing the failure in lib and bin that you are.

You could try running an strace on a new Konqueror instance to see if it is loading an older mime magic database from an alternate location.  Attach the strace here and I'll take a look at it.

Created attachment 2286 [details]
strace opening konqueror

At line 177 a call is made to /usr/lib/libmagic.so.1. The file exists on my system and has 755 permissions.

I see the same Unknown results with a fresh profile, thus the problem is not likely stale profile data.

(In reply to Darrell from comment #25)
> Created attachment 2286 [details]
> strace opening konqueror
> 
> At line 177 a call is made to /usr/lib/libmagic.so.1. The file exists on my
> system and has 755 permissions.
> 
> I see the same Unknown results with a fresh profile, thus the problem is not
> likely stale profile data.

Very strange.  I'm putting this on the R14 block list until I can figure out exactly what happened.

I have been surfing the web looking for clues. libmagic seems to be a very well kept secret --- I am not finding anything useful.

At this point I can only affirm that non TDE file managers on my system do not have the problem. No hidden magic files in TDE. Same results with a fresh profile.

I have not tested TDE Dolphin or TDE Krusader. I will do that today.

(In reply to Darrell from comment #27)
> I have been surfing the web looking for clues. libmagic seems to be a very
> well kept secret --- I am not finding anything useful.

libmagic is the library interface of the file(1) command.  If 'file --mime-type /bin/bash' returns something like 'application/x-executable' your system mime database is OK (though I suspect it is anyway seeing as other desktops work).

> At this point I can only affirm that non TDE file managers on my system do
> not have the problem. No hidden magic files in TDE. Same results with a
> fresh profile.
> 
> I have not tested TDE Dolphin or TDE Krusader. I will do that today.

Won't make any difference I'm afraid; all of these applications use the same KMimeResolver core class.

Tim

(In reply to Michele Calgaro from comment #19)
> > >is the extension of such files actually known to the system?
> > How does one learn that?
> Konqueror -> Settings -> Configure Konqueror -> File associations
> 
> > >I don't know what TDE used to do with unknown files before the change to
> > >libmagic, but if the system does not actually know what a file is, I would
> > >prefer it displays "Unknown type" instead of "Plain text".
> > What do you see when you view $TDEHOME/share/config? Plain text files or
> > Unknown files?
> I see a long list of Unknown files, as you see.
<snip>

This might be a clue; on my system I see most of the files in $TDEHOME/share/config identified as plain text.  Michele, you are testing with Debian as well, correct?  Can you check for the /opt/trinity/share/mimelnk/magic file on your system?

Thanks!

(In reply to Darrell from comment #25)
> Created attachment 2286 [details]
> strace opening konqueror
> 
> At line 177 a call is made to /usr/lib/libmagic.so.1. The file exists on my
> system and has 755 permissions.
> 
> I see the same Unknown results with a fresh profile, thus the problem is not
> likely stale profile data.

This does not appear to be a complete strace log.  Can you re-do it like this:
strace konqueror ~/.trinity/share/config &> ~/strace.log

wait for konqueror to load *fully* and list all files, then close it and post the ~/strace.log file?

Thanks!

Created attachment 2288 [details]
2nd strace log

>I suspect it is anyway seeing as other desktops work
Yes, the command line queries return expected results. That's a good start. :)

>I have not tested TDE Dolphin or TDE Krusader.
Useless tests as I cannot figure out how to get either app to display file types in the file panels. Using the Properties dialog in both apps on any file still results in Unknown.

>post the ~/strace.log file
Attached. Looks more helpful this time.

Thanks, much more helpful!

Looks like the system is attempting to append the contents of /opt/trinity/share/config/magic/ to the magic database.  This is then failing, and (I guess) corrupting the master database in memory as well.

Let me play with this a bit and see what I can do.

Due to the relatively poor documentation available for libmagic it seems I was using it wrong when more than one magic file was combined.

Specifically the load function loads the new magic file *in addition to* any and all existing magic files already loaded into memory.  This was not clear, so I assumed load overwrote any previously loaded files.  In doing so I made libmagic unable to load *any* magic files, hence the failure.

This is fixed in GIT hash 62bfcbe.  Thanks for the strace!

Created attachment 2289 [details]
crash backtrace attempting to start konqueror

I rebuilt tdelibs and tdebase with the latest commits. I get a similar crash report starting kate.

I get the same crash as root.

I did not rebuild any other package but the crash indicates something awry with libmagic.

(In reply to Darrell from comment #34)
> Created attachment 2289 [details]
> crash backtrace attempting to start konqueror
> 
> I rebuilt tdelibs and tdebase with the latest commits. I get a similar crash
> report starting kate.
> 
> I get the same crash as root.
> 
> I did not rebuild any other package but the crash indicates something awry
> with libmagic.

Well I can't seem to keep this bug closed, can I? :-P  Obviously it didn't crash on my development system.

Can you install debugging symbols for libmagic or, if they are not available, compile and install libmagic from source?  Afterwards please regenerate and post the new backtrace.

Thanks!

Slackware does not come with debugging symbols. I have to rebuild file-5.14. I am not having success finding how to enable debugging symbols with this package.

Created attachment 2290 [details]
konqueror crash backtrace with libmagic symbols

Never mind. There must be a universal law of physics that the answer appears two minutes after posting. Does this backtrace help?

Created attachment 2291 [details]
strace log

> Michele, you are testing with Debian as well, correct?  Can you check for the > /opt/trinity/share/mimelnk/magic file on your system?
Yes, testing in Debian/Jessie. If needed I can also test on a fresh new VM environment.
I don't have any /opt/trinity/share/mimelnk/magic in my system, nor a ~/.trinity/share/mimelnk/magic (same as Darrell)

I have attached a strace log as required. Keep in mind this is not using the latest commits of the last two days. I can rebuild and re do it if required.

(In reply to Darrell from comment #37)
> Created attachment 2290 [details]
> konqueror crash backtrace with libmagic symbols
> 
> Never mind. There must be a universal law of physics that the answer appears
> two minutes after posting. Does this backtrace help?

Yes it does.  Unfortunately it also points to an upstream bug against libmagic (figures):
http://bugs.gw.com/view.php?id=248

On your system, in the file /usr/include/magic.h what is the value of MAGIC_VERSION?  If I'm really lucky maybe I can just blacklist that libmagic version and take the slight functionality loss with broken versions.  If not, I'm pretty much stuck as the old code has its own flaws and libmagic is the proper way to do mimetype detection.

On my system MAGIC_VERSION is 518.
I have started a rebuild, but I will be able to complete it by tonight. Tomorrow I can report on a test using the latest GIT code.

(In reply to Michele Calgaro from comment #40)
> On my system MAGIC_VERSION is 518.
> I have started a rebuild, but I will be able to complete it by tonight.
> Tomorrow I can report on a test using the latest GIT code.

Yep, 518 here too.  I'd bet that Darrell's system is 517.

Tim

with the latest GIT sources (as of now), the situation improves. I still see several "Unknown" in /usr/bin or /bin, but now I also have several "Plain text files" displayed. I can post a screenshot if required.
$TDEHOME/share/config now shows "Plain text files" for all files.

I found a (potentially) new problem after the switch to libmagic. 
Select a folder that displays both "Unknown" and "known" type of files (for example /bin is a good choice on my system).
Try to open an "unknown" file (either double click on it or right click + Open with...). Notice that there is no option to "Remember application association for this type".
Try the same on a known type: the option appears.

So far so good, the above make sense even though it could be improved.
But now I have problems for example with pdf files. They are all displayed as "Unknown" even though in Konqueror -> Settings -> Configure Konqueror -> File associations, the entry for pdf exists.

I guess the same problem may also happen on some other extensions, even though I haven't noticed yet.

Can anyone reproduce this?

Slackware 14.1: MAGIC_VERSION 514

The next version of Slackware is the same. What is the latest (working) version?

On Slackware libmagic is not a separate package but part of the file package.

By coincidence, every time I open konqueror and experience the crash, I also receive the error message reported in bug 1718. Perhaps the backtrace provides a clue about resolving that bug too?

>Can anyone reproduce this?
I cannot test with the current sources because of the crashes. Reverting to builds from two days ago, before the latest libmagic commit in tdelibs, the option check box does not exist in the dialog.

>Yes it does.  Unfortunately it also points to an upstream bug against libmagic
There must be a work-around? Because no other file manager on my system crashes.

(In reply to Darrell from comment #46)
> >Yes it does.  Unfortunately it also points to an upstream bug against libmagic
> There must be a work-around? Because no other file manager on my system
> crashes.

OK, after much gnashing of teeth and reading the libmagic/file sources, I think I understand how to use libmagic properly.  Ignore my notes in Comment 33.

Both the crash and the issue noted by Michele should be fixed in GIT hash e5f8982.  It took a while to verify because I needed to test thoroughly with and without the extra TDE-specific magic definitions, but I'm fairly confident this should be the final patch needed.

Darrell, you are just lucky I guess.  Any libmime version between 5.12 and 5.17 inclusive will crash when reloading the magic database files.  I have resolved the crash by disabling additional definition loads if the system libmime version is in that range--you might notice a tiny drop in functionality (the extra magic files in /etc/trinity/magic will not be used), however this is a.) not an absolute drop in fuctionality vs. TDE 3.5.13.2 due to the thoroughly obsolete magic database TDE used to use and .b) easily resolved by upgrading your libmagic to >= 5.18.

I'm not going to jinx myself by closing the report; if both Darrell and Michele agree that the MIME detection system is working properly (e.g. same as or better than TDE 3.5.13.x) please close the report. :-)

Thanks!

Tim

Does the backtrace provide any clues for bug 1718 (comment 45)?

>easily resolved by upgrading your libmagic to >= 5.18.
While I personally could update the file package on Slackware, which includes libmagic, that will not help other Slackers. Generally, packages are updated in previous Slackware releases only for security updates, a common policy with most distros. Backporting fixes for crashes have to be proven severe and must affect the stock Slackware. As Trinity is not a stock software and not officially supported in Slackware, getting libmagic updated is unlikely (although I can always ask). The officially supported desktops in Slackware are KDE and Xfce. As none of those respective file managers have problems with the stock libmagic 5.14, crashes in TDE will not be convincing arguments to backport libmagic. :(

That said, the latest commit results in no crashes anyway. So back to discussing bugs. :)

>the issue noted by Michele
When I select a file listed as Unknown, the check box option to remember is not in the dialog. If the file is listed incorrectly but anything other than Unknown, the remember option appears in the dialog. But this was the behavior in TDE before the mime changes. Makes sense because there is no mime type of Unknown to associate an app. Thus there should not be a remember option.

>...should be fixed in GIT hash e5f8982.
After rebuilding tdelibs, on my system all shell scripts are listed as Unknown.

Other oddities are a few $TDEHOME/share/config files are not listed as plain text files but as C/C++/Pascal source files.

Looking at the same anomaly files in KDE Dolphin, and presuming Dolphin uses libmagic, indicates the problem probably is Trinity, since both file managers are using the same libmagic 5.14. Dolphin lists shell scripts as shell scripts. Dolphin lists a few $TDEHOME/share/config files as desktop configuration files, but does not list any as C/C++/Pascal source files.

Despite me using libmagic 5.14, that other file managers use libmagic and seem to have fewer problems leaves me hanging with my rhetorical questions in comment 46: why do they not see the same problems? After browsing several directories in Konqueror, my feeling is I now see fewer incorrect listings than with kmimemagic. That is good.

I remain puzzled why TDE lists obvious text files as C/C++/Pascal files and the other file managers do not. While I can live with those few anomalies, listing shell scripts as Unknown is a big backwards workflow step because often I edit shell scripts from the Konqueror context menus. I no longer can do that. I cannot even double-click on the files. I now have to interrupt that workflow to select a text editor.

I do not consider this bug report resolved. R14 blocker? Probably not, but will be embarrassing if the shell script Unknown problem hits other distros. I don't think many people will bother with a handful of Unknown files, but listing shell scripts as Unknown will only reopen this bug report or result in new bug reports.

A quick comparison note. I booted into LMDE, which has libmagic version 5.14 installed. The MATE Caja file manager lists shell scripts as shell scripts and all $TDEHOME/share/config files as plain text documents.

As noted in the dev mail list, magic.h is not a standard user installed file with some distros. Installing magic.h in LMDE (probably all Debian distros) requires the development package. :(

(In reply to Darrell from comment #49)
> >easily resolved by upgrading your libmagic to >= 5.18.
> While I personally could update the file package on Slackware, which
> includes libmagic, that will not help other Slackers. 
<snip>

Fully aware, hence the patch that resolved the crashes.  TDE has a feature that these other desktop environments don't, so it exposes a flaw in libmagic that has since been fixed in later libmagic versions.  Don't want to use the feature?  No problem, it's now automatically disable with the buggy libmagic versions.  Simple as that. :-)

> That said, the latest commit results in no crashes anyway. So back to
> discussing bugs. :)
> 
> >the issue noted by Michele
> When I select a file listed as Unknown, the check box option to remember is
> not in the dialog. If the file is listed incorrectly but anything other than
> Unknown, the remember option appears in the dialog. But this was the
> behavior in TDE before the mime changes. Makes sense because there is no
> mime type of Unknown to associate an app. Thus there should not be a
> remember option.

This is an old bug as you say, but it goes deeper than it first appears.  TDE gets a valid mimetype from libmagic, but doesn't know what to do with it, so it appears as Unknown.  I'd like to move this part of the report to a new bug as this one is getting quite cluttered.

> >...should be fixed in GIT hash e5f8982.
> After rebuilding tdelibs, on my system all shell scripts are listed as
> Unknown.

Ooops.  Seems in the intervening decade Linux moved to identifying said scripts as text/... instead of application/...  Fixed in GIT hash f8790c7.  If TDE doesn't have a mimetype .desktop file for a given MIME type it appears as Unknown--essentially TDE doesn't know what the file type is called or what to do with it even though libmagic spat out a mime string.  Fixing these is a simple matter of identifying them (e.g. verifying file(1) lists the file as something other than octet stream) and adding an appropriate mime definition file to tdelibs/mimetypes.

> Other oddities are a few $TDEHOME/share/config files are not listed as plain
> text files but as C/C++/Pascal source files.

What does 'file -k --mime-type <file name>' have to say about those files?

> Despite me using libmagic 5.14, that other file managers use libmagic and
> seem to have fewer problems leaves me hanging with my rhetorical questions
> in comment 46: why do they not see the same problems? After browsing several
> directories in Konqueror, my feeling is I now see fewer incorrect listings
> than with kmimemagic. That is good.

Glad to see you can at least see some progress!  Regarding comment 46, I can think of lots of ways for this to be the case, ranging from direct embedding of a known-good libmagic version and database (like we used to do; eventually leads to obsolescence issues) to detecting bad versions (like we do now) to simply not using any features that cause problems.

> I do not consider this bug report resolved. R14 blocker? Probably not, but
> will be embarrassing if the shell script Unknown problem hits other distros.

Well as long as the mime detection is "embarrassing" yes, this is an R14 blocker.  When it's on par with other desktops then this report can be closed.

Tim

>Ooops.  Seems in the intervening decade Linux moved to identifying said scripts as text/... instead of
>application/...  Fixed in GIT hash f8790c7.
Yay! Much better!

>What does 'file -k --mime-type <file name>' have to say about those files?
Same as Konqueror. Some examples (Konqueror, file command):

$TDEHOME/share/config/kcharselectrc: C source File, text/x-c
$TDEHOME/share/config/kdeglobals: Pascal Source File, text/x-pascal
$TDEHOME/share/config/kilerc: C++ source File, text/x-c++
$TDEHOME/share/config/konquerorrc: C source File, text/x-c
$TDEHOME/share/config/ksmserverrc: Pascal Source File, text/x-pascal
gtkrc, gtkrc-2.0: C++ source File, text/x-c++
.conkyrc: C source File, text/x-c

Problem likely is upstream, possibly cured by later versions of file/libmagic. Does anybody else see the same?

Overall, I think this bug report is resolved unless Michele sees other problems.

>Does anybody else see the same?
Out of curiosity, I build a file-5.19 package. All but the gtkrc* files are listed as text files. Thus the anomalies are upstream. Michele can close this report if he finds nothing awry.

I filed bug 2148 against kate and syntax highlighting. Definitely a mime issue.

Mime type recognition looks ok with the latest tdelibs code and the problem I highlighted yesterday is gone. Well done Tim!
As for me, the bug can be closed.

Closing bug (again).

Thanks for reporting and for all the assistance in catching breakage!

(In reply to Timothy Pearson from comment #56)
> Closing bug (again).
> 
> Thanks for reporting and for all the assistance in catching breakage!

Uh, I'm a little embarrassed, but I was forced to reopen this bug report.

I tried to give rebuild tdelibs for all debian / ubuntu distribution, but I stopped at the first - squeeze. libmagic 5.04 does not contains function magic_version. I checked further wheezy - libmagic 5.11 also not contains magic_version. Only libmagic 5.19 from jessie. Ubuntu versions I did not check.

What to do?

Cursing the libmagic documentation again I implemented a workaround in GIT hash 58c3aed (tdelibs).

Please test and confirm.

Francois confirmed on-list that compilation works now, so I'm closing this report yet again.

I apologize for that, but...

Ubuntu 10.04 - Lucid == libmagic 5.03-5ubuntu1

/tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp: In member function 'int KMimeMagic::apprentice(const TQString&)':
/tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp:167: error: 'magic_getpath' was not declared in this scope

(In reply to Slávek Banko from comment #60)
> I apologize for that, but...
> 
> Ubuntu 10.04 - Lucid == libmagic 5.03-5ubuntu1
> 
> /tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp: In
> member function 'int KMimeMagic::apprentice(const TQString&)':
> /tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp:167:
> error: 'magic_getpath' was not declared in this scope

I'm elbows deep in the TDENewStuff class right now, and have a horrid track record with this bug. :-)

Want to try fixing this?

(In reply to Timothy Pearson from comment #61)
> (In reply to Slávek Banko from comment #60)
> > I apologize for that, but...
> > 
> > Ubuntu 10.04 - Lucid == libmagic 5.03-5ubuntu1
> > 
> > /tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp: In
> > member function 'int KMimeMagic::apprentice(const TQString&)':
> > /tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp:167:
> > error: 'magic_getpath' was not declared in this scope
> 
> I'm elbows deep in the TDENewStuff class right now, and have a horrid track
> record with this bug. :-)
> 
> Want to try fixing this?

Actually, what is the first Ubuntu version that supports magic_getpath?  That function is used solely to allow multiple mime magic databases to be appended to the default database; that feature will likely need to be disabled for older libmagic versions.

Slowly I see why a known good libmagic version was embedded in TDE; however with the latest libmagic versions being far more functional than that old copy (and with the two being mutually exclusive) we are left in a very bad situation.

>and have a horrid track record with this bug.
I think you are doing a great job with this bug. The libmagic backend introduced some new holes, but you fixed them rapidly.

(In reply to Timothy Pearson from comment #62)
> (In reply to Timothy Pearson from comment #61)
> > (In reply to Slávek Banko from comment #60)
> > > I apologize for that, but...
> > > 
> > > Ubuntu 10.04 - Lucid == libmagic 5.03-5ubuntu1
> > > 
> > > /tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp: In
> > > member function 'int KMimeMagic::apprentice(const TQString&)':
> > > /tmp/buildd/tdelibs-trinity-14.0.0-s~889/tdeio/tdeio/kmimemagic.cpp:167:
> > > error: 'magic_getpath' was not declared in this scope
> > 
> > I'm elbows deep in the TDENewStuff class right now, and have a horrid track
> > record with this bug. :-)
> > 
> > Want to try fixing this?
> 
> Actually, what is the first Ubuntu version that supports magic_getpath? 
> That function is used solely to allow multiple mime magic databases to be
> appended to the default database; that feature will likely need to be
> disabled for older libmagic versions.
> 
> Slowly I see why a known good libmagic version was embedded in TDE; however
> with the latest libmagic versions being far more functional than that old
> copy (and with the two being mutually exclusive) we are left in a very bad
> situation.

Affected are Lucid and Maverick. Starting Natty magic_getpath function is available.

I am thinking about the following solution: Instead of magic_getpath use MAGIC environment variable and if it is not set, use file /usr/share/misc/magic.

Optionally I can add CMake test for magic_getpath existence and proposed solution use conditional upon this test.

(In reply to Slávek Banko from comment #64)
> Affected are Lucid and Maverick. Starting Natty magic_getpath function is
> available.
> 
> I am thinking about the following solution: Instead of magic_getpath use
> MAGIC environment variable and if it is not set, use file
> /usr/share/misc/magic.

This is not portable.  Be aware that if the default database is not found many core components of TDE will not work correctly (Konqueror, Kwrite, Kate, etc.).

Given that only Lucid and Maverick are affected an HAVE_OLD_LIBMAGIC define should be set by CMake/Autotools if magic_getpath is not present in libmagic.so.  This then would be used to disable the incremental magic database load.

Yes, I know it's a functionality loss, but it is quite minor compared to the serious consequences of not finding the mime database because a non-Debian distro put it somewhere else.  For such a critical library it was maintained in a very sloppy fashion until the past year or so.

(In reply to Timothy Pearson from comment #66)
> (In reply to Slávek Banko from comment #64)
> > Affected are Lucid and Maverick. Starting Natty magic_getpath function is
> > available.
> > 
> > I am thinking about the following solution: Instead of magic_getpath use
> > MAGIC environment variable and if it is not set, use file
> > /usr/share/misc/magic.
> 
> This is not portable.  Be aware that if the default database is not found
> many core components of TDE will not work correctly (Konqueror, Kwrite,
> Kate, etc.).
> 
> Given that only Lucid and Maverick are affected an HAVE_OLD_LIBMAGIC define
> should be set by CMake/Autotools if magic_getpath is not present in
> libmagic.so.  This then would be used to disable the incremental magic
> database load.
> 
> Yes, I know it's a functionality loss, but it is quite minor compared to the
> serious consequences of not finding the mime database because a non-Debian
> distro put it somewhere else.  For such a critical library it was maintained
> in a very sloppy fashion until the past year or so.


Another proposed solution: Add CMake test for magic_getpath existence and if it does not exist, read the list of databases using default "file --version":

# file --version
file-5.03
magic file from /etc/magic:/usr/share/misc/magic

I don't have standing in this discussion, but both releases are past end-of-life and no longer supported upstream:

Ubuntu 10.10 Maverick Meerkat April 10, 2012
Ubuntu 10.04 Lucid Lynx May 9, 2013

Would dropping TDE support for those releases be an option?

(In reply to Darrell from comment #68)
> I don't have standing in this discussion, but both releases are past
> end-of-life and no longer supported upstream:
> 
> Ubuntu 10.10 Maverick Meerkat April 10, 2012
> Ubuntu 10.04 Lucid Lynx May 9, 2013
> 
> Would dropping TDE support for those releases be an option?

No, not really.  Especially with Lucid you have to think like a typical user; Lucid has been supported for so long and Ubuntu had broken so much in their subsequent releases that there very well might be Lucid installations in the wild that *cannot* yet be upgraded.  Restricting TDE upgrades from such systems, when the only ill effect to TDE is a slight loss in mime magic functionality on such systems, doesn't make sense.

Besides, I floated the idea of dropping support for older Ubuntu versions earlier this year on the mailing list to reduce the load on the build farm--the consensus was release R14, then drop support.

Tim

(In reply to Darrell from comment #63)
> >and have a horrid track record with this bug.
> I think you are doing a great job with this bug. The libmagic backend
> introduced some new holes, but you fixed them rapidly.

Thanks for the vote of confidence!  I was growing concerned over the constant fix-->break-->fix cycles. ;-)

BTW Slavek I'm assigning this particular fix to you.  I need to poke around on the bugtracker and verify I haven't missed anything else for R14, and handle other pre-RC1 tasks.

Thanks!

Created attachment 2297 [details]
fix FTBFS on missing magic_getpath

I attach a patch for review. The findings default path by calling: "file --version" is done during build time, instead of at runtime.

(In reply to Slávek Banko from comment #71)
> Created attachment 2297 [details]
> fix FTBFS on missing magic_getpath
> 
> I attach a patch for review. The findings default path by calling: "file
> --version" is done during build time, instead of at runtime.

Excellent solution!  I've been fixing bugs too long; didn't think of it. :-P

Can you also add something similar to the Makefile.am for tdeio?  We haven't fully dropped automake support yet and we should therefore try to keep them somewhat consistent.

Once that is done, go ahead and push.

Thanks!

(In reply to Timothy Pearson from comment #72)
> (In reply to Slávek Banko from comment #71)
> > Created attachment 2297 [details]
> > fix FTBFS on missing magic_getpath
> > 
> > I attach a patch for review. The findings default path by calling: "file
> > --version" is done during build time, instead of at runtime.
> 
> Excellent solution!  I've been fixing bugs too long; didn't think of it. :-P
> 
> Can you also add something similar to the Makefile.am for tdeio?  We haven't
> fully dropped automake support yet and we should therefore try to keep them
> somewhat consistent.
> 
> Once that is done, go ahead and push.
> 
> Thanks!

I'm afraid it's more things that are not consistent in automake. However, yes, I can try to prepare a function check also for automake.

Comment on attachment 2297 [details]
fix FTBFS on missing magic_getpath

Added code for automake configure script (but not tested) and pushed to GIT in hash 03a61295.

Closing bug (again). Anyone else who would like to reopen this bug? :)