Bug 2221

Removed "1/2" from bug title.

(I had originally thought that Packages.gz was permanently redirected leading to a "2/2" bug filing but I was mistaken.)

obsolete

Not obsolete, just moved to the current TDE URL, e.g.

http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian

I don't quite get it. Whats the problem?
The closest mirror is chosen for faster downloads.
It's the responsibility of the site owner to have good mirrors.

Mirrors don't all update at the same rate, particularly during new releases when rollout takes several days.  But even during minor updates it takes several hours for all mirrors to fully sync.  (I know because I operate not Tim's master server but the master mirror which feeds the other mirrors.)

To this end Tim's master server maintains information about which mirrors have each regular file and only redirect regular file requests to mirrors which have the requested file.  This works fine.

However the master server also redirects requests for directories.  This can result in redirecting to an incomplete mirror of a directory, only to find that desired regular files or subdirectories are missing.  I know this happens because that's how I first stumbled over this bug.

The master server redirector should not redirect directory requests, only regular file requests.  Keep users on the master server until they ask for a regular file, and only then redirect them.

so in stupidspeach, you are saying,
the server is missconfigured.

Tim or Slavek know about this?

Bugs here tend to be berried and forgotten.
You posted this 3 years ago.
You should send them an email directly.

Tim has higher priorities.  Just leave this bug report open and if he finishes dealing with more important bugs this report will remind him about this rare but real problem.  G'nite.  --Mike

Maybe he should delegate the master server to you also?
Since he has no time?

I will not comment on the speeches wofgdkncxojef.

Mike, your proposal is very helpful. Regardless of the mirror synchronization status, if the user is redirected to one of the mirrors for a folder listing, then all subsequent downloads use URL to that mirror, instead of using the redirector. Because of this, the redirector can not be helpful with redirection for files that are missing for a specific mirror.

Because I'm administrator of the current main redirector on mirror.ppa.trinitydesktop.org, I tried to solve the problem.

Here are two major pitfalls:

1) Redirector is not the primary server == does not have the primary server content.

2) The redirector does not have its own complete mirror.

Because of this, on the redirector cannot be verified against the local file system whether the URL refers to a folder or a file. At the same time, it is not possible to make a folder listing locally on the redirector. However, I have prepared a solution that could be good.

When the redirector verifies the availability of a specific file on a specific mirror, the http head is fetched. Folder listings returns in http head content-type text/html. Therefore, for content-type text/html I have added special behavior to the redirector. Instead of a redirection, the best content available for that url is loaded and then served directly from the redirector.

Source selection is as follows:

1) If the URL refers to a specific html file or a folder containing the index.html file that is available in the local cache on the redirector, this is served. This is useful, for example, for http://mirror.ppa.trinitydesktop.org/trinity/trinity/rpm/

2) If the primary server is available, content from the primary server is loaded and served. Primary server unavailability may be caused by either a network problem or a ban on firewall that is sometimes inappropriately active - see bug 2938.

3) The content is loaded and served from the primary mirror.

4) The content is loaded and served from a mirror selected according to the user's location.

In all cases, the content is served directly from the redirector, so references from these folder listings will again refer to the redirector. And that was the primary purpose of this solution.

Please try to see if it works as expected.

Mike, Slavek, 
what is the status of this bug now? Still valid?

This issue seems to be partially fixed.

For example, this link works as desired.

http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian

However clicking on dists in the resulting page yields a blank page but should yield a page on the redirector with content similar to this:

http://tde-mirror.yosemite.net/trinity/trinity-r14.0.0/debian/dists/

The problem appears to be that the redirector handles URLs ending "dists" different from URLs ending "dists/".

Thanks Mike

(In reply to mgb-trinity from comment #11)
> This issue seems to be partially fixed.
> 
> For example, this link works as desired.
> 
> http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian
> 
> However clicking on dists in the resulting page yields a blank page but
> should yield a page on the redirector with content similar to this:
> 
> http://tde-mirror.yosemite.net/trinity/trinity-r14.0.0/debian/dists/
> 
> The problem appears to be that the redirector handles URLs ending "dists"
> different from URLs ending "dists/".

A moment ago, I found that issue with links in the folder listing (are relative) occurs if the url does not contain the final slash.

For this address does not work:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian

For this address works:
http://mirror.ppa.trinitydesktop.org/trinity/trinity-r14.0.0/debian/

I will prepare a solution to this problem.

I edited the redirector code, the problem with the missing final slash for the folders now should be resolved!

Please verify that everything works as expected.

Successfully tested on directories with and without trailing slashes, regular files, and symlinks to each of the above.

(Not tested on dangling symlinks as there are none in the archive at this time but I don't think this is a significant case.)

However for missing files it gives a blank page instead of a 404.  This might confuse diagnosis of user problems e.g. failure to apt-get update before upgrade.

(In reply to mgb-trinity from comment #15)
> Successfully tested on directories with and without trailing slashes,
> regular files, and symlinks to each of the above.
> 
> (Not tested on dangling symlinks as there are none in the archive at this
> time but I don't think this is a significant case.)
> 
> However for missing files it gives a blank page instead of a 404.  This
> might confuse diagnosis of user problems e.g. failure to apt-get update
> before upgrade.

Yes, you are right - although the status code 404 is returned correctly, no page content is returned, which could be confusing to users. Fixed - now a simple html page is returned.

Looks good.

So it is ok to close now? OR something else remains to be fixed?

IMHO you can mark this bug fixed.

(In reply to mgb-trinity from comment #19)
> IMHO you can mark this bug fixed.

Mike, thank you for your initial idea and comments during testing!