Hopefully finally sorted out the "http://www.xyz.cz?param" mess. The true

semantics turned out to be "http://www.xyz.cz/?param" and most web servers
really require "GET /?param".

I've changed the normalization rules to add the leading slash if needed
which also solves the relative URL problem I mentioned in the comments.

However, this means that the SEMANTICS OF NORMALIZED URL'S HAS CHANGED
and gatherer databases with URL's in the "http://www.xyz.cz?param" form
are now INVALID. I'm going to delete all such URL's from our gatherer now.

diff --git a/lib/url.c b/lib/url.c
index 105c7dd019e9f9d3c2de55a711b5cbd6eba390f8..cb577cb1625f7ad07e4adcf19fd32a406034bd08 100644 (file)
--- a/lib/url.c
+++ b/lib/url.c
@@ -11,9 +11,8 @@
  *
  *        o  Escaping of special characters still follows RFC 1738.
  *        o  Interpretation of path parameters follows RFC 1808.
- *        o  Parsing a relative URL "x" wrt. base "http://hell.org?y"
- *           gives an error, which might be wrong. However, I failed
- *           to find any rule applying to this case in the RFC.
+ *
+ *     XXX: The buffer handling in this module is really horrible, but it works.
  */
 
 #include "lib/lib.h"
@@ -396,6 +395,18 @@ url_normalize(struct url *u, struct url *b)
        }
     }
 
+  /* Change path "?" to "/?" because it's the true meaning */
+  if (u->rest[0] == '?')
+    {
+      int l = strlen(u->rest);
+      if (u->bufend - u->buf < l+1)
+       return URL_ERR_TOO_LONG;
+      u->buf[0] = '/';
+      memcpy(u->buf+1, u->rest, l+1);
+      u->rest = u->buf;
+      u->buf += l+2;
+    }
+
   /* Fill in missing info */
   if (u->port == ~0U)
     u->port = std_ports[u->protoid];

diff --git a/lib/url.h b/lib/url.h
index c01c1693f1fae26f16a7b057e9f86d8478adfd56..a9c4449876646fccff0dfc8766e54505d53ecb74 100644 (file)
--- a/lib/url.h
+++ b/lib/url.h
@@ -1,7 +1,7 @@
 /*
  *     Sherlock Library -- URL Functions
  *
- *     (c) 1997--2002 Martin Mares <mj@ucw.cz>
+ *     (c) 1997--2004 Martin Mares <mj@ucw.cz>
  *     (c) 2001 Robert Spalek <robert@ucw.cz>
  *
  *     This software may be freely distributed and used according to the terms