Frequently asked questions
Q: Will My Sitemaps Crawler recognize and obey rules in robots.txt files though intended for other crawlers?
Yes and No - My Sitemaps Crawler obeys rules targeted at the User-agent: mysitemaps. If this is not found, it will try to recognize rules targeted at the most well known search engines (mainly Google, Yahoo! and Microsoft) and obey those rules.
Q: Will My Sitemaps Crawler recognize and use the "allow" rule in robots.txt files?
Yes it will.
Q: Why is frame / iframe urls included in the sitemap.xml?
The search engines want to know about those too.
Q: Where do I put my sitemap.xml file?
If your sites url are http://mysite.com/ then you need to upload the sitemap.xml so that you can access it from http://mysite.com/sitemap.xml - if your sites url are http://mysite.com/mypath/ - then it has to be accessible from http://mysite.com/mypath/sitemap.xml
Q: Why do I have to register?
For convenience - once you are signed up, we can remember more than one site and have your sitemaps saved for you.
Q: Why do I have to provide a title for the pages found?
You don't! It is just to make it easier for you, if you for instance have urls that can be hard to recognise - like these: index.php?Page=1 and index.php?Page=2
Q: Do you have a privacy policy?
Yes - We collect statistical data but do not connect these to any persons data. Data gathered that can identify a person, will be held confident and will not be disclosed or sold to any third party.
Q: What do I do, if my sitemap contains more than 50.000 links?
You create a sitemap pr. directory. You can create a new site with the url:
http://www.mysite.com/my_directory/ and create a sitemap for that path - and a site with the url:
http://www.mysite.com/my_other_directory/ and create a sitemap for that path. When you have done this, you put each sitemap file in to those directories, so they are accessible from:
http://www.mysite.com/my_directory/sitemap.xml and
http://www.mysite.com/my_other_directory/sitemap.xml - when done with this, you can create an index file, that looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>http://www.mysite.com/my_directory/sitemap.xml</loc>
<lastmod>2004-10-01T18:23:17+00:00</lastmod>
</sitemap>
<sitemap>
<loc>http://www.mysite.com/my_other_directory/sitemap.xml</loc>
<lastmod>2005-01-01</lastmod>
</sitemap>
</sitemapindex>
This is also described here:
sitemaps.org sitemap index FAQ.
Q: Why does my url contain stuff like & instead of &?
.
Last updated: 27 May 2010
Also visit: Show IP
