Melimundo Web Crawler
Information for website owners
About Our Web Crawler
The Melimundo web crawler (also known as a bot or spider) periodically visits websites to gather and update information about travel regulations, visa requirements, and immigration processes. This helps us maintain accurate and up-to-date information for travelers using our services.
How to Identify Our Crawler
Our crawler identifies itself in the User-Agent string as:
MelimundoBot/1.0 (+https://melimundo.com/bot)
If you see this User-Agent in your server logs, it means our crawler has visited your site.
Our User-Agent Guidelines
When designing our crawler, we follow these best practices:
- Be Transparent: Our User-Agent clearly identifies itself as a crawler/bot.
- Provide Contact Information: We include our website URL where administrators can find more information and contact details.
- Respect Robots.txt: Our crawler strictly adheres to the rules specified in a site's robots.txt file to avoid unwanted crawling.
What We Collect
-
Visa Requirements: Information about necessary visas for different nationalities and destinations.
-
Application Processes: Steps and procedures for visa applications in different countries.
-
Document Requirements: Lists of required documents for different visa applications.
-
Travel Regulations: Current travel restrictions, requirements, and procedures.
-
Immigration Information: Updates on immigration policies and procedures.
Our Crawler Policies
-
We respect robots.txt: Our crawler adheres to the robots.txt protocol and will honor any crawling restrictions you've specified.
-
Reasonable crawl rate: We maintain a reasonable crawl rate to avoid overloading your servers.
-
Attribution: When displaying information gathered from your site, we provide proper attribution with links back to the source.
-
Data freshness: We periodically revisit sites to ensure our information remains current and accurate.
Controlling Our Crawler's Access
If you wish to control how our crawler accesses your site, you can use the following methods:
Using robots.txt
To block our crawler completely, add these lines to your robots.txt file:
User-agent: MelimundoBot
Disallow: /
To block specific sections:
User-agent: MelimundoBot
Disallow: /private-section/
Allow: /public-info/
Example robots.txt File
Here's an example of a typical robots.txt file structure that blocks specific system directories while allowing most content to be crawled:
User-agent: *
Disallow: /_layouts/
Disallow: /_vti_bin/
Disallow: /_catalogs/
Sitemap: https://www.example.com/sitemap.xml
Note: This example shows how to block all crawlers from accessing certain directories and points to a sitemap file.
Contact Us
If you have questions, concerns, or would like to request changes to how our crawler interacts with your site, please contact us at:
Email: [email protected]
Please include your domain name and specific concerns in your message.