Announcement

Collapse
No announcement yet.

Spiders crawling member URL's

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Spiders crawling member URL's

    All spiders are crawling member URL's like this:
    Code:
    /member/200-brent?200-Brent=&tab=visitor_messaging&page=150
    Is there a way to only let them crawl /member? Not quite sure what to put in my robots.txt.

  • #2
    Disallow: /*?*

    See: https://stackoverflow.com/questions/...fic-parameters
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API - Full / Mobile
    Vote for your favorite feature requests and the bugs you want to see fixed.

    Comment


    • #3
      Thanks Wayne!

      Comment

      widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
      Working...
      X