Announcement

Collapse
No announcement yet.

Spiders crawling member URL's

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Fleiding
    replied
    Thanks Wayne!

    Leave a comment:


  • Wayne Luke
    replied
    Disallow: /*?*

    See: https://stackoverflow.com/questions/...fic-parameters

    Leave a comment:


  • Fleiding
    started a topic Spiders crawling member URL's

    Spiders crawling member URL's

    All spiders are crawling member URL's like this:
    Code:
    /member/200-brent?200-Brent=&tab=visitor_messaging&page=150
    Is there a way to only let them crawl /member? Not quite sure what to put in my robots.txt.
widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
Working...
X