Announcement

Collapse
No announcement yet.

robots.txt

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • robots.txt

    Google and Inktomi spiders were indexing my forum pages and using lots of bandwidth, so I included a robots.txt file in my root directory:

    User-agent: *
    Disallow: /forum/attachment.php
    Disallow: /forum/avatar.php
    Disallow: /forum/editpost.php
    Disallow: /forum/member.php
    Disallow: /forum/member2.php
    Disallow: /forum/misc.php
    Disallow: /forum/moderator.php
    Disallow: /forum/newreply.php
    Disallow: /forum/newthread.php
    Disallow: /forum/online.php
    Disallow: /forum/poll.php
    Disallow: /forum/postings.php
    Disallow: /forum/printthread.php
    Disallow: /forum/private.php
    Disallow: /forum/private2.php
    Disallow: /forum/report.php
    Disallow: /forum/search.php
    Disallow: /forum/sendtofriend.php
    Disallow: /forum/threadrate.php
    Disallow: /forum/usercp.php
    Disallow: /forum/admincp/
    Disallow: /forum/modcp/
    Disallow: /forum/images/
    Disallow: /forum/sendmessage.php
    Disallow: /forum/register.php
    Disallow: /forum/subscription.php

    I've noticed over the past few days that I'm still seeing Inktomi spiders, but I haven't seen even ONE Google spider (I used to have tons of them every day).

    Did I do something wrong in my robots.txt file? I just want to disallow stuff that doesn't matter to search engines, not the content.

  • #2
    They cache the robots.txt so it might take a while for them to realise that you dont want any of the traffic.
    Scott MacVicar

    My Blog | Twitter

    Comment


    • #3
      Why have you disallowed register.php

      Having your register page indexed could be beneficial.IMHO.
      Techzonez - Tech News
      Techzonez Forums - Tech Community

      Comment


      • #4
        Actually, I just took a list from someone else on this forum, I wasn't really sure what to allow and what not to allow.


        I just didn't want them spidering what they didn't need to, I still want the search engines to spider the forum.

        Comment


        • #5
          Ive never used these robot.txt files before, do i jsut creat a file in my root directory called robots.txt and paste that code in???

          I must seem like a dumbass!!!

          Comment


          • #6
            I've always been afraid to modify the robots.txt in fear that I would either send away the robot or disallow a wrong directory.

            Comment


            • #7
              Well ive made it now and there are like 3 bots online at once compared to the 1 google bot i usually had, odd!

              Comment


              • #8
                (Currently running patched 3.0.6)

                I just did a search on this forum for "bad google indexing" and ended up at this thread.

                I've just run a site:mysite.co.uk check on Google. It turns out that i have 45,000 pages indexed, which some might say is good, but a lot of the search results didn't even include a description. Looking like they're penalised.

                The majority of the the indexed pages are newreply.php, showmessage.php and showthread.php. Would it be worth adding these to a robots.txt file? Surely they're worthless from a seach engine point of view.

                What does the vBulletin team suggest?
                "To err is human; to really mess up, you need a computer."

                Comment


                • #9
                  Let the bot decide what to index IMO. A listing is a listing. Put internal link backs to your site on those pages or in the footer like the footer navbar links so the bots at least get a link to follow back in.
                  Computer Help Forum
                  An informed rider makes their first destination the motorcycle forum at rider info.

                  Comment


                  • #10
                    Good point, noppid.
                    "To err is human; to really mess up, you need a computer."

                    Comment


                    • #11
                      Call me captain overkill, but if I don't want search engines to find my site, couldn't I just disallow everything? I just did, we'll find out.. I'm pushing 50 gigs this month, 98% are .php files, lol..

                      Comment


                      • #12
                        I put a robots.txt in the forum directory as well as the root a few days ago and now Google is hunting or trying to view the admin and other members profiles. Can they see anything if I have it closed to the public?

                        The little stop sign is there which means: Viewing 'No Permission' Message but why do they still come around? Unless you are registered AND validated then added by the admin (me) you cannot see anything. I have tested it and tested it. Google and Ask Jeeves were on earlier today.

                        Thanks!
                        ~*Sara*~

                        Comment


                        • #13
                          The search engines will only be able to view and index what you allow Guests to view.
                          Steve Machol, former vBulletin Customer Support Manager (and NOT retired!)
                          Change CKEditor Colors to Match Style (for 4.1.4 and above)

                          Steve Machol Photography


                          Mankind is the only creature smart enough to know its own history, and dumb enough to ignore it.


                          Comment


                          • #14
                            Ok, thanks Steve. Just kinda freaked me out a little when they were viewing members and it showed names.
                            ~*Sara*~

                            Comment

                            widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                            Working...
                            X