Announcement

Collapse
No announcement yet.

Robots and site indexing problems

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Robots and site indexing problems

    I am in over my head on this one - about 1 months ago I submitted my forum to google search console for indexing. When I used to google "V6 Camaro Forum" I used to see my forum in the search results in various locations on the first page with my meta description for my main forum page....now today when I google it this is what I see:

    Click image for larger version

Name:	camaro  no description.PNG
Views:	93
Size:	6.7 KB
ID:	4390913

    When I click on the learn why it takes me to this page: https://support.google.com/webmaster.../7489871?hl=en

    Which essentially says this message means that Google could not access the page in order to create a useful description. When I click the"how do I fix this link" It starts talking about robot txt files are blocking this page and telling google not to crawl it.

    Similarly Google Console has come back and sent me a message saying new issue found: indexed, though blocked by robots.txt

    So - how can I correct this issue in cloud vbulletin? Is there an area I can access robot.txt files? I can't see how asking google to index the site would have negatively done something to my existing site - there must have been a setting off before?
    Your V6 Camaro information source
    How to fix deep scratches in your cars paint

    Vote for:

    Ability to display ads for xx number of views / impressions: http://tracker.vbulletin.com/browse/VBV-18045

  • #2
    That isn't a valid vBulletin Cloud URL. We require a three-part domain name including a sub-domain such as WWW at all times. You need to make sure that the domain above redirects properly and does not try to obfuscate the URL for the user and they see the three-part domain in their browser's address bar.

    The only pages blocked by robots.txt are search results and an old vB4 MAPI url. Search Results only last one hour and should never be indexed. The vB4 MAPI URL doesn't actually exist on Cloud but is used by the mobile apps. Here is the entire content of the robots.txt file on Cloud:
    Code:
    User-agent:*
    #we're blocking /search requests since those search pages are temporary anyway.  
    Disallow: /search
    #we're blocking /forumdisplay.php since it's a vB4 MAPI method.  
    Disallow: /forumdisplay.php
    The same file is used by all Cloud Sites.

    Make sure you don't have a robots.txt at camarov6.com which is outside our cloud network.
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud demonstration site.
    vBulletin 5 API - Full / Mobile
    Vote for your favorite feature requests and the bugs you want to see fixed.

    Comment


    • #3
      What isn't a valid Vbulletin cloud URL? The Camarov6.com? So in order to fix this I need to have anything without www. redirect to www.camarov6.com? Would that be with http:// or https://

      How can I ensure this is happening? : "You need to make sure that the domain above redirects properly and does not try to obfuscate the URL" Is there a test or way I can verify I have redirected everything properly?

      Make sure you don't have a robots.txt at camarov6.com which is outside our cloud network.
      Prior to about 24 hours ago I didn't know much about anything on robot texts- is there a way to verify this? or is it safe to assume that if I didn't create one and put it in my host domain file directory that it wouldn't exist?

      Sorry if any of the above are dumb questions.
      Your V6 Camaro information source
      How to fix deep scratches in your cars paint

      Vote for:

      Ability to display ads for xx number of views / impressions: http://tracker.vbulletin.com/browse/VBV-18045

      Comment


      • #4
        Yes, you need to redirect everything from camarov6.com to the HTTPS version of your cloud URL.

        When you type camarov6.com in the browser's address bar, it should change to the HTTPS version of your cloud URL.

        To see if there is a robots.txt file at your host domain directory you would have to log in via FTP or talk to the provider of the domain.
        Translations provided by Google.

        Wayne Luke
        The Rabid Badger - a vBulletin Cloud demonstration site.
        vBulletin 5 API - Full / Mobile
        Vote for your favorite feature requests and the bugs you want to see fixed.

        Comment


        • #5
          So I just confirmed with hostgator my provider and they confirmed that all of my urls:

          camarov6.com
          www.camarov6.com
          forum.camarov6.com
          www.forum.camarov6.com

          All redirect to https://www.camarov6.com

          So I am confused again - I also checked my files in my control panel by my provider and did not find any robot.txt files pertaining to my forum site...
          Your V6 Camaro information source
          How to fix deep scratches in your cars paint

          Vote for:

          Ability to display ads for xx number of views / impressions: http://tracker.vbulletin.com/browse/VBV-18045

          Comment


          • #6
            In Webmaster Tools you need to tell Google that the URL is https://www.camarov6.com. Not https://camarov6.com.

            Also, going to https://camarov6.com does not redirect me to https://www.camarov6.com.
            MARK.B | vBULLETIN SUPPORT

            TalkNewsUK - My vBulletin 5.5.6 Demo
            AdminAmmo - My Cloud Demo

            Comment


            • #7
              Thank you for pointing that out - I had to purchase an SSL certificate in order to redirect https://camarov6.com and now I have done that - that redirects to https://www.camarov6.com

              In Google Search Console I have specified my preferred domain is https://www.camarov6.com (which is the new web master tools)

              I am still showing the same result when I google "v6 camaro forum" in that I have no details to display- I'll give it a few days for google to reindex the site, I am hoping this will fix the issue?
              Your V6 Camaro information source
              How to fix deep scratches in your cars paint

              Vote for:

              Ability to display ads for xx number of views / impressions: http://tracker.vbulletin.com/browse/VBV-18045

              Comment


              • #8
                Have you submitted an update XML Sitemap with the new URL? That can speed up the process a bit though it can take 4-6 weeks for Google to update depending on the size of your site and the traffic it gets. Larger sites are updated more frequently.
                Translations provided by Google.

                Wayne Luke
                The Rabid Badger - a vBulletin Cloud demonstration site.
                vBulletin 5 API - Full / Mobile
                Vote for your favorite feature requests and the bugs you want to see fixed.

                Comment


                • #9
                  Originally posted by Wayne Luke View Post
                  Have you submitted an update XML Sitemap with the new URL? That can speed up the process a bit though it can take 4-6 weeks for Google to update depending on the size of your site and the traffic it gets. Larger sites are updated more frequently.
                  Do you mean rebuild a new one now that I have the additional https://camarov6.com redirected to https://www.camarov6.com?

                  Would that make a difference in the sitemap? If so I will do that next
                  Your V6 Camaro information source
                  How to fix deep scratches in your cars paint

                  Vote for:

                  Ability to display ads for xx number of views / impressions: http://tracker.vbulletin.com/browse/VBV-18045

                  Comment


                  • #10
                    You need to build and submit a site map for https://www.camarov6.com that is your site. https://camarov6.com is just a redirect. You want the content on your site indexed, not a redirect.
                    Translations provided by Google.

                    Wayne Luke
                    The Rabid Badger - a vBulletin Cloud demonstration site.
                    vBulletin 5 API - Full / Mobile
                    Vote for your favorite feature requests and the bugs you want to see fixed.

                    Comment

                    Related Topics

                    Collapse

                    Working...
                    X