Announcement

Collapse
No announcement yet.

Well that was fun... (About downtime)

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Well that was fun... (About downtime)

    I got a call from Kier who was out, saying that ThePlanet said our main load balancer server was down.

    Sure enough, I couldn't SSH to it from the external NIC, but I tried one of the backend servers and was able to get in via the internal NIC.

    Logged in to Orbit (ThePlanet's support system) and saw 2 tickets. One about the downtime, and one saying that they noticed malicious traffic and enabled FloodGuard, DDoS protection.

    Well, this is around the time when everything went pear-shaped. One of their techs said:
    Your server was being hit with a 1.5gbps inbound attack.
    (Emphasis mine.)
    Err... wow. Though apparently it stopped so they disabled FloodGuard.

    When I was in via the internal NIC, I was able to run ethtool to see that there was no link detected on the external NIC. I spent the last few hours trying to get ThePlanet to look at the switch we're connected to.

    And now it's finally up. But just before it came up, they said this in a ticket:
    We are still investigating this issue at this point we cannot determine whether it is a bad switch port, or bad nic.
    We are taking the server down to install an extra nic to see if this can fix the issue.
    -Support
    Fun... Let's hope they don't take the server down.

  • #2
    Wow... Busy morning
    Sound District

    Comment


    • #3
      Didn't realise vBulletin.org was on the same servers as these =D

      Comment


      • #4
        well, ouch
        Peggy
        ~ normal is overrated ~

        One Buzy Mama!

        Comment


        • #5
          Originally posted by Shazz™ View Post
          Wow... Busy morning
          Except it was late afternoon and then evening over here

          Originally posted by Yours Truly View Post
          Didn't realise vBulletin.org was on the same servers as these =D
          Yep, has been for years.
          Baby, I was born this way

          Comment


          • #6
            Originally posted by Paul M View Post
            Except it was late afternoon and then evening over here

            Yep, has been for years.
            I thought it was a different part of vBulletin all together =]

            Comment


            • #7
              DDOS'd? Man that sucks
              Plan, Do, Check, Act!

              Comment


              • #8
                Originally posted by Paul M View Post
                Except it was late afternoon and then evening over here

                Yep, has been for years.
                It was busy for me too, I kept refreshing vb.org 1000 times
                Sound District

                Comment


                • #9
                  Originally posted by Shazz� View Post
                  It was busy for me too, I kept refreshing vb.org 1000 times
                  LOL!!
                  Peggy
                  ~ normal is overrated ~

                  One Buzy Mama!

                  Comment


                  • #10
                    That was a very annoying and LONG down time :P.

                    Comment


                    • #11
                      From 5 to 9 .. four hours. It felt even longer *my precious*

                      Comment


                      • #12
                        Originally posted by Floris View Post
                        From 5 to 9 .. four hours. It felt even longer *my precious*
                        Agreed .

                        Comment


                        • #13
                          I've received various concerns and comments from people if this was in relation with the latest IB news: No, it was not. It has nothing to do with IB, and we're not moving servers to a different provider or anything like that. I hope this answers those questions.

                          Comment


                          • #14
                            Well, at least we're back up and running here.
                            Congratulations on the death of vBulletin, Internet Brands.

                            Comment


                            • #15
                              need a new F5 key here

                              Comment

                              widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                              Working...
                              X