Announcement

Collapse
No announcement yet.

Are vBulletin "cron jobs" executed for (all?) non-logged in HTTP requests to a board?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Are vBulletin "cron jobs" executed for (all?) non-logged in HTTP requests to a board?

    I have a vBulletin 5 Connect board with very low daily visiting activity, but relatively many subscription notifications (emails) that should be sent out quickly once something is posted.

    Since such subscription notification emails are being queued up and sent using vBulletin "cron jobs" (right?), and such "cron jobs", as far as I have understood, are being triggered as "piggy-backed" hooks on other visitors' HTTP requests to all(?) normal PHP scripts of a board, I'm interested in triggering such requests myself using repeated automated HTTP requests to my board from another server that I manage.

    My questions about this are therefore:

    Are there some specific criteria that need to be fulfilled for these automated HTTP requests of mine made to the board, in order to trigger the execution of such cron jobs, for example that the HTTP requests need to be made only using authenticated sessions (i.e. by logged-in users), or that they need to be of the POST type rather than the GET type (e.g. submitting a new post, but NOT simply listing a forum index), or would it suffice for my script to simply request the forum index page as a non-logged in user every 5 minutes in order to make sure that all my vBulletin cron jobs are reliably triggered on a continuous basis (and therefore also all my subscription notification emails sent out in a reliable and quick fashion)?
    Last edited by vbSuperfan; Tue 25th Jun '19, 3:20am.

  • #2
    Each Scheduled Task has its own schedule as defined in the AdminCP. When a user visits your site, exactly one Scheduled Task will trigger if it is the time for it. If you have low traffic, this can cause a backlog of tasks. To counteract this, we have provided a file named vbcron.php in your do_not_upload folder. You can upload this to your server and set it up in your server's task system (crontab on linux, scheduled tasks in Windows) so that it is triggered every minute. Instructions are in the comments within the file. This removes the Scheduled Task system from user visits and ensures a smoother operation.

    Small sites will probably want to turn off the email queue system as well. This is only helpful if your provider meters your emails. Most don't or allow SMTP without metering. You can turn off the email queue in the AdminCP under Settings -> Options -> Email Settings. You can also switch to SMTP (the preferred method of sending email) here as well.
    Last edited by Wayne Luke; Tue 25th Jun '19, 7:38am.
    Translations provided by Google.

    Wayne Luke
    The Rabid Badger - a vBulletin Cloud customization and demonstration site.
    vBulletin 5 Documentation - Updated every Friday. Report issues here.
    vBulletin 5 API - Full / Mobile
    I am not currently available for vB Messenger Chats.

    Comment


    • #3
      Thanks for this information!

      My situation is unfortunately a little special though...

      Even though my board is relatively small (~200 users), all of these users are subscribed to multiple forums, so there will quite often be multiple batches of 200 emails (and Firebase push notifications) that are to be sent. I'm therefore guessing that the email queue is still a good thing to use, rather than sending all of the emails and Firebase push notifications at once?

      Also, the forum is not on a VPS, but rather simply on "web hosting", where I don't have access to things like the operating system cron functionality (i.e. I don't have shell access, but rather just FTP access). Therefore, the vbcron.php solution that you mention is unfortunately not an option either.

      This brings me back to my original question, in regards to my "poor man's cron automation", by using scripted requests to my board from another server that I manage, with the intention of in turn triggering the "Scheduled Tasks" in vBulletin:

      Are there any particular requests to the board that will or will not trigger the queued up "Scheduled Tasks" in vBulletin, or will any kind of request to the board whatsoever suffice?

      More specifically:

      1.
      Can I use normal non-logged in requests to the board, and it will still trigger the execution of queued up "Scheduled Tasks"?

      2.
      Will any simple request to the board do, like e.g. reloading the main page or some forum index page, or will only some specific targets for my automated requests have any effect for executing queued up "Scheduled Tasks", and in that case, more specifically which requests would that be?

      Comment


      • #4
        Originally posted by vbSuperfan View Post
        Even though my board is relatively small (~200 users), all of these users are subscribed to multiple forums, so there will quite often be multiple batches of 200 emails (and Firebase push notifications) that are to be sent. I'm therefore guessing that the email queue is still a good thing to use, rather than sending all of the emails and Firebase push notifications at once?
        Firebase Notifications are not involved with the Email Queue and have nothing to do with outgoing emails. Really, on most systems it is best to let the email handler queue the system. With the default settings, vBulletin sends 10 emails every 10 minutes using the queue system.

        Also, the forum is not on a VPS, but rather simply on "web hosting", where I don't have access to things like the operating system cron functionality (i.e. I don't have shell access, but rather just FTP access). Therefore, the vbcron.php solution that you mention is unfortunately not an option either.
        Most hosting plans offer access to Crontab either through SSH or cPanel. This is not a VPS feature.

        This brings me back to my original question, in regards to my "poor man's cron automation", by using scripted requests to my board from another server that I manage, with the intention of in turn triggering the "Scheduled Tasks" in vBulletin:

        Are there any particular requests to the board that will or will not trigger the queued up "Scheduled Tasks" in vBulletin, or will any kind of request to the board whatsoever suffice?

        More specifically:

        1.
        Can I use normal non-logged in requests to the board, and it will still trigger the execution of queued up "Scheduled Tasks"?

        2.
        Will any simple request to the board do, like e.g. reloading the main page or some forum index page, or will only some specific targets for my automated requests have any effect for executing queued up "Scheduled Tasks", and in that case, more specifically which requests would that be?

        All it takes to trigger a Scheduled Task is to load the page. One Scheduled Task will run per page load every 5 minutes. It isn't rocket science.
        Translations provided by Google.

        Wayne Luke
        The Rabid Badger - a vBulletin Cloud customization and demonstration site.
        vBulletin 5 Documentation - Updated every Friday. Report issues here.
        vBulletin 5 API - Full / Mobile
        I am not currently available for vB Messenger Chats.

        Comment


        • #5
          Originally posted by Wayne Luke View Post
          Firebase Notifications are not involved with the Email Queue and have nothing to do with outgoing emails.
          Yes, I know, but Firebase Notifications still take processing to send, and also make use of the Scheduled Tasks in vBulletin, that's what I meant.


          Originally posted by Wayne Luke View Post
          Really, on most systems it is best to let the email handler queue the system. With the default settings, vBulletin sends 10 emails every 10 minutes using the queue system.
          That's exactly what I'm doing, but the entire problem is that this requires active visitors, since it makes use of the Scheduled Tasks in vBulletin.


          Originally posted by Wayne Luke View Post
          Most hosting plans offer access to Crontab either through SSH or cPanel. This is not a VPS feature.
          SSH shell access is indeed in my experience a VPS feature, but I will look into the cPanel cron possibilities though, thanks.


          Originally posted by Wayne Luke View Post
          All it takes to trigger a Scheduled Task is to load the page. One Scheduled Task will run per page load every 5 minutes. It isn't rocket science.
          Ok, I take it then this means that I can have a script that makes simple (non-logged in) HTTP requests to my forum index every 5 minutes, in order to continuously trigger the Scheduled Tasks in vBulletin. Good, thanks!

          Comment


          • #6
            Originally posted by vbSuperfan View Post
            Ok, I take it then this means that I can have a script that makes simple (non-logged in) HTTP requests to my forum index every 5 minutes, in order to continuously trigger the Scheduled Tasks in vBulletin. Good, thanks!
            Yes. This basically what the vbcron.php script does.
            Translations provided by Google.

            Wayne Luke
            The Rabid Badger - a vBulletin Cloud customization and demonstration site.
            vBulletin 5 Documentation - Updated every Friday. Report issues here.
            vBulletin 5 API - Full / Mobile
            I am not currently available for vB Messenger Chats.

            Comment


            • #7
              You will need to automatically make the HTTP requests using a physical browser or a headless browser, otherwise, if you'll just do it via cURL or something similar, it would just give you the plain HTML output of the page but not execute the Javascript on the page. The Scheduled Task is triggered by an AJAX Javascript call that is executed by the browser on every page load.

              GIPHY for vB5 AutoLinker Auto-Create Flag Report Topic Social Icons in Postbit Clear Cache Cron DragDrop Upload Topic AJAX AutoUpdate Custom Avatars Selector Stop Links in Posts...and more!

              Comment


              • #8
                Originally posted by Glenn Vergara View Post
                You will need to automatically make the HTTP requests using a physical browser or a headless browser, otherwise, if you'll just do it via cURL or something similar, it would just give you the plain HTML output of the page but not execute the Javascript on the page. The Scheduled Task is triggered by an AJAX Javascript call that is executed by the browser on every page load.
                Thanks Glenn! I guess there was a bit of "rocket science" to it after all then...

                You (or anyone from the vBulletin support) wouldn't happen to know the target URL of that AJAX call (i.e. that triggers the actual Scheduled Tasks processing), would you?

                In that case I could just make the cURL request to that URL instead (I'm guessing that the whole AJAX design is just to avoid affecting the loading time of the page itself that the user is watching, and therefore instead making this background call that won't bother anyone even if it would take 5-10 seconds to return, right?).

                Comment


                • #9
                  Loading your site in a browser will trigger the AJAX call. You cannot call the AJAX script directly from the address bar of a browser or script.

                  To trigger a Scheduled Task on this site, you would point a browser to https://forum.vbulletin.com/

                  To call the API directly, you would use vbcron.php to do so.
                  Translations provided by Google.

                  Wayne Luke
                  The Rabid Badger - a vBulletin Cloud customization and demonstration site.
                  vBulletin 5 Documentation - Updated every Friday. Report issues here.
                  vBulletin 5 API - Full / Mobile
                  I am not currently available for vB Messenger Chats.

                  Comment


                  • #10
                    Originally posted by Wayne Luke View Post
                    Loading your site in a browser will trigger the AJAX call.
                    Sure, I understand that, but the problem is that I cannot automate this from an actual browser, but rather only from a script running in a shell on another server.


                    Originally posted by Wayne Luke View Post
                    You cannot call the AJAX script directly from the address bar of a browser or script.
                    It is indeed true that all AJAX HTTP requests cannot be mimiced directly from the address bar of a browser, but on the contrary, they can always be exactly replicated from a script using e.g. cURL or Python.


                    Originally posted by Wayne Luke View Post
                    To call the API directly, you would use vbcron.php to do so.
                    Ok, so you're saying that I can put this vbcron.php script in the vBulletin root dir in my web root, and then simply request it from the internet (like so: http://www.mydomain.com/forum/vbcron.php), and the Scheduled Tasks will be triggered? That would be really great if it was so!

                    Comment


                    • #11
                      vbcron.php will only work from a command line via crontab. You cannot call APIs directly as a web address due to security concerns. You cannot call the page via cURL because Javascript won't be processed. You may be able to do it with Python but if you can call a Python script from the command line, you can call vbcron.php from a command line.
                      Translations provided by Google.

                      Wayne Luke
                      The Rabid Badger - a vBulletin Cloud customization and demonstration site.
                      vBulletin 5 Documentation - Updated every Friday. Report issues here.
                      vBulletin 5 API - Full / Mobile
                      I am not currently available for vB Messenger Chats.

                      Comment


                      • #12
                        Originally posted by Wayne Luke View Post
                        You cannot call the page via cURL because Javascript won't be processed. You may be able to do it with Python but if you can call a Python script from the command line, you can call vbcron.php from a command line.
                        AJAX works like this:

                        1.
                        Javascript code is loaded (by the currently displayed web page) in the browser, on the client-side.

                        2.
                        This loaded Javascript code does some processing on the client-side, and then makes an HTTP request to the web server, in turn resulting in some processing on the server-side [this is where the Scheduled Tasks are executed], in turn normally also returning some kind of response to the client-side in the end.

                        3.
                        The data in the response is processed by the Javascript code on the client-side, normally finally resulting in the Javascript dynamically updating the DOM of the currently displayed web page in the browser on the client-side.


                        Thus, if I only want to achieve the processing in question on the server-side (see my bold text above), all that is necessary is to mimic the HTTP request being made by the Javascript in step 2 above, while ignoring/bypassing everything else that the Javascript does (both before and after it performs this HTTP request).

                        Thus, it is, again, indeed possible to do this, from the client-side, if I just know the target + parameters of the HTTP request being made by the Javascript code (i.e. the "AJAX request"), while at the same time not needing to load or execute any Javascript code whatsoever.

                        Believe me.

                        (and regarding the Python script that you commented, I'm not talking about a server-side Python script (which would indeed be pointless, just like you say), but rather a client-side Python script, which, just like cURL, can be used to mimic any HTTP request that a web browser can perform, including any such HTTP request being performed from Javascript code)

                        Having sorted that out, I would again just need to know which AJAX HTTP request that triggers the Scheduled Tasks on the server-side?

                        The only two AJAX requests I see when loading the vBulletin.com forum index are "https://forum.vbulletin.com/ajax/api...ns/fetchValues" and "https://forum.vbulletin.com/ajax/api/phrase/getPhrases". But then again, perhaps you are using the vbcron.php setup instead, which perhaps eliminates this AJAX call you are talking about altogether? Is that the case?

                        Any and all input from Mr vBulletin-low-level-ninja-knowledge himself, Glenn Vergara, here would be super appreciated!



                        EDIT:
                        Seems like I was right about the AJAX call not being present on vbulletin.com. I just took a deeper look at the HTTP traffic for my own forum, and found this AJAX HTTP call, which is very likely to be the one to call to trigger the processing of the Scheduled tasks:

                        https://www.mydomain.com/forum/ajax/apidetach/cron/run

                        And just to even more prove my point about the possibility to 100% mimic this call using cURL, here is the exact command-line to do this:

                        curl "https://www.mydomain.com/forum/ajax/apidetach/cron/run" -H "Accept: application/json, text/javascript, */*; q=0.01" -H "Referer: https://www.mydomain.com/forum/" -H "Origin: https://www.mydomain.com" -H "X-Requested-With: XMLHttpRequest" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" --data "securitytoken=guest" --compressed

                        Last edited by vbSuperfan; Wed 26th Jun '19, 3:36pm.

                        Comment


                        • #13
                          Originally posted by vbSuperfan View Post
                          Seems like I was right about the AJAX call not being present on vbulletin.com. I just took a deeper look at the HTTP traffic for my own forum, and found this AJAX HTTP call, which is very likely to be the one to call to trigger the processing of the Scheduled tasks:

                          https://www.mydomain.com/forum/ajax/apidetach/cron/run

                          And just to even more prove my point about the possibility to 100% mimic this call using cURL, here is the exact command-line to do this:

                          curl "https://www.mydomain.com/forum/ajax/apidetach/cron/run" -H "Accept: application/json, text/javascript, */*; q=0.01" -H "Referer: https://www.mydomain.com/forum/" -H "Origin: https://www.mydomain.com" -H "X-Requested-With: XMLHttpRequest" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" --data "securitytoken=guest" --compressed

                          You need to make it a POST request.

                          GIPHY for vB5 AutoLinker Auto-Create Flag Report Topic Social Icons in Postbit Clear Cache Cron DragDrop Upload Topic AJAX AutoUpdate Custom Avatars Selector Stop Links in Posts...and more!

                          Comment


                          • #14
                            I must be missing something. What is the point to jumping through all these hoops? Why is the scheduled tasks running every 60 seconds with the vbcron.php file insufficient?

                            Comment


                            • #15
                              Originally posted by Glenn Vergara View Post

                              You need to make it a POST request.
                              Thanks, but the --data parameter already does/implies that.

                              cURL only requires you to explicitly specify the POST method (using the -X / --request parameter) if the request contains no parameter data.

                              Comment

                              Related Topics

                              Collapse

                              Working...
                              X