Announcement

Collapse
No announcement yet.

Are vBulletin "cron jobs" executed for (all?) non-logged in HTTP requests to a board?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Wayne Luke
    replied
    Originally posted by vbSuperfan View Post

    What you're missing is that all web hosting packages don't allow you to execute server-side scripts like that, but rather simply provide you with FTP access and that's it. In that case you cannot use the vbcron.php file, and instead you have to McGyver yourself a script-based client-side-only solution to automate the vBulletin Scheduled Tasks processing, which is what we just did here.
    I'd find a new hosting provider. I have had SSH and Crontab support on every hosting provider that I have had in the last 20 years.

    Leave a comment:


  • vbSuperfan
    replied
    Originally posted by In Omnibus View Post
    I must be missing something. What is the point to jumping through all these hoops? Why is the scheduled tasks running every 60 seconds with the vbcron.php file insufficient?
    What you're missing is that all web hosting packages don't allow you to execute server-side scripts like that, but rather simply provide you with FTP access and that's it. In that case you cannot use the vbcron.php file, and instead you have to McGyver yourself a script-based client-side-only solution to automate the vBulletin Scheduled Tasks processing, which is what we just did here.

    Leave a comment:


  • vbSuperfan
    replied
    Originally posted by Glenn Vergara View Post

    You need to make it a POST request.
    Thanks, but the --data parameter already does/implies that.

    cURL only requires you to explicitly specify the POST method (using the -X / --request parameter) if the request contains no parameter data.

    Leave a comment:


  • In Omnibus
    replied
    I must be missing something. What is the point to jumping through all these hoops? Why is the scheduled tasks running every 60 seconds with the vbcron.php file insufficient?

    Leave a comment:


  • Glenn Vergara
    replied
    Originally posted by vbSuperfan View Post
    Seems like I was right about the AJAX call not being present on vbulletin.com. I just took a deeper look at the HTTP traffic for my own forum, and found this AJAX HTTP call, which is very likely to be the one to call to trigger the processing of the Scheduled tasks:

    https://www.mydomain.com/forum/ajax/apidetach/cron/run

    And just to even more prove my point about the possibility to 100% mimic this call using cURL, here is the exact command-line to do this:

    curl "https://www.mydomain.com/forum/ajax/apidetach/cron/run" -H "Accept: application/json, text/javascript, */*; q=0.01" -H "Referer: https://www.mydomain.com/forum/" -H "Origin: https://www.mydomain.com" -H "X-Requested-With: XMLHttpRequest" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" --data "securitytoken=guest" --compressed

    You need to make it a POST request.

    Leave a comment:


  • vbSuperfan
    replied
    Originally posted by Wayne Luke View Post
    You cannot call the page via cURL because Javascript won't be processed. You may be able to do it with Python but if you can call a Python script from the command line, you can call vbcron.php from a command line.
    AJAX works like this:

    1.
    Javascript code is loaded (by the currently displayed web page) in the browser, on the client-side.

    2.
    This loaded Javascript code does some processing on the client-side, and then makes an HTTP request to the web server, in turn resulting in some processing on the server-side [this is where the Scheduled Tasks are executed], in turn normally also returning some kind of response to the client-side in the end.

    3.
    The data in the response is processed by the Javascript code on the client-side, normally finally resulting in the Javascript dynamically updating the DOM of the currently displayed web page in the browser on the client-side.


    Thus, if I only want to achieve the processing in question on the server-side (see my bold text above), all that is necessary is to mimic the HTTP request being made by the Javascript in step 2 above, while ignoring/bypassing everything else that the Javascript does (both before and after it performs this HTTP request).

    Thus, it is, again, indeed possible to do this, from the client-side, if I just know the target + parameters of the HTTP request being made by the Javascript code (i.e. the "AJAX request"), while at the same time not needing to load or execute any Javascript code whatsoever.

    Believe me.

    (and regarding the Python script that you commented, I'm not talking about a server-side Python script (which would indeed be pointless, just like you say), but rather a client-side Python script, which, just like cURL, can be used to mimic any HTTP request that a web browser can perform, including any such HTTP request being performed from Javascript code)

    Having sorted that out, I would again just need to know which AJAX HTTP request that triggers the Scheduled Tasks on the server-side?

    The only two AJAX requests I see when loading the vBulletin.com forum index are "https://forum.vbulletin.com/ajax/api...ns/fetchValues" and "https://forum.vbulletin.com/ajax/api/phrase/getPhrases". But then again, perhaps you are using the vbcron.php setup instead, which perhaps eliminates this AJAX call you are talking about altogether? Is that the case?

    Any and all input from Mr vBulletin-low-level-ninja-knowledge himself, Glenn Vergara, here would be super appreciated!



    EDIT:
    Seems like I was right about the AJAX call not being present on vbulletin.com. I just took a deeper look at the HTTP traffic for my own forum, and found this AJAX HTTP call, which is very likely to be the one to call to trigger the processing of the Scheduled tasks:

    https://www.mydomain.com/forum/ajax/apidetach/cron/run

    And just to even more prove my point about the possibility to 100% mimic this call using cURL, here is the exact command-line to do this:

    curl "https://www.mydomain.com/forum/ajax/apidetach/cron/run" -H "Accept: application/json, text/javascript, */*; q=0.01" -H "Referer: https://www.mydomain.com/forum/" -H "Origin: https://www.mydomain.com" -H "X-Requested-With: XMLHttpRequest" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" --data "securitytoken=guest" --compressed

    Last edited by vbSuperfan; Wed 26 Jun '19, 3:36pm.

    Leave a comment:


  • Wayne Luke
    replied
    vbcron.php will only work from a command line via crontab. You cannot call APIs directly as a web address due to security concerns. You cannot call the page via cURL because Javascript won't be processed. You may be able to do it with Python but if you can call a Python script from the command line, you can call vbcron.php from a command line.

    Leave a comment:


  • vbSuperfan
    replied
    Originally posted by Wayne Luke View Post
    Loading your site in a browser will trigger the AJAX call.
    Sure, I understand that, but the problem is that I cannot automate this from an actual browser, but rather only from a script running in a shell on another server.


    Originally posted by Wayne Luke View Post
    You cannot call the AJAX script directly from the address bar of a browser or script.
    It is indeed true that all AJAX HTTP requests cannot be mimiced directly from the address bar of a browser, but on the contrary, they can always be exactly replicated from a script using e.g. cURL or Python.


    Originally posted by Wayne Luke View Post
    To call the API directly, you would use vbcron.php to do so.
    Ok, so you're saying that I can put this vbcron.php script in the vBulletin root dir in my web root, and then simply request it from the internet (like so: http://www.mydomain.com/forum/vbcron.php), and the Scheduled Tasks will be triggered? That would be really great if it was so!

    Leave a comment:


  • Wayne Luke
    replied
    Loading your site in a browser will trigger the AJAX call. You cannot call the AJAX script directly from the address bar of a browser or script.

    To trigger a Scheduled Task on this site, you would point a browser to https://forum.vbulletin.com/

    To call the API directly, you would use vbcron.php to do so.

    Leave a comment:


  • vbSuperfan
    replied
    Originally posted by Glenn Vergara View Post
    You will need to automatically make the HTTP requests using a physical browser or a headless browser, otherwise, if you'll just do it via cURL or something similar, it would just give you the plain HTML output of the page but not execute the Javascript on the page. The Scheduled Task is triggered by an AJAX Javascript call that is executed by the browser on every page load.
    Thanks Glenn! I guess there was a bit of "rocket science" to it after all then...

    You (or anyone from the vBulletin support) wouldn't happen to know the target URL of that AJAX call (i.e. that triggers the actual Scheduled Tasks processing), would you?

    In that case I could just make the cURL request to that URL instead (I'm guessing that the whole AJAX design is just to avoid affecting the loading time of the page itself that the user is watching, and therefore instead making this background call that won't bother anyone even if it would take 5-10 seconds to return, right?).

    Leave a comment:


  • Glenn Vergara
    replied
    You will need to automatically make the HTTP requests using a physical browser or a headless browser, otherwise, if you'll just do it via cURL or something similar, it would just give you the plain HTML output of the page but not execute the Javascript on the page. The Scheduled Task is triggered by an AJAX Javascript call that is executed by the browser on every page load.

    Leave a comment:


  • Wayne Luke
    replied
    Originally posted by vbSuperfan View Post
    Ok, I take it then this means that I can have a script that makes simple (non-logged in) HTTP requests to my forum index every 5 minutes, in order to continuously trigger the Scheduled Tasks in vBulletin. Good, thanks!
    Yes. This basically what the vbcron.php script does.

    Leave a comment:


  • vbSuperfan
    replied
    Originally posted by Wayne Luke View Post
    Firebase Notifications are not involved with the Email Queue and have nothing to do with outgoing emails.
    Yes, I know, but Firebase Notifications still take processing to send, and also make use of the Scheduled Tasks in vBulletin, that's what I meant.


    Originally posted by Wayne Luke View Post
    Really, on most systems it is best to let the email handler queue the system. With the default settings, vBulletin sends 10 emails every 10 minutes using the queue system.
    That's exactly what I'm doing, but the entire problem is that this requires active visitors, since it makes use of the Scheduled Tasks in vBulletin.


    Originally posted by Wayne Luke View Post
    Most hosting plans offer access to Crontab either through SSH or cPanel. This is not a VPS feature.
    SSH shell access is indeed in my experience a VPS feature, but I will look into the cPanel cron possibilities though, thanks.


    Originally posted by Wayne Luke View Post
    All it takes to trigger a Scheduled Task is to load the page. One Scheduled Task will run per page load every 5 minutes. It isn't rocket science.
    Ok, I take it then this means that I can have a script that makes simple (non-logged in) HTTP requests to my forum index every 5 minutes, in order to continuously trigger the Scheduled Tasks in vBulletin. Good, thanks!

    Leave a comment:


  • Wayne Luke
    replied
    Originally posted by vbSuperfan View Post
    Even though my board is relatively small (~200 users), all of these users are subscribed to multiple forums, so there will quite often be multiple batches of 200 emails (and Firebase push notifications) that are to be sent. I'm therefore guessing that the email queue is still a good thing to use, rather than sending all of the emails and Firebase push notifications at once?
    Firebase Notifications are not involved with the Email Queue and have nothing to do with outgoing emails. Really, on most systems it is best to let the email handler queue the system. With the default settings, vBulletin sends 10 emails every 10 minutes using the queue system.

    Also, the forum is not on a VPS, but rather simply on "web hosting", where I don't have access to things like the operating system cron functionality (i.e. I don't have shell access, but rather just FTP access). Therefore, the vbcron.php solution that you mention is unfortunately not an option either.
    Most hosting plans offer access to Crontab either through SSH or cPanel. This is not a VPS feature.

    This brings me back to my original question, in regards to my "poor man's cron automation", by using scripted requests to my board from another server that I manage, with the intention of in turn triggering the "Scheduled Tasks" in vBulletin:

    Are there any particular requests to the board that will or will not trigger the queued up "Scheduled Tasks" in vBulletin, or will any kind of request to the board whatsoever suffice?

    More specifically:

    1.
    Can I use normal non-logged in requests to the board, and it will still trigger the execution of queued up "Scheduled Tasks"?

    2.
    Will any simple request to the board do, like e.g. reloading the main page or some forum index page, or will only some specific targets for my automated requests have any effect for executing queued up "Scheduled Tasks", and in that case, more specifically which requests would that be?

    All it takes to trigger a Scheduled Task is to load the page. One Scheduled Task will run per page load every 5 minutes. It isn't rocket science.

    Leave a comment:


  • vbSuperfan
    replied
    Thanks for this information!

    My situation is unfortunately a little special though...

    Even though my board is relatively small (~200 users), all of these users are subscribed to multiple forums, so there will quite often be multiple batches of 200 emails (and Firebase push notifications) that are to be sent. I'm therefore guessing that the email queue is still a good thing to use, rather than sending all of the emails and Firebase push notifications at once?

    Also, the forum is not on a VPS, but rather simply on "web hosting", where I don't have access to things like the operating system cron functionality (i.e. I don't have shell access, but rather just FTP access). Therefore, the vbcron.php solution that you mention is unfortunately not an option either.

    This brings me back to my original question, in regards to my "poor man's cron automation", by using scripted requests to my board from another server that I manage, with the intention of in turn triggering the "Scheduled Tasks" in vBulletin:

    Are there any particular requests to the board that will or will not trigger the queued up "Scheduled Tasks" in vBulletin, or will any kind of request to the board whatsoever suffice?

    More specifically:

    1.
    Can I use normal non-logged in requests to the board, and it will still trigger the execution of queued up "Scheduled Tasks"?

    2.
    Will any simple request to the board do, like e.g. reloading the main page or some forum index page, or will only some specific targets for my automated requests have any effect for executing queued up "Scheduled Tasks", and in that case, more specifically which requests would that be?

    Leave a comment:

Related Topics

Collapse

Working...
X