Can I resume searchshell.php after fatal error, and how?

[DJ RRebel]

I have a relatively large forum: 380k posts, 45k threads, 35k users

I've just made the move from a shared server to a virtual private one.

Everytihng's been super smooth and decided to finally use what little I know about shell access to reindex my Search Index .. was going great .. but after 4-5 hours it crashed with the following error:

Processed post: 241486
Processed post: 241487
Processed post: 241488

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 35 bytes) in /var/www/****/******.com/httpdocs/vBforums/includes/functions.php on line 157

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 40 bytes) in /var/www/****/******.com/httpdocs/vBforums/includes/class_core.php on line 2968

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 16 bytes) in /var/www/****/******.com/httpdocs/vBforums/includes/class_core.php on line 822

1- Can I resume doing the search index from post 241489 using either: searchshell.php again or the admincp? Or is the process corrupted and even if I fix the root of the problem I'll still have to start reindexing again from scratch?


2- How do I do fix this? I've read a few similar threads that refer to php.ini .. but that term isn't searchable here and I'm having a hard time actually getting information on it because I can't get informative search results anywhere. Could someone please explain to me how to raise my PHP memory levels for shellaccess.php or point me to a specific page with the exact instructions.

Is it true all I really have to do is add the following code to config.php? Will this trick work with searchshell?

ini_set('memory_limit', 32 * 1024 * 1024);


Thanks tons!

=========== Edited addition ===========

3- NOTE .. I checked my "View PHP info" in the admincp and saw that my current memory_limit actually IS 32M already ... the value that is at 8M is a directive called max_post_size .. should I try the following instead:

ini_set('max_post_size', 32 * 1024 * 1024);

Thanks

[Steve Machol]

Searchshell does not use the vB config.php file. First, create an .htaccess file in your forum directory with this content:

php_value max_execution_time 600
php_value memory_limit 32M

That should overcome the memory limit.

Then you can set the post number to start at in searchshell:

What post number would you like to start at?

[DJ RRebel]

wow .. thanks tons for such a quick reply!

While I was waiting I grabbed all my other PHP limitations:

memory_limit 32M
max_input_time 60
max_execution_time 30
post_max_size 8M
upload_max_filesize 2M

Am I correct in assuming a .htaccess file is just a simple txt file (without .txt extention at the end obviously)? So basically I could just enter those two lines of code you mentioned? Or should I really try increase post_max_size to 16M or even 32M as it seems that was the only think that might have triggered a 8388608 byte error? Or maybe to be safe I could just increase them all as such:

php_value memory_limit 64M
php_value max_input_time 300
php_value max_execution_time 300
php_value post_max_size 32M
php_value upload_max_filesize 16M

Does each line have to end with a semicolon? or is that a non issue for .htaccess files?


It would be a pain for me to restart my search index .. but I'm moving the forum to a new server and now will be the best time for a long time .. would it be beneficial at all for any reason to simply purge the index and restart from 0? Or does it really make no difference at all as long as I enter 241489 as the starting post?


Thanks tons once again!

[Colin F]

No semicolons needed in .htaccess files

It shouldn't make a difference if you resume from the last number.

[DJ RRebel]

Muchos gracias guys!

Turns out after reading some here and looking at the variables and talking to my ISP .. that I actually have access to my php.ini file after all! lol

I can not beleive how easy the solution was for me. As a thanks to your help guys I'll post how I did it for future reference for other members with VSPs (Virtual Private Servers are sorta inbetween shared hosting and dedicated hosting) who might be also be having this issue.


1- Log into your Server Manager (in my case it was an application called Virtuozzo). You should find instructions on how to access it in your welcome e-mails from your ISP when you registered.

2- Go into: File Manager .. Note that you are not logging into your site .. the files and folders here are for your Virtual Private Server and will not be the same files and folders you see when you usually log into your site's FTP. This aspect of it might be a little confusing for people new to VPS's like me who only have one site on their VPS.

3- Go to a folder named: etc

4- In 'etc' you'll find php.ini lower down and posibly on a later page.

5- Click the download icon to download php.ini to your PC.

6- MAKE A COPY as a back-up just in case

7- Open php.ini in wordpad

8- Look down the file Edit these settings .. the numbers you see below were my defaults .. yours may be different .. that being said .. if you're having problems a good start would be to double your current number. Remember to keep your overall memory limits in mind if you have any. Meaning that if your server package allows 128M of memory, then don't set anything to 256M (or even 128M for that matter as you'd want to leave overhead for other stuff)!

memory_limit 32M
max_input_time 60
max_execution_time 30
post_max_size 8M
upload_max_filesize 2M

NOTE: I don't think max_input_time and upload_max_filesize affect searchshell.php at all but if you're running into trouble with some stuff chances are you might in other places to .. best to ask someone at vB first, but I'm guessing it can't hurt to up all of them (any experts here know if there's any other derectives [derective is simply like a PHP variable] that could be looked at for increasing at the same time?)

9- Save the new php.ini file (make sure wordpad didn't add .txt to the end of the file name .. if it did, then simply rename it back to php.ini)

10- Back in Virtuozzo you'll see a button at the top that says "upload file". you should still be within the 'etc' folder so simply press the button and upload the copy of php.ini from your PC in 'etc' .. it should automatically overwrite the old php.ini

11- In Virtuozzo go into "Start/Stop VPS"

12- Before step 13 make sure you aren't running anything too important on your server that shouldn't be interupted (very little qualifies as that .. but don't do this if you're in the middle of reindexing something or uploading/downling a very large file).

13- Press the "Restart VPS" button.

14- Wait a minute or two as your VPS will quickly shut down and restart to let the new PHP values to take effect.

15- Go into your vB admincp at the very bottom .. select:
"Maintenance" > "View PHP info" .. then search for the variables to see if the new values are in effect!


I know this is a very dumbed down broken down and simple instruction .. but I made it for those like me who are just starting with this sort of server side thing .. if there are any errors or things I've left out .. please by all means correct my information as I'm just saying what I did and not claiming it to be the best solution! Hope it helps someone eventually!


======================================


B- I also noticed some of you claiming you get the PHP of your forum's (closed) homepage as output .. it happened to me as well .. all I did to fix that problem was turn on my forum and try again. I don't think searchshell works with a closed forum!



C- General info that's imposible to find the answers for by searching:

1- searchshell is only slightly faster that using the admincp .. BUT .. the big difference is in stability .. which will make things faster in the long run .. the total really depends on your server strength and number of posts .. but but don't expect to save anything more than 10% of the usual time. You WILL save more time if you have a bad server that usually timed out a lot when you use the admincp meathod!

2- Once you're in your shell program the 'pwd' command tells your your current location. So type it before you run searchshell because it might ask you for the location of your forum and if you typed 'pwd' before hand, the result will already be there for you to see and all you have to do is look and retype it instead of trying to remember your cryptic directorypath. Also 'dir' gives you a directory of what's in your current location just like the DOS days .. and 'cd' and 'cd ..' work like DOS as well!

3- A good program to get shell access is called Putty

4- searchshell.php displays a line for each post with the post number .. so the output is sorta similar to what you'd get with the admincp way of doing things .. I just mention that becaue when I was planning I was actually expecting it to just be blank until it's finished like when you extract a database. Basically you will see it working (or not .. lol)!

[Solo Drakban]

Just as an aside note, if you have access to some sort of shell in VPS (Not really familiar with them) you should be able to have the PHP changes take effect without having to restart the entire server simply by restarting the web server. For most variants of Linux running apache it can be accomlished via the init.d script using "/etc/init.d/httpd restart" or from apachectl using "/usr/local/apache/bin/apachectl restart". Either method will require 'root' on the server.

[DJ RRebel]

lol .. who here wants a good laugh? I think the source of the problem could posibly not be related to the size of the sum of all your posts .. but posibly it could crash because of a single long post (regardless if it is at the start or end of your indexing). Although obviously the timing a partial factor in that the indexing slows down the further it gets .. and us such a long post near the end will have a much larger chance of timing out or crashing than if it was at the start. But I'm thikning a forum of 3 million small short posts would have a significantly easier time reindexing than a forum with only 30 thousand longer posts!

The post that crashed me before was so rediculously large that I remember I had to change vB's post size limits by a few times for it to pass. Anyone curious can still view it in my old forum here .. note that it's the 2nd post in the thread .. it's also my mission statement .. make sure you have some free time before you proceed!

http://www.homeofchat.com/vB/showthread.php?p=241489