Tweet
I am attempting to wget a large forum for future litigation. I need to login to get the information we require.
This is what I currently use:
wget --post-data 'forceredirect=1&do=login&vb_login_md5password=HASH!&vb_login_username=USERNAME&url=/forums/index.php&vb_login_password=&cookieuser=1' -r --save-cookies cookies.txt --keep-session-cookies -o site.log -N -E -b -k -R *login.php?do=logout* http://www.SITE.com/forums/login.php
It is successful for anywhere between 2,000 - 25,000 posts. Then I start getting 400 Bad Request errors. Can anyone help me narrow this problem down? Is it likely a session timeout? Apache stopping the mirror? Or is it vbulletin code that is set in place to stop it?
This is what I currently use:
wget --post-data 'forceredirect=1&do=login&vb_login_md5password=HASH!&vb_login_username=USERNAME&url=/forums/index.php&vb_login_password=&cookieuser=1' -r --save-cookies cookies.txt --keep-session-cookies -o site.log -N -E -b -k -R *login.php?do=logout* http://www.SITE.com/forums/login.php
It is successful for anywhere between 2,000 - 25,000 posts. Then I start getting 400 Bad Request errors. Can anyone help me narrow this problem down? Is it likely a session timeout? Apache stopping the mirror? Or is it vbulletin code that is set in place to stop it?
Comment