Announcement

Collapse
No announcement yet.

ABOUT TIME!!

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    Another thing is that when you load them in global.php -- the memory taken up by each HTTPD process increases accordingly -- loading all the templates seems like a good idea speed wise, until you have 150 processes running that are eating up more than 4MB RAM (as it was in our case, we had to upgrade from 512 to 1024MB RAM).
    I believe it's a hell of a lot more efficient than
    a) loading the templates one by one as you come across them
    b) not caching them so you have to load postbit 30 times.

    Unfortunately, a) will happen regardless if you move to flat files.

    Additionally, I see what you're saying, but my template dumps (from 2.0 which has more templates) are only about 250-300k and we aren't loading every template so I don't see how loading 12 templates would make it soar THAT much.

    Comment


    • #32
      Originally posted by Ed Sullivan
      I believe it's a hell of a lot more efficient than
      a) loading the templates one by one as you come across them
      It might be worthwhile looking into loading the templates for each page as you visit that page -- from looking at the table locks on template, I can see that the template table is being hit up quite a bit -- almost with each new user. So, given the fact that most people will get a new Apache process when they visit (unless you modify your HTTPD.conf to let them stick around longer / more of them), why not just have it load the templates that you *need* and not all of them?

      But really, my suggestion was to stick them in the PHP file themselves -- e.g. stick postbit in there, and have something like:
      Code:
      $postbit .= eval( $postbittemplate );
      Maybe retain a web-based interface -- have the template files stored in /template so that you can still edit them from the admin control panel, but when you hit "submit" it will open the PHP file(s) that use that template and then write the new stuff to a specific area...
      Maybe have a portion of the file delimited in such a manner that you know where to insert it --
      ### INSERT TEMPLATE HERE ###

      #### END TEMPLATE ####

      Then you'd just dump:
      $title = "$template";

      Originally posted by Ed Sullivan
      b) not caching them so you have to load postbit 30 times.

      Unfortunately, a) will happen regardless if you move to flat files.

      Additionally, I see what you're saying, but my template dumps (from 2.0 which has more templates) are only about 250-300k and we aren't loading every template so I don't see how loading 12 templates would make it soar THAT much.
      Our templates are a little larger -- but having each HTTPD process eat that much is pretty bad --we have 250 running now!
      Matt
      Sybase DBA / PHP fanatic
      Sybase v. MySQL v. Oracle | Why I don't like MySQL | Download Sybase TODAY! | Visit DBForums.com!

      Comment


      • #33
        why not just have it load the templates that you *need* and not all of them?
        That's what we are doing. In 1.1.4/5 it only did it for the "big 3", but 2.0 selectively loads templates on (almost) every page now, depending on action if possible (member.php is a good example) and I'm still going through adding calls to $DB_site->free_result and unset.

        Comment


        • #34
          Also, I'm not trying to ignore your suggestion. It would just be VERY hard to get working 100% correctly.

          Comment


          • #35
            Originally posted by Ed Sullivan
            That's what we are doing. In 1.1.4/5 it only did it for the "big 3", but 2.0 selectively loads templates on (almost) every page now, depending on action if possible (member.php is a good example) and I'm still going through adding calls to $DB_site->free_result and unset.
            That will help, depending on the size of the result set returned -- the only times I've ever needed the free_result( ) function was if I was querying 50,000 or more records at a time on the post table. However, as soon as the PHP script finishes executing, mysql_free_result will be executed automagically -- so I guess the concern is excess memory usage caused by the queries during execution time?

            Another concern I have (which I've been fixing as I come across them) are functions in which I see this:
            Code:
            function foo( ) {
            
               global $bar;
            
               if( $bar == "baz" ) {
                  $boo = "happyjoy";
               } else {
                  $boo = "bash";
               } // end if
            
               return $boo;
            
            } // end function foo
            Not only is it unnecessary to assign $boo a value, it is down right wasteful as well -- you create a variable, give it some memory, assign it the value of something else, then pass it on to the calling script, causing additional overhead. Just freakin return "happyjoy"; or return "bash"; . You still incur the overhead of returning the data, but you don't have the wasteful creation of needless variables.

            Another issue that would be solved by just returning a result are in places where there are a mass of if( ), elseif( ), else, etc. statements (if( 1 ) { $var = 42; } else...). If you follow the path of the conditional statements, you can see where the value of a returned variable wouldn’t change (e.g. all the rest of the if( ) statements would return false) – so we’re wasting CPU clock cycles to run 10 checks which we know (but the compiler doesn't) will all be false anyway.

            Further, I see functions that should *not* return anything at all – some that go:
            Code:
            function crappy( $someid ) {
              if( $someid != 0 ) {
                $DB_site->query( “UPDATE poop SET someid = $someid” );
              } // end if
             
               return 1;
            } // end function crappy
            That is also wasteful – there is no purpose for the return 1; -- the calling PHP file doesn’t even catch the value!! In C++ this is a no-no, PHP is a little more forgiving. However, it is still bad programming style / practice, and detrimental to performance to boot.

            Sheeh, I didn’t mean for this thread or this post to turn into a bashing of the code – think of that as constructive criticism, or more importantly – BUGS!

            Originally posted by Ed Sullivan
            Also, I'm not trying to ignore your suggestion. It would just be VERY hard to get working 100% correctly.
            I understand -- I'm thinking of people with higher loads and more users, where it would be beneficial for them to sqeeze every last ounce of performance from the machine. It might seem trivial that a new database connection takes .3 seconds to make, or that a large chunk of if( ) statments take 10 clock cycles to execute -- but if you have 1000 people making a new connection and runing that if block, it soon begins to add up.

            Most users wouldn't need, nor should they care, to think about such things. If Jelsoft and VB want to cater to people who are beginning to max out their current systems and don't have the cash to spend on brand new hardware, they need to show it in the form of better, tighter code. Even people who can afford Sybase or Oracle don't like to waste their money. Why buy a(nother) larger machine if it is caused by "bad" code?

            [Edited by mrogish on 01-13-2001 at 07:47 PM]
            Matt
            Sybase DBA / PHP fanatic
            Sybase v. MySQL v. Oracle | Why I don't like MySQL | Download Sybase TODAY! | Visit DBForums.com!

            Comment


            • #36
              Originally posted by mrogish
              That will help, depending on the size of the result set returned -- the only times I've ever needed the free_result( ) function was if I was querying 50,000 or more records at a time on the post table. However, as soon as the PHP script finishes executing, mysql_free_result will be executed automagically -- so I guess the concern is excess memory usage caused by the queries during execution time?
              Well, the memory usage just builds as the script goes, so I'm hoping to see if we can free it up so that at the end the process doesn't take up 37 GB of memory. Freeing the result sets for pre-cached templates for example and unset'ing parts of the arrays that aren't going to be used anymore (for example, specific forums and parentid's in the forum list)

              Code:
              function foo( ) {
              
                 global $bar;
              
                 if( $bar == "baz" ) {
                    $boo = "happyjoy";
                 } else {
                    $boo = "bash";
                 } // end if
              
                 return $boo;
              
              } // end function foo
              <snip other suggestions too>
              Good suggestions. Most have them have been implemented where possible now.

              Originally posted by Ed Sullivan
              Also, I'm not trying to ignore your suggestion. It would just be VERY hard to get working 100% correctly.
              Yeah yeah, quoting myself. This wouldn't be too hard to pull off in 1.1.x, but with the addition of template sets, I don't know exactly how we could ensure that the correct data is used. I have some ideas but they're just theory and this big of a template rewrite would delay 2.0 a lot.

              [Edited by Ed Sullivan on 01-13-2001 at 11:45 PM]

              Comment


              • #37
                That would prove difficult -- the only way around that I could think of would store it in an array, or .. well no, I guess the only clean way is to have it in the DB.

                I wrote multiple site templates into FanHome (like you guys probably are doing for 2.0) -- I have 1 set of PHP files for all 5 sites we have, and the ability to create a co-branded site at the click of a button (new replacement vars, templates, etc.). That would most definitely not be able to work with a PHP based solution, and would be difficult but not impossible to work with file-based systems. For users with a single site that don't need all of those bells and whistles though, a PHP based solution would work well and offer the most performance.

                Thinking about it more, I can't advocate the use of file-based solutions at all because they probably wouldn't ever be more efficient than in the DB -- because what is a database but a front end to a large file (or in the case of MySQL several files)? They’ve spent years working on concurrency issues, compression, indexes, etc. – why throw all that away for PHP’s file-handling routines, which as nice as they are, aren’t the focus of the software (PHP is first web page generation software, not file manipulation software).
                Matt
                Sybase DBA / PHP fanatic
                Sybase v. MySQL v. Oracle | Why I don't like MySQL | Download Sybase TODAY! | Visit DBForums.com!

                Comment


                • #38
                  Back in July, I made a quick hack for the in-line template storage. I don't think it'll work on 1.14/5, but it did work on previous versions. No, I don't reccommend using this in its current state, but if you're experienced with PHP, you might want to hack it in on your own if you don't plan on using template sets.

                  How it works? Create a directory called virgin, and move all your /forum/ files (non /admin/ or /images/ though) there. Run dosr.php in your /forum/ directory, and voila. Yes, its a bit oversimplified, but this is _really_ old and _really_ bad code which had some JS problems for templates.

                  Some of this search/replace code is now part of PEAR, if you want to integrate it with that...

                  Here is, the never-before released, tOpti 1.0:
                  Code:
                  <?php 
                  /******************************************************\
                  |         ______         __ __         __   __         |
                  | .--.--.|   __ \.--.--.|  |  |.-----.|  |_|__|.-----. |
                  | |  |  ||   __ <|  |  ||  |  ||  -__||   _|  ||     | |
                  |  \___/ |______/|_____||__|__||_____||____|__||__|__| |
                  |******************************************************|
                  |** tOpti - Template Optimizer for vBulletin Software *|
                  |**** Script Copyright (C) Chris & Mike Lambert, 2000 *|
                  |**** Special Thanks to Richard Heyes (some S&R code) *|
                  |**** For More Information, Email: [email protected] *|
                  \******************************************************/ 
                  
                  $virdir = "virgin";
                  if ($allfiles=opendir($virdir)) {
                   print "<b>Files restored</b> from <tt>$virdir</tt>: ";
                   $commaq = "";
                   while ($file=readdir($allfiles)) {
                    if (is_dir($file) == 0) {
                     if ($commaq) {
                      print ", ";
                     }
                     $commaq = "1";
                     copy("virgin/$file",$file);
                     print "<tt>$file</tt>";
                    }
                   }
                   print "<br>\n<br>\n";
                  }
                  
                  
                  /*************************************** 
                  ** The main search and replace routine. 
                  ***************************************/ 
                          function search_replace($filename,$templates,$ignore){ 
                                  $occurences = 0; 
                                  $file_array = file($filename); 
                                  $newfile = "";
                  		$newlines="";
                  		for($i=0; $i<count($file_array); $i++){ 
                         	                $continue_flag = 0; 
                                 	        if($ignore_lines != ''){ 
                                         	        for($j=0; $j<count($ignore); $j++){ 
                                                 	        if(substr($file_array[$i],0,strlen($ignore[$j])) == $ignore[$j]) $continue_flag = 1;
                                                  } 
                         	                } 
                                 	        if($continue_flag == 1) continue; 
                  			$newlines[] = $file_array[$i];
                  		}
                  		$newfile = implode('', $newlines);
                                  for($j=0; $j<count($templates); $j++){ 
                                          $newfile = str_replace('gettemplate("'.$templates[$j][title].'")', '"'.$templates[$j][doubletemplate].'"', $newfile); 
                                          $newfile = str_replace('gettemplate("'.$templates[$j][title].'",0)', '"'.$templates[$j][singletemplate].'"', $newfile); 
                                  } 
                  
                  #                if($occurences > 0) $return = array($occurences, implode('', $file_array)); else $return = FALSE; 
                                  return $newfile; 
                          } 
                  
                  /*************************************** 
                  ** Function for writing out a new file. 
                  ***************************************/ 
                          function writeout($filename, $contents){ 
                                  if($fp = fopen($filename, 'w')){ 
                                          fwrite($fp, $contents); 
                                          fclose($fp); 
                                  }else{ 
                                          die('Could not open file: '.$filename); 
                                  } 
                          } 
                  
                  /*************************************** 
                  ** First do whatever files are specified, 
                  ** and/or if directories are specified, 
                  ** do those too. 
                  ***************************************/ 
                  function dosr($templates,$directories='',$files='',$include_subdir=0,$ignore_lines='//'){
                          $occurences = 0; 
                          $ignore = explode(',', $ignore_lines); 
                                  if(!empty($files)){ 
                                          if(ereg(',', $files)) $files = explode(',', $files); else $files = array($files); 
                                          for($i=0;$i<count($files); $i++){ 
                                                  if($files[$i] == '.' OR $files[$i] == '..') continue; 
                                                  if(is_dir($files[$i]) == TRUE) continue; 
                                                  $newfile = search_replace($files[$i],$find,$replace,$ignore); 
                                                  if(is_array($newfile) == TRUE){ 
                                                          writeout($files[$i], $newfile[1]); 
                                                          $occurences += $newfile[0]; 
                                                  } 
                                          } 
                                  } 
                  
                                  if(!empty($directories)){ 
                                          if(ereg(',', $directories)) $directories = explode(',', $directories); else $directories = array($directories); 
                                          for($i=0;$i<count($directories); $i++){ 
                                                  $dh = opendir($directories[$i]); 
                                                  while($file = readdir($dh)){ 
                                                          if($file == '.' OR $file == '..') continue; 
                  
                                                          if(is_dir($directories[$i].$file) == TRUE AND $include_subdir == 1){ 
                                                                  $directories[] = $directories[$i].$file.'/'; 
                                                                  continue; 
                                                          }elseif(is_dir($directories[$i].$file) == TRUE){ 
                                                                  continue; 
                                                          } elseif (substr($file,-4) != ".php") {
                  						continue;
                                                          } elseif ($file == "dosr.php") {
                  						continue;
                  					}
                  
                  #                                        if(is_array($newfile) == TRUE){ 
                                                          $newfile = search_replace($directories[$i].$file,$templates,$ignore); 
                                                          writeout($directories[$i].$file, $newfile); 
                  echo "<b>Wrote</b> out <tt>".$directories[$i].$file."</tt><br>\n";
                  #                                        $occurences += $newfile[0]; 
                  #                                        } 
                                                  } 
                                          } 
                                  } 
                  #        } 
                  #        echo "Found ".$occurences." occurences replacing <tt>$find</tt>.<br>\n"; 
                  }
                  include("global.php");
                  
                  $templates = "";
                  $templateiterator = $DB_site->query("SELECT title,template FROM template");
                  while ($template=$DB_site->fetch_array($templateiterator)) {
                    $template[title]    = addslashes($template[title]);
                    $template[singletemplate] = str_replace("\$","\\\$",addslashes($template[template]));
                    $template[doubletemplate] = str_replace("\$","\\\$",addslashes(addslashes($template[template])));
                    $templates[]=$template;
                  }
                  
                  dosr($templates,'./');
                  
                  ?>

                  Comment


                  • #39
                    Your co-branding is how ours works too - you can specify both/either replacement vars and templates. The "Global templates" part would really cause a problem.

                    What I was originally thinking was still using the gettemplate function, but requiring some files that would just be like this:

                    <?php

                    $templatecache['title'] = 'template';

                    ?>

                    Of course, the use of some templates in more than one file kinda complicates this, but the template sets kill it... although, theoretically I could cache like $templatecache['title']['setid'] = 'template', and do an isset check on the user's styleid, then check the global one, or lastly try the db. Hmm...

                    Also, yes - MySQL's index files and data files are separate.

                    Comment

                    widgetinstance 262 (Related Topics) skipped due to lack of content & hide_module_if_empty option.
                    Working...
                    X