<?php
    // at the top of your template
    $headers=$this->getHeadData();
    $headers['scripts'] = array();
    $this->setHeadData($headers);
 
    // make sure to eliminate any of the following
    //JHTML::_('behavior.framework', true);
    //JHTML::_('behavior.mootools', true);
?>

In Joomla 1.6 this can remain the same if you adjust a tiny bit of Joomla internals. Specifically we need to modify the setHeadData() function in joomla16/libraries/joomla/document/html/html.php (around line 116).

// original version
$this->_scripts = (isset($data['scripts']) && !empty($data['scripts'])) ?
                  $data['scripts'] : $this->_scripts;
 
// updated version ( !empty() replaced with is_arrray() )
$this->_scripts = (isset($data['scripts']) && is_array($data['scripts'])) ?
                  $data['scripts'] : $this->_scripts;

What this does is allow Joomla to accept an empty value for $headers['scripts'] where it would previously reject it.

Let me know if you have any questions.

Happy coding!

Data caching takes the place of a database call and can be effective if there are a lot of database requests.  You would typically want to do this if you know that a piece of data is very common and unlikely to change. This saves time on one step of the process.

Fragment caching is a little more complicated.  Instead of caching the data that you would input into another process to build a layout (for example), you just cache that layout directly.  This is great for a "featured news" section that changes only once a day and is viewed often.  It would be silly to cache that data and calculate the layout for every view, so instead we would just cache the layout itself. Saves a bunch of work.

Moving on, we have page caching.  Which is the topic which I really want to go over. I have a warning though, page caching is dumb. REAL dumb. But, in that magnificent dumness is speed. Page caching is the most efficient of the three. It takes the gains we get from fragment caching and goes even further. Think about it, we hardly need to do anything—and therein lies the problem, we lose ALL dynamic ability of the page. This is not an issue for a lot of sites; news pages, catalogs, and blogs can all benefit from this method of caching.

. . .

The method I describe here I used allowed a real site to receive 250,000 real page-views in 40 min (~104 requests/sec). I expect you can do much more. (More number towards the end.)

What I haven't mentioned yet is that I'm using PHP (specifically Joomla). PHP is certainly not the fastest language around, but considering the whole spectrum of issues facing a web developer, script performance is minimal. And since I'm on this rabbit trail, I'll mention that Javascript/CSS is the first thing to fix, then caching. I don't need to explain more about this, there are volumes about it already.

. . .

Moving on, the method described for page caching is based on a simple fallback plan. Here is the psudo-code.

If the page requested is in cache
    serve cached version
otherwise
    serve it dynamically
    add result to memcached

. . .

Joomla Plugin

With Joomla we can actually do this very easily. We just need to hook into the onAfterRender event and grab the page data.  You might notice we're using the database name as the prefix to the URL in the key name. This is just to be friendly to any other sites hosted on the same server (or copies of the same site using a different database).

You also might notice that we're grabbing the gzipped version of the page. This is just to trim a little off what the user needs to download.

// no direct access
defined( '_JEXEC' ) or die( 'Restricted access' );

jimport( 'joomla.plugin.plugin' );

class  plgSystemFullPageMemcache extends JPlugin
{
    function plgSystemFullPageMemcache(& $subject, $config) {
        parent::__construct($subject, $config);
    }

    function onAfterRender() {
        global $mainframe;
        $config = new JConfig();
        $cache_id= $config->db . ":" . $_SERVER['REQUEST_URI'];

        $data = JResponse::toString($mainframe->getCfg('gzip'));

        $db = new Memcache;
        $db->addServer('127.0.0.1', '11211', true);
        $db->set($cache_id, $data, 1, 3600);
    }
}

Nginx configuration

server {

  listen 80;
  server_name  yourdomain.com www.yourdomain.com;

  # ...

  location ~ \.php$ {
    set $memcached_key "yourdomain:$request_uri";
    memcached_pass 127.0.0.1:11211;

    default_type       text/html;
    error_page 404 405 502 = @cache_miss;
  }

  location @cache_miss {
    fastcgi_pass 127.0.0.1:9002;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /var/www/yoursite/htdocs$fastcgi_script_name;
    include /etc/nginx/fastcgi_params;
  }

  # ...
}

. . .

Here you can see @cache_miss will get called when nginx can't find anything in the memcache for the given key (the url). This triggers our fallback plan where the cache will fill itself with any missing data. When we do hit the cache, we actually bypass PHP altogether. No initialization wait, no slowness, and no database calls. By educated guess, nginx and memcached in this setup can start serving content in under 15 milliseconds. Compare this with the 50-100 milliseconds we commonly see.

Theoretically you could serve content faster using this system than just using files due to time spent waiting on disk access. In my proof of concept, I was able to serve about 2500 requests/second but I have reason to believe that limit was imposed by the network card. The CPU was at 10% the whole time and I'm sure it could handle more.

Happy caching!

(Downloadable Plugin Coming Soon)

Beware, this is beta quality software. Use at your own risk.

As we understand already, Joomla doesn't have relevant searching. It's a shame since everything else is designed well, but this is a problem regarding implementation. This article explains how to setup sphinx to index your Joomla articles and use my plugin to search using Joomla.

In my own experiments I built a plugin that accomplished this. It works splendidly, and I had all intention of releasing the code to the public, but got distracted. This process is currently unrefined, but it works. I'm a fan of revision anyway, so if you see a way to improve this article, hit me up (email).

So why doesn't Joomla have relevant searching already? — Unfortunately it doesn't seem like they had much choice about it. Joomla relies on MySQL. MySQL is just now beginning to get some Full-Text searching mechanisms implemented and those that do exist are spotty. For example, it is only available in recent versions of the MyISAM database engine and not at all for InnoDB. Plus, I've heard some complaints about it not even being terribly good at it's own job.

A system like Sphinx can go beyond what MySQL supports anyway. It can use word stems and substitutions, so I could specify CPP to be used interchangeably with C++ in any search query and it doesn't get confused about plural or singular versions of words. It also supports weights, so I can tell Sphinx to prefer matches found in titles and move them higher in the search results.

Install & Configure Sphinx

There are a number of good articles on the web about this. I may revisit how to install Sphinx in detail, but for now defer to the follow links:

• Sphinx Homepage — Obviously the first place to start looking is the Sphinx homepage. Good documentation on Sphinx operation and internals.
• Build a custom search engine with PHP — IBM has a great article on using Sphinx in-the-field. This article describes setup of Sphinx as well as testing and integration with PHP.

Here is my configuration I've been using for my Joomla Setup. This is basically the same as any configuration setup on IBM's website, but adapted to Joomla. There are a few things you'll need to edit though:

• exampleuser — This should be the username you connect with to search your database
• examplepass — This should be the password for the username just described
• exampledb — This is the database sphinx will be searching on.
• Check your paths. I'm making some assumptions here. Your sphinx data should be outside of your web-root and readable/writable by whatever process searchd is running as.

/var/www/website/sphinx/config.conf:

source mywebsite_articles
{
    type            = mysql
    sql_host        = localhost
    sql_user        = exampleuser
    sql_pass        = examplepassword
    sql_db          = exampledb
    sql_sock        = /var/run/mysqld/mysqld.sock
    sql_port        = 3306

    # indexer query
    # document_id MUST be the very first field
    # document_id MUST be positive (non-zero, non-negative)
    # document_id MUST fit into 32 bits
    # document_id MUST be unique

    sql_query = \
        SELECT `id`, `title`, `alias`, `introtext`, `fulltext`, `created`, `modified`, `hits` \
        FROM jos_content ;

    # document info query
    # ONLY used by search utility to display document information
    # MUST be able to fetch document info by its id, therefore
    # MUST contain '$id' macro
    #

    sql_query_info = \
        SELECT * \
        FROM jos_content  \
        WHERE id=$id
}
index mywebsite_articles
{
    source                  = mywebsite_articles
    path                    = /var/www/website/data
    morphology              = none

    min_word_len            = 3
    min_prefix_len          = 0
    min_infix_len           = 3
}

searchd
{
    port          = 3312
    log           = /var/www/website/sphinx/search.log
    query_log     = /var/www/website/sphinx/query.log
    pid_file      = /var/www/website/sphinx/searchd.pid
}

Testing Sphinx

How do we know if Sphinx is setup correctly? We test it!

First we need to generate the index. So fire up your terminal and run the following (assuming your sphinx install lives in /usr/local/sphinx):

/usr/local/sphinx/bin/indexer \
--config /var/www/website/sphinx/config.conf \
--all --rotate

Second we need to start the search daemon. This will process search requests.

/usr/local/sphinx/bin/searchd \
--config /var/www/website/sphinx/config.conf

If you get any errors, I'd recommend going back and reviewing the configuration and re-running the indexer, then start searchd again.

If we don't have any errors, we can run a test outside of Joomla to see if everything is working correctly. For example:

/usr/local/sphinx/bin/search \
--config /var/www/website/sphinx/config.conf \
"Hello world"

Install & Configure Joomla Plugin

You can download the plugin here (Download) or at the bottom of this article.

Enable the plugin. Be sure to turn off other search modules while you're at it. It is much easier to determine if your search results are isolated to one plugin.

If you are hosting Sphinx on the same server as your web service, you'll probably want to keep these similar to what I have. Otherwise, be sure to fill-in the appropriate fields here and configure any firewalls that may be between the web server and the Sphinx server.

You will also want to be careful to use the same name of your index defined in your config file. In my example I'm using mywebsite_articles. This is what I fill-in for Resource Name.

Test Sphinx Plugin

The only thing left to do now is to actually test the thing. Go to your website and do a search!

Completing the solution

There are a few things still missing from this setup that I want you to be aware of.

First, if you don't update your index periodically, you will only see the content that you indexed the first time. One solution is to setup a scheduled task (or cron) to re-index your content. Notice as well I put --rotate at the end of my indexing operation. Once Sphinx has updated the index, it will begin to use the new index to search with. You don't need to restart your searchd process, just update the index.

Second, you should be aware that if your server restarts you'll need searchd to start automatically. An init script would work great, even something as simple as this:

/etc/init.d/searchd

#! /bin/sh
NAME=sphinx
DAEMON=/usr/local/sphinx/bin/searchd
CONFIG=/var/www/website/sphinx/config.conf
[ -x "$DAEMON" ] || exit 0
case "$1" in
  start)
    echo "Stopping any running daemons..."
    $DAEMON --config $CONFIG --stop
    echo "Starting sphinx search daemon..."
    $DAEMON --config $CONFIG
    ;;
  stop)
    echo "Stopping sphinx search daemon..."
    $DAEMON --config $CONFIG --stop
    ;;
  *)
    echo "Usage: $NAME {start|stop}"
    exit 3
    ;;
esac
:

plgSphinxContent-v2.zip (Sphinx Search Plugin)

Downloads: 162 File Size: 0.0 KB

If you've used Joomla before, you may be aware that there are built-in caching features which you can activate to reduce the number of expensive database calls needed to render a page. This feature is also available to your own custom code, whether that be in the form of components, modules, or plugins.

The actual mileage for this feature may vary due to the speed of the server and what Joomla decides to cache. Personally I think Joomla's use of caching, specifically how they use it for articles, leaves something to be desired. The downsides include content that doesn't get flushed out once it has changed or content that has too short of a lifetime to prove useful. Because of this, I don't think lot of developers realize that you can use it to cache your own components, plugins, and modules reliably and without affecting the user experience in a negative way, all the while in a modest environment (i.e. a server environment that doesn't have special software like memcached installed.)

You might consider implementing a little caching if:

• Your code is heavy and slow (Don't take it personally, mine is too sometimes)
• Your code is high-latency requiring network resources that may or may not actually be available
• You have a crazy number of visitors and your database is becoming a significant bottleneck

In my own recent project I was able to improve my access times of some SOAP requests by 500% (~1 second/request down to ~200ms/request).

Let's get our bearings

Here's a basic example of some code which initializes a JCache object and get's some data using it. You'll notice that there are two functions, one which just generically gets some data getData and another function getDataCached which instantiates the cache and calls getData. We need to do this so that our cache is aware of what the inputs and outputs are, that way it can do a lookup on the cache and return data without ever touching getData.

com_cachetest/cachetest.php:

class CacheTest {
    function getData($url) {
        // ... Get some data

        // in this case, I'm just returning some dummy data instead of
        // downloading a web page which if we plan for the worst case scenario,
        // might not be available.

        $data = $url;

        // send data back
        return $data;
    }
    function getDataCached($url) {

        $group = "cachetest";
        $function = array("CacheTest", "getData");

        // get a cache object and name it something useful
        $cache =& JFactory::getCache( $group );
        $cache->setCaching( 1 ); // enable caching (1 = enabled, 0 = disabled)

        // call our function "getData" through the cache
        $result = $cache->call( $function , $url);

        return $result;
    }
}

$data = CacheTest::getDataCached( "http://example.com" );
var_dump($data); // to see what we actually get

Fault Tolerant Caching

One issue in the above example is that if getData returns some bogus data, it's going to be stuck in the cache and you won't realize it until it's too late. This isn't a likely scenario for a quick database call but if you have a network request that is waiting and waiting and eventually reaches timeout, the cached data will possibly be empty. This can be problematic but it's also easily avoided. In the following example we do a quick check to see if the data is invalid, remove the old data and get a fresh copy.

com_cachetest/cachetest.php:

class CacheTest {
    function getData($url) {
        // ... Get some data

        // in this case, I'm just returning some dummy data instead of
        // downloading a web page which if we plan for the worst case scenario,
        // might not be available.

        $data = $url;

        // send data back
        return $data;
    }

    function getDataCached($url) {

        $group = "cachetest";
        $function = array("CacheTest", "getData");

        // get a cache object and name it something useful
        $cache =& JFactory::getCache( $group );
        $cache->setCaching( 1 ); // enable caching (1 = enabled, 0 = disabled)

        // call our function "getData" through the cache
        $result = $cache->call( $function, $url);

        // Check if we get sane data. Depending on your data this
        // could be more than just an empty check
        if ( empty( $result ) ) {

            // find the id to remove data for
            $id = $cache->_makeId( $function, $url );

            // invalidate this entry in the cache
            $cache->remove( $id, $group );

            // call our function again knowing the cache has been fixed
            $result = $cache->call( $function, $url );
        }
        return $result;
    }
}
$data = CacheTest::getDataCached( "http://example.com" );
var_dump($data); // to see what we actually get

In Closing

Joomla's caching API is simple and, by default, just writes these cache files to disk. Although not the fastest approach the default options are really handy for a less-than-stellar server setup. These caching options are available to even shared hosts. At the same time, I should mention that Joomla has done a fantastic job of integrating support for more advanced caching systems like memcache and xdebug so that the above code can be run even faster without any modifications.

joomla-caching-example.git (Programmatic Caching in Joomla 1.5 with Fault Tolerance)

Downloads: 159 File Size: 0.0 KB

I tried to find the source of the problem and always got to the JOIN statement where it would routinely take 10+ seconds to run one query—add in multiple queries and filters and it would take over a minute just to get 500 or so records from MySQL.

I almost gave up hope of finding a solution and resolved to rewrite the entire search & display module, but then I decided to check the indexes—an amateur mistake not looking at this sooner.

Sure enough, no indexes on the fields we join the tables on.

To fix the problem add the following indexes (remember, we're not talking about Primary Keys or Unique indexes here):

on jos_rsform_submissions (table name may be different) add an index to the FormId field.

on jos_rsform_submission_values (table name may also be different) add an index to the SubmissionId field.

or in SQL:

ALTER TABLE `jos_rsform_submissions` ADD INDEX ( `FormId` )
ALTER TABLE `jos_rsform_submission_values` ADD INDEX ( `SubmissionId` )

Those queries which were taking 1 minute to run now take roughly 0.5 seconds. The queries are really inefficient but I can live with requests under a second.

Now I can get back to enjoying the Joomla community—they really are awesome!

It works in conjunction with Sphinx to index and search Joomla content "the right way." (i.e. not using basic LIKE statements)

Edit (Feb. 27, 2010): I just released a beta version of this plugin here: Joomla Sphinx Search Plugin

Joomla has been notorious for not being able to sort search results by relevancy and really, it's not very fast either.

Sphinx allows for full-text searching on any type of database table (so you're not forced to use MyISAM), and can even work with PostgreSQL (which Joomla unfortunately doesn't support.) It can handle a number of advanced features like:

• Wildcard searching
• English word stems ('dogs' => 'dog')
• Stopwords ( so that words like 'the' don't get searched)
• Equivilent word matching ( cpp = c++ = cplusplus )
• Live index updates ( although my plugin does this manually )

The project I'm working on is a three part system, including:

• Search plugin – to take the search terms and call the Sphinx search daemon.
• Content plugin – to update the Sphinx index when a new article has been written or updated.
• Admin Component – to help maintain, help configure,  and assist in setup of the index.

I don't know when this project will be released in beta, but hopefully by next week I can get something cobbled together. Let me know (Email me) if you're interested because I want this project to be as usable a possible and would very much appreciate the input.