Find top cache identifiers

New functionalities in the „cachemgm“ extension.

Overview of the configured caches:

Analyse your caches with the cache log:

Dependency Injection Caching Analysis:

What is the goal of caching

1. Content that can be cached should be cached, and we want as less cache entries as possible. To cache content means its likely that the generated content is valid for a certain amount of visitors.
2. Parts of the content that should not be cached should not be cached, but the rest of the page should be cached. For example you might have dynamic elements like a shopping basket or a username.
3. We never want uncached page rendering.

1) Cache generated pages in TYPO3

• Diffrent usergroups
• the different states of all typoscript conditions. (This depends on your typoscript setup)
• Diffrent values of predefined Core parameters if they are valid (id, type, MP)
• Based on other incoming parameters if the cHash matches. This last point is important to understand - since this is what you need to make your extension output cacheable the way you want.

2) Have dynamic content

3) Avoid uncached contend:

cHash details

current cHash problems and limitations

The solution: Flexible cHash calculation

Request Handlers

Search Handlers

• q: The query string that is parsed using a query parser
• sort: For defining the sorting
• start, rows - to define offest and result count
• fq - to define filter queries - you can use multiples of them - and the results are cached. (The caching can also be disabled)
• fl - specify the list of fields that should be returned for the matched documents. For the upcoming SOLR 4.0 this parameter has nice additional features - like pseudo fields, function results as field etc [7]
• defType - specify the query parser
• debugQuery - set this to "on" to see the parsed query and details for scoring calculation.

Query Parsers:

• lucene - standard (the standard parser) - see below for details
• dismax - aims to deal with a "human query string" - see below for details. (there is also the extended edismax parser in newer versions)
• func - to build function queries. Not really useful as standalone parser - but often used together with others (_val_ hook for lucene parser and bf parameter for dismax parser)
• boost - to boost a query
• frange - can be used to speed up range queries
• field - simple field query useful in filter querys
• prefix - simple prefix query - useful e.g. in filter querys
• raw / term - create raw term query from input
• query - allow for combining different querytypes ("nesting")

Query Parsers at a closer look

the lucene query parser (standard)

• used to select on specific fields if you need fine grained control
• enables powerful usage of Lucene Query Parser with a lot of features [2]
• two hooks that allows more complex use cases

The dismax query parser

• Phrases (fo boosting documents where all terms match in close proximity)
  • pf - Phrase Fields (e.g. content^2)
  • ps - Phrase sloop (e.g. 15 )
• bf - boost function, shorthand for _val_ hook. (Simelar like for the lucene parser)
• bq - boost query - any additional query that is executed and added to the scoring

Boost Query Parsers:

Function Query Parser

• lucene parser: using the _val_ hook
• dismax parser: using the _val_ hook or the bf parameter

Combining dismax and lucene

The fq (filter query) parameter

• You can also use different query parsers (using localparam syntax). The default (and the only one that makes really sense) is lucene
• You can have multiple fq parameters in the url: all are evaluated
• The result of the query is cached (can be disabled)
• The filter query do not influence the scoring calculation therefore we can perform filtering without fearing to change the score value of individual documents

1. all filter queries are executed seperate. If caching is not disabled the resulting document ids are cached. So if the same filter query is used again its not executed - but retrieved from the cache.
2. Before the final result is returned, solr calculates the intersection between the main query result and all filter query results.

Search components

• FacetComponent - Adds informations that can be used to show filters (facets) with the search result.
• Highlighting - Adds preview snippets for the result documents based on the query
• StatsComponent - similar to FacetComponents but only returns infos like min and max values of a certain field. (Useful to display range filters)
• SpellCheckComponent - advanced spell checking for the query
• QueryElevantionComponent - adds or boost documents based on a editorial maintained file
• TermVectorComponent - can return infos like frequent terms in a field ( if the field is configured to store termVectors)
• TermsComponent - provide access to the indexed terms in a field - often used for autosuggest

var detection = DeviceDetection(); 
detection.isApple(); 
detection.isTouchDevice(); 
detection.isAndroid(); 
detection.isSmartPhone(); 
detection.isDesktop(); 
detection.isTablet(); 

class Tx_Extkey_System_Session_SessionStorage implements t3lib_Singleton {
	
	const SESSIONNAMESPACE = 'tx_extkey';
	
	/**
	 * Returns the object stored in the user´s session
	 * @param string $key
	 * @return Object the stored object
	 */
	public function get($key) {
		$sessionData = $this->getFrontendUser()->getKey('ses', self::SESSIONNAMESPACE.$key);
		if ($sessionData == '') {
			throw new LogicException('No value for key found in session '.$key);
		}
		return $sessionData;
	}
	
	/**
	 * checks if object is stored in the user´s session
	 * @param string $key
	 * @return boolean
	 */
	public function has($key) {
		$sessionData = $this->getFrontendUser()->getKey('ses', self::SESSIONNAMESPACE.$key);
		if ($sessionData == '') {
			return false;
		}
		return true;
	}
 
	/**
	 * Writes something to storage
	 * @param string $key
	 * @param string $value
	 * @return	void
	 */
	public function set($key,$value) {
		$this->getFrontendUser()->setKey('ses', self::SESSIONNAMESPACE.$key, $value);
		$this->getFrontendUser()->storeSessionData();
	}
	
	/**
	 * Writes a object to the session if the key is empty it used the classname
	 * @param object $object
	 * @param string $key
	 * @return	void
	 */
	public function storeObject($object,$key=null) {
		if (is_null($key)) {
			$key = get_class($object);
		}
		return $this->set($key,serialize($object));		 
	}
	
	/**
	 * Writes something to storage
	 * @param string $key
	 * @return	object
	 */
	public function getObject($key) {
		return unserialize($this->get($key));		 
	}
 
	/**
	 * Cleans up the session: removes the stored object from the PHP session
	 * @param string $key
	 * @return	void
	 */
	public function clean($key) {
		$this->getFrontendUser()->setKey('ses', self::SESSIONNAMESPACE.$key, NULL);
		$this->getFrontendUser()->storeSessionData();
	}
	
	/**
	 * Gets a frontend user which is taken from the global registry or as fallback from TSFE->fe_user.
	 *
	 * @return	ux_tslib_feUserAuth	The current extended frontend user object
	 * @throws	LogicException
	 */
	protected function getFrontendUser() {
		if ($GLOBALS ['TSFE']->fe_user) {
			return $GLOBALS ['TSFE']->fe_user;
		}
		throw new LogicException ( 'No Frontentuser found in session!' );
	}
}

class Tx_Extkey_Domain_UserSession implements t3lib_Singleton {
	/**
	 * @var Tx_Extkey_System_Session_SessionStorage
	 */
	private $sessionStorage;
	
...	

	public function __construct(Tx_Extkey_System_Session_SessionStorage $sessionStorage) {
		$this->sessionStorage = $sessionStorage;
	}
	

	public function getBasket() {
		if ($this->sessionStorage->has('Basket')) {
			return $this->sessionStorage->getObject('Basket');
		}
		else {
			return $this->objectManager->create('Basket');
		}
	}
	
	public function saveBasket(Basket $basket) {
		$this->sessionStorage->storeObject($basket);
	}
		
}

Serialisation drawbacks..

• Not all associated framework objects are serializable. For example you will have problems serializing domain entities (too big, lost connection to persitence manager...)
• Cyclic relations will cause problems
• The serialized version can simple be too big

• can store ids instead of entitys and try to get them with a repository on wakeup
• throw injected singletons away and try to get them back on __wakeup

Dependency Injection

1. in the __wakeup method using the Extbase ObjectManager (t3lib_div::makeInstance('Tx_Extbase_Object_ObjectManager')
2. explicit In the UserSession Class like:

	public function getBasket() {
 		if ($this->sessionStorage->has('Basket')) {
 			$basket =  $this->sessionStorage->getObject('Basket'); 
                        $basket->injectSomeThing($this->objectManager->get('Something'));
 		}
 		else {
 			return $this->objectManager->create('Basket');
 		}
 	} 

In FLOW3

1. You need to split the word "Autonavigation" into "Auto" and "navigation"
2. You need to search for the translated, synonyms and untranslated tokens.
3. Be fuzzy enough
4. Be relevant

Using DictionaryCompoundWordTokenFilterFactory to split nouns

<!-- 1 split subwords english nouns -->
<filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="wordlists/german-common-nouns.txt" minWordSize="5" minSubwordSize="4" maxSubwordSize="15" onlyLongestMatch="true"/>
<!-- 2 split subwords german nouns -->
<filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="wordlists/english-common-nouns.txt" minWordSize="5" minSubwordSize="4" maxSubwordSize="15" onlyLongestMatch="true"/>
			

Using SynonymFilterFactory to translate

<filter class="solr.SynonymFilterFactory" synonyms="wordlists/translationsgroups-english-german-nouns.txt" ignoreCase="true" expand="true"/>
				 

Using the correct Stemmer

• GermanStemFilterFactory
• SnowballPorterFilterFactory (language German)
• GermanLightStemFilterFactory
• GermanMinimalStemFilterFactory

Be relevant

<fieldType name="expandedtext" class="solr.TextField" positionIncrementGap="100">
	<analyzer type="index">
		<tokenizer class="solr.WhitespaceTokenizerFactory"/>
		<filter class="solr.HyphenatedWordsFilterFactory"/>
		<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
		<!-- Case insensitive stop word removal.
			add enablePositionIncrements=true in both the index and query
			analyzers to leave a 'gap' for more accurate phrase queries.
		-->                          	
		<filter class="solr.LowerCaseFilterFactory"/>
		<filter class="solr.StopFilterFactory"
				ignoreCase="true"
				words="stopwords_de.txt"
				enablePositionIncrements="true"
				/>
		<filter class="solr.StopFilterFactory"
				ignoreCase="true"
				words="stopwords_en.txt"
				enablePositionIncrements="true"
				/>
		<!-- 1 split subwords english nouns -->
		<filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="wordlists/german-common-nouns.txt"
           		minWordSize="5" minSubwordSize="4" maxSubwordSize="15" onlyLongestMatch="true"/>
           	<!-- 2 split subwords german nouns -->
           	<filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="wordlists/english-common-nouns.txt"
           		minWordSize="5" minSubwordSize="4" maxSubwordSize="15" onlyLongestMatch="true"/>
		<!-- 3 expand english words to include the german translation -->
		<filter class="solr.SynonymFilterFactory" synonyms="wordlists/translationsgroups-english-german-nouns.txt" ignoreCase="true" expand="true"/>
		<filter class="solr.LowerCaseFilterFactory"/>				
		<filter class="solr.GermanStemFilterFactory"/>		
		<filter class="solr.SnowballPorterFilterFactory" language="English"/>			
		<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
	</analyzer>
	<analyzer type="query">
		<tokenizer class="solr.WhitespaceTokenizerFactory"/>
		<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
		<filter class="solr.LowerCaseFilterFactory"/>
		<filter class="solr.GermanStemFilterFactory"/>	
		<filter class="solr.SnowballPorterFilterFactory" language="English"/>
		<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
	</analyzer>
</fieldType> 

Downloads:

• list of common english nouns
• list of common german nouns (Substantive)
• noun synonym list with german, english translation groups

Links for wordlists

SOLR related links

Other Links

1. Cache-Control Headers: If your website sends proper "Cache-Control: public" HTTP Headers together with a proper expire that's perfect for caching: Your browser will store the website in its internal cache - and there is nothing that could be faster than this.
2. Varnish: Varnish is a "web application accelerator" - its build to cache content based on powerful configurations. (It also supports loadbalancing and failover handling)
3. TYPO3 Page Cache: TYPO3 caching framework is very flexible and should be used of course. It can handle Full-page cache based on parameters, logins and conditions. It also supports dynamic marker replacement and dynamic content parts (USER_INT).
4. Application Caches (Feature specific caches in our application logic)

How dynamic is my website?

1. based on time (news articles)
2. based on a login (e.g. after login I see personalized informations)
3. based on the user (even without login - e.g. based on location, cookie informations, browser history...)

Decide on a cache strategie

1 Use Cache-Control where possible

Problem 1: "USER_INT":

• is cHash used correct to have different cacheentries based on parameters?
• Does the plugin only show dynamic content after login? Then this is a nice solution:
  

  plugin.tx_felogin_pi1 = USER 
  [usergroup = *]
   	plugin.tx_felogin_pi1 = USER_INT 
  [global]

• Does the plugin only show dynamic content if a certain parameter is set? Then this is a nice solution:
  
  [globalVar = GP:aoe_solr|sword=]
    plugin.aoe_solr=USER
  [else]
     plugin.aoe_solr=USER_INT
  [global]

Problem 2: Login and User specific content

• Use ajax calls to add dynamic content (same like above). Especially wen you have a highly personalized website with a lot of user specific content there are solutions around, that use search technologies to display dynamic user-specific content. (e.g. the searchperience recommendation engine and widget features)
• or take care that the urls look different after login (to force the client to load new content):
  • The best is to switch from http to https after login. You can then tell TYPO3 not to send Cache-Control Headers in HTTPS:  (the condition depends on the server - also if using varnish you  need to find another way of detecting https )
    
    [globalString = _SERVER|HTTPS=on]
      config.sendCacheHeaders=0
    [global]
    
    
  • You can also simply add an additional parameter to the URLs after login, and dont send Cache Headers if that parameter exists ( config.linkVars=login )

2. Use Varnish

Problem 1: Cookies

• If you dont need login you simple may drop the cookies in varnish or set this TYPO3 setting:
  $TYPO3_CONF_VARS['FE']['dontSetCookie'] = TRUE;
• Better is to tell TYPO3 only to send a cookie if there is a valid user logged in. This can be done by installing some of the extensions (moc_varnish or cacheinfo). You may want to look at http://forge.typo3.org/projects/show/extension-cacheinfo , since this extension not only modifies the cookie behavior, but also adds several X-T3Cache* HTTP Headers that helps a lot...

Problem 2: rsaauth and PHPSESSION cookie

ESI - Edge Side Includes?

• Varnish needs to wait till all ESI calls are finished before it can deliver the page. That means the page deliver time is much lower compared to a page without ESI.
• The same could be achieved using ajax calls: Then the user gets the page asap and a single ajax call can be done (instead of multiple ESI calls). I think for most websites that require something like this it's ok to require javascript support.

Website Languages

TSConfig

mod.SHARED { 		   
  defaultLanguageFlag  = us
  defaultLanguageLabel = English - US 	 
} 

mod.web_txtemplavoilaM1.enableLocalizationLinkForFCEs = 1

mod.web_txtemplavoilaM1.hideCopyForTranslation >

TypoScript

 config {
   sys_language_mode = ignore
   sys_language_overlay = hideNonTranslated
   sys_language_uid = 0
   language=en
   locale_all = en_US.utf8
...
 }
...
 [globalVar = GP:L = 1] 	
   config {
 	  sys_language_uid = 1
 	  language = de
 	  locale_all = de_DE.utf8
        } 
... 
 [end]
...

TemplaVoila Localisation Modi

1. "Disabled" ( langDisabled = 1 )
2. "Inlinetranslation" ( langChildren = 1 )
3. "Inlinetranslation-seperate" ( langChildren = 0 )
4. "databaseoverlay" ( langDatabaseOverlay =1 ) *

Container FCEs

<meta type="“array”" />
        <noEditOnCreation>1</noEditOnCreation>
  	<langDisable>1</langDisable>
        <default>
          <TCEForms>
  	  	<sys_language_uid>-1</sys_language_uid>
  	  </TCEForms>
        </default>
...  

Content FCEs and mixed FCEs

<meta type="“array”" />
	<langDisable>1</langDisable>
	<langDatabaseOverlay>1</langDatabaseOverlay>
...   

<field_content type="array">
...
 <TCEforms type="array">
     <l10n_mode>exclude</l10n_mode>
...
 </TCEforms>
</field_content>
...

<field_image type="array">
...
 <TCEforms type="array">
     <l10n_mode>mergeIfNotBlank</l10n_mode>
...
  </TCEforms>
</field_image>
...

page DS

<meta type="“array”" />
  	<langDisable>0</langDisable>
  	<langChildren>1</langChildren>
...   

Special Attention

• don't forget to check your correct UTF8 setup. (If TYPO3 version is below 4.5 you need to set "forceCharset = utf-8" and maybe also setDBInit )
• TYPO3 uses the language ISO code as key for inlinetranslation in flexforms. If you have multiple localizations per language you need to add additional pseudo ISO codes. ( Read more in a related article "multilanguage websites with same language")

Lucene Revolution

realtime search

Big Data

semantic (calais)

edismax in solr 3.1

Stemming and Language detection

TYPO3 Conference San Francisco

continuous delivery

FLOW 3

cloud deployment

Links:

• T3CON11 San Francisco Vimeo Channel: http://vimeo.com/channels/207300/
• Slideshare: http://www.slideshare.net/event/t3con11sf
• FLOW3 News: http://news.typo3.org/news/article/on-the-road-to-flow3-10-beta-1/
• FLOW 3 Tutorials from Thomas: http://www.layh.com/work/flow3-fluid.html