The problem was to have dynamic values in the frames (section_frame) field of content elements, depending on the value of the column (colPos) field. If normal column was choosen, frames should offer value 80 and 81. If right column was choosen, frames should offer 82 and 83.

The first thing I did was to add colPos to the list of fields, which trigger saving and reloading the content element on change. I kickstarted a new extension and added these lines to extTables.php:

$updateFields = t3lib_div::trimExplode(
  ',',
  $TCA['tt_content']['ctrl']['requestUpdate']
);
if (in_array('colPos', $updateFields)) {
  $TCA['tt_content']['ctrl']['requestUpdate'] .= ',colPos';
} 

Then I changed the TCA of section_frame to use itemsProcFunc to customize its items:

$TCA['tt_content']['columns']['section_frame']['config']['items'] = array();
$TCA['tt_content']['columns']['section_frame']['config']['itemsProcFunc'] = 'tx_myext_setsectionframeitems->main';
if (TYPO3_MODE == 'BE') {
  include_once(t3lib_extMgm::extPath($_EXTKEY).'class.tx_myext_setsectionframeitems.php');
} 

Next step was to implement the class, which controls the items. The class itself was stored in a seperate file called class.tx_myext_setsectionframeitems.php:

class tx_myext_setsectionframeitems {
  function main(&$params, &$pObj) {
    global $LANG;
    $selectOptions = array();

   // Allowed fields for normal column
    $normalFields = array(
      $LANG->sL("LLL:EXT:myext/locallang_tca.xml:tt_content.section_frame.I.80") => 80,
      $LANG->sL("LLL:EXT:myext/locallang_tca.xml:tt_content.section_frame.I.81") => 81
     );
    // Allowed fields for right column
    $rightFields = array(
      $LANG->sL("LLL:EXT:myext/locallang_tca.xml:tt_content.section_frame.I.82") => 82,
      $LANG->sL("LLL:EXT:myext/locallang_tca.xml:tt_content.section_frame.I.83") => 83
    );
    // Get value of tt_content.colPos
    $key = key($pObj->cachedTSconfig);
    $colPos = $pObj->cachedTSconfig[$key]['_THIS_ROW']['colPos'];
    // Add items
    if ($colPos == 2) {
      $selectOptions = $rightFields;
    } else {
      $selectOptions = $normalFields;
    }
    foreach ($selectOptions as $key => $value) {
      $params['items'][] = array(
        $key,
        $value
      );
    }
  }
}

if (defined('TYPO3_MODE') && $TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['ext/myext/class.tx_myext_setsectionframeitems.php'])  {
  include_once($TYPO3_CONF_VARS[TYPO3_MODE]['XCLASS']['ext/myext/class.tx_myext_setsectionframeitems.php']);
} 

The custom labels were stored in a usual locallang file.

Finally, I did some TSconfig tuning in extTables.php to prevent invalid field values:

t3lib_extMgm::addPageTSConfig('
   TCEFORM.tt_content.section_frame.disableNoMatchingValueElement = 1
'); 

That's it. The whole stuff looks a bit ugly to me. So if you know a smarter solution, I'd be glad if you leave a comment.

I already got in contact with FLOW3 at T3DD09, listening to several talks about its architecture and components and playing a bit with the code. This milestone release comes just in time with Extbase, the forthcoming backport of the MVC pattern of FLOW3 for TYPO3 version 4.x.

I am very curious about the progress of these projects, since I am already infected by the new design patterns and programming paradigms. I also see more and more people testing and writing FLOW3 and Extbase stuff which makes me confident about its success. Still there is a lot of criticism and scepticism about FLOW3, especially the question of performance. Anyway this is a first alpha release and there's still enough time and code left to solve these problems.

Keep on with your good work!

What are heatmaps and how are they generated?

There are different ways of heatmap data generation. I will explain two of them in context of website usage analysis.  The first one is a click-heatmap which tracks the coordinates of all mouse clicks users perform on a website. The second one is an eyetracking-heatmap, which tracks the eye movement of a user when he looks at a website. Both methods use colorization to visualize those areas where user clicked or looked at. The more clicks an area has or the more time was spend for watching an area, the more reddish the area is colorized. The visualization follows assumptions of the color theory, which associate red colors more hot than blue ones. So hot areas on heatmaps reveal more user attention than cold ones.

Click-heatmap usecase

A click-heatmap gives an impression about the regions of a website users click - no matter if there's anything to click or not. Heatmaps therefore are very useful to identify usability flaws. They reveal those parts of a website, where users expect something to click, but in fact nothing exists to click. A popular example are those hugh logo images at the top of a website, where you could expect a link to the root page. In fact you often click in vain.

Eyetracking-heatmap usecase

Heatmaps also can be used to describe users' eye movement and favorite coordinates on a website. Jakob Nielson demonstrated this by creating heatmaps which proof the so called F-pattern. He describes the F-Pattern as a dominant scheme how users track a website. His heatmaps revealed a F-shaped area, where users looked most frequently at.

The image shows an F-Pattern heatmap on a google search result page.

Why not simply install a heatmap software to find out what users behave like...

You need to be sceptical about the obvious nature of a visual representation like a heatmap:

• In the wild, websites users have different tools and interfaces which render your wesbite differently. Screen resolutions or size and browser rendering can vary and bias the results of a heatmap survey. The more diverse user environments are, the more bias your results will have. And in most cases it is hard to estimate the bias.
• Content changes, especially on dynamic root pages with latest news. The quality of heatmap results declines the more frequent content changes or the longer a survey takes on a dynamic website.

Conclusion

A heatmap is a handy tool to visualize website usage. It's results are real eyecatcher. The reliability of the results can suffer very much if you are not in full control of the users environment. The perfect usecase would be a website usability pretest with controlled laboratory conditions. Heatmaps are no substitution to visitor or website traffic statistics, but can cast light on some hidden aspects.

After installing and setting up extBase, Fluid and the blog_example extensions, I added a new view helper folder in the blog_example extension:

mkdir typo3conf/ext/blog_example/Classes/ViewHelpers

This folder is generally used as a container for view helpers which are shipped with extensions. In the next step, I created a new which contains the helper itself:

touch typo3conf/ext/blog_example/Classes/ViewHelpers/LoremIpsumViewHelper.php

It's important to stick to this naming scheme, because otherwise Fluid will not find the view helper. Let's have a closer look at it. Beside the .php extension at the end of the filename, it contains two main parts: LoremIpsum and ViewHelper. LoremIpsum is the very name of the view helper and ViewHelper is a constant which is a mandatory part. The filename reflects the naming scheme of the class name of the view helper. So let's see what the file content looks like. I removed some comments to increase readability:

<?php

/**
 * This class is a demo view helper for the Fluid templating engine.
 *
 * @package TYPO3
 * @subpackage Fluid
 * @version
 */
class Tx_BlogExample_ViewHelpers_LoremIpsumViewHelper extends Tx_Fluid_Core_AbstractViewHelper {

    /**
     * Renders some classic dummy content: Lorem Ipsum...
     *
     * @param int $lenght The number of characters of the dummy content
     * @return string dummy content, cropped after the given number of characters
     * @author Lorem Ipsum <lorem@example.com>
     */
    public function render($lenght) {
        $dummyContent = 'Lorem ipsum dolor sit amet.';
        return substr($dummyContent, 0, $lenght);
    }
}

?>

This is a very basic example. My class extends the abstract view helper class of Fluid. I will not explain the Fluid API here. Let's concentrate on the example. The only method of my class is render() which is used to render the content of my marker. It crops a "Lorem Ipsum" string to a given lenght by using the substr() function. Some PHPDOC parts in the comments of the render() method are mandatory to get the view helper working. For example the @param part defines the argument type (int). The argument is being checked by the framework to ensure that the render() method always gets integer values. That's all for now in this file.

In a next step, I added the view helper to the Fluid template of blog_example:

# vi typo3conf/ext/blog_example/Resources/Private/Templates/Blog/index.html
{namespace blog = Tx_BlogExample_ViewHelpers} 
<h1>
  <blog:loremIpsum lenght="5" />
</h1>

First I had to declare a namespace for my view helpers. I chose to use blog, to be used as a shortcut to all the view helpers of the blog_example extension. Next is the marker itself. The syntax is easy to spot: The namespace and the viewhelper name is seperated by a : The lenght arguments is given afterwards

The result of using this template marker is a level 1 header with "Lorem" as the text:

<h1>Lorem</h1>

That's all. I am really exited about how easy it is to write own view helpers. I hope this article could demonstrate that and motivate you to get into the FLOW3/Fluid/extBase universe.

Jochen and Sebastian have just introduced the basic principles which are realized in Extbase. Extbase is meant to be a new basis to create frontend extensions, which yet has been handled by pibase

Extbase promises to end writing spaghetti code which was common and sometimes unavoidable in pibase extensions. It tries to implement MVC pattern, dividing things into several logical layers.

The presentation is very inspiring, the concepts sound understandable and things are well visualized. Also Jochen is a great speaker who not only seems to know one's stuff but is also a very skilled teacher. The audience seems to be very interested, since a lot of discussion is going on.

The Extbase session will be follwed by an introduction to Fluid, the new templating engine which is going to be released wth Extbase. So far it looks very promising, making templating much more flexible and powerful than the marker based approach.

There are lots of tutorials, howtos, blog postings and documentations around which aim to help you converting a character set to UTF-8. Some are more helpful, some are less. But all have something in common: They have been written without considering how your system is configured. Analyzing your individual environment paired with a deep understanding of your components will probably serve much better than trial-and-error snippets. I'll try to give you an impression what that could mean.

Taking all components and their interaction into account

The architecture of a TYPO3 system is complex, since it consists of independent components, which are even interchangeable. The following questions are essentials when thinking about character set conversion:

• Operating system (Win, Linux, *BSD, ...): What's the default charset of your OS? Does it support UTF-8?
• Filesystem: What charset do the files have? Is it used consistently or mixed?
• Webserver: Which charset is used for serving static files?
• Database: This one is the most complex. The DB has various components, which can come with different charsets:
  
  
  • Client: Which charset does the client use to display/process data?
  • Server: How does the server store the data?
  • Connection: What charset is used for data transfer?
  • ...
• PHP: There is a bunch of modules, which try to help with on-the-fly charset conversion: recode, mbstring, iconv, ... Which one you have, depends if the module is available for your PHP version and if it's enabled.
• TYPO3: What version do you use? What charset configuration?
• The whole stuff mixed together: What charset do the components use to interact with each other?
• ...

Uff, much stuff! But this is just a loose and incomplete collection of items. I stopped brainstorming after a few minutes. I am sure there's much more to think (and write) about. I will not give you advices what things to do or not to do. As I already mentioned, there's a lot of writing out there in the web.

Demonstrating my favorite issue

This one was really tricky to find out. And it serves as a great example, how complex things can be:

Updating from TYPO3 4.1 (and earlier versions) to 4.2 (and later)

Some DB fields in 4.1 are of type BLOB (for example the TS templates). Most of these fields get converted to TEXT in 4.2. Now think about the following scenario, which seems to be common. The template was saved using TYPO3 4.1 and a DB using latin1 (ISO-8859-X) as charset. Then the DB converted to UTF-8 and TYPO3 was configured accordingly. You think you're finished because everything works will. But in most cases, there's still some latin1 formatted data in the BLOB fields. You just don't see. Once you upgrade TYPO3 to 4.2, these BLOBs get converted to TEXT, assuming the data is UTF-8. But it's latin1, because BLOB was not converted before. The result is a broken template. Lots of people in the mailing lists complain about whole parts missing. The reason is invalid non ascii characters (like umlauts äöüé¢ etc), which break the template view.

How to avoid that?

If you change the charset of TYPO3 and/or your DB, convert the BLOB fields to TEXT before charset conversion or make sure to convert the BLOB data otherwise.

Lessons learned?

• Don't believe that using snippets from the web will fit your needs 100%. Even if they are written by someone with good reputation and rated well, chances are that your system behaves different.
• Structured Analysis paired with sufficient knowledge about CMS components are the best basis for successful administration. Trial-and-error will not always lead to a lucky punch and can be very disappointing.
• Character set conversion is a complex task, even for the experienced administrators. Complex tasks are time-consuming, so calculate generously.
• Design flaws can cause a disaster which might come up at a point you can't foresee. Using BLOB for text in databases is such a design flaw.

Finaly, a tiny helper extension

Sometimes it's not easy to find out what charset TYPO3 uses when connecting to MySQL. Asking for help, you probably will be advised to provide the output of

SHOW VARIABLES LIKE '%CHARACTER_SET%'

But if you do that on the command line (using the mysql client) or in phpmyadmin, you might get a different result than TYPO3 would produce.

I have written a tiny charset helper extension, which shows the results from a true TYPO3 point-of-view (by using standard TYPO3 DB functions). It's available from TER and does not need any configuration. just install it and navigate to the module:

http://typo3.org/extensions/repository/view/sm_charsethelper/current/

First impression

The first positive impression when unpacking the book was that it covered only 217 pages.  It was looking like a book, which can be read entirely without getting tired of stuff you don't want to read anyway. As the title implies, the book addresses TYPO3 extension developers, especially those without much experience in extension development.
It guides the reader through the complete process of extension development, covering both frontend plugins and backend modules with well-chosen real world examples.
Although I feel quite familiar with the task of extension development, I was surprised that the book still revealed some gems of extension development for me, especially the in-depth study of frontend plugins (chapter 5). But more on that later.

Book structure

The book is well structured, starting with general topics on hacking with TYPO3 and getting more and more specific to the end. This reminded me of the talks Dmitry held at T3DD08, which were also very well organized and easy to follow.

Chapter 1 ("About TYPO3 API")

The first chapter concentrates on TYPO3 API. Since the API is very huge, the chapter concentrates on the some relevant topics, leaving out unnecessary details which anyway could be found on the official documentation on typo3.org or any doxygen created API docs. It's flavored with some code examples which helps to understand the use of the discussed items. Since this is the first chapter of the book, I would have expected a more general introduction in the task of extending TYPO3, especially for the unexperienced user.

Chapter 2 ("Anatomy of a TYPO3 extension")

The second chapter discusses the usual files inside an extension. It rather addresses people which are new to TYPO3 extension development, giving an overview about the purposes and uses of extension files and directories. Nevertheless it covers the mandatory knowledge for the following chapters.

Chapter 3 ("Planning Extensions")

If you have already heard about the author Dmitry Dulepov and follow his blog, you'll probably won't be surprised about chapter 3. Dmitry doesn't hide his passion for elaborate planning. Consequently, this topic got a separate chapter. It guides you through general and TYPO3 specific planning questions, which need some thinking before starting to code. In practice the chapter describes the goals of the example extension, which will be coded in the following chapters. So don't fear to be confronted with boring theoretical stuff, but with tips which help to code standard-compliant extensions.

Chapter 4 ("Generating Extensions")

This chapter is a guide on how to use the extension kickstarter for generating the base structure of an extension. Like chapter 2 it also addresses people without or with less experience in extension development. Screenshots and their description explain the main sections of the kickstarter. Unfortunately only some parts of the kickstarter are discussed. Creating services and clickmenu items for example are not, which personally left me with a mixed feeling. If you depend on these kind of extension features, you'll have to search elsewhere for the necessary information. Fortunately, François Suter has recently published a completely rewritten manual about TYPO3 services.

Chapter 5 ("Frontend-Plugin: An In-Depth Study")

Chapter 5 focuses on some particular parts of a TYPO3 extensions. These are plugin configuration, templating, localizing and caching, which are essential for frontend plugins. Each part and its particular difficulties are explained and demonstrated with code examples. Some of you probably got in contact with the deep jungle of TYPO3 caching, especially when reading the article "Mysteries of cHash" again and again, trying to finaly understand it. Dmitry manages to blaze a trail through that jungle with concise explanations and best practice examples. The only drawback in the caching chapter is a mistake in the noncached plugin example on page 111: it comes with USER instead of USER_INT. The alert reader will notice.

Chapter 6 ("Programming Frontend Plugins")

This section is the longest part of the book and discusses the coding of crucial parts of a plugin. It does so by using an example extension which presents website users. The reader learns how to create single and list views by writing and linking together PHP code, HTML templates, TS and FlexForm configuration and a simple (but very cool) AJAX portion. It's amazing to see how Dmitry manages to code rich features with that few lines of code. My favorite part is the use of stdWrap inside PHP.

Chapter 7 ("Programming Backend Plugins")

Chapter 7 is divided into two parts: database access by TCA (TYPO3 configuration array) and backend modules. Since most of the TCA parts are created by the kickstarter, the first part is accordingly thin. It describes only the essential parts of TCA. If you are interested in more details, the official core API documentation might be a better place to browse. Well, back to Dmitrys book: The second part begins with a short excursion on essential functions for backend modules, followed by a step-by-step howto about implementing a full featured BE module. The example which is used here displays user and page statistics. For my taste this part contains too much code and not enough explanation.

Chapter 8 ("Finalizing Extensions")

The last chapter leads the user through the documentation and publication process of an extension. A checklist helps to increase the overall quality of the extension.

Index

The book closes with an index, which is far too short. My first impression was that a page was missing in the book, but then I realized that the letters G, H, I, J, K, M, N ,O simply don't have any record in the index. That was very disappointing!

Conclusion

My overall impression of the book is that it contains many best practices in extension programming. It covers essential knowledge for coding a frontend extension with various features. Unfortunately the backend module part is too thin. Considering that the TYPO3 backend code is ugly and the API is hard to understand, this part should have deserved more attention. But altogether the book leaves the impression of a solid guide for extension developers. It was quite amusing to read because of its real-world examples. Finaly, don't forget to have a look at the various extensions Dmitry published and maintains.

After searching a while for an existing version of TYPO3 trunk without success, I decided to generate my own. You can find the TYPO3 core API of trunk on a seperate page of this website.

The search form allows you to find any function in the TYPO3 core within a few seconds and get all facts of its use, parameters and dependencies to other classes/functions. It gets updates on a regular basis via cron. If you like it, you can use it:

http://www.t3node.com/typo3api/trunk/

page = PAGE
page.10 = CONTENT
page.10 {
  table = tt_content
  select {
    pidInList = 123
    where = deleted=0
    andWhere.data = GP:my_test|uid
    andWhere.wrap = uid=|
  }
}

The content of page.10 is a content element of page 123. To choose a certain content element, you can use a GET/POST parameter as the uid.

See it working

To see the example working, add some content elements to a page. We use page 123 in this example. Then note the uid of one of these content elements. In the example we use 42. Go to the frontend page in your browser and query the page:

http: //yourdomain.tld/index.php?id=123&my_test[uid]=42

This will output the content element with uid 42 on any page.

See the SQL injection working

What's the matter with this? The GET variable my_test[uid] is used inside a SQL query without being sanitized, open for SQL injection! Just try this URL:

http: //yourdomain.tld/index.php?id=123&my_test[uid]=42 OR 1=1

You should now see ALL content elements of your website. This is because OR 1=1 is injected and processed in the SQL WHERE clause.

How to prevent this?

The GP parameter my_test[uid] should only accept integer values. In PHP one could use the function intval() to make sure the given value gets converted to type integer. Luckily, there is an equialent in TypoScript for that. select.andWhere has stdWrap properties, so we can use the intval property inside andWhere. The above TypoScript snippet would then look like this:

page = PAGE
page.10 = CONTENT
page.10 {
  table = tt_content
  select {
    pidInList = 123
    where = deleted=0
    andWhere.data = GP:my_test|uid
    andWhere.wrap = uid=|
    andWhere.intval = 1
   }
}

Be careful, whenever you create SQL queries which processes GET/POST parameters. Using TypoScript solely does not avoid the danger of SQL injection.

Since TYPO3 4.2 CSH is also supported for FlexForm fields. Integration is very easy since you only have to create a file with the CSH content and add a line to your FlexForm pointing to that file. Have a look at the following example.

As always the FlexForm is configured in XML. The item which points to the location of the CSH content must be placed inside the <ROOT><TCEforms> tags. The name of the element is cshFile (and it's case sensitive!).

The FlexForm file your_extkey/pi1/flexform_ds.xml looks like this:

<T3DataStructure>
  <meta>
    <langDisable>1</langDisable>
  </meta>
  <sheets>
    <sDEF>
      <ROOT>
        <TCEforms>
          <sheetTitle>The title of the sheet</sheetTitle>
          <cshFile>LLL:EXT:your_extkey/pi1/locallang_csh_flexform.xml</cshFile>
        </TCEforms>
        <type>array</type>
        <el>
          <yourField>
            <TCEforms>
              <label>Label of the the field</label>
              <config>
                ... field configuration ...
              </config>
            </TCEforms>
          </yourField>
        </el>
      </ROOT>
    </sDEF>
  </sheets>
</T3DataStructure>

The significant line is marked in strong letters. Next step is to add the CSH content as already shown in the previous article.

The file should be named your_extkey/pi1/locallang_csh_flexform.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<T3locallang>
  <meta type="array">
    <description>CSH labels for FlexForm fields</description>
    <type>CSH</type>
    <csh_table></csh_table>
    <fileId>EXT:your_extkey/pi1/locallang_csh_flexform.xml</fileId>
    <labelContext type="array"></labelContext>
  </meta>
  <data type="array">
    <languageKey index="default" type="array">
      <label index="yourField.alttitle">alternative title</label>
      <label index="yourField.description">description</label>
      <label index="yourField.details">details</label>
      <label index="yourField.syntax">syntax</label>
      <label index="yourField.image_descr">image caption</label>
      <label index="yourField.image">gfx/i/pages.gif</label>
      <label index="yourField.seeAlso">pages:starttime</label>
    </languageKey>
  </data>
</T3locallang>

That's even more simple than adding CSH to a BE module. At the time of writing this article, one features does not yet work as it does for TCA fields. Hovering the mouse pointer over a field title should trigger a popup which shows the title and description of the field. This does not work in FlexForms. I have opened a bug and submitted a patch to the core list to fix this issue and hope it will be resolved in the forthcoming 4.3 release. In the context of the bugreport you can also find a test extension, which serves as a showcase for FlexForm CSH integration.

All you need to have is:

1. the content itself, put into a classic locallang file called locallang_csh_txYOUREXTKEY.xml
2. the item, placed in mod/index.php
3. a reference to tell the extension manager about the locallang file

Let's start with the first one. I prepared a skeleton file, which explains the most significant fields:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<T3locallang>
   <meta type="array">
      <description>CSH contents of myextkey module</description>
      <type>CSH</type>
      <csh_table>_MOD_tools_txmyextkeyM1</csh_table>
      <fileId>EXT:myextkey/locallang_csh_txmyextkey.xml</fileId>
      <labelContext type="array"></labelContext>
   </meta>
   <data type="array">
      <languageKey index="default" type="array">
         <label index=".alttitle">Module title</label>
         <label index=".description">
            This part describes the module and is also shown
            when hovering over the CSH icon [hover]
         </label>
         <label index=".syntax">
            This part is a predefined section and
            can be used to demonstrate coding syntax.
         </label>
         <label index="mod_itemname.alttitle">Item title</label>
         <label index="mod_itemname.description">
            This part describes a certain item [hover]
         </label>
         <label index="mod_itemname.details">
            This part gives more details of the item
         </label>
         <label index="mod_itemname.image">
           EXT:myextkey/res/imagefile.png
         </label>
      </languageKey>
   </data>
</T3locallang>

Next, add a CSH item to the content of the module. Of course you can have more than one item:

$content .= t3lib_BEfunc::cshItem(
   '_MOD_'.$GLOBALS['MCONF']['name'],
   'mod_'.$str,
   $GLOBALS['BACK_PATH'],
   '|',
   FALSE,
   'margin-bottom:0px;'
);

The function cshItem() is well documented inside the source code. I was lazy and simply copied the descriptions from the sources:

string   Table name ('_MOD_'+module name)
string   Field name (CSH locallang main key)
string   Back path
string   Wrap code for icon-mode, splitted by "|".
         Not used for full-text mode.
boolean  If set, the full text will never be shown (only icon).
         Useful for places where it will break the page
         if the table with full text is shown.
string   Additional style-attribute content
         for wrapping table (full text mode only) 

Finally add a reference to ext_tables.php to tell the extension manager about the CSH:

t3lib_extMgm::addLLrefForTCAdescr(
   '_MOD_tools_txmyextkeyM1',
   'EXT:myextkey/locallang_csh_txmyextkey.xml'
);

That's all you need to do for a backend module. You can find out more about CSH in the Inside TYPO3 documentation.

To get a quick impression about what services are listening on what socket (IP/port), I use netstat:

spock@cayenne:~$ netstat -ltpn

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3690          0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:50001         0.0.0.0:*               LISTEN      11201/firefox   
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      - 

Whenever I am at home, OpenOffice.org and Zotero are running as expected. As soon as I leave home (e.g. to spend a day in the library) OpenOffice.org refuses to connect to Zotero, complaining that Zotero might not be running:

But Zotero was up and running?! I remembered I had this error before, when updating Zotero to a new version without updating the OpenOffice.org extension. But this time there was no update.

After some investigation, I found out that one can turn on the logging of Zotero using about:config in Firefox (extensions.zotero.debug.log = true). The logs revealed that Zotero wasn't able to to offer it's HTTP service, because Firefox is in offline mode:

(...)
Browser is offline -- not initializing integration HTTP server
(...)

netstat could also not find Firefox listening on port 50001:

spock@cayenne:~$ netstat -ltpn

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3690          0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      - 

The laptop I use for writing is always online at home. In the library it is usually offline except for the localhost dummy interface (127.0.0.1). Firefox on Ubuntu Intrepid seems to switch to offline mode, if no outbound network connection can be found. In offline mode, Zotero does not provide its service at all, although it does in no way depend on an outbound network.

The solution was to switch to online mode in Firefox, which lets Zotero start its service immediately.

What can we learn from the story?

• Do not blindly believe in error logs: They sometimes don't tell the whole truth. Instead of precisely describing the problem, the OpenOffice.org popup made useless suggestions ("Make sure Firefox is open").
• Do not blindly wipe away error logs: Chances are, that you find the reason for your problem inside. Zotero loggin was hard to find, but then revealed the root of my problem.

For a considerable time, TYPO3 ships an out-of-the-box function to add a page browser to your plugin. The function pi_list_browseresults is part of the cObject and is called by:

$content .= $this->pi_list_browseresults(
   $showResultCount=1,
   $tableParams='',
   $wrapArr=array(),
   $pointerName='pointer',
   $hscText=TRUE
);

Unfortunately the output of the function is a HTML table which is semantically not a proper form for a page browser. The configuration misses the power of common templating, although you could serve most needs with TypoScript wraps. To find out how to implement pi_list_browseresults, you'll either have to dig into some 3rd party extensions or search the archives for comments. Therefore we leave this function as is and turn to another approach.

Extension "Universal page browser" (pagebrowse)

The Universal page browser extension from TER is meant to be used inside your plugin as a powerful alternative to pi_list_browseresults. The styling of the page browser is done by HTML templating and is controlled by TypoScript setup. It can be used inside any extension by integrating it as a USER or USER_INT cObject. (Details on how to do that are perfectly explained in one of Dmitrys blog postings). Additional hooks give you the possibility to keep control of the generation of the page browser. The extension has a straight documentation, describing the implementation with commented code snippets.

Comparison result

Both approaches allow the integration of a page browser into your frontend plugin. The convincing documentation and the HTML styling ability of the "Universal page browser" extension finaly make the difference, especially if you rather rely on standard writings than on searching for code snippets.

1. odt2txt for extracting the content of an OpenOffice.org document. (A Debian package is available)
2. oodiff for comparing the content.

The real easter egg is that you can compare your local copy with the latest subversion revision. Well, oodiff does not really connect to subversion, but uses a local copy of the .svn directory. Git and Mercurial are also supported.

Today, I tried to see how it works. The setup for TYPO3 was really easy and it worked without any problems. Installing memcached and php module on Debian was a nobrainer, so no time was wasted.

But my first tests were a bit disappointing. First, I used memcached on a lightweight virtualbox and the gain of using memcached instead of DB tables for cache_* "tables" was zeeeeero:

 ab -c 100 -n 5000 http ://virtual.box.test/ 

The tests revealed ridiculous 8 req/s (mean) both using DB tables and memcached for caching. I realized, that the CPU could have been the bottleneck, because it immediately went to 100%. My conclusion was that the virtual box was too lame to get significant results. So I decided to switch to a real machine.

I decided to use an AMD BE-2350 machine for further tests.

 ab -c 100 -n 5000 http ://physical.box.test/ 

The same basic tests with ApacheBench resulted in overall better performance and a slightly difference between memcached (170 req/s) and classic caching (135 req/s). But still not the boost I had expected. Once again looking at the CPU load, it was 100%. Most load was caused by fcgid and mysql, but I couldn't do any further tests because midnight cronjobs started and I did'nt want to stop them.

The next step will be to replay the tests on a machine using mod_php instead of fcgid. But I'm also looking forward for some more reports from the community to crosscheck my own results.

Anyway, the backport of the caching framework is very good work. It spends hope to have a smooth migration path to FLOW3 and TYPO3 v.5. So a big thank you goes out to core devs  for their efforts.

This is the regular view on all threads of a newsgroup:

You can hide all threads which do not have any unread postings:

I did test it and it works as expected. Thanks, Dmitry. It really helps me.

The convention to wrap the plugin content with an additional div container is not fair in any case. It should be configurable.

We usually find the following piece of code at the end of the main() function:

 return $this->pi_wrapInBaseClass($content); 

The reason for this might be boilerplate code from the extension kickstarter.

Criticism

This is not only a lack in usability, but one more violation of the MVC pattern. The method adds additional HTML tags to the content. This should be part of the view and not the controller. One could make the objection that tslib_pibase anyway does not follow the MVC pattern. If so, we still have to face the usability aspect.

Solution

In TYPO3 the output can be controlled by TypoScript. We should make use of it to provide a baseWrap option making that stuff configurable:

public function main($content, $conf) {
  // (...)
  return $this->baseWrap($content);
}
 
protected function baseWrap($content) {
  if (isset($this->conf['baseWrap.'])) {
    return $this->cObj->stdWrap($content,$this->conf['baseWrap.']);
  } else {
    return $this->pi_wrapInBaseClass($content);
  }
} 

The content gets wrapped with the container by default, so backward compatibility is guaranteed. If user wishes to change that, he could use the stdWrap functions with baseWrap in his TS setup:

 plugin.tx_pluginname_pi1 {
   # Remove basewrap div container
   baseWrap.wrap = |
 } 

It takes you only a few minutes to do the magic.

Thanks to Francois Suter for his inspiration.

The idea is to create a link to a file, which is set in the media field of the page. You can find this field in the page properties below the ressources tab and its label in the backend is "Files".

Once you have chosen the file in the backend, it gets copied to the uploads/media/ directory. In the plugin, we need to get the filename and then simply build a link around it. "That's easy", you might think, "just get the filename from TSFE and put some a href tag around it and it's finished". But as this snippet will show, you don't need to think about HTML at all. The magic lies inside the filelink function of TypoScript.

// path to the file
$filePath = 'uploads/media/'.$GLOBALS['TSFE']->page['media'];
// use the filelink function to create a rich feature link
$content = $this->cObj->filelink(
  $filePath,
  $this->conf['fileLink.']
); 

This is only a few lines of PHP code! Next we need to configure the filelink with in the TypoScript setup of the plugin:

plugin.tx_myplugin_pi1 {
  fileLink {
    file.noTrimWrap = | | |
    labelStdWrap.data = page:media
    icon = 1
    icon_link = 1
    size = 1
    size.wrap = (|)
    size.bytes = 1
    size.bytes.labels = " B| KB| MB| GB"
  }
}

The result is a full featured link with an icon corresponding to the content type and the size of the file:

The ruleset hides at http://typo3.org/waf.txt

I didn't test it but had a quick look at the file. It's a quite short configuration and I could not spot any TYPO3 specific rules. One lines points to an external file called modsecurity_crs_9999_typo3.conf but I couldn't find that file. Well, waf.txt also reveals that the current version was written on September, 2007. So maybe a newer one is already released, but kept secret somewhere else ;-)

Not only are they breaking internet standards and abuse privacy, but they act as a content provider. And this may have adverse consequences for them:
In Germany, all content providers ("Anbieter von Mediendiensten") are forced by law ("Telemediengesetzes") to maintain an impress on their website. If they do not, they risk to get some kind of cease-and-desist order ("Abmahnung und Unterlassungsaufforderung"), which not only costs money but could turn press spotlights on this abuse. This hopefully attracts lawyers as well as the press and stops this nonsense.

Btw, Verisign once started a similar "service" which earned lots of critique. They could have learned from that.

Now finally, let's welcome Alice on my personal blacklist of companies-to-be-never-recommended.

For example, a friend of mine is collecting music records. He needs to order the records in his shelf to be able to quickly find a particular album. He chose the following hierarchical order:

 1. Genre
 2. Musician/Band name
 3. Album name 

This kind of order is sufficient to find most of his albums. But from time to time, he has to face the situation to be wary about which genre to choose for a new album. Dub, Reggae or Ska? The circumstance to decide in terms of either/or pushes him into the uncomfortable situation of making more or less poor decisions.

Now think about digitalizing all his records. The usual way to structure the digital copies is to save each song as a single file on the harddisk. These files are being stored inside directories, which represent the band name and/or the album name:

/Music/Bandname/Albumname/Songname.ogg

What pops up here is the habit to maintain structures of the analog world when transforming content into a digital form: A song belongs to a record and the record belongs to a band. This hierarchical paradigm seems to be widespread on desktop computers, although it is not mandatory. One can just as well use a database instead of a filesystem. In case of finding a certain song the represented file structure is not a limitation, because one can add any meta information to a song he likes. For example, a genre can be assigned to a song by using an id3 tag. Based on this meta data, a music library software can crawl all directories and list all Reggae songs. We even could assign more than a single genre, just like this blog article is tagged with multiple tags. No more decision in terms of either/or, but in terms of as-well-as. That means more freedom in classification and less misdeterminations caused by enforced yes-or-no decisions. In contrast to the analog world, digital content can be separated from its mode of presentation. In combination with networking, this opens a new kind of freedom when it comes to classify content. In particular, collaborate classification as realized in so called folksonomies seems to be very helpful and popular.

The other side of the coin is an increasing diffusion which may lead to disorientation. Human beings are heavily bound to the rules of the analog world and used to its limitations. Simple models of classification help to avoid uncertainty, even when they tend to cause wrong decisions. The challenge in a digitalized world is to make use of collaborate forces against the increasing confusion in a complex world. Folksonomies as a classification model seem to be a helpful step towards that. There might come a day when we have to skip using hierarchical ordered files and folders, because this kind of categorization gets too complex for handling the rapidly growing data.

Today, I did some tests on moving from outdated SuSE to Debian Etch. Cloning the database and the files was easy and went fine, but then I stumbled over wrong date localization of tt_news. The website comes with german as default language and english and french as secondary. Nevertheless, the fresh clone on the Debian server ignored all language settings, using english formats for all pages.

The reason was the config.locale_all setting in the TypoScript template. My configuration was:

config.locale_all = de_DE
[globalVar = GP:L = 1]
  config.locale_all = en_EN
[global]
[globalVar = GP:L = 2]
  config.locale_all = fr_FR
[global]

and this worked fine on SuSE. However, Debian does not come with any of these locales and TYPO3 switched to default (english). I found out, that the locales on Debian use a different naming scheme than SuSE does. Switching to the command line,

locale -a

revealed the names of all installed locales. That was de_DE.utf-8, en_US.utf-8 and fr_FR.utf-8. I changed my configuration accordingly and everything worked fine:

config.locale_all = de_DE.utf-8
[globalVar = GP:L = 1]
  config.locale_all = en_US.utf-8
[global]
[globalVar = GP:L = 2]
  config.locale_all = fr_FR.utf-8
[global]

Details about the issue:

I got a mail on June 30, 2008 from the security team about the extension. It allows BE users to enter username, password and hosts in the filelist module. The problem here was, that the host parameter was "not properly sanitized, making it possible to add arbitrary code lines to a htaccess file", the security team wrote. An example how to insert malicious values was appended:

POC: all%0DAddType application/x-httpd-php .txt

This could lead to arbitrary code execution, because user were then able to upload executable PHP code in the filelist module (under certain conditions, see below).

Description of the patch:

I added a routine to compare the input field against a whitelist of characters to prevent injection of control characters.

A fixed version was released on July 8, 2008. So far the technical facts.

Criticism:

What I do not understand, is the bulletin severity of HIGH:

The vulnerability only affects systems with unsafe Apache configuration, namely the infamous AllowOverride All for .htaccess context. It is even mentioned in the extension manual not to use that, because it opens doors for security holes. So no excuse if someone didn't give a f*** about this and was hit by the issue.

Also, why did the security team wait more than two month to release a bulletin earlier, if the severity is so high?

Please don't get me wrong. The issue was there and some user may have been affected. But there was no description at all in the bulletin, and I wanted to cast a light on this.

The first place to look for those features is the pending documentation section in the TYPO3 wiki. It's meant to be the place for documentation of all new features, which have been added to the trunk, but not yet released. The items are collected here, reviewed and then put into the docs at typo3.org. Sometimes, this process is a little bit delayed and the wiki page becomes the only referrer to new features of an actual release. In this context, we can see Dmitry bashing Michael in the core list about that: "(...) Stucki is responsible for [not] updating core documentation in time. kick him, I am already tired of doing it. (...)" (Dmitry, 2008/07/31).

Forge: Core docs in the subversion trunk

Francois kindly updated TSref with all pending parts from the wiki. To be release soon...

Bugtracker: documentation category

Another place is the bugtracker. There is a documentation section in mantis, where tough users could have reported items about missing documentation.

Mailinglists: recently committed features

Last but not least, you could RTFML, especially the core list. Use the search function of support.typo3.org for a full bodytext search. Although "we cannot expect from our users to read the Core list to learn about new features" (Masi, 2008/08/04), it however seems to be the price one has to pay for living on the bleeding edge.

• Use TYPO3 Version 4.2 (because HMENU does not use typolink in >=4.1)
• use RealURL Version >=1.4.0
• setup TypoScript of all rootpage templates (the first three items of course can be merged into a global template):



config {
   simulateStaticDocuments = 0
   tx_realurl_enable = 1
   # render internal links between domains
   typolinkCheckRootline = 1 
   baseURL = mydomain.com
}

• set domain record in every rootpage
• set the rootlevel flag in the rootpage templates
• set the "Is root of website" flag in the rootpage properties. This is essential for nested domains.
• set the rootpage_id for every domain in the RealURL configuration part in typo3conf/localconf.php:



 $TYPO3_CONF_VARS['EXTCONF']['realurl'] = array(
   'www.foo.com' => array(
     'pagePath' => array(
       'rootpage_id' = 1234
   ...
   ),
   'www.bar.com' => array (
     'pagePath' => array(
       'rootpage_id' = 5678
   ...
   ),
 

However, you could still run into problems. Keep your eyes open for recent patches in the TYPO3-core list and related bugs.

• The user interface looks intuitive and familiar with its three columns layout.
• The clear use of colors gives you a quick overview about important actions.
• Most important: I was able to recover the "do it, defer it, delegate it" pradigm at once.

The next days I'll try to find out how good it really fits my needs...

The bookmarks are encoded as JSON and stored as a single file. Fortunately, Stefan Macke (german!) now provides a PHP script to convert JSON to HTML to display your bookmarks on a website. Thank you, Stefan!

So if you run into trouble with Ubuntu 8.04, download the patch to your typo3_src directory and patch your sources:

patch  -p0 < 0006158_v2.patch

When in Insert mode, type the beginning of a PHP function, for example

array_s

Then hit <Control-X> <Control-O> and vim will show you all possible functions in a drop down menu. Just choose the one you were looking for and hit enter

array_search( 
array_search( f
array_shift(  f
array_slice(  f
array_splice( f
array_sum( f

Find out more about vim secrets: www.vim.org/tips/tip.php

A prominent example is config.linkVars. In most cases, it's used to append the global language parameter L to typolinks:

config.linkVars = L

However, if you need to append another value to that property using a basis template or an extension template, you can do this with := addToList():

config.linkVars := addToList(print)

Parsing both properties will have the same result as:

config.linkVars = L, print

List of functions

It's so easy, isn't it? Mind the := operator as a placeholder for the parser to process the function. The "TypoScript Syntax and In-depth Study" docs reveal the following list of functions which can be used here. There's even a hook to use your own functions.

prependString()

... adds a string before the existing one.

appendString()

... adds a string after the existing one.

removeString()

... removes a string from the existing one. str_replace() is used here.

replaceString()

... replaces an existing string with a new one. str_replace() is used here,with the pipe symbol | as a divider.

addToList()

... appends values to a comma separated list. The list will neither be sorted nor distinct. Multiple values are given as a comma separated list.

removeFromList()

... removes values from a comma separated list. Multiple values are given as a comma separated list.

Today, I had enough and tried to find out, what 43 stands for. Finaly, the source code documentation led the cat out of the bag: 43 is the uid of the static template 'content.default' and the function name means: Add PlugIn to Static Template #43. Now don't worry if you use css_styled_content instead of static templates: The docs promise it will work anyway.

HTML

<body id="papers">
...
<ul id="navMenu">
  <li class="blog">My blog</li>
  <li class="papers">My papers</li>
  <li class="books>My books</li>
  <li class="contact">Contact</li>
</ul>

CSS

#blog #navMenu .blog a,
#papers #navMenu .papers a,
#books #navMenu .books a,
#contact #navMenu .contact a, {
  color: red;
}

The first time I realizied those million ways to apply styles to links, was some years ago, when Wikipedia started to explicitly tag external links with png icons. Last week I found some time to dig into CSS and attribute selectors and the stuff I saw was amazing. Not only we can style one and the same HTML element with different ids or classes, but also by choosing any other attribute or part of it. On a small scale, a kind of Regex is allowed to differentiate cases more precisely:

HTML

<a href="http://www.domain.net">An external link</a>

CSS

a[href^="http:"] { background: url(ext.png) no-repeat right; }

This will add a little icon on the right side of the link. The attribute selectors [href^=value] limits the style to links starting with http:. If your browser supports CSS2, you can see the effect. Now have a look at the next example:

HTML

<a href="http://document.pdf">my Thesis</a>

CSS

a[href$=".pdf"] { background: url(pdf.png) no-repeat right; }

This will highlight links to pdf documents. The attribute selectors [href$=value] limits the style to links ending with .pdf. And here comes the example.

Now, that IE7 is going to replace its ugly ancestor, attribute selectors are also supported by the MS empire. The next step seems to be microformats, which mainly use attributes to semantically markup web content.

The solution to get some more readable output is so easy: Terminate your query with \G instead of ; or \g

• pdfinfo binary (gzipped, version 3.02)
• Xpdf sources as tar.gz archive (version 3.02)
• pdfinfo.cc source file, modificated by zotero dev, see Zotero svn

[UPDATE 2007/10/11]

The bug is already fixed and new binaries are provided. Nevertheless, if you still run into the described problems, it might be because the old file still exists in the Firefox cache. Clear the cache or alternatively get the file from:
www.zotero.org/download/xpdf/pdfinfo-Linux-i686-3.02

• Zotero comes as a Firefox extension. I like the idea, because the browser is my favorite tool for almost any research.
• It also ships an Openoffice package, to integrate the references into my favorite word processor.
• It's open source software.
• There's an active community.

Zotero does not only collect research sources on a meta level (author, title, year, ...) but also archives and indexes full text items (e.g. HTML, PDF). Tags and relations make it easy to structure and categorize items.

The installation is really easy and the documentation helps you to understand things within a short time. Give it a try.

Luc seems to like charts, since we already have some postfix statistics of the TYPO3 listserver and some numbers about list membership.

Fortunately, Andrew McMillan provides a fresh eAccelerator package for Etch. There have been some questions on the TYPO3-Debian mailing list about installing third party packages on Debian Linux. To clearify things, here comes a quick installation guide.

You need to import Andrews public key, because Etch now comes with signed packages and apt-get could complain about unverified signatures of 3rd party software.

gpg --keyserver pgp.mit.edu --recv-keys 8F068012
gpg --armor --export 8F068012  | apt-key add -

Ok. Now add the repository to your apt sources.

echo 'deb debian.mcmillan.net.nz/debian etch awm' \
>> /etc/apt/sources.list

Update your list of packages and install the eAccelerator module.

aptitude update 
aptitude install php5-eaccelerator 

You need to activate the module in /etc/php5/{cgi|apache|apache2}/conf.d/eaccelerator.ini by uncommenting the last line:

extension=eaccelerator.so 

If you use PHP5 via FastCGI/fcgid and Apache2 worker, it is recommended to use eAccelerator as an extension of the Zend engine. Replace the above line with:

zend_extension="/usr/lib/php5/20060613+lfs/eaccelerator.so" 

Finally, reload Apache

/etc/init.d/apache2 force-reload 

Use phpinfo() to check if eAccelerator is activated. Note that php-cli is not supported by eAccelerator.

<?php phpinfo(); ?>

The forum is shipped with a build-in crawler and search engine, optimized for searching forum postings - a wise decision, since indexed_search is almost infamous for causing heavy load and timeouts in large scale environments.

A first look at the database tables reveals the use of indexes for some columns/tables. Indexes can speed up SELECT statements but are yet rarely implemented in any extension. Anyway, thanks to the efforts of core-dev Dmitry Dulepov, they are meanwhile common in core and sysext tables.

The extension is available via extension repository or directly via typo3.org.

Statistics for nonspam.cfc:
-----------------------------------------------
Database version:                    OSBF-Bayes
Total buckets in database:                94321
Buckets used (%):                           9.0
Bucket size (bytes):                         12
Header size (bytes):                       4092
Number of chains:                          7444
Max chain len (buckets):                      5
Average chain length (buckets):             1.1
Max bucket displacement:                      3
Buckets unreachable:                          0
Trainings:                                    9
Classifications:                             88
Learned mistakes:                             7
Extra learnings:                              0
Ham accuracy (%):                         92.47
-----------------------------------------------

Statistics for spam.cfc:
-----------------------------------------------
Database version:                    OSBF-Bayes
Total buckets in database:                94321
Buckets used (%):                          62.6
Bucket size (bytes):                         12
Header size (bytes):                       4092
Number of chains:                         16430
Max chain len (buckets):                     60
Average chain length (buckets):             3.6
Max bucket displacement:                     29
Buckets unreachable:                          0
Trainings:                                   42
Classifications:                            134
Learned mistakes:                             2
Extra learnings:                              1
Spam accuracy (%):                        98.45
-----------------------------------------------
Spam rate (%):                            58.11
Global accuracy (%):                      95.95
-----------------------------------------------

Now it's up to Micr0s0ft, to fix their IE bugs for proper presentation of this page ;-)