initial patch proposal

[git.ikiwiki.info.git] / doc / plugins / po.mdwn

diff --git a/doc/plugins/po.mdwn b/doc/plugins/po.mdwn
index 09df263944b7ee1682a715346b89b92846f84ed7..9273da399a3207543802a6c4ca48bcc1d5924928 100644 (file)
--- a/doc/plugins/po.mdwn
+++ b/doc/plugins/po.mdwn
@@ -5,6 +5,8 @@ This plugin adds support for multi-lingual wikis, translated with
 gettext, using [po4a](http://po4a.alioth.debian.org/).
 
 It depends on the Perl `Locale::Po4a::Po` library (`apt-get install po4a`).
 gettext, using [po4a](http://po4a.alioth.debian.org/).
 
 It depends on the Perl `Locale::Po4a::Po` library (`apt-get install po4a`).

+As detailed bellow in the security section, `po4a` is subject to
+denial-of-service attacks before version 0.35.

 
 [[!toc levels=2]]
 
 
 [[!toc levels=2]]
 


@@ -31,6 +33,12 @@ Example: `bla/page.fr.po` is the PO "message catalog" used to
 translate `bla/page.mdwn` into French; if `usedirs` is enabled, it is
 rendered as `bla/page/index.fr.html`, else as `bla/page.fr.html`
 
 translate `bla/page.mdwn` into French; if `usedirs` is enabled, it is
 rendered as `bla/page/index.fr.html`, else as `bla/page.fr.html`
 

+(In)Compatibility
+=================
+
+This plugin does not support the `indexpages` mode. If you don't know
+what it is, you probably don't care.
+

 
 Configuration
 =============
 
 Configuration
 =============


@@ -47,7 +55,7 @@ Supported languages
 languages, such as:
 
         po_slave_languages => { 'fr' => 'Français',
 languages, such as:
 
         po_slave_languages => { 'fr' => 'Français',

-                                'es' => 'Castellano',
+                                'es' => 'Español',

                                 'de' => 'Deutsch',
         }
 
                                 'de' => 'Deutsch',
         }
 


@@ -64,39 +72,40 @@ worry about excluding them explicitly from this [[ikiwiki/PageSpec]].
 Internal links
 --------------
 
 Internal links
 --------------
 

+### Links targets
+

 The `po_link_to` option in `ikiwiki.setup` is used to decide how
 internal links should be generated, depending on web server features
 and site-specific preferences.
 
 The `po_link_to` option in `ikiwiki.setup` is used to decide how
 internal links should be generated, depending on web server features
 and site-specific preferences.
 

-### Default linking behavior
+#### Default linking behavior

 
 If `po_link_to` is unset, or set to `default`, ikiwiki's default
 linking behavior is preserved: `\[[destpage]]` links to the master
 language's page.
 
 
 If `po_link_to` is unset, or set to `default`, ikiwiki's default
 linking behavior is preserved: `\[[destpage]]` links to the master
 language's page.
 

-### Link to current language
+#### Link to current language

 
 If `po_link_to` is set to `current`, `\[[destpage]]` links to the
 `destpage`'s version written in the current page's language, if
 available, *i.e.*:
 
 
 If `po_link_to` is set to `current`, `\[[destpage]]` links to the
 `destpage`'s version written in the current page's language, if
 available, *i.e.*:
 

-- `foo/destpage/index.LL.html` if `usedirs` is enabled
-- `foo/destpage.LL.html` if `usedirs` is disabled
+* `foo/destpage/index.LL.html` if `usedirs` is enabled
+* `foo/destpage.LL.html` if `usedirs` is disabled

 
 

-### Link to negotiated language
+#### Link to negotiated language

 
 If `po_link_to` is set to `negotiated`, `\[[page]]` links to the
 negotiated preferred language, *i.e.* `foo/page/`.
 
 (In)compatibility notes:
 
 
 If `po_link_to` is set to `negotiated`, `\[[page]]` links to the
 negotiated preferred language, *i.e.* `foo/page/`.
 
 (In)compatibility notes:
 

-- if `usedirs` is disabled, it does not make sense to set `po_link_to`
+* if `usedirs` is disabled, it does not make sense to set `po_link_to`

   to `negotiated`; this option combination is neither implemented
   nor allowed.
   to `negotiated`; this option combination is neither implemented
   nor allowed.

-- if the web server does not support Content Negotiation, setting
+* if the web server does not support Content Negotiation, setting

   `po_link_to` to `negotiated` will produce a unusable website.
 
   `po_link_to` to `negotiated` will produce a unusable website.
 

-

 Server support
 ==============
 
 Server support
 ==============
 


@@ -105,7 +114,8 @@ Apache
 
 Using Apache `mod_negotiation` makes it really easy to have Apache
 serve any page in the client's preferred language, if available.
 
 Using Apache `mod_negotiation` makes it really easy to have Apache
 serve any page in the client's preferred language, if available.

-This is the default Debian Apache configuration.
+
+Add 'Options MultiViews' to the wiki directory's configuration in Apache.

 
 When `usedirs` is enabled, one has to set `DirectoryIndex index` for
 the wiki context.
 
 When `usedirs` is enabled, one has to set `DirectoryIndex index` for
 the wiki context.


@@ -114,6 +124,8 @@ Setting `DefaultLanguage LL` (replace `LL` with your default MIME
 language code) for the wiki context can help to ensure
 `bla/page/index.en.html` is served as `Content-Language: LL`.
 
 language code) for the wiki context can help to ensure
 `bla/page/index.en.html` is served as `Content-Language: LL`.
 

+For details, see [Apache's documentation](http://httpd.apache.org/docs/2.2/content-negotiation.html).
+

 lighttpd
 --------
 
 lighttpd
 --------
 


@@ -129,6 +141,10 @@ Usage
 Templates
 ---------
 
 Templates
 ---------
 

+When `po_link_to` is not set to `negotiated`, one should replace some
+occurrences of `BASEURL` with `HOMEPAGEURL` to get correct links to
+the wiki homepage.
+

 The `ISTRANSLATION` and `ISTRANSLATABLE` variables can be used to
 display things only on translatable or translation pages.
 
 The `ISTRANSLATION` and `ISTRANSLATABLE` variables can be used to
 display things only on translatable or translation pages.
 


@@ -137,43 +153,21 @@ display things only on translatable or translation pages.
 The `OTHERLANGUAGES` loop provides ways to display other languages'
 versions of the same page, and the translations' status.
 
 The `OTHERLANGUAGES` loop provides ways to display other languages'
 versions of the same page, and the translations' status.
 

-One typically adds the following code to `templates/page.tmpl`:
-
-       <TMPL_IF NAME="OTHERLANGUAGES">
-       <div id="otherlanguages">
-         <ul>
-         <TMPL_LOOP NAME="OTHERLANGUAGES">
-           <li>
-             <a href="<TMPL_VAR NAME="URL">"><TMPL_VAR NAME="LANGUAGE"></a>
-             <TMPL_UNLESS NAME="MASTER">
-               (<TMPL_VAR NAME="PERCENT">&nbsp;%)
-             </TMPL_UNLESS>
-           </li>
-         </TMPL_LOOP>
-         </ul>
-       </div>
-       </TMPL_IF>
-
-The following variables are available inside the loop (for every page in):
-
-- `URL` - url to the page
-- `CODE` - two-letters language code
-- `LANGUAGE` - language name (as defined in `po_slave_languages`)
-- `MASTER` - is true (1) if, and only if the page is a "master" page
-- `PERCENT` - for "slave" pages, is set to the translation completeness, in percents
+An example of its use can be found in the default
+`templates/page.tmpl`. In case you want to customize it, the following
+variables are available inside the loop (for every page in):
+
+* `URL` - url to the page
+* `CODE` - two-letters language code
+* `LANGUAGE` - language name (as defined in `po_slave_languages`)
+* `MASTER` - is true (1) if, and only if the page is a "master" page
+* `PERCENT` - for "slave" pages, is set to the translation completeness, in percents

 
 ### Display the current translation status
 
 The `PERCENTTRANSLATED` variable is set to the translation
 
 ### Display the current translation status
 
 The `PERCENTTRANSLATED` variable is set to the translation

-completeness, expressed in percent, on "slave" pages.
-
-One can use it this way:
-
-       <TMPL_IF NAME="ISTRANSLATION">
-       <div id="percenttranslated">
-         <TMPL_VAR NAME="PERCENTTRANSLATED">
-       </div>
-       </TMPL_IF>
+completeness, expressed in percent, on "slave" pages. It is used by
+the default `templates/page.tmpl`.

 
 Additional PageSpec tests
 -------------------------
 
 Additional PageSpec tests
 -------------------------


@@ -194,14 +188,18 @@ Also, when the plugin has just been enabled, or when a page has just
 been declared as being translatable, the needed POT and PO files are
 created, and the PO files are checked into version control.
 
 been declared as being translatable, the needed POT and PO files are
 created, and the PO files are checked into version control.
 

-Discussion pages
-----------------
+Discussion pages and other sub-pages
+------------------------------------

 
 Discussion should happen in the language in which the pages are
 written for real, *i.e.* the "master" one. If discussion pages are
 enabled, "slave" pages therefore link to the "master" page's
 discussion page.
 
 
 Discussion should happen in the language in which the pages are
 written for real, *i.e.* the "master" one. If discussion pages are
 enabled, "slave" pages therefore link to the "master" page's
 discussion page.
 

+Likewise, "slave" pages are not supposed to have sub-pages;
+[[WikiLinks|wikilink]] that appear on a "slave" page therefore link to
+the master page's sub-pages.
+

 Translating
 -----------
 
 Translating
 -----------
 


@@ -211,196 +209,167 @@ interface could also be implemented at some point).
 If [[tips/untrusted_git_push]] is setup, one can edit the PO files in one's
 preferred `$EDITOR`, without needing to be online.
 
 If [[tips/untrusted_git_push]] is setup, one can edit the PO files in one's
 preferred `$EDITOR`, without needing to be online.
 

-TODO
-====
-
-Security checks
----------------
-
-### Security history
-
-The only past security issues I could find in GNU gettext and po4a
-are:
-
-- [CVE-2004-0966](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2004-0966),
-  *i.e.* [Debian bug #278283](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=278283):
-  the autopoint and gettextize scripts in the GNU gettext package
-  1.14 and later versions, as used in Trustix Secure Linux 1.5
-  through 2.1 and other operating systems, allows local users to
-  overwrite files via a symlink attack on temporary files.
-- [CVE-2007-4462](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4462):
-  `lib/Locale/Po4a/Po.pm` in po4a before 0.32 allows local users to
-  overwrite arbitrary files via a symlink attack on the
-  gettextization.failed.po temporary file.
-
-**FIXME**: check whether this plugin would have been a possible attack
-vector to exploit these vulnerabilities.
+Markup languages support
+------------------------

 
 

-Depending on my mood, the lack of found security issues can either
-indicate that there are none, or reveal that no-one ever bothered to
-find (and publish) them.
+[[Markdown|mdwn]] is well supported. Some other markup languages supported
+by ikiwiki mostly work, but some pieces of syntax are not rendered
+correctly on the slave pages:

 
 

-### PO file features
+* [[reStructuredText|rst]]: anonymous hyperlinks and internal
+  cross-references
+* [[wikitext]]: conversion of newlines to paragraphs
+* [[creole]]: verbatim text is wrapped, tables are broken
+* [[html]] and LaTeX: not supported yet; the dedicated po4a modules
+  could be used to support them, but they would need a security audit
+* other markup languages have not been tested.

 
 

-Can any sort of directives be put in po files that will cause mischief
-(ie, include other files, run commands, crash gettext, whatever)?
+Security
+========

 
 

-> No [documented](http://www.gnu.org/software/gettext/manual/gettext.html#PO-Files)
-> directive is supposed to do so. [[--intrigeri]]
+[[po/discussion]] contains a detailed security analysis of this plugin
+and its dependencies.

 
 

-### Running po4a on untrusted content
+When using po4a older than 0.35, it is recommended to uninstall
+`Text::WrapI18N` (Debian package `libtext-wrapi18n-perl`), in order to
+avoid a potential denial of service.

 
 

-Are there any security issues on running po4a on untrusted content?
+TODO
+====

 
 

-To say the least, this issue is not well covered, at least publicly:
+Better links
+------------

 
 

-- the documentation does not talk about it;
-- grep'ing the source code for `security` or `trust` gives no answer.
+Once the fix to
+[[bugs/pagetitle_function_does_not_respect_meta_titles]] from
+[[intrigeri]]'s `meta` branch is merged into ikiwiki upstream, the
+generated links' text will be optionally based on the page titles set
+with the [[meta|plugins/meta]] plugin, and will thus be translatable.
+It will also allow displaying the translation status in links to slave
+pages. Both were implemented, and reverted in commit
+ea753782b222bf4ba2fb4683b6363afdd9055b64, which should be reverted
+once [[intrigeri]]'s `meta` branch is merged.

 
 

-On the other hand, a po4a developer answered my questions in
-a convincing manner, stating that processing untrusted content was not
-an initial goal, and analysing in detail the possible issues.
+An integration branch, called `meta-po`, merges [[intrigeri]]'s `po`
+and `meta` branches, and thus has this additional features.

 
 

-#### Already checked
+Self links
+----------

 
 

-- the core (`Po.pm`, `Transtractor.pm`) should be safe
-- po4a source code was fully checked for other potential symlink
-  attacks, after discovery of one such issue
-- the only external program run by the core is `diff`, in `Po.pm` (in
-  parts of its code we don't use)
-- `Locale::gettext`: only used to display translated error messages
-- Nicolas François "hopes" `DynaLoader` is safe, and has "no reason to
-  think that `Encode` is not safe"
-- Nicolas François has "no reason to think that `Encode::Guess` is not
-  safe". The po plugin nevertheless avoids using it by defining the
-  input charset (`file_in_charset`) before asking `Transtractor` to
-  read any file. NB: this hack depends on po4a internals to stay
-  the same.
+If a page contains a WikiLink to itself, ikiwiki does not normally
+turn that into a hyperlink. However, if a translated page contains a
+WikiLink to itself, a hyperlink is inserted, at least with the default
+`po_link_to` the link points to the English version of the page. Is there a
+good reason for that to be done? --[[Joey]]

 
 

-#### To be checked
+> The commit 0113c69d4fb in my po branch might fix this. --[[intrigeri]]

 
 

-##### Locale::Po4a modules
+Language display order
+----------------------

 
 

-The modules we want to use have to be checked, as not all are safe
-(e.g. the LaTeX module's behaviour is changed by commands included in
-the content); they may use regexps generated from the content.
+Jonas pointed out that one might want to control the order that links to
+other languages are listed, for various reasons. Currently, there is no
+order, as `po_slave_languages` is a hash. It would need to be converted
+to an array to support this. (If twere done, twere best done quickly.)
+--[[Joey]] 

 
 

-`Chooser.pm` only loads the plugin we tell it too: currently, this
-means the `Text` module only.
+Duplicate %links ?
+------------------

 
 

-`Text` module (I checked the CVS version):
+I notice code in the scan hook that seems to assume
+that %links will accumulate duplicate links for a page.
+That used to be so, but the bug was fixed. Does this mean
+that po might be replacing the only link on a page, in error? 
+--[[Joey]] 

 
 

-- it does not run any external program
-- only `do_paragraph()` builds regexp's that expand untrusted
-  variables; they seem safe to me, but someone more expert than me
-  will need to check. Joey?
+> It would replace it. The only problematic case is when another
+> plugin has its own reasons, in its `scan` hook, to add a page
+> that is already there to `$links{$page}`. This other plugin's
+> effect might then be changed by po's `scan` hook... which could
+> be either good (better overall l10n) or bad (break the other
+> plugin's goal). --[[intrigeri]]

 
 

-##### Text::WrapI18N
+Name of toplevel index page
+---------------------------

 
 

-`Text::WrapI18N` can cause DoS (see the
-[Debian bug #470250](http://bugs.debian.org/470250)), but it is
-optional and we do not need the features it provides.
+Normally at the top index page of a wiki, you see the wiki name at
+the top. However, at the top *translated* index page, you see something
+like "index.da". --[[Joey]]

 
 

-It is loaded if available by `Locale::Po4a::Common`; looking at the
-code, I'm not sure we can prevent this at all, but maybe some symbol
-table manipulation tricks could work; overriding
-`Locale::Po4a::Common::wrapi18n` may be easier. I'm no expert at all
-in this field. Joey? [[--intrigeri]]
+> I suggest changing `Render.pm`, line 115, to replace the `$page eq 'index'`
+> test with a predicate call such as isindexpage($page). Such a predicate
+> function could then be overriden by the po plugin. --[[intrigeri]]

 
 

-> Update: Nicolas François suggests we add an option to po4a to
-> disable it. It would do the trick, but only for people running
-> a brand new po4a (probably too late for Lenny). Anyway, this option
-> would have to take effect in a `BEGIN` / `eval` that I'm not
-> familiar with. I can learn and do it, in case no Perl wizard
-> volunteers to provide the po4a patch. [[--intrigeri]]
+Pagespecs
+---------

 
 

-##### Term::ReadKey
+I was suprised that, when using the map directive, a pagespec of "*"
+listed all the translated pages as well as regular pages. That can 
+make a big difference to an existing wiki when po is turned on,
+and seems generally not wanted.
+(OTOH, you do want to match translated pages by
+default when locking pages.) --[[Joey]]

 
 

-`Term::ReadKey` is not a hard dependency in our case, *i.e.* po4a
-works nicely without it. But the po4a Debian package recommends
-`libterm-readkey-perl`, so it will probably be installed on most
-systems using the po plugin.
+Edit links on untranslated pages
+--------------------------------

 
 

-If `$ENV{COLUMNS}` is not set, `Locale::Po4a::Common` uses
-`Term::ReadKey::GetTerminalSize()` to get the terminal size. How safe
-is this?
+If a page is not translated yet, the "translated" version of it
+displays wikilinks to other, existing (but not yet translated?)
+pages as edit links, as if those pages do not exist. 

 
 

-Part of `Term::ReadKey` is written in C. Depending on the runtime
-platform, this function use ioctl, environment, or C library function
-calls, and may end up running the `resize` command (without
-arguments).
+That's really confusing, especially as clicking such a link
+brings up an edit form to create a new, english page.

 
 

-IMHO, using Term::ReadKey has too far reaching implications for us to
-be able to guarantee anything wrt. security. Since it is anyway of no
-use in our case, I suggest we define `ENV{COLUMNS}` before loading
-`Locale::Po4a::Common`, just to be on the safe side. Joey?
-[[--intrigeri]]
+This is with po_link_to=current or negotiated. With default, it doesn't
+happen.. 

 
 

-> Update: adding an option to disable `Text::WrapI18N`, as Nicolas
-> François suggested, would as a bonus disable `Term::ReadKey`
-> as well. [[--intrigeri]]
+Also, this may only happen if the page being linked to is coming from an
+underlay, and the underlays lack translation to a given language.
+--[[Joey]] 

 
 

-### msgmerge
+recentchanges links to po files
+-------------------------------

 
 

-`refreshpofiles()` runs this external program. A po4a developer
-answered he does "not expect any security issues from it".
+When a po file is changed, the recentchanges page shows a link such as
+"sandbox.es". But, clicking on it goes to the English (or negotiated
+language) version of the page. It would be better in this one case if
+the link went direct to the translated version of the page. --[[Joey]] 

 
 

-### Fuzzing input
+> The (untested) commit 496e8523c670 in my po branch might fix this...
+> or at least, I believe, put someone on the right track.
+> --[[intrigeri]]

 
 

-I was not able to find any public information about gettext or po4a
-having been tested with a fuzzing program, such as `zzuf` or `fusil`.
-Moreover, some gettext parsers seem to be quite
-[easy to crash](http://fusil.hachoir.org/trac/browser/trunk/fuzzers/fusil-gettext),
-so it might be useful to bang msgmerge/po4a's heads against such
-a program in order to easily detect some of the most obvious DoS.
-[[--intrigeri]]
-
-> po4a was not fuzzy-tested, but according to one of its developers,
-> "it would be really appreciated". [[--intrigeri]]
-
-gettext/po4a rough corners
+Double commits of po files

 --------------------------
 
 --------------------------
 

-- fix infinite loop when synchronizing two ikiwiki (when checkouts
-  live in different directories): say bla.fr.po has been updated in
-  repo2; pulling repo2 from repo1 seems to trigger a PO update, that
-  changes bla.fr.po in repo1; then pushing repo1 to repo2 triggers
-  a PO update, that changes bla.fr.po in repo2; etc.; quickly fixed in
-  `629968fc89bced6727981c0a1138072631751fee`, by disabling references
-  in Pot files. Using `Locale::Po4a::write_if_needed` might be
-  a cleaner solution. (warning: this function runs the external
-  `diff` program, have to check security)
-- new translations created in the web interface must get proper
-  charset/encoding gettext metadata, else the next automatic PO update
-  removes any non-ascii chars; possible solution: put such metadata
-  into the Pot file, and let it propagate; should be fixed in
-  `773de05a7a1ee68d2bed173367cf5e716884945a`, time will tell.
-
-Misc. improvements
-------------------
+When adding a new english page, the po files are created, committed,
+and then committed again. The second commit makes this change:
+
+       -"Content-Type: text/plain; charset=utf-8\n"
+       -"Content-Transfer-Encoding: ENCODING"
+       +"Content-Type: text/plain; charset=UTF-8\n"
+       +"Content-Transfer-Encoding: ENCODING\n"

 
 

-### page titles
+Same thing happens when a change to an existing page triggers a po file
+update. --[[Joey]] 

 
 

-Use nice page titles from meta plugin in links, as inline already
-does. This is actually a duplicate for
-[[bugs/pagetitle_function_does_not_respect_meta_titles]], which might
-be fixed by something like [[todo/using_meta_titles_for_parentlinks]].
+Ugly messages with empty files
+------------------------------

 
 

-### source files format
+If there are empty .mdwn files, the po plugin displays some ugly messages.

 
 

-Markdown is supported, great, but what about others? The set of file
-formats supported both in ikiwiki and po4a probably is greater than
-`{markdown}`. Warning: the po4a modules are the place where one can
-expect security issues.
+Translation of directives
+-------------------------

 
 

-Translation quality assurance
------------------------------
+If a translated page contains a directive, it may expand to some english
+text, or text in whatever single language ikiwiki is configured to "speak".

 
 

-Modifying a PO file via the CGI must be forbidden if the new version
-is not a valid PO file. As a bonus, check that it provides a more
-complete translation than the existing one.
+Maybe there could be a way to switch ikiwiki to speaking another language
+when building a non-english page? Then the directives would get translated.

 
 

-A new `cansave` type of hook would be needed to implement this.
+Documentation
+-------------

 
 

-Note: committing to the underlying repository is a way to bypass
-this check.
+Maybe write separate documentation depending on the people it targets:
+translators, wiki administrators, hackers. This plugin may be complex
+enough to deserve this.