Templates
---------
+When `po_link_to` is not set to `negotiated`, one should replace some
+occurrences of `BASEURL` with `HOMEPAGEURL` to get correct links to
+the wiki homepage.
+
The `ISTRANSLATION` and `ISTRANSLATABLE` variables can be used to
display things only on translatable or translation pages.
been declared as being translatable, the needed POT and PO files are
created, and the PO files are checked into version control.
-Discussion pages
-----------------
+Discussion pages and other sub-pages
+------------------------------------
Discussion should happen in the language in which the pages are
written for real, *i.e.* the "master" one. If discussion pages are
enabled, "slave" pages therefore link to the "master" page's
discussion page.
+Likewise, "slave" pages are not supposed to have sub-pages;
+[[WikiLinks|wikilink]] that appear on a "slave" page therefore link to
+the master page's sub-pages.
+
Translating
-----------
If [[tips/untrusted_git_push]] is setup, one can edit the PO files in one's
preferred `$EDITOR`, without needing to be online.
+Markup languages support
+------------------------
+
+Markdown is well supported. Some other markup languages supported by
+ikiwiki mostly work, but some pieces of syntax are not rendered
+correctly on the slave pages:
+
+* [[reStructuredText|rst]]: anonymous hyperlinks and internal
+ cross-references
+* [[wikitext]]: conversion of newlines to paragraphs
+* [[creole]]: verbatim text is wrapped, tables are broken
+* [[html]] and LaTeX: not supported yet; the dedicated po4a modules
+ could be used to support them, but they would need a security audit
+* other markup languages have not been tested.
+
+
TODO
====
##### Locale::Po4a modules
-- the modules we want to use have to be checked, as not all are safe
- (e.g. the LaTeX module's behaviour is changed by commands included
- in the content); they may use regexps generated from the content; we
- currently only use the `Text` module
-- the `Text` module does not run any external program
-- check that no module is loaded by `Chooser.pm`, when we tell it to
- load the `Text` one
-- `nsgmls` is used by `Sgml.pm`
+The modules we want to use have to be checked, as not all are safe
+(e.g. the LaTeX module's behaviour is changed by commands included in
+the content); they may use regexps generated from the content.
+
+`Chooser.pm` only loads the plugin we tell it too: currently, this
+means the `Text` module only.
+
+`Text` module (I checked the CVS version):
+
+- it does not run any external program
+- only `do_paragraph()` builds regexp's that expand untrusted
+ variables; they seem safe to me, but someone more expert than me
+ will need to check. Joey?
+
+ > Freaky code, but seems ok due to use of `quotementa`.
##### Text::WrapI18N
`Locale::Po4a::Common::wrapi18n` may be easier. I'm no expert at all
in this field. Joey? [[--intrigeri]]
+> Update: Nicolas François suggests we add an option to po4a to
+> disable it. It would do the trick, but only for people running
+> a brand new po4a (probably too late for Lenny). Anyway, this option
+> would have to take effect in a `BEGIN` / `eval` that I'm not
+> familiar with. I can learn and do it, in case no Perl wizard
+> volunteers to provide the po4a patch. [[--intrigeri]]
+
+>> That doesn't really need to be in a BEGIN. This patch moves it to
+>> `import`, and makes this disable wrap18n:
+>> `use Locale::Po4a::Common q{nowrapi18n}` --[[Joey]]
+
+<pre>
+--- /usr/share/perl5/Locale/Po4a/Common.pm 2008-07-21 14:54:52.000000000 -0400
++++ Common.pm 2008-11-11 18:27:34.000000000 -0500
+@@ -30,8 +30,16 @@
+ use strict;
+ use warnings;
+
+-BEGIN {
+- if (eval { require Text::WrapI18N }) {
++sub import {
++ my $class=shift;
++ my $wrapi18n=1;
++ if ($_[0] eq 'nowrapi18n') {
++ shift;
++ $wrapi18n=0;
++ }
++ $class->export_to_level(1, $class, @_);
++
++ if ($wrapi18n && eval { require Text::WrapI18N }) {
+
+ # Don't bother determining the wrap column if we cannot wrap.
+ my $col=$ENV{COLUMNS};
+</pre>
+
##### Term::ReadKey
`Term::ReadKey` is not a hard dependency in our case, *i.e.* po4a
`Locale::Po4a::Common`, just to be on the safe side. Joey?
[[--intrigeri]]
+> Update: adding an option to disable `Text::WrapI18N`, as Nicolas
+> François suggested, would as a bonus disable `Term::ReadKey`
+> as well. [[--intrigeri]]
+
### msgmerge
`refreshpofiles()` runs this external program. A po4a developer
answered he does "not expect any security issues from it".
+### msgfmt
+
+`isvalidpo()` runs this external program. Its security should be checked.
+
### Fuzzing input
I was not able to find any public information about gettext or po4a
> po4a was not fuzzy-tested, but according to one of its developers,
> "it would be really appreciated". [[--intrigeri]]
+Test conditions:
+
+- a 21M file containing 100 concatenated copies of all the files in my
+ `/usr/share/common-licenses/`; I had no existing PO file or
+ translated versions at hand, which renders these tests
+ quite incomplete.
+- po4a was the Debian 0.34-2 package; the same tests were also run
+ after replacing the `Text` module with the CVS one (the core was not
+ changed in CVS since 0.34-2 was released), without any significant
+ difference in the results.
+- Perl 5.10.0-16
+
+#### po4a-gettextize
+
+`po4a-gettextize` uses more or less the same po4a features as our
+`refreshpot` function.
+
+Without specifying an input charset, zzuf'ed `po4a-gettextize` quickly
+errors out, complaining it was not able to detect the input charset;
+it leaves no incomplete file on disk.
+
+So I had to pretend the input was in UTF-8, as does the po plugin.
+
+Two ways of crashing were revealed by this command-line:
+
+ zzuf -vc -s 0:100 -r 0.1:0.5 \
+ po4a-gettextize -f text -o markdown -M utf-8 -L utf-8 \
+ -m LICENSES >/dev/null
+
+They are:
+
+ Malformed UTF-8 character (UTF-16 surrogate 0xdcc9) in substitution iterator at /usr/share/perl5/Locale/Po4a/Po.pm line 1443.
+ Malformed UTF-8 character (fatal) at /usr/share/perl5/Locale/Po4a/Po.pm line 1443.
+
+and
+
+ Malformed UTF-8 character (UTF-16 surrogate 0xdcec) in substitution (s///) at /usr/share/perl5/Locale/Po4a/Po.pm line 1443.
+ Malformed UTF-8 character (fatal) at /usr/share/perl5/Locale/Po4a/Po.pm line 1443.
+
+Perl seems to exit cleanly, and an incomplete PO file is written on
+disk. I not sure whether if this is a bug in Perl or in `Po.pm`.
+
+> It's fairly standard perl behavior when fed malformed utf-8. As long as it doesn't
+> crash ikiwiki, it's probably acceptable. Ikiwiki can do some similar things itself when fed malformed utf-8 (doesn't crash tho) --[[Joey]]
+
+#### po4a-translate
+
+`po4a-translate` uses more or less the same po4a features as our
+`filter` function.
+
+Without specifying an input charset, same behaviour as
+`po4a-gettextize`, so let's specify UTF-8 as input charset as of now.
+
+ zzuf -cv \
+ po4a-translate -d -f text -o markdown -M utf-8 -L utf-8 \
+ -k 0 -m LICENSES -p LICENSES.fr.po -l test.fr
+
+... prints tons of occurences of the following error, but a complete
+translated document is written (obviously with some weird chars
+inside):
+
+ Use of uninitialized value in string ne at /usr/share/perl5/Locale/Po4a/TransTractor.pm line 854.
+ Use of uninitialized value in string ne at /usr/share/perl5/Locale/Po4a/TransTractor.pm line 840.
+ Use of uninitialized value in pattern match (m//) at /usr/share/perl5/Locale/Po4a/Po.pm line 1002.
+
+While:
+
+ zzuf -cv -s 0:10 -r 0.001:0.3 \
+ po4a-translate -d -f text -o markdown -M utf-8 -L utf-8 \
+ -k 0 -m LICENSES -p LICENSES.fr.po -l test.fr
+
+... seems to lose the fight, at the `readpo(LICENSES.fr.po)` step,
+against some kind of infinite loop, deadlock, or any similar beast.
+It does not seem to eat memory, though.
+
+Whatever format module is used does not change anything. This is thus
+probably a bug in po4a's core or in a lib it depends on.
+
+The sub `read`, in `TransTractor.pm`, seems to be a good debugging
+starting point.
+
+#### msgmerge
+
+`msgmerge` is run in our `refreshpofiles` function. I did not manage
+to crash it with `zzuf`.
+
gettext/po4a rough corners
--------------------------
into the Pot file, and let it propagate; should be fixed in
`773de05a7a1ee68d2bed173367cf5e716884945a`, time will tell.
-Misc. improvements
-------------------
+Better links
+------------
+
+### Page title in links
+
+Using the fix to
+[[bugs/pagetitle_function_does_not_respect_meta_titles]] from
+[[intrigeri]]'s `meta` branch, the generated links' text is based on
+the page titles set with the [[meta|plugins/meta]] plugin. This has to
+be merged upstream, though.
+
+Robustness tests
+----------------
+
+### Enabling/disabling the plugin
+
+- enabling the plugin with `po_translatable_pages` set
+- enabling the plugin without `po_translatable_pages` set: **OK**
+- disabling the plugin: **OK**
-### page titles
+### Changing the plugin config
-Use nice page titles from meta plugin in links, as inline already
-does. This is actually a duplicate for
-[[bugs/pagetitle_function_does_not_respect_meta_titles]], which might
-be fixed by something like [[todo/using_meta_titles_for_parentlinks]].
+- adding existing pages to `po_translatable_pages`: **OK**
+- removing existing pages from `po_translatable_pages`: **OK**
+- adding a language to `po_slave_languages`: **OK**
+- removing a language from `po_slave_languages`: **OK**
+- changing `po_master_language`: **OK**
+- replacing `po_master_language` with a language previously part of
+ `po_slave_languages`: needs two rebuilds, but **OK** (this is quite
+ a perverse test actually)
-### source files format
+### Creating/deleting/renaming pages
-Markdown is supported, great, but what about others? The set of file
-formats supported both in ikiwiki and po4a probably is greater than
-`{markdown}`. Warning: the po4a modules are the place where one can
-expect security issues.
+All cases of master/slave page creation/deletion/rename, both via RCS
+and via CGI, have been tested.
-Translation quality assurance
------------------------------
+### Misc
-Modifying a PO file via the CGI must be forbidden if the new version
-is not a valid PO file. As a bonus, check that it provides a more
-complete translation than the existing one.
+- general test with `usedirs` disabled: **OK**
+- general test with `indexpages` enabled
+- general test with `po_link_to=default`
-A new `cansave` type of hook would be needed to implement this.
+Documentation
+-------------
-Note: committing to the underlying repository is a way to bypass
-this check.
+Maybe write separate documentation depending on the people it targets:
+translators, wiki administrators, hackers. This plugin may be complex
+enough to deserve this.