po(change): re-render all rendered files in one corner case

[git.ikiwiki.info.git] / doc / plugins / po.mdwn

diff --git a/doc/plugins/po.mdwn b/doc/plugins/po.mdwn
index 7d8da987e117c538a9f5120d8db6459242cc6bb3..92fba349fda19b8d29ac6fdab85a7538581e4754 100644 (file)
--- a/doc/plugins/po.mdwn
+++ b/doc/plugins/po.mdwn
@@ -31,6 +31,12 @@ Example: `bla/page.fr.po` is the PO "message catalog" used to
 translate `bla/page.mdwn` into French; if `usedirs` is enabled, it is
 rendered as `bla/page/index.fr.html`, else as `bla/page.fr.html`
 
 translate `bla/page.mdwn` into French; if `usedirs` is enabled, it is
 rendered as `bla/page/index.fr.html`, else as `bla/page.fr.html`
 

+(In)Compatibility
+=================
+
+This plugin does not support the `indexpages` mode. If you don't know
+what it is, you probably don't care.
+

 
 Configuration
 =============
 
 Configuration
 =============


@@ -219,6 +225,22 @@ interface could also be implemented at some point).
 If [[tips/untrusted_git_push]] is setup, one can edit the PO files in one's
 preferred `$EDITOR`, without needing to be online.
 
 If [[tips/untrusted_git_push]] is setup, one can edit the PO files in one's
 preferred `$EDITOR`, without needing to be online.
 

+Markup languages support
+------------------------
+
+Markdown is well supported. Some other markup languages supported by
+ikiwiki mostly work, but some pieces of syntax are not rendered
+correctly on the slave pages:
+
+* [[reStructuredText|rst]]: anonymous hyperlinks and internal
+  cross-references
+* [[wikitext]]: conversion of newlines to paragraphs
+* [[creole]]: verbatim text is wrapped, tables are broken
+* [[html]] and LaTeX: not supported yet; the dedicated po4a modules
+  could be used to support them, but they would need a security audit
+* other markup languages have not been tested.
+
+

 TODO
 ====
 
 TODO
 ====
 


@@ -285,8 +307,6 @@ an initial goal, and analysing in detail the possible issues.
   read any file. NB: this hack depends on po4a internals to stay
   the same.
 
   read any file. NB: this hack depends on po4a internals to stay
   the same.
 

-#### To be checked
-

 ##### Locale::Po4a modules
 
 The modules we want to use have to be checked, as not all are safe
 ##### Locale::Po4a modules
 
 The modules we want to use have to be checked, as not all are safe


@@ -305,52 +325,19 @@ means the `Text` module only.
 
   > Freaky code, but seems ok due to use of `quotementa`.
 
 
   > Freaky code, but seems ok due to use of `quotementa`.
 

+#### To be checked
+

 ##### Text::WrapI18N
 
 `Text::WrapI18N` can cause DoS (see the
 [Debian bug #470250](http://bugs.debian.org/470250)), but it is
 optional and we do not need the features it provides.
 
 ##### Text::WrapI18N
 
 `Text::WrapI18N` can cause DoS (see the
 [Debian bug #470250](http://bugs.debian.org/470250)), but it is
 optional and we do not need the features it provides.
 

-It is loaded if available by `Locale::Po4a::Common`; looking at the
-code, I'm not sure we can prevent this at all, but maybe some symbol
-table manipulation tricks could work; overriding
-`Locale::Po4a::Common::wrapi18n` may be easier. I'm no expert at all
-in this field. Joey? [[--intrigeri]]
-
-> Update: Nicolas François suggests we add an option to po4a to
-> disable it. It would do the trick, but only for people running
-> a brand new po4a (probably too late for Lenny). Anyway, this option
-> would have to take effect in a `BEGIN` / `eval` that I'm not
-> familiar with. I can learn and do it, in case no Perl wizard
-> volunteers to provide the po4a patch. [[--intrigeri]]
-
->> That doesn't really need to be in a BEGIN. This patch moves it to
->> `import`, and makes this disable wrap18n:
->> `use Locale::Po4a::Common q{nowrapi18n}` --[[Joey]]
-
-<pre>
---- /usr/share/perl5/Locale/Po4a/Common.pm     2008-07-21 14:54:52.000000000 -0400
-+++ Common.pm  2008-11-11 18:27:34.000000000 -0500
-@@ -30,8 +30,16 @@
- use strict;
- use warnings;
- 
--BEGIN {
--    if (eval { require Text::WrapI18N }) {
-+sub import {
-+    my $class=shift;
-+    my $wrapi18n=1;
-+    if ($_[0] eq 'nowrapi18n') {
-+      shift;
-+      $wrapi18n=0;
-+    }
-+    $class->export_to_level(1, $class, @_);
-+
-+    if ($wrapi18n && eval { require Text::WrapI18N }) {
-     
-         # Don't bother determining the wrap column if we cannot wrap.
-         my $col=$ENV{COLUMNS};
-</pre>
+> I proposed a patch based on Joey's to po4a-devel, allowing to fully
+> disable this module's use. When it is merged upstream, we'll need to add
+> `use Locale::Po4a::Common qw(nowrapi18n)` to `po.pm`, before loading
+> any other `Locale::Po4a` module. A versioned dependency may be needed.
+> --[[intrigeri]]

 
 ##### Term::ReadKey
 
 
 ##### Term::ReadKey
 


@@ -359,42 +346,29 @@ works nicely without it. But the po4a Debian package recommends
 `libterm-readkey-perl`, so it will probably be installed on most
 systems using the po plugin.
 
 `libterm-readkey-perl`, so it will probably be installed on most
 systems using the po plugin.
 

-If `$ENV{COLUMNS}` is not set, `Locale::Po4a::Common` uses
-`Term::ReadKey::GetTerminalSize()` to get the terminal size. How safe
-is this?
+`Term::ReadKey` has too far reaching implications for us to
+be able to guarantee anything wrt. security.

 
 

-Part of `Term::ReadKey` is written in C. Depending on the runtime
-platform, this function use ioctl, environment, or C library function
-calls, and may end up running the `resize` command (without
-arguments).
+> The option that disables `Text::WrapI18N` also disables
+> `Term::ReadKey` as a consequence. [[--intrigeri]]

 
 

-IMHO, using Term::ReadKey has too far reaching implications for us to
-be able to guarantee anything wrt. security. Since it is anyway of no
-use in our case, I suggest we define `ENV{COLUMNS}` before loading
-`Locale::Po4a::Common`, just to be on the safe side. Joey?
-[[--intrigeri]]
+### msgmerge

 
 

-> Update: adding an option to disable `Text::WrapI18N`, as Nicolas
-> François suggested, would as a bonus disable `Term::ReadKey`
-> as well. [[--intrigeri]]
+`refreshpofiles()` runs this external program.

 
 

-### msgmerge
+A po4a developer answered he does "not expect any security issues from
+it". I did not manage to crash it with `zzuf`, nor was able to find
+any past security holes.

 
 

-`refreshpofiles()` runs this external program. A po4a developer
-answered he does "not expect any security issues from it".
+### msgfmt

 
 

-### Fuzzing input
+`isvalidpo()` runs this external program.

 
 

-I was not able to find any public information about gettext or po4a
-having been tested with a fuzzing program, such as `zzuf` or `fusil`.
-Moreover, some gettext parsers seem to be quite
-[easy to crash](http://fusil.hachoir.org/trac/browser/trunk/fuzzers/fusil-gettext),
-so it might be useful to bang msgmerge/po4a's heads against such
-a program in order to easily detect some of the most obvious DoS.
-[[--intrigeri]]
+* I could not manage to make it behave badly using zzuf, it exits
+  cleanly when too many errors are detected.
+* I could not find any past security holes.

 
 

-> po4a was not fuzzy-tested, but according to one of its developers,
-> "it would be really appreciated". [[--intrigeri]]
+### Fuzzing input

 
 Test conditions:
 
 
 Test conditions:
 


@@ -469,18 +443,9 @@ While:
 
 ... seems to lose the fight, at the `readpo(LICENSES.fr.po)` step,
 against some kind of infinite loop, deadlock, or any similar beast.
 
 ... seems to lose the fight, at the `readpo(LICENSES.fr.po)` step,
 against some kind of infinite loop, deadlock, or any similar beast.

-It does not seem to eat memory, though.

 
 

-Whatever format module is used does not change anything. This is thus
-probably a bug in po4a's core or in a lib it depends on.
-
-The sub `read`, in `TransTractor.pm`, seems to be a good debugging
-starting point.
-
-#### msgmerge
-
-`msgmerge` is run in our `refreshpofiles` function. I did not manage
-to crash it with `zzuf`.
+The root of this bug lies in `Text::WrapI18N`, see above for
+possible solutions.

 
 gettext/po4a rough corners
 --------------------------
 
 gettext/po4a rough corners
 --------------------------


@@ -507,57 +472,17 @@ Better links
 
 Using the fix to
 [[bugs/pagetitle_function_does_not_respect_meta_titles]] from
 
 Using the fix to
 [[bugs/pagetitle_function_does_not_respect_meta_titles]] from

-intrigeri's `meta` branch, the generated links' text is based on the
-page titles set with the [[meta|plugins/meta]] plugin. This has to be
-merged upstream, though.
-
-Page formats
-------------
-
-Markdown is well supported, great, but what about others?
-
-The [[po|plugins/po]] uses `Locale::Po4a::Text` for every page format;
-this can be expected to work out of the box with most other wiki-like
-formats supported by ikiwiki. Some of their ad-hoc syntax might be
-parsed in a strange way, but the worst problems I can imagine would be
-wrapping issues; e.g. there is code in po4a dedicated to prevent
-re-wrapping the underlined Markdown headers.
-
-While it would be easy to better support formats such as [[html]] or
-LaTeX, by using for each one the dedicated po4a module, this can be
-problematic from a security point of view.
-
-**TODO**: test the more popular formats and write proper documentation
-about it.
-
-Translation quality assurance
------------------------------
-
-Modifying a PO file via the CGI must be forbidden if the new version
-is not a valid PO file. As a bonus, check that it provides a more
-complete translation than the existing one.
-
-A new `cansave` type of hook would be needed to implement this.
-
-Note: committing to the underlying repository is a way to bypass
-this check.
-
-Creating new pages on the web
------------------------------
-
-See [[contrib/po|contrib/po]].
-
-Renaming pages
---------------
-
-- Renaming a translation should be forbidden.
+[[intrigeri]]'s `meta` branch, the generated links' text is based on
+the page titles set with the [[meta|plugins/meta]] plugin. This has to
+be merged into ikiwiki upstream, though.

 
 Robustness tests
 ----------------
 
 
 Robustness tests
 ----------------
 

-### Disabling the plugin
+### Enabling/disabling the plugin

 
 

-- enabling the plugin with `po_translatable_pages` set
+- enabling the plugin with `po_translatable_pages` set to blacklist: **OK**
+- enabling the plugin with `po_translatable_pages` set to whitelist: **OK**

 - enabling the plugin without `po_translatable_pages` set: **OK**
 - disabling the plugin: **OK**
 
 - enabling the plugin without `po_translatable_pages` set: **OK**
 - disabling the plugin: **OK**
 


@@ -572,32 +497,20 @@ Robustness tests
   `po_slave_languages`: needs two rebuilds, but **OK** (this is quite
   a perverse test actually)
 
   `po_slave_languages`: needs two rebuilds, but **OK** (this is quite
   a perverse test actually)
 

-### Creating pages
-
-- creating a master page via RCS: **OK**
-- creating a master page via CGI: **OK**
+### Creating/deleting/renaming pages

 
 

-### Deleting pages
-
-- removing a master page via RCS: **OK**
-- removing a translation via RCS: **OK**
-- removing a master page via CGI: **OK**
-- removing a translation via CGI: **OK**
-
-### Renaming pages
-
-- renaming a master page via RCS: **OK** (but the old translations
-  are lost, because not all RCS track file renaming)
-- renaming a master page and its translations via RCS: **OK**
-- renaming a master page via CGI: **OK**
-- renaming a translation via RCS
-- renaming a translation via CGI
+All cases of master/slave page creation/deletion/rename, both via RCS
+and via CGI, have been tested.

 
 ### Misc
 
 - general test with `usedirs` disabled: **OK**
 
 ### Misc
 
 - general test with `usedirs` disabled: **OK**

-- general test with `indexpages` enabled
-- general test with `po_link_to=default`
+- general test with `indexpages` enabled: **not OK**
+- general test with `po_link_to=default` with `userdirs` enabled: **OK**
+- general test with `po_link_to=default` with `userdirs` disabled: **OK**
+
+Misc. bugs
+----------

 
 Documentation
 -------------
 
 Documentation
 -------------