Link to a new plugin solving my question.

[git.ikiwiki.info.git] / doc / tips / optimising_ikiwiki.mdwn

diff --git a/doc/tips/optimising_ikiwiki.mdwn b/doc/tips/optimising_ikiwiki.mdwn
index 14d619ff824db3662f497884a4d3e5b9301ba811..0c67e606ce2a65a3b5205134f4580eefc70108c6 100644 (file)
--- a/doc/tips/optimising_ikiwiki.mdwn
+++ b/doc/tips/optimising_ikiwiki.mdwn
@@ -1,3 +1,5 @@
+[[!meta date="2009-10-15 18:42:46 -0400"]]
+

 Ikiwiki is a wiki compiler, which means that, unlike a traditional wiki,
 all the work needed to display your wiki is done up front. Where you can
 see it and get annoyed at it. In some ways, this is better than a wiki
 Ikiwiki is a wiki compiler, which means that, unlike a traditional wiki,
 all the work needed to display your wiki is done up front. Where you can
 see it and get annoyed at it. In some ways, this is better than a wiki


@@ -17,19 +19,19 @@ it's slow, and get the problem fixed!)
 
 Are you building your wiki by running a command like this?
 
 
 Are you building your wiki by running a command like this?
 

-       ikiwiki -setup my.setup
+       ikiwiki --setup my.setup

 
 If so, you're always telling ikiwiki to rebuild the entire site, from
 scratch. But, ikiwiki is smart, it can incrementally update a site,
 building only things affected by the changes you make. You just have to let
 it do so:
 
 
 If so, you're always telling ikiwiki to rebuild the entire site, from
 scratch. But, ikiwiki is smart, it can incrementally update a site,
 building only things affected by the changes you make. You just have to let
 it do so:
 

-       ikiwiki -setup my.setup -refresh
+       ikiwiki --setup my.setup --refresh

 
 Ikiwiki automatically uses an incremental refresh like this when handing
 a web edit, or when run from a [[rcs]] post-commit hook. (If you've
 configured the hook in the usual way.) Most people who have run into this
 
 Ikiwiki automatically uses an incremental refresh like this when handing
 a web edit, or when run from a [[rcs]] post-commit hook. (If you've
 configured the hook in the usual way.) Most people who have run into this

-problem got in the habit of running `ikiwiki -setup my.setup` by hand
+problem got in the habit of running `ikiwiki --setup my.setup` by hand

 when their wiki was small, and found it got slower as they added pages.
 
 ## use the latest version
 when their wiki was small, and found it got slower as they added pages.
 
 ## use the latest version


@@ -38,6 +40,14 @@ If your version of ikiwiki is not [[!version]], try upgrading. New
 optimisations are frequently added to ikiwiki, some of them yielding
 *enormous* speed increases.
 
 optimisations are frequently added to ikiwiki, some of them yielding
 *enormous* speed increases.
 

+## run ikiwiki in verbose mode
+
+Try changing a page, and run ikiwiki with `-v` so it will tell you
+everything it does to deal with that changed page. Take note of
+which other pages are rebuilt, and which parts of the build take a long
+time. This can help you zero in on individual pages that contain some of
+the expensive things listed below. 
+

 ## expensive inlines
 
 Do you have an archive page for your blog that shows all posts, 
 ## expensive inlines
 
 Do you have an archive page for your blog that shows all posts, 


@@ -130,7 +140,7 @@ all the pages on a traditional, highly WikiLinked wiki, is asking for things
 to be slow. But using it to map a few related pages is probably fine.
 
 This site's own [[plugins/linkmap]] rarely slows it down, because it
 to be slow. But using it to map a few related pages is probably fine.
 
 This site's own [[plugins/linkmap]] rarely slows it down, because it

-only shows the [[index]] page, and the small set of pages that link to it.
+only shows the index page, and the small set of pages that link to it.

 That is accomplished as follows:
 
        \[[!linkmap pages="index or (backlink(index)"]]
 That is accomplished as follows:
 
        \[[!linkmap pages="index or (backlink(index)"]]


@@ -140,20 +150,22 @@ That is accomplished as follows:
 Be aware that the [[plugins/search]] plugin has to update the search index
 whenever any page is changed. This can slow things down somewhat.
 
 Be aware that the [[plugins/search]] plugin has to update the search index
 whenever any page is changed. This can slow things down somewhat.
 

-## profiling
+## cgi overload workaround

 
 

-If you have a repeatable change that ikiwiki takes a long time to build,
-and none of the above help, the next thing to consider is profiling
-ikiwiki. 
+If the ikiwiki.cgi takes a long time to run, it's possible
+that under load, your site will end up with many
+of them running, all waiting on some long-running thing,
+like a site rebuild. This can prevent the web server from doing anything
+else.

 
 

-The best way to do it is:
+A workaround for this problem is to set `cgi_overload_delay` to 
+a number of seconds. Now if ikiwiki.cgi would block waiting
+for something, it will instead display a Please wait message (configurable
+via `cgi_overload_message`, which can contain arbitrary html),
+and set the page to reload it after the configured number of seconds.

 
 

-* Install [[!cpan Devel::NYTProf]]
-* `PERL5OPT=-d:NYTProf`
-* `export PER5OPT`
-* Now run ikiwiki as usual, and it will generate a `nytprof.out` file.
-* Run `nytprofhtml` to generate html files.
-* Those can be examined to see what parts of ikiwiki are being slow.
+This takes very little load, as it all happens within compiled C code.
+Note that it is currently limited to GET requests, not POST requests.

 
 ## scaling to large numbers of pages
 
 
 ## scaling to large numbers of pages
 


@@ -163,6 +175,12 @@ Finally, let's think about how huge number of pages can affect ikiwiki.
   new and changed pages. This is similar in speed to running the `find`
   command. Obviously, more files will make it take longer.
 
   new and changed pages. This is similar in speed to running the `find`
   command. Obviously, more files will make it take longer.
 

+  You can avoid this scanning overhead, if you're using git, by setting
+  `only_committed_changes`. This makes ikiwiki --refresh query git for
+  changed files since the last time, which tends to be a lot faster.
+  However, it only works if all files in your wiki are committed to git
+  (or stored in the [[/plugins/transient]] underlay).
+

 * Also, to see what pages match a [[ikiwiki/PageSpec]] like "blog/*", it has
   to check if every page in the wiki matches. These checks are done quite
   quickly, but still, lots more pages will make PageSpecs more expensive.
 * Also, to see what pages match a [[ikiwiki/PageSpec]] like "blog/*", it has
   to check if every page in the wiki matches. These checks are done quite
   quickly, but still, lots more pages will make PageSpecs more expensive.


@@ -178,3 +196,18 @@ Finally, let's think about how huge number of pages can affect ikiwiki.
 
 If your wiki will have 100 thousand files in it, you might start seeing
 the above contribute to ikiwiki running slowly.
 
 If your wiki will have 100 thousand files in it, you might start seeing
 the above contribute to ikiwiki running slowly.

+
+## profiling
+
+If you have a repeatable change that ikiwiki takes a long time to build,
+and none of the above help, the next thing to consider is profiling
+ikiwiki. 
+
+The best way to do it is:
+
+* Install [[!cpan Devel::NYTProf]]
+* `PERL5OPT=-d:NYTProf`
+* `export PER5OPT`
+* Now run ikiwiki as usual, and it will generate a `nytprof.out` file.
+* Run `nytprofhtml` to generate html files.
+* Those can be examined to see what parts of ikiwiki are being slow.