Merge branch 'master' into cvs

[git.ikiwiki.info.git] / doc / todo / should_optimise_pagespecs.mdwn

diff --git a/doc/todo/should_optimise_pagespecs.mdwn b/doc/todo/should_optimise_pagespecs.mdwn
index 3dfa8e1f28a0c1a87bdfe4750ec319cc2f043286..728ab89943488e0656b13c7c2947b0d379a3475d 100644 (file)
--- a/doc/todo/should_optimise_pagespecs.mdwn
+++ b/doc/todo/should_optimise_pagespecs.mdwn
@@ -79,8 +79,6 @@ I can think about reducung the size of my wiki source and making it available on
 > 
 > --[[Joey]]
 
 > 
 > --[[Joey]]
 

-[[!template id=gitbranch branch=smcv/ready/optimize-depends author="[[smcv]]"]]
-

 >> I've been looking at optimizing ikiwiki for a site using
 >> [[plugins/contrib/album]] (which produces a lot of pages) and it seems
 >> that checking which pages depend on which pages does take a significant
 >> I've been looking at optimizing ikiwiki for a site using
 >> [[plugins/contrib/album]] (which produces a lot of pages) and it seems
 >> that checking which pages depend on which pages does take a significant


@@ -90,6 +88,8 @@ I can think about reducung the size of my wiki source and making it available on
 >> rather than a single pagespec. This does turn out to be faster, although
 >> not as much as I'd like. --[[smcv]]
 
 >> rather than a single pagespec. This does turn out to be faster, although
 >> not as much as I'd like. --[[smcv]]
 

+>>> [[Merged|done]] --[[smcv]]
+

 >>> I just wanted to note that there is a whole long discussion of dependencies and pagespecs on the [[todo/tracking_bugs_with_dependencies]] page. -- [[Will]]
 
 >>>> Yeah, I had a look at that (as the only other mention of `pagespec_merge`).
 >>> I just wanted to note that there is a whole long discussion of dependencies and pagespecs on the [[todo/tracking_bugs_with_dependencies]] page. -- [[Will]]
 
 >>>> Yeah, I had a look at that (as the only other mention of `pagespec_merge`).


@@ -98,8 +98,6 @@ I can think about reducung the size of my wiki source and making it available on
 >>>> I haven't actually deleted it), because the "or" operation is now done in
 >>>> the Perl code, rather than by merging pagespecs and translating. --[[smcv]]
 
 >>>> I haven't actually deleted it), because the "or" operation is now done in
 >>>> the Perl code, rather than by merging pagespecs and translating. --[[smcv]]
 

-[[!template id=gitbranch branch=smcv/ready/remove-pagespec-merge author="[[smcv]]"]]
-

 >>>>> I've now added a patch to the end of that branch that deletes
 >>>>> `pagespec_merge` almost entirely (we do need to keep a copy around, in
 >>>>> ikiwiki-transition, but that copy doesn't have to be optimal or support
 >>>>> I've now added a patch to the end of that branch that deletes
 >>>>> `pagespec_merge` almost entirely (we do need to keep a copy around, in
 >>>>> ikiwiki-transition, but that copy doesn't have to be optimal or support


@@ -235,6 +233,23 @@ master at the time of rebasing: 14.20s to rebuild, 10.04/12.07/14.01s to
 refresh. I think you can see the bug clearly here - the pagespecs are getting
 more complicated every time!
 
 refresh. I think you can see the bug clearly here - the pagespecs are getting
 more complicated every time!
 

+> I can totally see a bug here, and it's one I didn't think existed. Ie,
+> I thought that after the first refresh, the pagespec should stabalize,
+> and what it stabalized to was probably unnecessarily long, but not
+> growing w/o bounds!
+> 
+> a) Explains why ikiwiki.info has been so slow lately. Well that and some
+>    other things that overloaded the system.
+> b) Suggests to me we will probably want to force a rebuild on upgrade
+>    when fixing this (via the mechanism in the postinst).
+>
+> I've investigated why the pagespecs keep growing: When page A changes,
+> its old depends are cleared. Then
+> page B that inlines A gets rebuilt, and its old depends are also cleared.
+> But page B also inlines page C; which means C gets re-rendered. And this
+> happens w/o its old depends being cleared, so C's depends are doubled.
+> --[[Joey]]
+

 After the initial optimization: 14.27s to rebuild, 8.26/8.33/8.26 to refresh.
 Success!
 
 After the initial optimization: 14.27s to rebuild, 8.26/8.33/8.26 to refresh.
 Success!
 


@@ -243,17 +258,56 @@ I'm worried that duplicates will just build up (again) in less simple cases,
 though, so 0.2s is probably a small price to pay for that not happening (it
 might well be experimental error, for that matter).
 
 though, so 0.2s is probably a small price to pay for that not happening (it
 might well be experimental error, for that matter).
 

+> It's weird that the suggested optimisations to
+> `add_depends` had no effect. So, the commit message to
+> b6fcb1cb0ef27e5a63184440675d465fad652acf is actually wrong.. ? --[[Joey]] 
+
+>> I'll try benchmarking again on the non-public wiki where I had the 4%
+>> speedup. The docwiki is so small that 4% is hard to measure... --[[smcv]]
+

 Not saving {depends} to the index, using a hash instead of a list to
 de-duplicate, and allowing add_depends to take an arrayref instead of a single
 pagespec had no noticable positive or negative effect on this test.
 
 Not saving {depends} to the index, using a hash instead of a list to
 de-duplicate, and allowing add_depends to take an arrayref instead of a single
 pagespec had no noticable positive or negative effect on this test.
 

+> I see e4cd168ebedd95585290c97ff42234344bfed46c is still in your branch
+> though. I don't like using an arrayref, it could just take `($page, @depends)`.
+> and I don't see the need to keep it if it doesn't currently help.
+
+>> I'll drop it. --[[smcv]]
+
+> Is there any reason to keep 7227c2debfeef94b35f7d81f42900aa01820caa3
+> if it doesn't improve speed?
+> --[[Joey]] 
+
+>> I'll try benchmarking on a more complex wiki and see whether it has a
+>> positive or negative effect. It does avoid being O(n**2) in number of
+>> dependencies. --[[smcv]]
+

 Memoizing the results of pagename brought the rebuild time down to 14.06s
 and the refresh time down to 7.96/7.92/7.92, a significant win.
 
 Memoizing the results of pagename brought the rebuild time down to 14.06s
 and the refresh time down to 7.96/7.92/7.92, a significant win.
 

+> Ok, that seems safe to memoize. (It's a real function and it isn't
+> called with a great many inputs.) Why did you chose to memoize it
+> explicitly rather than adding it to the memoize list at the top?
+
+>> It does depend on global variables, so using Memoize seemed like asking for
+>> trouble. I suppose what I did is equivalent to Memoize though... --[[smcv]]
+

 Refactoring to use pagespec_match_list looks more risky from a code churn
 point of view; rebuild now takes 14.35s, but refresh is only 7.30/7.29/7.28,
 another significant win.
 
 --[[smcv]]
 
 Refactoring to use pagespec_match_list looks more risky from a code churn
 point of view; rebuild now takes 14.35s, but refresh is only 7.30/7.29/7.28,
 another significant win.
 
 --[[smcv]]
 

+> I had mostly convinced myself that
+> `pagespec_match_list` would not lead to a speed gain here. My reasoning
+> was that you want to stop after finding one match, while `pagespec_match_list`
+> checks all pages for matches. So what we're seeing is that
+> on a rebuild, `@changed` is all pages, and not short-circuiting leads
+> to unnecessary work. OTOH, on refresh, `@changed` is small and I suppose
+> `pagespec_match_list`'s other slight efficiencies win out somehow.
+> 
+> Welcome to the "I made ikiwiki twice as fast
+> and all I got was this lousy git sha1sum" club BTW :-) --[[Joey]] 
+

 [[!tag wishlist patch patch/core]]
 [[!tag wishlist patch patch/core]]