From: Joey Hess Date: Tue, 25 Aug 2009 21:21:19 +0000 (-0400) Subject: Merge commit 'remotes/smcv/ready/remove-pagespec-merge' X-Git-Tag: 3.1415926~65 X-Git-Url: http://git.vanrenterghem.biz/git.ikiwiki.info.git/commitdiff_plain/b3d65aee1822cce5d8612f05c972d8db32721bee?hp=848bb66c37b94b633a378e186e561aebb7bab43a Merge commit 'remotes/smcv/ready/remove-pagespec-merge' --- diff --git a/IkiWiki.pm b/IkiWiki.pm index 06a0b7a2c..5563a03af 100644 --- a/IkiWiki.pm +++ b/IkiWiki.pm @@ -1738,15 +1738,6 @@ sub add_depends ($$) { my $page=shift; my $pagespec=shift; - if (ref $pagespec eq 'ARRAY') { - foreach my $ps (@$pagespec) { - if (pagespec_valid($ps)) { - $depends{$page}{$ps} = 1; - } - } - return; - } - return unless pagespec_valid($pagespec); $depends{$page}{$pagespec} = 1; diff --git a/IkiWiki/Plugin/calendar.pm b/IkiWiki/Plugin/calendar.pm index ce0719404..5d16dff75 100644 --- a/IkiWiki/Plugin/calendar.pm +++ b/IkiWiki/Plugin/calendar.pm @@ -212,7 +212,9 @@ EOF add_depends($params{page}, $params{pages}); # Explicitly add all currently linked pages as dependencies, so # that if they are removed, the calendar will be sure to be updated. - add_depends($params{page}, \@list); + foreach my $p (@list) { + add_depends($params{page}, $p); + } return $calendar; } diff --git a/IkiWiki/Plugin/inline.pm b/IkiWiki/Plugin/inline.pm index b566d960f..a501566b5 100644 --- a/IkiWiki/Plugin/inline.pm +++ b/IkiWiki/Plugin/inline.pm @@ -251,7 +251,9 @@ sub preprocess_inline (@) { # Explicitly add all currently displayed pages as dependencies, so # that if they are removed or otherwise changed, the inline will be # sure to be updated. - add_depends($params{page}, $#list >= $#feedlist ? \@list : \@feedlist); + foreach my $p ($#list >= $#feedlist ? @list : @feedlist) { + add_depends($params{page}, $p); + } if ($feeds && exists $params{feedpages}) { @feedlist=pagespec_match_list(\@feedlist, $params{feedpages}, location => $params{page}); diff --git a/IkiWiki/Plugin/map.pm b/IkiWiki/Plugin/map.pm index cc977024d..54146dc46 100644 --- a/IkiWiki/Plugin/map.pm +++ b/IkiWiki/Plugin/map.pm @@ -73,7 +73,9 @@ sub preprocess (@) { add_depends($params{page}, $params{pages}); # Explicitly add all currently shown pages, to detect when pages # are removed. - add_depends($params{page}, [keys %mapitems]); + foreach my $item (keys %mapitems) { + add_depends($params{page}, $item); + } # Create the map. my $parent=""; diff --git a/debian/changelog b/debian/changelog index 5b021ec24..cfe45c3b3 100644 --- a/debian/changelog +++ b/debian/changelog @@ -20,6 +20,8 @@ ikiwiki (3.141593) UNRELEASED; urgency=low Closes: #542036 * po: Fixed to run rcs_add ralative to srcdir. * Italian program translation from Luca Bruno. + * Fix example blog's tags/life to not have a broken PageSpec. + Closes: #543510 -- Joey Hess Wed, 12 Aug 2009 12:25:30 -0400 diff --git a/doc/examples/blog/tags/life.mdwn b/doc/examples/blog/tags/life.mdwn index ddc2e646c..719f2b192 100644 --- a/doc/examples/blog/tags/life.mdwn +++ b/doc/examples/blog/tags/life.mdwn @@ -1,4 +1,4 @@ This feed contains pages in the "life" category. -[[!inline pages="link(tags/life) and ./posts/* and !*/Discussion" +[[!inline pages="link(tags/life) and !*/Discussion" show="10" actions=yes]] diff --git a/doc/todo/should_optimise_pagespecs.mdwn b/doc/todo/should_optimise_pagespecs.mdwn index 5ed24d333..4b4e267f0 100644 --- a/doc/todo/should_optimise_pagespecs.mdwn +++ b/doc/todo/should_optimise_pagespecs.mdwn @@ -125,7 +125,12 @@ uses it still), and otherwise just bloats the index. >> It is acceptable not to support downgrades. >> I don't think we need a NEWS file update since any sort of refresh, >> not just a full rebuild, will cause the indexdb to be loaded and saved, ->> enabling the optimisation. --[[Joey]] +>> enabling the optimisation. --[[Joey]] + +>>> A refresh will load the current dependencies from `{depends}` and save +>>> them as-is as a one-element `{dependslist}`; only a rebuild will replace +>>> the single complex pagespec with a long list of simpler pagespecs. +>>> --[[smcv]] Is an array the right data structure? `add_depends` has to loop through the array to avoid dups, it would be better if a hash were used there. Since @@ -151,7 +156,9 @@ to avoid.. >> a bit faster. --[[smcv]] >>> It depends, really. And it'd certianly make sense to benchmark such a ->>> change. --[[Joey]] +>>> change. --[[Joey]] + +>>>> Benchmarked, below. --[[smcv]] Also, since a lot of places are calling add_depends in a loop, it probably makes sense to just make it accept a list of dependencies to add. It'll be @@ -165,7 +172,10 @@ when adding a lot of depends at once. >> Well, I was thinking that it might be sufficient to build a `%seen` >> hash of dependencies inside `add_depends`, if the places that call >> it lots were changed to just call it once. Of course the only way to ->> tell is benchmarking. --[[Joey]] +>> tell is benchmarking. --[[Joey]] + +>>> It doesn't seem that it significantly affects performance either way. +>>> --[[smcv]] In Render.pm, we now have a triply nested loop, which is a bit scary for efficiency. It seems there should be a way to @@ -182,7 +192,126 @@ out. >> run more often than before. That function is pretty inexpensive, but.. >> --[[Joey]] +>>> I don't see anything that can be hoisted without significant refactoring, +>>> actually. Beware that there are two pagename calls in the loop: one for +>>> `$f` (which is the page we might want to rebuild), and one for `$file` +>>> (which is the changed page that it might depend on). Note that I didn't +>>> choose those names! +>>> +>>> The three loops are over source files, their lists of dependency pagespecs, +>>> and files that might have changed. I see the following things we might be +>>> doing redundantly: +>>> +>>> * If `$file` is considered as a potential dependency for more than +>>> one `$f`, we evaluate `pagename($file)` more than once. Potential fix: +>>> cache them (this turns out to save about half a second on the docwiki, +>>> see below). +>>> * If several pages depend on the same pagespec, we evaluate whether each +>>> changed page matches that pagespec more than once: however, we do so +>>> with a different location parameter every time, so repeated calls are, +>>> in the general case, the only correct thing to do. Potential fix: +>>> perhaps special-case "page x depends on page y and nothing else" +>>> (i.e. globs that have no wildcards) into a separate hash? I haven't +>>> done anything in this direction. +>>> * Any preparatory work done by pagespec_match (converting the pagespec +>>> into Perl, mostly?) is done in the inner loop; switching to +>>> pagespec_match_list (significant refactoring) saves more than half a +>>> second on the docwiki. +>>> +>>> --[[smcv]] + Very good catch on img/meta using the wrong dependency; verified in the wild! (I've cherry-picked those bug fixes.) +---- + +Benchmarking results: I benchmarked by altering docwiki.setup to switch off +verbose, running "make clean && ./Makefile.PL && make", and timing one rebuild +of the docwiki followed by three refreshes. Before each refresh I used +`touch plugins/*.mdwn` to have something significant to refresh. + +I'm assuming that "user" CPU time is the important thing here (system time was +relatively small in all cases, up to 0.35 seconds per run). + +master at the time of rebasing: 14.20s to rebuild, 10.04/12.07/14.01s to +refresh. I think you can see the bug clearly here - the pagespecs are getting +more complicated every time! + +> I can totally see a bug here, and it's one I didn't think existed. Ie, +> I thought that after the first refresh, the pagespec should stabalize, +> and what it stabalized to was probably unnecessarily long, but not +> growing w/o bounds! +> +> a) Explains why ikiwiki.info has been so slow lately. Well that and some +> other things that overloaded the system. +> b) Suggests to me we will probably want to force a rebuild on upgrade +> when fixing this (via the mechanism in the postinst). +> +> I've investigated why the pagespecs keep growing: When page A changes, +> its old depends are cleared. Then +> page B that inlines A gets rebuilt, and its old depends are also cleared. +> But page B also inlines page C; which means C gets re-rendered. And this +> happens w/o its old depends being cleared, so C's depends are doubled. +> --[[Joey]] + +After the initial optimization: 14.27s to rebuild, 8.26/8.33/8.26 to refresh. +Success! + +Not pre-joining dependencies actually took about ~0.2s more; I don't know why. +I'm worried that duplicates will just build up (again) in less simple cases, +though, so 0.2s is probably a small price to pay for that not happening (it +might well be experimental error, for that matter). + +> It's weird that the suggested optimisations to +> `add_depends` had no effect. So, the commit message to +> b6fcb1cb0ef27e5a63184440675d465fad652acf is actually wrong.. ? --[[Joey]] + +>> I'll try benchmarking again on the non-public wiki where I had the 4% +>> speedup. The docwiki is so small that 4% is hard to measure... --[[smcv]] + +Not saving {depends} to the index, using a hash instead of a list to +de-duplicate, and allowing add_depends to take an arrayref instead of a single +pagespec had no noticable positive or negative effect on this test. + +> I see e4cd168ebedd95585290c97ff42234344bfed46c is still in your branch +> though. I don't like using an arrayref, it could just take `($page, @depends)`. +> and I don't see the need to keep it if it doesn't currently help. + +>> I'll drop it. --[[smcv]] + +> Is there any reason to keep 7227c2debfeef94b35f7d81f42900aa01820caa3 +> if it doesn't improve speed? +> --[[Joey]] + +>> I'll try benchmarking on a more complex wiki and see whether it has a +>> positive or negative effect. It does avoid being O(n**2) in number of +>> dependencies. --[[smcv]] + +Memoizing the results of pagename brought the rebuild time down to 14.06s +and the refresh time down to 7.96/7.92/7.92, a significant win. + +> Ok, that seems safe to memoize. (It's a real function and it isn't +> called with a great many inputs.) Why did you chose to memoize it +> explicitly rather than adding it to the memoize list at the top? + +>> It does depend on global variables, so using Memoize seemed like asking for +>> trouble. I suppose what I did is equivalent to Memoize though... --[[smcv]] + +Refactoring to use pagespec_match_list looks more risky from a code churn +point of view; rebuild now takes 14.35s, but refresh is only 7.30/7.29/7.28, +another significant win. + +--[[smcv]] + +> I had mostly convinced myself that +> `pagespec_match_list` would not lead to a speed gain here. My reasoning +> was that you want to stop after finding one match, while `pagespec_match_list` +> checks all pages for matches. So what we're seeing is that +> on a rebuild, `@changed` is all pages, and not short-circuiting leads +> to unnecessary work. OTOH, on refresh, `@changed` is small and I suppose +> `pagespec_match_list`'s other slight efficiencies win out somehow. +> +> Welcome to the "I made ikiwiki twice as fast +> and all I got was this lousy git sha1sum" club BTW :-) --[[Joey]] + [[!tag wishlist patch patch/core]]