X-Git-Url: http://git.vanrenterghem.biz/git.ikiwiki.info.git/blobdiff_plain/66e894c8771c5a5dfbd4cb92f826d4d397b2256f..808c699961eae0de7125812d4f1c51ecd5fc6c18:/doc/todo/dependency_types.mdwn?ds=sidebyside diff --git a/doc/todo/dependency_types.mdwn b/doc/todo/dependency_types.mdwn index 9d649e1e0..97cff97c5 100644 --- a/doc/todo/dependency_types.mdwn +++ b/doc/todo/dependency_types.mdwn @@ -188,7 +188,8 @@ before and it is present now. Should this cause a re-build of any page that has > Yes, a presence dep will trigger when a page is added, or removed. > Your example is valid.. but it's also not handled right by normal, -> (content) dependencies, for the same reasons. --[[Joey]] +> (content) dependencies, for the same reasons. Still, I think I've +> addressed it with the pagespec influence stuff below. --[[Joey]] I think that is another version of the problem you encountered with meta-data. @@ -221,7 +222,7 @@ ShavedByBob.mdwn: Does ShavedByBob.mdwn include itself? -(Yeah - in IkiWiki currently links are included by include, but the idea holds. I had a good example a while back, but I can't think of it right now.) +(Yeah - in IkiWiki currently links are *not* included by include, but the idea holds. I had a good example a while back, but I can't think of it right now.) sigh. @@ -229,16 +230,14 @@ sigh. > I have also been thinking about some sort of analysis pass over pagespecs > to determine what metadata, pages, etc they depend on. It is indeed -> tricky to do. Even if it's just limited to returning a list of pages -> as you suggest. -> -> Consider: For a `*` glob, it has to return a list of all pages -> in the wiki. Which is expensive. And what if the pagespec is -> something like `* and backlink(index)`? Without analyising the -> boolean relationship between terms, the returned list -> will have many more items in it than it should. Or do we not make -> globs return their matches? (If so we have to deal with those -> with one of the other methods disucssed.) --[[Joey]] +> tricky to do. More thoughts on influence lists a bit below. --[[Joey]] + +>> The big part of what makes this tricky is that there may be cycles in the +>> dependency graph. This can lead to situations where the result is just not +>> well defined. This is what I was trying to get at above. -- [[Will]] + +>>> Hmm, I'm not seeing cycles be a problem, at least with the current +>>> pagespec terms. --[[Joey]] ---- @@ -246,10 +245,7 @@ sigh. * `add_depends($page, $spec, links => 1, presence => 1)` adds a links + presence dependency. -* `refresh` only rebuilds a page with a links dependency if - pages matched by the pagespec gain or lose links. (What the link - actually points to may change independent of this, due to changes - elsewhere, without it firing.) +* Use backlinks change code to detect changes to link dependencies too. * So, brokenlinks can fire whenever any links in any of the pages it's tracking change, or when pages are added or removed. @@ -283,7 +279,7 @@ One way to fix this is to include with each dependency, a list of pages that currently match it. If the list changes, the dependency is triggered. Should be doable, but may involve more work than -currently. Consider that a dependency on "bugs/*" currently +currently. Consider that a dependency on `bugs/*` currently is triggered by just checking until *one* page is found to match it. But to store the list, *every* page would have to be tried against it. Unless the list can somehow be intelligently updated, looking at only the @@ -291,13 +287,202 @@ changed pages. ---- -What if there were a function that added a dependency, and at the same time -returned a list of pages matching the pagespec? Plugins that use this would -be exactly the ones, like inline and map, for which this is a problem, and -which already do a match pass over all pages. +Found a further complication in presence dependencies. Map now uses +presence dependencies when adding its explicit dependencies on pages. But +this defeats the purpose of the explicit dependencies! Because, now, +when B is changed to not match a pagespec, the A's presence dep does +not fire. + +I didn't think things through when switching it to use presence +dependencies there. But, if I change it to use full dependencies, then all +the work that was done to allow map to use presence dependencies for its +main pagespec is for naught. The map will once again have to update +whenever *any* content of the page changes. + +This points toward the conclusion that explicit dependencies, however they +are added, are not the right solution at all. Some other approach, such as +maintaining the list of pages that match a dependency, and noticing when it +changes, is needed. + +---- + +### pagespec influence lists + +I'm using this term for the concept of a list of pages whose modification +can indirectly influence what pages a pagespec matches. + +> Trying to make a formal definition of this: (Note, I'm using the term sets rather than lists, but they're roughly equivalent) +> +> * Let the *matching set* for a pagespec be the set of pages that the pagespec matches. +> * Let a *complete influence set* for a pagespec be the set of all pages whose alteration might change the matching set of that pagespec. +> * Let the *direct influence set* be the intersection of the matching set and the complete influence set. +> * Let the *indirect influence set* be the compliment of the direct influence set with respect to the complete influence set. +> +> Is that a fair definition? I don't think it quite matches your examples below unfortunately. +> I was unsure if I should insert the word 'existing' in there in a few places. As it stands, these definitions could include sets of pages that don't exist, e.g. "*". +> The one I'm least sure of is the definition of the direct influence set. It feels like you want something +> like "the traditional set of things we thought about that could cause a pagespec to change", but that definition +> is not very formal and I suspect will lead to problems. Something like "The set of pages matched by the globs in the pagespec" might be closer? +> --[[Will]] + +>> I appreciate the formalism! +>> +>> Only existing pages need to be in these sets, because if a page is added +>> in the future, the existing dependency code will always test to see +>> if it matches. So it will be in the maching set (or not) at that point. +>> +>> The problem with your definition of direct influence set seems to be +>> that it doesn't allow `link()` and `title()` to have as an indirect +>> influence, the page that matches. But I'm quite sure we need those. +>> --[[Joey]] + +#### Examples + +* The pagespec "created_before(foo)" has an influence list that contains foo. + The removal or (re)creation of foo changes what pages match it. + +* The pagespec "foo" has an empty influence list. This is because a + modification/creation/removal of foo directly changes what the pagespec + matches. + +* The pagespec "*" has an empty influence list, for the same reason. + Avoiding including every page in the wiki into its influence list is + very important! + +* The pagespec "title(foo)" has an influence list that contains every page + that currently matches it. A change to any matching page can change its + title, making it not match any more, and so the list is needed due to the + removal problem. + +* The pagespec "backlink(index)" has an influence list + that contains index (because a change to index changes the backlinks). + +* The pagespec "link(done)" has an influence list that + contains every page that it matches. A change to any matching page can + remove a link and make it not match any more, and so the list is needed + due to the removal problem. + +>> Why doesn't this include every page? If I change a page that doesn't have a link to +>> 'done' to include a link to 'done', then it will now match... or is that considered a +>> 'direct match'? -- [[Will]] + +>>> The regular dependency calculation code will check if every changed +>>> page matches every dependency. So it will notice the link was added. +>>> --[[Joey]] + +#### Low-level Calculation + +One way to calculate a pagespec's influence would be to +expand the SuccessReason and FailReason objects used and returned +by `pagespec_match`. Make the objects be created with an +influence list included, and when the objects are ANDed or ORed +together, combine the influence lists. + +That would have the benefit of allowing just using the existing `match_*` +functions, with minor changes to a few of them to gather influence info. + +But does it work? Let's try some examples: + +Consider "bugs/* and link(done) and backlink(index)". + +Its influence list contains index, and it contains all pages that the whole +pagespec matches. It should, ideally, not contain all pages that link +to done. There are a lot of such pages, and only a subset influence this +pagespec. + +When matching this pagespec against a page, the `link` will put the page +on the list. The `backlink` will put index on the list, and they will be +anded together and combined. If we combine the influences from each +successful match, we get the right result. + +Now consider "bugs/* and link(done) and !backlink(index)". + +It influence list is the same as the previous one, even though a term has +been negated. Because a change to index still influences it, though in a +different way. + +If negation of a SuccessReason preserves the influence list, the right +influence list will be calculated. + +Consider "bugs/* and (link(done) or backlink(index))" +and "bugs/* and (backlink(index) or link(done))' + +Its clear that the influence lists for these are identical. And they +contain index, plus all matching pages. + +When matching the first against page P, the `link` will put P on the list. +The OR needs to be a non-short-circuiting type. (In perl, `or`, not `||` -- +so, `pagespec_translate` will need to be changed to not use `||`.) +Given that, the `backlink` will always be evalulated, and will put index +onto the influence list. If we combine the influences from each +successful match, we get the right result. + +> This is implemented, seems to work ok. --[[Joey]] + +> `or` short-circuits too, but the implementation correctly uses `|`, +> which I assume is what you meant. --[[smcv]] + +#### High-level Calculation and Storage + +Naively calculating the full influence list for a pagespec requires trying +to match it against every page in the wiki. I'd like to avoid doing such +expensive matching redundantly. + +It may be possible, for some types of pagespecs, to just try matching a +single, arbitrary page against it, and know the full influence list has +been obtained. It seems to be that case that if a pagespec has any +influences, matching any page will return at least one. So if none are +returned, we can skip trying other pages. + +If the influence list does not include the page that was tried, we know +that the pagespec does not things like `link()` and `title()`, that are +influenced by the page's own content. So it *might* be safe to not try +matching any more pages in this case too. I think it would work for all +current pagespec terms. There might be a hypothetical term where this +optimisation doesn't work. We could add a special case to ensure it can +work: If a term declares it is unfluenced by "", then it means it is +always influenced by the matching page. + +Anyway, this seems worth doing: Add a `pagespec_match_all`, which returns a +list of all pages in the whole wiki that match the pagespec, and also adds +the pagespec as a dependency, and while it's at it, calculates and stores +the influence list. + +It could have an optional sort parameter, and limit parameter, to control +how many items to return and the sort order. So when inline wants to +display the 10 newest, only the influence lists for those ten are added. + +If `pagespec_match_depends` can be used by all plugins, then great, +influences are automatically calculated, no extra work needs to be done. + +If not, and some plugins still need to use `pagespec_match_list` or +`pagespec_match`, and `add_depends`, then I guess that `add_depends` can do +a slightly more expensive influence calculation. + +Bonus: If `add_depends` is doing an influence calculation, then I can remove +the nasty hack it currently uses to decide if a given pagespec is safe to use +with an existence or links dependency. + +Where to store the influence list? Well, it appears that we can just add +(content) dependencies for each item on the list, to the page's +regular list of simple dependencies. So, the data stored ends up looking +just like what is stored today by the explicit dependency hacks. Except, +it's calculated more smartly, and is added automatically. + +> I've implemented influence calculation in `add_depends`. As expected, +> it means rather a lot more work, and makes some things much slower. +> Optimisations next.. --[[Joey]] + +#### Influence types + +Note that influences can also have types, same as dependency types. +For example, "backlink(foo)" has an influence of foo, of type links. +"created_before(foo)" also is influenced by foo, but it's a presence +type. Etc. + +> This is an interesting concept that I hadn't considered. It might +> allow significant computational savings, but I suspect will be tricky +> to implement. -- [[Will]] -Adding explicit dependencies during this pass would thus be nearly free. -Not 100% free since it would add explicit deps for things that are not -shown on an inline that limits its display to the first sorted N items. -I suppose we could reach 100% free by making the function also handle -sorting and limiting, though that could be overkill. +>> It was actually really easy to implement it, assuming I picked the right +>> dependency types of course. --[[Joey]]