X-Git-Url: http://git.vanrenterghem.biz/git.ikiwiki.info.git/blobdiff_plain/c940cabee5811a954bdcd44c4882501c126b5408..512d860c169d776885922b7eaff0bcb2b722f12a:/doc/plugins/aggregate/discussion.mdwn

diff --git a/doc/plugins/aggregate/discussion.mdwn b/doc/plugins/aggregate/discussion.mdwn
index b61e0d879..a3336ee08 100644
--- a/doc/plugins/aggregate/discussion.mdwn
+++ b/doc/plugins/aggregate/discussion.mdwn
@@ -35,7 +35,7 @@ Two things aren't working as I'd expect:
 > problem. You can see the feed validator complain about it here:
 > <http://feedvalidator.org/check.cgi?url=http%3A%2F%2Fwww.davidj.org%2Frss.xml>
 > 
-> It's sorta unfortunate that [[cpan XML::Feed]] doesn't just assume the
+> It's sorta unfortunate that [[!cpan XML::Feed]] doesn't just assume the
 > un-esxaped html is part of the description field. Probably other feed
 > parsers are more lenient. --[[Joey]]
 
@@ -49,3 +49,75 @@ Two things aren't working as I'd expect:
 
 >>>>> I think that both issues are now fixed. Thanks for testing.
 >>>>> --[[Joey]]
+
+>>>>>> I can confirm, they're fixed on my end. --[[schmonz]]
+
+New bug: new posts aren't getting displayed (or cached for aggregation). After fixing his feed, David posted a new item today, and the aggregator is convinced there's nothing to do, whether by cronjob or webtrigger. I verified that it wasn't another problem with his feed by adding another of my ikiwiki's feed to the planet, running the aggregator, posting a new item, and running the aggregator again: no new item. --[[schmonz]]
+
+> Even if you start it more frequently, aggregation will only occur every
+> `updateinterval` minutes (default 15), maximum. Does this explain what
+> you're seeing? --[[Joey]]
+
+>> Crap, right, and my test update has since made it into the planet. His post still hasn't. So it must be something with David's feed again? A quick test with XML::Feed looks like it's parsing just fine: --[[schmonz]]
+
+    $ perl 
+    use XML::Feed;
+    my $feed = XML::Feed->parse(URI->new('http://www.davidj.org/rss.xml')) or die XML::Feed->errstr;
+    print $feed->title, "\n";
+    for my $entry ($feed->entries) {
+    print $entry->title, ": ", $entry->issued, "\n";
+    }
+    ^D
+    davidj.org
+    Amway Stories - Refrigerator Pictures: 2008-09-19T00:12:27
+    Amway Stories - Coffee: 2008-09-13T10:08:17
+    Google Alphabet Update: 2008-09-11T22:55:37
+    Writing for writing's sake: 2008-09-09T23:39:05
+    Google Chrome: 2008-09-02T23:12:26
+    Mister Casual: 2008-07-25T09:01:17
+    Parental Conversations: 2008-07-24T10:44:44
+    Place Of George Orwell: 2008-06-03T22:11:07
+    The Raw Beauty Of A National Duolian: 2008-05-31T12:41:06
+
+> I had no problem getting the "Refrigerator Pictures" post to aggregate
+> here, though without a copy of the old feed I can't be 100% sure I've
+> reproduced your ikiwiki's state. --[[Joey]]
+
+>> Okay, I blew away the cached entries and aggregator state files and reran the aggregator and all appears well again. If the problem recurs I'll be sure to post here. :-) --[[schmonz]]
+
+>>> On the off chance that you retained a copy of the old state, I'd not
+>>> mind having a copy to investigate. --[[Joey]]
+
+>>>> Didn't think of that, will keep a copy if there's a next time. -- [[schmonz]]
+
+-----
+
+In a corporate environment where feeds are generally behind
+authentication, I need to prime the aggregator's `LWP::UserAgent`
+with some cookies. What I've done is write a custom plugin to populate
+`$config{cookies}` with an `HTTP::Cookies` object, plus this diff:
+
+    --- /var/tmp/pkg/lib/perl5/vendor_perl/5.10.0/IkiWiki/Plugin/aggregate.pm  2010-06-24 13:03:33.000000000 -0400
+    +++ aggregate.pm    2010-06-24 13:04:09.000000000 -0400
+    @@ -488,7 +488,11 @@
+                        }
+                        $feed->{feedurl}=pop @urls;
+                }
+    -           my $res=URI::Fetch->fetch($feed->{feedurl});
+    +           my $res=URI::Fetch->fetch($feed->{feedurl},
+    +                                     UserAgent => LWP::UserAgent->new(
+    +                                           cookie_jar => $config{cookies},
+    +                                     ),
+    +           );
+                if (! $res) {
+                        $feed->{message}=URI::Fetch->errstr;
+                        $feed->{error}=1;
+
+It works, but I have to remember to apply the diff whenever I update
+ikiwiki.  Can you provide a more elegant means of allowing cookies and/or
+the user agent to be programmatically manipulated? --[[schmonz]]
+
+> Ping -- is the above patch perhaps acceptable (or near-acceptable)? -- [[schmonz]]
+
+>> Pong.. I'd be happier with a more 100% solution that let cookies be used
+>> w/o needing to write a custom plugin to do it. --[[Joey]]