X-Git-Url: http://git.vanrenterghem.biz/git.ikiwiki.info.git/blobdiff_plain/bff9a883768038c0cd6cd3aca46a637ac6767898..3a04e96389def78bcb873a4487b85f4d75653199:/doc/todo/git-annex_support.mdwn?ds=sidebyside diff --git a/doc/todo/git-annex_support.mdwn b/doc/todo/git-annex_support.mdwn index 78219625b..f1a4e787c 100644 --- a/doc/todo/git-annex_support.mdwn +++ b/doc/todo/git-annex_support.mdwn @@ -101,3 +101,144 @@ An alternative implementation, which remains to be detailed but is mentionned in Also note that ikiwiki-hosting has a [patch waiting](https://ikiwiki-hosting.branchable.com/todo/git-annex_support) to allow pushes to work with git-annex. This could potentially be expanded to sync content to the final checkout properly, avoiding some of the problems above (esp. wrt to non-annex bare repos). Combined with the [[underlay]] feature, this could work very nicely indeed... --[[anarcat]] + +Here's an attempt: + +
+cd /home/user +git clone source.git source.annex +cd source.annex +git annex direct +cd ../source.git +git annex group . transfer +git remote add annex ../source.annex +git annex sync annex ++ +Make sure the `hardlink` setting is enabled, and add the annex as an underlay, in `ikiwiki.setup`: + +
+hardlink: 1 +add_underlays: +- /home/w-anarcat/source.annex ++ +Then moving files to the underlay is as simple as running this command in the bare repo: + +
+#!/bin/sh + +echo "moving big files to annex repository..." +git annex move --to annex ++ +I have added this as a hook in `$HOME/source.git/hooks/post-receive` (don't forget to `chmod +x`). + +The problem with the above is that the underlay wouldn't work: for some reason it wouldn't copy those files in place properly. Maybe it's freaking out because it's a full copy of the repo... My solution was to make the source repository itself a direct repo, and then add it as a remote to the bare repo. --[[anarcat]] + +Back from the top +================= + +Obviously, the final approach of making the `source` repository direct mode will fail because ikiwiki will try to commit files there from the web interface which will fail (at best) and (at worst) add big files into git-annex (or vice-versa, not sure what's worse actually). + +Also, I don't know how others here made the underlay work, but it didn't work for me. I think it's because in the "source" repository, there are (dead) symlinks for the annexed files. This overrides the underlay, because of [[security]] - although I am unclear as to why this is discarded so early. So in order to make the original idea above work properly (ie. having a separate git-annex repo in direct mode) work, we must coerce ikiwiki into tolerating symlinks in the srcdir a little more: + +
+diff --git a/IkiWiki.pm b/IkiWiki.pm +index 1043ef4..949273c 100644 +--- a/IkiWiki.pm ++++ b/IkiWiki.pm +@@ -916,11 +916,10 @@ sub srcfile_stat { + my $file=shift; + my $nothrow=shift; + +- return "$config{srcdir}/$file", stat(_) if -e "$config{srcdir}/$file"; +- foreach my $dir (@{$config{underlaydirs}}, $config{underlaydir}) { +- return "$dir/$file", stat(_) if -e "$dir/$file"; ++ foreach my $dir ($config{srcdir}, @{$config{underlaydirs}}, $config{underlaydir}) { ++ return "$dir/$file", stat(_) if (-e "$dir/$file" && ! -l "$dir/$file"); + } +- error("internal error: $file cannot be found in $config{srcdir} or underlay") unless $nothrow; ++ error("internal error: $file cannot be found in $config{srcdir} or underlays @{$config{underlaydirs}} $config{underlaydir}") unless $nothrow; + return; + } + +diff --git a/IkiWiki/Render.pm b/IkiWiki/Render.pm +index 9d6f636..e0b4cf8 100644 +--- a/IkiWiki/Render.pm ++++ b/IkiWiki/Render.pm +@@ -337,7 +337,7 @@ sub find_src_files (;$$$) { + + if ($underlay) { + # avoid underlaydir override attacks; see security.mdwn +- if (! -l "$abssrcdir/$f" && ! -e _) { ++ if (1 || ! -l "$abssrcdir/$f" && ! -e _) { + if (! $pages{$page}) { + push @files, $f; + push @IkiWiki::underlayfiles, $f; ++ +
+../IkiWiki/Render.pm:421: find_src_files(1, \@files, \%pages); +../IkiWiki/Render.pm:846: ($files, $pages)=find_src_files(); +../po/po2wiki:18:my ($files, $pages)=IkiWiki::find_src_files(); ++ +The first occurence is in `IkiWiki::Render::process_changed_files`, where it is used mostly for populating `@IkiWiki::underlayfiles`, the only side effect of +`find_src_files`. The second occurence is in `IkiWiki::Render::refresh`. There things are a little more complicated (to say the least) and a lot of stuff happens. To put it in broad terms, first it does a `IkiWiki::Render::scan` and then a `IkiWiki::Render::render`. The last two call `srcfile()` appropriately (where i put an extra symlink check), except for `will_render()` in `scan`, which I can't figure out right now and that seems to have a lot of global side effects. It still looks fairly safe at first glance. The `rcs_get_current_rev`, `refresh`, `scan` and `rendered` hooks are also called in there, but I assume those to be safe, since they are called with sanitized values already. + +The patch does work: the files get picked up from the underlay and properly hardlinked into the target `public_html` directory! So with the above patch, then the following hook in `source.git/hooks/post-receive`: + +
+#!/bin/sh + +OLD_GIT_DIR="$GIT_DIR" +unset GIT_DIR +echo "moving big files to annex repository..." +git annex copy --to annex +git annex sync annex ++ +(I am not sure anymore why GIT_DIR is necessary, but I remember it destroyed all files in my repo because git-annex synced against the `setup` branch in the parent directory. fun times.) + +Then the `annex` repo is just a direct clone of the source.git: + +
+cd /home/user +git clone --shared source.git annex +cd annex +git annex direct +cd ../source.git +git remote add annex ../annex ++ +And we need the following config: + +
+hardlink: 1 +add_underlays: +- /home/w-anarcat/annex +add_plugins: +- underlay ++ +... and the `ikiwiki-hosting` patch mentionned earlier to allow git-annex-shell to run at all. Also, the `--shared` option will [make git-annex use hardlinks itself between the two repos](https://git-annex.branchable.com/todo/wishlist:_use_hardlinks_for_local_clones/), so the files will be available for download as well. --[[anarcat]] + +>