Repository navigation
Description
Description
My goal is to build a static website using go-task.
I have a hierarchy like this:
.
├── bin
├── output
│ └── public
│ └── html
│ ├── essays
│ └── notes
├── private
│ ├── resume
│ │ └── Taskfile.yml
│ └── Taskfile.yml
├── public
│ ├── essays
│ │ └── Taskfile.yml
│ ├── notes
│ │ └── Taskfile.yml
│ └── Taskfile.yml
└── Taskfile.yml
I want to use fingerprinting to avoid unnecessary work. I don't want conversion to HTML to happen sequentially because it will take forever.
The meat of my solution is as follows:
compile:
internal: true
label: compile-essay-{{.FILE}}
requires:
vars: [FILE, OUT_DIR, OUT]
sources:
- '{{.FILE}}'
generates:
- '{{.OUT}}'
cmds:
- pandoc {{.PANDOC_FLAGS}} {{.FILE}} -o {{.OUT}}
build-all:
desc: Compile all .md files in parallel with fingerprinting
silent: true
vars:
FILES:
sh: find {{.SRC_DIR}} -name '*.md' -printf "%f\n"
deps:
- for:
var: FILES
split: "\n"
as: FILE
task: compile
vars:
FILE: '{{.FILE}}'
OUT_DIR: '{{.OUT_DIR}}'
OUT: '{{.OUT_DIR}}{{base .FILE | trimSuffix ".md"}}.html'
Since I have essays and notes folders, I copied this approach to Taskfile.yml files in both directories so that I can have distinct handling for the different files../bin/task public:essays worked fine, but ./bin/task public:notes would SIGSEGV.
Note that the label is different for essays and notes, the namespace is different, but the task is the same.
The root cause ended up being that both folders had index.md files. Deleting the latter file fixed the issue, changing the label did not. Renaming the file still created the error unless I modified the filetype. These files are in separate namespaces. I still don't really understand-- I would like for the fingerprinting to be based on the label or the namespace or the path. I'm still not quite sure what the workaround is yet.
EDIT: I'm still experiencing the issue so I was wrong about finding a fix. I think the state is getting tangled up somehow. I think using the same VARs in two separate dependencies tasks was causing issues. I switched them to cmds tasks and things are better, but something with the fingerprinting is still failing even if I add a clean step that finds all the .task folders and deletes them.
I'll have to dig into this more to figure out what's going wrong.
I did eventually find the note in the documentation about this but I don't think it should error this way and the logging could be improved.
https://taskfile.dev/docs/guide#by-fingerprinting-locally-generated-files-and-their-sources
Here is what the error looks like. The point that it happens is racey because the file order is different each time and there is no help from echo debugging or --verbose because I think the fingerprinting happens before all that.
(>10 successes)
...
task: [compile-note-200904-0627-change_youtube_speed.md] pandoc --lua-filter=fix-links.lua '200904-0627-change_youtube_speed.md' -o '../../output/public/html/notes/200904-0627-change_youtube_speed.html'
task: [compile-note-240904-1055-mildred.md] pandoc --lua-filter=fix-links.lua '240904-1055-mildred.md' -o '../../output/public/html/notes/240904-1055-mildred.html'
task: [compile-note-20230903-1359_AUTOMATED_way_station_clifford_simak_hugo_award.md] pandoc --lua-filter=fix-links.lua '20230903-1359_AUTOMATED_way_station_clifford_simak_hugo_award.md' -o '../../output/public/html/notes/20230903-1359_AUTOMATED_way_station_clifford_simak_hugo_award.html'
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x62671b]
goroutine 478 [running]:
mvdan.cc/sh/v3/expand.FieldsSeq.func1(0x131e5e200e40)
mvdan.cc/sh/v3@v3.12.1-0.20260124232039-e74afc18e65b/expand/expand.go:459 +0x9b
mvdan.cc/sh/v3/expand.Fields(0x131e5e0ce140, {0x131e5c282508, 0x1, 0x1})
mvdan.cc/sh/v3@v3.12.1-0.20260124232039-e74afc18e65b/expand/expand.go:442 +0x125
github.com/go-task/task/v3/internal/execext.ExpandFields({0x131e5e0aa1e0?, 0x39?})
github.com/go-task/task/v3@v3.49.1/internal/execext/exec.go:139 +0x18e
github.com/go-task/task/v3/internal/fingerprint.glob({0x131e5cf1a980?, 0x131e5d833570?}, {0x131e5c2d9080?, 0x18?})
github.com/go-task/task/v3@v3.49.1/internal/fingerprint/glob.go:29 +0x37
github.com/go-task/task/v3/internal/fingerprint.Globs({0x131e5cf1a980, 0x39}, {0x131e5c2823c0, 0x1, 0x0?})
github.com/go-task/task/v3@v3.49.1/internal/fingerprint/glob.go:15 +0xfb
github.com/go-task/task/v3/internal/fingerprint.(*ChecksumChecker).checksum(0x425d85?, 0x18?)
github.com/go-task/task/v3@v3.49.1/internal/fingerprint/sources_checksum.go:91 +0x45
github.com/go-task/task/v3/internal/fingerprint.(*ChecksumChecker).Value(0x0?, 0x131e5da93080?)
github.com/go-task/task/v3@v3.49.1/internal/fingerprint/sources_checksum.go:76 +0x13
github.com/go-task/task/v3.(*Executor).compiledTask(0x131e5c638f00, 0x131e5c6056e0, 0x0)
github.com/go-task/task/v3@v3.49.1/variables.go:193 +0xfd4
github.com/go-task/task/v3.(*Executor).FastCompiledTask(...)
github.com/go-task/task/v3@v3.49.1/variables.go:30
github.com/go-task/task/v3.(*Executor).RunTask(0x131e5c638f00, {0x209ae80, 0x3385860}, 0x131e5c6056e0)
github.com/go-task/task/v3@v3.49.1/task.go:142 +0x1a8
github.com/go-task/task/v3.(*Executor).runDeps.func1()
github.com/go-task/task/v3@v3.49.1/task.go:329 +0x9a
golang.org/x/sync/errgroup.(*Group).Go.func1()
golang.org/x/sync@v0.19.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
golang.org/x/sync@v0.19.0/errgroup/errgroup.go:78 +0x95
Version
3.49.1
Operating system
Linux ARCH 6.19.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 14 Mar 2026 01:07:43 +0000 x86_64 GNU/Linux
Experiments Enabled
No response
Example Taskfile
# yaml-language-server: $schema=https://taskfile.dev/schema.json
# copy this file into two separate directories at the same level, add them as includes, then execute as cmd tasks
version: '3'
vars:
OUT_DIR: ../../output/public/html/notes/
tasks:
default:
vars:
FILES:
sh: find . -name '*.md' -printf "%f\n"
deps:
- for:
var: FILES
split: "\n"
as: FILE
task: compile
vars:
FILE: '{{.FILE}}'
OUT_DIR: '{{.OUT_DIR}}'
compile:
internal: true
label: 'compile-{{.FILE}}'
requires:
vars: [FILE, OUT_DIR]
sources:
- '{{.FILE}}'
generates:
- '{{.OUT_DIR}}/{{base .FILE | trimSuffix ".md"}}.html'
cmds:
- >
pandoc --lua-filter=fix-links.lua
-o '{{.OUT_DIR}}/{{base .FILE | trimSuffix ".md"}}.html'
'{{.FILE}}'
Activity
[-]SIGSEGV in fingerprint if duplicate filenames in different directories[/-][+]SIGSEGV in fingerprint possibly due to duplicate filenames throughout different dirs and namespaces[/+]timrulebosch commentedon Mar 20, 2026
Is failing while expanding sources. You could try this to understand the glob pattern that causes the crash:
That should not crash, and may give a clue as to what is happening.
loganfred commentedon Mar 21, 2026
I updated the
public:essays:compileandpublic:notes:compilelike so:Looks like the
build-all -> deps -> FILES varis only run once, for thepublic:essays:compile, and not forpublic:notes:compileCould that be it? I don't feel like I have the right mental model of scoping.
Here is the output
And here are the task files used.
trulede commentedon Mar 22, 2026
@loganfred the
sourcesis causing the crash, so you need to capture that in the log (you commented it out). It should not crash.Its helpful to have a reliable way to reproduce the problem. So if we discover the exact trigger of that ... helpful.
I do similar things with Task, so this should work. The error itself indicates something in particular, but need to know the condition which triggers it to understand why.
loganfred commentedon Mar 22, 2026
I found some time to switch to a minimal example.
I think I'm seeing that explicit
dir:is not being honored for multipleincludes:As
deps:which dir is a race and ascmds:the first dir is taken for allI haven't worked up to
sources:yet. I would guess that the fingerprinting is all messed up each time I switch from the leaf node tasks (which work fine) to running in parallel (deps) or sequence (cmds).Root
Taskfile.yml:alpha/Taskfile.yml:beta/Taskfile.yml: (identical structure,MY_VAR: BETA)Expected behavior
Each included taskfile's
sh:var should execute relative to thedir:specified in itsincludes:block.Actual behavior
Sequential (
go-task all) — beta'ssh:command runs in alpha's directory:Parallel (
go-task all-parallel) — race condition, both get the same files:loganfred commentedon Mar 22, 2026
FWIW the bots suggest review in these areas:
taskfile/ask/taskfile.go:70-72may have a copy-paste error wheret1.varsare being merged intot1instead oft2.varscompiler.go:160may be caching dynamic vars by shell string, which may not permit re-using the same shell commands to create dynamic variables.taskfile/ask/tasks.go:179may not allow tasks that already got their vars from a nested include to keep themI have a patch I can PR if it would be helpful. It passes my minimal example, solves the real-world problem, and the test suite passes, but I'll be honest I'm not a go programmer it was 3% me and 97% claude. I just wanted a workaround to keep tinkering.
[-]SIGSEGV in fingerprint possibly due to duplicate filenames throughout different dirs and namespaces[/-][+]SIGSEGV in fingerprint; explicit dir include dynamic vars are not isolated and caching prohibits exact shell command re-use[/+]loganfred commentedon Mar 23, 2026
Ugh, I'm still getting the SIGSEGV actually, even with the patched binary that isolates the dynamic variables.
I'll try to learn how to use delve so I can see what's happening and report back.
trulede commentedon Mar 23, 2026
The 'dir' problem, possibly a clear bug. Just a bit surprised no one else has noticed.
The 'files' ... I would think, with the dir problem fixed, that you would not see any problem with the sequential task calls.
For the dep case, the tasks get compiled first. Could be a problem there if for some reason the templater cache was shared between them.
You could try to form the sources differently, and see if that prevents the problem. That would be a proof. For instance "{{.SRC_DIR}}" and "./{{.SRC_DIR}}".
trulede commentedon Mar 23, 2026
With this setup I don't observe the directory problem. Be careful with global vars (taskfile level) when importing, I suggest not doing that. Try hardcoding or task level vars.
This bit, move them to the task and see if it helps.
I will try the file listing a bit later.
loganfred commentedon Mar 23, 2026
I think it has to do with
include:since I ran the simpler scenario from the previous message on my machine and it worked as expected.I re-ran my minimalist example.
3.49.1+aa83651dis producing the same result with task level vars as with the global vars.Hard-coding does fix the
dir:withinclude:issue, I used../alphaand../betaas arguments tofindOutput from:
3.49.1+aa83651dversion with global vars (bug result)3.49.1+aa83651dversion with local vars (bug result)3.49.1+aa83651dversion../alphaand../betafor distinct paths (expected result)And the
Taskfile.ymlfiles3.49.1+aa83651dversion with global vars (bug result)Patched Version with global vars (expected result)
3.49.1+aa83651dversion with local vars (bug result)3.49.1+aa83651dversion../alphaand../betafor distinct paths (expected result)loganfred commentedon Mar 24, 2026
AHA, the segfault is unrelated.
Here is the summary from my side so far:
include:-ed tasks, each with their owndir:, those tasks aren't running from the correct folders.dir: {{.FOLDER}}may not workincluded:tasks that re-use the same exactsh:VAR(e.g.find . -name '*.md') in a differentdir:may get the cache value from another taskroot -> parent -> A / Bsituation with nested includes, the outer merge may be clobbering existing inner mergessh:VARScreate segfaults (e.g. pandoc chesterton's_fence.md)Shame on me for overlooking a stupid filename, but it probably shouldn't segfault
#2757 seems to fix 1, 2, 3, 4
Segfault minimal reproduction
As a workaround, it seems like you can hard-code it or do something lovecraftian like this:
trulede commentedon Mar 24, 2026
Template function
quoteorshellQuotemight also help. We would have to look at mvdan to understand exactly what is going on, the shell obviously expects something, so it needs escaping.This:
is evaluated before the task runs, so the
dirthe command runs in may not be what you expect. There is a issue open about that ... its not so easy to fix.There are issues for all of these things already. And PR's too in some cases.
trulede commentedon Apr 15, 2026
@loganfred Did you try quote or shellQuote? Quoting the file name correctly seems to be the root cause of your problem.
I'm not sure if task should quote/escape this automatically, perhaps your AI bot can find the place in the code that does escaping and write a PR there? There is a place ....
trulede commentedon Apr 25, 2026
@loganfred I may have also found a solution to the panic.
#2810
loganfred commentedon Apr 25, 2026
Yes, the fixing the shell quoting solved the panic, but I'm glad #2810 will handle it.
It seems like the other footguns I stumbled into are covered by other issues, so I'll update the title to be clearer and close this one.
Thanks for the assistance, I think the project is neat.
[-]SIGSEGV in fingerprint; explicit dir include dynamic vars are not isolated and caching prohibits exact shell command re-use[/-][+]SIGSEGV / panic in fingerprint from apostrophe in env var shell command[/+][-]SIGSEGV / panic in fingerprint from apostrophe in env var shell command[/-][+]SIGSEGV / panic in fingerprint from apostrophe in env var shell command filename[/+]