To Makefile or not to Makefile, that is the question

[Shadowfiend]

Pulling this discussion into here so we don't get too far off topic in @rargulati's PR.

Quick breakdown from #26 :

@mhluongo:

Am I the only one who wants to use the Docker build in place of eg make? It's already standardized and having two build systems seems strange.

@pschlump:

I have used both. IMHO it is much harder to get docker do do simple
tasks. Before we are done we should create a Docker base build because it
captures all the dependencies and is fully reproducible.

@mhluongo:

Agreed that make is simpler, but we already have a Dockerfile and are using it to do CI and (eventually) reproducible builds- why would we maintain two? It'd lead to some people updating the Makefile and others updating the Dockerfile, I expect.

@Shadowfiend:

I don't consider docker build a build system for our application, I consider it the bit that sets up our docker image. The Dockerfile should just run make IMO.

I see make as answering “how do we build our project”.

I see docker as answering one variation of “how do we distribute our project”.

@mhluongo:

... I get the appeal, I really do. But I've seen this show before. The Dockerfile needs to understand as much as a Makefile to properly cache, and we already have all make-like functionality we need in CircleCI. Using both will lead to a dual build system.

🙏 spend a little time cozying up with Docker and CircleCI before we pull the trigger on adding make to the mix.

@Shadowfiend:

to properly cache

Can you explain some more about this/point to relevant docs? Not sure I follow.

@keep-network/go

[Shadowfiend]

Put together some prior breadcrumbs and found this bit on multi-stage builds, which I'm guessing is what the caching bit refers to.

Core concern multi-stage builds help with is reducing image size, but doing that requires pulling in dependencies, executing build steps in stages, and ultimately creating a final image. Docker can cache along the way, but to do that correctly you have to be copying the bits you want to continue working on and leaving behind the bits you want to be able to cache in previous stages, and thus reproduce some or all of the dependency logic that ends up in the Makefile.

Is that a correct understanding of your concern @mhluongo ?

What it comes down to on my end is, I don't love requiring docker to build. Docker isn't a build system, and you can recognize it by, for example, the fact that you can't parallelize your build tasks. It feels like we should be able to abstract stuff into make targets that each stage of the docker build can execute… Similarly, into targets that circle CI can execute. And simply declare “docker and CI should only be running make or docker commands” (which should be fairly easy to enforce during code review).

That said, I'm very conscious of the note “I've seen this show before”---I don't want to repeat mistakes you've already seen. If you think those mistakes are inherent to the approach of trying to keep build concerns in make and deploy concerns in docker, I'm willing to defer (though would also love to hear more details for my own edification). I'm wondering if they are not inherent, but rather if it's something that can be done well or poorly, and we just need to carefully navigate the “well” path.

[rargulati]

Here's what Shantanu and I did for our YC app that let us move quickly - we used both make and docker compose and things just worked (tm): https://gist.github.com/rargulati/6d0933beb5746e86536142e90fb20d1c

The above was going to call out to terraform to provision our application and we were looking at Mesos/kubes instead of docker-compose, but all from a simple make XXX command.

Of course, I haven't done any of this at scale, and I don't know if this will break down at scale.

[Shadowfiend]

I read a bit more yesterday, and it looks like docker also caches any time you do a COPY command. Going to detail that out for myself and anyone reading who didn't know this…

So our current Dockerfile, for example, does one copy, moving the source code into the image. Before this, it has fetched dependencies and cloned and installed the bn library. That means we won't rebuild the bn library unless you force docker to start from scratch: any further docker build will kick off with the COPY ./go/ ./ line, pulling the sources into the image.

We can actually take this one step further, by doing a COPY of just the Gopkg.toml and Gopkg.lock files, running dep ensure, and then doing the COPY ./go/ ./. That would add a second cached image after the dep ensure, so that we only rerun dep ensure if our dependencies change.

So on and so forth, the point is if we want to fully leverage this, we encode a partial version of our dependencies in the Dockerfile so that docker can build as little of the image as possible along the way. Even if we use a Makefile, we would be explicitly reproducing some of the dependency info that the Makefile contains in the Dockerfile, so as to minimize image build times.

Is that a more correct description of what you're worried about, @mhluongo ?

[mhluongo]

Yep, much closer @Shadowfiend, though note it's more than just COPY. Big idea- let's do the docker part right and make sure everyone's got some expertise and knows what goes where before introducing something like make, especially since the default dev behavior will be toward a complex Makefile and oversimplified Dockerfile, impacting build performance

[Shadowfiend]

The docs I could find on Docker build caching are in the best practices document and describe the other moving parts of caching that @mhluongo references above.

---

The tradeoff I see here is relatively simple, and as long as we commit to it I'm down either way: by using the Dockerfile exclusively, we give up the ability to have one command that can produce a native build for your current platform. That is, if we used make, locally I can run make and have a macOS binary. In this sense, if we use the Dockerfile exclusively, we commit to docker as our only supported distribution strategy. This circles back to:

I see docker as answering one variation of “how do we distribute our project”.

If we want more than one variation, we cannot only use Dockerfile to drive the builds. If we are saying we want to shuck native builds for now and circle back to revisit the decision later, we can.

---

Quick assessment on tradeoffs from my perspective: if we want someone to be able to run a DO droplet with our bare-metal client, we need native builds. If we want folks to be able to run our client on their desktop, we need native builds. If we want folks to be able to develop locally without docker, we need native builds.

If, on the other hand, we envision development requiring docker, and are okay with deployment also requiring docker, at least for now, then we can stick to Dockerfile as our build and distribution strategy.

[l3x]

I used some of the Docker related work while Dockerizing the local group relay swarm ... to suggest an improvement to our build process. Please see #36

I included a detailed write up in the comments of that PR.

[mhluongo]

Getting out of the way here- basically I think a Makefile is fine, as long as we go Docker-first and make sure the Dockerfile works without make. If things get out of sync... well, things can't get out of sync.

[Shadowfiend]

Ok, for now we'll duplicate work a little and have make and docker do the same fundamental things as needed.

For the most part I expect it'll be simple. We'll need to stick the go generate stuff in both, and maybe add a check in the Makefile to at least warn you if you're trying to roll without mcl and bn built/available as libs. Other than that I think we're leaning almost entirely on go build.

Closing this issue for now, we'll figure out the best way to use the Makefile as we go and keep the principle of Dockerfile not depending on make at the fore.