Skip to content

Multi-stage Docker builds #37

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
l3x merged 2 commits into from
Feb 15, 2018
Merged

Multi-stage Docker builds #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
985ccae
Multi-stage builds, take two
mhluongo Feb 15, 2018
67417ee
Added mv keep-client /usr/local/bin command to Dockerfile's RUN command.
l3x Feb 15, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
37 changes: 26 additions & 11 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,26 @@ ENV GOPATH=/go \
APP_REPO_DIR=/go/src/keep-network/keep-client \
APP_NAME=keep-client

COPY ./go $APP_REPO_DIR

@Shadowfiend Shadowfiend Feb 15, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has to happen after the dep ensure or a *.go file change will rerun the whole apk + dep ensure flow.

WORKDIR $APP_REPO_DIR

RUN apk add --update --no-cache \
git && \
mkdir -p /go/src && \
rm -rf /var/cache/apk && mkdir /var/cache/apk && \
rm -rf /usr/share/man

RUN go get -u github.com/golang/dep/cmd/dep
COPY ./go/Gopkg.toml ./go/Gopkg.lock ./
RUN dep ensure --vendor-only

RUN CGO_ENABLED=1 GOOS=linux go build -a -installsuffix cgo -o $APP_NAME . && \
mv $APP_NAME /usr/local/bin/

@Shadowfiend Shadowfiend Feb 15, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. So this will build without bn because bn is dynamically linked? May want to investigate (not right now haha) how we might link it statically. Do we think it's worth filing an issue?

Out of curiosity, what's the benefit of doing this before we do all the updates below, vs after? It feels like the right flow organizationally (to me!) to handle dependencies from furthest away (non-Go) to closest (Go) and then conclude by building our own executable, rather than building the executable up top and pulling in additional dependencies further down, but it's entirely possible we gain something I missed :)

@mhluongo mhluongo Feb 15, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to get it statically linked- that way the client could work standalone. I spent a ton of time on it early on but couldn't figure it out.

It feels like the right flow organizationally (to me!) to handle dependencies from furthest away (non-Go) to closest (Go) and then conclude by building our own executable

This is where we're getting a benefit. This Dockerfile builds two images, and in the second we have way less stuff, but take the static build artifact from the first. If the library could be statically built and easily moved, we'd build the lib and client in the first and copy both to the new image.

Basically, takeaway- this pattern is two different containers with different base images, installs, etc

@Shadowfiend Shadowfiend Feb 15, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha! So basically if we built the bn image first, we'd need to pull a bunch of Go gunk in to do our Go build after, and then it'd be in the final image. Here, we do the Go stuff, create a basic Alpine image, pull in the stuff we need to compile bn + bn itself, build it, nuke the packages that we pulled in to build it, and pull in our static Go artifact. End result: we have the bn compile installed in the system, plus our static Go artifact, and nothing else.

@mhluongo mhluongo Feb 15, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly


FROM alpine:3.7

ENV BN_VERSION=d1a44d2f242692601b3e150b59044ab82f265b65

RUN apk add --update --no-cache \
bash \
clang \
Expand All @@ -18,24 +38,19 @@ RUN apk add --update --no-cache \
make \
openssl \
openssl-dev && \
git clone https://github.com/dfinity/bn /bn && \
mkdir -p /go/src && \
git clone https://github.com/dfinity/bn /bn && \
cd /bn && \
git reset --hard $BN_VERSION && \
make install && make && \
rm -rf /bn && \
mkdir -p /go/src && \
rm -rf /var/cache/apk && mkdir /var/cache/apk && \
rm -rf /usr/share/man && \
apk del git make clang llvm && \
mkdir -p $APP_REPO_DIR
apk del git make clang llvm

ENV BN_VERSION=d1a44d2f242692601b3e150b59044ab82f265b65
COPY --from=0 /usr/local/bin/keep-client /usr/local/bin/

COPY ./go $APP_REPO_DIR
WORKDIR $APP_REPO_DIR

RUN CGO_ENABLED=1 GOOS=linux go build -a -installsuffix cgo -o $APP_NAME . && \
mv $APP_NAME /usr/local/bin && \
rm -rf $APP_REPO_DIR
ENV LD_LIBRARY_PATH=/usr/local/lib/

ENTRYPOINT ["keep-client"]

Expand Down
28 changes: 10 additions & 18 deletions build-keep-client-docker-img.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,25 @@
#!/usr/bin/env bash
# Build vendor directory from latest source (assumes vendor directory is checked into github)
# Usage: build-keep-client-docker-img.sh [--force]
# Note: Only use the --force flag locally when testing. Your last test before pushing to github
# should be run w/o the --force flag to ensure your latest updates will work in production.
FORCE=$1
if [ "$(basename $(pwd))" != "keep-core" ] || [ ! -d ./go ]; then
echo "You should run $(basename $0) from the github.com/keep-net/keep-core directory"
if [ "$FORCE" != "--force" ]; then exit 2;fi
echo "WARNING: You should run $(basename $0) from the github.com/keep-net/keep-core directory"
fi

if [ $(git status | grep "On branch master" | wc | awk '{print $1}') == "0" ]; then
echo "WARNING: You are not on the master branch! Are you sure you want to continue? (CTRL+C to abort)"
read x
fi
if [ $(git status | grep "Your branch is up to date with 'origin/master'." | wc | awk '{print $1}') == "0" ]; then
echo "Your branch (local file system) is NOT up-to-date with the master branch."
if [ "$FORCE" != "--force" ]; then exit 2;fi
echo "WARNING: Your branch (local file system) is NOT up-to-date with the master branch."
fi
if [ $(git status | grep "nothing to commit, working tree clean" | wc | awk '{print $1}') == "0" ]; then
echo "Have you committed all of your changes?"
if [ "$FORCE" != "--force" ]; then exit 2;fi
echo "WARNING: Have you committed all of your changes?"
fi

# Go to source directory
cd go
# Build vendor from current source
dep ensure
# Back to the keep-core directory that has the Dockerfile.
cd ..
# Here's how to update the vendor directory:
## Go to source directory
#cd go
## Build vendor from current source
#dep ensure
## Back to the keep-core directory that has the Dockerfile.
#cd ..

IMG=keep-client
DOCKERFILE=Dockerfile
Expand Down
36 changes: 0 additions & 36 deletions go/vendor/github.com/dfinity/go-dfinity-crypto/.travis.yml
View file
Open in desktop

This file was deleted.

27 changes: 0 additions & 27 deletions go/vendor/github.com/dfinity/go-dfinity-crypto/README.md
View file
Open in desktop

This file was deleted.

Loading