Skip to content

Improve dev/Dockerfile #1527

Closed
#2540
Closed
Improve dev/Dockerfile#1527
#2540

Description

@kevinjqliu
kevinjqliu
opened on Jan 16, 2025

Feature Request / Improvement

We had issues with downloading spark for the integration test docker image.

The issue comes from the need to download spark from an URL.

Since dev/Dockerfile is building a spark docker image with some extra configs.
I was wondering if we can use the dockerhub spark image as the base and configure it.

Activity

  1. kevinjqliu commented on Jan 17, 2025

    @kevinjqliu
    kevinjqliu
    on Jan 17, 2025
    ContributorAuthor

    possibly reuse for rust as well apache/iceberg-rust#894 (comment)

  2. github-actions commented on Jul 17, 2025

    @github-actions
    github-actionsbot
    on Jul 17, 2025 – with GitHub Actions

    This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

  3. github-actions
    added
    stale
    on Jul 17, 2025
  4. kevinjqliu
    removed
    stale
    on Jul 18, 2025

  5. jayceslesar commented on Sep 24, 2025

    @jayceslesar
    jayceslesar
    on Sep 24, 2025
    Contributor

    I think its as easy as doing a multi stage build where we first build from

    FROM apache/spark-py:v3.5.6 AS spark-dist

# stuff in between the existing start of the Dockerfile to COPY spark-defaults.conf /opt/spark/conf

COPY --from=spark-dist /opt/spark /opt/spark
COPY spark-defaults.conf /opt/spark/conf

# stuff after spark-defaults.conf /opt/spark/conf

    Would need some ARG/ENV handling so we dont break anything

  6. kevinjqliu commented on Sep 26, 2025

    @kevinjqliu
    kevinjqliu
    on Sep 26, 2025
    ContributorAuthor

    a couple progress on this

    We're on our way to use the default Spark docker image without any modifications :)

  7. Fokko
    added a commit that references this issue on Oct 2, 2025

    CI: Use Spark base image for Docker (#2540)

    5ee5eea
  8. Fokko
    closed this as completedin #2540on Oct 2, 2025
  9. Fokko
    added a commit that references this issue on Oct 3, 2025

    infra: group spark files into `dev/spark/` (#2563)

    3e46596
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions