Skip to content

Add documentation and sample DAG for ExasolToS3Operator - #73940

Merged
vincbeck merged 13 commits into
apache:mainfrom
Mola-maker:docs-exasol-to-s3
Oct 8, 2026
Merged

vincbeck merged 13 commits into
apache:mainfrom
Mola-maker:docs-exasol-to-s3

Conversation

@Mola-maker

@Mola-maker Mola-maker commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Add a documentation page and an example (system test) DAG for the ExasolToS3Operator transfer, which previously had neither, plus a small docstring clarification in the operator itself discovered while testing against real infrastructure.

What this PR contains

  1. providers/amazon/docs/transfer/exasol_to_s3.rst — documentation page, following the same structure as the other transfer-to-S3 pages (e.g. mongo_to_s3.rst).
  2. providers/amazon/tests/system/amazon/aws/example_exasol_to_s3.py — example DAG following the standard Amazon provider system test pattern (create bucket → transfer → delete bucket, with watcher).
  3. providers/amazon/src/airflow/providers/amazon/aws/transfers/exasol_to_s3.py — docstring clarification for query_or_table: a table name must be a single plain identifier. Dotted names such as SCHEMA.TABLE are rejected by pyexasol's export_to_file as unsafe identifiers, so schema-qualified tables must be exported via a SQL query (e.g. SELECT * FROM SCHEMA.TABLE). Verified empirically against Exasol 2026.1.2 / pyexasol 2.4.1: query string ✅, (schema, table) tuple ✅, dotted string ❌ ExaExportError.

Tested against real infrastructure ✅

As requested in the review, the system test was executed end-to-end against real infrastructure — not mocked:

  • Exasol: exasol/docker-db:latest-8 community edition (2026.1.2) in Docker, with a TEST.EMPLOYEES table (3 rows)
  • AWS: real S3 bucket in us-east-2 (created and deleted by the DAG itself)
  • Result: 1 passed — repeated 3 times, including a final run with the full teardown path
  • Content verification: with the teardown task temporarily disabled, the exported object was downloaded from S3 and contained exactly the 3 expected rows (1,Alice / 2,Bob / 3,Carol), matching the Exasol source table

Environment: Python 3.12.13, pyexasol 2.4.1, --system pytest flag. Screenshots and details are in this comment.

Note: the run above used EXASOL_TABLE='SELECT * FROM TEST.EMPLOYEES' (query form) rather than the dotted table name, per the docstring clarification added in this PR.

closes: #22632


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below): Kimi (AI coding assistant)

Generated-by: Kimi (AI coding assistant) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg

boring-cyborg Bot commented Sep 30, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@vincbeck

Copy link
Copy Markdown
Contributor

I would like to see it tested with real infrastructure. Example Dags/system tests are used as documentation and example by users so I think it would be better with real testing

@o-nikolas

Copy link
Copy Markdown
Contributor

I would like to see it tested with real infrastructure. Example Dags/system tests are used as documentation and example by users so I think it would be better with real testing

Agreed

A dotted table name such as SCHEMA.TABLE is rejected by pyexasol's
export_to_file as an unsafe identifier. Verified against Exasol
2026.1.2 with pyexasol 2.4.1 while running the example system test
against real infrastructure; document that a SQL query should be
used for schema-qualified tables.
@Mola-maker

Copy link
Copy Markdown
Contributor Author

System test executed against real infrastructure ✅

@vincbeck @o-nikolas — following up on the review feedback: the example system test has now been run end-to-end against real infrastructure (no mocks).

Result

providers/amazon/tests/system/amazon/aws/example_exasol_to_s3.py::test_run PASSED [100%]
======================== 1 passed, 1 warning in 13.28s ========================

Repeated 3 times, including a final run with the complete teardown path (delete_s3_bucket with TriggerRule.ALL_DONE).

system test passed

Environment

Component Version / Detail
Exasol exasol/docker-db:latest-8 community edition, 2026.1.2 (Docker, --privileged)
pyexasol 2.4.1
Python 3.12.13
Airflow this PR branch (docs-exasol-to-s3)
AWS real S3 bucket in us-east-2 (created + deleted by the DAG)
EXASOL_TABLE SELECT * FROM TEST.EMPLOYEES
Test data TEST.EMPLOYEES(id, name) — 3 rows

Content verification

To prove the exported data actually lands in S3, I temporarily disabled the delete_s3_bucket teardown task, re-ran the test (passed again), and downloaded the object:

$ aws s3 ls s3://pr73940-exasol-to-s3-bucket/
2026-10-02 02:21:55         22 pr73940-exasol-to-s3-key

$ aws s3 cp s3://pr73940-exasol-to-s3-bucket/pr73940-exasol-to-s3-key -
1,Alice
2,Bob
3,Carol

s3 content verification

The CSV content matches the Exasol source table exactly. The bucket was deleted manually afterwards, and the final rerun with the restored DAG passed with full teardown (bucket lifecycle fully managed by the DAG).

One finding, now documented in this PR

While setting this up I hit a real usability issue: passing a schema-qualified table name as a string (query_or_table="TEST.EMPLOYEES") fails — pyexasol's export_to_file rejects dotted identifiers as unsafe (ExaExportError). Verified all three forms against Exasol 2026.1.2 / pyexasol 2.4.1:

  • "SELECT * FROM TEST.EMPLOYEES" (query) — ✅ works
  • ("TEST", "EMPLOYEES") (tuple) — ✅ works (but not expressible via the operator's str parameter)
  • "TEST.EMPLOYEES" (dotted string) — ❌ ExaExportError

So for schema-qualified tables, users must pass a query. I added a docstring clarification to ExasolToS3Operator in commit 8634fae, and the system test was run with the query form accordingly. This is exactly the kind of thing that only surfaces when running against a real instance — thanks for pushing for a real test run.

@vincbeck vincbeck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

The new example_exasol_to_s3 system test now covers ExasolToS3Operator,
so keeping it in MISSING_EXAMPLES_FOR_CLASSES fails the
test_missing_examples check ("Operator listed in missing examples but
is used in example dag").
The new example_exasol_to_s3 system test now covers ExasolToS3Operator,
so keeping it in MISSING_EXAMPLES_FOR_CLASSES fails the
test_missing_examples check ("Operator listed in missing examples but
is used in example dag"). This also reverts the accidental unrelated
edits from the previous commit.
@Mola-maker

Copy link
Copy Markdown
Contributor Author

Note on the failed CI run (for whoever approves the next workflow run): Non-DB tests: core / Non-DB-core::3.10:Always failed in test_missing_examples (airflow-core/tests/unit/always/test_project_structure.py) — now that example_exasol_to_s3.py covers ExasolToS3Operator, the operator had to be removed from MISSING_EXAMPLES_FOR_CLASSES ("Operator listed in missing examples but is used in example dag"). Fixed in 114cb8c; the current diff only touches the two new files, the operator docstring clarification, and that exemption entry.

@vincbeck

vincbeck commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

CI is failing:

suspended_providers  set()
    Found 1 error in providers
    Error: Discrepancies between documentation/logos for providers and provider.yaml files How to fix it: Please synchronize the docs/logs.
       Left set:Document urls found in airflow/docs
       Right set:Document urls configured in provider.yaml files
          Items in the left set but not the right:
             '/docs/apache-airflow-providers-amazon/transfer/exasol_to_s3.rst'

    Error 1 returned

The new docs page providers/amazon/docs/transfer/exasol_to_s3.rst was
not listed in provider.yaml, failing the docs/provider.yaml
synchronization check. Also reverts accidental content changes from
the previous two commits.
@Mola-maker

Copy link
Copy Markdown
Contributor Author

@vincbeck Fixed — the new docs page transfer/exasol_to_s3.rst wasn't registered in the Amazon provider.yaml transfers section, which tripped the docs/provider.yaml synchronization check. Added the missing entry:

  - source-integration-name: Exasol
    target-integration-name: Amazon Simple Storage Service (S3)
    how-to-guide: /docs/apache-airflow-providers-amazon/transfer/exasol_to_s3.rst
    python-module: airflow.providers.amazon.aws.transfers.exasol_to_s3

The provider.yaml diff is now exactly that one added line; nothing else in the file changed.

@Mola-maker
Mola-maker requested a review from vincbeck October 7, 2026 14:05
@vincbeck

vincbeck commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

CI is still failing

…ider_info

- Replace pyexasol /master/ documentation URLs with the versionless
  root URL to satisfy check-for-inclusive-language
- Add the how-to-guide entry for the Exasol to S3 transfer in
  get_provider_info.py as generated by the provider info sync hook
…ider_info

- Replace pyexasol /master/ documentation URLs with the versionless
  root URL to satisfy check-for-inclusive-language
- Add the how-to-guide entry for the Exasol to S3 transfer in
  get_provider_info.py as generated by the provider info sync hook
The previous push accidentally altered unrelated auto-generated config
description strings in this file. This restores them byte-for-byte and
keeps only the intended how-to-guide addition for exasol_to_s3.
@Mola-maker

Copy link
Copy Markdown
Contributor Author

OK now I have updated the branch <0.0>

@vincbeck
vincbeck merged commit 42b25f5 into apache:main Oct 8, 2026
85 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 8, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing doc and sample dag for ExasolToS3Operator

3 participants