Repository navigation
Add documentation and sample DAG for ExasolToS3Operator - #73940
Conversation
Added documentation for Exasol to Amazon S3 data transfer.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
|
I would like to see it tested with real infrastructure. Example Dags/system tests are used as documentation and example by users so I think it would be better with real testing |
Agreed |
A dotted table name such as SCHEMA.TABLE is rejected by pyexasol's export_to_file as an unsafe identifier. Verified against Exasol 2026.1.2 with pyexasol 2.4.1 while running the example system test against real infrastructure; document that a SQL query should be used for schema-qualified tables.
System test executed against real infrastructure ✅@vincbeck @o-nikolas — following up on the review feedback: the example system test has now been run end-to-end against real infrastructure (no mocks). ResultRepeated 3 times, including a final run with the complete teardown path ( Environment
Content verificationTo prove the exported data actually lands in S3, I temporarily disabled the The CSV content matches the Exasol source table exactly. The bucket was deleted manually afterwards, and the final rerun with the restored DAG passed with full teardown (bucket lifecycle fully managed by the DAG). One finding, now documented in this PRWhile setting this up I hit a real usability issue: passing a schema-qualified table name as a string (
So for schema-qualified tables, users must pass a query. I added a docstring clarification to |
The new example_exasol_to_s3 system test now covers ExasolToS3Operator,
so keeping it in MISSING_EXAMPLES_FOR_CLASSES fails the
test_missing_examples check ("Operator listed in missing examples but
is used in example dag").
The new example_exasol_to_s3 system test now covers ExasolToS3Operator,
so keeping it in MISSING_EXAMPLES_FOR_CLASSES fails the
test_missing_examples check ("Operator listed in missing examples but
is used in example dag"). This also reverts the accidental unrelated
edits from the previous commit.
|
Note on the failed CI run (for whoever approves the next workflow run): |
|
CI is failing: |
The new docs page providers/amazon/docs/transfer/exasol_to_s3.rst was not listed in provider.yaml, failing the docs/provider.yaml synchronization check. Also reverts accidental content changes from the previous two commits.
|
@vincbeck Fixed — the new docs page - source-integration-name: Exasol
target-integration-name: Amazon Simple Storage Service (S3)
how-to-guide: /docs/apache-airflow-providers-amazon/transfer/exasol_to_s3.rst
python-module: airflow.providers.amazon.aws.transfers.exasol_to_s3The |
|
CI is still failing |
…ider_info - Replace pyexasol /master/ documentation URLs with the versionless root URL to satisfy check-for-inclusive-language - Add the how-to-guide entry for the Exasol to S3 transfer in get_provider_info.py as generated by the provider info sync hook
…ider_info - Replace pyexasol /master/ documentation URLs with the versionless root URL to satisfy check-for-inclusive-language - Add the how-to-guide entry for the Exasol to S3 transfer in get_provider_info.py as generated by the provider info sync hook
The previous push accidentally altered unrelated auto-generated config description strings in this file. This restores them byte-for-byte and keeps only the intended how-to-guide addition for exasol_to_s3.
|
OK now I have updated the branch <0.0> |
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |


Add a documentation page and an example (system test) DAG for the
ExasolToS3Operatortransfer, which previously had neither, plus a small docstring clarification in the operator itself discovered while testing against real infrastructure.What this PR contains
providers/amazon/docs/transfer/exasol_to_s3.rst— documentation page, following the same structure as the other transfer-to-S3 pages (e.g.mongo_to_s3.rst).providers/amazon/tests/system/amazon/aws/example_exasol_to_s3.py— example DAG following the standard Amazon provider system test pattern (create bucket → transfer → delete bucket, withwatcher).providers/amazon/src/airflow/providers/amazon/aws/transfers/exasol_to_s3.py— docstring clarification forquery_or_table: a table name must be a single plain identifier. Dotted names such asSCHEMA.TABLEare rejected bypyexasol'sexport_to_fileas unsafe identifiers, so schema-qualified tables must be exported via a SQL query (e.g.SELECT * FROM SCHEMA.TABLE). Verified empirically against Exasol 2026.1.2 / pyexasol 2.4.1: query string ✅,(schema, table)tuple ✅, dotted string ❌ExaExportError.Tested against real infrastructure ✅
As requested in the review, the system test was executed end-to-end against real infrastructure — not mocked:
exasol/docker-db:latest-8community edition (2026.1.2) in Docker, with aTEST.EMPLOYEEStable (3 rows)us-east-2(created and deleted by the DAG itself)1 passed— repeated 3 times, including a final run with the full teardown path1,Alice / 2,Bob / 3,Carol), matching the Exasol source tableEnvironment: Python 3.12.13, pyexasol 2.4.1,
--systempytest flag. Screenshots and details are in this comment.Note: the run above used
EXASOL_TABLE='SELECT * FROM TEST.EMPLOYEES'(query form) rather than the dotted table name, per the docstring clarification added in this PR.closes: #22632
Was generative AI tooling used to co-author this PR?
Generated-by: Kimi (AI coding assistant) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.