HIVE-29419: Provide a Hive-specific docker image for Tez AM #6435
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -88,4 +88,25 @@ | |
| <name>hive.query.results.cache.directory</name> | ||
| <value>${HIVE_QUERY_RESULTS_CACHE_DIRECTORY}</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.server2.tez.initialize.default.sessions</name> | ||
| <value>false</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.server2.tez.use.external.sessions</name> | ||
| <value>${HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS}</value> | ||
| </property> | ||
| <!-- | ||
| A registry namespace prefix is a hardcoded prefix for Tez external sessions. | ||
| The actual tez.am.registry.namespace value is appended to this prefix. | ||
| Once hive can use the registry client that Tez provides (ZkAMRegistryClient), this property will be removed. | ||
|
Member
There was a problem hiding this comment. is there a ticket for that? needs Tez upgrade?
Contributor
Author
There was a problem hiding this comment. for the release we already have: https://issues.apache.org/jira/browse/TEZ-4701 |
||
| --> | ||
| <property> | ||
| <name>hive.server2.tez.external.sessions.namespace</name> | ||
| <value>/tez-external-sessions${TEZ_AM_REGISTRY_NAMESPACE}</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.server2.tez.external.sessions.registry.class</name> | ||
| <value>org.apache.hadoop.hive.ql.exec.tez.ZookeeperExternalSessionsRegistryClient</value> | ||
| </property> | ||
| </configuration> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "taskSchedulerDescriptors": [ | ||
| { | ||
| "className": "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService", | ||
| "entityName": "LLAP" | ||
| } | ||
| ], | ||
| "containerLauncherDescriptors": [ | ||
| { | ||
| "className": "org.apache.hadoop.hive.llap.tezplugins.LlapContainerLauncher", | ||
| "entityName": "LLAP" | ||
| } | ||
| ], | ||
| "taskCommunicatorDescriptors": [ | ||
| { | ||
| "className": "org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator", | ||
| "entityName": "LLAP" | ||
| } | ||
| ], | ||
| "enableContainers": false, | ||
| "enableUber": false | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| log4j.rootLogger=INFO, console | ||
|
Member
There was a problem hiding this comment. maybe, simply |
||
| log4j.appender.console=org.apache.log4j.ConsoleAppender | ||
| log4j.appender.console.Target=System.err | ||
| log4j.appender.console.layout=org.apache.log4j.PatternLayout | ||
| log4j.appender.console.layout.ConversionPattern=%d{ISO8601} %-5p %c{1} - %m%n | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| <?xml version="1.0"?> | ||
| <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --> | ||
| <configuration> | ||
| <property> | ||
| <name>tez.am.mode.session</name> | ||
| <value>true</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.framework.mode</name> | ||
| <value>${TEZ_FRAMEWORK_MODE}</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.zookeeper.quorum</name> | ||
| <value>${TEZ_AM_ZOOKEEPER_QUORUM}</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.registry.namespace</name> | ||
| <value>${TEZ_AM_REGISTRY_NAMESPACE}</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.local.mode</name> | ||
| <value>false</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.tez-ui.webservice.enable</name> | ||
| <value>false</value> | ||
| </property> | ||
| <!-- Tez AM should not timeout in ZK Mode --> | ||
| <property> | ||
| <name>tez.session.am.dag.submit.timeout.secs</name> | ||
| <value>-1</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.ignore.lib.uris</name> | ||
| <value>true</value> | ||
| </property> | ||
| <property> | ||
| <name>tez.am.disable.client-version-check</name> | ||
| <value>true</value> | ||
| </property> | ||
|
|
||
| <!-- | ||
| In standalone AM mode with LLAP, the task scheduler (LlapTaskSchedulerService) and task | ||
| communicator (LlapTaskCommunicator) are instantiated at AM startup from service_plugins_descriptor.json | ||
| on the classpath — not from the DAG payload as in YARN-submitted mode. These plugins read their | ||
| configuration from TezConfiguration, which is built exclusively from tez-site.xml on the classpath. | ||
| Hive-specific properties must therefore appear here so the AM can find LLAP daemons and connect | ||
| to them, even though they are conceptually Hive settings. | ||
| --> | ||
| <property> | ||
| <name>hive.zookeeper.quorum</name> | ||
| <value>${HIVE_ZOOKEEPER_QUORUM}</value> | ||
| </property> | ||
| <property> | ||
| <name>hive.llap.daemon.service.hosts</name> | ||
| <value>${HIVE_LLAP_DAEMON_SERVICE_HOSTS}</value> | ||
| </property> | ||
| </configuration> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -132,14 +132,35 @@ services: | |
| - zookeeper_datalog:/datalog | ||
| - zookeeper_logs:/logs | ||
|
|
||
| tezam: | ||
| profiles: | ||
| - llap | ||
|
Member
There was a problem hiding this comment. should we add standalone support later? without llap daemons -> Tez containers?
Contributor
Author
There was a problem hiding this comment. what would this standalone mode look like? you mean: hs2 + tezam + tezchild (classic single task executor)?
Member
There was a problem hiding this comment. yea, something like that. not useful?
Contributor
Author
There was a problem hiding this comment. sorry, forgot to respond: short term, it's not worth the effort from Hive project's point of view, given that we adapted at least one of our main execution modes (LLAP vs. Tez container mode), LLAP, to cloud, but long term, it's a good-to-have and not only for Hive, but Tez also, that's what TEZ-4662 is about |
||
| image: apache/hive:${HIVE_VERSION} | ||
|
Member
There was a problem hiding this comment. is it Ok to reuse hive image for TezAM? IDK just asking. is is the same downstream?
Contributor
Author
There was a problem hiding this comment. I think it's ok to reuse, actually, we build both (Hive specific TezAM image + Tez AM image in Tez project), see the motivation on HIVE-29419 |
||
| container_name: tezam | ||
| hostname: tezam | ||
| depends_on: | ||
| - zookeeper | ||
| restart: on-failure:3 | ||
| environment: | ||
| USER: hive | ||
| SERVICE_NAME: 'tezam' | ||
|
|
||
| TEZ_FRAMEWORK_MODE: STANDALONE_ZOOKEEPER | ||
| TEZ_AM_ZOOKEEPER_QUORUM: zookeeper:2181 | ||
|
|
||
| # LLAP daemon discovery | ||
| HIVE_ZOOKEEPER_QUORUM: zookeeper:2181 | ||
| HIVE_LLAP_DAEMON_SERVICE_HOSTS: '@llap0' | ||
|
|
||
| # Directories shared between HiveServer2 and LLAP daemon | ||
| HIVE_SCRATCH_DIR: /opt/hive/scratch | ||
| HIVE_QUERY_RESULTS_CACHE_DIRECTORY: /opt/hive/scratch/_resultscache_ | ||
|
|
||
| volumes: | ||
| - warehouse:/opt/hive/data/warehouse | ||
| - scratch:/opt/hive/scratch | ||
| networks: | ||
| - hive | ||
|
|
||
| llapdaemon: | ||
| profiles: | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why not initialize ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's a yarn-based thing that HS2 initializes a few sessions on startup
in the standalone mode, HS2 has no control over them, just discovers them via ZK