diff --git a/flink-contrib/docker-flink/LICENSE b/flink-contrib/docker-flink/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..e06d2081865a766a8668acc12878f98b27fc9ea0 --- /dev/null +++ b/flink-contrib/docker-flink/LICENSE @@ -0,0 +1,202 @@ +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + diff --git a/flink-contrib/docker-flink/README.md b/flink-contrib/docker-flink/README.md new file mode 100644 index 0000000000000000000000000000000000000000..9dff93095555a616c0c905245ad88d2533abcd7f --- /dev/null +++ b/flink-contrib/docker-flink/README.md @@ -0,0 +1,80 @@ +#Apache Flink cluster deployment on Docker using Docker-Compose + +##Installation +###Install Docker + +https://docs.docker.com/installation/ + +if you have issues with Docker-Compose versions incompatible with your version of Docker try + +`curl -sSL https://get.docker.com/ubuntu/ | sudo sh` + +###Install Docker-Compose + +``` +curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose + +chmod +x /usr/local/bin/docker-compose +``` + +###Get the repo + +###Build the images + +Images are based on Ubuntu Trusty 14.04 and run Supervor to stay alive when running containers. + +The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config. + +- Run `./build.sh` + +###Deploy + +- Deploy cluster and see config/setup log output (best run in a screen session) + +`docker-compose up` + +- Deploy as a daemon (and return) + +`docker-compose up -d` + +- Scale the cluster up or down to *N* TaskManagers + +`docker-compose scale taskmanager=` + +- Access the JobManager node with SSH (exposed on Port 220) + +`ssh root@localhost -p 220` + +or on Mac OS X with boot2docker + +`ssh root@$(boot2docker ip) -p 220` + +The password is 'secret' + +- Kill the cluster + +`docker-compose kill` + +- Upload a jar to the cluster + +`scp -P 220 root@localhost:/` + +- Run a topology + +`ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c ` + +or + +ssh to the job manager and run the topology from there. + +###Ports + +- The Web Dashboard is on port `48080` +- The Web Client is on port `48081` +- JobManager RPC port `6123` (default, not exposed to host) +- TaskManagers RPC port `6121` (default, not exposed to host) +- TaskManagers Data port `6122` (default, not exposed to host) +- JobManager SSH `220` +- TaskManagers SSH: randomly assigned port, check wih `docker ps` + +Edit the `docker-compose.yml` file to edit port settings. diff --git a/flink-contrib/docker-flink/base/Dockerfile b/flink-contrib/docker-flink/base/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..67a295322f001f40d5b1af51413dc3dde6116ce2 --- /dev/null +++ b/flink-contrib/docker-flink/base/Dockerfile @@ -0,0 +1,49 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +FROM ubuntu:trusty + +#requirements +RUN apt-get update; apt-get install -y curl wget supervisor openssh-server openssh-client nano + +#priviledge separation directory +RUN mkdir /var/run/sshd + +#install Java 7 Oracle JDK +RUN mkdir -p /usr/java/default && \ + curl -Ls 'http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz' -H 'Cookie: oraclelicense=accept-securebackup-cookie' | \ + tar --strip-components=1 -xz -C /usr/java/default/ +ENV JAVA_HOME /usr/java/default/ + +#Install Java Open JDK +#RUN apt-get install -y unzip openjdk-7-jre-headless +#ENV JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/ + + +RUN echo 'root:secret' | chpasswd + +#SSH as root... probably needs to be revised for security! +RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config + +EXPOSE 22 + +# supervisor base configuration +ADD supervisor.conf /etc/supervisor/ + +# default command / not overridable CMD needed for supervisord +#CMD ["supervisord", "-c", "/etc/supervisor.conf"] \ No newline at end of file diff --git a/flink-contrib/docker-flink/base/supervisor.conf b/flink-contrib/docker-flink/base/supervisor.conf new file mode 100644 index 0000000000000000000000000000000000000000..ab152226e4bfd68402ac375991b15a4fb0dfb957 --- /dev/null +++ b/flink-contrib/docker-flink/base/supervisor.conf @@ -0,0 +1,21 @@ +; Licensed to the Apache Software Foundation (ASF) under one +; or more contributor license agreements. See the NOTICE file +; distributed with this work for additional information +; regarding copyright ownership. The ASF licenses this file +; to you under the Apache License, Version 2.0 (the +; "License"); you may not use this file except in compliance +; with the License. You may obtain a copy of the License at +; +; http://www.apache.org/licenses/LICENSE-2.0 +; +; Unless required by applicable law or agreed to in writing, software +; distributed under the License is distributed on an "AS IS" BASIS, +; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +; See the License for the specific language governing permissions and +; limitations under the License. + +[supervisord] +nodaemon=true + +[include] +files = /etc/supervisor/conf.d/*.conf diff --git a/flink-contrib/docker-flink/build.sh b/flink-contrib/docker-flink/build.sh new file mode 100755 index 0000000000000000000000000000000000000000..d732a9396863711e0fa3d43b31d41fbd9622f92b --- /dev/null +++ b/flink-contrib/docker-flink/build.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash + +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +#Kill cluster and remove all containers +docker-compose kill +#docker rm $(docker ps -a -q) + +#make sure the config file script is executable +chmod +x flink/config-flink.sh + +#rebuild images +docker build -t="base" base +docker build -t="flink" flink diff --git a/flink-contrib/docker-flink/docker-compose.yml b/flink-contrib/docker-flink/docker-compose.yml new file mode 100644 index 0000000000000000000000000000000000000000..8fee23b9b9a06f95c59184e888b4223f02d5226b --- /dev/null +++ b/flink-contrib/docker-flink/docker-compose.yml @@ -0,0 +1,37 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +jobmanager: + image: flink + ports: + - "48080:8080" + - "48081:8081" + - "220:22" + expose: + - "6123" + - "22" + command: /usr/local/flink/bin/config-flink.sh jobmanager +taskmanager: + image: flink + ports: + - "22" + expose: + - "6121" + - "6122" + links: + - jobmanager:jobmanager \ No newline at end of file diff --git a/flink-contrib/docker-flink/flink/Dockerfile b/flink-contrib/docker-flink/flink/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..8ed7a10b99db77d57bbeaac46bbce729eed7bcf0 --- /dev/null +++ b/flink-contrib/docker-flink/flink/Dockerfile @@ -0,0 +1,43 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +FROM base + +#add passless key to ssh +RUN ssh-keygen -f ~/.ssh/id_rsa -t rsa -N '' +RUN cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/* + +##Flink 0.8.1 Installation +###Download: +RUN mkdir ~/downloads && cd ~/downloads && \ + wget -q -O - http://mirrors.advancedhosters.com/apache/flink/flink-0.8.1/flink-0.8.1-bin-hadoop2.tgz | tar -zxvf - -C /usr/local/ +RUN cd /usr/local && ln -s ./flink-0.8.1 flink + +ENV FLINK_HOME /usr/local/flink +ENV PATH $PATH:$FLINK_HOME/bin + +#config files (template) +ADD conf/flink-conf.yaml /usr/local/flink/conf/ + +ADD config-flink.sh /usr/local/flink/bin/ +RUN chmod +x /usr/local/flink/bin/config-flink.sh + +EXPOSE 6123 +EXPOSE 22 + +CMD ["/usr/local/flink/bin/config-flink.sh", "taskmanager"] diff --git a/flink-contrib/docker-flink/flink/conf/flink-conf.yaml b/flink-contrib/docker-flink/flink/conf/flink-conf.yaml new file mode 100644 index 0000000000000000000000000000000000000000..e7136e5401a421d133dec3bc4770842f8a02f04f --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/flink-conf.yaml @@ -0,0 +1,37 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +jobmanager.rpc.address: %jobmanager% +jobmanager.rpc.port: 6123 +jobmanager.heap.mb: 128 + +taskmanager.rpc.port: 6121 +taskmanager.data.port: 6122 +taskmanager.heap.mb: 256 +taskmanager.numberOfTaskSlots: %nb_slots% + +parallelization.degree.default: %parallelism% + +env.java.home: /usr/java/default + +#============================================================================== +# Web Frontend +#============================================================================== + +jobmanager.web.port: 8080 +webclient.port: 8081 \ No newline at end of file diff --git a/flink-contrib/docker-flink/flink/conf/log4j.properties b/flink-contrib/docker-flink/flink/conf/log4j.properties new file mode 100644 index 0000000000000000000000000000000000000000..172f566a99682bfec795ec1e6609978001adee0d --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/log4j.properties @@ -0,0 +1,26 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +log4j.rootLogger=INFO, file + +# Log all infos in the given file +log4j.appender.file=org.apache.log4j.FileAppender +log4j.appender.file.file=${log.file} +log4j.appender.file.append=false +log4j.appender.file.layout=org.apache.log4j.PatternLayout +log4j.appender.file.layout.ConversionPattern=%d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n \ No newline at end of file diff --git a/flink-contrib/docker-flink/flink/conf/logback-yarn.xml b/flink-contrib/docker-flink/flink/conf/logback-yarn.xml new file mode 100644 index 0000000000000000000000000000000000000000..c5511b3b151db53f186762038f9c087295ff6302 --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/logback-yarn.xml @@ -0,0 +1,39 @@ + + + + + ${log.file} + false + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n + + + + + + %d{HH:mm:ss} %-5level %logger{60} %X{sourceThread} - %msg%n + + + + + + + + + \ No newline at end of file diff --git a/flink-contrib/docker-flink/flink/conf/logback.xml b/flink-contrib/docker-flink/flink/conf/logback.xml new file mode 100644 index 0000000000000000000000000000000000000000..9305b115cd14415ad271718ff6e1632b296f0123 --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/logback.xml @@ -0,0 +1,31 @@ + + + + + ${log.file} + false + + %d{HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n + + + + + + + \ No newline at end of file diff --git a/flink-contrib/docker-flink/flink/conf/slaves b/flink-contrib/docker-flink/flink/conf/slaves new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/flink-contrib/docker-flink/flink/config-flink.sh b/flink-contrib/docker-flink/flink/config-flink.sh new file mode 100755 index 0000000000000000000000000000000000000000..72f28287f59ec3e0910f3df3b2d271fe28311c03 --- /dev/null +++ b/flink-contrib/docker-flink/flink/config-flink.sh @@ -0,0 +1,55 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +#!/bin/bash + +CONF=/usr/local/flink/conf +EXEC=/usr/local/flink/bin + +#set nb_slots = nb CPUs +#let "nbslots=$2 * `nproc`" +sed -i -e "s/%nb_slots%/`nproc`/g" $CONF/flink-conf.yaml + +#set parallelism +sed -i -e "s/%parallelism%/1/g" $CONF/flink-conf.yaml + +if [ "$1" = "jobmanager" ]; then + echo "Configuring Job Manager on this node" + sed -i -e "s/%jobmanager%/`hostname -i`/g" $CONF/flink-conf.yaml + $EXEC/jobmanager.sh start cluster + $EXEC/start-webclient.sh + +elif [ "$1" = "taskmanager" ]; then + echo "Configuring Task Manager on this node" + sed -i -e "s/%jobmanager%/$JOBMANAGER_PORT_6123_TCP_ADDR/g" $CONF/flink-conf.yaml + $EXEC/taskmanager.sh start +fi + +#print out config - debug +echo "config file: " && cat $CONF/flink-conf.yaml + +#add ENV variable to shell for ssh login +echo "export JAVA_HOME=/usr/java/default;" >> ~/.profile +echo "export PATH=$PATH:$JAVA_HOME/bin;" >> ~/.profile +echo "export F=/usr/local/flink/;" >> ~/.profile +#Uncomment for SSH connection between nodes without prompts +#echo 'export FLINK_SSH_OPTS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"' >> ~/.profile + +#run ssh server and supervisor to keep container running. +/usr/sbin/sshd && supervisord -c /etc/supervisor/supervisor.conf \ No newline at end of file diff --git a/pom.xml b/pom.xml index d534eee446c6918b16b2c7395752ba17eb333aad..ec063cff66ce95b3571c0150cb0cde74e6bec6de 100644 --- a/pom.xml +++ b/pom.xml @@ -710,6 +710,7 @@ under the License. out/test/flink-avro/avro/user.avsc **/flink-bin/conf/slaves + flink-contrib/docker-flink/flink/conf/slaves **/README.md CHANGELOG