fix according to code review and fix broken examples

docs/content/Configuration.md

@@ -15,7 +15,7 @@ There are three JVM parameters that we set on all of our processes:
 Modules
 =======
 
-As of Druid v0.6.2, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules.
+As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules.
 
 Configuration of the various modules is done via Java properties. These can either be provided as `-D` system properties on the java command line or they can be passed in via a file called `runtime.properties` that exists on the classpath.
 

docs/content/Tutorial:-The-Druid-Cluster.md

@@ -202,7 +202,7 @@ Usually, when new segments are created, these MySQL entries are created directly
 
 ``` sql
 use druid;
-INSERT INTO segments (id, dataSource, created_date, start, end, partitioned, version, used, payload) VALUES ('wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z', 'wikipedia', '2013-08-08T21:26:23.799Z', '2013-08-01T00:00:00.000Z', '2013-08-02T00:00:00.000Z', '0', '2013-08-08T21:22:48.989Z', '1', '{\"dataSource\":\"wikipedia\",\"interval\":\"2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z\",\"version\":\"2013-08-08T21:22:48.989Z\",\"loadSpec\":{\"type\":\"s3_zip\",\"bucket\":\"static.druid.io\",\"key\":\"data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip\"},\"dimensions\":\"dma_code,continent_code,geo,area_code,robot,country_name,network,city,namespace,anonymous,unpatrolled,page,postal_code,language,newpage,user,region_lookup\",\"metrics\":\"count,delta,variation,added,deleted\",\"shardSpec\":{\"type\":\"none\"},\"binaryVersion\":9,\"size\":24664730,\"identifier\":\"wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z\"}');
+INSERT INTO druid_segments (id, dataSource, created_date, start, end, partitioned, version, used, payload) VALUES ('wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z', 'wikipedia', '2013-08-08T21:26:23.799Z', '2013-08-01T00:00:00.000Z', '2013-08-02T00:00:00.000Z', '0', '2013-08-08T21:22:48.989Z', '1', '{\"dataSource\":\"wikipedia\",\"interval\":\"2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z\",\"version\":\"2013-08-08T21:22:48.989Z\",\"loadSpec\":{\"type\":\"s3_zip\",\"bucket\":\"static.druid.io\",\"key\":\"data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip\"},\"dimensions\":\"dma_code,continent_code,geo,area_code,robot,country_name,network,city,namespace,anonymous,unpatrolled,page,postal_code,language,newpage,user,region_lookup\",\"metrics\":\"count,delta,variation,added,deleted\",\"shardSpec\":{\"type\":\"none\"},\"binaryVersion\":9,\"size\":24664730,\"identifier\":\"wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z\"}');
 ```
 
 If you look in your coordinator node logs, you should, after a maximum of a minute or so, see logs of the following form:
@@ -237,6 +237,8 @@ druid.port=8083
 
 druid.zk.service.host=localhost
 
+druid.extensions.coordinates=["io.druid.extensions:druid-examples:0.6.2-SNAPSHOT"]
+
 druid.db.connector.connectURI=jdbc\:mysql\://localhost\:3306/druid
 druid.db.connector.user=druid
 druid.db.connector.password=diurd

docs/content/Twitter-Tutorial.textile

@@ -61,12 +61,12 @@ Once the node starts up you will see a bunch of logs about setting up properties
 <pre><code>2013-05-17 23:04:40,934 INFO [main] org.mortbay.log - Started SelectChannelConnector@0.0.0.0:8080
 2013-05-17 23:04:40,935 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.druid.http.FileRequestLogger.start()] on object[com.metamx.druid.http.FileRequestLogger@42bb0406].
 2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - Connection established.
-2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] druid.examples.twitter.TwitterSpritzerFirehoseFactory - Connected_to_Twitter
+2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] io.druid.examples.twitter.TwitterSpritzerFirehoseFactory - Connected_to_Twitter
 2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - Receiving status stream.
 </code></pre>
 
 Periodically, you'll also see messages of the form:
-<pre><code>2013-05-17 23:04:59,793 INFO [chief-twitterstream] druid.examples.twitter.TwitterSpritzerFirehoseFactory - nextRow() has returned 1,000 InputRows
+<pre><code>2013-05-17 23:04:59,793 INFO [chief-twitterstream] io.druid.examples.twitter.TwitterSpritzerFirehoseFactory - nextRow() has returned 1,000 InputRows
 </code></pre>
 
 These messages indicate you are ingesting events. The Druid real time-node ingests events in an in-memory buffer. Periodically, these events will be persisted to disk. Persisting to disk generates a whole bunch of logs:

examples/bin/run_example_client.sh

@@ -18,7 +18,7 @@ EXAMPLE=$1
 if [ -z ${EXAMPLE} ] ; then
     echo "Please specify an example type."
     echo "Examples availables:"
-    echo `ls ${EXAMPLES_DIR}`
+    echo `ls ${EXAMPLES_DIR} | grep -v indexing`
     read -p "> " EXAMPLE
     echo " "
 fi

examples/bin/run_example_server.sh

@@ -19,7 +19,7 @@ EXAMPLE=$1
 if [ -z ${EXAMPLE} ] ; then
     echo "Please specify an example type."
     echo "Examples availables:"
-    echo `ls ${EXAMPLES_DIR}`
+    echo `ls ${EXAMPLES_DIR} | grep -v indexing`
     read -p "> " EXAMPLE
     echo " "
 fi

examples/config/historical/runtime.properties

@@ -4,7 +4,7 @@ druid.port=8081
 
 druid.zk.service.host=localhost
 
-druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions:0.6.2"]
+druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions:0.6.2-SNAPSHOT"]
 
 # Dummy read only AWS account (used to download example data)
 druid.s3.secretKey=QyyfVZ7llSiRg6Qcrql1eEUG7buFpAK6T6engr1b

examples/config/realtime/runtime.properties

@@ -4,6 +4,8 @@ druid.port=8083
 
 druid.zk.service.host=localhost
 
+druid.extensions.coordinates=["io.druid.extensions:druid-examples:0.6.2-SNAPSHOT"]
+
 druid.db.connector.connectURI=jdbc\:mysql\://localhost\:3306/druid
 druid.db.connector.user=druid
 druid.db.connector.password=diurd

examples/pom.xml

@@ -20,7 +20,7 @@
 
 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
     <modelVersion>4.0.0</modelVersion>
-    <groupId>io.druid</groupId>
+    <groupId>io.druid.extensions</groupId>
     <artifactId>druid-examples</artifactId>
     <name>druid-examples</name>
     <description>druid-examples</description>

examples/src/main/java/druid/examples/ExamplesDruidModule.java → examples/src/main/java/io/druid/examples/ExamplesDruidModule.java

@@ -17,16 +17,16 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples;
+package io.druid.examples;
 
 import com.fasterxml.jackson.databind.Module;
 import com.fasterxml.jackson.databind.jsontype.NamedType;
 import com.fasterxml.jackson.databind.module.SimpleModule;
 import com.google.inject.Binder;
-import druid.examples.flights.FlightsFirehoseFactory;
-import druid.examples.rand.RandomFirehoseFactory;
-import druid.examples.twitter.TwitterSpritzerFirehoseFactory;
-import druid.examples.web.WebFirehoseFactory;
+import io.druid.examples.flights.FlightsFirehoseFactory;
+import io.druid.examples.rand.RandomFirehoseFactory;
+import io.druid.examples.twitter.TwitterSpritzerFirehoseFactory;
+import io.druid.examples.web.WebFirehoseFactory;
 import io.druid.initialization.DruidModule;
 
 import java.util.Arrays;

examples/src/main/java/druid/examples/flights/FlightsConverter.java → examples/src/main/java/io/druid/examples/flights/FlightsConverter.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.flights;
+package io.druid.examples.flights;
 
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.google.common.io.Closeables;

examples/src/main/java/druid/examples/flights/FlightsFirehoseFactory.java → examples/src/main/java/io/druid/examples/flights/FlightsFirehoseFactory.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.flights;
+package io.druid.examples.flights;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;

examples/src/main/java/druid/examples/rabbitmq/RabbitMQProducerMain.java → examples/src/main/java/io/druid/examples/rabbitmq/RabbitMQProducerMain.java

-package druid.examples.rabbitmq;
+package io.druid.examples.rabbitmq;
 
 import com.rabbitmq.client.Channel;
 import com.rabbitmq.client.Connection;

examples/src/main/java/druid/examples/rand/RandomFirehoseFactory.java → examples/src/main/java/io/druid/examples/rand/RandomFirehoseFactory.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.rand;
+package io.druid.examples.rand;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;

examples/src/main/java/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java → examples/src/main/java/io/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.twitter;
+package io.druid.examples.twitter;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;

examples/src/main/java/druid/examples/web/InputSupplierUpdateStream.java → examples/src/main/java/io/druid/examples/web/InputSupplierUpdateStream.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.fasterxml.jackson.core.JsonParseException;
 import com.fasterxml.jackson.core.type.TypeReference;

examples/src/main/java/druid/examples/web/InputSupplierUpdateStreamFactory.java → examples/src/main/java/io/druid/examples/web/InputSupplierUpdateStreamFactory.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.google.common.io.InputSupplier;
 

examples/src/main/java/druid/examples/web/RenamingKeysUpdateStream.java → examples/src/main/java/io/druid/examples/web/RenamingKeysUpdateStream.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.google.common.collect.Maps;
 

examples/src/main/java/druid/examples/web/RenamingKeysUpdateStreamFactory.java → examples/src/main/java/io/druid/examples/web/RenamingKeysUpdateStreamFactory.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import java.util.Map;
 

examples/src/main/java/druid/examples/web/UpdateStream.java → examples/src/main/java/io/druid/examples/web/UpdateStream.java

@@ -16,7 +16,7 @@
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
-package druid.examples.web;
+package io.druid.examples.web;
 
 import java.util.Map;
 import java.util.concurrent.TimeUnit;

examples/src/main/java/druid/examples/web/UpdateStreamFactory.java → examples/src/main/java/io/druid/examples/web/UpdateStreamFactory.java

@@ -16,7 +16,7 @@
  * along with this program; if not, write to the Free Software
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
-package druid.examples.web;
+package io.druid.examples.web;
 
 public interface UpdateStreamFactory
 {

examples/src/main/java/druid/examples/web/WebFirehoseFactory.java → examples/src/main/java/io/druid/examples/web/WebFirehoseFactory.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;

examples/src/main/java/druid/examples/web/WebJsonSupplier.java → examples/src/main/java/io/druid/examples/web/WebJsonSupplier.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.google.common.io.InputSupplier;
 import com.metamx.emitter.EmittingLogger;

examples/src/main/resources/META-INF/services/io.druid.initialization.DruidModule

+io.druid.examples.ExamplesDruidModule
\ No newline at end of file

examples/src/test/java/druid/examples/web/InputSupplierUpdateStreamTest.java → examples/src/test/java/io/druid/examples/web/InputSupplierUpdateStreamTest.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.google.common.io.InputSupplier;
 import junit.framework.Assert;

examples/src/test/java/druid/examples/web/RenamingKeysUpdateStreamTest.java → examples/src/test/java/io/druid/examples/web/RenamingKeysUpdateStreamTest.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.google.common.io.InputSupplier;
 import junit.framework.Assert;

examples/src/test/java/druid/examples/web/TestCaseSupplier.java → examples/src/test/java/io/druid/examples/web/TestCaseSupplier.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.google.common.io.InputSupplier;
 

examples/src/test/java/druid/examples/web/WebFirehoseFactoryTest.java → examples/src/test/java/io/druid/examples/web/WebFirehoseFactoryTest.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.Lists;

examples/src/test/java/druid/examples/web/WebJsonSupplierTest.java → examples/src/test/java/io/druid/examples/web/WebJsonSupplierTest.java

@@ -17,7 +17,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  */
 
-package druid.examples.web;
+package io.druid.examples.web;
 
 import org.junit.Test;
 

indexing-service/src/main/java/io/druid/indexing/common/index/EventReceiverFirehoseFactory.java

@@ -63,6 +63,9 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
   private final MapInputRowParser parser;
   private final Optional<ChatHandlerProvider> chatHandlerProvider;
 
+  @Deprecated
+  private final EventReceiverFirehoseFactoryConfig config;
+
   @JsonCreator
   public EventReceiverFirehoseFactory(
       @JsonProperty("serviceName") String serviceName,
@@ -75,14 +78,17 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
   {
     // This code is here for backwards compatibility
     if (serviceName == null) {
-      serviceName = String.format("%s:%s",
+      this.serviceName = String.format(
+          "%s:%s",
           config.getFirehoseIdPrefix(),
           Preconditions.checkNotNull(firehoseId, "firehoseId")
       );
+      this.config = config;
+    } else {
+      this.serviceName = Preconditions.checkNotNull(serviceName, "serviceName");
+      this.config = null;
     }
 
-    this.serviceName = Preconditions.checkNotNull(serviceName, "serviceName");
-
     this.bufferSize = bufferSize == null || bufferSize <= 0 ? DEFAULT_BUFFER_SIZE : bufferSize;
     this.parser = Preconditions.checkNotNull(parser, "parser");
     this.chatHandlerProvider = Optional.fromNullable(chatHandlerProvider);
@@ -98,7 +104,6 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
     if (chatHandlerProvider.isPresent()) {
       log.info("Found chathandler of class[%s]", chatHandlerProvider.get().getClass().getName());
       chatHandlerProvider.get().register(serviceName, firehose);
-      chatHandlerProvider.get().register(serviceName.replaceAll(".*:", ""), firehose); // rofl
     } else {
       log.info("No chathandler detected");
     }
@@ -112,6 +117,17 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
     return serviceName;
   }
 
+  @Deprecated
+  @JsonProperty("firehoseId")
+  public String getFirehoseId()
+  {
+    if (config != null) {
+      return serviceName.replaceFirst(config.getFirehoseIdPrefix(), "");
+    }
+
+    return null;
+  }
+
   @JsonProperty
   public int getBufferSize()
   {

services/pom.xml

@@ -17,7 +17,8 @@
   ~ along with this program; if not, write to the Free Software
   ~ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
   -->
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <groupId>io.druid</groupId>
     <artifactId>druid-services</artifactId>
@@ -82,6 +83,16 @@
                             <outputFile>
                                 ${project.build.directory}/${project.artifactId}-${project.version}-selfcontained.jar
                             </outputFile>
+                            <filters>
+                                <filter>
+                                    <artifact>*:*</artifact>
+                                    <excludes>
+                                        <exclude>META-INF/*.SF</exclude>
+                                        <exclude>META-INF/*.DSA</exclude>
+                                        <exclude>META-INF/*.RSA</exclude>
+                                    </excludes>
+                                </filter>
+                            </filters>
                         </configuration>
                     </execution>
                 </executions>