support oal list includes and excludes & add tags to some source (#5739)

* add oal excludes and includes to support tags
Co-authored-by: 吴晟 Wu Sheng <wu.sheng@foxmail.com>

docs/en/concepts-and-designs/oal.md

@@ -34,7 +34,7 @@ Read [Scope Definitions](scope-definitions.md), you can find all existing Scopes
 Use filter to build the conditions for the value of fields, by using field name and expression. 
 
 The expressions support to link by `and`, `or` and `(...)`. 
-The OPs support `==`, `!=`, `>`, `<`, `>=`, `<=`, `in [...]` ,`like %...`, `like ...%` and `like %...%`, with type detection based of field type. Trigger compile
+The OPs support `==`, `!=`, `>`, `<`, `>=`, `<=`, `in [...]` ,`like %...`, `like ...%` , `like %...%` , `contain` and `not contain`, with type detection based of field type. Trigger compile
  or code generation error if incompatible. 
 
 ## Aggregation Function
@@ -105,33 +105,39 @@ In default, no one is being disable.
 
 ## Examples
 ```
-// Caculate p99 of both Endpoint1 and Endpoint2
+// Calculate p99 of both Endpoint1 and Endpoint2
 endpoint_p99 = from(Endpoint.latency).filter(name in ("Endpoint1", "Endpoint2")).summary(0.99)
 
-// Caculate p99 of Endpoint name started with `serv`
+// Calculate p99 of Endpoint name started with `serv`
 serv_Endpoint_p99 = from(Endpoint.latency).filter(name like "serv%").summary(0.99)
 
-// Caculate the avg response time of each Endpoint
+// Calculate the avg response time of each Endpoint
 endpoint_avg = from(Endpoint.latency).avg()
 
-// Caculate the p50, p75, p90, p95 and p99 of each Endpoint by 50 ms steps.
+// Calculate the p50, p75, p90, p95 and p99 of each Endpoint by 50 ms steps.
 endpoint_percentile = from(Endpoint.latency).percentile(10)
 
-// Caculate the percent of response status is true, for each service.
+// Calculate the percent of response status is true, for each service.
 endpoint_success = from(Endpoint.*).filter(status == true).percent()
 
-// Caculate the sum of response code in [404, 500, 503], for each service.
+// Calculate the sum of response code in [404, 500, 503], for each service.
 endpoint_abnormal = from(Endpoint.*).filter(responseCode in [404, 500, 503]).sum()
 
-// Caculate the sum of request type in [RequestType.PRC, RequestType.gRPC], for each service.
+// Calculate the sum of request type in [RequestType.PRC, RequestType.gRPC], for each service.
 endpoint_rpc_calls_sum = from(Endpoint.*).filter(type in [RequestType.PRC, RequestType.gRPC]).sum()
 
-// Caculate the sum of endpoint name in ["/v1", "/v2"], for each service.
+// Calculate the sum of endpoint name in ["/v1", "/v2"], for each service.
 endpoint_url_sum = from(Endpoint.*).filter(endpointName in ["/v1", "/v2"]).sum()
 
-// Caculate the sum of calls for each service.
+// Calculate the sum of calls for each service.
 endpoint_calls = from(Endpoint.*).sum()
 
+// Calculate the CPM with the GET method for each service.The value is made up with `tagKey:tagValue`.
+service_cpm_http_get = from(Service.*).filter(tags contain "http.method:GET").cpm()
+
+// Calculate the CPM with the HTTP method except for the GET method for each service.The value is made up with `tagKey:tagValue`.
+service_cpm_http_other = from(Service.*).filter(tags not contain "http.method:GET").cpm()
+
 disable(segment);
 disable(endpoint_relation_server_side);
 disable(top_n_database_statement);

docs/en/concepts-and-designs/scope-definitions.md

@@ -12,6 +12,7 @@ By using Aggregation Function, the requests will group by time and **Group Key(s
 | status  | Represent whether success or fail of the request.  |   | bool(true for success)  |
 | responseCode | Represent the response code of HTTP response, if this request is the HTTP call. e.g. 200, 404, 302| | int |
 | type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
+| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
 
 ### SCOPE `Service`
 
@@ -27,6 +28,7 @@ Calculate the metrics data from each request of the service.
 | status | Represent whether success or fail of the request. | | bool(true for success)  |
 | responseCode | Represent the response code of HTTP response, if this request is the HTTP call | | int|
 | type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
+| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
 
 ### SCOPE `ServiceInstance`
 
@@ -42,6 +44,7 @@ Calculate the metrics data from each request of the service instance.
 | status | Represent whether success or fail of the request. | | bool(true for success) |
 | responseCode | Represent the response code of HTTP response, if this request is the HTTP call. | | int |
 | type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
+| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
 
 #### Secondary scopes of `ServiceInstance` 
 
@@ -113,6 +116,7 @@ Calculate the metrics data from each request of the endpoint in the service.
 | status | Represent whether success or fail of the request.| | bool(true for success) |
 | responseCode | Represent the response code of HTTP response, if this request is the HTTP call. | | int |
 | type | Represent the type of each request. Such as: Database, HTTP, RPC, gRPC. | | enum |
+| tags | Represent the labels of each request and each value is made up with the `TagKey:TagValue` in the segment. | | `List<String>` |
 
 ### SCOPE `ServiceRelation`
 

oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/MultiScopesAnalysisListener.java

@@ -245,6 +245,7 @@ public class MultiScopesAnalysisListener implements EntryAnalysisListener, ExitA
                     log.warn("span {} has illegal status code {}", span, tag.getValue());
                 }
             }
+            sourceBuilder.setTag(tag);
         });
 
         sourceBuilder.setStatus(!span.getIsError());

oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/trace/parser/listener/SourceBuilder.java

@@ -18,9 +18,12 @@
 
 package org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener;
 
+import java.util.ArrayList;
+import java.util.List;
 import lombok.Getter;
 import lombok.RequiredArgsConstructor;
 import lombok.Setter;
+import org.apache.skywalking.apm.network.common.v3.KeyStringValuePair;
 import org.apache.skywalking.apm.util.StringUtil;
 import org.apache.skywalking.oap.server.core.analysis.NodeType;
 import org.apache.skywalking.oap.server.core.config.NamingControl;
@@ -120,6 +123,8 @@ class SourceBuilder {
     @Getter
     @Setter
     private long timeBucket;
+    @Getter
+    private final List<String> tags = new ArrayList<>();
 
     /**
      * The global level metrics source
@@ -134,6 +139,7 @@ class SourceBuilder {
         all.setResponseCode(responseCode);
         all.setType(type);
         all.setTimeBucket(timeBucket);
+        all.setTags(tags);
         return all;
     }
 
@@ -150,6 +156,7 @@ class SourceBuilder {
         service.setStatus(status);
         service.setResponseCode(responseCode);
         service.setType(type);
+        service.setTags(tags);
         service.setTimeBucket(timeBucket);
         return service;
     }
@@ -190,6 +197,7 @@ class SourceBuilder {
         serviceInstance.setStatus(status);
         serviceInstance.setResponseCode(responseCode);
         serviceInstance.setType(type);
+        serviceInstance.setTags(tags);
         serviceInstance.setTimeBucket(timeBucket);
         return serviceInstance;
     }
@@ -232,6 +240,7 @@ class SourceBuilder {
         endpoint.setStatus(status);
         endpoint.setResponseCode(responseCode);
         endpoint.setType(type);
+        endpoint.setTags(tags);
         endpoint.setTimeBucket(timeBucket);
         return endpoint;
     }
@@ -294,4 +303,8 @@ class SourceBuilder {
         databaseAccess.setTimeBucket(timeBucket);
         return databaseAccess;
     }
+
+    public void setTag(KeyStringValuePair tag) {
+        tags.add(tag.getKey().trim() + ":" + tag.getValue().trim());
+    }
 }

oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4

@@ -90,6 +90,8 @@ LESS_EQUAL:                          '<=';
 NOT_EQUAL:                           '!=';
 LIKE:                                'like';
 IN:                                  'in';
+CONTAIN:                            'contain';
+NOT_CONTAIN:                        'not contain';
 
 // Literals
 

oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4

@@ -90,7 +90,15 @@ literalExpression
     ;
 
 expression
-    : booleanMatch | stringMatch | greaterMatch | lessMatch | greaterEqualMatch | lessEqualMatch | notEqualMatch | booleanNotEqualMatch | likeMatch | inMatch
+    : booleanMatch | stringMatch | greaterMatch | lessMatch | greaterEqualMatch | lessEqualMatch | notEqualMatch | booleanNotEqualMatch | likeMatch | inMatch | containMatch | notContainMatch
+    ;
+
+containMatch
+    : conditionAttribute CONTAIN stringConditionValue
+    ;
+
+notContainMatch
+    : conditionAttribute NOT_CONTAIN stringConditionValue
     ;
 
 booleanMatch

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java

@@ -153,6 +153,16 @@ public class OALListener extends OALParserBaseListener {
         conditionExpression.setExpressionType("likeMatch");
     }
 
+    @Override
+    public void enterContainMatch(final OALParser.ContainMatchContext ctx) {
+        conditionExpression.setExpressionType("containMatch");
+    }
+
+    @Override
+    public void enterNotContainMatch(final OALParser.NotContainMatchContext ctx) {
+        conditionExpression.setExpressionType("notContainMatch");
+    }
+
     @Override
     public void enterInMatch(final OALParser.InMatchContext ctx) {
         conditionExpression.setExpressionType("inMatch");

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/ContainMatch.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.FilterMatcher;
+
+@FilterMatcher
+public class ContainMatch {
+    public boolean match(List<String> left, String right) {
+        if (Objects.isNull(left)) {
+            return false;
+        }
+        if (right.startsWith("\"") && right.endsWith("\"")) {
+            right = right.substring(1, right.length() - 1);
+        }
+        return left.contains(right);
+    }
+}

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/NotContainMatch.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
+
+import java.util.List;
+import java.util.Objects;
+import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.FilterMatcher;
+
+@FilterMatcher
+public class NotContainMatch {
+    public boolean match(List<String> left, String right) {
+        if (Objects.isNull(left)) {
+            return false;
+        }
+        if (right.startsWith("\"") && right.endsWith("\"")) {
+            right = right.substring(1, right.length() - 1);
+        }
+        return !left.contains(right);
+    }
+}

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/All.java

@@ -18,6 +18,7 @@
 
 package org.apache.skywalking.oap.server.core.source;
 
+import java.util.List;
 import lombok.Getter;
 import lombok.Setter;
 
@@ -56,4 +57,7 @@ public class All extends Source {
     @Getter
     @Setter
     private RequestType type;
+    @Getter
+    @Setter
+    private List<String> tags;
 }

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Endpoint.java

@@ -18,6 +18,7 @@
 
 package org.apache.skywalking.oap.server.core.source;
 
+import java.util.List;
 import lombok.Getter;
 import lombok.Setter;
 import lombok.extern.slf4j.Slf4j;
@@ -75,6 +76,9 @@ public class Endpoint extends Source {
     @Getter
     @Setter
     private RequestType type;
+    @Getter
+    @Setter
+    private List<String> tags;
 
     @Override
     public void prepare() {

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/Service.java

@@ -18,6 +18,7 @@
 
 package org.apache.skywalking.oap.server.core.source;
 
+import java.util.List;
 import lombok.Getter;
 import lombok.Setter;
 import org.apache.skywalking.oap.server.core.analysis.IDManager;
@@ -64,4 +65,8 @@ public class Service extends Source {
     @Getter
     @Setter
     private RequestType type;
+    @Getter
+    @Setter
+    private List<String> tags;
+
 }

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/source/ServiceInstance.java

@@ -18,6 +18,7 @@
 
 package org.apache.skywalking.oap.server.core.source;
 
+import java.util.List;
 import lombok.Getter;
 import lombok.Setter;
 import org.apache.skywalking.oap.server.core.analysis.IDManager;
@@ -67,6 +68,9 @@ public class ServiceInstance extends Source {
     @Getter
     @Setter
     private RequestType type;
+    @Getter
+    @Setter
+    private List<String> tags;
 
     @Override
     public void prepare() {

oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/ContainMatchTest.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
+
+import java.util.Arrays;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class ContainMatchTest {
+    @Test
+    public void match() {
+        ContainMatch containMatch = new ContainMatch();
+        Assert.assertFalse(containMatch.match(null, "http.method:GET"));
+        Assert.assertTrue(containMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:GET"));
+        Assert.assertFalse(
+            containMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:PUT"));
+    }
+}
\ No newline at end of file

oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/analysis/metrics/expression/NotContainMatchTest.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.core.analysis.metrics.expression;
+
+import java.util.Arrays;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class NotContainMatchTest {
+
+    @Test
+    public void match() {
+        NotContainMatch notContainMatch = new NotContainMatch();
+        Assert.assertFalse(notContainMatch.match(null, "http.method:GET"));
+        Assert.assertFalse(
+            notContainMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:GET"));
+        Assert.assertTrue(notContainMatch.match(Arrays.asList("http.method:GET", "http.method:POST"), "http.method:PUT"));
+    }
+}
\ No newline at end of file

oap-server/server-receiver-plugin/skywalking-trace-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/trace/provider/parser/listener/MultiScopesAnalysisListenerTest.java

@@ -190,6 +190,10 @@ public class MultiScopesAnalysisListenerTest {
                                           .setIsError(true)
                                           .setSpanType(SpanType.Entry)
                                           .setSpanLayer(SpanLayer.RPCFramework)
+                                          .addTags(KeyStringValuePair.newBuilder()
+                                                                     .setKey("http.method")
+                                                                     .setValue("GET")
+                                                                     .build())
                                           .addRefs(
                                               SegmentReference.newBuilder()
                                                               .setRefType(RefType.CrossProcess)
@@ -226,6 +230,11 @@ public class MultiScopesAnalysisListenerTest {
         Assert.assertEquals(serviceInstance.getName(), serviceInstanceRelation.getDestServiceInstanceName());
         Assert.assertEquals("downstream-endpoint", endpointRelation.getEndpoint());
         Assert.assertEquals(endpoint.getName(), endpointRelation.getChildEndpoint());
+        // tags test
+        Assert.assertEquals("http.method:GET", all.getTags().get(0));
+        Assert.assertEquals("http.method:GET", service.getTags().get(0));
+        Assert.assertEquals("http.method:GET", serviceInstance.getTags().get(0));
+        Assert.assertEquals("http.method:GET", endpoint.getTags().get(0));
     }
 
     /**