Support `(str->long)` and `(str->int)` cast statements in OAL core engine. (#7907)

Enhance OAL engine core
* Fix `funcParamExpression` and `literalExpression` can't be used in the same aggregation function.
* Support cast statement in the OAL core engine.
* Support `(str->long)` and `(long)` for string to long cast statement.
* Support `(str->int)` and `(int)` for string to int cast statement.
* Support Long literal number in the OAL core engine.
* Support literal `string` as parameter of aggregation function.
* Add `attributeExpression` and `attributeExpressionSegment` in the OAL grammar tree to support `map` type for the
  attribute expression.

Besides added UTs, I locally tested these OAL scripts
```
mq_consume_latency = from((str->long)Service.tag["transmission.latency"]).longAvg();

mq_consume_latency = from((long)Service.tag["transmission.latency"]).longAvg();

mq_consume_latency = from((str->long)Service.tag["transmission.latency"]).filter((str->long)tag["transmission.latency"] > 0L).longAvg();
```

CHANGES.md

@@ -6,6 +6,7 @@ Release Notes.
 ------------------
 
 #### OAP Server
+
 * Add component definition for `Jackson`.
 * Fix that zipkin-receiver plugin is not packaged into dist.
 * Replace e2e cases to e2e-v2: Lua Nginx/SelfObservability.
@@ -13,8 +14,18 @@ Release Notes.
 * Add component definition for `Apache-Kylin`.
 * Enhance `get` generation mechanism of OAL engine, support map type of source's field.
 * Replace e2e cases to e2e-v2: Gateway.
+* Add `tag`(Map) into All, Service, ServiceInstance and Endpoint sources.
+* Fix `funcParamExpression` and `literalExpression` can't be used in the same aggregation function.
+* Support cast statement in the OAL core engine.
+* Support `(str->long)` and `(long)` for string to long cast statement.
+* Support `(str->int)` and `(int)` for string to int cast statement.
+* Support Long literal number in the OAL core engine.
+* Support literal `string` as parameter of aggregation function.
+* Add `attributeExpression` and `attributeExpressionSegment` in the OAL grammar tree to support `map` type for the
+  attribute expression.
 
 #### UI
+
 * Optimize endpoint dependency.
 * Show service name by hovering nodes in the sankey chart.
 * Add Apache Kylin logo.
@@ -22,7 +33,6 @@ Release Notes.
 
 #### Documentation
 
-
 All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/101?closed=1)
 
 ------------------

docs/en/concepts-and-designs/oal.md

@@ -15,15 +15,17 @@ You can open set `SW_OAL_ENGINE_DEBUG=Y` at system env to see which classes are
 Scripts should be named `*.oal`
 ```
 // Declare the metrics.
-METRICS_NAME = from(SCOPE.(* | [FIELD][,FIELD ...]))
-[.filter(FIELD OP [INT | STRING])]
+METRICS_NAME = from(CAST SCOPE.(* | [FIELD][,FIELD ...]))
+[.filter(CAST FIELD OP [INT | STRING])]
 .FUNCTION([PARAM][, PARAM ...])
 
 // Disable hard code 
 disable(METRICS_NAME);
 ```
 
-## Scope
+## From
+The **from** statement defines the data source of this OAL expression.
+
 Primary **SCOPE**s are `All`, `Service`, `ServiceInstance`, `Endpoint`, `ServiceRelation`, `ServiceInstanceRelation`, and `EndpointRelation`.
 There are also some secondary scopes which belong to a primary scope. 
 
@@ -95,6 +97,23 @@ All metrics data will be grouped by Scope.ID and min-level TimeBucket.
 
 - In the `Endpoint` scope, the Scope.ID is same as the Endpoint ID (i.e. the unique ID based on service and its endpoint).
 
+## Cast
+Fields of source are static type. In some cases, the type required by the filter expression and aggregation function doesn't 
+match the type in the source, such as tag value in the source is String type, most aggregation calculation requires numeric.
+
+Cast expression is provided to do so. 
+- `(str->long)` or `(long)`, cast string type into long.
+- `(str->int)` or `(int)`, cast string type into int.
+
+```
+mq_consume_latency = from((str->long)Service.tag["transmission.latency"]).longAvg(); // the value of tag is string type.
+```
+
+Cast statement is supported in
+1. **From statement**. `from((cast)source.attre)`. 
+2. **Filter expression**. `.filter((cast)tag["transmission.latency"] > 0)`
+3. **Aggregation function parameter**. `.longAvg((cast)strField1== 1,  (cast)strField2)`
+
 ## Disable
 `Disable` is an advanced statement in OAL, which is only used in certain cases.
 Some of the aggregation and metrics are defined through core hard codes. Examples include `segment` and `top_n_database_statement`.

oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4

@@ -122,7 +122,7 @@ fragment HexDigit
     ;
 
 fragment Digits
-    : [0-9] ([0-9_]* [0-9])?
+    : [0-9] ([0-9_]* [0-9])? ('l'|'L')?
     ;
 
 fragment LetterOrDigit
@@ -135,3 +135,9 @@ fragment Letter
     | ~[\u0000-\u007F\uD800-\uDBFF] // covers all characters above 0x7F which are not a surrogate
     | [\uD800-\uDBFF] [\uDC00-\uDFFF] // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
     ;
+
+// Type cast rule
+STRING_TO_LONG:                                '(str->long)';
+STRING_TO_LONG_SHORT:                          '(long)';
+STRING_TO_INT:                                 '(str->int)';
+STRING_TO_INT_SHORT:                           '(int)';
\ No newline at end of file

oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4

@@ -38,7 +38,7 @@ disableStatement
     ;
 
 metricStatement
-    : FROM LR_BRACKET source (sourceAttributeStmt+) RR_BRACKET (filterStatement+)? DOT aggregateFunction
+    : FROM LR_BRACKET (sourceAttrCast)? source (sourceAttributeStmt+) RR_BRACKET (filterStatement+)? DOT aggregateFunction
     ;
 
 filterStatement
@@ -69,7 +69,7 @@ sourceAttributeStmt
     ;
 
 sourceAttribute
-    : IDENTIFIER | ALL
+    : IDENTIFIER | ALL | mapAttribute
     ;
 
 variable
@@ -77,7 +77,7 @@ variable
     ;
 
 aggregateFunction
-    : functionName LR_BRACKET ((funcParamExpression (COMMA funcParamExpression)?) | (literalExpression (COMMA literalExpression)?))? RR_BRACKET
+    : functionName LR_BRACKET ((funcParamExpression|literalExpression|attributeExpression) (COMMA (funcParamExpression|literalExpression|attributeExpression))?)? RR_BRACKET
     ;
 
 functionName
@@ -89,7 +89,15 @@ funcParamExpression
     ;
 
 literalExpression
-    : BOOL_LITERAL | NUMBER_LITERAL | IDENTIFIER
+    : BOOL_LITERAL | NUMBER_LITERAL | STRING_LITERAL
+    ;
+
+attributeExpression
+    : functionArgCast? attributeExpressionSegment (DOT attributeExpressionSegment)*
+    ;
+
+attributeExpressionSegment
+    : (IDENTIFIER | mapAttribute)
     ;
 
 expression
@@ -153,11 +161,11 @@ multiConditionValue
     ;
 
 conditionAttributeStmt
-    : conditionAttribute ((DOT conditionAttribute)*)
+    : (expressionAttrCast)? conditionAttribute ((DOT conditionAttribute)*)
     ;
 
 conditionAttribute
-    : IDENTIFIER | mapAttribute
+    : (IDENTIFIER | mapAttribute)
     ;
 
 mapAttribute
@@ -179,3 +187,19 @@ enumConditionValue
 numberConditionValue
     : NUMBER_LITERAL
     ;
+
+sourceAttrCast
+    : castStmt
+    ;
+
+expressionAttrCast
+    : castStmt
+    ;
+
+functionArgCast
+    : castStmt
+    ;
+
+castStmt
+    : STRING_TO_LONG | STRING_TO_LONG_SHORT | STRING_TO_INT | STRING_TO_INT_SHORT
+    ;
\ No newline at end of file

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java

@@ -46,6 +46,8 @@ public class AnalysisResult {
 
     private List<String> sourceAttribute = new ArrayList<>();
 
+    private String sourceCastType;
+
     private String aggregationFunctionName;
 
     private String metricsClassName;
@@ -64,6 +66,8 @@ public class AnalysisResult {
 
     private int argGetIdx = 0;
 
+    private String nextArgCast = null;
+
     private List<DataColumn> persistentFields;
 
     private List<SourceColumn> fieldsFromSource;
@@ -107,9 +111,17 @@ public class AnalysisResult {
         if (funcArgs == null) {
             funcArgs = new LinkedList<>();
         }
+        if (nextArgCast != null) {
+            argument.setCastType(nextArgCast);
+            nextArgCast = null;
+        }
         funcArgs.add(argument);
     }
 
+    public Argument getLastArgument() {
+        return funcArgs.get(funcArgs.size() - 1);
+    }
+
     public Argument getNextFuncArg() {
         return funcArgs.get(argGetIdx++);
     }

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java

@@ -21,6 +21,7 @@ package org.apache.skywalking.oal.rt.parser;
 import java.util.List;
 import lombok.Getter;
 import lombok.RequiredArgsConstructor;
+import lombok.Setter;
 
 /**
  * Function argument.
@@ -32,4 +33,11 @@ public class Argument {
     private final int type;
 
     private final List<String> text;
+
+    @Setter
+    private String castType;
+
+    public void addText(String text) {
+        this.text.add(text);
+    }
 }

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java

@@ -36,6 +36,7 @@ public class ConditionExpression {
     private String value;
     private List<String> values;
     private boolean number;
+    private String castType;
 
     public ConditionExpression(final String expressionType, final String attributes, final String value) {
         this.expressionType = expressionType;

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java

@@ -24,6 +24,7 @@ import java.lang.reflect.Method;
 import java.lang.reflect.Parameter;
 import java.util.List;
 import org.apache.skywalking.oal.rt.util.ClassMethodUtil;
+import org.apache.skywalking.oal.rt.util.TypeCastUtil;
 import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
 import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Arg;
 import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.ConstOne;
@@ -56,7 +57,7 @@ public class DeepAnalysis {
 
                 final Expression filterExpression = new Expression();
                 filterExpression.setExpressionObject(matcherInfo.getMatcher().getName());
-                filterExpression.setLeft("source." + getter);
+                filterExpression.setLeft(TypeCastUtil.withCast(expression.getCastType(), "source." + getter));
                 filterExpression.setRight(expression.getValue());
                 result.addFilterExpressions(filterExpression);
             }
@@ -94,7 +95,12 @@ public class DeepAnalysis {
             Annotation annotation = parameterAnnotations[0];
             if (annotation instanceof SourceFrom) {
                 entryMethod.addArg(
-                    parameterType, "source." + ClassMethodUtil.toGetMethod(result.getSourceAttribute()));
+                    parameterType,
+                    TypeCastUtil.withCast(
+                        result.getSourceCastType(),
+                        "source." + ClassMethodUtil.toGetMethod(result.getSourceAttribute())
+                    )
+                );
             } else if (annotation instanceof ConstOne) {
                 entryMethod.addArg(parameterType, "1");
             } else if (annotation instanceof org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Expression) {
@@ -113,7 +119,7 @@ public class DeepAnalysis {
                     final Expression argExpression = new Expression();
                     argExpression.setRight(expression.getValue());
                     argExpression.setExpressionObject(matcherInfo.getMatcher().getName());
-                    argExpression.setLeft("source." + getter);
+                    argExpression.setLeft(TypeCastUtil.withCast(expression.getCastType(), "source." + getter));
 
                     entryMethod.addArg(argExpression);
                 }

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java

@@ -23,12 +23,13 @@ import java.util.List;
 import lombok.Getter;
 import lombok.Setter;
 import org.apache.skywalking.oal.rt.util.ClassMethodUtil;
+import org.apache.skywalking.oal.rt.util.TypeCastUtil;
 
 @Getter
 @Setter
 public class EntryMethod {
     static final int LITERAL_TYPE = 1;
-    static final int IDENTIFIER_TYPE = 2;
+    static final int ATTRIBUTE_EXP_TYPE = 2;
     static final int EXPRESSION_TYPE = 3;
 
     private String methodName;
@@ -42,9 +43,9 @@ public class EntryMethod {
             return;
         }
         addArg(parameterType, arg.getType(), parameterType.equals(boolean.class) ?
-            "source." + ClassMethodUtil.toIsMethod(arg.getText())
+            TypeCastUtil.withCast(arg.getCastType(), "source." + ClassMethodUtil.toIsMethod(arg.getText()))
             :
-            "source." + ClassMethodUtil.toGetMethod(arg.getText()));
+            TypeCastUtil.withCast(arg.getCastType(), "source." + ClassMethodUtil.toGetMethod(arg.getText())));
     }
 
     void addArg(Class<?> parameterType, String expression) {

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java

@@ -18,6 +18,7 @@
 
 package org.apache.skywalking.oal.rt.parser;
 
+import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
 import org.antlr.v4.runtime.misc.NotNull;
@@ -63,6 +64,11 @@ public class OALListener extends OALParserBaseListener {
         current.getSourceAttribute().add(ctx.getText());
     }
 
+    @Override
+    public void enterSourceAttrCast(OALParser.SourceAttrCastContext ctx) {
+        current.setSourceCastType(ctx.getText());
+    }
+
     @Override
     public void enterVariable(OALParser.VariableContext ctx) {
     }
@@ -205,6 +211,11 @@ public class OALListener extends OALParserBaseListener {
         enterConditionValue(ctx.getText());
     }
 
+    @Override
+    public void enterExpressionAttrCast(final OALParser.ExpressionAttrCastContext ctx) {
+        conditionExpression.setCastType(ctx.getText());
+    }
+
     private void enterConditionValue(String value) {
         if (value.split("\\.").length == 2 && !value.startsWith("\"")) {
             // Value is an enum.
@@ -219,11 +230,22 @@ public class OALListener extends OALParserBaseListener {
 
     @Override
     public void enterLiteralExpression(OALParser.LiteralExpressionContext ctx) {
-        if (ctx.IDENTIFIER() == null) {
-            current.addFuncArg(new Argument(EntryMethod.LITERAL_TYPE, Arrays.asList(ctx.getText())));
-            return;
-        }
-        current.addFuncArg(new Argument(EntryMethod.IDENTIFIER_TYPE, Arrays.asList(ctx.getText().split("\\."))));
+        current.addFuncArg(new Argument(EntryMethod.LITERAL_TYPE, Arrays.asList(ctx.getText())));
+    }
+
+    @Override
+    public void enterAttributeExpression(final OALParser.AttributeExpressionContext ctx) {
+        current.addFuncArg(new Argument(EntryMethod.ATTRIBUTE_EXP_TYPE, new ArrayList<>(3)));
+    }
+
+    @Override
+    public void enterAttributeExpressionSegment(OALParser.AttributeExpressionSegmentContext ctx) {
+        current.getLastArgument().addText(ctx.getText());
+    }
+
+    @Override
+    public void enterFunctionArgCast(final OALParser.FunctionArgCastContext ctx) {
+        current.getLastArgument().setCastType(ctx.getText());
     }
 
     private String metricsNameFormat(String source) {

oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/util/TypeCastUtil.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oal.rt.util;
+
+public class TypeCastUtil {
+    /**
+     * @param castType           to change the value of given original expression.
+     * @param originalExpression to read the value
+     * @return cast expression if cast type exists and is legal.
+     */
+    public static String withCast(String castType, String originalExpression) {
+        if (castType == null) {
+            return originalExpression;
+        }
+        switch (castType) {
+            case "(str->long)":
+            case "(long)":
+                return "Long.parseLong(" + originalExpression + ")";
+            case "(str->int)":
+            case "(int)":
+                return "Integer.parseInt(" + originalExpression + ")";
+            default:
+                throw new IllegalArgumentException(
+                    "castType:" + castType + " is legal, context expression:" + originalExpression);
+        }
+    }
+}

oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java

@@ -253,7 +253,9 @@ public class ScriptParserTest {
     @Test
     public void testParse9() throws IOException {
         ScriptParser parser = ScriptParser.createFromScriptText(
-            "ServicePercent = from(Service.sidecar.internalError).filter(sidecar.internalError == \"abc\").percent(sidecar.internalError != \"\");", TEST_SOURCE_PACKAGE);
+            "ServicePercent = from(Service.sidecar.internalError).filter(sidecar.internalError == \"abc\").percent(sidecar.internalError != \"\");",
+            TEST_SOURCE_PACKAGE
+        );
         List<AnalysisResult> results = parser.parse().getMetricsStmts();
 
         AnalysisResult servicePercent = results.get(0);
@@ -290,12 +292,34 @@ public class ScriptParserTest {
     @Test
     public void testParse11() throws IOException {
         ScriptParser parser = ScriptParser.createFromScriptText(
-            "GetCallTraffic = from(Service.*).filter(tag[\"http.method\"] == \"get\").cpm();", TEST_SOURCE_PACKAGE);
+            "GetCallTraffic = from(Service.*).filter(tag[\"http.method\"] == \"get\").cpm(tag[\"http.method\"]);",
+            TEST_SOURCE_PACKAGE
+        );
         List<AnalysisResult> results = parser.parse().getMetricsStmts();
         AnalysisResult clientCpm = results.get(0);
         final List<Expression> filterExpressions = clientCpm.getFilterExpressions();
         Assert.assertEquals(1, filterExpressions.size());
         Assert.assertEquals("source.getTag(\"http.method\")", filterExpressions.get(0).getLeft());
+        Assert.assertEquals(1, clientCpm.getFuncArgs().size());
+        Assert.assertEquals("[tag[\"http.method\"]]", clientCpm.getFuncArgs().get(0).getText().toString());
+    }
+
+    @Test
+    public void testParse12() throws IOException {
+        ScriptParser parser = ScriptParser.createFromScriptText(
+            "cast_metrics = from((str->long)Service.tag[\"transmission.latency\"]).filter((str->long)tag[\"transmission.latency\"] > 0).longAvg((str->long)strField1== 1,  (str->long)strField2);",
+            TEST_SOURCE_PACKAGE
+        );
+        List<AnalysisResult> results = parser.parse().getMetricsStmts();
+        AnalysisResult castExp = results.get(0);
+        Assert.assertEquals("(str->long)", castExp.getSourceCastType());
+        final List<Expression> filterExpressions = castExp.getFilterExpressions();
+        Assert.assertEquals(1, filterExpressions.size());
+        Assert.assertEquals(
+            "Long.parseLong(source.getTag(\"transmission.latency\"))", filterExpressions.get(0).getLeft());
+        Assert.assertEquals("(str->long)", castExp.getFuncConditionExpressions().get(0).getCastType());
+        Assert.assertEquals(EntryMethod.ATTRIBUTE_EXP_TYPE, castExp.getFuncArgs().get(0).getType());
+        Assert.assertEquals("(str->long)", castExp.getFuncArgs().get(0).getCastType());
     }
 
     @Test