Running SAM backbone on frontend (#6019)

<!-- Raise an issue to propose your change
(https://github.com/opencv/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution
guide](https://opencv.github.io/cvat/docs/contributing/). -->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
Resolved #5984 
Resolved #6049
Resolved #6041

- Compatible only with ``sam_vit_h_4b8939.pth`` weights. Need to
re-export ONNX mask decoder with some custom model changes (see below)
to support other weights (or just download them using links below)
- Need to redeploy the serverless function because its interface has
been changed.

Decoders for other weights:
sam_vit_l_0b3195.pth:
[Download](https://drive.google.com/file/d/1Nb5CJKQm_6s1n3xLSZYso6VNgljjfR-6/view?usp=sharing)
sam_vit_b_01ec64.pth:
[Download](https://drive.google.com/file/d/17cZAXBPaOABS170c9bcj9PdQsMziiBHw/view?usp=sharing)

Changes done in ONNX part:
```
git diff scripts/export_onnx_model.py
diff --git a/scripts/export_onnx_model.py b/scripts/export_onnx_model.py
index 8441258..18d5be7 100644
--- a/scripts/export_onnx_model.py
+++ b/scripts/export_onnx_model.py
@@ -138,7 +138,7 @@ def run_export(

     _ = onnx_model(**dummy_inputs)

-    output_names = ["masks", "iou_predictions", "low_res_masks"]
+    output_names = ["masks", "iou_predictions", "low_res_masks", "xtl", "ytl", "xbr", "ybr"]

     with warnings.catch_warnings():
         warnings.filterwarnings("ignore", category=torch.jit.TracerWarning)
bsekachev@DESKTOP-OTBLK26:~/sam$ git diff segment_anything/utils/onnx.py
diff --git a/segment_anything/utils/onnx.py b/segment_anything/utils/onnx.py
index 3196bdf..85729c1 100644
--- a/segment_anything/utils/onnx.py
+++ b/segment_anything/utils/onnx.py
@@ -87,7 +87,15 @@ class SamOnnxModel(nn.Module):
         orig_im_size = orig_im_size.to(torch.int64)
         h, w = orig_im_size[0], orig_im_size[1]
         masks = F.interpolate(masks, size=(h, w), mode="bilinear", align_corners=False)
-        return masks
+        masks = torch.gt(masks, 0).to(torch.uint8)
+        nonzero = torch.nonzero(masks)
+        xindices = nonzero[:, 3:4]
+        yindices = nonzero[:, 2:3]
+        ytl = torch.min(yindices).to(torch.int64)
+        ybr = torch.max(yindices).to(torch.int64)
+        xtl = torch.min(xindices).to(torch.int64)
+        xbr = torch.max(xindices).to(torch.int64)
+        return masks[:, :, ytl:ybr + 1, xtl:xbr + 1], xtl, ytl, xbr, ybr

     def select_masks(
         self, masks: torch.Tensor, iou_preds: torch.Tensor, num_points: int
@@ -132,7 +140,7 @@ class SamOnnxModel(nn.Module):
         if self.return_single_mask:
             masks, scores = self.select_masks(masks, scores, point_coords.shape[1])

-        upscaled_masks = self.mask_postprocessing(masks, orig_im_size)
+        upscaled_masks, xtl, ytl, xbr, ybr = self.mask_postprocessing(masks, orig_im_size)

         if self.return_extra_metrics:
             stability_scores = calculate_stability_score(
@@ -141,4 +149,4 @@ class SamOnnxModel(nn.Module):
             areas = (upscaled_masks > self.model.mask_threshold).sum(-1).sum(-1)
             return upscaled_masks, scores, stability_scores, areas, masks

-        return upscaled_masks, scores, masks
+        return upscaled_masks, scores, masks, xtl, ytl, xbr, ybr
```

### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [x] I have added a description of my changes into the
[CHANGELOG](https://github.com/opencv/cvat/blob/develop/CHANGELOG.md)
file
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](

https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [x] I have increased versions of npm packages if it is necessary

([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning),

[cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning),

[cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning)
and

[cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))

### License

- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the
project.
  Feel free to contact the maintainers if that's a concern.

CHANGELOG.md

@@ -10,7 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - TDB
 
 ### Changed
-- TDB
+- Running SAM masks decoder on frontend (<https://github.com/opencv/cvat/pull/6019>)
 
 ### Deprecated
 - TDB

cvat-canvas/package.json

 {
   "name": "cvat-canvas",
-  "version": "2.16.4",
+  "version": "2.16.5",
   "description": "Part of Computer Vision Annotation Tool which presents its canvas library",
   "main": "src/canvas.ts",
   "scripts": {

cvat-canvas/src/typescript/interactionHandler.ts

 // Copyright (C) 2020-2022 Intel Corporation
+// Copyright (C) 2023 CVAT.ai Corporation
 //
 // SPDX-License-Identifier: MIT
 
@@ -146,13 +147,13 @@ export class InteractionHandlerImpl implements InteractionHandler {
                         _e.stopPropagation();
                         self.remove();
                         this.shapesWereUpdated = true;
-                        const shouldRaiseEvent = this.shouldRaiseEvent(_e.ctrlKey);
                         this.interactionShapes = this.interactionShapes.filter(
                             (shape: SVG.Shape): boolean => shape !== self,
                         );
                         if (this.interactionData.startWithBox && this.interactionShapes.length === 1) {
                             this.interactionShapes[0].style({ visibility: '' });
                         }
+                        const shouldRaiseEvent = this.shouldRaiseEvent(_e.ctrlKey);
                         if (shouldRaiseEvent) {
                             this.onInteraction(this.prepareResult(), true, false);
                         }
@@ -314,7 +315,7 @@ export class InteractionHandlerImpl implements InteractionHandler {
                 'pointer-events': 'none',
                 opacity: 0.5,
             }).addClass('cvat_canvas_interact_intermediate_shape');
-            image.move(this.geometry.offset, this.geometry.offset);
+            image.move(this.geometry.offset + left, this.geometry.offset + top);
             this.drawnIntermediateShape = image;
 
             imageDataToDataURL(

cvat-core/package.json

 {
   "name": "cvat-core",
-  "version": "9.0.1",
+  "version": "9.1.0",
   "description": "Part of Computer Vision Tool which presents an interface for client-side integration",
   "main": "src/api.ts",
   "scripts": {

cvat-core/src/plugins.ts

 // Copyright (C) 2019-2022 Intel Corporation
+// Copyright (C) 2023 CVAT.ai Corporation
 //
 // SPDX-License-Identifier: MIT
 
 import { PluginError } from './exceptions';
 
 const plugins = [];
+
+export interface APIWrapperEnterOptions {
+    preventMethodCall?: boolean;
+}
+
 export default class PluginRegistry {
     static async apiWrapper(wrappedFunc, ...args) {
-        // I have to optimize the wrapper
         const pluginList = await PluginRegistry.list();
+        const aggregatedOptions: APIWrapperEnterOptions = {
+            preventMethodCall: false,
+        };
+
         for (const plugin of pluginList) {
             const pluginDecorators = plugin.functions.filter((obj) => obj.callback === wrappedFunc)[0];
             if (pluginDecorators && pluginDecorators.enter) {
                 try {
-                    await pluginDecorators.enter.call(this, plugin, ...args);
+                    const options: APIWrapperEnterOptions | undefined = await pluginDecorators
+                        .enter.call(this, plugin, ...args);
+                    if (options?.preventMethodCall) {
+                        aggregatedOptions.preventMethodCall = true;
+                    }
                 } catch (exception) {
                     if (exception instanceof PluginError) {
                         throw exception;
@@ -24,7 +37,10 @@ export default class PluginRegistry {
             }
         }
 
-        let result = await wrappedFunc.implementation.call(this, ...args);
+        let result = null;
+        if (!aggregatedOptions.preventMethodCall) {
+            result = await wrappedFunc.implementation.call(this, ...args);
+        }
 
         for (const plugin of pluginList) {
             const pluginDecorators = plugin.functions.filter((obj) => obj.callback === wrappedFunc)[0];

cvat-ui/package.json

 {
   "name": "cvat-ui",
-  "version": "1.50.9",
+  "version": "1.51.0",
   "description": "CVAT single-page application",
   "main": "src/index.tsx",
   "scripts": {
@@ -22,6 +22,7 @@
   "dependencies": {
     "@ant-design/icons": "^4.6.3",
     "@types/lodash": "^4.14.172",
+    "@types/lru-cache": "^7.10.10",
     "@types/platform": "^1.3.4",
     "@types/react": "^16.14.15",
     "@types/react-color": "^3.0.5",
@@ -41,8 +42,10 @@
     "dotenv-webpack": "^8.0.1",
     "error-stack-parser": "^2.0.6",
     "lodash": "^4.17.21",
+    "lru-cache": "^9.1.1",
     "moment": "^2.29.2",
     "mousetrap": "^1.6.5",
+    "onnxruntime-web": "^1.14.0",
     "platform": "^1.3.6",
     "prop-types": "^15.7.2",
     "react": "^16.14.0",

cvat-ui/plugins/sam_plugin/src/ts/index.tsx

+// Copyright (C) 2023 CVAT.ai Corporation
+//
+// SPDX-License-Identifier: MIT
+
+import { InferenceSession, Tensor } from 'onnxruntime-web';
+import { LRUCache } from 'lru-cache';
+import { PluginEntryPoint, APIWrapperEnterOptions, ComponentBuilder } from 'components/plugins-entrypoint';
+
+interface SAMPlugin {
+    name: string;
+    description: string;
+    cvat: {
+        lambda: {
+            call: {
+                enter: (
+                    plugin: SAMPlugin,
+                    taskID: number,
+                    model: any,
+                    args: any,
+                ) => Promise<null | APIWrapperEnterOptions>;
+                leave: (
+                    plugin: SAMPlugin,
+                    result: any,
+                    taskID: number,
+                    model: any,
+                    args: any,
+                ) => Promise<any>;
+            };
+        };
+    };
+    data: {
+        core: any;
+        task: any;
+        modelID: string;
+        modelURL: string;
+        embeddings: LRUCache<string, Tensor>;
+        lowResMasks: LRUCache<string, Tensor>;
+        session: InferenceSession | null;
+    };
+    callbacks: {
+        onStatusChange: ((status: string) => void) | null;
+    };
+}
+
+interface ONNXInput {
+    image_embeddings: Tensor;
+    point_coords: Tensor;
+    point_labels: Tensor;
+    orig_im_size: Tensor;
+    mask_input: Tensor;
+    has_mask_input: Tensor;
+    readonly [name: string]: Tensor;
+}
+
+interface ClickType {
+    clickType: -1 | 0 | 1,
+    height: number | null,
+    width: number | null,
+    x: number,
+    y: number,
+}
+
+function getModelScale(w: number, h: number): number {
+    // Input images to SAM must be resized so the longest side is 1024
+    const LONG_SIDE_LENGTH = 1024;
+    const samScale = LONG_SIDE_LENGTH / Math.max(h, w);
+    return samScale;
+}
+
+function modelData(
+    {
+        clicks, tensor, modelScale, maskInput,
+    }: {
+        clicks: ClickType[];
+        tensor: Tensor;
+        modelScale: { height: number; width: number; samScale: number };
+        maskInput: Tensor | null;
+    },
+): ONNXInput {
+    const imageEmbedding = tensor;
+
+    const n = clicks.length;
+    // If there is no box input, a single padding point with
+    // label -1 and coordinates (0.0, 0.0) should be concatenated
+    // so initialize the array to support (n + 1) points.
+    const pointCoords = new Float32Array(2 * (n + 1));
+    const pointLabels = new Float32Array(n + 1);
+
+    // Add clicks and scale to what SAM expects
+    for (let i = 0; i < n; i++) {
+        pointCoords[2 * i] = clicks[i].x * modelScale.samScale;
+        pointCoords[2 * i + 1] = clicks[i].y * modelScale.samScale;
+        pointLabels[i] = clicks[i].clickType;
+    }
+
+    // Add in the extra point/label when only clicks and no box
+    // The extra point is at (0, 0) with label -1
+    pointCoords[2 * n] = 0.0;
+    pointCoords[2 * n + 1] = 0.0;
+    pointLabels[n] = -1.0;
+
+    // Create the tensor
+    const pointCoordsTensor = new Tensor('float32', pointCoords, [1, n + 1, 2]);
+    const pointLabelsTensor = new Tensor('float32', pointLabels, [1, n + 1]);
+    const imageSizeTensor = new Tensor('float32', [
+        modelScale.height,
+        modelScale.width,
+    ]);
+
+    const prevMask = maskInput ||
+        new Tensor('float32', new Float32Array(256 * 256), [1, 1, 256, 256]);
+    const hasMaskInput = new Tensor('float32', [maskInput ? 1 : 0]);
+
+    return {
+        image_embeddings: imageEmbedding,
+        point_coords: pointCoordsTensor,
+        point_labels: pointLabelsTensor,
+        orig_im_size: imageSizeTensor,
+        mask_input: prevMask,
+        has_mask_input: hasMaskInput,
+    };
+}
+
+const samPlugin: SAMPlugin = {
+    name: 'Segmeny Anything',
+    description: 'Plugin handles non-default SAM serverless function output',
+    cvat: {
+        lambda: {
+            call: {
+                async enter(
+                    plugin: SAMPlugin,
+                    taskID: number,
+                    model: any, { frame }: { frame: number },
+                ): Promise<null | APIWrapperEnterOptions> {
+                    if (model.id === plugin.data.modelID) {
+                        if (!plugin.data.session) {
+                            throw new Error('SAM plugin is not ready, session was not initialized');
+                        }
+
+                        const key = `${taskID}_${frame}`;
+                        if (plugin.data.embeddings.has(key)) {
+                            return { preventMethodCall: true };
+                        }
+                    }
+
+                    return null;
+                },
+
+                async leave(
+                    plugin: SAMPlugin,
+                    result: any,
+                    taskID: number,
+                    model: any,
+                    { frame, pos_points, neg_points }: {
+                        frame: number, pos_points: number[][], neg_points: number[][],
+                    },
+                ): Promise<
+                    {
+                        mask: number[][];
+                        bounds: [number, number, number, number];
+                    }> {
+                    if (!plugin.data.task || plugin.data.task.id !== taskID) {
+                        [plugin.data.task] = await plugin.data.core.tasks.get({ id: taskID });
+                    }
+                    const { height: imHeight, width: imWidth } = await plugin.data.task.frames.get(frame);
+                    const key = `${taskID}_${frame}`;
+                    if (model.id !== plugin.data.modelID) {
+                        return result;
+                    }
+
+                    if (result) {
+                        const bin = window.atob(result.blob);
+                        const uint8Array = new Uint8Array(bin.length);
+                        for (let i = 0; i < bin.length; i++) {
+                            uint8Array[i] = bin.charCodeAt(i);
+                        }
+                        const float32Arr = new Float32Array(uint8Array.buffer);
+                        plugin.data.embeddings.set(key, new Tensor('float32', float32Arr, [1, 256, 64, 64]));
+                    }
+
+                    const modelScale = {
+                        width: imWidth,
+                        height: imHeight,
+                        samScale: getModelScale(imWidth, imHeight),
+                    };
+
+                    const composedClicks = [...pos_points, ...neg_points].map(([x, y], index) => ({
+                        clickType: index < pos_points.length ? 1 : 0 as 0 | 1 | -1,
+                        height: null,
+                        width: null,
+                        x,
+                        y,
+                    }));
+
+                    const feeds = modelData({
+                        clicks: composedClicks,
+                        tensor: plugin.data.embeddings.get(key) as Tensor,
+                        modelScale,
+                        maskInput: plugin.data.lowResMasks.has(key) ? plugin.data.lowResMasks.get(key) as Tensor : null,
+                    });
+
+                    function toMatImage(input: number[], width: number, height: number): number[][] {
+                        const image = Array(height).fill(0);
+                        for (let i = 0; i < image.length; i++) {
+                            image[i] = Array(width).fill(0);
+                        }
+
+                        for (let i = 0; i < input.length; i++) {
+                            const row = Math.floor(i / width);
+                            const col = i % width;
+                            image[row][col] = input[i] * 255;
+                        }
+
+                        return image;
+                    }
+
+                    function onnxToImage(input: any, width: number, height: number): number[][] {
+                        return toMatImage(input, width, height);
+                    }
+
+                    const data = await (plugin.data.session as InferenceSession).run(feeds);
+                    const { masks, low_res_masks: lowResMasks } = data;
+                    const imageData = onnxToImage(masks.data, masks.dims[3], masks.dims[2]);
+                    plugin.data.lowResMasks.set(key, lowResMasks);
+
+                    const xtl = Number(data.xtl.data[0]);
+                    const xbr = Number(data.xbr.data[0]);
+                    const ytl = Number(data.ytl.data[0]);
+                    const ybr = Number(data.ybr.data[0]);
+
+                    return {
+                        mask: imageData,
+                        bounds: [xtl, ytl, xbr, ybr],
+                    };
+                },
+            },
+        },
+    },
+    data: {
+        core: null,
+        task: null,
+        modelID: 'pth-facebookresearch-sam-vit-h',
+        modelURL: '/api/lambda/sam_detector.onnx',
+        embeddings: new LRUCache({
+            // float32 tensor [256, 64, 64] is 4 MB, max 512 MB
+            max: 128,
+            updateAgeOnGet: true,
+            updateAgeOnHas: true,
+        }),
+        lowResMasks: new LRUCache({
+            // float32 tensor [1, 256, 256] is 0.25 MB, max 32 MB
+            max: 128,
+            updateAgeOnGet: true,
+            updateAgeOnHas: true,
+        }),
+        session: null,
+    },
+    callbacks: {
+        onStatusChange: null,
+    },
+};
+
+const SAMModelPlugin: ComponentBuilder = ({ core }) => {
+    samPlugin.data.core = core;
+    InferenceSession.create(samPlugin.data.modelURL).then((session) => {
+        samPlugin.data.session = session;
+        core.plugins.register(samPlugin);
+    });
+
+    return {
+        name: 'Segment Anything model',
+        destructor: () => {},
+    };
+};
+
+function register(): void {
+    if (Object.prototype.hasOwnProperty.call(window, 'cvatUI')) {
+        (window as any as { cvatUI: { registerComponent: PluginEntryPoint } })
+            .cvatUI.registerComponent(SAMModelPlugin);
+    }
+}
+
+window.addEventListener('plugins.ready', register, { once: true });

cvat-ui/react_nginx.conf

@@ -6,12 +6,16 @@ server {
         try_files $uri $uri/ /index.html;
         add_header  Cache-Control "no-cache, no-store, must-revalidate";
         add_header  Pragma "no-cache";
+        add_header  Cross-Origin-Opener-Policy "same-origin";
+        add_header  Cross-Origin-Embedder-Policy "credentialless";
         add_header  Expires 0;
     }
 
     location /assets {
         expires 1y;
         add_header Cache-Control "public";
+        add_header Cross-Origin-Embedder-Policy "require-corp";
+
         access_log off;
     }
 }

cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx

@@ -30,7 +30,7 @@ import { Canvas, convertShapesForInteractor } from 'cvat-canvas-wrapper';
 import {
     getCore, Attribute, Label, MLModel,
 } from 'cvat-core-wrapper';
-import openCVWrapper from 'utils/opencv-wrapper/opencv-wrapper';
+import openCVWrapper, { MatType } from 'utils/opencv-wrapper/opencv-wrapper';
 import {
     CombinedState, ActiveControl, ObjectType, ShapeType, ToolsBlockerState, ModelAttribute,
 } from 'reducers';
@@ -211,6 +211,7 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
         latestResponse: {
             mask: number[][],
             points: number[][],
+            bounds?: [number, number, number, number],
         };
         lastestApproximatedPoints: number[][];
         latestRequest: null | {
@@ -375,6 +376,13 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
                 const response = await core.lambda.call(jobInstance.taskId, interactor,
                     { ...data, job: jobInstance.id });
 
+                // if only mask presented, let's receive points
+                if (response.mask && !response.points) {
+                    const left = response.bounds ? response.bounds[0] : 0;
+                    const top = response.bounds ? response.bounds[1] : 0;
+                    response.points = await this.receivePointsFromMask(response.mask, left, top);
+                }
+
                 // approximation with cv.approxPolyDP
                 const approximated = await this.approximateResponsePoints(response.points);
 
@@ -383,10 +391,7 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
                     return;
                 }
 
-                this.interaction.latestResponse = {
-                    mask: response.mask,
-                    points: response.points,
-                };
+                this.interaction.latestResponse = response;
                 this.interaction.lastestApproximatedPoints = approximated;
 
                 this.setState({ pointsReceived: !!response.points.length });
@@ -400,10 +405,14 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
             }
 
             if (this.interaction.lastestApproximatedPoints.length) {
-                const height = this.interaction.latestResponse.mask.length;
-                const width = this.interaction.latestResponse.mask[0].length;
                 const maskPoints = this.interaction.latestResponse.mask.flat();
-                maskPoints.push(0, 0, width - 1, height - 1);
+                if (this.interaction.latestResponse.bounds) {
+                    maskPoints.push(...this.interaction.latestResponse.bounds);
+                } else {
+                    const height = this.interaction.latestResponse.mask.length;
+                    const width = this.interaction.latestResponse.mask[0].length;
+                    maskPoints.push(0, 0, width - 1, height - 1);
+                }
                 canvasInstance.interact({
                     enabled: true,
                     intermediateShape: {
@@ -830,7 +839,7 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
         }
     }
 
-    private constructFromPoints(points: number[][]): void {
+    private async constructFromPoints(points: number[][]): Promise<void> {
         const { convertMasksToPolygons } = this.state;
         const {
             frame, labels, curZOrder, jobInstance, activeLabelID, createAnnotations,
@@ -849,10 +858,15 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
 
             createAnnotations(jobInstance, frame, [object]);
         } else {
-            const height = this.interaction.latestResponse.mask.length;
-            const width = this.interaction.latestResponse.mask[0].length;
             const maskPoints = this.interaction.latestResponse.mask.flat();
-            maskPoints.push(0, 0, width - 1, height - 1);
+            if (this.interaction.latestResponse.bounds) {
+                maskPoints.push(...this.interaction.latestResponse.bounds);
+            } else {
+                const height = this.interaction.latestResponse.mask.length;
+                const width = this.interaction.latestResponse.mask[0].length;
+                maskPoints.push(0, 0, width - 1, height - 1);
+            }
+
             const object = new core.classes.ObjectState({
                 frame,
                 objectType: ObjectType.SHAPE,
@@ -867,18 +881,50 @@ export class ToolsControlComponent extends React.PureComponent<Props, State> {
         }
     }
 
+    private async initializeOpenCV(): Promise<void> {
+        if (!openCVWrapper.isInitialized) {
+            const hide = message.loading('OpenCV client initialization..', 0);
+            try {
+                await openCVWrapper.initialize(() => {});
+            } catch (error: any) {
+                notification.error({
+                    message: 'Could not initialize OpenCV',
+                    description: error.toString(),
+                });
+            } finally {
+                hide();
+            }
+        }
+    }
+
+    private async receivePointsFromMask(
+        mask: number[][],
+        left: number,
+        top: number,
+    ): Promise<number[]> {
+        await this.initializeOpenCV();
+
+        const src = openCVWrapper.mat.fromData(mask[0].length, mask.length, MatType.CV_8UC1, mask.flat());
+        const contours = openCVWrapper.matVector.empty();
+        try {
+            const polygons = openCVWrapper.contours.findContours(src, contours);
+            return polygons[0].map((val: number, idx: number) => {
+                if (idx % 2) {
+                    return val + top;
+                }
+
+                return val + left;
+            });
+        } finally {
+            src.delete();
+            contours.delete();
+        }
+    }
+
     private async approximateResponsePoints(points: number[][]): Promise<number[][]> {
         const { approxPolyAccuracy } = this.state;
         if (points.length > 3) {
-            if (!openCVWrapper.isInitialized) {
-                const hide = message.loading('OpenCV.js initialization..', 0);
-                try {
-                    await openCVWrapper.initialize(() => {});
-                } finally {
-                    hide();
-                }
-            }
-
+            await this.initializeOpenCV();
             const threshold = thresholdFromAccuracy(approxPolyAccuracy);
             return openCVWrapper.contours.approxPoly(points, threshold);
         }

cvat-ui/src/components/plugins-entrypoint.tsx

@@ -7,7 +7,7 @@ import { Dispatch, AnyAction } from 'redux';
 import { useDispatch } from 'react-redux';
 
 import { PluginsActionTypes, pluginActions } from 'actions/plugins-actions';
-import { getCore } from 'cvat-core-wrapper';
+import { getCore, APIWrapperEnterOptions } from 'cvat-core-wrapper';
 
 const core = getCore();
 
@@ -28,6 +28,9 @@ export type ComponentBuilder = ({
 };
 
 export type PluginEntryPoint = (componentBuilder: ComponentBuilder) => void;
+export type {
+    APIWrapperEnterOptions,
+};
 
 function PluginEntrypoint(): null {
     const dispatch = useDispatch();

cvat-ui/src/cvat-core-wrapper.ts

@@ -23,6 +23,7 @@ import Comment from 'cvat-core/src/comment';
 import User from 'cvat-core/src/user';
 import Organization from 'cvat-core/src/organization';
 import { Dumper } from 'cvat-core/src/annotation-formats';
+import { APIWrapperEnterOptions } from 'cvat-core/src/plugins';
 
 const cvat: any = _cvat;
 
@@ -66,4 +67,5 @@ export type {
     SerializedLabel,
     StorageData,
     ModelProvider,
+    APIWrapperEnterOptions,
 };

cvat-ui/src/utils/opencv-wrapper/opencv-wrapper.ts

 // Copyright (C) 2020-2022 Intel Corporation
+// Copyright (C) 2023 CVAT.ai Corporation
 //
 // SPDX-License-Identifier: MIT
 
@@ -15,7 +16,17 @@ export interface Segmentation {
     intelligentScissorsFactory: (onChangeToolsBlockerState:(event:string)=>void) => IntelligentScissors;
 }
 
+export interface MatSpace {
+    empty: () => any;
+    fromData: (width: number, height: number, type: MatType, data: number[]) => any;
+}
+
+export interface MatVectorSpace {
+    empty: () => any;
+}
+
 export interface Contours {
+    findContours: (src: any, contours: any) => number[][];
     approxPoly: (points: number[] | any, threshold: number, closed?: boolean) => number[][];
 }
 
@@ -27,6 +38,12 @@ export interface Tracking {
     trackerMIL: OpenCVTracker;
 }
 
+export enum MatType {
+    CV_8UC1,
+    CV_8UC3,
+    CV_8UC4,
+}
+
 export class OpenCVWrapper {
     private initialized: boolean;
     private cv: any;
@@ -40,6 +57,12 @@ export class OpenCVWrapper {
         this.injectionProcess = null;
     }
 
+    private checkInitialization() {
+        if (!this.initialized) {
+            throw new Error('Need to initialize OpenCV first');
+        }
+    }
+
     private async inject(): Promise<void> {
         const response = await fetch(`${baseURL}/opencv/opencv.js`);
         if (response.status !== 200) {
@@ -109,13 +132,55 @@ export class OpenCVWrapper {
         return !!this.injectionProcess;
     }
 
-    public get contours(): Contours {
-        if (!this.initialized) {
-            throw new Error('Need to initialize OpenCV first');
-        }
+    public get mat(): MatSpace {
+        this.checkInitialization();
+        const { cv } = this;
+        return {
+            empty: () => new cv.Mat(),
+
+            fromData: (width: number, height: number, type: MatType, data: number[]) => {
+                const typeToCVType = {
+                    [MatType.CV_8UC1]: cv.CV_8UC1,
+                    [MatType.CV_8UC3]: cv.CV_8UC3,
+                    [MatType.CV_8UC4]: cv.CV_8UC4,
+                };
 
+                const mat = cv.matFromArray(height, width, typeToCVType[type], data);
+                return mat;
+            },
+        };
+    }
+
+    public get matVector(): MatVectorSpace {
+        this.checkInitialization();
         const { cv } = this;
         return {
+            empty: () => new cv.MatVector(),
+        };
+    }
+
+    public get contours(): Contours {
+        this.checkInitialization();
+        const { cv } = this;
+        return {
+            findContours: (src: any, contours: any): number[][] => {
+                const jsContours: number[][] = [];
+                const hierarchy = new cv.Mat();
+                try {
+                    cv.findContours(src, contours, hierarchy, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_NONE);
+                    for (let i = 0; i < contours.size(); i++) {
+                        const contour = contours.get(i);
+                        jsContours.push(Array.from(contour.data32S));
+                        contour.delete();
+                    }
+                } finally {
+                    hierarchy.delete();
+                }
+
+                const longest = jsContours.sort((arr1, arr2) => arr2.length - arr1.length)[0];
+                return [longest];
+            },
+
             approxPoly: (points: number[] | number[][], threshold: number, closed = true): number[][] => {
                 const isArrayOfArrays = Array.isArray(points[0]);
                 if (points.length < 3) {
@@ -143,10 +208,7 @@ export class OpenCVWrapper {
     }
 
     public get segmentation(): Segmentation {
-        if (!this.initialized) {
-            throw new Error('Need to initialize OpenCV first');
-        }
-
+        this.checkInitialization();
         return {
             intelligentScissorsFactory:
             (onChangeToolsBlockerState:
@@ -155,18 +217,14 @@ export class OpenCVWrapper {
     }
 
     public get imgproc(): ImgProc {
-        if (!this.initialized) {
-            throw new Error('Need to initialize OpenCV first');
-        }
+        this.checkInitialization();
         return {
             hist: () => new HistogramEqualizationImplementation(this.cv),
         };
     }
 
     public get tracking(): Tracking {
-        if (!this.initialized) {
-            throw new Error('Need to initialize OpenCV first');
-        }
+        this.checkInitialization();
         return {
             trackerMIL: {
                 model: () => new TrackerMImplementation(this.cv),

cvat-ui/tsconfig.json

@@ -16,5 +16,5 @@
     "jsx": "preserve",
     "baseUrl": "src"
   },
-  "include": ["./index.d.ts", "src/index.tsx", "src/assets/index.d.ts", "src"]
+  "include": ["./index.d.ts", "src/index.tsx", "src/assets/index.d.ts", "plugins/**/*", "src"]
 }

cvat-ui/webpack.config.js

@@ -14,9 +14,9 @@ const CopyPlugin = require('copy-webpack-plugin');
 
 module.exports = (env) => {
     const defaultAppConfig = path.join(__dirname, 'src/config.tsx');
-    const defaultPlugins = [];
+    const defaultPlugins = ['plugins/sam_plugin'];
     const appConfigFile = process.env.UI_APP_CONFIG ? process.env.UI_APP_CONFIG : defaultAppConfig;
-    const pluginsList = process.env.CLIENT_PLUGINS ? process.env.CLIENT_PLUGINS.split(':')
+    const pluginsList = process.env.CLIENT_PLUGINS ? [...defaultPlugins, process.env.CLIENT_PLUGINS.split(':')]
         .map((s) => s.trim()).filter((s) => !!s) : defaultPlugins
 
     const transformedPlugins = pluginsList
@@ -46,6 +46,9 @@ module.exports = (env) => {
             publicPath: '/',
         },
         devServer: {
+            devMiddleware: {
+                writeToDisk: true,
+            },
             compress: false,
             host: process.env.CVAT_UI_HOST || 'localhost',
             client: {
@@ -56,6 +59,12 @@ module.exports = (env) => {
             static: {
                 directory: path.join(__dirname, 'dist'),
             },
+            headers: {
+                // to enable SharedArrayBuffer and ONNX multithreading
+                // https://cloudblogs.microsoft.com/opensource/2021/09/02/onnx-runtime-web-running-your-machine-learning-model-in-browser/
+                'Cross-Origin-Opener-Policy': 'same-origin',
+                'Cross-Origin-Embedder-Policy': 'credentialless',
+            },
             proxy: [
                 {
                     context: (param) =>
@@ -191,6 +200,10 @@ module.exports = (env) => {
                         from: '../cvat-data/src/ts/3rdparty/avc.wasm',
                         to: 'assets/3rdparty/',
                     },
+                    {
+                        from: '../node_modules/onnxruntime-web/dist/*.wasm',
+                        to  : 'assets/[name][ext]',
+                    },
                 ],
             }),
         ],

cvat/apps/lambda_manager/urls.py

@@ -25,5 +25,6 @@ router.register('requests', views.RequestViewSet, basename='request')
 # GET  /api/lambda/requests/<int:rid> - get status of the request
 # DEL  /api/lambda/requests/<int:rid> - cancel a request (don't delete)
 urlpatterns = [
-    path('api/lambda/', include(router.urls))
+    path('api/lambda/', include(router.urls)),
+    path('api/lambda/sam_detector.onnx', views.ONNXDetector)
 ]

cvat/apps/lambda_manager/views.py

 # Copyright (C) 2022 Intel Corporation
-# Copyright (C) 2022 CVAT.ai Corporation
+# Copyright (C) 2022-2023 CVAT.ai Corporation
 #
 # SPDX-License-Identifier: MIT
 
@@ -7,6 +7,8 @@ import base64
 import json
 import os
 import textwrap
+import glob
+from django_sendfile import sendfile
 from copy import deepcopy
 from enum import Enum
 from functools import wraps
@@ -861,3 +863,11 @@ class RequestViewSet(viewsets.ViewSet):
         queue = LambdaQueue()
         job = queue.fetch_job(pk)
         job.delete()
+
+def ONNXDetector(request):
+    dirname = os.path.join(settings.STATIC_ROOT, 'lambda_manager')
+    pattern = os.path.join(dirname, 'decoder.onnx')
+    path = glob.glob(pattern)[0]
+    response = sendfile(request, path)
+    response['Cache-Control'] = "public, max-age=604800"
+    return response

package.json

@@ -125,5 +125,6 @@
     "@types/react-redux": "^7.1.24",
     "@types/react-router-dom": "^5.3.3",
     "@types/prettier": "2.4.1"
-  }
+  },
+  "dependencies": {}
 }

serverless/pytorch/facebookresearch/sam/nuclio/main.py

@@ -17,15 +17,13 @@ def init_context(context):
 def handler(context, event):
     context.logger.info("call handler")
     data = event.body
-    pos_points = data["pos_points"]
-    neg_points = data["neg_points"]
     buf = io.BytesIO(base64.b64decode(data["image"]))
     image = Image.open(buf)
     image = image.convert("RGB")  #  to make sure image comes in RGB
-    mask, polygon = context.user_data.model.handle(image, pos_points, neg_points)
+    features = context.user_data.model.handle(image)
+
     return context.Response(body=json.dumps({
-            'points': polygon,
-            'mask': mask.tolist(),
+            'blob': base64.b64encode((features.cpu().numpy() if features.is_cuda else features.numpy())).decode(),
         }),
         headers={},
         content_type='application/json',

serverless/pytorch/facebookresearch/sam/nuclio/model_handler.py

@@ -32,37 +32,11 @@ class ModelHandler:
         self.sam_checkpoint = "/opt/nuclio/sam/sam_vit_h_4b8939.pth"
         self.model_type = "vit_h"
         self.latest_image = None
-        self.latest_low_res_masks = None
         sam_model = sam_model_registry[self.model_type](checkpoint=self.sam_checkpoint)
         sam_model.to(device=self.device)
         self.predictor = SamPredictor(sam_model)
 
-    def handle(self, image, pos_points, neg_points):
-        # latest image is kept in memory because function is always run-time after startup
-        # we use to avoid computing emeddings twice for the same image
-        is_the_same_image = self.latest_image is not None and np.array_equal(np.array(image), self.latest_image)
-        if not is_the_same_image:
-            self.latest_low_res_masks = None
-            numpy_image = np.array(image)
-            self.predictor.set_image(numpy_image)
-            self.latest_image = numpy_image
-        # we assume that pos_points and neg_points are of type:
-        # np.array[[x, y], [x, y], ...]
-        input_points = np.array(pos_points)
-        input_labels = np.array([1] * len(pos_points))
-
-        if len(neg_points):
-            input_points = np.concatenate([input_points, neg_points], axis=0)
-            input_labels = np.concatenate([input_labels, np.array([0] * len(neg_points))], axis=0)
-
-        masks, _, low_res_masks = self.predictor.predict(
-            point_coords=input_points,
-            point_labels=input_labels,
-            mask_input = self.latest_low_res_masks,
-            multimask_output=False
-        )
-        self.latest_low_res_masks = low_res_masks
-        object_mask = np.array(masks[0], dtype=np.uint8)
-        cv2.normalize(object_mask, object_mask, 0, 255, cv2.NORM_MINMAX)
-        polygon = convert_mask_to_polygon(object_mask)
-        return object_mask, polygon
+    def handle(self, image):
+        self.predictor.set_image(np.array(image))
+        features = self.predictor.get_image_embedding()
+        return features

yarn.lock

@@ -1434,6 +1434,59 @@
     "@nodelib/fs.scandir" "2.1.5"
     fastq "^1.6.0"
 
+"@protobufjs/aspromise@^1.1.1", "@protobufjs/aspromise@^1.1.2":
+  version "1.1.2"
+  resolved "https://registry.yarnpkg.com/@protobufjs/aspromise/-/aspromise-1.1.2.tgz#9b8b0cc663d669a7d8f6f5d0893a14d348f30fbf"
+  integrity sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==
+
+"@protobufjs/base64@^1.1.2":
+  version "1.1.2"
+  resolved "https://registry.yarnpkg.com/@protobufjs/base64/-/base64-1.1.2.tgz#4c85730e59b9a1f1f349047dbf24296034bb2735"
+  integrity sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==
+
+"@protobufjs/codegen@^2.0.4":
+  version "2.0.4"
+  resolved "https://registry.yarnpkg.com/@protobufjs/codegen/-/codegen-2.0.4.tgz#7ef37f0d010fb028ad1ad59722e506d9262815cb"
+  integrity sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==
+
+"@protobufjs/eventemitter@^1.1.0":
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz#355cbc98bafad5978f9ed095f397621f1d066b70"
+  integrity sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==
+
+"@protobufjs/fetch@^1.1.0":
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/@protobufjs/fetch/-/fetch-1.1.0.tgz#ba99fb598614af65700c1619ff06d454b0d84c45"
+  integrity sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==
+  dependencies:
+    "@protobufjs/aspromise" "^1.1.1"
+    "@protobufjs/inquire" "^1.1.0"
+
+"@protobufjs/float@^1.0.2":
+  version "1.0.2"
+  resolved "https://registry.yarnpkg.com/@protobufjs/float/-/float-1.0.2.tgz#5e9e1abdcb73fc0a7cb8b291df78c8cbd97b87d1"
+  integrity sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==
+
+"@protobufjs/inquire@^1.1.0":
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/@protobufjs/inquire/-/inquire-1.1.0.tgz#ff200e3e7cf2429e2dcafc1140828e8cc638f089"
+  integrity sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==
+
+"@protobufjs/path@^1.1.2":
+  version "1.1.2"
+  resolved "https://registry.yarnpkg.com/@protobufjs/path/-/path-1.1.2.tgz#6cc2b20c5c9ad6ad0dccfd21ca7673d8d7fbf68d"
+  integrity sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==
+
+"@protobufjs/pool@^1.1.0":
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/@protobufjs/pool/-/pool-1.1.0.tgz#09fd15f2d6d3abfa9b65bc366506d6ad7846ff54"
+  integrity sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==
+
+"@protobufjs/utf8@^1.1.0":
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/@protobufjs/utf8/-/utf8-1.1.0.tgz#a777360b5b39a1a2e5106f8e858f2fd2d060c570"
+  integrity sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==
+
 "@sinclair/typebox@^0.25.16":
   version "0.25.24"
   resolved "https://registry.yarnpkg.com/@sinclair/typebox/-/typebox-0.25.24.tgz#8c7688559979f7079aacaf31aa881c3aa410b718"
@@ -1700,6 +1753,18 @@
   resolved "https://registry.yarnpkg.com/@types/lodash/-/lodash-4.14.191.tgz#09511e7f7cba275acd8b419ddac8da9a6a79e2fa"
   integrity sha512-BdZ5BCCvho3EIXw6wUCXHe7rS53AIDPLE+JzwgT+OsJk53oBfbSmZZ7CX4VaRoN78N+TJpFi9QPlfIVNmJYWxQ==
 
+"@types/long@^4.0.1":
+  version "4.0.2"
+  resolved "https://registry.yarnpkg.com/@types/long/-/long-4.0.2.tgz#b74129719fc8d11c01868010082d483b7545591a"
+  integrity sha512-MqTGEo5bj5t157U6fA/BiDynNkn0YknVdh48CMPkTSpFTVmvao5UQmm7uEF6xBEo7qIMAlY/JSleYaE6VOdpaA==
+
+"@types/lru-cache@^7.10.10":
+  version "7.10.10"
+  resolved "https://registry.yarnpkg.com/@types/lru-cache/-/lru-cache-7.10.10.tgz#3fa937c35ff4b3f6753d5737915c9bf8e693a713"
+  integrity sha512-nEpVRPWW9EBmx2SCfNn3ClYxPL7IktPX12HhIoSc/H5mMjdeW3+YsXIpseLQ2xF35+OcpwKQbEUw5VtqE4PDNA==
+  dependencies:
+    lru-cache "*"
+
 "@types/mdast@^3.0.0":
   version "3.0.10"
   resolved "https://registry.yarnpkg.com/@types/mdast/-/mdast-3.0.10.tgz#4724244a82a4598884cbbe9bcfd73dff927ee8af"
@@ -1732,6 +1797,11 @@
   resolved "https://registry.yarnpkg.com/@types/node/-/node-18.11.18.tgz#8dfb97f0da23c2293e554c5a50d61ef134d7697f"
   integrity sha512-DHQpWGjyQKSHj3ebjFI/wRKcqQcdR+MoFBygntYOZytCqNfkd2ZC4ARDJ2DQqhjH5p85Nnd3jhUJIXrszFX/JA==
 
+"@types/node@>=13.7.0":
+  version "18.15.11"
+  resolved "https://registry.yarnpkg.com/@types/node/-/node-18.15.11.tgz#b3b790f09cb1696cffcec605de025b088fa4225f"
+  integrity sha512-E5Kwq2n4SbMzQOn6wnmBjuK9ouqlURrcZDVfbo9ftDDTFt3nk7ZKK4GMOzoYgnpQJKcxwQw+lGaBvvlMo0qN/Q==
+
 "@types/normalize-package-data@^2.4.0":
   version "2.4.1"
   resolved "https://registry.yarnpkg.com/@types/normalize-package-data/-/normalize-package-data-2.4.1.tgz#d3357479a0fdfdd5907fe67e17e0a85c906e1301"
@@ -1752,7 +1822,7 @@
   resolved "https://registry.yarnpkg.com/@types/polylabel/-/polylabel-1.0.5.tgz#9262f269de36f1e9248aeb9dee0ee9d10065e043"
   integrity sha512-gnaNmo1OJiYNBFAZMZdqLZ3hKx2ee4ksAzqhKWBxuQ61PmhINHMcvIqsGmyCD1WFKCkwRt9NFhMSmKE6AgYY+w==
 
-"@types/prettier@2.4.1", "@types/prettier@^2.1.5":
+"@types/prettier@^2.1.5":
   version "2.4.1"
   resolved "https://registry.yarnpkg.com/@types/prettier/-/prettier-2.4.1.tgz#e1303048d5389563e130f5bdd89d37a99acb75eb"
   integrity sha512-Fo79ojj3vdEZOHg3wR9ksAMRz4P3S5fDB5e/YWZiFnyFQI1WY2Vftu9XoXVVtJfxB7Bpce/QTqWSSntkz2Znrw==
@@ -1785,12 +1855,12 @@
     "@types/react" "*"
     "@types/reactcss" "*"
 
-"@types/react-dom@^16.9.14", "@types/react-dom@^18.0.5":
-  version "18.0.11"
-  resolved "https://registry.yarnpkg.com/@types/react-dom/-/react-dom-18.0.11.tgz#321351c1459bc9ca3d216aefc8a167beec334e33"
-  integrity sha512-O38bPbI2CWtgw/OoQoY+BRelw7uysmXbWvw3nLWO21H1HSh+GOlqPuXshJfjmpNlKiiSDG9cc1JZAaMmVdcTlw==
+"@types/react-dom@^16.9.14":
+  version "16.9.19"
+  resolved "https://registry.yarnpkg.com/@types/react-dom/-/react-dom-16.9.19.tgz#6a139c26b02dec533a7fa131f084561babb10a8f"
+  integrity sha512-xC8D280Bf6p0zguJ8g62jcEOKZiUbx9sIe6O3tT/lKfR87A7A6g65q13z6D5QUMIa/6yFPkNhqjF5z/VVZEYqQ==
   dependencies:
-    "@types/react" "*"
+    "@types/react" "^16"
 
 "@types/react-grid-layout@^1.3.2":
   version "1.3.2"
@@ -1799,7 +1869,7 @@
   dependencies:
     "@types/react" "*"
 
-"@types/react-redux@^7.1.18", "@types/react-redux@^7.1.24":
+"@types/react-redux@^7.1.18":
   version "7.1.25"
   resolved "https://registry.yarnpkg.com/@types/react-redux/-/react-redux-7.1.25.tgz#de841631205b24f9dfb4967dd4a7901e048f9a88"
   integrity sha512-bAGh4e+w5D8dajd6InASVIyCo4pZLJ66oLb80F9OBLO1gKESbZcRCJpTT6uLXX+HAB57zw1WTdwJdAsewuTweg==
@@ -1809,7 +1879,7 @@
     hoist-non-react-statics "^3.3.0"
     redux "^4.0.0"
 
-"@types/react-router-dom@^5.1.9", "@types/react-router-dom@^5.3.3":
+"@types/react-router-dom@^5.1.9":
   version "5.3.3"
   resolved "https://registry.yarnpkg.com/@types/react-router-dom/-/react-router-dom-5.3.3.tgz#e9d6b4a66fcdbd651a5f106c2656a30088cc1e83"
   integrity sha512-kpqnYK4wcdm5UaWI3fLcELopqLrHgLqNsdpHauzlQktfkHL3npOSwtj1Uz9oKBAzs7lFtVkV8j83voAz2D8fhw==
@@ -1833,10 +1903,19 @@
   dependencies:
     "@types/react" "*"
 
-"@types/react@*", "@types/react@^16.14.15", "@types/react@^17.0.30":
-  version "17.0.55"
-  resolved "https://registry.yarnpkg.com/@types/react/-/react-17.0.55.tgz#f94eac1a37929cd86d1cc084c239c08dcfd10e5f"
-  integrity sha512-kBcAhmT8RivFDYxHdy8QfPKu+WyfiiGjdPb9pIRtd6tj05j0zRHq5DBGW5Ogxv5cwSKd93BVgUk/HZ4I9p3zNg==
+"@types/react@*":
+  version "17.0.59"
+  resolved "https://registry.yarnpkg.com/@types/react/-/react-17.0.59.tgz#5aa4e161a356fcb824d81f166e01bad9e82243bb"
+  integrity sha512-gSON5zWYIGyoBcycCE75E9+r6dCC2dHdsrVkOEiIYNU5+Q28HcBAuqvDuxHcCbMfHBHdeT5Tva/AFn3rnMKE4g==
+  dependencies:
+    "@types/prop-types" "*"
+    "@types/scheduler" "*"
+    csstype "^3.0.2"
+
+"@types/react@^16", "@types/react@^16.14.15":
+  version "16.14.41"
+  resolved "https://registry.yarnpkg.com/@types/react/-/react-16.14.41.tgz#55c4e7ebb736ca4e2379e97a4e0c1803150ed8d4"
+  integrity sha512-h+joCKF2r5rdECoM1U8WCEIHBp5/0TSR5Nyq8gtnnYY1n2WqGuj3indYqTjMb2/b5g2rfxJV6u4jUFq95lbT6Q==
   dependencies:
     "@types/prop-types" "*"
     "@types/scheduler" "*"
@@ -3825,7 +3904,7 @@ custom-error-instance@2.1.1:
     three "^0.126.1"
 
 "cvat-canvas@link:./cvat-canvas":
-  version "2.16.3"
+  version "2.16.4"
   dependencies:
     "@types/fabric" "^4.5.7"
     "@types/polylabel" "^1.0.5"
@@ -5008,6 +5087,11 @@ flat-cache@^3.0.4:
     flatted "^3.1.0"
     rimraf "^3.0.2"
 
+flatbuffers@^1.12.0:
+  version "1.12.0"
+  resolved "https://registry.yarnpkg.com/flatbuffers/-/flatbuffers-1.12.0.tgz#72e87d1726cb1b216e839ef02658aa87dcef68aa"
+  integrity sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ==
+
 flatted@^3.1.0:
   version "3.2.7"
   resolved "https://registry.yarnpkg.com/flatted/-/flatted-3.2.7.tgz#609f39207cb614b89d0765b477cb2d437fbf9787"
@@ -5346,6 +5430,11 @@ graceful-fs@^4.1.15, graceful-fs@^4.1.2, graceful-fs@^4.2.4, graceful-fs@^4.2.6,
   resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.2.10.tgz#147d3a006da4ca3ce14728c7aefc287c367d7a6c"
   integrity sha512-9ByhssR2fPVsNZj478qUUbKfmL0+t5BDVyjShtyZZLiK7ZDAArFFfopyOTj0M05wE2tJPisA4iTnnXl2YoPvOA==
 
+guid-typescript@^1.0.9:
+  version "1.0.9"
+  resolved "https://registry.yarnpkg.com/guid-typescript/-/guid-typescript-1.0.9.tgz#e35f77003535b0297ea08548f5ace6adb1480ddc"
+  integrity sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==
+
 handle-thing@^2.0.0:
   version "2.0.1"
   resolved "https://registry.yarnpkg.com/handle-thing/-/handle-thing-2.0.1.tgz#857f79ce359580c340d43081cc648970d0bb234e"
@@ -7182,6 +7271,11 @@ log-update@^4.0.0:
     slice-ansi "^4.0.0"
     wrap-ansi "^6.2.0"
 
+long@^4.0.0:
+  version "4.0.0"
+  resolved "https://registry.yarnpkg.com/long/-/long-4.0.0.tgz#9a7b71cfb7d361a194ea555241c92f7468d5bf28"
+  integrity sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA==
+
 longest-streak@^2.0.0:
   version "2.0.4"
   resolved "https://registry.yarnpkg.com/longest-streak/-/longest-streak-2.0.4.tgz#b8599957da5b5dab64dee3fe316fa774597d90e4"
@@ -7211,6 +7305,11 @@ lowercase-keys@^2.0.0:
   resolved "https://registry.yarnpkg.com/lowercase-keys/-/lowercase-keys-2.0.0.tgz#2603e78b7b4b0006cbca2fbcc8a3202558ac9479"
   integrity sha512-tqNXrS78oMOE73NMxK4EMLQsQowWf8jKooH9g7xPavRT706R6bkQJ6DY2Te7QukaZsulxa30wQ7bk0pm4XiHmA==
 
+lru-cache@*, lru-cache@^9.1.1:
+  version "9.1.1"
+  resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-9.1.1.tgz#c58a93de58630b688de39ad04ef02ef26f1902f1"
+  integrity sha512-65/Jky17UwSb0BuB9V+MyDpsOtXKmYwzhyl+cOa9XUiI4uV2Ouy/2voFP3+al0BjZbJgMBD8FojMpAf+Z+qn4A==
+
 lru-cache@^5.1.1:
   version "5.1.1"
   resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-5.1.1.tgz#1da27e6710271947695daf6848e847f01d84b920"
@@ -8255,6 +8354,30 @@ onetime@^6.0.0:
   dependencies:
     mimic-fn "^4.0.0"
 
+onnx-proto@^4.0.4:
+  version "4.0.4"
+  resolved "https://registry.yarnpkg.com/onnx-proto/-/onnx-proto-4.0.4.tgz#2431a25bee25148e915906dda0687aafe3b9e044"
+  integrity sha512-aldMOB3HRoo6q/phyB6QRQxSt895HNNw82BNyZ2CMh4bjeKv7g/c+VpAFtJuEMVfYLMbRx61hbuqnKceLeDcDA==
+  dependencies:
+    protobufjs "^6.8.8"
+
+onnxruntime-common@~1.14.0:
+  version "1.14.0"
+  resolved "https://registry.yarnpkg.com/onnxruntime-common/-/onnxruntime-common-1.14.0.tgz#2bb5dac5261269779aa5fb6536ca379657de8bf6"
+  integrity sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==
+
+onnxruntime-web@^1.14.0:
+  version "1.14.0"
+  resolved "https://registry.yarnpkg.com/onnxruntime-web/-/onnxruntime-web-1.14.0.tgz#c8cee538781b1d4c1c6b043934f4a3e6ddf1466e"
+  integrity sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==
+  dependencies:
+    flatbuffers "^1.12.0"
+    guid-typescript "^1.0.9"
+    long "^4.0.0"
+    onnx-proto "^4.0.4"
+    onnxruntime-common "~1.14.0"
+    platform "^1.3.6"
+
 open@^8.0.9:
   version "8.4.0"
   resolved "https://registry.yarnpkg.com/open/-/open-8.4.0.tgz#345321ae18f8138f82565a910fdc6b39e8c244f8"
@@ -9108,6 +9231,25 @@ property-information@^6.0.0:
   resolved "https://registry.yarnpkg.com/property-information/-/property-information-6.2.0.tgz#b74f522c31c097b5149e3c3cb8d7f3defd986a1d"
   integrity sha512-kma4U7AFCTwpqq5twzC1YVIDXSqg6qQK6JN0smOw8fgRy1OkMi0CYSzFmsy6dnqSenamAtj0CyXMUJ1Mf6oROg==
 
+protobufjs@^6.8.8:
+  version "6.11.3"
+  resolved "https://registry.yarnpkg.com/protobufjs/-/protobufjs-6.11.3.tgz#637a527205a35caa4f3e2a9a4a13ddffe0e7af74"
+  integrity sha512-xL96WDdCZYdU7Slin569tFX712BxsxslWwAfAhCYjQKGTq7dAU91Lomy6nLLhh/dyGhk/YH4TwTSRxTzhuHyZg==
+  dependencies:
+    "@protobufjs/aspromise" "^1.1.2"
+    "@protobufjs/base64" "^1.1.2"
+    "@protobufjs/codegen" "^2.0.4"
+    "@protobufjs/eventemitter" "^1.1.0"
+    "@protobufjs/fetch" "^1.1.0"
+    "@protobufjs/float" "^1.0.2"
+    "@protobufjs/inquire" "^1.1.0"
+    "@protobufjs/path" "^1.1.2"
+    "@protobufjs/pool" "^1.1.0"
+    "@protobufjs/utf8" "^1.1.0"
+    "@types/long" "^4.0.1"
+    "@types/node" ">=13.7.0"
+    long "^4.0.0"
+
 proxy-addr@~2.0.7:
   version "2.0.7"
   resolved "https://registry.yarnpkg.com/proxy-addr/-/proxy-addr-2.0.7.tgz#f19fe69ceab311eeb94b42e70e8c2070f9ba1025"