CVAT-3D-M3 (#2768)

* CVAT-3D Updated the Mime Types with Bin Support, added dependency of open3D

* CVAT-3D Added additional column as Dimension for engine_task table and created a relatedfiles table for PCD to Image mapping.

* Added Support for 3D file Upload in BIN and PCD.

* Added Dimension attribute defaulting to 2D for importer and exporter.

* Added props passing for dimension attribute, filtering of import, Migration Scripts and Dimension attribute for MpegChunk Writers

* Modified code as per review comments

* Updated Unit test cases for 3D task creation

* Refactored Dimension Enum in UI and backend code

* Resolving conflicts

* Updated Unit Test Case

* Refactored TaskDimension to DimensionType, Simplified usage of Dimension accross classes

* Removing manually created test files

* Removing old pcd mime-type mapping

* Added test files generated by synthetic data using open3d

* Merged with develop branch latest changes

* Added libraries required for open3d

* Added files

* Added synthethic pcd,bin and img test files

* Modified test file name

* Trigger travis ci

* Modified test case to ignore 3D preview images

* Trigger notification

* Deleting DS Store files

* Modified test cases as per review comments

* Checking pre-commit hook

* Fixed Lint issues - precommit hook verification

* Added changes for CVAT-3D Milestone2 changes - Frame Navigation, photo context hide and show

* Modified changes

* Added canvas3D for 3D Perspective

* Added missing files

* Added code to get image context for 3D view

* Codacy check for stylesheet

* Modified frame navigantion for 3D View

* Modified style for context-image

* Trigger notification

* Added Support for 3D file Upload in BIN and PCD.

* Added props passing for dimension attribute, filtering of import, Migration Scripts and Dimension attribute for MpegChunk Writers

* Modified code as per review comments

* Refactored Dimension Enum in UI and backend code

* Merged with develop branch latest changes

* Added files

* Added changes for CVAT-3D Milestone2 changes - Frame Navigation, photo context hide and show

* Modified changes

* Added canvas3D for 3D Perspective

* Added missing files

* Added code to get image context for 3D view

* Codacy check for stylesheet

* Modified frame navigantion for 3D View

* Modified style for context-image

* Changed cvat-data lint issues

* Modified to use opencv as per review comments

* Removed unwanted imports

* Fixed css and added usage of hooks

* Merged Develop branch code

* Removed unused data structures

* Removed unused data structures

* Refactored unused data structures

* Added three js dependency in cvat ui package-lock.json

* Merged develop branch code and refactored code

* Fixed snyk issue

* Modified Camera Icon in photo-context

* Update icons.tsx

* Remove unused svg file

* Modified changelog file

* Added changes for CVAT-3D Milestone3

* Added missing line as per codacy check

Co-authored-by: cdp <cdp123>

CHANGELOG.md

@@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - CVAT-3D: Load all frames corresponding to the job instance
   (<https://github.com/openvinotoolkit/cvat/pull/2645>)
 - Intelligent scissors with OpenCV javascript (<https://github.com/openvinotoolkit/cvat/pull/2689>)
+- CVAT-3D: Visualize 3D point cloud spaces in 3D View, Top View Side View and Front View (<https://github.com/openvinotoolkit/cvat/pull/2768>)
 - [Inside Outside Guidence](https://github.com/shiyinzhang/Inside-Outside-Guidance) serverless
   function for interative segmentation
 - Pre-built [cvat_server](https://hub.docker.com/r/openvino/cvat_server) and

components/analytics/docker-compose.analytics.yml

@@ -70,9 +70,9 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
     environment:
-        LOGSTASH_OUTPUT_HOST: elasticsearch:9200
-        LOGSTASH_OUTPUT_USER:
-        LOGSTASH_OUTPUT_PASS:
+      LOGSTASH_OUTPUT_HOST: elasticsearch:9200
+      LOGSTASH_OUTPUT_USER:
+      LOGSTASH_OUTPUT_PASS:
     depends_on: ['cvat_elasticsearch']
     restart: always
 

cvat-canvas3d/README.md

@@ -28,7 +28,6 @@ npm run build -- --mode=development     # without a minification
 interface Canvas3d {
   html(): HTMLDivElement;
   setup(frameData: any): void;
-  fitCanvas(): void;
   mode(): Mode;
   isAbleToChangeFrame(): boolean;
   render(): void;
@@ -46,5 +45,4 @@ console.log('Current mode is ', window.canvas.mode());
 
 // Put canvas to a html container
 htmlContainer.appendChild(canvas.html());
-canvas.fitCanvas();
 ```

cvat-canvas3d/package-lock.json

@@ -1771,6 +1771,11 @@
             "node-releases": "^1.1.42"
           }
         },
+        "camera-controls": {
+          "version": "1.25.3",
+          "resolved": "https://registry.npmjs.org/camera-controls/-/camera-controls-1.25.3.tgz",
+          "integrity": "sha512-hcPgd3ly69N+HU8LRsyCmDh4N5YAbyDLB2LwbwOJdFoXomvpEi1BPGIJCbHbzp/n+CrisEjgeCHJprqy7GT2Dg=="
+        },
         "caniuse-lite": {
           "version": "1.0.30001016",
           "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001016.tgz",
@@ -2321,6 +2326,11 @@
         }
       }
     },
+    "camera-controls": {
+      "version": "1.25.3",
+      "resolved": "https://registry.npmjs.org/camera-controls/-/camera-controls-1.25.3.tgz",
+      "integrity": "sha512-hcPgd3ly69N+HU8LRsyCmDh4N5YAbyDLB2LwbwOJdFoXomvpEi1BPGIJCbHbzp/n+CrisEjgeCHJprqy7GT2Dg=="
+    },
     "caniuse-lite": {
       "version": "1.0.30000985",
       "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30000985.tgz",

cvat-canvas3d/package.json

@@ -38,6 +38,7 @@
     "webpack-dev-server": "^3.11.0"
   },
   "dependencies": {
-    "three": "^0.125.0"
+    "three": "^0.125.0",
+    "camera-controls": "^1.25.3"
   }
 }

cvat-canvas3d/src/typescript/canvas3d.ts

@@ -5,18 +5,18 @@
 import pjson from '../../package.json';
 import { Canvas3dController, Canvas3dControllerImpl } from './canvas3dController';
 import { Canvas3dModel, Canvas3dModelImpl, Mode } from './canvas3dModel';
-import { Canvas3dView, Canvas3dViewImpl } from './canvas3dView';
+import { Canvas3dView, Canvas3dViewImpl, ViewsDOM } from './canvas3dView';
 import { Master } from './master';
 
 const Canvas3dVersion = pjson.version;
 
 interface Canvas3d {
-    html(): any;
+    html(): ViewsDOM;
     setup(frameData: any): void;
     isAbleToChangeFrame(): boolean;
-    fitCanvas(): void;
     mode(): Mode;
     render(): void;
+    keyControls(keys: KeyboardEvent): void;
 }
 
 class Canvas3dImpl implements Canvas3d {
@@ -30,10 +30,14 @@ class Canvas3dImpl implements Canvas3d {
         this.view = new Canvas3dViewImpl(this.model, this.controller);
     }
 
-    public html(): any {
+    public html(): ViewsDOM {
         return this.view.html();
     }
 
+    public keyControls(keys: KeyboardEvent): void {
+        this.view.keyControls(keys);
+    }
+
     public render(): void {
         this.view.render();
     }
@@ -49,10 +53,6 @@ class Canvas3dImpl implements Canvas3d {
     public isAbleToChangeFrame(): boolean {
         return this.model.isAbleToChangeFrame();
     }
-
-    public fitCanvas(): void {
-        this.model.fitCanvas(this.view.html().clientWidth, this.view.html().clientHeight);
-    }
 }
 
 export { Canvas3dImpl as Canvas3d, Canvas3dVersion };

cvat-canvas3d/src/typescript/canvas3dModel.ts

@@ -45,27 +45,26 @@ export enum Mode {
     INTERACT = 'interact',
 }
 
-export interface Canvas3dModel {
+export interface Canvas3dDataModel {
+    canvasSize: Size;
+    image: Image | null;
+    imageID: number | null;
+    imageOffset: number;
+    imageSize: Size;
+    drawData: DrawData;
     mode: Mode;
+    exception: Error | null;
+}
 
+export interface Canvas3dModel {
+    mode: Mode;
+    data: Canvas3dDataModel;
     setup(frameData: any): void;
-
     isAbleToChangeFrame(): boolean;
-
-    fitCanvas(width: number, height: number): void;
 }
 
 export class Canvas3dModelImpl extends MasterImpl implements Canvas3dModel {
-    private data: {
-        canvasSize: Size;
-        image: Image | null;
-        imageID: number | null;
-        imageOffset: number;
-        imageSize: Size;
-        drawData: DrawData;
-        mode: Mode;
-        exception: Error | null;
-    };
+    public data: Canvas3dDataModel;
 
     public constructor() {
         super();
@@ -92,36 +91,32 @@ export class Canvas3dModelImpl extends MasterImpl implements Canvas3dModel {
 
     public setup(frameData: any): void {
         if (this.data.imageID !== frameData.number) {
-            if ([Mode.EDIT, Mode.DRAG, Mode.RESIZE].includes(this.data.mode)) {
-                throw Error(`Canvas is busy. Action: ${this.data.mode}`);
-            }
+            this.data.imageID = frameData.number;
+            frameData
+                .data((): void => {
+                    this.data.image = null;
+                    this.notify(UpdateReasons.IMAGE_CHANGED);
+                })
+                .then((data: Image): void => {
+                    if (frameData.number !== this.data.imageID) {
+                        // already another image
+                        return;
+                    }
+
+                    this.data.imageSize = {
+                        height: frameData.height as number,
+                        width: frameData.width as number,
+                    };
+
+                    this.data.image = data;
+                    this.notify(UpdateReasons.IMAGE_CHANGED);
+                })
+                .catch((exception: any): void => {
+                    this.data.exception = exception;
+                    this.notify(UpdateReasons.DATA_FAILED);
+                    throw exception;
+                });
         }
-
-        this.data.imageID = frameData.number;
-        frameData
-            .data((): void => {
-                this.data.image = null;
-                this.notify(UpdateReasons.IMAGE_CHANGED);
-            })
-            .then((data: Image): void => {
-                if (frameData.number !== this.data.imageID) {
-                    // already another image
-                    return;
-                }
-
-                this.data.imageSize = {
-                    height: frameData.height as number,
-                    width: frameData.width as number,
-                };
-
-                this.data.image = data;
-                this.notify(UpdateReasons.IMAGE_CHANGED);
-            })
-            .catch((exception: any): void => {
-                this.data.exception = exception;
-                this.notify(UpdateReasons.DATA_FAILED);
-                throw exception;
-            });
     }
 
     public set mode(value: Mode) {
@@ -138,16 +133,4 @@ export class Canvas3dModelImpl extends MasterImpl implements Canvas3dModel {
 
         return !isUnable;
     }
-
-    public fitCanvas(width: number, height: number): void {
-        this.data.canvasSize.height = height;
-        this.data.canvasSize.width = width;
-
-        this.data.imageOffset = Math.floor(
-            Math.max(this.data.canvasSize.height / FrameZoom.MIN, this.data.canvasSize.width / FrameZoom.MIN),
-        );
-
-        this.notify(UpdateReasons.FITTED_CANVAS);
-        this.notify(UpdateReasons.OBJECTS_UPDATED);
-    }
 }

cvat-canvas3d/src/typescript/canvas3dView.ts

@@ -4,21 +4,58 @@
 
 import * as THREE from 'three';
 import { PCDLoader } from 'three/examples/jsm/loaders/PCDLoader';
+import { OrbitControls } from 'three/examples/jsm/controls/OrbitControls';
+import CameraControls from 'camera-controls';
 import { Canvas3dController } from './canvas3dController';
 import { Listener, Master } from './master';
-
+import CONST from './consts';
 import { Canvas3dModel, UpdateReasons, Mode } from './canvas3dModel';
 
 export interface Canvas3dView {
-    html(): HTMLDivElement;
+    html(): ViewsDOM;
     render(): void;
+    keyControls(keys: KeyboardEvent): void;
+}
+
+enum CAMERA_ACTION {
+    ZOOM_IN = 'KeyI',
+    MOVE_UP = 'KeyU',
+    MOVE_DOWN = 'KeyO',
+    MOVE_LEFT = 'KeyJ',
+    ZOOM_OUT = 'KeyK',
+    MOVE_RIGHT = 'KeyL',
+    TILT_UP = 'ArrowUp',
+    TILT_DOWN = 'ArrowDown',
+    ROTATE_RIGHT = 'ArrowRight',
+    ROTATE_LEFT = 'ArrowLeft',
+}
+
+export interface Views {
+    perspective: RenderView;
+    top: RenderView;
+    side: RenderView;
+    front: RenderView;
+}
+
+export interface RenderView {
+    renderer: THREE.WebGLRenderer;
+    scene: THREE.Scene;
+    camera?: THREE.PerspectiveCamera | THREE.OrthographicCamera;
+    controls?: CameraControls | OrbitControls;
+}
+
+export interface ViewsDOM {
+    perspective: HTMLCanvasElement;
+    top: HTMLCanvasElement;
+    side: HTMLCanvasElement;
+    front: HTMLCanvasElement;
 }
 
 export class Canvas3dViewImpl implements Canvas3dView, Listener {
     private controller: Canvas3dController;
-    private renderer: any;
-    private scene: any;
-    private camera: any;
+    private views: Views;
+    private clock: THREE.Clock;
+    private speed: number;
 
     private set mode(value: Mode) {
         this.controller.mode = value;
@@ -30,22 +67,91 @@ export class Canvas3dViewImpl implements Canvas3dView, Listener {
 
     public constructor(model: Canvas3dModel & Master, controller: Canvas3dController) {
         this.controller = controller;
+        this.clock = new THREE.Clock();
+        this.speed = CONST.MOVEMENT_FACTOR;
+        this.views = {
+            perspective: {
+                renderer: new THREE.WebGLRenderer({ antialias: true }),
+                scene: new THREE.Scene(),
+            },
+            top: {
+                renderer: new THREE.WebGLRenderer({ antialias: true }),
+                scene: new THREE.Scene(),
+            },
+            side: {
+                renderer: new THREE.WebGLRenderer({ antialias: true }),
+                scene: new THREE.Scene(),
+            },
+            front: {
+                renderer: new THREE.WebGLRenderer({ antialias: true }),
+                scene: new THREE.Scene(),
+            },
+        };
+        CameraControls.install({ THREE });
 
         this.mode = Mode.IDLE;
 
-        this.scene = new THREE.Scene();
-        this.scene.background = new THREE.Color(0x000000);
+        Object.keys(this.views).forEach((view: string): void => {
+            this.views[view as keyof Views].scene.background = new THREE.Color(0x000000);
+        });
+
+        const viewSize = CONST.ZOOM_FACTOR;
+        const height = window.innerHeight;
+        const width = window.innerWidth;
+        const aspectRatio = window.innerWidth / window.innerHeight;
 
         // setting up the camera and adding it in the scene
-        this.camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 1, 500);
-        this.camera.position.set(-15, 0, 4);
-        this.camera.up.set(0, 0, 1);
-        this.camera.lookAt(0, 0, 0);
-        this.scene.add(this.camera);
+        this.views.perspective.camera = new THREE.PerspectiveCamera(50, aspectRatio, 1, 500);
+        this.views.perspective.camera.position.set(-15, 0, 4);
+        this.views.perspective.camera.up.set(0, 0, 1);
+        this.views.perspective.camera.lookAt(10, 0, 0);
 
-        this.renderer = new THREE.WebGLRenderer({ antialias: true });
-        this.renderer.setPixelRatio(window.devicePixelRatio);
-        this.renderer.setSize(window.innerWidth, window.innerHeight);
+        this.views.top.camera = new THREE.OrthographicCamera(
+            (-aspectRatio * viewSize) / 2 - 2,
+            (aspectRatio * viewSize) / 2 + 2,
+            viewSize / 2 + 2,
+            -viewSize / 2 - 2,
+            -10,
+            10,
+        );
+
+        this.views.side.camera = new THREE.OrthographicCamera(
+            (-aspectRatio * viewSize) / 2,
+            (aspectRatio * viewSize) / 2,
+            viewSize / 2,
+            -viewSize / 2,
+            -10,
+            10,
+        );
+        this.views.side.camera.position.set(0, 5, 0);
+        this.views.side.camera.lookAt(0, 0, 0);
+        this.views.side.camera.up.set(0, 0, 1);
+
+        this.views.front.camera = new THREE.OrthographicCamera(
+            (-aspectRatio * viewSize) / 2,
+            (aspectRatio * viewSize) / 2,
+            viewSize / 2,
+            -viewSize / 2,
+            -10,
+            10,
+        );
+        this.views.front.camera.position.set(-7, 0, 0);
+        this.views.front.camera.up.set(0, 0, 1);
+        this.views.front.camera.lookAt(0, 0, 0);
+
+        Object.keys(this.views).forEach((view: string) => {
+            const viewType = this.views[view as keyof Views];
+            viewType.renderer.setSize(width, height);
+            if (view !== 'perspective') {
+                viewType.controls = new OrbitControls(viewType.camera, viewType.renderer.domElement);
+                viewType.controls.enableRotate = false;
+                viewType.controls.enablePan = false;
+            } else {
+                viewType.controls = new CameraControls(viewType.camera, viewType.renderer.domElement);
+            }
+            viewType.controls.minDistance = CONST.MIN_DISTANCE;
+            viewType.controls.maxDistance = CONST.MAX_DISTANCE;
+        });
 
         model.subscribe(this);
     }
@@ -58,13 +164,16 @@ export class Canvas3dViewImpl implements Canvas3dView, Listener {
             loader.load(objectURL, this.addScene.bind(this));
             URL.revokeObjectURL(objectURL);
             const event: CustomEvent = new CustomEvent('canvas.setup');
-            this.renderer.domElement.dispatchEvent(event);
+            this.views.perspective.renderer.domElement.dispatchEvent(event);
         }
     }
 
     private clearScene(): void {
-        for (let i = this.scene.children.length - 1; i >= 0; i--) {
-            this.scene.remove(this.scene.children[i]);
+        for (let i = this.views.perspective.scene.children.length - 1; i >= 0; i--) {
+            this.views.perspective.scene.remove(this.views.perspective.scene.children[i]);
+            this.views.top.scene.remove(this.views.top.scene.children[i]);
+            this.views.side.scene.remove(this.views.side.scene.children[i]);
+            this.views.front.scene.remove(this.views.front.scene.children[i]);
         }
     }
 
@@ -73,14 +182,88 @@ export class Canvas3dViewImpl implements Canvas3dView, Listener {
         points.material.size = 0.03;
         // eslint-disable-next-line no-param-reassign
         points.material.color = new THREE.Color(0x0000ff);
-        this.scene.add(points);
+        this.views.perspective.scene.add(points);
+        this.views.top.scene.add(points.clone());
+        this.views.side.scene.add(points.clone());
+        this.views.front.scene.add(points.clone());
+    }
+
+    private static resizeRendererToDisplaySize(view: RenderView): void {
+        const canvas = view.renderer.domElement;
+        const width = canvas.parentElement.clientWidth;
+        const height = canvas.parentElement.clientHeight;
+        const needResize = canvas.clientWidth !== width || canvas.clientHeight !== height;
+        if (needResize) {
+            if (!(view.camera instanceof THREE.OrthographicCamera)) {
+                // eslint-disable-next-line no-param-reassign
+                view.camera.aspect = width / height;
+            }
+            view.camera.updateProjectionMatrix();
+            view.renderer.setSize(width, height);
+        }
     }
 
     public render(): void {
-        this.renderer.render(this.scene, this.camera);
+        Object.keys(this.views).forEach((view: string) => {
+            const viewType = this.views[view as keyof Views];
+            Canvas3dViewImpl.resizeRendererToDisplaySize(viewType);
+            viewType.controls.update(this.clock.getDelta());
+            viewType.renderer.render(viewType.scene, viewType.camera);
+        });
+    }
+
+    public keyControls(key: any): void {
+        if (!(this.views.perspective.controls instanceof OrbitControls)) {
+            const { controls } = this.views.perspective;
+            switch (key.code) {
+                case CAMERA_ACTION.ROTATE_RIGHT:
+                    controls.rotate(0.1 * THREE.MathUtils.DEG2RAD * this.speed, 0, true);
+                    break;
+                case CAMERA_ACTION.ROTATE_LEFT:
+                    controls.rotate(-0.1 * THREE.MathUtils.DEG2RAD * this.speed, 0, true);
+                    break;
+                case CAMERA_ACTION.TILT_UP:
+                    controls.rotate(0, -0.05 * THREE.MathUtils.DEG2RAD * this.speed, true);
+                    break;
+                case CAMERA_ACTION.TILT_DOWN:
+                    controls.rotate(0, 0.05 * THREE.MathUtils.DEG2RAD * this.speed, true);
+                    break;
+                default:
+                    break;
+            }
+            if (key.altKey === true) {
+                switch (key.code) {
+                    case CAMERA_ACTION.ZOOM_IN:
+                        controls.dolly(CONST.DOLLY_FACTOR, true);
+                        break;
+                    case CAMERA_ACTION.ZOOM_OUT:
+                        controls.dolly(-CONST.DOLLY_FACTOR, true);
+                        break;
+                    case CAMERA_ACTION.MOVE_LEFT:
+                        controls.truck(-0.01 * this.speed, 0, true);
+                        break;
+                    case CAMERA_ACTION.MOVE_RIGHT:
+                        controls.truck(0.01 * this.speed, 0, true);
+                        break;
+                    case CAMERA_ACTION.MOVE_DOWN:
+                        controls.truck(0, -0.01 * this.speed, true);
+                        break;
+                    case CAMERA_ACTION.MOVE_UP:
+                        controls.truck(0, 0.01 * this.speed, true);
+                        break;
+                    default:
+                        break;
+                }
+            }
+        }
     }
 
-    public html(): any {
-        return this.renderer.domElement;
+    public html(): ViewsDOM {
+        return {
+            perspective: this.views.perspective.renderer.domElement,
+            top: this.views.top.renderer.domElement,
+            side: this.views.side.renderer.domElement,
+            front: this.views.front.renderer.domElement,
+        };
     }
 }

cvat-canvas3d/src/typescript/consts.ts

@@ -3,7 +3,17 @@
 // SPDX-License-Identifier: MIT
 
 const BASE_GRID_WIDTH = 2;
+const MOVEMENT_FACTOR = 200;
+const DOLLY_FACTOR = 5;
+const MAX_DISTANCE = 100;
+const MIN_DISTANCE = 0;
+const ZOOM_FACTOR = 7;
 
 export default {
     BASE_GRID_WIDTH,
+    MOVEMENT_FACTOR,
+    DOLLY_FACTOR,
+    MAX_DISTANCE,
+    MIN_DISTANCE,
+    ZOOM_FACTOR,
 };

cvat-ui/package-lock.json

@@ -3977,6 +3977,25 @@
         "array-find-index": "^1.0.1"
       }
     },
+    "cvat-canvas3d": {
+      "version": "file:../cvat-canvas3d",
+      "requires": {
+        "three": "^0.125.0",
+        "camera-controls": "1.25.3"
+      },
+      "dependencies": {
+        "three": {
+          "version": "0.125.2",
+          "resolved": "https://registry.npmjs.org/three/-/three-0.125.2.tgz",
+          "integrity": "sha512-7rIRO23jVKWcAPFdW/HREU2NZMGWPBZ4XwEMt0Ak0jwLUKVJhcKM55eCBWyGZq/KiQbeo1IeuAoo/9l2dzhTXA=="
+        },
+        "camera-controls": {
+          "version": "1.25.3",
+          "resolved": "https://registry.npmjs.org/camera-controls/-/camera-controls-1.25.3.tgz",
+          "integrity": "sha512-hcPgd3ly69N+HU8LRsyCmDh4N5YAbyDLB2LwbwOJdFoXomvpEi1BPGIJCbHbzp/n+CrisEjgeCHJprqy7GT2Dg=="
+        }
+      }
+    },
     "cvat-canvas": {
       "version": "file:../cvat-canvas",
       "requires": {
@@ -12870,19 +12889,6 @@
         }
       }
     },
-    "cvat-canvas3d": {
-      "version": "file:../cvat-canvas3d",
-      "requires": {
-        "three": "^0.125.0"
-      },
-      "dependencies": {
-        "three": {
-          "version": "0.125.2",
-          "resolved": "https://registry.npmjs.org/three/-/three-0.125.2.tgz",
-          "integrity": "sha512-7rIRO23jVKWcAPFdW/HREU2NZMGWPBZ4XwEMt0Ak0jwLUKVJhcKM55eCBWyGZq/KiQbeo1IeuAoo/9l2dzhTXA=="
-        }
-      }
-    },
     "cvat-core": {
       "version": "file:../cvat-core",
       "requires": {

cvat-ui/src/components/annotation-page/canvas/canvas-wrapper3D.tsx

@@ -5,7 +5,9 @@
 import React, { ReactElement, useEffect, useRef } from 'react';
 import { GlobalHotKeys } from 'react-hotkeys';
 import Layout from 'antd/lib/layout/layout';
-
+import {
+    ArrowUpOutlined, ArrowRightOutlined, ArrowLeftOutlined, ArrowDownOutlined,
+} from '@ant-design/icons';
 import { Workspace } from 'reducers/interfaces';
 import { Canvas3d } from 'cvat-canvas3d-wrapper';
 import ContextImage from '../standard3D-workspace/context-image/context-image';
@@ -29,17 +31,15 @@ interface Props {
 
 const CanvasWrapperComponent = (props: Props): ReactElement => {
     const animateId = useRef(0);
-    const cvatCanvasContainerRef = useRef();
+    const perspectiveView = useRef<HTMLCanvasElement>(null);
+    const topView = useRef<HTMLCanvasElement>(null);
+    const sideView = useRef<HTMLCanvasElement>(null);
+    const frontView = useRef<HTMLCanvasElement>(null);
 
     const {
         frameData, contextImageHide, getContextImage, loaded, data, annotations, curZLayer,
     } = props;
 
-    const fitCanvas = (): void => {
-        const { canvasInstance } = props;
-        canvasInstance.fitCanvas();
-    };
-
     const onCanvasSetup = (): void => {
         const { onSetupCanvas } = props;
         onSetupCanvas();
@@ -63,26 +63,46 @@ const CanvasWrapperComponent = (props: Props): ReactElement => {
     const initialSetup = (): void => {
         const { canvasInstance } = props;
 
-        // Size
-        window.addEventListener('resize', fitCanvas);
-        fitCanvas();
-
+        const canvasInstanceDOM = canvasInstance.html();
         // Events
-        canvasInstance.html().addEventListener('canvas.setup', onCanvasSetup);
+        canvasInstanceDOM.perspective.addEventListener('canvas.setup', onCanvasSetup);
+    };
+
+    const keyControls = (key: KeyboardEvent): void => {
+        const { canvasInstance } = props;
+        canvasInstance.keyControls(key);
     };
 
     useEffect(() => {
         const { canvasInstance } = props;
 
-        cvatCanvasContainerRef.current.appendChild(canvasInstance.html());
+        const canvasInstanceDOM = canvasInstance.html();
+
+        if (
+            perspectiveView &&
+            perspectiveView.current &&
+            topView &&
+            topView.current &&
+            sideView &&
+            sideView.current &&
+            frontView &&
+            frontView.current
+        ) {
+            perspectiveView.current.appendChild(canvasInstanceDOM.perspective);
+            topView.current.appendChild(canvasInstanceDOM.top);
+            sideView.current.appendChild(canvasInstanceDOM.side);
+            frontView.current.appendChild(canvasInstanceDOM.front);
+        }
+
+        document.addEventListener('keydown', keyControls);
 
         initialSetup();
         updateCanvas();
         animateCanvas();
 
         return () => {
-            canvasInstance.html().removeEventListener('canvas.setup', onCanvasSetup);
-            window.removeEventListener('resize', fitCanvas);
+            canvasInstanceDOM.perspective.removeEventListener('canvas.setup', onCanvasSetup);
+            document.removeEventListener('keydown', keyControls);
             cancelAnimationFrame(animateId.current);
         };
     });
@@ -91,9 +111,52 @@ const CanvasWrapperComponent = (props: Props): ReactElement => {
         updateCanvas();
     }, [frameData, annotations, curZLayer]);
 
+    const renderArrowGroup = (): ReactElement => (
+        <span className='cvat-canvas3d-perspective-arrow-directions'>
+            <button type='button' className='cvat-canvas3d-perspective-arrow-directions-icons-up'>
+                <ArrowUpOutlined className='cvat-canvas3d-perspective-arrow-directions-icons-color' />
+            </button>
+            <br />
+            <button type='button' className='cvat-canvas3d-perspective-arrow-directions-icons-bottom'>
+                <ArrowLeftOutlined className='cvat-canvas3d-perspective-arrow-directions-icons-color' />
+            </button>
+            <button type='button' className='cvat-canvas3d-perspective-arrow-directions-icons-bottom'>
+                <ArrowDownOutlined className='cvat-canvas3d-perspective-arrow-directions-icons-color' />
+            </button>
+            <button type='button' className='cvat-canvas3d-perspective-arrow-directions-icons-bottom'>
+                <ArrowRightOutlined className='cvat-canvas3d-perspective-arrow-directions-icons-color' />
+            </button>
+        </span>
+    );
+
+    const renderControlGroup = (): ReactElement => (
+        <span className='cvat-canvas3d-perspective-directions'>
+            <button type='button' className='cvat-canvas3d-perspective-directions-icon'>
+                U
+            </button>
+            <button type='button' className='cvat-canvas3d-perspective-directions-icon'>
+                I
+            </button>
+            <button type='button' className='cvat-canvas3d-perspective-directions-icon'>
+                O
+            </button>
+            <br />
+            <button type='button' className='cvat-canvas3d-perspective-directions-icon'>
+                J
+            </button>
+            <button type='button' className='cvat-canvas3d-perspective-directions-icon'>
+                K
+            </button>
+            <button type='button' className='cvat-canvas3d-perspective-directions-icon'>
+                L
+            </button>
+        </span>
+    );
+
     return (
-        <Layout.Content style={{ position: 'relative' }}>
+        <Layout.Content className='cvat-canvas3d-fullsize'>
             <GlobalHotKeys />
+
             <ContextImage
                 frame={frameData}
                 contextImageHide={contextImageHide}
@@ -101,7 +164,27 @@ const CanvasWrapperComponent = (props: Props): ReactElement => {
                 loaded={loaded}
                 data={data}
             />
-            <div ref={cvatCanvasContainerRef} className='cvat-canvas-container cvat-canvas-container-overflow' />
+
+            <div className='cvat-canvas3d-perspective'>
+                <div className='cvat-canvas-container cvat-canvas-container-overflow' ref={perspectiveView} />
+                {renderArrowGroup()}
+                {renderControlGroup()}
+            </div>
+            <div className='cvat-canvas3d-orthographic-views'>
+                <div className='cvat-canvas3d-orthographic-view cvat-canvas3d-topview'>
+                    <div className='cvat-canvas3d-header'>TOP VIEW</div>
+                    <div className='cvat-canvas3d-fullsize' ref={topView} />
+                </div>
+                <div className='cvat-canvas3d-orthographic-view cvat-canvas3d-sideview'>
+                    <div className='cvat-canvas3d-header'>SIDE VIEW</div>
+                    <div className='cvat-canvas3d-fullsize' ref={sideView} />
+                </div>
+                <div className='cvat-canvas3d-orthographic-view cvat-canvas3d-frontview'>
+                    <div className='cvat-canvas3d-header'>FRONT VIEW</div>
+                    <div className='cvat-canvas3d-fullsize' ref={frontView} />
+                </div>
+                <div className='cvat-canvas3d-view-slider' />
+            </div>
         </Layout.Content>
     );
 };

cvat-ui/src/components/annotation-page/standard3D-workspace/styles.scss

@@ -171,3 +171,95 @@
 .cvat-control-side-bar-icon-size {
     font-size: $grid-unit-size * 5;
 }
+
+.cvat-canvas3d-perspective {
+    height: 50%;
+    width: 100%;
+    position: relative;
+    padding: $grid-unit-size/2;
+}
+
+.cvat-canvas3d-orthographic-views {
+    height: 50%;
+    position: relative;
+    display: flex;
+    flex-direction: row;
+    width: 100%;
+}
+
+.cvat-canvas3d-perspective-arrow-directions {
+    position: absolute;
+    padding: $grid-unit-size;
+    bottom: 0;
+    right: 0;
+}
+
+.cvat-canvas3d-perspective-arrow-directions-icons-up {
+    margin-left: $grid-unit-size * 5.25;
+}
+
+.cvat-canvas3d-perspective-arrow-directions-icons-bottom {
+    margin: $grid-unit-size/2;
+}
+
+.cvat-canvas3d-perspective-arrow-directions-icons-color {
+    color: black;
+}
+
+.cvat-canvas3d-perspective-directions {
+    padding: $grid-unit-size;
+    position: absolute;
+    bottom: 0;
+    left: 0;
+}
+
+.cvat-canvas3d-perspective-directions-icon {
+    margin: $grid-unit-size/2 $grid-unit-size/4;
+    width: $grid-unit-size * 4;
+}
+
+.cvat-canvas3d-orthographic-view {
+    width: 33.333%;
+    height: 100%;
+    padding-top: $grid-unit-size/2;
+    padding-bottom: $grid-unit-size/2;
+}
+
+.cvat-canvas3d-topview {
+    padding-left: $grid-unit-size/2;
+}
+
+.cvat-canvas3d-sideview {
+    padding-left: $grid-unit-size/2;
+    padding-right: $grid-unit-size/2;
+}
+
+.cvat-canvas3d-frontview {
+    padding-right: $grid-unit-size/2;
+}
+
+.cvat-canvas3d-fullsize {
+    position: relative;
+    width: 100%;
+    height: 100%;
+}
+
+.cvat-canvas3d-view-slider {
+    position: absolute;
+    margin-left: auto;
+    width: $grid-unit-size * 10;
+    margin-right: auto;
+    right: 0;
+    top: 0;
+    left: 0;
+    background-color: grey;
+    height: $grid-unit-size/2;
+}
+
+.cvat-canvas3d-header {
+    height: $grid-unit-size * 4;
+    width: 100%;
+    background-color: $background-color-2;
+    text-align: center;
+    vertical-align: middle;
+}

cvat-ui/src/components/header/settings-modal/player-settings.tsx

@@ -115,22 +115,38 @@ export default function PlayerSettingsComponent(props: Props): JSX.Element {
                             onChangeFrameSpeed(speed);
                         }}
                     >
-                        <Select.Option key='fastest' value={FrameSpeed.Fastest} className='cvat-player-settings-speed-fastest'>
+                        <Select.Option
+                            key='fastest'
+                            value={FrameSpeed.Fastest}
+                            className='cvat-player-settings-speed-fastest'
+                        >
                             Fastest
                         </Select.Option>
                         <Select.Option key='fast' value={FrameSpeed.Fast} className='cvat-player-settings-speed-fast'>
                             Fast
                         </Select.Option>
-                        <Select.Option key='usual' value={FrameSpeed.Usual} className='cvat-player-settings-speed-usual'>
+                        <Select.Option
+                            key='usual'
+                            value={FrameSpeed.Usual}
+                            className='cvat-player-settings-speed-usual'
+                        >
                             Usual
                         </Select.Option>
                         <Select.Option key='slow' value={FrameSpeed.Slow} className='cvat-player-settings-speed-slow'>
                             Slow
                         </Select.Option>
-                        <Select.Option key='slower' value={FrameSpeed.Slower} className='cvat-player-settings-speed-slower'>
+                        <Select.Option
+                            key='slower'
+                            value={FrameSpeed.Slower}
+                            className='cvat-player-settings-speed-slower'
+                        >
                             Slower
                         </Select.Option>
-                        <Select.Option key='slowest' value={FrameSpeed.Slowest} className='cvat-player-settings-speed-slowest'>
+                        <Select.Option
+                            key='slowest'
+                            value={FrameSpeed.Slowest}
+                            className='cvat-player-settings-speed-slowest'
+                        >
                             Slowest
                         </Select.Option>
                     </Select>

cvat-ui/src/components/labels-editor/common.ts

-// Copyright (C) 2020 Intel Corporation
+// Copyright (C) 2021 Intel Corporation
 //
 // SPDX-License-Identifier: MIT
 
@@ -35,15 +35,11 @@ function validateParsedAttribute(attr: Attribute): void {
     }
 
     if (typeof attr.mutable !== 'boolean') {
-        throw new Error(
-            `Attribute: "${attr.name}". Mutable flag must be a boolean value. Got value ${attr.mutable}`,
-        );
+        throw new Error(`Attribute: "${attr.name}". Mutable flag must be a boolean value. Got value ${attr.mutable}`);
     }
 
     if (!Array.isArray(attr.values)) {
-        throw new Error(
-            `Attribute: "${attr.name}". Attribute values must be an array. Got type ${typeof attr.values}`,
-        );
+        throw new Error(`Attribute: "${attr.name}". Attribute values must be an array. Got type ${typeof attr.values}`);
     }
 
     if (!attr.values.length) {

cvat/apps/documentation/installation.md

@@ -17,6 +17,7 @@
         - [1. Make the proxy listen on 80 and 443 ports](#1-make-the-proxy-listen-on-80-and-443-ports)
         - [2. Issue a certificate and run HTTPS versions with `acme.sh` helper](#2-issue-a-certificate-and-run-https-versions-with-acmesh-helper)
           - [Create certificate files using an ACME challenge on docker host](#create-certificate-files-using-an-acme-challenge-on-docker-host)
+
 # Quick installation guide
 
 Before you can use CVAT, you’ll need to get it installed. The document below
@@ -510,6 +511,7 @@ rm -r /root/.acme.sh/CVAT.example.com
 ```
 
 ####### Issue a production certificate
+
 ```
 ~/.acme.sh/acme.sh --issue -d CVAT.example.com -w $HOME/cvat/letsencrypt-webroot --debug
 ```

cvat/apps/documentation/user_guide.md

@@ -322,14 +322,15 @@ Once created, the project will appear on the projects page. To open a project, j
 ![](static/documentation/images/image192_mapillary_vistas.jpg)
 
 Here you can do the following:
-  1. Change the project's title.
-  1. Open the `Actions` menu.
-  1. Change issue tracker or open issue tracker if it is specified.
-  1. Change labels.
-  You can add new labels or add attributes for the existing labels in the Raw mode or the Constructor mode. 
-  You can also change the color for different labels. By clicking `Copy` you can copy the labels to the clipboard.
-  1. Assigned to — is used to assign a project to a person. Start typing an assignee's name and/or choose the right person out of the dropdown list.
-  1. `Tasks` — is a list of all tasks for a particular project.
+
+1. Change the project's title.
+1. Open the `Actions` menu.
+1. Change issue tracker or open issue tracker if it is specified.
+1. Change labels.
+   You can add new labels or add attributes for the existing labels in the Raw mode or the Constructor mode. 
+   You can also change the color for different labels. By clicking `Copy` you can copy the labels to the clipboard.
+1. Assigned to — is used to assign a project to a person. Start typing an assignee's name and/or choose the right person out of the dropdown list.
+1. `Tasks` — is a list of all tasks for a particular project.
 
 You can remove the project and all related tasks through the Action menu.
 
@@ -756,7 +757,7 @@ Button assignment:
   - [MS COCO](http://cocodataset.org/#format-data)
   - [YOLO](https://pjreddie.com/darknet/yolo/)
 - `Open the task` — opens a page with details about the task.
-- `Request a review` - calls up the form to submit the job for a review, read more in the [review](#review)  section.
+- `Request a review` - calls up the form to submit the job for a review, read more in the [review](#review) section.
 - `Finish the job` - changes the status of the job to `completed` and returns to the task page without review.
 - `Submit the review` - (available during the review) calls up the form to submit a review, read more in the [review](#review) section.
 
@@ -1146,7 +1147,7 @@ The tool based on [Open CV](https://opencv.org/) Computer Vision library which i
 
 First step to work with OpenCV is to load it into CVAT. Click on the toolbar icon, then click `Load OpenCV`.
 
-  ![](static/documentation/images/image198.jpg)
+![](static/documentation/images/image198.jpg)
 
 Once it is loaded, the tool's functionality will be available.
 
@@ -1155,24 +1156,26 @@ Once it is loaded, the tool's functionality will be available.
 Intelligent scissors is an CV method of creating a polygon by placing points with automatic drawing of a line between them.
 The distance between the adjacent points is limited by the threshold of action,
 displayed as a red square which is tied to the cursor.
- 
+
+
+
 - First, select the label and then click on the `intelligent scissors` button.
 
   ![](static/documentation/images/image199.jpg)
 
 - Create the first point on the boundary of the allocated object.
-You will see a line repeating the outline of the object.
+  You will see a line repeating the outline of the object.
 - Place the second point, so that the previous point is within the restrictive threshold.
-After that a line repeating the object boundary will be automatically created between the points.
+  After that a line repeating the object boundary will be automatically created between the points.
 
-  ![](static/documentation/images/image200_detrac.jpg) 
+  ![](static/documentation/images/image200_detrac.jpg)
 
   To increase or lower the action threshold, hold `Ctrl` and scroll the mouse wheel.
   Increasing action threshold will affect the performance.
   During the drawing process you can remove the last point by clicking on it with the left mouse button.
- 
+
 - Once all the points are placed, you can complete the creation of the object by clicking on the icon or clicking `N`.
-As a result, a polygon will be created (read more about the polygons in the [annoation with polygons](#annotation-with-polygons)).
+  As a result, a polygon will be created (read more about the polygons in the [annoation with polygons](#annotation-with-polygons)).
 
 ## Annotation with rectangle by 4 points
 
@@ -1474,18 +1477,19 @@ or areas in the frame and describe the problem.
   ![](static/documentation/images/image195.jpg)
 
 - Then click on an object in the frame to highlight the object or highlight the area by holding the left mouse button
-and describe the problem. The object or area will be shaded in red.
+  and describe the problem. The object or area will be shaded in red.
 - The created issue will appear in the workspace and in the `issues` tab on the objects sidebar.
 - After you save the annotation, other users will be able to see the problem, comment on each issue
-and change the status of the problem to `resolved`.
+  and change the status of the problem to `resolved`.
 - You can use the arrows on the issues tab to navigate the frames that contain problems.
 
   ![](static/documentation/images/image196_detrac.jpg)
 
 - Once all the problems are marked, save the annotation, open the menu and select "submit the review". After that you'll see a form containing the verification statistics, here you can give an assessment of the job and choose further actions:
-    - Accept - changes the status of the job to `completed`.
-    - Review next – passes the job to another user for re-review.
-    - Reject - changes the status of the job to `annotation`.
+
+  - Accept - changes the status of the job to `completed`.
+  - Review next – passes the job to another user for re-review.
+  - Reject - changes the status of the job to `annotation`.
 
   ![](static/documentation/images/image197.jpg)