Skip to content
体验新版
项目
组织
正在加载...
登录
切换导航
打开侧边栏
jobily
Questdb
提交
4127c9ac
Q
Questdb
项目概览
jobily
/
Questdb
11 个月 前同步成功
通知
1
Star
0
Fork
0
代码
文件
提交
分支
Tags
贡献者
分支图
Diff
Issue
0
列表
看板
标记
里程碑
合并请求
0
DevOps
流水线
流水线任务
计划
Wiki
0
Wiki
分析
仓库
DevOps
项目成员
Pages
Q
Questdb
项目概览
项目概览
详情
发布
仓库
仓库
文件
提交
分支
标签
贡献者
分支图
比较
Issue
0
Issue
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
Pages
DevOps
DevOps
流水线
流水线任务
计划
分析
分析
仓库分析
DevOps
Wiki
0
Wiki
成员
成员
收起侧边栏
关闭侧边栏
动态
分支图
创建新Issue
流水线任务
提交
Issue看板
体验新版 GitCode,发现更多精彩内容 >>
提交
4127c9ac
编写于
12月 18, 2019
作者:
V
Vlad Ilyushchenko
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
fix(http): NPE when content-disposition is missing or malformed. Added more tests.
上级
c85d4afc
变更
7
隐藏空白更改
内联
并排
Showing
7 changed file
with
169 addition
and
25 deletion
+169
-25
core/src/main/java/io/questdb/cutlass/http/HttpConnectionContext.java
...n/java/io/questdb/cutlass/http/HttpConnectionContext.java
+8
-1
core/src/main/java/io/questdb/cutlass/http/HttpMultipartContentListener.java
...io/questdb/cutlass/http/HttpMultipartContentListener.java
+3
-2
core/src/main/java/io/questdb/cutlass/http/HttpMultipartContentParser.java
...a/io/questdb/cutlass/http/HttpMultipartContentParser.java
+3
-1
core/src/main/java/io/questdb/cutlass/http/HttpRequestProcessor.java
...in/java/io/questdb/cutlass/http/HttpRequestProcessor.java
+2
-1
core/src/main/java/io/questdb/cutlass/http/processors/TextImportProcessor.java
.../questdb/cutlass/http/processors/TextImportProcessor.java
+20
-20
core/src/main/java/io/questdb/network/ServerDisconnectException.java
...in/java/io/questdb/network/ServerDisconnectException.java
+29
-0
core/src/test/java/io/questdb/cutlass/http/IODispatcherTest.java
...c/test/java/io/questdb/cutlass/http/IODispatcherTest.java
+104
-0
未找到文件。
core/src/main/java/io/questdb/cutlass/http/HttpConnectionContext.java
浏览文件 @
4127c9ac
...
...
@@ -146,6 +146,9 @@ public class HttpConnectionContext implements IOContext, Locality, Mutable {
}
catch
(
PeerIsSlowToReadException
ignore
)
{
LOG
.
debug
().
$
(
"peer is slow writer"
).
$
();
dispatcher
.
registerChannel
(
this
,
IOOperation
.
READ
);
}
catch
(
ServerDisconnectException
ignore
)
{
LOG
.
info
().
$
(
"kicked out [fd="
).
$
(
fd
).
$
(
']'
);
dispatcher
.
disconnect
(
this
);
}
break
;
case
IOOperation
.
WRITE
:
...
...
@@ -159,6 +162,9 @@ public class HttpConnectionContext implements IOContext, Locality, Mutable {
dispatcher
.
registerChannel
(
this
,
IOOperation
.
WRITE
);
}
catch
(
PeerDisconnectedException
ignore
)
{
dispatcher
.
disconnect
(
this
);
}
catch
(
ServerDisconnectException
ignore
)
{
LOG
.
info
().
$
(
"kicked out [fd="
).
$
(
fd
).
$
(
']'
);
dispatcher
.
disconnect
(
this
);
}
}
else
{
assert
false
;
...
...
@@ -192,7 +198,8 @@ public class HttpConnectionContext implements IOContext, Locality, Mutable {
}
}
private
void
handleClientRecv
(
HttpRequestProcessorSelector
selector
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
private
void
handleClientRecv
(
HttpRequestProcessorSelector
selector
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
try
{
final
long
fd
=
this
.
fd
;
// this is address of where header ended in our receive buffer
...
...
core/src/main/java/io/questdb/cutlass/http/HttpMultipartContentListener.java
浏览文件 @
4127c9ac
...
...
@@ -26,11 +26,12 @@ package io.questdb.cutlass.http;
import
io.questdb.network.PeerDisconnectedException
;
import
io.questdb.network.PeerIsSlowToReadException
;
import
io.questdb.network.ServerDisconnectException
;
public
interface
HttpMultipartContentListener
{
void
onChunk
(
long
lo
,
long
hi
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
;
void
onPartBegin
(
HttpRequestHeader
partHeader
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
;
void
onPartBegin
(
HttpRequestHeader
partHeader
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
;
void
onPartEnd
()
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
;
void
onPartEnd
()
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
;
}
core/src/main/java/io/questdb/cutlass/http/HttpMultipartContentParser.java
浏览文件 @
4127c9ac
...
...
@@ -26,6 +26,7 @@ package io.questdb.cutlass.http;
import
io.questdb.network.PeerDisconnectedException
;
import
io.questdb.network.PeerIsSlowToReadException
;
import
io.questdb.network.ServerDisconnectException
;
import
io.questdb.std.Mutable
;
import
io.questdb.std.Unsafe
;
import
io.questdb.std.str.DirectByteCharSequence
;
...
...
@@ -89,7 +90,8 @@ public class HttpMultipartContentParser implements Closeable, Mutable {
return
this
;
}
public
boolean
parse
(
long
lo
,
long
hi
,
HttpMultipartContentListener
listener
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
public
boolean
parse
(
long
lo
,
long
hi
,
HttpMultipartContentListener
listener
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
long
_lo
=
Long
.
MAX_VALUE
;
long
ptr
=
lo
;
while
(
ptr
<
hi
)
{
...
...
core/src/main/java/io/questdb/cutlass/http/HttpRequestProcessor.java
浏览文件 @
4127c9ac
...
...
@@ -26,6 +26,7 @@ package io.questdb.cutlass.http;
import
io.questdb.network.PeerDisconnectedException
;
import
io.questdb.network.PeerIsSlowToReadException
;
import
io.questdb.network.ServerDisconnectException
;
public
interface
HttpRequestProcessor
{
void
onHeadersReady
(
HttpConnectionContext
context
);
...
...
@@ -35,6 +36,6 @@ public interface HttpRequestProcessor {
default
void
resumeRecv
(
HttpConnectionContext
context
)
{
}
default
void
resumeSend
(
HttpConnectionContext
context
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
default
void
resumeSend
(
HttpConnectionContext
context
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
}
}
core/src/main/java/io/questdb/cutlass/http/processors/TextImportProcessor.java
浏览文件 @
4127c9ac
...
...
@@ -34,10 +34,7 @@ import io.questdb.cutlass.text.TextException;
import
io.questdb.cutlass.text.TextLoader
;
import
io.questdb.log.Log
;
import
io.questdb.log.LogFactory
;
import
io.questdb.network.IOOperation
;
import
io.questdb.network.NoSpaceLeftInResponseBufferException
;
import
io.questdb.network.PeerDisconnectedException
;
import
io.questdb.network.PeerIsSlowToReadException
;
import
io.questdb.network.*
;
import
io.questdb.std.*
;
import
io.questdb.std.str.CharSink
;
...
...
@@ -275,9 +272,10 @@ public class TextImportProcessor implements HttpRequestProcessor, HttpMultipartC
}
@Override
public
void
onPartBegin
(
HttpRequestHeader
partHeader
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
LOG
.
debug
().
$
(
"part begin [name="
).
$
(
partHeader
.
getContentDispositionName
()).
$
(
']'
).
$
();
if
(
Chars
.
equals
(
"data"
,
partHeader
.
getContentDispositionName
()))
{
public
void
onPartBegin
(
HttpRequestHeader
partHeader
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
final
CharSequence
contentDisposition
=
partHeader
.
getContentDispositionName
();
LOG
.
debug
().
$
(
"part begin [name="
).
$
(
contentDisposition
).
$
(
']'
).
$
();
if
(
Chars
.
equalsNc
(
"data"
,
contentDisposition
))
{
final
HttpRequestHeader
rh
=
transientContext
.
getRequestHeader
();
CharSequence
name
=
rh
.
getUrlParam
(
"name"
);
...
...
@@ -285,10 +283,8 @@ public class TextImportProcessor implements HttpRequestProcessor, HttpMultipartC
name
=
partHeader
.
getContentDispositionFilename
();
}
if
(
name
==
null
)
{
transientContext
.
simpleResponse
().
sendStatus
(
400
,
"no name given"
);
// we have to disconnect to interrupt potentially large upload
transientContext
.
getDispatcher
().
disconnect
(
transientContext
);
return
;
transientContext
.
simpleResponse
().
sendStatus
(
400
,
"no file name given"
);
throw
ServerDisconnectException
.
INSTANCE
;
}
transientState
.
analysed
=
false
;
...
...
@@ -304,17 +300,21 @@ public class TextImportProcessor implements HttpRequestProcessor, HttpMultipartC
transientState
.
forceHeader
=
Chars
.
equalsNc
(
"true"
,
rh
.
getUrlParam
(
"forceHeader"
));
transientState
.
messagePart
=
MESSAGE_DATA
;
}
else
if
(
Chars
.
equals
(
"schema"
,
partHeader
.
getContentDispositionName
()
))
{
}
else
if
(
Chars
.
equals
Nc
(
"schema"
,
contentDisposition
))
{
transientState
.
textLoader
.
setState
(
TextLoader
.
LOAD_JSON_METADATA
);
transientState
.
messagePart
=
MESSAGE_SCHEMA
;
}
else
{
// todo: disconnect
transientState
.
messagePart
=
MESSAGE_UNKNOWN
;
if
(
partHeader
.
getContentDisposition
()
==
null
)
{
transientContext
.
simpleResponse
().
sendStatus
(
400
,
"'Content-Disposition' multipart header missing'"
);
}
else
{
transientContext
.
simpleResponse
().
sendStatus
(
400
,
"invalid value in 'Content-Disposition' multipart header"
);
}
throw
ServerDisconnectException
.
INSTANCE
;
}
}
@Override
public
void
onPartEnd
()
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
public
void
onPartEnd
()
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
try
{
LOG
.
debug
().
$
(
"part end"
).
$
();
transientState
.
textLoader
.
wrapUp
();
...
...
@@ -351,14 +351,14 @@ public class TextImportProcessor implements HttpRequestProcessor, HttpMultipartC
@Override
public
void
resumeSend
(
HttpConnectionContext
context
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
doResumeSend
(
LV
.
get
(
context
),
context
.
getChunkedResponseSocket
());
}
private
void
doResumeSend
(
TextImportProcessorState
state
,
HttpChunkedResponseSocket
socket
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
try
{
if
(
state
.
json
)
{
...
...
@@ -374,8 +374,7 @@ public class TextImportProcessor implements HttpRequestProcessor, HttpMultipartC
// is larger that response content buffer
// all we can do in this scenario is to log appropriately
// and disconnect socket
// todo: this is a force disconnect
throw
PeerDisconnectedException
.
INSTANCE
;
throw
ServerDisconnectException
.
INSTANCE
;
}
}
...
...
@@ -410,7 +409,8 @@ public class TextImportProcessor implements HttpRequestProcessor, HttpMultipartC
socket
.
done
();
}
private
void
sendResponse
(
HttpConnectionContext
context
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
{
private
void
sendResponse
(
HttpConnectionContext
context
)
throws
PeerDisconnectedException
,
PeerIsSlowToReadException
,
ServerDisconnectException
{
TextImportProcessorState
state
=
LV
.
get
(
context
);
// todo: may be set this up when headers are ready?
state
.
json
=
Chars
.
equalsNc
(
"json"
,
context
.
getRequestHeader
().
getUrlParam
(
"fmt"
));
...
...
core/src/main/java/io/questdb/network/ServerDisconnectException.java
0 → 100644
浏览文件 @
4127c9ac
/*******************************************************************************
* ___ _ ____ ____
* / _ \ _ _ ___ ___| |_| _ \| __ )
* | | | | | | |/ _ \/ __| __| | | | _ \
* | |_| | |_| | __/\__ \ |_| |_| | |_) |
* \__\_\\__,_|\___||___/\__|____/|____/
*
* Copyright (c) 2014-2019 Appsicle
* Copyright (c) 2019-2020 QuestDB
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
******************************************************************************/
package
io.questdb.network
;
public
class
ServerDisconnectException
extends
Exception
{
public
static
final
ServerDisconnectException
INSTANCE
=
new
ServerDisconnectException
();
}
core/src/test/java/io/questdb/cutlass/http/IODispatcherTest.java
浏览文件 @
4127c9ac
...
...
@@ -409,6 +409,110 @@ public class IODispatcherTest {
"\r\n"
+
"--------------------------27d997ca93d2689d\r\n"
+
"content-disposition: form-data; name=\"data\"; filename=\"fhv_tripdata_2017-02.csv\"\r\n"
+
"content-type: application/octet-stream\r\n"
+
"\r\n"
+
"9988"
+
"\r\n"
+
"--------------------------27d997ca93d2689d--"
,
NetworkFacadeImpl
.
INSTANCE
,
true
,
1
);
}
@Test
public
void
testMissingContentDisposition
()
throws
Exception
{
testImport
(
"HTTP/1.1 400 Bad request\r\n"
+
"Server: questDB/1.0\r\n"
+
"Date: Thu, 1 Jan 1970 00:00:00 GMT\r\n"
+
"Transfer-Encoding: chunked\r\n"
+
"Content-Type: text/html; charset=utf-8\r\n"
+
"\r\n"
+
"31\r\n"
+
"'Content-Disposition' multipart header missing'\r\n"
+
"\r\n"
+
"00\r\n"
+
"\r\n"
,
"POST /upload HTTP/1.1\r\n"
+
"host: localhost:9001\r\n"
+
"User-Agent: curl/7.64.0\r\n"
+
"Accept: */*\r\n"
+
"Content-Length: 437760673\r\n"
+
"Content-Type: multipart/form-data; boundary=------------------------27d997ca93d2689d\r\n"
+
"Expect: 100-continue\r\n"
+
"\r\n"
+
"--------------------------27d997ca93d2689d\r\n"
+
"Content-Type: application/octet-stream\r\n"
+
"\r\n"
+
"9988"
+
"\r\n"
+
"--------------------------27d997ca93d2689d--"
,
NetworkFacadeImpl
.
INSTANCE
,
true
,
1
);
}
@Test
public
void
testMissingContentDispositionName
()
throws
Exception
{
testImport
(
"HTTP/1.1 400 Bad request\r\n"
+
"Server: questDB/1.0\r\n"
+
"Date: Thu, 1 Jan 1970 00:00:00 GMT\r\n"
+
"Transfer-Encoding: chunked\r\n"
+
"Content-Type: text/html; charset=utf-8\r\n"
+
"\r\n"
+
"39\r\n"
+
"invalid value in 'Content-Disposition' multipart header\r\n"
+
"\r\n"
+
"00\r\n"
+
"\r\n"
,
"POST /upload HTTP/1.1\r\n"
+
"host: localhost:9001\r\n"
+
"User-Agent: curl/7.64.0\r\n"
+
"Accept: */*\r\n"
+
"Content-Length: 437760673\r\n"
+
"Content-Type: multipart/form-data; boundary=------------------------27d997ca93d2689d\r\n"
+
"Expect: 100-continue\r\n"
+
"\r\n"
+
"--------------------------27d997ca93d2689d\r\n"
+
"content-disposition: ; filename=\"fhv_tripdata_2017-02.csv\"\r\n"
+
"Content-Type: application/octet-stream\r\n"
+
"\r\n"
+
"9988"
+
"\r\n"
+
"--------------------------27d997ca93d2689d--"
,
NetworkFacadeImpl
.
INSTANCE
,
true
,
1
);
}
@Test
public
void
testMissingContentDispositionFileName
()
throws
Exception
{
testImport
(
"HTTP/1.1 400 Bad request\r\n"
+
"Server: questDB/1.0\r\n"
+
"Date: Thu, 1 Jan 1970 00:00:00 GMT\r\n"
+
"Transfer-Encoding: chunked\r\n"
+
"Content-Type: text/html; charset=utf-8\r\n"
+
"\r\n"
+
"14\r\n"
+
"no file name given\r\n"
+
"\r\n"
+
"00\r\n"
+
"\r\n"
,
"POST /upload HTTP/1.1\r\n"
+
"host: localhost:9001\r\n"
+
"User-Agent: curl/7.64.0\r\n"
+
"Accept: */*\r\n"
+
"Content-Length: 437760673\r\n"
+
"Content-Type: multipart/form-data; boundary=------------------------27d997ca93d2689d\r\n"
+
"Expect: 100-continue\r\n"
+
"\r\n"
+
"--------------------------27d997ca93d2689d\r\n"
+
"content-disposition: form-data; name=\"data\"\r\n"
+
"Content-Type: application/octet-stream\r\n"
+
"\r\n"
+
"9988"
+
...
...
编辑
预览
Markdown
is supported
0%
请重试
或
添加新附件
.
添加附件
取消
You are about to add
0
people
to the discussion. Proceed with caution.
先完成此消息的编辑!
取消
想要评论请
注册
或
登录