regular commit

README.md

 # candock
 这是一个用于记录毕业设计的日志仓库，其目的是尝试多种不同的深度神经网络结构(如LSTM,RESNET,DFCNN等)对单通道EEG进行自动化睡眠阶段分期.我们相信这些代码同时可以用于其他生理信号(如ECG,EMG等)的分类.希望这将有助于您的研究.<br>
 ## 数据集
-使用了三个睡眠数据集进行测试,分别是: [CinC Challenge 2018](https://physionet.org/physiobank/database/challenge/2018/#files)   [sleep-edf](https://www.physionet.org/physiobank/database/sleep-edf/)   [sleep-edfx](https://www.physionet.org/physiobank/database/sleep-edfx/) <br>
+使用了三个睡眠数据集进行测试,分别是:   [[CinC Challenge 2018]](https://physionet.org/physiobank/database/challenge/2018/#files)    [[sleep-edf]](https://www.physionet.org/physiobank/database/sleep-edf/)   [[sleep-edfx]](https://www.physionet.org/physiobank/database/sleep-edfx/) <br>
 对于CinC Challenge 2018数据集,使用其C4-M1通道<br>对于sleep-edfx与sleep-edf数据集,使用Fpz-Cz通道<br>
 值得注意的是:sleep-edfx是sleep-edf的扩展版本.<br>
 
 ## 一些说明
-* 数据集分割<br>
-  读取数据集各样本后分割为30s/Epoch作为一个输入,共有5个标签,分别是Sleep stage 3,2,1,R,W,将分割后的eeg信号与睡眠阶段标签进行对应后,打乱其顺序,并将80%的数据用于训练,20%的数据用于测试.
+* 对数据集进行的处理<br>
+  读取数据集各样本后分割为30s/Epoch作为一个输入,共有5个标签,分别是Sleep stage 3,2,1,R,W,将分割后的eeg信号与睡眠阶段标签进行对应后,打乱其顺序,并将80%的数据用于训练,20%的数据用于测试.<br>
+
+  注意:对于sleep-edfx数据集,我们仅仅截取了入睡前30分钟到醒来后30分钟之间的睡眠区间作为读入数据(实验结果中用only sleep time 进行标注),目的是平衡各睡眠时期的比例并加快训练速度.
+
 * 数据预处理<br>
   对于不同的网络结构,对原始eeg信号采取了预处理,使其拥有不同的shape:<br>
   LSTM:将30s的eeg信号进行FIR带通滤波,获得θ,σ,α,δ,β波,并将它们进行连接后作为输入数据<br>
   resnet_1d:这里使用resnet的一维形式进行实验,(修改nn.Conv2d为nn.Conv1d).<br>
   DFCNN:将30s的eeg信号进行短时傅里叶变换,并生成频谱图作为输入,并使用resnet网络进行分类.<br>
+
 * EEG频谱图<br>
   这里展示5个睡眠阶段对应的频谱图,它们依次是wake, stage 1, stage 2, stage 3, REM
   ![image](https://github.com/HypoX64/candock/blob/master/image/spectrum_wake.png)
@@ -20,15 +24,26 @@
   ![image](https://github.com/HypoX64/candock/blob/master/image/spectrum_N2.png)
   ![image](https://github.com/HypoX64/candock/blob/master/image/spectrum_N3.png)
   ![image](https://github.com/HypoX64/candock/blob/master/image/spectrum_REM.png)
+
 * 关于代码<br>
   目前的代码仍然在不断修改与更新中,不能确保其能工作.详细内容将会在毕业设计完成后抽空更新.<br>
 ## 部分实验结果
-该部分将持续更新... ...
+该部分将持续更新... ...<br>
+[[Confusion matrix]](https://github.com/HypoX64/candock/blob/master/image/confusion_mat)<br>
 * sleep-edf<br>
-  [[Confusion matrix]](https://github.com/HypoX64/candock/blob/master/image/confusion_mat)<br>
-  
   | Network        | Label average recall | Label average accuracy | error rate |
   | :------------- | :------------------- | ---------------------- | ---------- |
   | lstm           | 0.8342               | 0.9611                 | 0.0974     |
-  | resnet18_1d    | 0.8434               | 0.9627                 | 0.093      |
+  | resnet18_1d    | 0.8434               | 0.9627                 | 0.0930     |
   | DFCNN+resnet18 | 0.8567               | 0.9663                 | 0.0842     |
+  | DFCNN+resnet50 | 0.7916               | 0.9607                 | 0.0983     |
+<br>
+* sleep-edfx(only sleep time)<br>
+  | Network        | Label average recall | Label average accuracy | error rate |
+  | :------------- | :------------------- | ---------------------- | ---------- |
+  | lstm           | 0.7864               | 0.9166                 | 0.2085     |
+  | resnet18_1d    | xxxxxx               | xxxxxx                 | xxxxxx     |
+  | DFCNN+resnet18 | xxxxxx               | xxxxxx                 | xxxxxx     |
+  | DFCNN+resnet50 | xxxxxx               | xxxxxx                 | xxxxxx     |
+<br>
+* CinC Challenge 2018<br>
\ No newline at end of file

data.py

@@ -49,8 +49,8 @@ def random_transform_1d(data,finesize,test_flag):
         result = data[move:move+finesize]
 
         #random flip
-        # if random.random()<0.5:
-        #     result = result[::-1]
+        if random.random()<0.5:
+            result = result[::-1]
 
         #random amp
         result = result*random.uniform(0.95,1.05)

dataloader.py

@@ -72,7 +72,7 @@ def loaddata(dirpath,signal_name,BID = 'median',filter = True):
     # print(stages.shape,signals.shape)
     return signals,stages
 
-def loaddata_sleep_edf(filedir,filenum,signal_name,BID = 'median',filter = True):
+def loaddata_sleep_edf(opt,filedir,filenum,signal_name,BID = 'median',filter = True):
     filenames = os.listdir(filedir)
     for filename in filenames:
         if str(filenum) in filename and 'Hypnogram' in filename:
@@ -84,7 +84,7 @@ def loaddata_sleep_edf(filedir,filenum,signal_name,BID = 'median',filter = True)
     raw_data= mne.io.read_raw_edf(os.path.join(filedir,f_signal_name),preload=True)
     raw_annot = mne.read_annotations(os.path.join(filedir,f_stage_name))
     eeg = raw_data.pick_channels([signal_name]).to_data_frame().values.T
-    signals = eeg.reshape(-1)
+    eeg = eeg.reshape(-1)
 
     raw_data.set_annotations(raw_annot, emit_warning=False)
     event_id = {'Sleep stage 4': 0,
@@ -97,14 +97,21 @@ def loaddata_sleep_edf(filedir,filenum,signal_name,BID = 'median',filter = True)
                   'Movement time': 5}
     events, _ = mne.events_from_annotations(
         raw_data, event_id=event_id, chunk_duration=30.)
-    
-    signals = signals[events[0][0]:events[-1][0]]
-    events = np.array(events)
-    signals = signals.reshape(-1,3000)
-    # signals = signals*13/np.median(np.abs(signals))
-    stages = events[:,2]
-    stages = stages[:len(signals)]
 
+    stages = []
+    signals =[]
+    for i in range(len(events)-1):
+        stages.append(events[i][2])
+        signals.append(eeg[events[i][0]:events[i][0]+3000])
+    stages=np.array(stages)
+    signals=np.array(signals)
+    signals = signals*13/np.median(np.abs(signals))
+
+    # #select sleep time 
+    if opt.select_sleep_time:
+        if 'SC' in f_signal_name:
+            signals = signals[np.clip(int(raw_annot[0]['duration'])//30-60,0,9999999):int(raw_annot[-2]['onset'])//30+60]
+            stages = stages[np.clip(int(raw_annot[0]['duration'])//30-60,0,9999999):int(raw_annot[-2]['onset'])//30+60]
 
     stages_copy = stages.copy()
     cnt = 0
@@ -147,7 +154,7 @@ def loaddata_sleep_edf(filedir,filenum,signal_name,BID = 'median',filter = True)
     return signals.astype(np.int16),stages.astype(np.int16)
 
 
-def loaddataset(filedir,dataset_name = 'CinC_Challenge_2018',signal_name = 'C4-M1',num = 100 ,BID = 'median',shuffle = True):
+def loaddataset(opt,filedir,dataset_name = 'CinC_Challenge_2018',signal_name = 'C4-M1',num = 100 ,BID = 'median',shuffle = True):
     print('load dataset, please wait...')
     filenames = os.listdir(filedir)
 
@@ -180,7 +187,7 @@ def loaddataset(filedir,dataset_name = 'CinC_Challenge_2018',signal_name = 'C4-M
         cnt = 0
         for filename in filenames:
             if 'PSG' in filename:
-                signal,stage = loaddata_sleep_edf(filedir,filename[2:6],signal_name = 'EEG Fpz-Cz')
+                signal,stage = loaddata_sleep_edf(opt,filedir,filename[2:6],signal_name = 'EEG Fpz-Cz')
                 if cnt == 0:
                     signals =signal.copy()
                     stages =stage.copy()

image/confusion_mat

@@ -20,7 +20,16 @@ confusion_mat:
  [   3    1   51   18 1558]]
 
 
-DFresnet18
+resnet18_1d_sleep-edf
+avg_recall:0.8434  avg_acc:0.9627  error:0.0930
+confusion_mat:
+[[ 225   37    1    0    1]
+ [  31  653   27   31    0]
+ [   0    4   85   28    1]
+ [   0   19   43  261    3]
+ [   1    3   44    9 1533]]
+
+DFCNN+resnet18
 avg_recall:0.8567  avg_acc:0.9663  error:0.0842
 confusion_mat:
 [[ 238    9    2    1    0]
@@ -29,14 +38,22 @@ confusion_mat:
  [   0   17   17  261    2]
  [   1    1   27    7 1621]]
 
-
-resnet18_1d_sleep-edf
-avg_recall:0.8434  avg_acc:0.9627  error:0.0930
+DFCNN+resnet50
+avg_recall:0.7916  avg_acc:0.9607  error:0.0983
 confusion_mat:
-[[ 225   37    1    0    1]
- [  31  653   27   31    0]
- [   0    4   85   28    1]
- [   0   19   43  261    3]
- [   1    3   44    9 1533]]
+[[ 178   64    2    0    1]
+ [  15  653    7   24    1]
+ [   0   15   49   42    7]
+ [   0   20    6  320    1]
+ [   4   11   40   38 1534]]
+
+-------------------sleep-edfx(only sleep time)-------------------
 
--------------------sleep-edfx-------------------
+LSTM
+avg_recall:0.7864  avg_acc:0.9166  error:0.2085
+confusion_mat:
+[[ 3495   396    20     5     4]
+ [ 1323 13432  2033  1024    82]
+ [   26   699  3159   775   384]
+ [    5   380   971  5377   111]
+ [   13    34  1490   191 12379]]

options.py

@@ -4,7 +4,7 @@ import numpy as np
 import torch
 #filedir = '/media/hypo/Hypo/physionet_org_train'
 # filedir ='E:\physionet_org_train'
-
+#python3 train.py --dataset_name sleep-edf --model_name resnet50 --batchsize 4 --epochs 50 --pretrained
 #'/media/hypo/Hypo/physionet_org_train'
 class Options():
     def __init__(self):
@@ -19,6 +19,7 @@ class Options():
         self.parser.add_argument('--dataset_dir', type=str, default='./datasets/sleep-edfx/',
                                 help='your dataset path')
         self.parser.add_argument('--dataset_name', type=str, default='sleep-edf',help='Choose dataset')
+        self.parser.add_argument('--select_sleep_time', action='store_true', help='if input, for sleep-cassette only use sleep time to train')
         self.parser.add_argument('--signal_name', type=str, default='EEG Fpz-Cz',help='Choose the EEG channel C4-M1|EEG Fpz-Cz')
         self.parser.add_argument('--sample_num', type=int, default=20,help='the amount you want to load')
         self.parser.add_argument('--model_name', type=str, default='resnet18',help='Choose model')
@@ -27,6 +28,7 @@ class Options():
         self.parser.add_argument('--network_save_freq', type=int, default=5,help='the freq to save network')
 
 
+
         self.initialized = True
 
     def getparse(self):
@@ -38,17 +40,19 @@ class Options():
             self.opt.sample_num = 8
 
 
-        if self.opt.weight_mod == 'normal':
-            weight = np.array([1,1,1,1,1])
-        elif self.opt.weight_mod == 'avg_best':
-            if self.opt.dataset_name == 'CinC_Challenge_2018':
-                weight = np.log(1/np.array([0.15,0.3,0.08,0.13,0.18]))
-            elif self.opt.dataset_name == 'sleep-edfx':
-                weight = np.log(1/np.array([0.04,0.20,0.04,0.08,0.63]))
-            elif self.opt.dataset_name == 'sleep-edf':
-                weight = np.log(1/np.array([0.08,0.23,0.01,0.10,0.53]))
+        # if self.opt.weight_mod == 'normal':
+        #     weight = np.array([1,1,1,1,1])
+        # elif self.opt.weight_mod == 'avg_best':
+        #     if self.opt.dataset_name == 'CinC_Challenge_2018':
+        #         weight = np.log(1/np.array([0.15,0.3,0.08,0.13,0.18]))
+        #     elif self.opt.dataset_name == 'sleep-edfx':
+        #         weight = np.log(1/np.array([0.04,0.20,0.04,0.08,0.63]))
+        #     elif self.opt.dataset_name == 'sleep-edf':
+        #         weight = np.log(1/np.array([0.08,0.23,0.01,0.10,0.53]))
+        #         if self.opt.select_sleep_time:
+        #             weight = np.log(1/np.array([0.16,0.44,0.05,0.19,0.53]))
       
-        self.opt.weight = weight
+        # self.opt.weight = weight
 
 
         return self.opt
\ No newline at end of file

train.py

@@ -21,7 +21,7 @@ localtime = time.asctime(time.localtime(time.time()))
 statistics.writelog('\n\n'+str(localtime)+'\n'+str(opt))
 
 t1 = time.time()
-signals,stages = dataloader.loaddataset(opt.dataset_dir,opt.dataset_name,opt.signal_name,opt.sample_num,shuffle=True,BID='median')
+signals,stages = dataloader.loaddataset(opt,opt.dataset_dir,opt.dataset_name,opt.signal_name,opt.sample_num,shuffle=True,BID='median')
 stage_cnt_per = statistics.stage(stages)[1]
 print('stage_cnt_per:',stage_cnt_per,'\nlength of dataset:',len(stages))
 signals_train,stages_train,signals_eval,stages_eval, = data.batch_generator(signals,stages,opt.batchsize,shuffle = True)
@@ -38,7 +38,12 @@ net=models.CreatNet(opt.model_name)
 if opt.pretrained:
     net.load_state_dict(torch.load('./checkpoints/pretrained/'+opt.model_name+'.pth'))
 
-weight = torch.from_numpy(opt.weight).float()
+weight = np.array([1,1,1,1,1])
+if opt.weight_mod == 'avg_best':
+    weight = np.log(1/stage_cnt_per)
+    weight[2] = weight[2]+1
+print(weight)
+weight = torch.from_numpy(weight).float()
 # print(net)
 if not opt.no_cuda:
     net.cuda()