This commit update coh check assertion to enable aggressive
prefetch miss req / store miss req merge. Previous wrong
assertion forbids store req from stoping a previous prefetch

For example, consider 2 reqs with the same p address fire in order:
1) A prefetch with alias bit 00 need to change coh state from N->T
2) A store with alias bit 11 need to change coh state from B->T
Then prefetch and store miss req will be merged in the same missq
entry. Store req (2) should be able to stop prefetch (1) so that a
ping-pong process will not start

Note that Huancun have not been updated in this commit

Added meta_prefetch and meta_access related sim perf counter

For now, optional dcache meta prefetch and access can be removed safely

This PR remove data in lq.

All cache miss load instructions will be replayed by lq, and the forward path to the D channel
and mshr is added to the pipeline.
Special treatment is made for uncache load. The data is no longer stored in the datamodule
but stored in a separate register. ldout is only used as uncache writeback, and only ldout0
will be used. Adjust the priority so that the replayed instruction has the highest priority in S0.

Future work:
1. fix `milc` perf loss
2. remove data from MSHRs

* difftest: monitor cache miss latency

* lq, ldu, dcache: remove lq's data

* lq's data is no longer used
* replay cache miss load from lq (use counter to delay)
* if dcache's mshr gets refill data, wake up lq's missed load
* uncache load will writeback to ldu using ldout_0
* ldout_1 is no longer used

* lq, ldu: add forward port

* forward D and mshr in load S1, get result in S2
* remove useless code logic in loadQueueData

* misc: revert monitor

* misc: add utility submodule

* misc: adjust to new utility framework

* bump utility: revert resetgen

* bump huancun

* pipelineReg in miss queue
* translated_cache_req_opCode and io_cache_req_valid_reg in cacheOpDecoder
* r_way_en_reg in bankedDataArray

This commit and an extra cycle for miss queue store data and mask write.
For now, there are 16 missqueue entries. Each entry has a 512 bit store
data reg and a 64 bit store mask. If we update miss queue data in 1
cycle, the fanout will be at least 16x(512+64) = 9216.

Now missqueue req meta update is unchanged, however, store data and mask
update will happen 1 cycle after primary fire or secondary fire (T0).
In T0, store data and meta will be written to a buffer in missqueue.
In T1, s_write_storedata in each missqueue entry will be used as store
data and mask wen.

Miss queue entry data organization is also optimized. 512 bit
req.store_data is removed from miss queue entry. It should save
8192 bits in total.

replace_pipe_req is causing timing problem as vaddr in it is used to
generate mainpipe block signal. Unfortunately, vaddr from
replace_pipe_req is selected form all miss queue entries
(16 by default), which caused timing problem

refill_pipe_req will not be scheduled until dcache main pipe s3 reports
that replace_pipe_req has been finished. Thus it is legal to add a
pipe reg for replace_pipe_req

Now ALL mainpipe req candidates come from pipe reg. At the entry of
main pipe, 1 req is selected from 4 main pipe reqs, and its vaddr is
used to calcuate set block condition

* dcache: fix bug that a block could be released twice

* MainPipe: fix bug in way_en of miss_req

* MainPipe: fix bug

* mem: fix error csr update

* dcache: l2 error will now trigger atom error

* chore: fix cache error debug decoder

* mem: split L1CacheErrorInfo and L1BusErrorUnitInfo

* dcache: let ecc error and l2 corrupt raise load af

If CSR.smblockctl.cache_error_enable is disabled, ecc error and l2 corrupt
will not raise any exception.

* mem: enable cache error by default

* mem: support store ecc check, add ecc error csr

Support store / atom ecc check (early version)
Add ecc error csr to distingush ecc error and other access fault

Timing opt and unit tests to be added.

* MissQueue: loose merging condition to ease timing stress

* MissQueue: remove grant_beats

* MissQueue: compare block addr, not the whole addr bits

* dcache: optimize timing for generating ready to sbuffer
Co-authored-by: NWilliam Wang <zeweiwang@outlook.com>

This commit optimizes the coding style and timing for hardware
performance counters.

By default, performance counters are RegNext(RegNext(_)).

Now we RegNext(refill_req) for 1 cycle. It will provide more
time for refillShouldBeBlocked calcuation

* Add 1 cycle in refill pipe
Co-authored-by: Nzhanglinjuan <zhanglinjuan20s@ict.ac.cn>

* sbuffer: do flush correctly while draining sbuffer

* ci: enable ci for timing-memblock branch

* mem: disable EnableFastForward for timing reasons

* sbuffer: optimize forward mask gen timing

* dcache: block main pipe req if refill req is valid

Refill req comes from refill arbiter. There is not time left for index
conflict check. Now we simplily block all main pipe req when refill
req comes from miss queue.

* dcache: delay some resp signals for better timing

* dcache: optimize wbq enq entry select timing

* WritebackQueue: optimize enqueue logic fir timing

* WritebackQueue: always reject a req when wbq is full

* Revert "ci: enable ci for timing-memblock branch"

This reverts commit 32453dc4.

* WritebackQueue: fix bug in secondary_valid
Co-authored-by: Nzhanglinjuan <zhanglinjuan20s@ict.ac.cn>

* dcache: merge replace pipe with main pipe for timing reason

* MainPipe: fix bug in s3_fire

* MainPipe: fix bug in delay_release sent to wbq

* MainPipe: fix bug in blocking policy

* MainPipe: send io.replace_resp in stage 3

* MainPipe: fix bug in miss_id sent to wbq

* MainPipe: fix bug
Co-authored-by: NWilliam Wang <zeweiwang@outlook.com>

* sbuffer: do flush correctly while draining sbuffer

* mem: disable EnableFastForward for timing reasons

* sbuffer: optimize forward mask gen timing

* dcache: block main pipe req if refill req is valid

Refill req comes from refill arbiter. There is not time left for index
conflict check. Now we block all main pipe req when refill
req comes from miss queue.

* dcache: delay some resp signals for better timing

* dcache: optimize wbq enq entry select timing

* dcache: decouple missq req.valid to valid & cancel

* valid is fast, it is used to select which miss req will be sent to
miss queue
* cancel can be slow to generate, it will cancel miss queue req in the
last moment

* sbuffer: optimize noSameBlockInflight check timing

* FDivSqrt: use hierarchy API to avoid dedup bug

* Dedup: use hartId from io port instead of core parameters

* Bump fudian

DCache timing problem has not been solved yet. DCache structure will be further changed.

* sbuffer: add extra perf counters

* sbuffer: optmize timeout replay check timing

* sbuffer: optmize do_uarch_drain check timing

Now we only compare merge entry's vtag, check will not start until
mergeIdx is generated by PriorityEncoder

* mem, lq: optmize writeback select logic timing

* dcache: replace missqueue reill req arbiter

* dcache: refactor missqueue entry select logic

* mem: add comments for lsq data

* dcache: give amo alu an extra cycle

* sbuffer: optmize sbuffer forward data read timing

* disable log as default
* code clean up

* dcache: do not check readline rmask

This should opt bank_conflict check timing

* dcache: block replace if store s1 valid

It takes quite long to generate way_en in mainpipe s1. As a result,
use s1 way_en to judge if replace should be blocked will cause severe
timing problem

Now we simply block replace if mainpipe.s1.valid

Refill timing to be optmized later

* sbuffer: delay sbuffer enqueue for 1 cycle

With store queue growing larger, read data from datamodule nearly
costs a whole cycle. Hence we delay sbuffer enqueue for 1 cycle
for better timing.

* dcache: reduce probe queue size

* dcache: replace probe pipe req RRArbiter with Arbiter

* dcache: reduce writeback queue size for timing opt

* dcache: delay wbqueue enqueue req for 1 cycle

Addr enqueue req will compare its addr with addrs in all writeback
entries to check if it should be blocked. Delay enqueue req will
give that process more time.

* dcache: set default replacer to setplru

It does not change current design

* dcache: fix wbqueue req_delayed deadlock

We delayed writeback queue enq for 1 cycle, missQ req does not
depend on wbQ enqueue. As a result, missQ req may be blocked
in req_delayed. When grant comes, that req should also be updated

* dcache: remove outdated require

* dcache: replace missReqArb RRArbiter with Arbiter

* perf: add detailed histogram for low dcache latency

* dcache: fix wbqueue entry alloc logic

* dcache: opt probe req timing

In current design, resv_set is maintained in dcache. All probe req
will be blocked if that addr is in resv_set.

However, checking if that addr is in resv_set costs almost half a cycle,
which causes severe timing problem.

Now when we update update_resv_set, all probe reqs will be blocked
in the next cycle. It should give Probe reservation set addr compare an
independent cycle, which will lead to better timing

* Add perf counters
* add reg from hpm counter source
* add print perfcounter enable

* L1D: provide independent meta array for load pipe

* misc: reorg files in cache dir

* chore: reorg l1d related files

* bump difftest: use clang to compile verialted files

* dcache: add BankedDataArray

* dcache: fix data read way_en

* dcache: fix banked data wmask

* dcache: replay conflict correctly

 When conflict is detected:
* Report replay
* Disable fast wakeup

* dcache: fix bank addr match logic

* dcache: add bank conflict perf counter

* dcache: fix miss perf counters

* chore: make lsq data print perttier

* dcache: enable banked ecc array

* dcache: set dcache size to 128KB

* dcache: read mainpipe data from banked data array

* dcache: add independent mainpipe data read port

* dcache: revert size change

* Size will be changed after main pipe refactor

* Merge remote-tracking branch 'origin/master' into l1-size

* dcache: reduce banked data load conflict

* MainPipe: ReleaseData for all replacement even if it's clean

* dcache: set dcache size to 128KB

BREAKING CHANGE: l2 needed to provide right vaddr index to probe l1,
and it has to help l1 to avoid addr alias problem

* chore: fix merge conflict

* Change L2 to non-inclusive / Add alias bits in L1D

* debug: hard coded dup data array for debuging

* dcache: fix ptag width

* dcache: fix amo main pipe req

* dcache: when probe, use vaddr for main pipe req

* dcache: include vaddr in atomic unit req

* dcache: fix get_tag() function

* dcache: fix writeback paddr

* huancun: bump version

* dcache: erase block offset bits in release addr

* dcache: do not require probe vaddr != 0

* dcache: opt banked data read timing

* bump huancun

* dcache: fix atom unit pipe req vaddr

* dcache: simplify main pipe writeback_vaddr

* bump huancun

* dcache: remove debug data array

* Turn on all usr bits in L1

* Bump huancun

* Bump huancun

* enable L2 prefetcher

* bump huancun

* set non-inclusive L2/L3 + 128KB L1 as default config

* Use data in TLBundleB to hint ProbeAck beeds data

* mmu.l2tlb: mem_resp now fills multi mq pte buffer

mq entries can just deq without accessing l2tlb cache

* dcache: handle dirty userbit

* bump huancun

* chore: l1 cache code clean up

* Remove l1plus cache
* Remove HasBankedDataArrayParameters

* Add bus pmu between L3 and Mem

* bump huncun

* IFU: add performance counters and mmio af

* icache replacement policy moniter

* ifu miss situation moniter

* icache miss rate

* raise access fault when found mmio req

* Add framework for seperated main pipe and reg meta array

* Rewrite miss queue for seperated pipes

* Add RefillPipe

* chore: rename NewSbuffer.scala

* cache: add CacheInstruction opcode and reg list

* CSR: add cache control registers

* Add Replace Pipe

* CacheInstruction: add CSRs for cache instruction

* mem: remove store replay unit

* Perf counter to be added
* Timing opt to be done

* mem: update sbuffer to support new dcache

* sbuffer: fix missqueue time out logic

* Merge remote-tracking branch 'origin/master' into dcache-rm-sru

* chore: fix merge conflict, remove nStoreReplayEntries

* Temporarily disable TLMonitor

* Bump huancun (L2/L3 MSHR bug fix)

* Rewrite main pipe

* ReplacePipe: read meta to decide whether data should be read

* RefillPipe: add a store resp port

* MissQueue: new req should be rejected according to set+way

* Add replacement policy interface

* sbuffer: give missq replay the highest priority

Now we give missqReplayHasTimeOut the highest priority, as eviction
has already happened
Besides, it will fix the problem that fix dcache eviction generate logic
gives the wrong sbuffer id

* Finish DCache framework

* Split meta & tag and use regs to build meta array

* sbuffer: use new dcache io

* dcache: update dcache resp in memblock and fake d$

* Add atomics processing flow

* Refactor Top

* Bump huancun

* DCacheWrapper: disable ld fast wakeup only when bank conflict

* sbuffer: update dcache_resp difftest io

* MainPipe: fix combinational loop

* Sbuffer: fix bug in assert

* RefillPipe: fix bug of getting tag from addr

* dcache: ~0.U should restrict bit-width

* LoadPipe: fix bug in assert

* ReplacePipe: addr to be replaced should be block-aligned

* MainPipe: fix bug in required coh sending to miss queue

* DCacheWrapper: tag write in refill pipe should always be ready

* MainPipe: use replacement way_en when the req is from miss queue

* MissQueue: refill data should be passed on to main pipe

* MainPipe: do not use replacement way when tag match

* CSR: clean up cache op regs

* chore: remove outdated comments

* ReplacePipe: fix stupid bug

* dcache: replace checkOneHot with assert

* alu: fix bug of rev8 & orc.b instruction

* MissQueue: fix bug in the condition of mshr accepting a req

* MissQueue: add perf counters

* chore: delete out-dated code

* chore: add license

* WritebackQueue: distinguish id from miss queue

* AsynchronousMetaArray: fix bug

* Sbuffer: fix difftest io

* DCacheWrapper: duplicate one more tag copy for main pipe

* Add perf cnt to verify whether replacing is too early

* dcache: Release needs to wait for refill pipe

* WritebackQueue: fix accept condition

* MissQueue: remove unnecessary assert

* difftest: let refill check ingore illegal mem access

* Parameters: enlarge WritebackQueue to break dead-lock

* DCacheWrapper: store hit wirte should not be interrupted by refill

* Config: set nReleaseEntries to twice of nMissEntries

* DCacheWrapper: main pipe read should block refill pipe by set
Co-authored-by: NWilliam Wang <zeweiwang@outlook.com>
Co-authored-by: NLinJiawei <linjiav@outlook.com>
Co-authored-by: NTangDan <tangdan@ict.ac.cn>
Co-authored-by: NLinJiawei <linjiawei20s@ict.ac.cn>
Co-authored-by: NZhangZifei <zhangzifei20z@ict.ac.cn>
Co-authored-by: Nwangkaifan <wangkaifan@ict.ac.cn>
Co-authored-by: NJinYue <jinyue20s@ict.ac.cn>
Co-authored-by: Zhangfw <471348957@qq.com>

* bump chisel to 3.5.0-RC1

We don't want to use SNAPSHOT version any more because we don't know
what will happen when we wake up in the morning.

* misc: remove TMA_* to avoid conflicts

* add soft prefetch
Add the softprefetch. Actually, prefetch.r&w are an ORI which's ldest is x0, we distinguish it in decodeUnit and send it to ld func unit. Then, we modified some interaction signals in ordinary Load steps.

* rename Roq to Rob

* remove trailing whitespaces

* remove unused parameters

* L1D: provide independent meta array for load pipe

* misc: reorg files in cache dir

* chore: reorg l1d related files

* bump difftest: use clang to compile verialted files

* dcache: add BankedDataArray

* dcache: fix data read way_en

* dcache: fix banked data wmask

* dcache: replay conflict correctly

 When conflict is detected:
* Report replay
* Disable fast wakeup

* dcache: fix bank addr match logic

* dcache: add bank conflict perf counter

* dcache: fix miss perf counters

* chore: make lsq data print perttier

* dcache: enable banked ecc array

* dcache: set dcache size to 128KB

* dcache: read mainpipe data from banked data array

* dcache: add independent mainpipe data read port

* dcache: revert size change

* Size will be changed after main pipe refactor

* Merge remote-tracking branch 'origin/master' into l1-size

* dcache: reduce banked data load conflict

* MainPipe: ReleaseData for all replacement even if it's clean

* dcache: set dcache size to 128KB

BREAKING CHANGE: l2 needed to provide right vaddr index to probe l1,
and it has to help l1 to avoid addr alias problem

* chore: fix merge conflict

* Change L2 to non-inclusive / Add alias bits in L1D

* debug: hard coded dup data array for debuging

* dcache: fix ptag width

* dcache: fix amo main pipe req

* dcache: when probe, use vaddr for main pipe req

* dcache: include vaddr in atomic unit req

* dcache: fix get_tag() function

* dcache: fix writeback paddr

* huancun: bump version

* dcache: erase block offset bits in release addr

* dcache: do not require probe vaddr != 0

* dcache: opt banked data read timing

* bump huancun

* dcache: fix atom unit pipe req vaddr

* dcache: simplify main pipe writeback_vaddr

* bump huancun

* dcache: remove debug data array

* Turn on all usr bits in L1

* Bump huancun

* Bump huancun

* enable L2 prefetcher

* bump huancun

* set non-inclusive L2/L3 + 128KB L1 as default config

* Use data in TLBundleB to hint ProbeAck beeds data

* mmu.l2tlb: mem_resp now fills multi mq pte buffer

mq entries can just deq without accessing l2tlb cache

* dcache: handle dirty userbit

* bump huancun

* chore: l1 cache code clean up

* Remove l1plus cache
* Remove HasBankedDataArrayParameters

* Add bus pmu between L3 and Mem

* bump huncun

* dcache: fix l1 probe index generate logic

* Now right probe index will be used according to the len of alias bits

* dcache: clean up amo pipeline

* DCacheParameter rowBits will be removed in the future, now we set it to 128
to make dcache work

* dcache: fix amo word index

* bump huancun
Co-authored-by: NWilliam Wang <zeweiwang@outlook.com>
Co-authored-by: Nzhanglinjuan <zhanglinjuan20s@ict.ac.cn>
Co-authored-by: NTangDan <tangdan@ict.ac.cn>
Co-authored-by: NZhangZifei <zhangzifei20z@ict.ac.cn>
Co-authored-by: Nwangkaifan <wangkaifan@ict.ac.cn>