DDR3 Test doesn't do anything

[tudang]

Hi Jamey,

I'm able to build the tests/ddr3 but I observe that the test doesn't do anything. The test is wrapped in an IF statement which never fires.

https://github.com/cambridgehackers/connectal/blob/master/tests/ddr3/testddr3.cpp#L56
I've removed the ``if'' to see how it works. The program just hangs forever.

I would like to read and write to DDR3 but it's difficult to follow the test. It's great if you could help me to rewrite the test or an example to use DDR3 with Connectal.

[jameyhicks]

Hi,

If it hangs forever, something is wrong. That being said, I never actually finished this test.

Looking at the test, it does seem to be more complicated than it needs to be. Let's see what I can do.

Also, the modules and interfaces used in the test are documented here: http://www.connectal.org/doc/current/ref/bsv/memreadengine.html

[jameyhicks]

heh, now I understand what some of that code was for.

[jameyhicks]

Pushed a simpler version of the test. Building it for vc707 now. (vc707g2)

[jameyhicks]

I had to add some more cross clock domain constraints to pcie-clocks.xdc.

After running the example, I discovered it did not read from system memory, which it should have done. I used pcieflat to look at PCIE transactions.

So then I looked at the Makefile, which had the read client commented out. Fixed that and rebuilding...

[tudang]

I've built the previous commit on nfsume and got some timing violation errors. I include the log here in case you're interested.

*** timing violation ***
Slack (VIOLATED) :        -3.378ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            host_pcieHostTop_pciehost/traceif/fromPcieTraceBram_cbram_bram/RAM_reg_4/REGCEAREGCE
                            (rising edge-triggered cell RAMB36E1 clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

*** timing violation ***
Slack (VIOLATED) :        -2.928ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_block0_status_reg_replica/C
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_memc_ui_top_axi/u_axi_mc/axi_mc_aw_channel_0/axi_mc_cmd_translator_0/axi_mc_incr_cmd_0/axaddr_incr_reg[9]/D
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -4.924ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_block1_status_reg/C
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})
  Destination:            tile_0_lDdr3Test_dramWriteGearbox_elem_2_reg[110]/CE
                            (rising edge-triggered cell FDRE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

*** timing violation ***
Slack (VIOLATED) :        -6.616ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_elem1_status_0_reg/C
                            (rising edge-triggered cell FDRE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_memc_ui_top_axi/u_axi_mc/axi_mc_aw_channel_0/axi_mc_cmd_translator_0/axi_mc_incr_cmd_0/axaddr_incr_reg[9]/D
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -0.364ns  (arrival time - required time)
  Source:                 tile_0_lDdr3Test_dramWriteGearbox_elem_0_reg[112]/C
                            (rising edge-triggered cell FDRE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_memc_ui_top_axi/u_axi_mc/axi_mc_w_channel_0/mc_app_wdf_data_reg_reg[112]/D
                            (rising edge-triggered cell FDRE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -2.613ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_awfifo/dGDeqPtr_reg[1]/CLR
                            (recovery check against rising-edge clock clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -1.705ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})
  Destination:            tile_0_lDdr3Test_ddr3Controller/rst200/reset_hold_reg[8]/CLR
                            (recovery check against rising-edge clock sys_clk  {rise@0.000ns fall@2.500ns period=5.000ns})

*** timing violation ***
Slack (VIOLATED) :        -3.273ns  (required time - arrival time)
  Source:                 tile_0_lDdr3Test_ddr3Controller/mc/u_axiddr3_mig/u_ddr3_infrastructure/rstdiv0_sync_r1_reg/C
                            (rising edge-triggered cell FDPE clocked by clk_pll_i  {rise@0.000ns fall@2.500ns period=5.000ns})
  Destination:            tile_0_lDdr3Test_dramWriteGearbox_sInReset_pre_isInReset_reg/PRE
                            (recovery check against rising-edge clock userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

*** timing violation ***
Slack (VIOLATED) :        -1.441ns  (required time - arrival time)
  Source:                 host_pcieHostTop_ep7/pcieReset250/reset_hold_reg[3]/C
                            (rising edge-triggered cell FDCE clocked by userclk2  {rise@0.000ns fall@2.000ns period=4.000ns})

[jameyhicks]

I pushed a change to pcie-clocks.xdc to fix those timing violation.

After building c60f0d5 it ran to completion but the data read back from DRAM did not match what I thought I wrote. Please give it a try -- I may not have time to look at it again until tomorrow.

[tudang]

The timing issue has gone. I got the following output when run the program.

src dram[0000]=00000000
src dram[0004]=00000001
src dram[0008]=00000002
...
src dram[03f8]=000000fe
src dram[03fc]=000000ff

The writing task may be blocked.

[jameyhicks]

Please run pcieflat to see if it is doing any DMA.

[tudang]

I guess it does. Here are the first few lines of the output of running pcieflat

             ts     delta   response                     XXX          tlp          address  off   be       tag     clid  nosnp  laddr        data
                                pkttype format               foo (be hit eof sof)            (1st last)        req     stat  bcnt    length
TXqq  450636380          0 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 0000000275287000 be(1st: f last:f) tag:00 reqid:0000 length:32 
RXpp  450636550        170 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:00 8300 00ff 0 00 080 00  16 00000000 
RXcc  450636551          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000004000000030000000200000001 
RXcc  450636552          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000008000000070000000600000005 
RXcc  450636553          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000000c0000000b0000000a00000009 
RXcc  450636554          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:000000000000000f0000000e0000000d 
RXpp  450636555          1 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:00 8300 00ff 0 00 040 40  16 10000000 
RXcc  450636556          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000014000000130000001200000011 
RXcc  450636557          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000018000000170000001600000015 
RXcc  450636558          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000001c0000001b0000001a00000019 
RXcc  450636559          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:000000000000001f0000001e0000001d 
TXqq  450636601         42 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 0000000275287080 be(1st: f last:f) tag:01 reqid:0000 length:32 
...

[jameyhicks]

That is good -- at least it did some reads.

Usually it's the end of the trace that is more interesting.

[jameyhicks]

Have you used Vivado Integrated Logic Analyzer? This might be a good time for it.

If you add mkProbe in your BSV code, its signals will have (* mark_debug="true" *) so that they show up in the netlist after synthesis.

I am interested in how many times rl_req_aw and rl_wdata fire. I would reduce 1024 in Ddr3Test.bsv to 256 or 128.

[tudang]

This is the end of the trace

TXqq  450637930         43 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 0000000275287380 be(1st: f last:f) tag:07 reqid:0000 length:32 
RXpp  450638097        167 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:07 8300 00ff 0 00 080 00  32 e0000000 
RXcc  450638098          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000e4000000e3000000e2000000e1 
RXcc  450638099          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000e8000000e7000000e6000000e5 
RXcc  450638100          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000ec000000eb000000ea000000e9 
RXcc  450638101          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000f0000000ef000000ee000000ed 
RXcc  450638102          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000f4000000f3000000f2000000f1 
RXcc  450638103          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000f8000000f7000000f6000000f5 
RXcc  450638104          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000fc000000fb000000fa000000f9 
RXcc  450638105          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:00000000000000ff000000fe000000fd 
TXqq 3035184128 2584546023 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 00000002747a7000 be(1st: f last:f) tag:08 reqid:0000 length:32 
RXpp 3035184301        173 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 1 0 1)                        tag:08 8300 00ff 0 00 080 00  32 00000000 
RXcc 3035184302          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000004000000030000000200000001 
RXcc 3035184303          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000008000000070000000600000005 
RXcc 3035184304          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000000c0000000b0000000a00000009 
RXcc 3035184305          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:000000100000000f0000000e0000000d 
RXcc 3035184306          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000014000000130000001200000011 
RXcc 3035184307          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:00000018000000170000001600000015 
RXcc 3035184308          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:0000001c0000001b0000001a00000019 
RXcc 3035184309          1 DmaRCon                         08 0x4 tlp(fff0 0 1 0)                            data:000000000000001f0000001e0000001d 
{'DmaRReq': 9, 'DmaRRsp': 14, 'DmaRCon': 72}
95

[pietrobressana]

Hi Jamey,

I'm working on Tu's DDR3 project, in order to add DDR3 capability to our Paxos implementation.

Can you please answer the following questions?

• Is Xilinx MIG supported into NetFPGA SUME framework? Is there any project that enables DDR3 capability on SUME?
• Is connectal's DDR3 test application working with SUME?
• Which DDR3 controller do you use in connectal? Do you use Xilinx MIG, or a custom one? Is connectal's controller ready to be used with SUME?

Thanks

[jameyhicks]

Hi Pietro,

I have not used DDR3 on the SUME board. All the projects using connectal that use DDR3 use the Xilinx MIG. Some of these projects use the DDR3 code in the Bluespec distribution, which was generated by Xilinx MIG. For example: bluespec/lib/board_support/bluenoc/xilinx/VC709/verilog/ddr3_v2_1/ddr3_v2_1

Because connectal is open source, I cannot really check in the output of the MIG, so my approach is to generate the cores as needed.

This test application has never been fully debugged, but I think we're close to having it working now. I will spend some more time on it today.