About the support for nutshell

[zhengkeng]

I am currently working on enabling xfuzz support for NutShell(as DUT) and have made the following modifications:
I added ccover support in build.sc:

import mill., scalalib.
import coursier.maven.MavenRepository
import mill._
import mill.scalalib._
import mill.scalalib.publish._
import $file.difftest.build

object ivys {
val scala = "2.13.10"
val chisel_v = "3.6.0"
val chisel = ivy"edu.berkeley.cs::chisel3:3.6.0"
val chiselPlugin = ivy"edu.berkeley.cs:::chisel3-plugin:3.6.0"
}

trait CommonModule extends ScalaModule {
override def scalaVersion = ivys.scala

override def scalacOptions = Seq("-Ymacro-annotations")
}

trait HasChisel extends ScalaModule {
override def ivyDeps = Agg(ivys.chisel)
override def scalacPluginIvyDeps = Agg(ivys.chiselPlugin)
}

trait CcoverModule extends SbtModule
with HasChisel{

def scalaVersion: T[String] = T(ivys.scala)

def sourceRoot = T.sources { T.workspace / "ccover" / "instrumentation" / "src" }

private def getSources(p: PathRef) = if (os.exists(p.path)) os.walk(p.path) else Seq()

def allSources = T { sourceRoot().flatMap(getSources).map(PathRef(_)) }

def chiselModule: Option[ScalaModule] = None

def chiselPluginJar: T[Option[PathRef]] = None

def chiselIvy = Some(ivys.chisel3)

def chiselPluginIvy = Some(ivys.chisel3Plugin)

def ivyDeps = super.ivyDeps() ++ Agg(ivy"edu.berkeley.cs::chiseltest:0.6.2")

}

object ccover extends CcoverModule

trait CommonNS extends SbtModule with CommonModule with HasChisel

object difftest extends CommonNS {
override def millSourcePath = os.pwd / "difftest"
override def ivyDeps = Agg(ivys.chisel)

override def scalacPluginIvyDeps = Agg(ivys.chiselPlugin)

override def scalacOptions = T(SeqString)
}

object generator extends CommonNS {

override def millSourcePath = os.pwd

val isChisel3 = true
override def moduleDeps = super.moduleDeps ++ Seq(
difftest
) ++
Option.when(isChisel3)(ccover)

object test extends SbtModuleTests with TestModule.ScalaTest

}

Then, integrated ccover into the build process by modifying the generator object.
I modify TopMain.scala located in src/test/scala/TopMain.scala.

/**************************************************************************************

• Copyright (c) 2020 Institute of Computing Technology, CAS
• Copyright (c) 2020 University of Chinese Academy of Sciences
• NutShell is licensed under Mulan PSL v2.
• You can use this software according to the terms and conditions of the Mulan PSL v2.
• You may obtain a copy of Mulan PSL v2 at:

•         http://license.coscl.org.cn/MulanPSL2
  

• THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER
• EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR
• FIT FOR A PARTICULAR PURPOSE.
• See the Mulan PSL v2 for more details.
  ***************************************************************************************/

package top

import chisel3._
import chisel3.stage.ChiselGeneratorAnnotation
import circt.stage._
import device.AXI4VGA
import difftest.DifftestModule
import nutcore.NutCoreConfig
import sim.SimTop
import system.NutShell
import xfuzz.CoverPoint

class Top extends Module {
val io = IO(new Bundle{})
val nutshell = Module(new NutShell()(NutCoreConfig()))
val vga = Module(new AXI4VGA)

nutshell.io := DontCare
vga.io := DontCare
dontTouch(nutshell.io)
dontTouch(vga.io)
}

object TopMain extends App {
def parseArgs(info: String, args: Array[String]): String = {
var target = ""
for (arg <- args) { if (arg.startsWith(info + "=") == true) { target = arg } }
require(target != "")
target.substring(info.length()+1)
}

val board = parseArgs("BOARD", args)
val core = parseArgs("CORE", args)

val s = (board match {
case "sim" => Nil
case "pynq" => PynqSettings()
case "axu3cg" => Axu3cgSettings()
case "PXIe" => PXIeSettings()
} ) ++ ( core match {
case "inorder" => InOrderSettings()
case "ooo" => OOOSettings()
case "embedded"=> EmbededSettings()
} )
s.foreach{Settings.settings += } // add and overwrite DefaultSettings
println("====== Settings = (" + board + ", " + core + ") ======")
Settings.settings.toList.sortBy(._1)(Ordering.String).foreach {
case (f, v: Long) =>
println(f + " = 0x" + v.toHexString)
case (f, v) =>
println(f + " = " + v)
}

val generator = if (board == "sim") {
ChiselGeneratorAnnotation(() => new SimTop)
}
else {
ChiselGeneratorAnnotation(() => new Top)
}
var exe_args = args.filter{
value => value.forall(char => char!='=')
}

(new ChiselStage).execute(args, Seq(generator) ++ CoverPoint.getTransforms(args)._2
:+ CIRCTTargetAnnotation(CIRCTTarget.SystemVerilog)
:+ FirtoolOption("--disable-annotation-unknown")
)

}

Finally, I executed the following command in the NutShell(https://github.com/OpenXiangShan/xs-env) directory:

make emu XFUZZ=1 FIRRTL_COVER=mux

However, I encountered the following error:

fatal error: 'firrtl-cover.h' file not found

I checked the build directory and found that the corresponding file was indeed not generated. Could you advise if there are any additional modifications I need to make to resolve this issue?

[zhengkeng]

After some investigation, I realized that generating firrtl-cover.h requires executing the execute function in ccover/instrumentation/src/xfuzz/CoverPointTransform.scala. It appears that this function was not executed during the build process.

[poemonsense]

After some investigation, I realized that generating firrtl-cover.h requires executing the execute function in ccover/instrumentation/src/xfuzz/CoverPointTransform.scala. It appears that this function was not executed during the build process.

This transform will be enabled via the CoverPoint.getTransforms shown in your code.

(new ChiselStage).execute(args, Seq(generator) ++ CoverPoint.getTransforms(args)._2
:+ CIRCTTargetAnnotation(CIRCTTarget.SystemVerilog)
:+ FirtoolOption("--disable-annotation-unknown")
)

I think the missing thing is to pass the coverage arguments to mill/Scala. Could you please have a try to add the COVER=??? to Scala build argument (mill -i [append it here])? You may refer to the code here: https://github.com/OpenXiangShan/rocket-chip/blob/dev-difftest/Makefile#L24-L27

This part seems missing in the README of this repo. Thanks for reporting this. I'll get it fixed after your confirmation

[poemonsense]

For example:

# Coverage support
ifneq ($(FIRRTL_COVER),)
MILL_ARGS += COVER=$(FIRRTL_COVER)
endif

This is the missing part in the NutShell Makefile.

[zhengkeng]

I apologize for not mentioning this earlier. I have already added this part to the Makefile. The complete mill command I am using is:

mill -i generator.test.runMain top.TopMain -td /home/xs-env/NutShell/build/rtl BOARD=sim CORE=inorder COVER=mux --split-verilog

I also printed the results obtained from CoverPoint.getTransforms(args)._2:

RunFirrtlTransformAnnotation(rfuzz.NoDedupTransform@4bbfb90a)
RunFirrtlTransformAnnotation(xfuzz.DontTouchClockAndResetTransform@7c29daf3)
RunFirrtlTransformAnnotation(xfuzz.CoverPointTransform@9660f4e)
RunFirrtlTransformAnnotation(rfuzz.SplitMuxConditions@5a8806ef)
RunFirrtlTransformAnnotation(xfuzz.MuxCoverTransform@6c49835d)

However, firrtl-cover.h still has not been successfully generated.

[poemonsense]

I've noticed that you have already changed NutShell to Chisel 3.6.0. I believe this version is well supported.

Maybe you can try adding some println for debugging in MuxCoverTransform.execute and CoverPointTransform.execute? I've no idea what is going on in your example.

I just released my experimental source code for NutShell at https://github.com/poemonsense/FuzzingNutShell. It was using a legacy version of difftest, and its ccover directory is put in difftest/instrumentation. However, I'm not sure I'll have time to fix the code and migrate the code to the latest difftest in the short time. You may use its setup as a reference design.

[poemonsense]

I've updated https://github.com/poemonsense/FuzzingNutShell

After make emu -j16 FIRRTL_COVER=mux XFUZZ=1, you can get the following output as expected:

➜  NutShell git:(a8ff113) ✗ ./build/fuzzer -c firrtl.mux -- ./ready-to-run/microbench.bin -e 0
The image is ./ready-to-run/microbench.bin
nemu does not enable flash fetch!
======= Running MicroBench [input *test*] =======
[qsort] Quick sort: * Passed.
[queen] Queen placement: * Passed.
[bf] Brainf**k interpreter: * Passed.
[fib] Fibonacci number: * Passed.
[sieve] Eratosthenes sieve: * Passed.
[15pz] A* 15-puzzle search: * Passed.
[dinic] Dinic's maxflow algorithm: * Passed.
[lzip] Lzip compression: * Passed.
[ssort] Suffix sort: * Passed.
[md5] MD5 digest: * Passed.
==================================================
MicroBench PASS
Total time: 0 ms
Core 0: HIT GOOD TRAP at pc = 0x80003a8a
Core-0 instrCnt = 363,318, cycleCnt = 1,063,874, IPC = 0.341505
SimMemory: img_size 3071, req_all 11128, req_in_range 2495
COVERAGE: mux, 1410, 1034, 1034
ExitCode: 0
Total Coverage:       73.333%

Please refer to the FuzzingNutShell repo for details

[figuresudo]

@poemonsense
I encountered a similar issue while working on adapting XiangShan for xfuzz. I modified build.sc and src/main/scala/top/Generator.scala. The Generator.scala file was updated as follows:

object Generator {
def execute(args: Array[String], mod: => chisel3.RawModule, firtoolOpts: Array[String]) = {
val annotations = chisel3.BuildInfo.version match {
case "3.6.0" => Seq(
RunFirrtlTransformAnnotation(new PrintControl),
RunFirrtlTransformAnnotation(new PrintModuleName)
)
case _ => Seq(CIRCTTargetAnnotation(CIRCTTarget.Verilog)) ++ firtoolOpts.map(FirtoolOption.apply)
}

//(new XiangShanStage).execute(args, ChiselGeneratorAnnotation(mod _) +: annotations)
(new XiangShanStage).execute(args, 
  Seq(ChiselGeneratorAnnotation(mod _))  ++ annotations ++ CoverPoint.getTransforms(args)._2 
  )

}
}

However, when I executed the command: make emu XFUZZ=1 FIRRTL_COVER=toggle CONFIG=MinimalConfig EMU_THREADS=2

I received an error indicating a potential cyclic dependency:

Exception in thread "main" firrtl.options.DependencyManagerException: No transform ordering possible due to cyclic dependency in invalidates with cycles:
- List(xfuzz.DontTouchClockAndResetTransform@6ee2d138, xiangshan.transforms.PrintControl@9d251c6, rfuzz.NoDedupTransform@329a8775)

Do you have any suggestions on how to resolve this issue?

[poemonsense]

@poemonsense

I encountered a similar issue while working on adapting XiangShan for xfuzz. I modified build.sc and src/main/scala/top/Generator.scala. The Generator.scala file was updated as follows:

object Generator {

def execute(args: Array[String], mod: => chisel3.RawModule, firtoolOpts: Array[String]) = {

val annotations = chisel3.BuildInfo.version match {

  case "3.6.0" => Seq(

    RunFirrtlTransformAnnotation(new PrintControl),

    RunFirrtlTransformAnnotation(new PrintModuleName)

  )

  case _ => Seq(CIRCTTargetAnnotation(CIRCTTarget.Verilog)) ++ firtoolOpts.map(FirtoolOption.apply)

}



//(new XiangShanStage).execute(args, ChiselGeneratorAnnotation(mod _) +: annotations)

(new XiangShanStage).execute(args, 

  Seq(ChiselGeneratorAnnotation(mod _))  ++ annotations ++ CoverPoint.getTransforms(args)._2 

  )

}

}

However, when I executed the command: make emu XFUZZ=1 FIRRTL_COVER=toggle CONFIG=MinimalConfig EMU_THREADS=2

I received an error indicating a potential cyclic dependency:

**Exception in thread "main" firrtl.options.DependencyManagerException: No transform ordering possible due to cyclic dependency in invalidates with cycles:

- List(xfuzz.DontTouchClockAndResetTransform@6ee2d138, xiangshan.transforms.PrintControl@9d251c6, rfuzz.NoDedupTransform@329a8775)**

Do you have any suggestions on how to resolve this issue?

I have not tried fuzzing XiangShan for like about 10 months. But you may find the branch I was using last year. Probably search branches with fuzz in its name in OpenXiangShan/XiangShan for reference.

For the specific question you raised, probably the easiest way to solve it is to simply comment out the annotation for running the PrintControl transform. This transform has not been used in XiangShan for a long time. I recommend just removing it.

However, if you still need to enable them, maybe you should check the dependency specified in these transforms. They are defined by several methods in the transform class. I don't have time to review it these days but in few weeks maybe I can provide an example on fuzzing the latest XiangShan.

[figuresudo]

ok, thanks a lot.