Wednesday, November 22, 2017

Connecting Zeppelin, Spark, and MongoDB

It took me a few hours to connect Zeppelin, Spark, and MongoDB.  I didn't find a solution to this problem online; thus the short entry.

First, I added a dependency to the MongoDB Connector for Spark in my Zeppelin notebook.


%dep
z.reset()
z.load("org.mongodb.spark:mongo-spark-connector_2.10:2.2.0")

%spark
import com.mongodb.spark._
import com.mongodb.spark.rdd.MongoRDD
val rdd = MongoSpark.load(sc)

This gave :

java.lang.IllegalArgumentException: Missing database name. Set via the 'spark.mongodb.input.uri' or 'spark.mongodb.input.database' property

Then, after realizing,  that you cannot dynamically reconfigure the SparkContext.  I used the GUI to set the property.


It is working well now!

rdd: com.mongodb.spark.rdd.MongoRDD[org.bson.Document] = MongoRDD[0] at RDD at MongoRDD.scala:47

Tuesday, November 7, 2017

Early November update: DARPA, GCE, open source release

It has been a busy few weeks, and it is not going to get better in November.  No big deal because I am enjoying startup life. My biggest challenge is to not work too much.  I am in the process of applying for a DARPA research grant for the company

I have been looking to scale my design environment to the cloud.  I followed a friend's recommendation and eval'ed Google Compute Engine. Wow! I was up and running in seconds.  I intended to buy an additional workstation to serve as a build server, but the $20,000 in free startup credits have me thinking otherwise.

I open sourced my design environment to support the effort.  Here is a link to the public repository https://bitbucket.org/ecote/nade.

Here is a picture of an FPGA that I produced.


I also recently subscribed to Grammarly.  I am incredibly impressed.

Saturday, October 21, 2017

Script to generate Verilog ports from Chisel source

Here's a script I wrote that will (naively) parse a Verilog file and output it's corresponding ports in Chisel format.  This is useful to import a Verilog module as a blackbox in a Chisel design.


#!/bin/sh
exec scala "$0" "$@"
!#

import scala.util.{Success, Try}
import scala.util.matching.Regex

val port = raw"\s*(input|output|inout)\s+(wire\s*|reg\s*)?\s*(\[(\d+):(\d+)\])?\s*(\w+)?\s*;?".r

val p = scala.io.Source.fromFile(args(0)).getLines.foreach { f: String =>
  val b = f match {
    case port(d, tzpe, _, msb, lsb, name) =>
      val dir = d match {
        case "input" => "Input"
        case "output" => "Output"
        case "inout" => "Analog"
        case _ =>
      }
      val w = Try({msb.toInt - lsb.toInt + 1}) match {
        case Success(s) => s"%d".W.format(s)
        case _ => "1.W"
      }
      s"val $name = $dir(UInt($w))"
    case _ => ""
  }
  if (b.nonEmpty) println(b)
}

The chosen regular expression was testing using regex101.com, a populate online regular expression tester.

Monday, October 16, 2017

My Clocking and Reset Strategy with RawModule

This post is originally a reply to: https://groups.google.com/forum/#!topic/chisel-users/LTujWW6DtI4

The following snippet of code illustrates my top level clocking and reset strategy.  I opt to use LazyRawModuleImp and explicitly specify the clocks and reset.

class LazyTop(implicit p: Parameters) extends LazyModule {
  lazy val module = new LazyRawModuleImp(this) {
    val io = IO(new Bundle {      
      val ref_clk_p = Input(Clock())
      val rst_sw_n = Input(Bool()) // async reset, debounced on board?
      val led_n = Output(Bits(8.W))
    })

    val GSR_INST = Module(new GSR()) // this is blackbox module
    GSR_INST.io.GSR := io.rst_sw_n

    val global_reset = Wire(Bool())
    val global_reset_n = Wire(Bool())
    global_reset_n := GSR_INST.io.GSR // appears GSR is active low
    global_reset := !GSR_INST.io.GSR

    // use active high clock
    withClockAndReset(ref_clk_p, global_reset) {
      io.led_n := RegNext("h5A".asUInt(8.W), init = 0.U)
    }
  }
}

Module GSR is a Chisel BlackBox for my FPGA's GSR (global set reset) library cell.


class GSR extends BlackBox {
  val io = IO(new Bundle {
    val GSR = Input(Bool())
  })
}

Wednesday, October 4, 2017

To link Chisel annotations and circuit info

Here's another snippet of Chisel code.  It was well worth the (ashamed to admit) 4+ hrs to write.  Admittedly, I am still getting a hand of functional programming.  I am particularly proud of the recursive function call to populate an immutable val.  The function's use case is to return Chisel circuit level information from an Annotation object.  It is necessary because Annotation does not contain (do best of my knowledge) a reference to a Data object.


  private def getDirection(circuit: Circuit, a: Annotation): ActualDirection = {
    // get net info from annotation object
    val thisModName = a.target.asInstanceOf[ComponentName].module.name
    val thisPortName = a.target.name.replace("io.", "") // FIXME: this is a HACK ..

    // find matching module in Chisel circuit
    val module = circuit.components.flatMap {
      case m: DefModule => Some(m)
      case b: DefBlackBox => None
    }.find {
      _.name == thisModName
    }.head

    // recursively return Element objects
    def elemsFromData(d: Data): Seq[Element] = d match {
      case e: Element => Seq(e)
      case r: Record =>
        r.elements.flatMap((e: (String, Data)) => {
          elemsFromData(e._2)
        }).toSeq
    }

    // all elements for all module port
    val elems: Seq[Element] = module.ports.flatMap((p: Port) => {
      elemsFromData(p.id)
    })

    // use DataMirror to find direction, have yet to test whether fully accurate
    // FIXME: obviously not optimal, should filter above
    val dir = DataMirror.directionOf(elems.find(_.name == thisPortName).head)
    dir
  }