Thursday, June 12, 2014

Wednesday Night Hack #3 - BARF is done

I am done working on my Python-based build and run flow.

While doing this hack, I took a small detour and evaluated SCons for use with EDA tools.  I was successful in understanding the declarative nature of the tool and even wrote some custom builders, however, it quickly became clear that I was trying to fit a square peg into a round hole and abandoned the effort.  I was hoping that I could adopt a tried-and-tested solution and would love to hear from others who have been successful, if any.

The feature list for BARF is limited.  It has the ability to collect groups of files into components and provides a wrapper to execute commands.

Each component is specified using a YAML file.  The component contains a list of files, options, and requires.  Requires are needed to indicate parent-child relationship between components.  For example, a block may require a ram component or the chip-level wrapper requires all block components.
name: led
files: [led.v]
options: []
requires: [ram]
I will point out that the use of YAML is a reversal from the previous blog post where I experienced with making the mechanism to declare a component be a Python script itself.

I kept the API for execute commands extremely straight forward.  Each custom job inherits from a base class.  Since each job is modeled as a Python object, artefacts from each job can be collected by the specialized class and the collected Python can easily passed from one job to another.

I am pasting the relevant source code below.  Maybe next week, I'll set up a GitHub account.
class Barf(object):
    """ Build and Run Flow """

    def post_order(self, node):
        """ Recursive post-order tree traversal """
        if not node:
            return
        for child_name in node['requires']:
            child_node = self.comp[child_name]
            self.post_order(child_node)
        if node['visited'] == 0:
            self.flist_obj.append(node)
            node['visited'] = 1

    def load_comps(self, top_node):
        """ Load components from yaml files """
        self.comp = {}

        for root,dirs,files in os.walk(os.environ.get('WS')):
            for file in files:
                if file == "comp.yml":
                    full_path = os.path.join(root, file)
                    stream = yaml.load(open(full_path))

                    # use list comprehension (!)
                    stream['files'] = [ root+'/'+x for x in stream['files'] ]

                    name = stream['name']
                    self.comp[name] = {}
                    self.comp[name]['files'] = stream['files']
                    self.comp[name]['options'] = stream['options']
                    self.comp[name]['requires'] = stream['requires']
                    self.comp[name]['visited'] = 0

        self.flist_obj = []
        self.post_order(self.comp[top_node])
  
class Job(object):
    """ Base class for job object """

    def exec_cmd(self,cmd,wdir=os.environ.get('WSTMP')):
        """ Execute shell command """
        p = subprocess.Popen('cd {0} && {1}'.format(wdir,cmd),stdout=subprocess.PIPE,shell=True)
        (stdout, stderr) = p.communicate()
        if p.returncode != 0: raise Exception("Command {0} failed ".format(cmd))
        return (stdout, stderr)

    def cyg_to_win_path(self,cyg_path):
        """ Convert cygwin path to windows path """
        p = subprocess.Popen('cygpath -w '+cyg_path,stdout=subprocess.PIPE,shell=True)
        return '"'+p.communicate()[0].rstrip()+'"' #--HACK: cygwin

class CleanTmp(Job):
    def execute(self,lib_name='work'):
        self.exec_cmd('rm -rf {0}/*'.format(os.environ.get('WSTMP')))

class RunVlib(Job):
    def execute(self,lib_name='work'):
        self.exec_cmd('vlib {0}'.format(lib_name))

class RunVlog(Job):
    def execute(self,flist_obj):
        files = []
        for obj in flist_obj:
            files +=  obj['files']
        files = [ self.cyg_to_win_path(x) for x in files ]

        options = []
        for obj in flist_obj:
            options +=  obj['options']

        self.exec_cmd('vlog -sv2k5 {0} {1}'.format(' '.join(files),
                                                   ' '.join(options)))

Wednesday, June 4, 2014

Wednesday Night Hack #2 - BARF

Tonight, I continued to work on the front end tool described in last week's blog post: http://blog.edmondcote.com/2014/05/wednesday-hack-1.html.  I found a clever name for the script: barf (build and run flow).  It's too late in the evening to post details, so I will be brief.  Besides the name change, I implemented an object oriented build pipeline.  The user can describe the stages of their build and run flow using Python objects.  The execution of the objects is controlled by a centralized class.  While it may be outside of the scope of this (yet to be determined effort), this would allow the jobs to be executed in parallel.  I also added a mechanism for a job stage to pass information from one another.  An example use case is for the synthesis step to pass an object to the place and route step.  Finally, I created two build stages.  The first to create an Active-HDL work library (vlib) and the second to compile Verilog file (vlib).

Wednesday, May 28, 2014

Wednesday Night Hack #1

This week, I began the development of basic a front end tool to wrap the functionality of the CAD tools bundled with my LatticeECP3 FPGA development board.  The packaged tools are Aldec-HDL, Synplify Pro, and Lattice Diamond (the FPGA implementation tool).  Active-HDL, by all respects, have a pretty solid GUI, but my preference is to maintain control from the command line.

Rather than implement a custom Makefile library or a traditional Perl script, I chose to develop an API using Python that a user could use to build up their own flow apps.  The idea is to replace configuration files or application-specific dynamic scripting languages with an actual scripting language.  This approach scales better over time.

The first step to this process is project management. We want the ability to manage groups of files.  I call these groups of files components.  Examples of components include RTL unit (Verilog module and its sub modules), UVC (Universal Verification Component), design IP library (RAM library, FPGA megafunctions), etc.

We also want the ability to establish "depends on" relationships between components.  Here is an example component definition written in Python

c.set_name('top')
c.add_file('top.v')
c.add_require('led')
c.add_require('ram')

For a simple project consisting of three components. Top, RAM, and LED where top depends on RAM and LED and LED also depends on RAM.  The flist would need to resemble following.  The script below is able to produce this.  In fact, below is the actual output of the script.

# Component: ram
/cygdrive/d/Projects/system/rtl/ram/ram.v

# Component: led
# Requires: ram
/cygdrive/d/Projects/system/rtl/led/led.v

# Component: top
# Requires: led,ram
/cygdrive/d/Projects/system/rtl/top/top.v

I achieve this functionality by implementing two classes in Python: Component and Go.  Here are some code snippets.  First, I build a tree data structure in process_requires function.  I traverse the tree in function get_flist using a simple recursive algorithm.

Here is a snippet of the source code.  There's no error checking, so YMMV.

class Component:
    """ Exposed to user """
    def add_file(self, file):
        self.files.append(self.root_dir+'/'+file)

    """ Exposed to user """
    def add_option(self, option):
        self.options.append(option)

    """ Exposed to user """
    def add_require(self, require):
        self.requires.append(require)

    def get_flist(self):
        flist = '#' * 80 + '\n' # 80 character comment line
        flist +=  '# Component: ' + self.get_name() + '\n'
        if self.requires:
            flist +=  '# Requires: '+','.join(self.requires)+'\n'
        if self.files:
            flist +=  '\n'.join(self.files) + '\n'
        if self.options:
            flist +=  '\n'.join(self.options) + '\n'
        return flist

class Go:
    """ Load all component under root_dir """
    def load_comps(self):
        for root,dirs,files in os.walk(self.root_dir):
            for f in files:
                if f == 'comp.py':
                    c = Component()
                    c.set_root_dir(root)
                    execfile(root+'/'+f)
                    self.component[c.get_name()] = c

    def process_requires(self,node):
        for require in node.requires:
            child_node = self.component[require]
            self.process_requires(child_node)
            node.child_nodes.append(child_node)

    def init(self):
        self.load_comps()
        self.process_requires(self.get_top_node())

    def get_flist(self):
        self.visited = {}
        self.do_get_flist(self.get_top_node())

    """ Traverse tree using recursive post-order search """
    def do_get_flist(self,node):
        for child in node.child_nodes:
            if not child.get_name() in self.visited:
                self.do_get_flist(child)
        if not node.get_name() in self.visited:
           print(node.get_flist())
           self.visited[node.get_name()] = 1 # mark visited

Wednesday, May 21, 2014

Back to Basics

I have been working in hardware design verification since I graduated from school.  It has been nearly 7 years now, time flies, I can hardly believe it.

Hardware design verification is an extremely exciting field.  The field requires knowledge spanning multiple disciplines: software development, hardware implementation, and (in my case) computer/graphics systems.

My job at Qualcomm is also extremely challenging, we are at the forefront of all the above.

At school, I wasn't interested so much in academics or research as I was building things.  I spent a substantial amount of time working on overly ambitious projects.

The best example of that, perhaps, was my digital VLSI circuits class where I spent more time playing with tools to generate GDS-II from RTL than focusing on the class assignment which was to design and layout a custom digital cell.

In my undergraduate computer graphics class, I went way beyond the defacto rendered-with-OpenGL textured cube or simple ray tracer. I wrote my own scene graph, associated rendering engine, and custom Python-based Blender3D export plugin.  Not only that, but I took it a step further and wrote some custom shaders in ARB assembly language.  This was before there were high-level shader languages.

Fast forward to grad school.  I had taken a keen interest in computer architecture.  The thought and opportunity to understand these systems from the ground up was (and still is) very exciting for me.

As you can imagine, I wasn't interested so much in research as I was in hacking.  I spent a substantial amount of time building my own multiprocessor system-on-a-chip from the ground up and prototyped it on an FPGA.

The final product was documented in my thesis and in two conference papers.  Granted, my contributions weren't ground breaking, but in retrospec, it was a very rewarding experience.

Over the past few years, I haven't tinkered so much. I worked. I lived. I got married. I am still learning about work-life balance. But, the bug to tinker is still in me.

This long winded rant to say that I have recently been researching FPGA prototype boards to purchase and settled on this one the other day.  Besides cost, I was motivated by the availability of Ethernet ports and PCIe connector.  I/O is an area of system design that I know next to nothing about.

http://www.latticesemi.com/en/Products/DevelopmentBoardsAndKits/LatticeECP3VersaDevelopmentKit.aspx


I do not plan to make any commitments regarding my tinkering, but it is a part of me that I hope to resurrect.

Saturday, February 4, 2012

Vim Setup for Python - Part 2

A quick follow up of my previous post:

http://blog.edmondcote.com/2013/10/vim-setup-for-python.html

Tagbar install and shortcut

1. Install ctags
2. Download Tagbar from http://www.vim.org/scripts/download_script.php?src_id=21362
3. Install Tagbar using vim -c 'so %' -c 'q' tagbar.vmb
4. Add nmap <F8> :TagbarToggle<CR> to your $HOME/.vimrc

Shortcut to fix indentation

http://vim.wikia.com/wiki/Fix_indentation

1. Add map <F7> mzgg=G`z<CR> to your $HOME/.vimrc

Tab-completion of Python code

1. Download latest version of Pydiction http://www.vim.org/scripts/script.php?script_id=850
2. Install Pydiction