Coverage driven Random Verification

Coverage-driven random verification methods are becoming recognized as one of the best ways to verify complex IC designs. Cadence Design Systems, announced that new technologies have been integrated into the Cadence® Incisive® Enterprise verification family that enable engineering teams to address increasingly complex chip design. Incisive technologies now offer support for the newly developed Open Verification Methodology (OVM), a new aspect-oriented generation engine, and the second generation of Cadence transaction-based acceleration (TBA) with native support of multiple test-bench languages and numerous productivity enhancements.

To understand how to take advantage of this solution, cdn (cadence designer network) talked to Mr. Apurva Kalia, VP of R&D.

You can read his interview here.

Developing Test Plan

What is the difference between Test Specification and a Test Plan?

Test Specification – A detailed summary of what scenarios will be tested, how they will be tested, how often they will be tested and so on. Example of a given feature include, if the USB2.0 hub receives the token packet followed by a data packet with a payload of 64 bytes for the bulk endpoint, it has to give the acknowledge back the host controller within the specified time as per USB2.0 specification.

Test Plan - A collection of all test specifications for a given area. The Test Plan contains a high-level overview of what is tested and what is tested by others for the given feature area.

If you ask a tester on another team what is the difference between the two, you might receive different answers. In addition, I use the terms interchangeably all the time at work, so if you see me using the term test plan, think of test specification.

Parts of a Test Specification

A Test Specification should consists of the following parts:

• History / Revision - Who created the test spec? Who were the developers and Program Managers (Usability Engineers, Documentation Writers, etc) at the time when the test specification was created? When was it created? When was the last time it was updated? What were the major changes at the time of the last update?
  

• Feature Description – A brief description of what area is being tested.
  

• What is tested? – A quick overview of what scenarios are tested, so people looking through this specification know that they are at the correct place.
  

• What is not tested? - Are there any areas being covered by different people or different test specs? If so, include a pointer to these test specifications.
  

• "Nightly" Test Cases – A list of the test cases and high-level description of what is tested each night or whenever a new release becomes available.
  

• Breakout of Major Test Areas - This section is the most interesting part of the test specification where test plan writers arrange test cases according to what they are testing.

Setting Test Case Priority

A Test Specification may have a couple of hundred test cases, depending on how the test cases were defined, how large the feature area is, and so forth. It is important to be able to query for the most important test cases (nightly), the next most important test cases (weekly), the next most important test cases (full test pass) and so forth. A sample prioritization for test cases may look like:

• Highest priority (Nightly) – Must run whenever a new release is available
  

• Second highest priority (Weekly) – Other major functionality tests run once every three or four release
  

• Lower priority – Run once every major coding milestone
  

Planning for Verification

INTRODUCTION

Verification planning is most important part of verification, irrespective of the size of the design. Since, about 70% of the design cycle time is spent on verification, with proper verification planning some of the issues faced during the later stages of the design cycle can be easily avoided earlier. Verification planning is described as a set of goals that needs to be verified. A verification plan would consists of:

1. Functional requirements
2. Design requirements
3. Defining coverage goals and
4. Embedded firmware requirements
   

Apart from these requirements, the verification plan should also focus on reusable methodology.

SPECIFICATION

The specification is for capturing the requirements of the design. It is required to split the specification into Functional and Design requirements. Functional requirements can be defined as the behavior required by the system while the Design requirements is used to check the implementation of the function against the design specification. For verifying the design requirements, bottom-up verification approach can be adapted to reduce the effort spent on SOC level. The Design requirements need to be split into system-specific requirements and peripheral specific requirements . The peripheral-specific requirements are those which can be verified in module level. Examples of peripheral-specific requirements are

• Verifying all possible packet types for all HS/FS/LS for USB
  

• Verifying all possible baud rate for a UART
  

System specific requirements focus more on system level issues such as reset generation logic and clock generation logic can be completely verified only on system level.

Examples of system-specific requirements are

1. Verifying the system is properly reset for the different types of reset

2. Verifying the system for different types of power saving modes

3. Verifying the interconnectivity of clock

4. Verifying the connectivity to pads, interrupts, debug interfaces.

With this approach, the peripheral is verified completely in the module level. The SOC level verification could then focus more on top-level issues such as interconnectivity, interrupt system behavior (response of interrupt for that peripheral), bus interfaces, I/O interfaces. The system level verification could focus more on system level issues and speed up the overall verification process.

COVERAGE HOLES

It is good to have a complete coverage metric for all the peripherals, but this would mean additional overhead for simulators which would slow down the simulation speed drastically. There has always been a compromise for coverage against the simulation speed. Hence it becomes necessary to understand the complexity of the design before identifying the coverage points.

For SOC verification it would be good to have interconnectivity coverage, coverage for interrupts, system-specific behavior such as power saving features, recovery sequences, reset and clock (power saving features), and system buses.

EMBEDDED F/W REQUIREMENTS

An embedded firmware can be described as a piece of software embedded into the ROM/EPROM which would initialize the chip into a defined state on reset. This piece of software could contain Startup sequences, bootstrap loaders, memory test routines, tests for production etc. It can be seen that this software is very complex and the corresponding verification is a complex task. Unlike the verification of peripherals, firmware verification is restricted to SOC level which further increases the complexity of verification. So how do we ensure this piece of software works? For firmware verification, it is necessary to have a good coverage metric. During the verification planning phase, we need to identify all the coverage points with iterative process of review from the members involved in the development of concept and firmware. Normally the firmware verification plan is a list of coverage items that needs to be addressed in the process.

Figure shows the coverage points for firmware ( Thanks to http://www.us.design-reuse.com)

An example of Firmware code

If (HWCFG == “010”) then
MEM(status) = software_boot;
Software_start();
else if (HWCFG == “011”) then
MEM(status) = ext_boot;
Ext_start();
else
MEM(status) = int_boot;
Internal_boot();
end if;

The firmware code can then be translated to a flow chart which gives the verification engineer an overview of the software flow.

Conclusion

Verification plan gives an early estimate on effort, resource required, reuse percentage and coverage goals. Verification closure is an iterative process where the plan is measured against the implementation and coverage goals. Verification reuse can be achieved with proper planning, reusable verification environment and proper documentation.

SOC Verification 3

In the last post, we saw the traditional SOC verification approach. In this post, we are going to see the unique approaches in SOC verification. SOC verification becomes more complex because of many different kinds of IPs on the chip. A good understanding of the overall application of SOC is essential. The more extensive the knowledge of external interfaces, the more complete the SOC verification will be.

Verification Planning Guidelines

The following should be considered in the verification planning.

External Interface Emulation When you verify the complex SOCs, you should consider the full chip emulation. The external interface of each and every IPs on the SOC as well as the SOC data interfaces should also be examined. This should be performed simultaneously for all cores.

Unit level to Top level SOC designs are built from bottom to top. The truth is that the unit level must be used in any of the design hierarchy imposes a need to verify these modules in any possible scenarios.

Re-Use the verification components As the leaf modules are assembled to create the SOC, many of the leaf module interfaces are internal interfaces between various modules of SOC, and there is no longer need to drive their inputs. However other interfaces are external interfaces to SOC. If the test generators for external interfaces are independent components, then most system level stimuli can be taken as is from the various module environment.

Many components in SOC can work independently and work in parallel with other components. In order to exercise the SOC in corner cases, the tests should be able to describe parallel streams of activity for each component separately.

Integration Monitors The primary focus of SOC verification is on integration. Most bugs appear in the integration b/w blocks. An integration monitor that comes with an IP can be great help to find the integration problem. It can be hooked in to the simulation environment and just run to see any integration violation appears on the monitor. This can save the time dramatically. This kind of IP monitors can bring lot of benefits in quality of SOC.

Coverage It is important tool for identifying areas that were never exercised. Both code and toggle coverage are the first indication for areas that were never exercised. However they never tell you that you achieved the full verification. Functional coverage allows you to define what functionality of the device should be monitored.

Looking at functional coverage reports, you may conclude that certain features were already exercised and focus your efforts on the areas that were neglected. But most significant impact of functional coverage in context of SOC verification is in eliminating the need to write many of the most time consuming and hard to write tests.

Conclusion The main focus of SOC verification needs to be on the integration of the many blocks it is composed of. There is a need for welldefined ways for the IP developer to communicate the integration rules in an executable way, and to help the integrator verify that the IP was incorporated correctly. The complexity introduced by the many hardware blocks and by the software running on the processor points out the need to change some of the traditional verification schemes.

Will we ever get a handle on SoC verification cost?

In a presentation at International Symposium on Quality Electronic Design last week, Mentor Graphics verification and test division GM & VP Robert Hum presented an interesting good-news/bad-news scenario. On the good-news side, Hum said that after years of verification costs gobbling a larger and larger fraction of the total engineering budget for chips, and in fact after years of verification cost rising faster than revenue per design, the rate of increase in verification cost is finally starting to moderate.

http://www.edn.com/blog/1690000169/post/1710023971.html

SOC Verification 2

Traditional SOC Verification

Write a detailed test plan document

We usually write hundreds of directed tests to verify the specific scenarios and all sort of activities a verification engineer can think it is an important. But there are some limitations:

• The complexity of SOC is such that, many important specific scenarios in which the bug might hide are never thought of.
  

• As the complexity of SOC increases, it become difficult to write a directed test that reach the goals.

Test Generations

Each directed tests that we write, check the specific scenarios only once. But this is not advisable. Since we need to exercise these specific scenarios with different combination of inputs, then only we can find the hiding bugs. Many of us write a random test case to find the hiding bugs, but these are exercised only at the end of the verification cycle. Though these tests reach most of the unexpected corners, we will be verifying the same scenarios again and again and still tend to miss a lot of bugs. But what we actually need is to focus on the particular area of interest in the design. So ....... We need a generic test generators that can easily directed into areas of interest.

Integration

Test bench development for SOC design requires more efforts than the design itself. Many SOC verification test benches doesn't have a means for verifying the correctness of the integration of various modules. Instead the DUT is exercised as a whole unit. The main draw back to this approach is finding the source of the problems by tracing the signals all the way back to where it originated from takes much time. This leads to the need for integration monitors that could identify integration problems at the source.

Tape out..... Tape out..... Tape out.....

Every design and verification team needs an answer for the Million dollar question...... When are we ready for tape out?

To answer for this question is very tough as the verification quality is very hard to measure. Every one's answer would be different. My answer would be depends on code, branch, expression and toggle coverage, functional coverage and bug rates. To solve this dilemma, there is need for coverage metrics that will measure progress in a more precise way.

To summarize, there is always an element of spray and pray(luck) in verification, we are hoping that we will hit and identify most bugs. In SOCs, where so many independent components are integrated, the uncertainty in results is greater. There are new technologies and methodologies available today that offer a more dependable process, with less praying and less time that needs to be invested. In the next post i'll explain unique approaches in SOC verification.

SOC Verification 1

The typical System-On-Chip (SOC) may contain the following components. The processor (ARM or DSP), the processor bus, many peripherals like USB and UART, peripheral bus, the bridge which connects the buses and a Controller. The verification of SOC is a challenging one because of the following reasons.

Integration of various modules : The main focus on verification of SOC is to check the integration between the various modules. The SOC verification engineers assumes that each module was independently verified by the module level verification engineers.

IP block re-use : IP reuse was indeed seen as a way to foster development productivity and output that would eventually offset the design productivity gap. Many companies treat their IPs as an asset.

HW/SW co-verification : An SOC is really ready to ship when the complete application works, not just when hardware simulations pass in regressions. In other words, the ultimate test for a chip is to see it performs its application correctly and completely. That means execute the software together with the RTL. So we need a way to capture both HW and SW activities in the tests we write to verify the SOC.

Some of the SOC bugs might hide in the following areas.

• Interactions b/w the various blocks.
• Unexpected SW/HW handling

All the challenges above indicates that we need a rigorous verification of each of the SOC components separately. I'll explain the trends in traditional SOC verification methodology in the next post.

Functional Coverage vs Code Coverage

Basic definition of Functional coverage

Functional coverage is the determination of how much functionality of the design has been exercised by the verification environment. Let us explain this with a simple example:

If your manager told you to prove that the set of regression tests created for a particular design was exercising all the functionality defined in the specification. How would you go about proving?

• You would show the list of tests in the regression suite and correlation of those tests to the functionality defined in the specification.
  

• You would need to prove that the test executed the functionality, it is supposed to check.
  

• Finally, you would create a list showing each function and check off those that were exercised.
  

• From this list you would extract a metric showing number of functions exercised divided by total number of functions to be checked.

This is probably what you would present to your manager. This is functional coverage. The difficulty is that it is a too much of manual process; Today's design requires more structured approach.

There are two magical questions that every design team ask and answer.

1. Is my chip functioning properly?
2. Am I done verifying the chip?

Proper execution of each test in a test suite is measure of functional coverage. Each test is created to check the particular functionality of a specification. Therefore, it is natural we assume that if it were proven that each test is completed properly, then the entire set of functionality is verified. This assumes that each test has been verified to exercise the functionality for which it was created. In many cases this verification is performed manually. This type of manual checking is both time consuming and error prone. There appears to be a confusion in the industry what constitutes a functional coverage.

Code coverage

This will give information about how many lines are executed, how many times expressions, branches executed. This coverage is collected by the simulation tools. Users use this coverage to reach those corner cases which are not hit by the random test cases. Users have to write the directed test cases to reach the missing code coverage areas.

Both of them have equal importance in the verification. 100% functional coverage does not mean that the DUT is completely exercised and vice-versa. Verification engineers will consider both coverages to measure the verification progress.

I would like to explain this difference with a simple example. Let's say, the specification talks about 3 features, A, B, and C. And let's say that the RTL designed coded only feature A and B.If the test exercises only feature A and B, then you can 100% code coverage. Thus, even if you have 100% code coverage, you have a big hole (feature C) in the design.So, the verification engineer, has to write functional coverage code for A, B and C and 100% functional coverage means, there are tests for all the features, which the verification engineer has thought of.

The role of coverage in verification environment

Both functional and code coverage are complementary to each other, meaning that 100% functional coverage doesn't imply 100% code coverage; 100% code coverage - still has to achieve functional coverage goals.

Identify the coverage holes

One of the most important of functional verification is to identify the coverage holes in the coverage space The goal for any successful verification is to achieve the specified target goals with least amount of simulation cycle.

Limitation of functional coverage

• There is not a defined list of 100% functionality of the design is and therefore, there may be a missing functionality in the list.
  

• There is no real way to check that the coverage model is correct, manual check is the only way.
  

Who is better Verification Engineer? You or Your Customer?

Why is it that after months of directed and random testing, we were not able to find a bug that our customer found within days of receiving samples? Is there anything wrong with our directed and random testing? Could it be that we didn't run our simulation long enough? Could the bug have been discovered by using better functional coverage?

This article will answers to all those questions by analyzing past mistakes and proposing an effective way of writing a verification plan.

Using System Verilog for Functional Verification

New to SystemVerilog? Mentor Graphics' Tom Fitzpatrick presents some simple techniques that will get you started right now with System Verilog for functional verification.

http://www.eetimes.com/showArticle.jhtml;jsessionid=P5SZ1MPHAHIZYQSNDLPCKHSCJUNN2JVN?articleID=174900365