Contents
========
	Introduction
	Manifest		- List of files in this suite and their purposes
	Test Set		- Requirements and instructions to set up the test.
	Running Tests	- Instructions for running the tests
	Known Test		- A list of tests that are known to fail as of
	Failures		  revision 3.0.M3.  All other tests pass.
	Debugging Test	- How to determine if a test is really failing or
	Failures		  if something else is causing the failure.
	Appendix A		- Instructions for OpenHPI configuration specific
					  to running these tests.
	Appendix B		- Instructions for OpenSAF configuration specific
					  to running these tests.

Introduction
============
This file provides a manifest, test-setup guide, known-bug list and
test-debug information for HISv tests on HP ProLiant BL blades in a
c-Class chassis.  More information on BL blades and c-Class chassis at
http://h18004.www1.hp.com/products/blades/components/c-class-components.html

Manifest
========

  src directory
  -------------
  --------------------------------------------------------------------
  |         File        |                Description                 |
  --------------------------------------------------------------------
  |libcli               | Command-line interface routines            |
  --------------------------------------------------------------------
  |Makefile.am          | Automake configuration file                |
  --------------------------------------------------------------------
  |hisv_power_ctrl.c    | hpl_resource_power_set() API-test code     |
  --------------------------------------------------------------------
  |hisv_reset_ctrl.c    | hpl_resource_reset() API-test code         |
  --------------------------------------------------------------------
  |hisv_test.h          | #include's and definitions for API tests   |
  --------------------------------------------------------------------
  |hisv_test_common.c   | Common test functions for HISv-API tests   |
  --------------------------------------------------------------------
  |hpl_entity_lookup.c  | hpl_entity_path_lookup() API-test code     |
  --------------------------------------------------------------------

  lib directory
  -------------
  --------------------------------------------------------------------
  |         File        |                  Description               |
  --------------------------------------------------------------------
  |libcli               | Common CLI functions (start, stop, run     |
  |                     | command).                                  |
  --------------------------------------------------------------------
  |Globals.pm           | Common global variables for Perl utilities |
  --------------------------------------------------------------------
  |defaults             | Sets default variable values for Expect    |
  |                     | utilities.                                 |
  --------------------------------------------------------------------
  |libHisvApi           | Common subroutines related to testing HISv |
  |                     | APIs.                                      |
  --------------------------------------------------------------------
  |libTestUtils         | Subroutines for error reporting, test-     |
  |                     | cleanup, and statistics reporting.         |
  --------------------------------------------------------------------
  |libcmdline           | Parses common command-line options         |
  --------------------------------------------------------------------
  |libilo               | iLO and OA control utilities               |
  --------------------------------------------------------------------
  |liblinux             | Run Linux commands (login, check command   |
  |                     | status, etc.)                              |
  --------------------------------------------------------------------
  |libsm                | Odds and ends -- connect to system console |
  |                     | run command, etc.                          |
  --------------------------------------------------------------------
  |libssh               | Library of ssh-related commands (login,    |
  |                     | run remote command, etc.)                  |
  --------------------------------------------------------------------
  |libstate             | Determine system state (used to determine  |
  |                     | if a system is rebooting, dead, etc.)      |
  --------------------------------------------------------------------
  |stdlib               | push, shift, etc.                          |
  --------------------------------------------------------------------

  suites directory
  ----------------
  Note that tests are run from this directory.
  --------------------------------------------------------------------
  |         File            |               Description              |
  --------------------------------------------------------------------
  |Run_all                  | Test harness that will run tests in    |
  |                         | test.list, logging the results to      |
  |                         | ../results/hisv.<hostname>.log.<date>  |
  --------------------------------------------------------------------
  |test.list                | List of tests to run (used by Run_all) |
  --------------------------------------------------------------------
  |bug_295                  | Regression test for bug 295            |
  --------------------------------------------------------------------
  |bug_306                  | Regression test for bug 306            |
  --------------------------------------------------------------------
  |getState                 | Utility to determine what state a      |
  |                         | system in (booting, login prompt, etc.)|
  |                         | Logs into the system console, sends a  |
  |                         | carriage return and returns BOOTING if |
  |                         | grub or init messages are detecte,     |
  |                         | LINUX if the root prompt or a user     |
  |                         | prompt are detected; LOGIN if a login  |
  |                         | prompt is detected; SHUTDOWN if        |
  |                         | messages related to shutdown are       |
  |                         | detected.                              |
  --------------------------------------------------------------------
  |rssh                     | Utility to log into or run remote      |
  |                         | commands on a system                   |
  --------------------------------------------------------------------
  |readOaPower              | Utility to read the power state of all |
  |                         | the systems in a c-Class enclosure.    |
  |                         | Logs into the OA and queries systems   |
  |                         | using the OA command,                  |
  |                         |   show server list                     |
  --------------------------------------------------------------------
  |sahpi_parser             | Parses /etc/openhpi/SaHpi.h to get     |
  |                         | entity-path information.               |
  --------------------------------------------------------------------
  |setPower                 | Set the power state of a system via the|
  |                         | OA or iLO.  Logs into OA or iLO and    |
  |                         | sends the appropriate command to power |
  |                         | a system off or on.                    |
  --------------------------------------------------------------------
  |runCli                   | Utitity to run CLI commands.  Starts   |
  |                         | the CLI, gets to the appropriate       |
  |                         | service and sends the requeste command.|
  --------------------------------------------------------------------
  |power_api                | Functional tests of the                |
  |                         | hpl_resource_power_set() API.          |
  --------------------------------------------------------------------
  |power_api_error_handling | Test of hpl_resource_power_set() error |
  |                         | handling.                              |
  --------------------------------------------------------------------
  |power_cli                | Functional tests of power-control using|
  |                         | the command-line interface.            |
  --------------------------------------------------------------------
  |power_cli_error_handling | Tests of error handling of admreq      |
  |                         | commands in the command-line interface |
  --------------------------------------------------------------------
  |reset_api                | Functional tests of the                |
  |                         | hpl_resource_reset() API.              |
  --------------------------------------------------------------------
  |reset_api_error_handling | Tests of the hpl_resource_reset() API's|
  |                         | error handling.                        |
  --------------------------------------------------------------------
  |reset_cli                | Functional tests of reset using the    |
  |                         | command-line interface.                |
  --------------------------------------------------------------------
  |reset_cli_error_handling | Tests of error handling of reset       |
  |                         | commands in the command-line interface |
  --------------------------------------------------------------------
  |test.dat                 | Sample test-data file                  |
  --------------------------------------------------------------------

Test Setup
==========

  Requirements
  ------------
    1.  Make sure your OA is running Firmware revision 2.21 or newer.
        Make sure that the iLO revision on your BL46* systems is 1.70
        or newer.
    2.  Install, configure and start OpenHPI, revision 2.11.3 or newer.
        See Appendix A for test-specific details of the set up.
    3.  This suite is intended to run on HP c-Class blades and enclosures.
        - The tests are set up to run on a two-controller, one payload
          cluster.
    4.  The test suite was written and tested on RHEL 5.x with HP bl46x
        blades in a c-7000 enclosure.
    5.  Install expect.  Expect is used in all scripts with names like
        readOaPower, setPower and runCli that need to interact with a
        command line.
  OpenSAF Set Up
  --------------
    1.  Install OpenSAF and perform the configuration-specific set
        up outlined in Appendix B.
    2.	Install opensaf-devel package on the controller nodes.
    3.  Start OpenSAF on both controllers and on the payloads.
  Compile Tests
  -------------
    This step is done automatically if the tests are run using Run_all.
    Otherwise, use the GNU autotools procedure,
      1.  bootstrap.sh
      2.  configure
      3.  make
  Create and Edit the test-configuration file
  -------------------------------------------
  The following set up is performed in the suites directory.
      1.  The default test-configuration file is test.dat.  If you use
          another name, be sure to add -f <data_file> to the command
          lines for the tests (this can be added to test.list if
          running the tests via Run_all).
      2.  The required test information is,
          OA - the hostname or IP address of the OA monitoring the
               enclosure.  This is used to verify that the systems'
               power states are correct after the API- or CLI-
               commands are applied.
          Chassis - The chassis ID for the enclosure.  This should
               match the entity_root entry in the OpenHPI config
               file (see Appendix A) and the value in the OpenSAF
               chassis_id file (see Appendix B).
          ControllerBay - The bay in the c7000 enclosure where the
               standby controller node is located.
          PayloadBay - The bay in the c7000 enclosure where the
               payload node is located.
      3.  Other parameters that can be set in the test-configuration
          file:
          The following parameters can be also be set in lib/defaults:
          OA_user - An OA user with administrative permissions.
          OA_password - The password for that user.
           NOTE: The correct user and password are required, or the
           tests will fail, since the tests connect to the OA to
           verify the power states of the systems in the enclosure.

          Optional parameters that can be added to the test-parameter
          file:
          Cluster - Enter a comma separated list of systems included
            in the test cluster.  This parameter restricts checks of
            the power states of systems in the enclosure to only those
            nodes in the cluster.  This useful if other systems in the
            test enclosure, but not part of the test cluster are
            undergoing power-state changes.

Running Tests
=============
Tests are run from the suites directory.  The tests can be run using
the Run_all script, which will read a test list specified on the
command line using -f <test_list_file> (default test.list).  Run_all
will run the tests and automatically log the results in the results
directory (<opensaf_src_dir>/tests/hisv/results) in a file named
hisv.<hostname>.log.<mmdd_HHMMSS>.  To vary the tests automatically
run by Run_all, you can edit test.list.  Note that lines in test.list
can be commented using a '#' character.

The tests can also be run from the common TET test-harness.  See the
README in the directory immediately above the hisv-test directory
(<opensaf_src_dir>/tests/README).

IMPORTANT: Whether running the tests using Run_all or tet, the test.dat
file must be set up as described under the Test Setup section.

Known Test Failures
===================

  power_api test failures:
  ------------------------
    Bug 306: HISv power-set for power-cycle powers down a blade
             and does not bring it back up.  This is a known bug
             in OpenHPI as of the 2.12.0 release.  It is fixed in
             OpenHPI revision 2.13.3 and newer.

    Bug 317 (related): Some power-off tests will fail with
             "wrong reset applied".  After a system is powered
             off, it should miss a heartbeat, causing OpenSAF to
             apply a reset to bring it back to life.  However,
             the reset applied by OpenSAF is the non-existent
             HISV_RES_GRACEFUL_REBOOT, which generates an HPI
             error and a warning that OpenSAF will try a cold
             reset of the system (which will fail, since the
             system is powered OFF).

  power_cli_error_handling failures:
  ----------------------------------
    Bug 323: The CLI will report success in setting the admin state
             for an entity path containing an invalid blade ID.

    Bug 417: The CLI will report a cold reset was applied to a system
             with a non-existent chassis ID and blade ID.

    Sometimes, a test will fail because the system cannot be put
    into the correct power state (usually OFF), because the power
    state can't be set via the system's iLO or from the OA.  Some
    light testing shows that iLO firmware version 1.77 (from
    http://support.hp.com) fixes this issue.

    Also, the test can be rerun after turning the failing system
    off using the "/sbin/poweroff" command in linux.  (This command
    always works.)


  reset_cli failures:
  -------------------
    Bug 317 (related): OpenSAF defines HISV_RES_GRACEFUL_REBOOT, which
             is not defined by OpenHPI, so these resets will fail, and
             OpenSAF will substitute a cold reset instead.

Debugging Test Failures
=======================

General
-------
1.  The tests require OpenHPI revison 2.13.3 or newer.
2.  If OpenSAF hangs at "Starting SCAP", there could be
    several causes.
    - DRDB: DRDB is complicated and problematic, disable DRDB,
      edit /var/lib/opensaf/pssv_spcn.list to start OpenSAF using
      the XML files (replace PSS everywhere with XML), and try
      restarting.
    - On some platforms (RHEL 5.x, for example), explicitly
      specify no optimization when configuring OpenSAF to build,
      i.e., use the command
           CFLAGS='-O0' ./configure --with-hpi-interface=B02
      to configure compilation of OpenSAF.

3.  Tests fail because get_resourceid fails (message in the
    HISv log file, /var/lib/opensaf/stdouts/ncs_hisv.log):
    - It is safest to explicitly give the OpenHPI specification
      when building the tests.  Use the command line
           CFLAGS='-O0' ./configure --with-hprev=B02
      when configuring openSAF. 

4.  Tests will fail if the incorrect OA user/password are not
    configured in the test-parameter file (default: test.dat),
    The signiture of such a failure are log file entries like,

5.  Tests will fail if opensaf is not running or if HISv is not
    running as a service.  The signiture of this type of failure
    is tests that fail because the HISv log,
    /var/lib/opensaf/stdouts/ncs_hisv.log, does not exist.  e.g.,
    cp: cannot stat `/var/lib/opensaf/stdouts/ncs_hisv.log': No such file or directory 

6.  Error messages from the test similar to,
    "**> ERROR: Blade N is ON and can't turn it off fot test", or
    "**> ERROR: Failed to turn system N to run test"
    indicate that the test was unable to set the system power to the
    correct state (opposite of what the test is trying to set it to).
    The test uses the OA or iLO console to power the system off or on.
    Sometimes setting the power this way does not succeed.  To remedy
    this, log in directly to the OA or iLO and set the power -- it may
    take multiple tries.


-----------------------------------------------------------------------------
WARNING: This is a private system.  Do not attempt to login unless you are an
authorized user.  Any authorized or unauthorized access and use may be moni-
tored and can result in criminal or civil prosecution under applicable law.
-----------------------------------------------------------------------------
Firmware Version: 2.21
Built: 06/13/2008 @ 07:29
OA Bay Number:  1
OA Role:        Active
Admin@saf2-oa.saf.telco's password:
Permission denied, please try again.
Admin@saf2-oa.saf.telco's password:
Permission denied, please try again.
Admin@saf2-oa.saf.telco's password:
Permission denied (publickey,password).

ERROR: Failed to connect to oa

API tests:
----------
For the discussion below, the API applications used in this test suite
are in src, and include
    hisv_power_ctrl
    hisv_reset_ctrl
    hpl_entity_lookup

1. OpenSAF is not running if any of the API applications, fail with a
   message similar to,
    "<test_name>: src/mds_dt_tipc.c:240: mdtm_tipc_init: Assertion `0' failed.
     Aborted"

2. The HISv service is not running if any of the API applications fails
   with an error similar to, "No HAM managing chassis 2".  For example
   the output of hisv_power_ctrl with valid parameters is,

   check_user_input:
        SpecifiedChassis : 1
        SpecifiedBlade : 1
        chassis : 2
        blade : 12
        Initializing the event subsystem
        Initializing the client-side library
   Looking up path for chassis 2 / blade 12
        hpl_entity_path_lookup(3, chassis, blade, hisv_entity_path, sizeof(hisv_entity_path))
   No HAM managing chassis 2
   ERROR: Failed to get an entity path for blade 12 at chassis 2
   Unregistering HPI Private Library for this application


CLI tests:
----------
1. If OpenSAF is not running, starting the CLI will print the
   following error message:
     /usr/bin/ncs_cli_maa
     ncs_cli_maa: src/mds_dt_tipc.c:240: mdtm_tipc_init: Assertion `0' failed.


2. If HISv is not running, cli operations for a valid system path
   will print out an error similar to, "No HAM managing chassis 2".
   This message will likely be embedded in the output of the cli,

     /usr/bin/ncs_cli_maa

     cli idle time is 300 seconds
     ncs cli user is an authenticated super user.
     NCS>enable
     NCS#configure terminal
     NCS-config#avsv
     NCS-config-avsv#admreq /2/12 operation unlockNo HAM managing chassis 2
     =================================
     MIB_ARG  PP @ address b7e7f29c
     =================================
     Op: SET Req, Tbl Id: 39, Exch Id: 0
     User Key: 4288675841
     MIB Key: 4288675841
     Source: 4
     Index vals as int: , 29, 123, 123, 54, 53, 53, 53, 55, 44, 49, 50, 125, 44, 123, 50, 51, 44, 50, 125, 44, 123, 54, 53, 53, 51, 53, 44, 48, 125, 125
     Index vals as ascii: , , {, {, 6, 5, 5, 5, 7, ,, 1, 2, }, ,, {, 2, 3, ,, 2, }, ,, {, 6, 5, 5, 3, 5, ,, 0, }, }
     Param Val: Id 11, Integer, Val 3
     
     Success: Admin State Set Successfully

3.  CLI tests will fail if the locations of the test systems are
    not configured in the BOM file (/etc/opensaf/NCSSystemBOM.xml )
    as described in Appendix B.


======================================================================
                              Appendix A
                        OpenHPI Configuration
======================================================================

OpenHPI revision 2.13.3 or newer must be installed.  If your
distribution does not include OpenHPI of the proper vintage,
download the source from http://openhpi.org/Downloads and build
it according to the instructions in the README file.  To configure
OpenHPI for OpenSAF, 

  1.  Comment out the simulator plug-in (put '#' characters at the
      beginning of each line, i.e.,

  ## You can load multiple copies of the simulator plugin but each
  ## copy must have a unique name.
  #handler libsimulator {
  #        entity_root = "{SYSTEM_CHASSIS,1}"
  #        name = "simulator"
  #}

  2.  Edit the OA-SOAP section of the configuration file,
      /etc/openhpi/openhpi.conf.  The section should look like,

  handler liboa_soap {
          entity_root = "{SYSTEM_CHASSIS,2}"
          OA_User_Name = "Admin"   # OA user name with admin privileges (required)
          OA_Password = "Admin"  # OA password for above user (required)
          ACTIVE_OA = "10.10.1.50"  # Active OA hostname or IP address (required)
          STANDBY_OA = "" # Standby OA hostname or IP address (optional)
  }

Note that the entity_root value should match the value in
/etc/opensaf/chassis_id and in the system BOM (see Appendix B).

======================================================================
                              Appendix B
                        OpenSAF Configuration
======================================================================

Build OpenSAF 3.0 or newer following the instructions in the
README included.  Copy the resultant RPM files to the test systems,
installing the opensaf-libs and opensaf-common packages on all
systems, the opensaf-controller and opensaf-devel packages on the
controller nodes, and the opensaf-payload package on the payload
node.  Perform the configuration indicated in the installation
guide, then configure HISv services for HP BL46* blades in a c-7000
enclosure, edit /etc/opensaf/script.conf and change the line defining
the target architecture to "HP_CCLASS".  For OpenSAF 3.0.0.m3,

  export OPENSAF_TARGET_SYSTEM_ARCH="HP_CCLASS"

This configuration file should be pushed to all systems in the cluster.
To configure the HISv service to run, modify the system BOM file on
the controllers, /etc/opensaf/NCSSystemBOM.xml.  It is simplest to
copy NCSSystemBOM.xml.hp.c-class to NCSSystemBOM.xml.

To be able to run the CLI tests, the system BOM file must be further
customized to match the test-hardware configuration.  In order to
correctly target CLI commands to the correct system, the system
locations must be correctly recorded in the system BOM.

In the section of the system BOM file that defines hardware-deployment
configuration, <Hardware-Deployment-Config>, where the definitions 
for the cluster systems are listed, i.e., <Name>Payload_3</Name>,
<Name>Payload_4</Name>, ...  <Name>Controller_1</Name> and
<Name>Controller_2</Name>, put the correct bay number where the
<EntityLocation> tags are.  For concreteness, suppose the cluster has
systems in bays 9, 10, 11 and 12.  The systems in bays 9 and 10 are 
the controllers, while systems 11 and 12 are the payloads.  Further-
more, suppose that the chassis ID is 2 (the default OpenSAF chassis
ID).  Examine the Hardware-Deployment-Config for this cluster below
and note the following: 
   1.  The chassis_descriptor matches the value in
       /etc/opensaf/chassis_id, 
   2.  the entity location for Payload_3 is 11,
   3.  the entity location for Payload_4 is 12,
   4.  the entity location for Controller_1 is 9,
   4.  the entity location for Controller_2 is 10

<Hardware-Deployment-Config>
   <EntityDeploymentInstance>
      <Name>chassis_descriptor</Name>
      <EntityTypeInstanceName>CHASSIS</EntityTypeInstanceName>
      <NodeName>safNode=Chassis_Pc_Cluster</NodeName>
      <EntityLocation>2</EntityLocation>
      <HPIEntityType>SAHPI_ENT_SYSTEM_CHASSIS</HPIEntityType>
      <isActivationSourceNCS>1</isActivationSourceNCS>
      <contains>
         <EntityDeploymentInstance>
             <Name>Payload_3</Name>
             <EntityTypeInstanceName>PAYLOAD_PC</EntityTypeInstanceName>
             <NodeName>safNode=PL_2_3</NodeName>
             <EntityLocation>11</EntityLocation>
             <HPIEntityType>SAHPI_ENT_SYSTEM_BLADE</HPIEntityType>
             <isActivationSourceNCS>1</isActivationSourceNCS>
         </EntityDeploymentInstance>

         <EntityDeploymentInstance>
             <Name>Payload_4</Name>
             <EntityTypeInstanceName>PAYLOAD_PC</EntityTypeInstanceName>
             <NodeName>safNode=PL_2_4</NodeName>
             <EntityLocation>12</EntityLocation>
             <HPIEntityType>SAHPI_ENT_SYSTEM_BLADE</HPIEntityType>
             <isActivationSourceNCS>1</isActivationSourceNCS>
         </EntityDeploymentInstance>

         <EntityDeploymentInstance>
             <Name>Controller_1</Name>
             <EntityTypeInstanceName>CONTROLLER_1</EntityTypeInstanceName>
             <NodeName>safNode=SC_2_1</NodeName>
             <EntityLocation>9</EntityLocation>
             <HPIEntityType>SAHPI_ENT_SYSTEM_BLADE</HPIEntityType>
             <isActivationSourceNCS>1</isActivationSourceNCS>

             <diskBoot/>

         </EntityDeploymentInstance>

         <EntityDeploymentInstance>
             <Name>Controller_2</Name>
             <EntityTypeInstanceName>CONTROLLER_2</EntityTypeInstanceName>
             <NodeName>safNode=SC_2_2</NodeName>
             <EntityLocation>10</EntityLocation>
             <HPIEntityType>SAHPI_ENT_SYSTEM_BLADE</HPIEntityType>
             <isActivationSourceNCS>1</isActivationSourceNCS>

             <diskBoot/>

         </EntityDeploymentInstance>

      </contains>
   </EntityDeploymentInstance>
</Hardware-Deployment-Config>

=========
vim: tabstop=4
-*- tab-width:4 -*-

