InfoServer Document fins/I0616-1


FIN #: I0616-1

SYNOPSIS: Ecache Memory Parity Error

DATE: Sep/11/00

KEYWORDS: Ecache Memory Parity Error


---------------------------------------------------------------------
- Sun Proprietary/Confidential: Internal Use Only -
---------------------------------------------------------------------  
                            FIELD INFORMATION NOTICE
                  (For Authorized Distribution by SunService)



SYNOPSIS: Solaris kernel patches provide improved handling and reduction
          of CPU, Ecache, and main memory errors in UltraSPARC systems.
              
TOP FIN/FCO REPORT: Yes
 
PRODUCT_REFERENCE:  Solaris 2.5.1, 2.6, 7, and 8
 
PRODUCT CATEGORY:   Software / Solaris

PRODUCTS AFFECTED:
  
Mkt_ID   Platform   Model   Description              Serial Number
------   --------   -----   -----------              -------------
Systems Affected
----------------
  -      E10000-HPC  ALL    Ultra Enterprise 10000 HPC     -
  -      E10000      ALL    Ultra Enterprise 10000         -
  -      E6500-HPC   ALL    Ultra Enterprise 6500 HPC      -
  -      E6500       ALL    Ultra Enterprise 6500          -
  -      E5500-HPC   ALL    Ultra Enterprise 5500 HPC      -
  -      E5500       ALL    Ultra Enterprise 5500          -
  -      E4500-HPC   ALL    Ultra Enterprise 4500 HPC      -
  -      E4500       ALL    Ultra Enterprise 4500          -
  -      E3500-HPC   ALL    Ultra Enterprise 3500 HPC      -
  -      E3500       ALL    Ultra Enterprise 3500          -
  -      E450-HPC    ALL    Ultra Enterprise 450 HPC       -
  -      A25         ALL    Enterprise 450                 -
  -      A33         ALL    Enterprise 420R                -
  -      A26         ALL    Enterprise 250                 -
  -      A34         ALL    Enterprise 220R                -
  -      N14         ALL    Netra T-1405                   -
  -      N15         ALL    Netra T-1400                   -
  -      N06         ALL    Netra T1 AC                    -
  -      N04         ALL    Netra T-1125                   -
  -      N03         ALL    Netra T-1120                   -
  -      A27         ALL    Ultra 80                       -
  -      A23         ALL    Ultra 60                       -
  -      A20         ALL    Ultra 450                      -
  -      A16         ALL    Ultra 30                       -
  -      A14         ALL    Ultra 2                        -
  -      E6000       ALL    Ultra Enterprise 6000          -
  -      E5000       ALL    Ultra Enterprise 5000          -
  -      E4000       ALL    Ultra Enterprise 4000          -
  -      E3000       ALL    Ultra Enterprise 3000          -
  -      A12         ALL    Ultra 1E                       -
  -      A11         ALL    Ultra 1                        -
  -      A22         ALL    Ultra 10                       -
  -      A21         ALL    Ultra 5                        -

  
X-Options Affected
------------------
X2248A     -         -      480Mhz UltraSPARC II Module 8MB Cache   -
X2244A     -         -      400Mhz UltraSPARC II Module 4MB Cache   -
X1994A     -         -      400Mhz UltraSPARC II Module 2MB Cache   -
X2240A     -         -      300MHz UltraSPARC II Module 2MB Cache   -
X2230A     -         -      250MHz UltraSPARC II Module 1MB Cache   -
X1995A     -         -      450Mhz UltraSPARC II Module 4MB Cache   -
X1997A     -         -      440Mhz UltraSPARC II Module 4MB Cache   -
X2580A     -         -      400MHz UltraSPARC II Module 8MB cache   -
X2570A     -         -      400MHz UltraSPARC II Module 4MB cache   -
X1993A     -         -      400Mhz UltraSPARC II Module 2MB Cache   -
X1992A     -         -      360Mhz UltraSPARC II Module 4MB Cache   -
X2560A     -         -      336MHz UltraSPARC II Module 4MB Cache   -
X1991A     -         -      300Mhz UltraSPARC II Module 1MB Cache   -
X2550A     -         -      250MHz UltraSPARC II Module 4MB Cache   -
X1990A     -         -      250Mhz UltraSPARC II Module 1MB Cache   -
X2530A     -         -      250MHz UltraSPARC II Module 1MB Cache   -
X1188A     -         -      200MHz UltraSPARC I Module 1MB Cache    -
X2510A     -         -      167MHz UltraSPARC I Module 1MB Cache    -
X1187A     -         -      167MHz UltraSPARC I Module .5MB Cache   -
X2500A     -         -      167MHz UltraSPARC I Module .5MB Cache   -


PART NUMBERS AFFECTED:

Part Number   Description                              Model
-----------   -----------                              -----
501-5729-0X   480 MHz UltraSPARC II Module 8MB Cache     -
501-5344-0X   450 MHz UltraSPARC II Module 4MB Cache     -
501-5539-0X   450 MHz UltraSPARC II Module 4MB Cache     -
501-5682-0X   440 MHz UltraSPARC II Module 4MB Cache     -
501-5235-0X   400 MHz UltraSPARC II Module 8MB Cache     -
501-5661-0X   400 MHz UltraSPARC II Module 8MB Cache     -
501-5762-0X   400 MHz UltraSPARC II Module 8MB Cache     -
501-4995-0X   400 MHz UltraSPARC II Module 4MB Cache     -
501-5239-0X   400 MHz UltraSPARC II Module 4MB Cache     -
501-5420-0X   400 MHz UltraSPARC II Module 4MB Cache     -
501-5425-0X   400 MHz UltraSPARC II Module 4MB Cache     -
501-5446-0X   400 MHz UltraSPARC II Module 4MB Cache     -
501-5500-0X   400 MHz UltraSPARC II Module 4MB Cache     -
501-5585-0X   400 MHz UltraSPARC II Module 4MB Cache     -
501-5237-0X   400 MHz UltraSPARC II Module 2MB Cache     -
501-5445-0X   400 MHz UltraSPARC II Module 2MB Cache     -
501-5541-0X   400 MHz UltraSPARC II Module 2MB Cache     -
501-5545-0X   400 MHz UltraSPARC II Module 2MB Cache     -
501-4781-0X   360 MHz UltraSPARC II Module 4MB Cache     -
501-5129-0X   360 MHz UltraSPARC II Module 4MB Cache     -
501-5552-0X   360 MHz UltraSPARC II Module 4MB Cache     -
501-4363-0X   336 MHz UltraSPARC II Module 4MB Cache     -
501-4196-0X   300 MHz UltraSPARC II Module 2MB Cache     -
501-4849-0X   300 MHz UltraSPARC II Module 2MB Cache     -
501-4249-0X   250 MHz UltraSPARC II Module 4MB Cache     -
501-4836-0X   250 MHz UltraSPARC II Module 4MB Cache     -
501-4178-0X   250 MHz UltraSPARC II Module 1MB Cache     -
501-4278-0X   250 MHz UltraSPARC II Module 1MB Cache     -
501-4857-0X   250 MHz UltraSPARC II Module 1MB Cache     -

501-3041-0X   200 MHz UltraSPARC I Module 1MB Cache      -
501-4791-0X   200 MHz UltraSPARC I Module 1MB Cache      -
501-2959-0X   167 MHz UltraSPARC I Module 1MB Cache      -
501-2702-03   167 MHz UltraSPARC I Module .5MB Cache     -
501-2941-0X   167 MHz UltraSPARC I Module .5MB Cache     -
501-2942-0X   167 MHz UltraSPARC I Module .5MB Cache     -

501-5149-0X   440 MHz UltraSPARC IIi Module 2MB Cache    -
501-5740-0X   400 MHz UltraSPARC IIi Module 2MB Cache    -
501-5741-0X   400 MHz UltraSPARC IIi Module 2MB Cache    -
501-5148-0X   360 MHz UltraSPARC IIi Module 256KB Cache  -
501-5222-0X   360 MHz UltraSPARC IIi Module 2MB Cache    -
501-5090-0X   333 MHz UltraSPARC IIi Module 2MB Cache    -
501-5568-0X   333 MHz UltraSPARC IIi Module 2MB Cache    -
501-4379-0X   300 MHz UltraSPARC IIi Module 512KB Cache  -
501-5040-0X   300 MHz UltraSPARC IIi Module 512KB Cache  -
501-4477-0X   270 MHz UltraSPARC IIi Module 256KB Cache  -
501-5039-0X   270 MHz UltraSPARC IIi Module 256KB Cache  -

(SCSI Devices)
Type  Vendor  Model    Serial Number(Min)  Serial Number(Max)  Firmware
----  ------  -------  ------------------  ------------------  --------
N/A

REFERENCES:

FIN: I0570-3
FIN: I0593-1
Sun Alert: SA 24669 - Possible WAIT_MBOX_DONE System Panics With Recent        
                      Kernel Update Patches
DOC: 806-5118-13  Best Practices Guide Addressing: E-cache Parity Errors
PatchId: 103640-34  Kernel Patch (Solaris 2.5.1)
PatchId: 105181-23  Kernel Patch (Solaris 2.6)
PatchId: 106541-13  Kernel Patch (Solaris 7)
PatchId: 108528-04  Kernel Patch (Solaris 8)
PatchId: 110151-01  SunMC 2.1 FCS Patch (Solaris 2.6)
PatchId: 110152-01  SunMC 2.1 L10N Patch (Solaris 2.6)
PatchId: 110094-01  SunMC 2.1.1 FCS Patch (Solaris 2.6)
PatchId: 103346-26  Exx00 flashprom update
URL: http://bestpractices.central/
URL: http://cte-www.uk/cgi-bin/afsr/afsr.pl
URL: http://cte-www.eng/cgi-bin/afsr/afsr.pl


PROBLEM DESCRIPTION: 

Solaris Kernel patches are available (see "Features Table" below for
availability details) that provide improved handling and reduction of CPU,
Ecache, and main memory errors in systems using UltraSPARC-I, -II, -IIi,
and -IIe processors.   All customers on Solaris 2.5.1, 2.6, 7 and 8 are
encouraged to consider upgrading to these kernel patches as they become
available.

Table Of Contents
*****************

    Kernel Patch Features Overview
    
       Cache Scrubber
       Improved Error Handling
       Improved Error Messages
       Performance Considerations

    Kernel Patch Features Details
        
       Features Table
       Details on the Cache Scrubber
       Errors and Events
       Details on Improved Error Handling
       Details on Improved Error Messages
          Messages that identify the type and source of an error
          Messages that supply a cache line or memory dump
          Messages from the kernel error recovery code
          Messages that indicate the disposition of an error
       Error Messages Examples
          EDP Event - Ecache Data Parity Event
          WP Event - Writeback Data Parity Error
          CP Event - Copyout Data Parity Error
          UE Event - Uncorrectable Memory Error
          BERR Event - Bus Error
          CE Event - Correctable Memory Error

    Starfire Specific
        
       Arbstop
       Recordstop
       DTag Considerations


Kernel Patch Features Overview
******************************
               
With the patches listed below, one or more of the following features
become available in the Solaris operating system (see "Features Table"
below to determine the features delivered with each patch):

 1. Cache Scrubber
    ==============

    To reduce the likelihood of Ecache Data, Writeback and CopyOut
    Parity errors, a "Cache Scrubber" has been implemented in the
    Solaris Kernel that periodically flushes modified cache lines out
    to main memory and invalidates cache lines that have not been
    modified.  By reducing the likelihood that an otherwise nonfatal
    error in the Ecache will result in a system failure, this procedure
    improves the system's reliability.

 2. Improved Error Handling
    =======================

    Each error reported by the CPU is now evaluated to determine
    whether it is fatal to the operating system, only fatal to a
    user process, or of no immediate consequence.  Fatal errors in
    the kernel result in a system panic, as they did before.  Fatal
    errors within user space will now cause the machine to reboot
    instead of panic, allowing file systems to be fully synched and
    also preventing the creation of unnecessary kernel core files.
    Events that do not affect the integrity of either the kernel or
    user processes are logged, but otherwise ignored.

    Because UltraSPARC-IIi and UltraSPARC-IIe use simplified error
    reporting logic (as compared to UltraSPARC-II), the error
    handling behavior for UltraSPARC-IIi and UltraSPARC-IIe based
    systems has not been changed.  Those systems will still panic
    on most CPU, Ecache, or uncorrectable memory errors.

 3. Improved Error Messages
    =======================

    The CPU, Ecache, and memory error messages have been improved
    to be more accurate and complete.  Text descriptions have been
    rewritten to emphasize the important parameters associated with
    each event.  Also, the logic for reporting hardware errors has
    changed to ensure that error events are reported accurately,
    completely, and in the order they occurred.  These new error
    messages will make it easier to determine the CPU that has
    encountered an error.

    There are related patches to SunMC so that it will recognize
    the improved error messages; without them, the management
    console will under-report the occurrence of corrected main
    memory errors.  See "Corrective Action" item 3, below, for a list
    of the related patches.

Performance Considerations
==========================

The above changes can slightly degrade system performance.  The primary
cause of this is the Improved Error Handling, which required inserting
membars in the kernel to properly isolate user-encountered errors from
kernel-encountered ones.  (A membar is an UltraSPARC instruction that
stalls the CPU pipeline until all outstanding memory operations have
completed, and any errors that may result from them have been reported.
Any errors reported after the execution of a membar completes can only
result from instructions that follow the membar in the instruction stream.)
In addition, the Cache Scrubber consumes 0.4% of CPU cycles in scanning
the Ecache.

Measurements using industry standard benchmarks have shown a decrease
in TPC-C performance of about 2% and in one kenbus configuration a
decrease in performance of about 5%.  Performance degradation of most
of the other benchmarks in the performance suite was indistinguishable
from measurement noise.  We do not expect most customers to notice
significant performance degradation.


Kernel Patch Features Details
*****************************

Features Table
==============

The following list gives details about the features delivered with each
of the patches:

     Solaris 2.5.1 with patch 103640-34 will introduce:
     - Cache Scrubber

     Solaris 2.6 with patch 105181-23 will introduce:
     - Cache Scrubber
     - Improved Error Messages
     - Improved Error Handling [1]

     Solaris 7 with patch 106541-13 (est. Nov/10/2000) will introduce:
     - Cache Scrubber
     - Improved Error Messages
     - Improved Error Handling [1]

     Solaris 8 with patch 108528-04 (est. Oct/27/2000) will introduce:
     - Cache Scrubber             Only for UltraSPARC-I, -II, -IIi
     - Improved Error Messages    Only for UltraSPARC-I, -II, -IIi, -IIe
     - Improved Error Handling [1] Only for UltraSPARC-I, -II

     Solaris 8 Update 3 (est. Dec/2000) will introduce:
     - Cache Scrubber             Only for UltraSPARC-I, -II, -IIi, -IIe
     - Improved Error Messages    Only for UltraSPARC-I, -II, -IIi, -IIe
     - Improved Error Handling [1] Only for UltraSPARC-I, -II

NOTE [1]: Due to hardware limitations there is no improved error handling
          for UltraSPARC-IIi and UltraSPARC-IIe based systems.


Details on the Cache Scrubber
=============================

The cache scrubber reduces the likelihood of EDP, WP, and CP events by
shortening the data lifetime in the Ecache, and by eliminating parity
errors where possible.  (See "Errors and Events" below for an
explanation
of the EDP, WP, and CP event types.)

The cache scrubber is enabled by default.  It scans the entire Ecache
of every CPU in the system once every ten seconds.

On an idle CPU, it scrubs all clean lines (lines that are identical to
the system memory from where they came), and dirty lines (lines that
have newer data than the system memory from where they came) that have
good parity.  This reduces the lifetime of data in the Ecache on an
idle CPU, reducing the likelihood that a parity error will affect
critical system or user data.

On a busy CPU, it only scrubs clean lines with bad parity (which might
otherwise lead to EDP or CP events).  Clean lines with good parity and
dirty lines are left in the Ecache so as to not adversely impact system
performance.

The cache scrubber never scrubs dirty lines with bad parity to avoid
causing WP events.  These bad lines could get overwritten by the
program using them before they are accessed or flushed, thereby
eliminating a bad parity event from occurring at all.  (This is
sometimes referred to as the natural scrubbing behavior of a busy
system.)


Errors and Events
=================

UltraSPARC processors can detect errors that are reported in the
following types of events (as detailed in the UltraSPARC-I/II User's
Manual, P/N 802-7220-02):

ETP   A parity error was detected by the CPU when reading from the
      Ecache Tag SRAM.  This is a fatal error because system coherency
      has been lost.  The system will reset (POR) and Starfire domains
      will arbstop (UPA Fatal error).  No Solaris error message will be
      generated.

EDP   A parity error was detected by the CPU when reading from the
      Ecache Data SRAM on a cache hit.

LDP   A parity error was detected by the CPU while reading main
      memory through its Ultra Data Buffer (UDB) chip on an Ecache
      miss.  Note that the Ecache itself is not involved.  This can occur
      when the CPU is reading non-cacheable data (for example, a frame
      buffer or I/O device), or when filling a line of cache from main
      memory.

WP    A parity error was detected by one of the UDB chips while data
      was being written back from the Ecache into main memory.  The UDB
      chips convert the data with bad parity into data with bad ECC, so
      that a subsequent access to the same physical address will result
      in a UE.  (See UE below.)  (The conversion of a parity error to a
      latent UE does not occur on either UltraSPARC-IIi or -IIe, which
      is one of the reasons why improved error handling is not
      available on those processors.)

CP    A parity error was detected during a copyout transaction; that
      is, a data transfer from one CPU's Ecache to another CPU.  This
      error is detected by the UDB chips of the providing CPU,
      resulting in the CP event.  The providing CPU's UDB chips convert
      the data with bad parity to data with bad ECC, so that the UDBs
      of the receiving CPU will report a UE event.  (See UE below.)

UE    An uncorrectable memory error has occurred.  This event refers to
      an error in the main system memory, reported by the system data
      bus on a read access.  The underlying source of this error could
      be main memory, another CPU module (see CP above), or another UPA
      device (for example, the I/O controller).  The UDB chips detect
      this error.

CE    A correctable error was detected when reading from main memory,
      or when reading from another CPU's UDB chips.  The data read has
      been corrected and valid data is given to the CPU and the CPU's
      Ecache.  This error is detected by the UDB chips.

BERR  A bus error has occurred during an attempt to read from a memory
      address.  Either there is no device at that address, or the
      device at that address has returned a bus error.  Therefore, bus
      errors are caused by a programming error or by a corrupted or
      defective device.

TO    A bus timeout was encountered during an attempt to read from a
      memory address.  Too much time has elapsed waiting for a device
      at that address to respond.


Details on Improved Error Handling
==================================

Any of the above mentioned errors can occur in kernel instruction
space, kernel data space, user instruction space, user data space, or
when the kernel reads or writes user data (as in copyin).  Depending on
these different states, the operating system will react differently so
as to maximize system availability.

On EDP, LDP, CP, UE, BERR, and TO events, the system will panic if the
affected data is in kernel space or if the error occurs while the CPU is
at a trap level greater than zero.  Otherwise, the process that caused the
error will be killed immediately (sent SIGKILL) and the system will be
rebooted (as if a privileged user had entered "init 6").  [2]

On WP events, an error is reported, and the memory scrubber is notified
to scan all of system memory for the latent UE the hardware has written
to memory (see below for the behavior of the memory scrubber on
encountering UE events).  If some CPU later attempts to read this
location (other than on behalf of the memory scrubber), a UE event will
occur.  Hence, when a UE event is encountered, it is recommended that the
log be checked for an earlier WP event that may have in fact caused the
UE event.

If the memory scrubber detects a UE event the system will neither panic
nor reboot but trigger a recovery mechanism instead.  If the page
containing the corrupted data is not in use, it will be retired and the
error will be cleared.  If it is in use, it will be marked for
retirement and clearing if and when it is no longer in use.

NOTE [2]: An active SC2.X cluster node will panic with a "Failfast
          timeout" (usually with "Device closed while Armed") when
          rebooted.  It is therefore useful to check the system messages
          for EDP, LDP, CP, UE, BERR, and TO events while encountering
          "Failfast timeout" panics.


Details on Improved Error Messages
==================================

For each error that is detected, the kernel generates an individual
report.  This is a major change; previously, some errors would hide other
errors, and some errors were combined into a single message.  The report
typically consists of several error messages.  Each message [3] contains
an AFT ("Asynchronous Fault Trap") tag that eases filtering, and an
errID
code that associates all of the messages emitted for the same event.  The
errID is a 64-bit code that corresponds to a specific set of error bits in
the Asynchronous Fault Status Register (AFSR) at a specific instance in
time; the value has no intrinsic meaning.

Each message may be longer than one physical line; long messages are
folded using embedded newlines.  Each folded line begins with four
space characters.

NOTE [3]: Because of the introduction of improved error messages, any tool
          using the affected error messages may have to be modified.
          Neither the format nor the content of kernel error messages are
          committed interfaces, and both may change without notice.
          Users (both internal and external) who rely on the exact format
          and/or content do so at their own risk.

The error messages can be grouped into four categories:


   Category 1: Messages that identify the type and source of an error
   ------------------------------------------------------------------

   Example:

      WARNING: [AFT1] EDP event on CPU1 Instruction access at TL=0, errID 
0x0000ad88.6cd9989f
         AFSR 0x00000000.80408000<PRIV,EDP> AFAR 0x00000000.0f0c8080
         AFSR.PSYND 0x8000(Score 95) AFSR.ETS 0x00 FAULT_PC 0x780b481c
         UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00

   Either the [AFT0] tag (for correctable errors) or the [AFT1] tag (for
   uncorrectable errors) is present in the message.  An "errID" field
   appears at the end of the first line of the message.  Messages from
   this category are displayed on the console and collected in the log
   file.  [4]

   To aid diagnosis of an Ecache-related error, especially if multiple
   components are involved, a heuristic algorithm has been included that
   automates analysis of the P_SYND bytes.  Every component reporting a
   failure has its AFSR decoded and a score ranging from 5 to 95 is
   assigned ("Score 95" in the above example).

   The Score indicates the likelihood that this component was the original
   source of the bad parity.  The higher the value, the higher the
   likelihood that this component was the original source.

NOTE [4]: This is the default behavior.  The /etc/system setting 
          report_ce_console is no longer referenced and should therefore
          be removed.


   Category 2: Messages that supply a cache line or memory dump
   ------------------------------------------------------------

   Example:

      [AFT2] errID 0x0000ad88.6cd9989f PA 0x00000000.0f0c8080 E$tag 
0x00000000.0bc001e1 E$State: Modified E$parity 0x05
      [AFT2] E$Data (0x00): 0xffffffff.beefface *Bad* PSYND=0x8000
      [AFT2] E$Data (0x08): 0x00000000.00000000
      [AFT2] E$Data (0x10): 0x6d656d6d.6f727920
      [AFT2] E$Data (0x18): 0x6572726f.7220696e
      [AFT2] E$Data (0x20): 0x6a656374.6f720000
      [AFT2] E$Data (0x28): 0x6d656d74.65737420
      [AFT2] E$Data (0x30): 0x6d757465.780059f8
      [AFT2] E$Data (0x38): 0x00000300.00c11000
      [AFT2] Event PA displayed in AFAR was derived from E$Tag

   Messages from this category are targeted for Sun Microsystems support
   staff to be used in backline diagnosis and for statistics.
   
   The [AFT2] tag is always present in these messages.  The "errID"
field
   appears at the beginning of the first line of the message.  Messages
   from this category are by default only collected in the log file.


   Category 3: Messages from the kernel error recovery code
   --------------------------------------------------------

   Example:

      [AFT3] errID 0x00000058.0d0dc830 Above Error detected by protected Kernel

code
          that will try to clear error from system

   Messages from this category supply analysis information from the
   kernel error recovery code, thereby indicating the actions the kernel
   took to contain the error.

   The [AFT3] tag is always present in these messages.  An "errID"
   field appears at the beginning of the first line of the message.
   Messages from this category are by default only collected in the log
   file.


   Category 4: Messages that indicate the disposition of an error
   --------------------------------------------------------------

   Example:

      panic[CPU1]/thread=30000670800: [AFT1] errID 0x00000392.89cbfefc EDP 
Error(s)
         See previous message(s) for details

   Messages from this category state the final handling (like panic or
   reboot) of a previously encountered error.

   Either the [AFT0] tag (for correctable errors) or the [AFT1] tag (for
   uncorrectable errors) is present in the message.  The "errID" field
   appears at the beginning of the first line of the message.  Messages from
   this category are displayed on the console and collected in the log file.


Error Messages Examples
=======================

The following compares previous messages with the new, improved error
messages.  Note that this is not an exhaustive list, but a sampling of
possible messages for each event type.  This also just shows what
appears on the console; the log-only messages are not shown.

Lines are shown exactly as they appear on the console.  If you print
this file, you will need to either use software that wraps long lines,
or print in landscape mode.


EDP Event - Ecache Data Parity Event
------------------------------------

* Solaris 8 Message - Kernel Data:

panic[CPU1]/thread=3000225bcc0: CPU1 Ecache SRAM Data Parity Error: AFSR 
0x00000000.80408000 AFAR 0x00000000.0bd83bd0

* Improved Message - Kernel Data:

WARNING: [AFT1] EDP event on CPU1 Data access at TL=0, errID
0x00000093.6323e6f8
    AFSR 0x00000000.80408000<PRIV,EDP> AFAR 0x00000000.06901980
    AFSR.PSYND 0x8000(Score 95) AFSR.ETS 0x00 Fault_PC 0x78128a84
    UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00

panic[cpu1]/thread=30000ae5000: [AFT1] errID 0x00000093.6323e6f8 EDP Error(s)
    See previous message(s) for details

* Solaris 8 Message - User Data:

panic[CPU3]/thread=30001f4fa00: CPU3 Ecache SRAM Data Parity Error: AFSR 
0x00000000.00400080 AFAR 0x00000000.01820000

* Improved Message - User Data (Reboot):

Aug 16 16:47:20 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] EDP event on CPU3 
Data access at TL=0, errID 0x00000057.d35eff81
Aug 16 16:47:20 thishost     AFSR 0x00000000.00400080<EDP> AFAR 
0x00000000.05e24418
Aug 16 16:47:20 thishost     AFSR.PSYND 0x0080(Score 95) AFSR.ETS 0x00 Fault_PC

0x11ce8
Aug 16 16:47:20 thishost     UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND

0x00
Aug 16 16:47:20 thishost unix: NOTICE: Scheduling clearing of error on page 
0x00000000.05e24000
Aug 16 16:47:20 thishost unix: WARNING: [AFT1] initiating reboot due to above 
error in pid 309 (mtst)
Aug 16 16:47:23 thishost unix: NOTICE: Previously reported error on page 
0x00000000.05e24000 cleared

INIT: New run level: 6
The system is coming down.  Please wait.
System services are now being stopped.
Print services stopped.
Aug 16 16:47:27 thishost syslogd: going down on signal 15
The system is down.
syncing file systems... done
rebooting...
Resetting ...

* Solaris 8 Message - Kernel Data at TL=1:

panic[CPU3]/thread=30001cfabe0:  Async data error at tl1:  AFAR 
0x00000000.0ab8f760 AFSR 0x00000000.80400080

* Improved Message - Kernel Data at TL=1 (Panic):

WARNING: [AFT1] EDP event on CPU3 Data access at TL>0, errID
0x00000111.53a7b8dd
    AFSR 0x00000000.80408000<PRIV,EDP> AFAR 0x00000000.01f47dc0
    AFSR.PSYND 0x8000(Score 95) AFSR.ETS 0x00 Fault_PC 0x1002fe20
    UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00

panic[cpu3]/thread=30000a4e040: [AFT1] errID 0x00000111.53a7b8dd EDP Error(s)
    See previous message(s) for details

* Solaris 8 Message - Kernel Instruction at TL=1:

panic[CPU3]/thread=3000226a140:  Async instruction error at tl1:  AFAR 
0x00000000.0dd55f70 AFSR 0x00000000.80408000

* Improved Message - Kernel Instruction at TL=1 (Panic):

WARNING: [AFT1] EDP event on CPU3 Instruction access at TL>0, errID 
0x00000043.24bfd349
    AFSR 0x00000000.80400800<PRIV,EDP> AFAR 0x00000000.0605c790
    AFSR.PSYND 0x0800(Score 95) AFSR.ETS 0x00 Fault_PC 0x1002fe20
    UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00

panic[cpu3]/thread=30000ad05c0: [AFT1] errID 0x00000043.24bfd349 EDP Error(s)
    See previous message(s) for details


WP Event - Writeback Data Parity Error
--------------------------------------

* Solaris 8 Message:

panic[CPU1]/thread=30001b26640:  CPU1 Ecache Writeback Data Parity Error: AFSR 
0x00000000.00800080 AFAR 0x00000000.0d5010f0

* Improved Message:

Aug 16 16:50:56 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] WP event on CPU1, 
errID 0x0000002b.3c7cd6d9
Aug 16 16:50:56 thishost     AFSR 0x00000000.00800080<WP> AFAR 
0x000001c8.01802800
Aug 16 16:50:56 thishost     AFSR.PSYND 0x0080(Score 95) AFSR.ETS 0x00 Fault_PC

0x11d7c
Aug 16 16:50:56 thishost     UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND

0x00
Aug 16 16:50:56 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] Uncorrectable 
Memory Error on CPU3 Data access at TL=0, errID 0x0000002b.45daae92
Aug 16 16:50:56 thishost     AFSR 0x00000000.80200000<PRIV,UE> AFAR 
0x00000000.03824418
Aug 16 16:50:56 thishost     AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC

0x10023414
Aug 16 16:50:56 thishost     UDBH 0x0000 UDBH.ESYND 0x00 UDBL
0x0203<UE> 
UDBL.ESYND 0x03
Aug 16 16:50:56 thishost     UDBL Syndrome 0x3 Memory Module 190x
Aug 16 16:50:56 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] errID 
0x0000002b.45daae92 Syndrome 0x3 indicates that this may not be a memory module

problem
Aug 16 16:50:56 thishost unix: NOTICE: Scheduling clearing of error on page 
0x00000000.03824000
Aug 16 16:50:58 thishost unix: NOTICE: Previously reported error on page 
0x00000000.03824000 cleared


NOTE: The last message (reporting clearing of the error) may appear much later,
      or may never appear, as the page may never drop out of use.  Also, the
      message reporting scheduling of clearing may occur more than once, as the
      memory scrubber may encounter the particular UE more than once before it
      can be cleared.


CP Event - Copyout Data Parity Error
------------------------------------

* Solaris 8 Message:

panic[CPU3]/thread=2a100105d40:  CPU3 UE Error:  Ecache Copyout on CPU1: AFSR 
0x00000000.01000080 AFAR 0x00000000.06c53090

* Improved Message - Kernel (Panic):

WARNING: [AFT1] Uncorrectable Memory Error on CPU3 Data access at TL=0, errID 
0x0000003a.30aafcba
    AFSR 0x00000000.80200000<PRIV,UE> AFAR 0x00000000.00347dc0
    AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC 0x78067b54
    UDBH 0x0203<UE> UDBH.ESYND 0x03 UDBL 0x0000 UDBL.ESYND 0x00
    UDBH Syndrome 0x3 Memory Module 190x
WARNING: [AFT1] errID 0x0000003a.30aafcba Syndrome 0x3 indicates that this may 
not be a memory module problem
WARNING: [AFT1] CP event on CPU1 (caused Data access error on CPU3), errID 
0x0000003a.30aafcba
    AFSR 0x00000000.01008000<CP> AFAR 0x00000000.00347dc0
    AFSR.PSYND 0x8000(Score 95) AFSR.ETS 0x00
    UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00

panic[cpu3]/thread=2a100157d40: [AFT1] errID 0x0000003a.30aafcba UE Error(s)
    See previous message(s) for details

* Improved Message - User (Reboot):

Aug 16 17:06:44 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] Uncorrectable 
Memory Error on CPU3 Data access at TL=0, errID 0x0000002b.963a3d3c
Aug 16 17:06:44 thishost     AFSR 0x00000000.00200000<UE> AFAR 
0x00000000.00224418
Aug 16 17:06:44 thishost     AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC

0x12380
Aug 16 17:06:44 thishost     UDBH 0x0000 UDBH.ESYND 0x00 UDBL
0x0203<UE> 
UDBL.ESYND 0x03
Aug 16 17:06:44 thishost     UDBL Syndrome 0x3 Memory Module 190x
Aug 16 17:06:44 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] errID 
0x0000002b.963a3d3c Syndrome 0x3 indicates that this may not be a memory module

problem
Aug 16 17:06:44 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] CP event on CPU1 
(caused Data access error on CPU3), errID 0x0000002b.963a3d3c
Aug 16 17:06:44 thishost     AFSR 0x00000000.01000080<CP> AFAR 
0x00000000.00224418
Aug 16 17:06:44 thishost     AFSR.PSYND 0x0080(Score 95) AFSR.ETS 0x00
Aug 16 17:06:44 thishost     UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND

0x00
Aug 16 17:06:44 thishost unix: NOTICE: Scheduling clearing of error on page 
0x00000000.00224000
Aug 16 17:06:44 thishost unix: WARNING: [AFT1] initiating reboot due to above 
error in pid 304 (mtst)
Aug 16 17:06:46 thishost unix: NOTICE: Previously reported error on page 
0x00000000.00224000 cleared

INIT: New run level: 6
The system is coming down.  Please wait.
System services are now being stopped.
Print services stopped.
Aug 16 17:06:50 thishost syslogd: going down on signal 15
The system is down.
syncing file systems... done
rebooting...
Resetting ...


NOTE: Due to a coding error, early versions of some of the patches produce
      the string "CP Error" instead of "CP event"; programs
that parse the
      messages must be prepared to deal with both.


UE Event - Uncorrectable Memory Error
-------------------------------------

* Solaris 8 Message - CPU Reference to Memory:

panic[CPU1]/thread=2a1000R7dd40: UE Error: AFSR 0x00000000.80200000 AFAR 
0x00000000.089cd740 Id 0 Inst 0 MemMod U0501 U0401

* Improved Message - CPU Reference to Memory - Kernel (Panic):

WARNING: [AFT1] Uncorrectable Memory Error on CPU1 Instruction access at TL=0, 
errID 0x0000004f.818d9280
    AFSR 0x00000000.80200000<PRIV,UE> AFAR 0x00000000.0685c7a0
    AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC 0x7815c7a0
    UDBH 0x0203<UE> UDBH.ESYND 0x03 UDBL 0x0000 UDBL.ESYND 0x00
    UDBH Syndrome 0x3 Memory Module 190x
WARNING: [AFT1] errID 0x0000004f.818d9280 Syndrome 0x3 indicates that this may 
not be a memory module problem

panic[cpu1]/thread=30000ad6320: [AFT1] errID 0x0000004f.818d9280 UE Error(s)
    See previous message(s) for details

* Improved Message - CPU Reference to Memory - User (Reboot):

Aug 16 17:03:04 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] Uncorrectable 
Memory Error on CPU1 Instruction access at TL=0, errID 0x00000032.593d8229
Aug 16 17:03:04 thishost     AFSR 0x00000000.00200000<UE> AFAR 
0x00000000.04921bf0
Aug 16 17:03:04 thishost     AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC

0x11bf0
Aug 16 17:03:04 thishost     UDBH 0x0203<UE> UDBH.ESYND 0x03 UDBL
0x0000 
UDBL.ESYND 0x00
Aug 16 17:03:04 thishost     UDBH Syndrome 0x3 Memory Module 190x
Aug 16 17:03:04 thishost SUNW,UltraSPARC-II: WARNING: [AFT1] errID 
0x00000032.593d8229 Syndrome 0x3 indicates that this may not be a memory module

problem
Aug 16 17:03:04 thishost unix: NOTICE: Scheduling clearing of error on page 
0x00000000.04920000
Aug 16 17:03:07 thishost unix: NOTICE: Previously reported error on page 
0x00000000.04920000 cleared
Aug 16 17:03:07 thishost unix: WARNING: [AFT1] initiating reboot due to above 
error in pid 304 (mtst)

INIT: New run level: 6
The system is coming down.  Please wait.
System services are now being stopped.
Print services stopped.
Aug 16 17:03:13 thishost syslogd: going down on signal 15
The system is down.
syncing file systems... done
rebooting...
Resetting ...

* Solaris 8 Message - SBus I/O Reference to Memory:

panic[CPU1]/thread=2a10007dd40:  SBus0 UE Primary Error DMA read:  AFSR 
0x40001be0.00000000 AFAR 0x00000000.02818000 MemMod U0501 U0401 Id 31

* Improved Message - SBus I/O Reference to Memory:

WARNING: SBus0 UE Primary Error DMA read: AFSR 0x40001be0.00000000 AFAR 
0x00000000.0d25c000 MemMod U0501 U0401 Id 31

panic[cpu0]/thread=2a10007dd40: Fatal Sbus0 UE Error


BERR Event - Bus Error
----------------------

* Solaris 8 Message:

panic[CPU1]/thread=30000d2c300:  CPU1 Privileged Bus Error:  AFSR 
0x00000000.84000000 AFAR 0x00000000.03422000

* Improved Message - Kernel (Panic):

WARNING: [AFT1] Bus Error on System Bus in privileged mode from CPU1 Data
access 
at TL=0, errID 0x0000002c.52b3d2c8
    AFSR 0x00000000.84000000<PRIV,BERR> AFAR 0x00000000.05224410
    AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC 0x780671a4
    UDBH 0x0000 UDBH.ESYND 0x00 UDBL 0x0000 UDBL.ESYND 0x00

panic[cpu1]/thread=30000b06080: [AFT1] errID 0x0000002c.52b3d2c8 BERR Error(s)
    See previous message(s) for details


CE Event - Correctable Memory Error
-----------------------------------

* Solaris 8 Message:

May  8 14:35:30 thishost SUNW,UltraSPARC-II: CPU1 CE Error: AFSR 
0x00000000.00100000 AFAR 0x00000000.8abb5a00 UDBH Syndrome 0x85 MemMod U0904
May  8 14:35:30 thishost SUNW,UltraSPARC-II:     ECC Data Bit 63 was corrected
May  8 14:35:30 thishost unix: Softerror: Intermittent ECC Memory Error, U0904

* Improved Message:

Aug 16 16:34:48 thishost SUNW,UltraSPARC-II: [AFT0] Corrected Memory Error on 
CPU1, errID 0x00000036.629edc25
Aug 16 16:34:48 thishost     AFSR 0x00000000.00100000<CE> AFAR 
0x00000000.00347dc0
Aug 16 16:34:48 thishost     AFSR.PSYND 0x0000(Score 05) AFSR.ETS 0x00 Fault_PC

0x1002fe20
Aug 16 16:34:48 thishost     UDBH Syndrome 0x85 Memory Module 1904
Aug 16 16:34:48 thishost SUNW,UltraSPARC-II: [AFT0] errID 0x00000036.629edc25 
Corrected Memory Error on 1904 is Intermittent
Aug 16 16:34:48 thishost SUNW,UltraSPARC-II: [AFT0] errID 0x00000036.629edc25 
ECC Data Bit 63 was in error and corrected


Starfire Specific
*****************

Arbstop
=======

STag Parity Errors on an E10000 almost always result in a "UPA Fatal
Error"
Arbstop dump.  Although these can also be caused by poor VCore voltage
power pucks on a System Board, error trends have shown that these errors
are generally an "ETP Event", caused by the CPU identified in the
Arbstop
dump file.

Recordstop
==========

Recordstop dump files will be generated anytime data is transferred
through the crossbar of the Starfire centerplane.  This means that
a recordstop is likely to occur during WP, CP, and LDP events.
As always, the "psi" reported error is an extremely strong indication
of the source of the "UE ECC Error" as reported in the wfail output
of redx.  The reporting XDB can be associated with one or two CPUs,
but which CPU actually sourced the data cannot be determined from the
recordstop itself, unless only one of the two possible CPUs are
present at the time of the error.  In these cases, the syndrome of 03
is always present in the XDB Error report.  Use the recordstop dump
to complement and confirm information provided by Solaris in the
message and console logs.  Expect Solaris to report a relatively
high "score" against one of the CPUs attached to the reporting XDB
within the AFT messages previously described in this document.

Note of Caution: Conversely, an XDB could report an "ldat" error with
a syndrome of 03, which includes the same data pattern and xmux_par
values.  In these cases, the XDB that reports the "ldat" error is the
XDB for the "victim" CPU in a copyback (CP) event.  In essence, an
"ldat" error reported by an XDB will actually prove that the CPUs it
services are victims of another CPU's Cache Parity Error, and therefore
can be used to exonerate the attached CPUs.

These XDB reported "ldat" are extremely rare, but can occur due to
other variables in a Starfire platform.  These errors may or may not
be reported with a complementary "psi" error, but the XDBs will
continue
to report a "UE ECC" error in the wfail output, along with a syndrome
value of 0x03.  For these events, the "ldat" error exonerates the
attached CPUs, and might be traced back through the Centerplane X-Bar
to the System Board where the data originated from.  However, it is
likely that the error will not be traceable back to a CPU on the
sourcing System Board, unless a corresponding "psi"-side error
is reported by an XDB from that System Board.

For all XDB-reported "ldat" errors, expect Solaris to report a low
"score" against one of the CPUs attached to the reporting XDB within
the AFT messages previously described in this document.

DTag Considerations
===================

A rumor has been circulating that this patch increases the rate of
DTag parity errors on E10000 systems. That rumor is false.

The development team observed two customer systems (out of 30) using
the USER-level scrubber that experienced an increased rate of DTag
parity errors.  It was determined that the combination of the USER-
level scrubber plus a certain customer-dependent application mix
(which we have yet to characterize) tickles marginal E10000 boards into
producing DTag parity errors.  The KERNEL level scrubber that is
contained in the patch uses a completely different algorithm, and does
not have this tickling effect.  There have been no reports of increased
DTag parity errors with the KERNEL scrubber.

If a customer experiences a DTag parity error with or without the USER
or KERNEL level scrubber, standard replacement policies apply.



IMPLEMENTATION: (T) (R) (Proactive vs Reactive) 
 
         ---
        |   |   MANDATORY (Fully Pro-Active)
         ---    
         
  
         ---
        | X |   CONTROLLED PRO-ACTIVE (per Sun Geo Plan) 
         --- 
         
                                
         ---
        |   |   REACTIVE (As Required)
         ---
         

CORRECTIVE ACTION: 

The following recommendations are provided as a guide for authorized
Enterprise Services Field Representatives and Enterprise Customers on
UltraSPARC based platforms running Solaris versions 2.5.1, 2.6, 7, and 8;

1.  If this system is running the user level cache scrubber, remove it.
    To determine whether a system is running the user level cache scrubber,
    enter the command:

        /usr/lib/cachescrubber -V
    
    If the response is "Command not found," then the user level
scrubber is
    not installed on this system. If the response is a message containing
    the current version of the user level scrubber, then the user level
    scrubber is installed on this system and must be removed. 

    To remove the user level cache scrubber, follow the removal procedure
    as described in the README file for the user level cache scrubber. The
    removal procedure varies for different versions of the scrubber.

2.  Apply the appropriate Kernel Patch for the version of Solaris per the
    chart below.

     PatchId      Solaris Release    Availability (estimated)
     ---------    ---------------    ------------
     103640-34    Solaris 2.5.1      Now
     105181-23    Solaris 2.6        Now
     106541-13    Solaris 7          Nov/10/2000
     108528-04    Solaris 8          Nov/15/2000

3.  If the system is running SunMC, apply the appropriate SunMC patches,
    per the table below.  This is necessary to maintain SunMC's ability to
    report corrected memory errors.  If the system is running SyMon, it will
    be necessary to upgrade to SunMC and then apply the appropriate patch.
    
                                Solaris Release
                                ---------------
                           2.6          7  [6]       8  [6]
                        ---------    ---------    ---------
    SunMC 2.1   FCS     110151-01    110213-01    110216-01
    SunMC 2.1   L10N    110152-01    110214-01    110217-01
    SunMC 2.1.1 FCS     110094-01    110215-01    110218-01

    (SunMC patches are not needed for 2.5.1 as the 2.5.1 patch does not
    contain improved error messages.)

NOTE [6]: The patches for Solaris 7 and Solaris 8 are not yet available.

4.  To ensure proper preservation of system error messages across a panic or
    or reboot:

    - E3x00, E4x00, E5x00, E6x00 systems must apply OBP patch 103346-25
      (or higher).

    - E10000 systems must activate netcon logging, as described in
      FIN I0593-1.

5.  If the system has operations personnel that have been trained to respond
    to the older system error and panic messages, these personnel must be
    notified, and become familiar with, the changed error messages that are
    described in this document (see "Details on Improved Error
Messages"
    and "Error Messages Examples" sections, above).  See the comment on
    "Customer White Paper," below.

6.  If the system employs custom software tools that extract system messages
    from kernel core dumps or log files (like /var/adm/messages), these tools
    will have to be modified to recognize the new messages.  See the comment
    on "Customer White Paper," below.

7.  For FRU replacement guidelines, refer to the Best Practices Guide:

        http://bestpractices.central.sun.com/BestPrac_Sept11_2000.ps

8.  For those systems where the appropriate above listed Kernel Patches
    have not yet been applied, FINI0570-3 will remain the reference
    document for troubleshooting Ecache errors.

9.  A mailing list has been set up to address the KJP.  Any bugs filed
    against the cache scrubber or error recovery mechanisms should include
    this mailing list on the interest list of the bug.  In addition, any
    unexplained system behavior changes should be directed to this mailing
    list as well.

        [email protected]


COMMENTS:

User Level Cache Scrubber
-------------------------

The user level cache scrubber was an early process-level implementation of
the cache scrubber, deployed by a small number of customers as an interim
measure.  Its functionality is superceded by that of the kernel level
cache scrubber that is provided in the Kernel Patch.  Running the user
level scrubber on a system that has the Kernel Patch applied may degrade
performance and will defeat some of the functionality of the kernel cache
scrubber.  For this reason, the user level scrubber should be removed
(uninstalled) prior to applying the Kernel Patch.

Systems that are currently running the user level cache scrubber and are
not applying the Kernel Patch (for example, on platforms where the Kernel
Patch is not yet available) should continue to run the user level
scrubber.  The user level scrubber should be removed only in preparation
for installing the Kernel Patch.


AFSR Decoder Tool
-----------------

The SPG-CTE AFSR decoder is available at the following URLs:

    http://cte-www.uk/cgi-bin/afsr/afsr.pl
    http://cte-www.eng/cgi-bin/afsr/afsr.pl

An equivalent output as provided by AFSR decode is now immediately
available in the [AFTn] messages.  However, the tool remains useful while
troubleshooting I/O related problems (DVMA transaction) and it has been
updated to reflect the Score parameter.  (See "Details on Improved Error
Messages" Category 1, above, for details on Score.)


Customer White Paper
--------------------

A customer white paper is being written that describes the improved error
handling capabilities of the Solaris Operating System.  The document will
assist customer operations personnel and monitoring tools developers who
need to become familiar with the new error messages.


--------------------------------------------------------------------------
Implementation Footnote:

i)   In case of MANDATORY FINs, Enterprise Services will attempt to    
     contact all affected customers to recommend implementation of 
     the FIN. 
   
ii)  For CONTROLLED PROACTIVE FINs, Enterprise Services mission critical    
     support teams will recommend implementation of the FIN  (to their  
     respective accounts), at the convenience of the customer. 

iii) For REACTIVE FINs, Enterprise Services will implement the FIN as the   
     need arises.
--------------------------------------------------------------------------

All released FINs and FCOs can be accessed using your favorite network 
browser as follows:
 
SunWeb Access:
-------------- 
* Access the top level URL of http://sdpsweb.ebay/FIN_FCO/

* From there, select the appropriate link to query or browse the FIN and
  FCO Homepage collections.
 
SunSolve Online Access:
-----------------------
* Access the SunSolve Online URL at http://sunsolve.Corp/

* From there, select the appropriate link to browse the FIN or FCO index.

Supporting Documents:
---------------------
* Supporting documents for FIN/FCOs can be found on Edist.  Edist can be 
  accessed internally at the following URL: http://edist.corp/.
  
* From there, follow the hyperlink path of "Enterprise Services Documenta- 
  tion" and click on "FIN & FCO attachments", then choose the
appropriate   
  folder, FIN or FCO.  This will display supporting directories/files for 
  FINs or FCOs.
   
Internet Access:
----------------
* Access the top level URL of https://infoserver.Sun.COM
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to [email protected]
--------------------------------------------------------------------------



------------- End Forwarded Message -------------



Questions or comments? [email protected]
Copyright (c) 1997-8 Sun Microsystems, Inc.